hernani/taler-monitoring
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
taler-monitoring/README.md
Hernâni Marques 757090cbff
release 1.9.0: monpages top/bottom content markers + surface required 
Require meaningful monitoring HTML: sticky-bar/generated at top, footer
or taler-mon:bottom at end; ERROR if body is progress-only. Mid-run
(incomplete meta or progress without complete foot) relaxes bottom only.
GOA always checks taler-monitoring-surface. Generator emits taler-mon
markers and taler-mon-page complete|incomplete meta.
2026-07-19 00:58:09 +02:00

423 lines
19 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# taler-monitoring
Checks for **GNU Taler** stacks (bank, exchange, merchant; optional surface inventory).
| Mode | What |
|------|------|
| **Outside-only** | Public HTTPS from anywhere — no SSH, no host-agent. Phases such as `urls`, `monpages`, `surface`, … |
| **Inside / local** | Where you operate the stack: podman/SSH (`inside`, `versions`, `e2e`, `ladder`, …) |
| **Host-agent + HTML** | Optional: timers run the suite and publish sticky-bar HTML (see simplified page layout below) |
Outside-only example: `./taler-monitoring.sh -d demo.taler.net urls`
Surface (ecosystem + mail + mattermost + versions): `./taler-monitoring.sh surface mattermost mail versions`
Full local GOA example: `./taler-monitoring.sh -d hacktivism.ch urls inside versions`
Repo: https://git.hacktivism.ch/hernani/taler-monitoring · tree: `~/src/taler-monitoring`
Version: `./taler-monitoring.sh --ver` (or `--version` / `-V`)
## Monitoring pages — simplified overview (v1.8.0+)
Public HTML is **not** an external addon. Layout is intentionally small:
| Kind | Where | Purpose |
|------|--------|---------|
| **Landing-stack report** | `https://{host}/monitoring/` and `/monitoring_err/` | Per **landing** front (9 stacks): GOA bank/exchange/taler · FP stage bank/exchange/monnaie · FP prod bank/exchange/monnaie |
| **Ecosystem surface** | **only** `https://taler.hacktivism.ch/taler-monitoring-surface/` and `…/taler-monitoring-surface_err/` | Software/versions found in the Taler universe (ports, nmap OS fingerprint, Mattermost, mail firefly+anastasis, package signals) |
There are **no** first-class public pages for `/taler-monitoring-mail*`, `/taler-monitoring-mattermost*`, or similar — those checks run **inside the surface job** and appear on the surface page.
Related:
- **Phase `monpages`:** obligatory **ERROR** if a page is missing, not suite HTML, or **content markers fail** (v1.9.0): top (`sticky-bar` / `taler-mon:top` / generated) and bottom (`footer` / `taler-mon:bottom:complete`). Mid-run pages (progress <100% or `incomplete` meta) may skip bottom ERROR. Body must contain real checks — progress bars alone → ERROR. On GOA, `/taler-monitoring-surface/` is always required.
- **Publish:** staging `~/monitoring-sites-staging` → `DEPLOY_WWW_ROOT` + reverse-proxy handles.
```bash
./taler-monitoring.sh monpages
./taler-monitoring.sh -d hacktivism.ch monpages
```
Host packages (koopa / runners): **[DEPENDENCIES.md](./DEPENDENCIES.md)**.
Host-agents (GOA + FP; shared report pipeline): **[host-agent/README.md](./host-agent/README.md)**.
Public HTML site-gen (same timeout/log/HTML defaults): **[site-gen/README.md](./site-gen/README.md)**.
FP host-agent: `host-agent/run-fp-prod-monitoring.sh` / `run-fp-stage-monitoring.sh` + `host-agent/env/*.env.example` (see `host-agent/README.md`).
Report for the **GOA** stack with boxed severity badges and **grouped test IDs**
`area.group-NN` (e.g. `www.exchange-01`, `e2e.pay-03`):
| Tag | Meaning |
|-----|---------|
| `┌ OK ┐` | check passed |
| `┌ INFO ┐` | status / note |
| `┌ WARN ┐` | degraded but maybe not fatal |
| `┌ ERROR ┐` | component problem |
| `┌ BLOCKER ┐` | **withdraw/pay path cannot complete** because of this |
IDs reset per **group** inside an area — so “too many www tests” become
`www.exchange-*`, `www.bank-*`, `www.landing-*`, etc.
Catalog: **[TESTS.md](./TESTS.md)**.
Git / tags / suite tree: **[VERSIONS.md](./VERSIONS.md)**.
GUI Android notes: **[android-test/GUI-AUTOMATION-NOTES.md](./android-test/GUI-AUTOMATION-NOTES.md)**.
CLI / wallet-cli recurring issues: **[CLI-AUTOMATION-NOTES.md](./CLI-AUTOMATION-NOTES.md)**.
```text
┌ OK ┐ www.exchange-01 exchange /config · HTTP 200
┌ BLOCKER┐ e2e.prereq-02 merchant HTTP 502
```
End of each phase: totals + list of **BLOCKERS** and **ERRORS** (ids included).
## Secrets (local GOA e2e)
**Do not commit passwords.** Suite tree: `~/src/taler-monitoring` · host ops (optional):
sibling `~/src/koopa/koopa-admin-secrets`.
### Local file (recommended on laptop)
```bash
cd ~/src/taler-monitoring
cp secrets.env.example secrets.env # gitignored
$EDITOR secrets.env
./taler-monitoring.sh e2e
```
See **[secrets.env.example](./secrets.env.example)** for every variable that
may appear (admin pass, merchant token, SECRETS_ROOT, progress knobs).
### Search order
1. Already-exported env (`E2E_BANK_ADMIN_PASS`, …)
2. `secrets.env` next to this suite (`MONITORING_SECRETS_ENV=` override)
3. `SECRETS_ROOT` / `KOOPA_ADMIN_SECRETS` file tree
4. Sibling `../koopa-admin-secrets/koopa/host-root`
5. `$HOME/src/koopa/koopa-admin-secrets/koopa/host-root`
6. `$HOME/.config/taler-landing/bank-admin-password.txt`
7. SSH koopa `/root/…` or bank container
### Line IDs + progress
Each check prints **global** and **grouped** ids:
```text
┌ OK ┐ #014 www.exchange-03 exchange /config · HTTP 200
┌ WARN ┐ #015 www.stats-02 stats age soft · yellow badge (not red)
┌ 42% ┐ ████████░░░░░░░░░░░░░░░░░░░░ 14/55
```
- `#014` — run-wide counter (whole `./taler-monitoring.sh` invocation)
- `www.exchange-03` — area.group-NN
- **Colours (left badge + label):**
- **OK** green · **INFO** blue/cyan · **WARN** yellow (never red)
- **ERROR** red · **BLOCKER** magenta
- Progress: badge shows **percent** (not “PROG”); bar fills **left → right**;
`done/total` on the right. Estimate auto-grows if short. Override:
`PROGRESS_TOTAL=N` / `PROGRESS_OFF=1`
- **QR (urls):** harvest `taler://` / `payto://` (+ app store `data-qr-url`) from landings and
`demo-withdraw.json` / `auto-account.json` → validate form → `qrencode` PNG → `zbarimg`
decode must match. Requires packages **`qrencode`** + **`zbar-tools`**.
## Commands
```bash
# Local GOA stack (may use SSH for inside/e2e)
./taler-monitoring.sh # urls + inside + versions + e2e
./taler-monitoring.sh inside # containers on koopa
./taler-monitoring.sh versions # deb.taler.net + package versions vs trixie
./taler-monitoring.sh sanity
./taler-monitoring.sh e2e
# FrancPaysan STAGE (TESTPAYSAN) — like hacktivism breadth:
# default phases: urls + inside + versions + e2e
./taler-monitoring.sh -d stage.lefrancpaysan.ch
./taler-monitoring.sh -d stage.lefrancpaysan.ch inside # stagepaysan only
# inside: low-priv SSH francpaysan-stage-user (User stagepaysan) → podman
# containers stage-lfp-{bank,exchange-ansible,merchant} · ports 9032/9031/9030
# stats: outside-in public …/intro/stats.json (age via generated_at_unix)
# Admin credit for e2e: E2E_BANK_ADMIN_PASS=… or francpaysan-host secrets
# (bank-admin-password.txt under /mnt/data/stagepaysan/bank/secrets/)
# Override: INSIDE_SSH=… INSIDE_PROFILE=stage-lfp
# Other domains — public HTTPS only (pass e2e/inside explicitly)
./taler-monitoring.sh -d taler.net
./taler-monitoring.sh --domain taler-ops.ch
./taler-monitoring.sh -d my.taler-ops.ch urls
./taler-monitoring.sh -d demo.taler.net urls
./taler-monitoring.sh taler.net # bare domain = same as -d
# Explicit endpoints (any mix; override profile or skip -d)
./taler-monitoring.sh --bank https://bank.demo.taler.net \
--exchange https://exchange.demo.taler.net \
--merchant https://backend.demo.taler.net --currency KUDOS urls
# Landing stats.json freshness (outside-in; age via generated_at_unix)
STATS_STALE_SECS=900 STATS_FAIL_SECS=3600 ./taler-monitoring.sh -d stage.lefrancpaysan.ch urls
# --- Full load (GOA / hacktivism) ---
# all = urls + inside + versions + sanity + e2e
# full = all + server + ladder + auth401 (long; needs secrets.env)
./taler-monitoring.sh -d hacktivism.ch full
./taler-monitoring.sh -d hacktivism.ch urls inside versions sanity server e2e ladder auth401
# SPA pin after selfbuild-webui.sh (container):
EXPECT_WEBUI_VERSION=1.6.11 EXPECT_WEBUI_OVERLAY=selfbuild-v1.6.11 \
WEBUI_OVERLAY_DENY='c778af04|master-11340' \
./taler-monitoring.sh -d hacktivism.ch urls
# Only 401 matrix (may create throwaway instance):
./taler-monitoring.sh -d hacktivism.ch auth401
# Stage FrancPaysan — durable mon401 account (password in francpaysan-secrets):
./taler-monitoring.sh -d stage.monnaie.lefrancpaysan.ch auth401
# secrets: ~/francpaysan-secrets/stage/instances/mon401.password
# ~/francpaysan-secrets/stage/default-instance-token.txt
# force throwaway create instead: AUTH401_PREFER_DURABLE=0 AUTH401_CREATE=1 …
```
### Landing `stats.json` (www.stats)
Public `GET …/intro/stats.json` must expose timestamps so monitoring can tell
whether the collector is still updating:
| Field | Role |
|-------|------|
| `generated_at_unix` | epoch seconds — **preferred** for age = now − unix |
| `generated_at` | ISO-8601 (fallback if unix missing) |
| `generated_at_human` | UI “updated” / footer string |
Also checks display payloads (bank-shape `withdraws` / accounts, or exchange-db /
merchant-db counters). Soft WARN if age > `STATS_STALE_SECS` (default 900);
ERROR if age > `STATS_FAIL_SECS` (default 3600; set `0` to disable).
Stage shared feed: same `gen_unix` on bank + exchange + monnaies.
## Domain profiles (`domains.conf`)
Each monitored stack must say **what is bank, exchange, and merchant-backend**.
Edit **[domains.conf](./domains.conf)** (or set `TALER_DOMAINS_CONF=`):
```text
# name bank exchange merchant-backend currency local [canonical]
taler-ops.ch bank.taler-ops.ch exchange.taler-ops.ch my.taler-ops.ch CHF 0
```
| Field | Meaning |
|-------|---------|
| **name** | `-d NAME` / `TALER_DOMAIN` match (add alias lines as needed) |
| **bank** | libeufin/bank public host or `https://…` |
| **exchange** | exchange public host or URL |
| **merchant** | merchant **backend** (SPA/API), e.g. `my.taler-ops.ch` |
| **currency** | `GOA` / `KUDOS` / `CHF` / … |
| **local** | `1` = koopa SSH stack; `0` = public only |
| **landing** | `1` = check `/intro` landings + assets; `0` = skip (TOPS / mytops) |
| **canonical** | optional label after alias (e.g. `my.taler-ops.ch` → `taler-ops.ch`) |
Then:
```bash
./taler-monitoring.sh -d taler-ops.ch urls
```
Overrides (always win after profile load):
```bash
./taler-monitoring.sh -d taler-ops.ch \
--merchant https://my.taler-ops.ch \
--exchange https://exchange.taler-ops.ch \
--bank https://bank.taler-ops.ch urls
```
Helpers in `lib.sh`: `set_taler_stack`, `load_domain_profile`, `apply_taler_domain`.
### Built-in profiles (see `domains.conf`)
| Domain / alias | Bank | Exchange | Merchant | Currency |
|----------------|------|----------|----------|----------|
| `hacktivism.ch` / `koopa` | bank.hacktivism.ch | exchange.hacktivism.ch | taler.hacktivism.ch | GOA |
| `taler.net` / `demo.taler.net` | bank.demo.taler.net | exchange.demo.taler.net | backend.demo.taler.net | KUDOS |
| `test.taler.net` | bank.test… | exchange.test… | backend.test… | **TESTKUDOS** · no landings |
| `lefrancpaysan.ch` | bank.… | exchange.… | **monnaie.…** (merchant) | report-only · no landings |
| **`stage.lefrancpaysan.ch`** | stage.bank.… | stage.exchange.… | **stage.monnaie.…** | **TESTPAYSAN** · landings · **inside via stagepaysan** |
| `taler-ops.ch` / `my.taler-ops.ch` | bank.taler-ops.ch | exchange.taler-ops.ch | **my.taler-ops.ch** | CHF · **no landings** |
| `stage.taler-ops.ch` | bank.stage… | exchange.stage… | **my.stage…** | CHF · **no landings** |
| unknown | bank.DOMAIN | exchange.DOMAIN | my.DOMAIN then probe | any · no landings |
**Landings:** GOA, stage LFP (`TESTPAYSAN`), and optionally demo use public `/intro` pages. **taler-ops.ch does not** — `CHECK_LANDING=0` skips intro crawl.
**SSH:**
| Stack | Host alias | User | Scope |
|-------|------------|------|--------|
| koopa / hacktivism | `koopa` | hernani | full host + containers |
| stage LFP | `francpaysan-stage-user` | **stagepaysan** | own podman only (no sudo / no prod data) |
Other domains: never SSH. Optional **e2e** aborts cleanly on login/KYC.
### E2E amounts (variable)
| | Local (koopa) | Remote |
|--|---------------|--------|
| **ATM withdraw** | 20 · 50 · 100 · 200 · **4200** (paivana) | 10 · 20 · 50 |
| **Pay ladder** | 0.01 … 10 | 0.01 … 1 |
| **Paivana** | HTTP paywall + template pay **GOA:4200** (`goa-shop` / `paivana`) | skipped |
```bash
./taler-monitoring.sh e2e
./taler-monitoring.sh -d taler.net urls e2e
# Amount ladder (withdraw 0 → … → max, then pay): same script for GOA + stage
./taler-monitoring.sh ladder # GOA / current domain
./taler-monitoring.sh -d stage.lefrancpaysan.ch ladder # TESTPAYSAN maxima
# Stage auto-reads bank max_wire_transfer_amount (e.g. 2000) and min denom (0.01).
# Explorer password: francpaysan-stage-user …/bank-explorer-password.txt (or EXP_PW=).
# Override: LADDER_MAX_AMOUNT=100 LADDER_STEPS=7 LADDER_PAY=0 …
# customize e2e:
E2E_WITHDRAW_VALUES="20 50 100" E2E_PAY_VALUES="0.05 1 5" ./taler-monitoring.sh e2e
E2E_VARIABLE=0 WITHDRAW_AMT=GOA:50 PAY_AMT=GOA:1 ./taler-monitoring.sh e2e # single fixed
# GOA shop catalog (local hacktivism): full list in E2E_SHOP_PRODUCTS; each run
# shuffles and pays E2E_SHOP_PICK_N products (default 2).
# E2E_SHOP_PRODUCTS lines: id|Product name|GOA:amount
# E2E_SHOP_PICK_N=2
# (landing QR = taler://pay-template/…/{id}; popup = live taler://pay after POST templates/{id})
# Paivana paywall (local GOA only):
# E2E_PAIVANA=1 (default) PAIVANA_PUBLIC=https://paivana.hacktivism.ch
# E2E_PAIVANA_TEMPLATE=paivana E2E_PAIVANA_AMOUNT=GOA:4200 E2E_PAIVANA_INSTANCE=goa-shop
# E2E_PAIVANA=0 # skip
# remote secrets:
# E2E_BANK_ADMIN_PASS=… E2E_MERCHANT_TOKEN=…
```
## Phases
| Phase | What |
|-------|------|
| **inside** | SSH koopa: processes, postgres, local /config,/keys, wirewatch, recent log ERRORs |
| **versions** | `deb.taler.net` reachable (InRelease/Packages/pool `.deb`); containers can reach it + have apt source; installed Taler packages vs **trixie** |
| **sanity** | bank · exchange · merchant sections (public + server) |
| **e2e** | account → credit → withdraw → wallet → confirm → order → pay |
| **ladder** | GOA withdraw: fixed **0** + low bands + **random high ranges** + fixed **max** |
```bash
# package suite (default trixie on deb.taler.net)
TALER_APT_SUITE=trixie ./taler-monitoring.sh versions
TALER_PKG_BEHIND=error ./taler-monitoring.sh versions # any behind = ERROR
```
E2E maps failures to blockers, e.g.:
- `bank-confirm HTTP 409` → wallet did not select exchange
- `no GOA balance` → wirewatch / transfer
- `create order failed` → merchant auth/instance
- `Alarm clock` during pay → usually missing `handle-uri --yes` or wrong pay URI (must be `…/instances/{inst}/{oid}/?c={token}` from merchant `taler_pay_uri`)
- `insufficient balance` → withdraw incomplete
## Performance (urls phase)
Outside-in HTTPS RTT for bank / exchange / merchant critical paths. Each probe prints
`HTTP … · N ms` on the `[OK]`/`[WARN]`/`[ERROR]` line; end of the block prints
`perf summary` (n / min / p50 / avg / max).
| Env | Default | Meaning |
|-----|---------|---------|
| `PERF_WARN_MS` | `8000` | WARN if latency ≥ this |
| `PERF_FAIL_MS` | `20000` | ERROR if latency ≥ this |
| `PERF_CURL_TIMEOUT` | `25` | curl max seconds per probe |
```bash
PERF_WARN_MS=3000 PERF_FAIL_MS=10000 ./taler-monitoring.sh urls
```
## Wallet coins (e2e · ladder)
At every relevant withdraw/pay step the wallet is snapped via `advanced dump-coins`.
Amounts use exchange **`alt_unit_names`** (Kilo-GOA, Mega-GOA, …) with the base
**GOA value in parentheses**:
```text
[INFO] coins after-ATM — amount_circ=2 Kilo-GOA (GOA:2000)
[INFO] coins after-ATM — denoms_in_circ 1 Kilo-GOA (GOA:1000)×2 (=2 Kilo-GOA (GOA:2000))
== ladder · rung 12 GOA:1000000 · 1 Mega-GOA (GOA:1000000) ==
```
Also: coin counts (in circulation / spent), status histogram, Δ vs previous snap.
History TSV: `$METRICS_DIR/coins-history.tsv`. Map loaded from
`${EXCHANGE_PUBLIC}/config` → `currency_specification.alt_unit_names`.
### Final statistics dashboard (e2e · ladder)
At the end of **e2e** and **ladder**, a boxed **FINAL STATISTICS** block prints:
| Section | Content |
|---------|---------|
| **Coins** | in circulation / spent counts + amounts + denoms (alt names) |
| **Money flow** | withdrawn events/amount, spent (paid), net (wd − spent) |
| **Tendency** | coin/amount trend over snaps (↑↓→), slope, last history rows |
| **Performance** | min / p50 / avg / max per phase; first-half vs second-half latency |
| **Load** | host loadavg + mem before/after; per-container RSS/CPU; tendencies |
Successful withdraw/pay steps append to `flow-withdrawn.txt` / `flow-spent.txt` (ladder also uses its TSVs).
## Load / memory (inside · e2e · ladder)
Host **loadavg**, **RAM used/avail**, and per-container **RSS / CPU / block I/O**
(plus postgres DB sizes and role RSS: libeufin, exchange, merchant, …) are
snapshotted via SSH whenever the stack is under load:
| Phase | When |
|-------|------|
| **inside** | end of container status |
| **e2e** | before work · after ATM withdraws · after payments · after shop · e2e-end (+ delta) |
| **ladder** | before first rung · after last rung (+ delta + timing overall) |
```bash
# LAN down? force WAN DNAT host
KOOPA_SSH=koopa-external ./taler-monitoring.sh inside
# or rely on automatic fallback (default KOOPA_SSH_FALLBACKS=koopa-external)
METRICS_LOAD=0 ./taler-monitoring.sh e2e # skip host probes
LADDER_LOAD=0 ./taler-monitoring.sh ladder
```
| Env | Default | Meaning |
|-----|---------|---------|
| `KOOPA_SSH` | `koopa` | primary SSH host |
| `KOOPA_SSH_FALLBACKS` | `koopa-external` | tried if primary fails |
| `METRICS_LOAD` | `1` | `0` = skip all host/container load probes |
| `LADDER_LOAD` | `1` | `0` = skip load in ladder only |
| `LADDER_PAY` | `1` | `0` = skip payment ladder after withdraws |
| `LADDER_WITHDRAW_SCALE` | `1.5` | withdraw mids / pay mids ratio (funding headroom) |
| `LADDER_STEPS` | `23` | rungs for both withdraw and pay (0 + mids + max−1 + max) |
| `METRICS_LOAD_SSH_TIMEOUT` | `90` | seconds for remote load python |
## Needs
- SSH `koopa` or `koopa-external` (inside / load / sanity server bits)
- secrets under `koopa-admin-secrets/...` for e2e
- `taler-wallet-cli` for e2e
## Languages (i18n) — **v1.2+**
Default **English**. Select language with `--lang en|fr` or `TALER_MON_LANG`.
FrancPaysan stacks auto-select **French** (`*lefrancpaysan*`). See `VERSIONS.md` and `i18n.sh`.
## Public monitoring pages via FQDN (v1.3.1+ / obligatory v1.7.6+ / layout v1.8.0+)
Phase **`monpages`** is **obligatory**: missing or non-suite HTML is **ERROR**.
Soft mode only with `MONPAGES_REQUIRE_PUBLIC=0` (escape hatch).
| Stack | What monpages expects (v1.9.0) |
|-------|--------------------------------|
| **GOA** | 3× `/monitoring/` + **surface** (existence **and** top/bottom/content markers) |
| **FrancPaysan** | Stack mon hosts `/monitoring/` with same content rules |
| **Nine landings (model)** | Each landing front: `/monitoring` + `/monitoring_err` (err when a failed run exists) |
```bash
./taler-monitoring.sh -d hacktivism.ch monpages
./taler-monitoring.sh -d lefrancpaysan.ch monpages
```
If pages are missing publicly: `sudo ~/koopa-caddy/apply-monitoring-live.sh` on koopa
(FP: Infomaniak vhost for `/monitoring*`).
## Version on monitoring pages (v1.3+ / sticky bar)
The sticky bar shows the suite **tag** (e.g. `v1.4.1`) and links to that tag on Forgejo.
Planned **v1.4.0**: fuller L10n (fr-CH for FrancPaysan, de-CH for hacktivism/Taler CH).
Reference in a new issue View git blame Copy permalink