diff --git a/CLI-AUTOMATION-NOTES.md b/CLI-AUTOMATION-NOTES.md index 3cd311a..22f75e5 100644 --- a/CLI-AUTOMATION-NOTES.md +++ b/CLI-AUTOMATION-NOTES.md @@ -5,9 +5,11 @@ stage TESTPAYSAN via **CLI** (`taler-wallet-cli`, bank/merchant HTTP, monitoring e2e/ladder). Ordered roughly by **how often they hurt automation** and how useful a fix in **wallet-cli / wallet-core** (or clearer APIs) would be. -Companion: Android GUI → [`android-test/GUI-AUTOMATION-NOTES.md`](./android-test/GUI-AUTOMATION-NOTES.md). +Companions: +- Android GUI → [`android-test/GUI-AUTOMATION-NOTES.md`](./android-test/GUI-AUTOMATION-NOTES.md) +- Git / suite tree → [`VERSIONS.md`](./VERSIONS.md) -Scripts: `check_e2e.sh`, `check_goa_ladder.sh`, `taler-monitoring.sh`. +Scripts: `check_e2e.sh`, `check_amount_ladder.sh`, `taler-monitoring.sh`. Legend: @@ -45,15 +47,82 @@ Legend: --- -## 3. `reserve_pub` / force-select is fragile (5114) +## 3. `reserve_pub` / force-select is fragile (5114 + empty scrape) | | | |--|--| -| **Seen** | ladder force-select, e2e confirm path, Android spinner issues | -| **Area** | **cli** / **core** / **bank** | -| **Problem** | Cumulative wallets re-emit old reserves; binding the wrong `reserve_pub` yields **HTTP 409 / code 5114** (“already bound”). Automation scrapes accept/tx dumps for candidates. | -| **Workaround** | Score pubs by WID/amount; try several; treat 5114 as “stale reserve” not “out of money”. | -| **Wanted** | CLI returns **the** `reserve_pub` for the just-accepted withdraw in structured JSON; `force-select` / bank-side select API documented with idempotency. | +| **Seen** | ladder force-select, e2e confirm path, Android spinner issues; **repro 2026-07-19 magikoopa** GOA ladder: `force-select skipped · no reserve_pub for this withdraw (WID=…); wallet may not have selected yet` | +| **Area** | **cli** / **core** / **bank** / **ops** | +| **Problem** | (a) Cumulative wallets re-emit old reserves; binding the wrong `reserve_pub` yields **HTTP 409 / code 5114** (“already bound”). (b) Automation scrapes accept/tx dumps for candidates — when the dump has **no** pub at all, ladder WARNs and skips force-select (soft), then often **SKIP_CONFIRM**. (c) Empty scrape is **not** the same as “bank out of money”; do not treat as balance failure. (d) **Mon bug (fixed v1.20.2):** extract used `json.loads(file_from_first_brace)` on wallet dumps that append log lines after JSON → **zero** objects parsed; regex only scanned accept (which has no `reservePub`) while the pub lives under `transactions[].withdrawalDetails.reservePub`. | +| **Workaround** | **Mon ≥1.20.2:** `JSONDecoder.raw_decode` + regex on accept **and** transactions; score by WID in `bankConfirmationUrl` / amount. Still: score pubs by WID/amount; try several; treat 5114 as “stale reserve” not “out of money”. Refresh `transactions` before each force-select try. Soft-skip confirm if still pending after polls. Fix wallet DB / helper first if accept/tx only print FATAL (see §15–16). | +| **Wanted** | CLI returns **the** `reserve_pub` for the just-accepted withdraw in structured JSON; `force-select` / bank-side select API documented with idempotency. Optional: bank integration already exposes `selected_reserve_pub` when wallet selected — prefer that over scraping wallet logs. | + +### 3a. Ladder pin amounts hide mid-range behaviour + +| | | +|--|--| +| **Seen** | `LADDER_STEPS=3` plan collapses to pins only: `GOA:0`, `max-1`, `max` (no random mid rungs) | +| **Area** | **ops** / **bank** / **cli** | +| **Problem** | **Zero** withdraw often never produces a usable `reserve_pub` / coins (wallet 7006 or no denoms). **Absolute max** and sometimes **max-1** hit **mint HTTP 500 / code 5110** SQL `P0001` (ceiling probe). Reproducing “real” force-select needs a **mid** amount (e.g. `GOA:1` … exchange min denom scale), not only pins. | +| **Workaround** | Use `LADDER_STEPS≥5` (or higher) so random mids exist; treat zero / absolute-max / max-1-5110 as probes (soft `CEILING_REJECT` in mon ≥1.20.2). | +| **Wanted** | Ladder plan docs: pin vs mid semantics; CLI structured error for amount-too-large / zero. | + +### 3b. We do **not** manage to withdraw **in time** (GOA) — bank path unreliable, not load + +| | | +|--|--| +| **Bottom line** | On live **hacktivism GOA**, automation **does not get spendable coins in time** for the pay phase (default wait **90 s**). Max-search may report a “best” amount and bank may even show `transfer_done`; pay still sees **`available=GOA:0`** with large **`pendingIncoming`**. Coins might arrive **later** (wirewatch / wallet) — **at least not within mon’s budget**. Recurring (several runs, 2026-07-19). | +| **Area** | **bank** + wire/wallet settle — **not** mon overload, **not** “laptop too slow” | +| **Not the cause: ecosystem load** | Outside-in check **2026-07-19 ~14:56 CEST**: bank/exchange/merchant all **HTTP 200**, config ~**50–150 ms**, exchange `/keys` ~**0.5 s**. Container **loadavg ~1.5–1.7**, RSS bank **~737 MiB** / exchange **~670 MiB**, bank probes **47 ms**, **`wirewatch_running: true`**. Stack is **healthy under moderate load** — failures are **application/SQL/protocol**, not “host on fire.” | +| **Call site (mint)** | `POST ${BANK}/accounts/explorer/withdrawals` → often HTTP **500** `{ code: 5110, hint: "Unexpected sql error with state P0001" }` (**`BANK_UNMANAGED_EXCEPTION`**; Postgres **`P0001`** = `RAISE EXCEPTION`). Should often be clean **409** (insufficient funds); instead opaque **500**. | +| **Config** | `/config` still allows ~**2⁵²** wire max; live mint fails **far below** that. Explorer **balance_explorer** on landing still **~4.5 Peta-GOA**. | + +#### Analysis — where withdraw dies + +| Stage | What we see | Manage to withdraw? | +|-------|-------------|---------------------| +| **Mint (high amounts)** | 5110/P0001 while hi falls Tera→Giga→Mega→Kilo with **lo=0** | **No** — withdrawal never created | +| **Mint (some mid amounts)** | occasional **HTTP 200** + URI | Mint only — not done | +| **Confirm** | force-select 200 then stuck **`selected`**, confirm **HTTP 500** → mon **`SKIP_CONFIRM`** (bounds unchanged) | **No** — bank never wires; **not** out-of-money (§3 / 5114 class) | +| **Bank settle** | rare full path: confirm 204 + `transfer_done` | Bank side “OK_BANK” | +| **Wallet coins** | almost always **`available=0`** within wait, large **`pendingIncoming`**, board **`done=0`** | **Not in time** for pay → **SKIP_BALANCE** / skip max-pay | + +**Pay-wait board (same cumulative wallet), example:** + +```text +summary: tsv_ok=2 · wallet_wd=14 (pendingIncoming=14 done=0) +timer 0s left — available still GOA:0 +pendingIncoming=GOA:335776 +``` + +Reading: only **2** TSV rows are full OK/OK_BANK, but the wallet holds **14** withdrawal txs still **pendingIncoming** and **none done** after the wait. Max-search also leaves many **mint-OK / confirm-failed** withdraws that **inflate pendingIncoming** and may never complete. After **90 s** + short `run-pending`: **still no spendable balance** → **we do not manage to withdraw in time** for pay (whether coins would appear later is unknown to the job). + +#### What *did* work (sparse) — for contrast + +| When / run | What worked | Amounts | +|------------|-------------|---------| +| Landing **2026-07-19 14:19–14:29 CEST** | Explorer singles **recorded** on bank intro stats | **~1.5–2 Giga-GOA**, **~145–202 Mega-GOA**, **~1.8–3.4 Mega-GOA** | +| Max-search run A (same day) | Full bank path (mint→confirm→`transfer_done`) for **two** amounts only; mon best | **`GOA:424114`**, **`GOA:822256`** (~822 Kilo best) | +| Max-search run B (later) | Even fewer full successes; many mint OK + **SKIP_CONFIRM**; mon best collapsed | e.g. full success around **`GOA:12276`** / **`GOA:17257`** (~**17 Kilo** best) | +| Pay after either | Spendable **available** within **90 s** | **`GOA:0`** (not in time) | + +So: **landing/history prove large withdraws were possible earlier the same day**; **automation does not manage a usable withdraw in time** (best amount drops run-to-run; coins not available inside mon’s wait). + +#### Mon behaviour (workaround only) + +- 5110 → soft **`CEILING_REJECT`** / max-search **`hi:=`** +- Full OK only raises **`lo`**; **SKIP_CONFIRM** does not +- Pay waits up to **`LADDER_PAY_WAIT_AVAILABLE_S`** (default 90) with withdraw board; then **SKIP_BALANCE** if still 0 +- **Does not fix** bank SQL mapping, stuck confirm, or wirewatch→coins lag + +#### Wanted (bank / exchange ops) + +1. Map mint **P0001** to structured errors (not **5110**). +2. Confirm after force-select must not stick **selected** + HTTP 500. +3. After `transfer_done`, coins must become wallet **available** (or document delay + API to wait). +4. File bugs with: max-search “what worked” table + landing `recent_withdraws`/`balance_explorer` + load note (healthy) + pay board `tsv_ok` vs `wallet_wd`/`pendingIncoming`/`done`. + +Sources: `https://bank.hacktivism.ch/intro/stats.json`, exchange/merchant intro stats, mon max-ladder logs 2026-07-19. --- @@ -64,7 +133,7 @@ Legend: | **Seen** | Android endless spinner; GOA `bank.hacktivism.ch` withdraw; fixed in app branch `fix/bank-withdraw-auto-exchange` | | **Area** | **core** / **cli** | | **Problem** | Withdraw URI names an exchange the wallet has never added → stuck loading / empty exchange list. CLI requires explicit `exchanges add` + `update` + `accept-tos` every fresh DB. | -| **Workaround** | e2e always adds exchange before ATM ladder; Android needs ensureExchange-style fix. | +| **Workaround** | e2e always adds exchange before bank withdraw ladder; Android needs ensureExchange-style fix. | | **Wanted** | CLI/core: on `accept-uri` withdraw, **auto-add** exchange from URI/bank details (`allowCompletion`), then surface ToS if needed in one command. | --- @@ -76,8 +145,8 @@ Legend: | **Seen** | ladder/e2e strip ports; QR checks; mint URIs with `:443` | | **Area** | **cli** / **bank** / **doc** | | **Problem** | Some bank/wallet outputs include `taler://…host:443/…`. QR validators and some clients reject or double-normalize inconsistently. | -| **Workaround** | `sed 's/:443\//\//g'` everywhere in automation. | -| **Wanted** | Canonical form without default ports in **all** wallet-cli and bank integration outputs; document as invariant. | +| **Workaround** | mon ≥1.23.4: `normalize_taler_uri` (lib.sh) on ladder/e2e/pay; bank landing `demo-withdraw-api.py` normalizes mint JSON; **urls** phase fails GOA/local if public landings still emit `:443`/`:80` in `taler://` (`www.taler-uri-ports-*`, unit `tests/test_normalize_taler_uri.sh`). | +| **Wanted** | libeufin never emits default ports (source fix). | --- @@ -85,7 +154,7 @@ Legend: | | | |--|--| -| **Seen** | e2e uses bank admin + ATM create; ladder/gui-chain reimplement explorer mint in Python | +| **Seen** | e2e uses bank admin + bank withdraw create; ladder/gui-chain reimplement explorer mint in Python | | **Area** | **cli** / **ops** | | **Problem** | Automation always hand-rolls Basic auth → token → `POST …/withdrawals` → `taler_withdraw_uri`. Easy to get wrong (auth header, amount currency). | | **Workaround** | Shared Python in `goa-chain` / ladder / e2e; secrets files. | @@ -189,6 +258,59 @@ Legend: --- +## 15. `--wallet-db=*.json` is rejected (“memory backend not supported”) + +| | | +|--|--| +| **Seen** | **2026-07-19 magikoopa** wallet-cli **1.6.8** / **1.6.11** bundle: any command with `--wallet-db=/tmp/foo.json` fails immediately with `Error: memory backend not supported` | +| **Area** | **cli** / **ops** | +| **Problem** | Current node host treats storage paths ending in **`.json` as unsupported memory backend** and throws before opening sqlite. Old notes / local tests that used `WALLET_DB=….json` break. Accept/transactions then never emit `reserve_pub` → ladder §3 empty-scrape WARNs. | +| **Workaround** | Use a **non-`.json` path**, e.g. `--wallet-db=$SCRATCH/wallet.sqlite3` (ladder already prefers `wallet.sqlite3`). Never pass `….json` as wallet-db. | +| **Wanted** | Clearer error: `wallet-db path must not end in .json; use a directory or .sqlite3 path`. Document legal path forms in `--help`. | + +--- + +## 16. `taler-helper-sqlite3` requires Python ≥ 3.11 on PATH + +| | | +|--|--| +| **Seen** | magikoopa: helper is a **Python** script (`#!/usr/bin/env python3`); with Apple `/usr/bin/python3` **3.9.6** → `FATAL: python version >=3.11 required but running on 3.9.6` in accept/tx logs; wallet may still exit 0 on some paths while stdout is FATAL-only | +| **Area** | **ops** / **cli** | +| **Problem** | Wallet spawns `taler-helper-sqlite3` from **PATH**. Wrong `python3` ⇒ sqlite backend fails ⇒ no real wallet state ⇒ no `reserve_pub` for force-select. | +| **Workaround** | Put **Python ≥3.11** (or 3.12+) first on PATH for mon jobs; ensure `command -v taler-helper-sqlite3` works; smoke: `taler-helper-sqlite3` starts without FATAL. Prefer Homebrew / user `python3.11+` over macOS system 3.9 for agent PATH. | +| **Wanted** | Helper shebang or wrapper that finds a suitable python; wallet preflight error if helper HELLO fails (not silent empty logs). | + +--- + +## 17. Portable wall-clock for scripts (no GNU `timeout` assumed) + +| | | +|--|--| +| **Seen** | macOS agent shells: `timeout: command not found`; e2e already tries `timeout` then `gtimeout` then `perl -e alarm` | +| **Area** | **ops** / **doc** | +| **Problem** | GNU coreutils `timeout` is not on stock macOS. Hard-wiring it breaks portable repro and CI on Darwin. | +| **Workaround** | Prefer **python3** `subprocess.Popen(…).wait(timeout=…)` for outer budgets; for wallet-cli inner calls keep e2e order: `timeout` → `gtimeout` → perl alarm. Document both. | +| **Wanted** | One suite helper `run_with_budget SECS cmd…` in `lib.sh` used by ladder/e2e/smoke. | + +--- + +## 18. How to withdraw (operator checklist, GOA laptop) + +Portable path that works when §§15–16 are fixed: + +1. **PATH:** `taler-helper-sqlite3` + **python3 ≥ 3.11** ahead of system 3.9. +2. **Wallet DB:** `WDB=$TMP/wallet.sqlite3` — **not** `*.json`. +3. **Exchange:** `exchanges add` → `update -f` → `accept-tos` for `https://exchange.hacktivism.ch/`. +4. **Bank mint (explorer):** token with explorer password from `SECRETS_ROOT` / `EXP_PW`; `POST …/accounts/explorer/withdrawals` with e.g. `{"amount":"GOA:1"}` (avoid pure **0** and absolute **max** pins for happy-path tests). +5. **Strip** `:443` from `taler_withdraw_uri`. +6. **`withdraw accept-uri --exchange URL URI`**. +7. **Bank:** poll `…/taler-integration/withdrawal-operation/{WID}` → when `selected`, confirm; if stuck `pending`, force-select with scraped `reserve_pub` (§3). +8. **Settle:** poll wallet balance / bank `transfer_done` — **no** `run-until-done` (§1). + +If step 6’s stdout is only `FATAL: python version…` or `memory backend not supported`, **stop** and fix §15–16 before debugging force-select. + +--- + ## Priority shortlist for `taler-wallet-cli` / core If only a few changes land, these unlock the most automation: @@ -198,6 +320,7 @@ If only a few changes land, these unlock the most automation: 3. **Auto-add exchange + ToS on accept-uri** (§4, §8) 4. **Canonical URIs without :443** (§5) 5. **Emit reserve_pub for last withdraw** (§3) +6. **Clear wallet-db path rules + helper preflight** (§15, §16) --- @@ -210,7 +333,12 @@ If only a few changes land, these unlock the most automation: | exchange | Explicit add/update/accept-tos | | ladder explorer | `read_secret` / EXP_PW_FILE | | stage maxima | bank `max_wire` + keys min denom | -| force-select | multi-rpub try; soft-skip confirm | +| force-select | multi-rpub try; soft-skip confirm; empty scrape → WARN not balance fail | +| higher amounts / withdraw fail | **§3b: we do not manage to withdraw in time** on GOA (healthy load); mint 5110/P0001 + SKIP_CONFIRM + pendingIncoming; pay still available=0 after 90s; landing had Giga/Mega earlier same day | +| pay vs balance | never order more than wallet **available** (soft `SKIP_BALANCE`); pendingIncoming does not count as spendable | +| pay wait UX | board: OK/OK_BANK start order + live pendingIncoming/`done`; often `tsv_ok≪wallet_wd` and **done=0** after 90s → skip pay | +| wallet-db path | Prefer `*.sqlite3` (never `*.json` as db path) | +| helper / python | mon PATH must include helper + python≥3.11 | --- diff --git a/DEPENDENCIES.md b/DEPENDENCIES.md index 940dea3..8e07f59 100644 --- a/DEPENDENCIES.md +++ b/DEPENDENCIES.md @@ -7,7 +7,8 @@ | Binary | Package (zypper) | Role | |--------|------------------|------| | `bash` | (base) | runner | -| `curl` | `curl` | HTTPS/API probes | +| `curl` | `curl` | HTTPS/API probes + **monpages** FQDN checks | +| `nmap` | `nmap` | **surface** OS/service fingerprint (`SURFACE_NMAP=1`, v1.8.0+; OS detect may need root) | | `python3` | `python313-base` (provides `/usr/bin/python3`) | JSON, HTML site-gen, metrics | | `podman` | `podman` | **inside**: `podman exec` into taler-hacktivism* | | `git` | `git` | pin commit SHA in monitoring HTML footer | @@ -40,9 +41,13 @@ command -v qrencode zbarimg | `jq` | nice-to-have; suite uses python3 for JSON | | `dig` / `bind-utils` | debug DNS; not required for default phases | | `ssh` | only for access mode **ssh** (laptop → `KOOPA_SSH` / stage `INSIDE_SSH`); not needed for **host-podman** | -| `node` / `pnpm` + wallet-cli | **e2e** / ladder only (+ secrets) | +| `node` + `taler-wallet-cli` (.mjs) | **e2e** / ladder only (+ secrets) | +| **`taler-helper-sqlite3` on PATH** | **required for e2e** — wallet-core spawns it for withdraw accept-uri / pay; missing → ENOENT mid-ladder. Put next to wallet bindir (`~/.local/bin` on stagepaysan) | +| `timeout` (coreutils) | preferred hard wall for `wcli` (suite falls back to perl only if missing) | | `secrets.env` / admin passwords | e2e only — never system packages | +**Preflight (agents / ops):** before trusting e2e on a host, smoke on the **same PATH** mon uses (`run-host-report` / systemd user): `command -v node taler-helper-sqlite3`, `node "$WALLET_CLI" --version`, short temp-DB `balance`. Do not wait for a full mon run to discover ENOENT. + ### systemd user (no extra packages) - `systemd` user session + **`loginctl enable-linger hernani`** (already **Linger=yes** on koopa) @@ -55,7 +60,7 @@ command -v qrencode zbarimg --- -## firecuda-external (macOS, outside-only timer) +## outside-runner (macOS, outside-only timer) | Need | Notes | |------|--------| @@ -65,6 +70,17 @@ command -v qrencode zbarimg --- -## Franc Paysan hosts (francpaysan / stagepaysan) +## Franc Paysan stage host (stagepaysan) -Document package lists under **`francpaysan-admin-log`** when those host-agents are installed (same idea: curl, python3, podman if inside, qrencode/zbar for full urls). +FP stage (`stagepaysan` via SSH alias `francpaysan-stage-user`): curl, python3, podman if inside, qrencode/zbar for full urls; suite at `~/src/taler-monitoring`. + +**e2e on stagepaysan** also needs user-local: + +| Path | Role | +|------|------| +| `~/.local/bin/node` | Node for wallet-cli | +| `~/.local/bin/taler-wallet-cli.mjs` (or `WALLET_CLI=`) | wallet CLI | +| `~/.local/bin/taler-helper-sqlite3` | **required** — accept-uri / coins | +| `~/francpaysan-secrets/stage/*` | bank admin + merchant token (layout like laptop `francpaysan-secrets`) | + +Host-agent PATH already puts `$HOME/.local/bin` first; still smoke-test under that PATH. diff --git a/README.md b/README.md index 63871da..2c08f4e 100644 --- a/README.md +++ b/README.md @@ -1,15 +1,51 @@ # taler-monitoring -Standalone repository: https://git.hacktivism.ch/hernani/taler-monitoring (tag **v1.0**). +Checks for **GNU Taler** stacks (bank, exchange, merchant; optional surface inventory). -Formerly: `koopa-admin-log/scripts/taler-monitoring/`. +| Mode | What | +|------|------| +| **Outside-only** | Public HTTPS from anywhere — no SSH, no host-agent. Phases such as `urls`, `monpages`, `surface`, … | +| **Inside / local** | Where you operate the stack: podman/SSH (`inside`, `versions`, `e2e`, `ladder`, …) | +| **Host-agent + HTML** | Optional: timers run the suite and publish sticky-bar HTML (see simplified page layout below) | + +Outside-only example: `./taler-monitoring.sh -d demo.taler.net urls` +Surface (ecosystem + mail + mattermost + versions): `./taler-monitoring.sh surface mattermost mail versions` +Full local GOA example: `./taler-monitoring.sh -d hacktivism.ch urls inside versions` + +Repo: https://git.hacktivism.ch/hernani/taler-monitoring · tree: `~/src/taler-monitoring` +Version: `./taler-monitoring.sh --ver` (or `--version` / `-V`) + + +## Monitoring pages — public URL families (only these three) + +**`taler-monitoring` is the suite name (git / CLI), not a public URL path.** +There is **no** mon page at `/taler-monitoring` or `/taler-monitoring/`. + +Exactly **three** first-class public path families (each must work **bare and with trailing slash**; reverse-proxy should `redir` bare → slash). Error companions use `…_err` when the last run failed: + +| Family | Paths (bare + slash) | Where | Purpose | +|--------|----------------------|--------|---------| +| **monitoring** | `/monitoring` · `/monitoring/` · optional `/monitoring_err` · `/monitoring_err/` | Per **landing** host (GOA bank/exchange/taler · FP · mytops stage, …) | Stack report for that host | +| **taler-monitoring-surface** | `/taler-monitoring-surface` · `/…/` · optional `…_err` | **only** `taler.hacktivism.ch` | Ecosystem surface (ports, nmap, mail, mattermost, versions) | +| **taler-monitoring-aptdeploy** | `/taler-monitoring-aptdeploy` · `/…/` · optional `…_err` | **only** `taler.hacktivism.ch` | apt-src merchant deploy tests (podman on koopa) | + +**Not** public mon pages: `/taler-monitoring-mail*`, `/taler-monitoring-mattermost*`, bare `/taler-monitoring` — mail/mm are folded into **surface**. + +Related: + +- **Phase `monpages`:** obligatory **ERROR** if a listed page is missing, bootstrap stub, not suite HTML, or **content markers fail**. Bare paths that fall through to merchant **code 21** are ERROR (not “OK because slash works”). +- **Publish:** staging `~/monitoring-sites-staging` → `DEPLOY_WWW_ROOT` + reverse-proxy handles for **only** the three families above. + +```bash +./taler-monitoring.sh monpages +./taler-monitoring.sh -d hacktivism.ch monpages +``` -# taler-monitoring Host packages (koopa / runners): **[DEPENDENCIES.md](./DEPENDENCIES.md)**. Host-agents (GOA + FP; shared report pipeline): **[host-agent/README.md](./host-agent/README.md)**. Public HTML site-gen (same timeout/log/HTML defaults): **[site-gen/README.md](./site-gen/README.md)**. -FP install notes: `francpaysan-admin-log/docs/taler-monitoring-host-agents.md`. +FP stage host-agent: `host-agent/run-fp-stage-monitoring.sh` + `host-agent/env/stagepaysan.env.example` (see `host-agent/README.md`). Report for the **GOA** stack with boxed severity badges and **grouped test IDs** `area.group-NN` (e.g. `www.exchange-01`, `e2e.pay-03`): @@ -25,9 +61,11 @@ Report for the **GOA** stack with boxed severity badges and **grouped test IDs** IDs reset per **group** inside an area — so “too many www tests” become `www.exchange-*`, `www.bank-*`, `www.landing-*`, etc. Catalog: **[TESTS.md](./TESTS.md)**. -Android / tooling: **[VERSIONS.md](./VERSIONS.md)**. +Git / tags / suite tree: **[VERSIONS.md](./VERSIONS.md)**. +Optional / deprecated tooling: **[meta/](./meta/)** (firecuda, legacy site-gen, old mail/mm units). + GUI Android notes: **[android-test/GUI-AUTOMATION-NOTES.md](./android-test/GUI-AUTOMATION-NOTES.md)**. -CLI / wallet-cli recurring issues (for upstream improvements): **[CLI-AUTOMATION-NOTES.md](./CLI-AUTOMATION-NOTES.md)**. +CLI / wallet-cli recurring issues: **[CLI-AUTOMATION-NOTES.md](./CLI-AUTOMATION-NOTES.md)**. ```text ┌ OK ┐ www.exchange-01 exchange /config · HTTP 200 @@ -38,13 +76,13 @@ End of each phase: totals + list of **BLOCKERS** and **ERRORS** (ids included). ## Secrets (local GOA e2e) -**Do not commit passwords.** Canonical trees: `~/src/koopa/koopa-admin-log` + +**Do not commit passwords.** Suite tree: `~/src/taler-monitoring` · host ops (optional): sibling `~/src/koopa/koopa-admin-secrets`. ### Local file (recommended on laptop) ```bash -cd ~/src/koopa/src/taler-monitoring +cd ~/src/taler-monitoring cp secrets.env.example secrets.env # gitignored $EDITOR secrets.env ./taler-monitoring.sh e2e @@ -102,7 +140,7 @@ Each check prints **global** and **grouped** ids: # inside: low-priv SSH francpaysan-stage-user (User stagepaysan) → podman # containers stage-lfp-{bank,exchange-ansible,merchant} · ports 9032/9031/9030 # stats: outside-in public …/intro/stats.json (age via generated_at_unix) -# Admin credit for e2e: E2E_BANK_ADMIN_PASS=… or francpaysan-host secrets +# Admin credit for e2e: E2E_BANK_ADMIN_PASS=… or stage host secrets # (bank-admin-password.txt under /mnt/data/stagepaysan/bank/secrets/) # Override: INSIDE_SSH=… INSIDE_PROFILE=stage-lfp @@ -165,8 +203,9 @@ Each monitored stack must say **what is bank, exchange, and merchant-backend**. Edit **[domains.conf](./domains.conf)** (or set `TALER_DOMAINS_CONF=`): ```text -# name bank exchange merchant-backend currency local [canonical] -taler-ops.ch bank.taler-ops.ch exchange.taler-ops.ch my.taler-ops.ch CHF 0 +# name bank exchange merchant currency local landing lang locale [canonical] +taler-ops.ch bank.… exchange.… my.… CHF 0 0 en de-CH +stage.lefrancpaysan.ch stage.bank.… stage.exchange.… stage.monnaie.… TESTPAYSAN 0 1 fr fr-CH ``` | Field | Meaning | @@ -178,8 +217,12 @@ taler-ops.ch bank.taler-ops.ch exchange.taler-ops.ch my.taler-ops.ch CHF 0 | **currency** | `GOA` / `KUDOS` / `CHF` / … | | **local** | `1` = koopa SSH stack; `0` = public only | | **landing** | `1` = check `/intro` landings + assets; `0` = skip (TOPS / mytops) | +| **lang** | UI language **`en`** / **`fr`** / **`de`** when `TALER_MON_LANG_SET≠1`. Stock profiles use **en** or **fr** only — **`de` is never a default** (opt-in via `--lang de` or an explicit profile edit). | +| **locale** | Regional l10n (BCP 47): **`fr-CH`** for FrancPaysan (Suisse romande); **`de-CH`** for other Swiss stacks. Aliases `ch-FR` / `ch-DE` accepted. Not the same as UI `lang`. | | **canonical** | optional label after alias (e.g. `my.taler-ops.ch` → `taler-ops.ch`) | +Workstation `TALER_MON_LANG=en` does **not** override a profile with `lang=fr`. Force English on FP only with `--lang en` / `TALER_MON_LANG_SET=1`. + Then: ```bash @@ -224,19 +267,21 @@ Other domains: never SSH. Optional **e2e** aborts cleanly on login/KYC. | | Local (koopa) | Remote | |--|---------------|--------| -| **ATM withdraw** | 20 · 50 · 100 · 200 · **4200** (paivana) | 10 · 20 · 50 | +| **Bank withdraw** | 20 · 50 · 100 · 200 · **4200** (paivana) | 10 · 20 · 50 | | **Pay ladder** | 0.01 … 10 | 0.01 … 1 | | **Paivana** | HTTP paywall + template pay **GOA:4200** (`goa-shop` / `paivana`) | skipped | ```bash ./taler-monitoring.sh e2e ./taler-monitoring.sh -d taler.net urls e2e -# Amount ladder (withdraw 0 → … → max, then pay): same script for GOA + stage -./taler-monitoring.sh ladder # GOA / current domain +# Amount ladder (any currency): classic 0 → … → max, then optional pay +./taler-monitoring.sh ladder # classic (current domain) ./taler-monitoring.sh -d stage.lefrancpaysan.ch ladder # TESTPAYSAN maxima +# Max-search: highest withdrawable + highest payable (≤ available; GOA free template goa-free) +./taler-monitoring.sh max-ladder # Stage auto-reads bank max_wire_transfer_amount (e.g. 2000) and min denom (0.01). # Explorer password: francpaysan-stage-user …/bank-explorer-password.txt (or EXP_PW=). -# Override: LADDER_MAX_AMOUNT=100 LADDER_STEPS=7 LADDER_PAY=0 … +# Override: LADDER_MODE=max LADDER_MAX_PROBES=16 LADDER_MAX_AMOUNT=100 LADDER_STEPS=7 LADDER_PAY=0 … # customize e2e: E2E_WITHDRAW_VALUES="20 50 100" E2E_PAY_VALUES="0.05 1 5" ./taler-monitoring.sh e2e E2E_VARIABLE=0 WITHDRAW_AMT=GOA:50 PAY_AMT=GOA:1 ./taler-monitoring.sh e2e # single fixed @@ -261,7 +306,7 @@ E2E_VARIABLE=0 WITHDRAW_AMT=GOA:50 PAY_AMT=GOA:1 ./taler-monitoring.sh e2e # s | **versions** | `deb.taler.net` reachable (InRelease/Packages/pool `.deb`); containers can reach it + have apt source; installed Taler packages vs **trixie** | | **sanity** | bank · exchange · merchant sections (public + server) | | **e2e** | account → credit → withdraw → wallet → confirm → order → pay | -| **ladder** | GOA withdraw: fixed **0** + low bands + **random high ranges** + fixed **max** | +| **ladder** | amount ladder modules: classic 23-rung wd+pay, or max-search wd+pay (`max-ladder`) | ```bash # package suite (default trixie on deb.taler.net) @@ -300,8 +345,8 @@ Amounts use exchange **`alt_unit_names`** (Kilo-GOA, Mega-GOA, …) with the bas **GOA value in parentheses**: ```text -[INFO] coins after-ATM — amount_circ=2 Kilo-GOA (GOA:2000) -[INFO] coins after-ATM — denoms_in_circ 1 Kilo-GOA (GOA:1000)×2 (=2 Kilo-GOA (GOA:2000)) +[INFO] coins after-bank-wd — amount_circ=2 Kilo-GOA (GOA:2000) +[INFO] coins after-bank-wd — denoms_in_circ 1 Kilo-GOA (GOA:1000)×2 (=2 Kilo-GOA (GOA:2000)) == ladder · rung 12 GOA:1000000 · 1 Mega-GOA (GOA:1000000) == ``` @@ -332,13 +377,13 @@ snapshotted via SSH whenever the stack is under load: | Phase | When | |-------|------| | **inside** | end of container status | -| **e2e** | before work · after ATM withdraws · after payments · after shop · e2e-end (+ delta) | +| **e2e** | before work · after bank withdraws · after payments · after shop · e2e-end (+ delta) | | **ladder** | before first rung · after last rung (+ delta + timing overall) | ```bash # LAN down? force WAN DNAT host KOOPA_SSH=koopa-external ./taler-monitoring.sh inside -# or rely on automatic fallback (default KOOPA_SSH_FALLBACKS=koopa-external) +# or rely on automatic fallback (KOOPA_SSH_FALLBACKS from env) METRICS_LOAD=0 ./taler-monitoring.sh e2e # skip host probes LADDER_LOAD=0 ./taler-monitoring.sh ladder @@ -347,20 +392,71 @@ LADDER_LOAD=0 ./taler-monitoring.sh ladder | Env | Default | Meaning | |-----|---------|---------| | `KOOPA_SSH` | `koopa` | primary SSH host | -| `KOOPA_SSH_FALLBACKS` | `koopa-external` | tried if primary fails | +| `KOOPA_SSH_FALLBACKS` | *(from env)* | tried if primary fails | | `METRICS_LOAD` | `1` | `0` = skip all host/container load probes | | `LADDER_LOAD` | `1` | `0` = skip load in ladder only | | `LADDER_PAY` | `1` | `0` = skip payment ladder after withdraws | -| `LADDER_WITHDRAW_SCALE` | `1.5` | withdraw mids / pay mids ratio (funding headroom) | -| `LADDER_STEPS` | `23` | rungs for both withdraw and pay (0 + mids + max−1 + max) | +| `LADDER_MODE` | `classic` | default **classic** 23-rung wd+pay; `max` = max-search wd+pay (`max-ladder`) | +| `LADDER_MAX_PROBES` | `32` | max-search probe budget; `0` = until converge / timeout | +| `LADDER_HIGH_RUNGS` | `6` | max-search: first N probes random-high, then bisect | +| `LADDER_MAX_TOL_REL` | `0.001` | max-search relative lo/hi gap to stop | +| `LADDER_FREE_TEMPLATE` | `goa-free` | public free-amount pay template (hacktivism) | +| `LADDER_USE_FREE_TEMPLATE` | `1` | create pays via template POST `{"amount":…}` | +| `LADDER_WITHDRAW_SCALE` | `1.5` | classic: withdraw mids / pay mids ratio (funding headroom) | +| `LADDER_STEPS` | `23` | classic: rungs for withdraw and pay (0 + mids + max−1 + max) | | `METRICS_LOAD_SSH_TIMEOUT` | `90` | seconds for remote load python | ## Needs -- SSH `koopa` or `koopa-external` (inside / load / sanity server bits) +- SSH via `KOOPA_SSH` / `KOOPA_SSH_FALLBACKS` (env) (inside / load / sanity server bits) - secrets under `koopa-admin-secrets/...` for e2e - `taler-wallet-cli` for e2e -## Languages (i18n) +## Languages (i18n) — **v1.2+** -Default **English**. FrancPaysan stacks use **French** (`TALER_MON_LANG=fr`, auto for `*lefrancpaysan*`). See `VERSIONS.md` and `i18n.sh`. +Default **English**. Select language with `--lang en|fr` or `TALER_MON_LANG`. +FrancPaysan stacks auto-select **French** (`*lefrancpaysan*`). See `VERSIONS.md` and `i18n.sh`. + +## Public monitoring pages via FQDN (v1.3.1+ / obligatory v1.7.6+ / layout v1.8.0+) + +Phase **`monpages`** is **obligatory**: missing or non-suite HTML is **ERROR**. +Soft mode only with `MONPAGES_REQUIRE_PUBLIC=0` (escape hatch). + +| Stack | What monpages expects (v1.9.0) | +|-------|--------------------------------| +| **GOA** | 3× `/monitoring/` + **surface** (existence **and** top/bottom/content markers) | +| **FrancPaysan** | Stack mon hosts `/monitoring/` with same content rules | +| **Nine landings (model)** | Each landing front: `/monitoring` + `/monitoring_err` (err when a failed run exists) | + +```bash +./taler-monitoring.sh -d hacktivism.ch monpages +./taler-monitoring.sh -d lefrancpaysan.ch monpages +``` + +If pages are missing publicly: `sudo $APPLY_MONITORING_LIVE` on koopa +(FP: Infomaniak vhost for `/monitoring*`). + +## Devtesting · fake-franken CHF (v1.13+) + +Phase **`devtesting`** (aliases: `franken`, `fake-franken`) exercises the +Taler Operations **rusty** host (`taler-devtesting` forced command): + +```bash +./taler-monitoring.sh -d stage.taler-ops.ch --currency CHF devtesting +``` + +Needs SSH to `devtesting@rusty.taler-ops.ch` (Host `taler-rusty-devtesting`). +Probes: CLI help, `geniban` (CH IBAN / synthetic debit funding), baseline `fake-incoming`, +optional **amount ladder** (`DEVTESTING_LADDER=1`, default **23** rungs up to **CHF:50000**). +Each rung reuses geniban debit payto so volume covers the max rung without a real bank withdraw. +`fake-incoming` CHF to exchange credit +IBAN. A nexus bounce *missing reserve public key* is **OK** for the plumbing +probe (no real wallet reserve required). + +## Version on monitoring pages (v1.3+ / sticky bar) + +The sticky bar shows the suite **tag** (e.g. `v1.11.0`) and links to that tag on Forgejo. + +**v1.12.0:** Run header (**monitoring env context**: env_file, suite pin, domain/hosts/flags) is a collapsed `
` block above the console log — expand to inspect agent context. + +**v1.11.0:** Click the sticky **error** or **warning** count to filter the console log to only those lines (other log lines hide; sticky bar, stats, and top overviews stay). Same via `#filter-error` / `#filter-warn` (or `#first-error` / `#first-warn`). Click again or **Show all** to clear. diff --git a/TESTS.md b/TESTS.md index a762f39..7364968 100644 --- a/TESTS.md +++ b/TESTS.md @@ -20,13 +20,15 @@ Every check line has a **global** run number and a **grouped** id: | **versions** | `check_versions.sh` | `outside` `inside` `compare` | | **aptdeploy** | `check_apt_deploy.sh` | `trixie` `trixie-testing` (koopa podman apt-src merchant smoke) | | **surface** | `check_surface.sh` | **remote-only** ecosystem / `-d` domain inventory (not in default/all/full) | +| **monpages** | `check_monitoring_pages.sh` | public HTML via FQDN — **ERROR** if missing or markers fail (top+bottom+body); surface required on GOA (v1.9.0) | +| **surface** | `check_surface.sh` | ecosystem inventory + **nmap** OS fingerprint; mail/mattermost content reported via surface page | | **sanity** | `check_sanity.sh` | `bank` `exchange` `merchant` | | **server** | `check_server.sh` | (flat `server-NN` or host groups) | -| **e2e** | `check_e2e.sh` | `prereq` `load` `bank` `wallet` `atm` `settle` `pay` `shop` `paivana` `dig` `report` | -| **ladder** | `check_goa_ladder.sh` | `plan` `load` `withdraw` `pay` `report` | +| **e2e** | `check_e2e.sh` | `prereq` `load` `bank` `wallet` `bankwd` `settle` `pay` `shop` `paivana` `dig` `report` | +| **ladder** | `check_amount_ladder.sh` + `ladder/lib_pay.sh` | `plan` `load` `withdraw` `pay` `report` — classic / max-search; phases `ladder` · `max-ladder` · `goa-ladder` (alias) | **Why groups:** flat `www-001`…`www-080` was hard to talk about. -`www.bank-04` / `e2e.atm-02` pin the failure to a logical block. +`www.bank-04` / `e2e.bankwd-02` pin the failure to a logical block. Usage in scripts: @@ -57,10 +59,13 @@ Numbering follows **executed** checks (early skip may shift later NN inside the | **www.webui-** | SPA fingerprints: **`/webui/version.txt`**, **`/webui/version-overlay.txt`**, `index.html` / `index.js`; optional pin via `EXPECT_WEBUI_VERSION` / `EXPECT_WEBUI_OVERLAY` / `WEBUI_OVERLAY_DENY` | | **www.paivana-** | local GOA paywall front (redirect to template) | | **www.landing-** | own-stack intro links; static assets (`qrcode.min.js` hard; `og-goa-shop.png` hard only GOA/local); **demo-withdraw.json** GOA-only; shop-pay soft; stage merchant shop: `shop-ui.js` + `shops.css` + `qrcode.min.js`; cross-links local | +| **www.taler-uri-ports-** | **no default `:443`/`:80`** in public `taler://` URIs — bank/exchange/merchant `/intro/` HTML, `demo-withdraw.json`, `auto-account.json`, bank `stats.json` (ERROR on GOA/local). Unit: `tests/test_normalize_taler_uri.sh` | | **www.qr-** | QR payloads: harvest `taler://` / `payto://` / app `data-qr-url` from landings + mint JSON; **form** check; **qrencode → zbarimg** exact roundtrip; optional static QR images. Needs `qrencode` + `zbar-tools`. Skip: `QR_CHECK=0` | **QR rule:** `taler://withdraw/HOST/taler-integration/UUID` (no default `:443`/`:80`); `taler://pay/` / `pay-template/`; `payto://…` shape OK; decoded PNG must equal payload. Form errors on withdraw/pay/payto are **ERROR** on local/GOA. +**Default port rule:** public and automation URIs must never keep libeufin-style `host:443` / `host:80`. Consumers use `normalize_taler_uri` (lib.sh); bank landing API normalizes in `demo-withdraw-api.py`. + **Legal docs rule:** HTTP 200, non-empty body, not plain `not configured`, not merchant API JSON `code:21`. Local stack may require content needle (terms/privacy/FADP/GOA…). **Performance rule:** Measured from the **monitoring runner** (public via Caddy), not container loopback. Latency (ms) on every perf line + **perf summary** (n / min / p50 / avg / max). @@ -95,10 +100,10 @@ SSH: `KOOPA_SSH` (default `koopa`), then `KOOPA_SSH_FALLBACKS` when LAN is unrea | Group | Checks | |-------|--------| | **e2e.prereq-** | wallet-cli, currency, budgets, secrets, public reachability | -| **e2e.load-** | host/container load snapshots (before / after ATM / after pay) | +| **e2e.load-** | host/container load snapshots (before / after bank withdraw / after pay) | | **e2e.bank-** | account, credit, withdraw ops | | **e2e.wallet-** | exchange + ToS, wallet setup | -| **e2e.atm-** | ATM withdraw ladder rungs | +| **e2e.bankwd-** | Bank withdraw ladder rungs | | **e2e.settle-** | wallet settlement wait / spendable balance | | **e2e.pay-** | variable payments | | **e2e.shop-** | GOA shop product catalog pays | @@ -107,7 +112,7 @@ SSH: `KOOPA_SSH` (default `koopa`), then `KOOPA_SSH_FALLBACKS` when LAN is unrea | **e2e.report-** | final tallies | When filing an issue, quote the full id + label, e.g. -`e2e.atm-03 ATM withdraw 5 GOA · bank confirmed, wallet empty`. +`e2e.bankwd-03 Bank withdraw 5 GOA · bank confirmed, wallet empty`. --- @@ -124,7 +129,7 @@ When filing an issue, quote the full id + label, e.g. | **aptdeploy.trixie-upgrade-** | upgrade-track `…-trixie-upgrade` (install once, then mytops-style upgrade) | | **aptdeploy.trixie-testing-upgrade-** | upgrade-track `…-trixie-testing-upgrade` | -Standalone: `host-agent/run-aptdeploy.sh` (on koopa local; else `ssh koopa-external`). +Standalone: `host-agent/run-aptdeploy.sh` (on koopa local; else ssh `$DEPLOY_SSH` / `$APT_DEPLOY_SSH`). ### surface (remote-only public inventory) @@ -139,7 +144,7 @@ Standalone: `host-agent/run-aptdeploy.sh` (on koopa local; else `ssh koopa-exter - OSV CVE query when package+version known → **ERROR** on hits - Never SSH / never podman on the *targets* | **sanity.bank-** / **.exchange-** / **.merchant-** | public + optional server-side per component | -| **ladder.plan-** / **.load-** / **.withdraw-** / **.pay-** / **.report-** | amount ladder (GOA ceiling or stage TESTPAYSAN `max_wire`) | +| **ladder.plan-** / **.load-** / **.withdraw-** / **.pay-** / **.report-** | amount ladder classic or max-search (any currency; stage uses `max_wire`) | | **server-** | SSH host ports / processes (flat unless grouped later) | | **auth401.*** | merchant 401 / case matrix (WebUI APIs): **signup** `POST /instances` MIX id, **login** Basic user low vs MIX/UPPER, full **case** matrix, **password** bad/empty, **pwchange** `POST private/auth` + re-login, **admin-create** `POST /management/instances` (if admin token), **bearer**, **durable mon401**, **webui** SPA `toLowerCase` probe (`check_auth401.sh`) | @@ -162,7 +167,7 @@ Skip SPA block: `CHECK_WEBUI_SPA=0`. | **`full`** | `urls inside versions sanity server e2e ladder auth401` | ```bash -cd ~/src/koopa/src/taler-monitoring +cd ~/src/taler-monitoring # secrets.env with bank admin + merchant tokens (see secrets.env.example) ./taler-monitoring.sh -d hacktivism.ch full # same, explicit: @@ -191,3 +196,14 @@ totals: 40 OK, 1 ERROR, 2 WARN, 5 INFO `NO_COLOR=1` or `CLICOLOR=0` disables boxes/colours (ASCII `[ OK ]` + `-- group --` headers). Always use `printf --` friendly plain headers when colour is off (leading `---` is not a printf option). + + +## devtesting · fake-franken (v1.13+) + +| ID prefix | What | +|-----------|------| +| **devtesting.cli-** | SSH to rusty · `taler-devtesting` CLI | +| **devtesting.geniban-** | synthetic CH IBAN | +| **devtesting.credit-payto-** | exchange credit IBAN for nexus | +| **devtesting.fake-incoming-** / **baseline-** | CHF IN via nexus; bounce missing reserve pub = plumbing OK | +| **devtesting.ladder-** | CHF fake-incoming amount ladder (default 23 rungs → CHF:50000; synthetic geniban debit each rung) | diff --git a/VERSION b/VERSION index 1cc5f65..27ddcc1 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.1.0 \ No newline at end of file +1.23.4 diff --git a/VERSIONS.md b/VERSIONS.md index c53eac5..1da8044 100644 --- a/VERSIONS.md +++ b/VERSIONS.md @@ -3,34 +3,145 @@ Release history for the standalone **taler-monitoring** suite (https://git.hacktivism.ch/hernani/taler-monitoring). +## Version scheme + +Tags and `VERSION` use **MAJOR.FEATURE.FIX** (three components), introduced in **v1.4.0**: + +| Part | Meaning | +|------|---------| +| **MAJOR** | Breaking changes to CLI, layout, or host-agent contract | +| **FEATURE** | New capabilities (phases, HTML features, i18n packs, …) | +| **FIX** | Bugfixes and docs that do not add a feature | + +Git tags: `vMAJOR.FEATURE.FIX` (e.g. `v1.8.0`). File `VERSION` omits the `v` prefix. + | Tag | Date (UTC) | Notes | |-----|------------|--------| -| **v1.1** | 2026-07-18 | **i18n**: default UI/console language **English**; **French** for FrancPaysan (`TALER_MON_LANG=fr`, auto on `*lefrancpaysan*`). Sticky bar “what this monitors” EN/FR; badge/summary chrome + message translation via `i18n.sh`. | -| **v1.0** | 2026-07-18 | Initial standalone release. Split from `koopa-admin-log` (`scripts/taler-monitoring/`). Host-agent HTML, surface, aptdeploy, sticky bar. | +| **v1.23.4** | 2026-07-19 | **Bugfix / mon:** shared **normalize_taler_uri** (strip taler:// host :443/:80); ladder/e2e/pay use it. **urls** scans bank+exchange+merchant intros + demo-withdraw/auto-account/stats for default ports (**ERROR** on GOA). Unit test tests/test_normalize_taler_uri.sh. | +| **v1.23.3** | 2026-07-19 | **Bugfix:** payments stale because max-ladder never got coins — root cause on koopa was **taler-wallet-cli missing** (all mint-OK → SKIP_ACCEPT). Hard-fail ladder prereq if CLI/helper missing; settle keeps polling after transfer_done + short run-pending; pay wait default 180s; max-ladder wrapper sets PATH/WALLET_CLI. | +| **v1.23.2** | 2026-07-19 | **Bugfix:** max-ladder host-agent forces **RUN_TIMEOUT=10800** (koopa env had 600 → run aborted mid-ladder); install starts oneshots with **--no-block**. | +| **v1.23.1** | 2026-07-19 | **Fix / scope:** only **`/monitoring-max-ladder/`** for GOA (drop classic `/monitoring-ladder`). Daily timer + install first-run. monpages catalog/Caddy bare redirs for max-ladder only. | +| **v1.23.0** | 2026-07-19 | **Feature:** GOA daily mon page **`/monitoring-max-ladder/`** (max-search) on bank/exchange/taler.hacktivism.ch *(1.23.0 briefly also had classic ladder; removed in 1.23.1)*. | +| **v1.22.1** | 2026-07-19 | **Bugfix (UX):** pay wait after withdraw no longer looks hung — shows **withdraw board in start order** (OK/OK_BANK from TSV) with live wallet status (`pendingIncoming` → `done`) + timer + aggregate available/pendingIncoming on the **same** wallet DB. Short `run-pending` nudge only (no run-until-done). Default wait 90s (`LADDER_PAY_WAIT_AVAILABLE_S`). | +| **v1.22.0** | 2026-07-19 | **Feature:** modular amount ladder — `ladder/lib_pay.sh` (pay) + withdraw in `check_amount_ladder.sh`; `ladder/extract_rpubs.py`. **max-ladder** hunts highest **withdrawable** and highest **payable** (≤ wallet available). GOA pays use public free-amount template **`goa-free`** (`LADDER_FREE_TEMPLATE`). Classic still default 23-rung wd+pay. Phase alias `goa-ladder` → amount ladder (shim removed). | +| **v1.21.0** | 2026-07-19 | **Feature:** generic **amount ladder** (`check_amount_ladder.sh`; compat shim `check_goa_ladder.sh`). Default **classic** 23-rung withdraw (+ pay when enabled); **max-search** (`max-ladder` / `LADDER_MODE=max`) hunts highest mintable amount (probes default 32, `0`=until timeout). **Pay:** never overspend — soft `SKIP_BALANCE` if amount > wallet available. **Alt units:** report/logs show human names (e.g. `8.15 Tera-GOA (GOA:…)`) + raw. **Fixes bundled:** external extract (no stdin SyntaxError); force-select multi-rpub; soft `CEILING_REJECT` on mint 5110/P0001; crash-proof report. CLI-AUTOMATION-NOTES §3/3a/3b/§15–18. | +| **v1.20.1** | 2026-07-19 | **Docs:** CLI-AUTOMATION-NOTES — force-select empty scrape + ladder pins (§3/3a); wallet-db must not be `*.json` (§15); `taler-helper-sqlite3` needs Python ≥3.11 (§16); portable wall-clock (§17); GOA withdraw checklist (§18). No code change. *(Superseded notes folded into **v1.21.0** feature release.)* | +| **v1.20.0** | 2026-07-19 | **Feature:** domains.conf **`locale`** (l10n) — FP **`fr-CH`**, others **`de-CH`** (aliases ch-FR/ch-DE); UI **`lang=de`** supported (sticky + console tags) but **never** a stock profile default; env `TALER_MON_LOCALE` / `TALER_DOMAIN_LOCALE`. | +| **v1.19.1** | 2026-07-19 | **Bugfix / docs:** DEPENDENCIES — e2e requires `taler-helper-sqlite3` on mon PATH + preflight; README domains.conf documents **`lang`** column. | +| **v1.19.0** | 2026-07-19 | **Feature:** domains.conf **`lang`** column (en\|fr) is the global UI language default per stack; i18n + HTML read `TALER_DOMAIN_LANG` / conf (FP profiles `lang=fr`). | +| **v1.18.9** | 2026-07-19 | **Bugfix:** UI lang — *lefrancpaysan*/*francpaysan* always default **fr** unless `TALER_MON_LANG_SET=1` / `--lang`; workstation `TALER_MON_LANG=en` no longer locks English (host-agent + console_to_html). | +| **v1.18.8** | 2026-07-19 | **Bugfix:** e2e wallet — require `taler-helper-sqlite3` on PATH; use absolute `node` + `timeout` instead of fragile `perl -e alarm` for wcli; clearer accept-uri errors (no raw alarm-shift noise). | +| **v1.18.7** | 2026-07-19 | **Bugfix:** mail/surface catalogs — TSA/Anastasis MX is **mail.anastasis.lu** (→ pixel), not non-resolving `anastasis.taler-systems.com`; add `anastasis.lu` + pixel host probes (fixes surface_err mail.*). **Also:** FP stage host-agent defaults include **e2e** + 1800s timeout; stagepaysan.env.example wires francpaysan-secrets + wallet-cli. | +| **v1.18.6** | 2026-07-19 | **Bugfix:** TESTS.md table cell for `e2e.bankwd-` markdown typo from rename. | +| **v1.18.5** | 2026-07-19 | **Bugfix:** TESTS.md e2e group list `atm` → `bankwd` (missed in 1.18.4). | +| **v1.18.4** | 2026-07-19 | **Bugfix:** e2e group id **atm → bankwd** (`e2e.bankwd-*`); finish ATM wording cleanup so mon logs match **Bank withdraw** (no leftover “atm” phase label). Env `E2E_ATM_MAX` still accepted as alias. | +| **v1.18.3** | 2026-07-19 | **Bugfix / wording:** e2e user-facing **ATM withdraw** → **Bank withdraw** (amounts still CHF-style cash-note ladder 20/50/100/200/… — “ATM” was only a metaphor for those note sizes, not a physical ATM). Env: `E2E_BANK_WITHDRAW_MAX` (legacy alias `E2E_ATM_MAX`). Internal group id `e2e.atm-*` kept for check-id stability. | +| **v1.18.2** | 2026-07-19 | **Bugfix:** never publish / accept **stale mon HTML assets** — refuse pre-1.15.8 box-drawing sticky JS in site-gen + generated pages; monpages ERROR on public pages still carrying that JS or teed `SyntaxWarning: invalid escape sequence`; purge `site-gen/__pycache__` after suite reset; convert with `PYTHONWARNINGS=error::SyntaxWarning`. | +| **v1.18.1** | 2026-07-19 | **Bugfix:** (1) mon HTML sticky/err counts honor French badges **ERREUR** / **AVERT** (were plain → 0 errors on FP pages); (2) `stage-lfp` + `INSIDE_PODMAN=1` uses **host-podman** (no empty `INSIDE_SSH` fail); (3) reaffirm filter JS has no box-glyph/`\\-` regex (charCodeAt only — kills SyntaxWarning “geheimsprache” noise). | +| **v1.18.0** | 2026-07-19 | **Feature:** mattermost — client compatibility matrix: **mobile**, **desktop**, **vendor support** floors (default ≥10.11.0, ERROR) + **web** WARN on EOL; env `MATTERMOST_DESKTOP/SUPPORT_MIN_SERVER`, `MATTERMOST_CLIENT_CHECK`. | +| **v1.17.0** | 2026-07-19 | **Feature:** mattermost phase — check **server version** (`X-Version-Id`) against **mobile store min** (default **≥10.11.0**); ERROR when too old (newest Android/iOS apps break); env `MATTERMOST_MOBILE_MIN_SERVER` / `MATTERMOST_MOBILE_CHECK=0`. | +| **v1.16.0** | 2026-07-19 | **Feature:** mon sticky — blue **info** + green **ok** filter badges; suite **version tag under status pill** (not blue badge); filters `#filter-info` / `#filter-ok`. | +| **v1.15.8** | 2026-07-19 | **Bugfix:** mon HTML filter JS — drop literal box-drawing / `\-` regex that caused Python SyntaxWarning (and weird glyphs in suite logs); use Unicode code-point loop instead. | +| **v1.15.7** | 2026-07-19 | **Bugfix:** mon HTML — do not paint «perf/ladder summary» as SUMMARY header (restore 2 INFO lines); drop converter SyntaxWarning/noise from public log; «wrote» on stderr; host-agent PYTHONWARNINGS for convert. | +| **v1.15.6** | 2026-07-19 | **Bugfix:** aptdeploy ERROR jump list is self-contained — systemd httpd failures include before/after state, Result/SubState, failed units, and journal snippets (no more vague «see systemd lines»); ldd/--version errors embed concrete missing libs / exit output. | +| **v1.15.5** | 2026-07-19 | **Bugfix:** devtesting CHF ladder — `ssh` was draining ladder stdin (only rung 1/23 ran → progress 45/98 then snap 52/52); use `ssh -n` + `mapfile`; incomplete ladder is WARN not false complete. | +| **v1.15.4** | 2026-07-19 | **Bugfix:** French sticky titles — page label *surveillance*, stack summary *public + interne*, SUMMARY verdict/totals via i18n (`tout est clair`, `totaux :`). | +| **v1.15.3** | 2026-07-19 | **Bugfix:** console language auto-**fr** for `*lefrancpaysan*` domains again — env `TALER_MON_LANG=en` no longer locks English; only `--lang` / `TALER_MON_LANG_SET=1` is explicit. | +| **v1.15.2** | 2026-07-19 | **Bugfix:** warn/error filter — drop blank log lines from HTML; compact console (no black voids above/mid/below); context skips SUMMARY chrome & pure box frames; tighter filter-gap. | +| **v1.15.1** | 2026-07-19 | **Bugfix:** mon HTML — phase `section()` boxes no longer painted as SUMMARY (were blue/green `sum-header`); only real SUMMARY rows get solid severity backgrounds (OK/WARN/ERROR/INFO). | +| **v1.15.0** | 2026-07-19 | **Feature:** CHF **fake-incoming amount ladder** in `devtesting` (`DEVTESTING_LADDER=1`, default **23** log-spaced rungs **CHF:0.01 → CHF:50000**); synthetic geniban debit funds each rung; baseline plumbing probe kept. | +| **v1.14.0** | 2026-07-19 | **Feature:** SUMMARY box rows are **colour-distinct** (per-severity badge + tinted background + bold counts; verdict-coloured header); HTML mon pages style `sum-ok` / `sum-warn` / `sum-err` / … panels the same way. | +| **v1.13.13** | 2026-07-19 | **Bugfix:** progress estimate for **devtesting** / fake-franken is 8 (happy-path numbered checks), not 10 — avoids mid-run 42/48 then end snap 46/46. | +| **v1.13.12** | 2026-07-19 | **Bugfix:** SUMMARY chrome (`[ WARN ] 2`, `warnings only`, OK/ERROR counts without `#NNN`) is **meta**, not warn/error — sticky counts + warn filter no longer double SUMMARY rows; filter context skips blank lines. | +| **v1.13.11** | 2026-07-19 | **Bugfix:** sticky error/warn filter keeps **±3 dimmed context lines** around each match (gray/grayscale) so phase/group context stays readable; dashed gap between distant clusters. | +| **v1.13.10** | 2026-07-19 | **Bugfix:** multi-phase runs print **one global SUMMARY** at parent end (aggregated OK/ERROR/WARN/INFO/BLOCK via `TALER_MON_STATE`); no phase SUMMARY at all (not even last phase); mid-phase still lists ERRORS/BLOCKERS; solo `check_*.sh` unchanged. | +| **v1.13.9** | 2026-07-19 | **Bugfix:** no mid-run SUMMARY / progress standings between phases (no more 34/52 + SUMMARY after urls while monpages still runs); full stand only on last phase or solo check script; `progress_finish` remains. | +| **v1.13.8** | 2026-07-19 | **Bugfix:** monpages **TOP** meta/marker failures (missing generated/version-link/taler-mon-page, empty race body) are **WARN only** — no longer self-red the run; optional `MONPAGES_TOP_STRICT=1` restores ERROR. | +| **v1.13.7** | 2026-07-19 | **Bugfix:** move optional/meta scripts under `meta/` (deprecated mail/mm units, firecuda outside-runner, legacy multi-host site-gen); production path stays host-agent + `site-gen/console_to_html.py`. | +| **v1.13.6** | 2026-07-19 | **Bugfix:** machine paths and SSH hosts only via env (`~/.config/taler-monitoring/env` / `taler-monitoring-env`); suite loads env before defaults; no `/home/hernani` or bogus host hardcodes. | +| **v1.13.5** | 2026-07-19 | **Bugfix:** progress bar — env-aware phase estimates (CHECK_BANK/LANDING, mon hosts); re-fit total after each phase (done+remaining); drop fake +12 for host-agent monpages post-check (fixes stage 42/139→46/46 snap). | +| **v1.13.4** | 2026-07-19 | **Bugfix:** monpages race-proof — atomic HTML write; rsync `--delay-updates` + settle; fetch retries; larger head windows; pre-publish soft content (`MONPAGES_PRE_PUBLISH`); bootstrap markers. | +| **v1.13.3** | 2026-07-19 | **Bugfix:** sticky **generated** time in **Europe/Zurich** (CEST/CET · MESZ/MEZ), not UTC `…Z`; ISO keeps offset for age JS. | +| **v1.13.2** | 2026-07-19 | **Bugfix:** sticky version link uses **commit** URL (`/src/commit/`) not `/src/tag/v…` (tags not yet pushed → Forgejo fat 404); tags v1.11–v1.13.1 pushed to origin. | +| **v1.13.1** | 2026-07-19 | **Bugfix:** progress bar **global** done/total — phase `summary()` no longer snaps total to phase-local N (fixes 34/34 then 38/47); final `progress_finish` snaps once; header shows 0/N. | +| **v1.13.0** | 2026-07-19 | **Feature:** phase **devtesting** / **franken** — fake-franken CHF via `rusty.taler-ops.ch` (`taler-devtesting` geniban + fake-incoming); stage host-agent can include it. | +| **v1.12.0** | 2026-07-19 | **Feature:** collapsible **monitoring env context** (agent header: env_file, suite pin, domain/hosts/flags) — `
` default closed; summary shows agent · domain · suite. | +| **v1.11.0** | 2026-07-19 | **Feature:** sticky **error/warn filter** — click counts to show only error or only warn log lines; sticky bar + overviews stay; `#filter-error` / `#filter-warn` (+ `#first-*` still works). | +| **v1.10.11** | 2026-07-19 | **Bugfix:** surface/aptdeploy wrappers **force** MON_HOSTS+PHASES+MONPAGES_INVENTORY (env cannot expand job to bank/exchange); wrap more mon knobs. | +| **v1.10.10** | 2026-07-19 | **Bugfix:** monpages bare code 21 is WARN again by default (slash form remains ERROR); bare redir still needs live Caddy `redir *` — set MONPAGES_BARE_STRICT=1 to enforce. | +| **v1.10.9** | 2026-07-19 | **Bugfix:** stats policy INFO no longer says bare “ERROR” (false #first-error on www.stats-01); extract_errors only from `┌ ERROR` badges. | +| **v1.10.8** | 2026-07-19 | **Docs/clarity:** only three public mon URL families (`/monitoring`, `/taler-monitoring-surface`, `/taler-monitoring-aptdeploy`) each bare+slash(+_err); suite name ≠ path; no `/taler-monitoring` page. | +| **v1.10.7** | 2026-07-19 | **Bugfix:** stop always-rsync of **stale suite-overlay** over Forgejo tip (broke stage sticky / new site-gen). Overlay only when suite missing phases. | +| **v1.10.6** | 2026-07-19 | **Fix:** stage mon HTML quality — suite update out of public log; filter git noise; stage-specific sticky (not GOA); `CHECK_BANK=0` / `MERCHANT_REQUIRED` for mytops stage scope. | +| **v1.10.5** | 2026-07-19 | **Fix:** aptdeploy monpages **job-only** (own `/taler-monitoring-aptdeploy*`); full GOA mon inventory + bare URL checks stay on **hacktivism** general run — no bank/exchange/surface monpages noise on aptdeploy HTML. | +| **v1.10.4** | 2026-07-19 | **Fix:** suite **auto-upgrade strict** (`SUITE_UPDATE_STRICT=1` default — abort on fetch fail); monpages ERROR on **bootstrap** HTML; ship mytops-stage timer/service units; host-agent refuses stale tree. | +| **v1.10.3** | 2026-07-19 | **Bugfix:** sticky ERROR only from check badges (`┌ ERROR`) / `ERROR monpages`; ignore git `HEAD is now at …` commit subjects that mention ERROR. | +| **v1.10.2** | 2026-07-19 | **Bugfix:** sticky ERROR count — classify with word-boundary `\bERROR\b` (not substring); avoids false red from git “wrote-errors” commit lines and similar. | +| **v1.10.1** | 2026-07-19 | **Bugfix:** HTML converter must not treat teed `wrote … (errors=N)` lines as ERROR (false sticky red after clean host-agent runs). | +| **v1.10.0** | 2026-07-19 | **Feature:** aptdeploy **container overview board** — inventory + per-pod matrix (state, merchant, httpd, ldd, OK/ERROR) in log and HTML sticky/top (`#container-overview`). | +| **v1.9.4** | 2026-07-19 | **Fix:** restore **taler-monitoring-aptdeploy(+_err)** as first-class GOA mon page (catalog + monpages path allow + surface docs). Mail/mattermost stay folded into surface. | +| **v1.9.3** | 2026-07-19 | **Bugfix (suite):** monpages merchant **code 21** always ERROR (bare no longer soft-WARN); apply hints use absolute `/home/hernani/koopa-caddy/…`; ship `host-agent/apply-monitoring-live.sh` (smoke array, bare check); fix ROOT-ON-KOOPA wrong `redir /path/ 302` examples. | +| **v1.9.2** | 2026-07-19 | **Bugfix:** Caddy bare-path redir footgun — inside `handle` use `redir * /path/ 302` (not `redir /path/ 302`, which becomes `Location: 302` and bare URLs fall through to merchant **code 21**). Snippet + ROOT-APPLY: absolute apply path (`/home/hernani/koopa-caddy/…`, not `~` as root). | +| **v1.9.1** | 2026-07-19 | **Bugfix:** stale path confusions in repo — sticky `pages_i3` no longer lists mail/mattermost/aptdeploy mon pages; lib.sh/android-test drop `scripts/taler-monitoring/`; monpages/host-agent hints match v1.8 layout; clarify GIT vs GUI vs CLI `*AUTOMATION-NOTES.md` (three files by design). | +| **v1.9.0** | 2026-07-19 | **monpages content:** ERROR if page missing or markers incomplete — top (sticky-bar/status-bar/generated) + bottom (footer/`taler-mon:bottom`); mid-run (progress <100% / incomplete meta) relaxes bottom only. Surface required on normal GOA runs with meaningful body (not progress-only). HTML emits `taler-mon:top` / `taler-mon:bottom:complete|incomplete`. | +| **v1.8.0** | 2026-07-19 | **Simplified public pages:** only `/taler-monitoring-surface(+_err)` on `taler.hacktivism.ch` for ecosystem software/versions; **9 landing stacks** each keep `/monitoring(+_err)`. Mail (firefly.gnunet.org + anastasis.taler-systems.com) and Mattermost folded into surface job; separate mail/mattermost mon pages + timers deprecated. **nmap** OS fingerprinting on surface (`SURFACE_NMAP`); **ERROR** when packages are behind (`TALER_PKG_BEHIND=error` default) or OS fingerprint is EOL. | +| **v1.8.1** | *planned* | **L10n** packs: **fr-CH** (FrancPaysan) and **de-CH** (hacktivism / Taler CH German). | +| **v1.7.7** | 2026-07-18 | **domains:** stage mytops merchant FQDN is `stage.my.taler-ops.ch` (betel mon sites). | +| **v1.9.1** | 2026-07-19 | **Bugfix:** stale path confusions in repo — sticky `pages_i3` no longer lists mail/mattermost/aptdeploy mon pages; lib.sh/android-test drop `scripts/taler-monitoring/`; monpages/host-agent hints match v1.8 layout; clarify GIT vs GUI vs CLI `*AUTOMATION-NOTES.md` (three files by design). | +| **v1.9.0** | 2026-07-19 | **monpages content:** ERROR if page missing or markers incomplete — top (sticky-bar/status-bar/generated) + bottom (footer/`taler-mon:bottom`); mid-run (progress <100% / incomplete meta) relaxes bottom only. Surface required on normal GOA runs with meaningful body (not progress-only). HTML emits `taler-mon:top` / `taler-mon:bottom:complete|incomplete`. | +| **v1.8.0** | 2026-07-18 | **monpages obligatory (ERROR):** GOA checks full suite inventory (bank/exchange/taler `/monitoring/` + surface/aptdeploy/mattermost/mail + on-disk discovery); FP only its own mon hosts; `MONPAGES_REQUIRE_PUBLIC=1` default (escape hatch `=0`). **Bugfix:** `with_timeout` no longer re-runs a phase when it exits non-zero. | +| **v1.9.1** | 2026-07-19 | **Bugfix:** stale path confusions in repo — sticky `pages_i3` no longer lists mail/mattermost/aptdeploy mon pages; lib.sh/android-test drop `scripts/taler-monitoring/`; monpages/host-agent hints match v1.8 layout; clarify GIT vs GUI vs CLI `*AUTOMATION-NOTES.md` (three files by design). | +| **v1.9.0** | 2026-07-19 | **monpages content:** ERROR if page missing or markers incomplete — top (sticky-bar/status-bar/generated) + bottom (footer/`taler-mon:bottom`); mid-run (progress <100% / incomplete meta) relaxes bottom only. Surface required on normal GOA runs with meaningful body (not progress-only). HTML emits `taler-mon:top` / `taler-mon:bottom:complete|incomplete`. | +| **v1.8.0** | 2026-07-18 | **Bugfix:** monpages soft mode `MONPAGES_REQUIRE_PUBLIC=0` (FP wrappers default) — missing public HTML is WARN until vhost publish; generic publish hints. | +| **v1.7.4** | 2026-07-18 | **Bugfix:** monpages bare URLs (`/monitoring` without `/`) WARN by default (trailing slash remains ERROR); avoids false red when slash form is live. | +| **v1.7.3** | 2026-07-18 | Monitoring **pages themselves** as suite scope: README section + sticky “What this monitors” block (monpages / public HTML paths). | +| **v1.7.2** | 2026-07-18 | **Bugfix:** Caddy static handles — `root` must be host dir + matcher `/monitoring*` (not leaf dir + `/monitoring/`), else file_server returns empty 404 after apply. | +| **v1.7.1** | 2026-07-18 | Host-agent default `STRICT_EXIT=1`; monpages staging-vs-public diagnosis; re-htmlify after public check fails. | +| **v1.7.0** | 2026-07-18 | CLI `--ver` / `--version` / `-V`: suite version, git commit, repo/tag/commit URLs. | +| **v1.6.0** | 2026-07-18 | **Mail monitoring:** phase `mail` (firefly + pixel/TSA); MX/SMTP/IMAP/SPF/DMARC; `/taler-monitoring-mail*`. | +| **v1.5.0** | 2026-07-18 | **Mattermost monitoring:** phase `mattermost` for `mattermost.taler.net`; `/taler-monitoring-mattermost*`. | +| **v1.4.1** | 2026-07-18 | Generic docs (no client hostnames); canonical install path `~/src/taler-monitoring`. | +| **v1.4.0** | 2026-07-18 | **MAJOR.FEATURE.FIX** version scheme formalized (three-component tags + `VERSION` file). | +| **v1.3.4** | 2026-07-18 | Summary i18n, monpages code-21 detection, aptdeploy rsync path. | +| **v1.3.3** | 2026-07-18 | GIT notes vs GUI notes; monpages in TESTS. | +| **v1.3.2** | 2026-07-18 | Suite paths only `~/src/taler-monitoring` (no admin-log suite tree). | +| **v1.3.1** | 2026-07-18 | monpages FQDN check + deploy no longer silent-skip. | +| **v1.3.0** | 2026-07-18 | Sticky bar suite version + Forgejo tag link. | +| **v1.2.0** | 2026-07-18 | Language selection: `--lang` / `TALER_MON_LANG` (en\|fr). | +| **v1.1.0** | 2026-07-18 | i18n chrome: English default; French for FrancPaysan. | +| **v1.0.0** | 2026-07-18 | Initial standalone release. | -## Language (v1.1+) +## Sticky bar version (v1.3.0+) -| Context | Language | -|---------|----------| -| Default / GOA / hacktivism | **English** | -| FrancPaysan prod + stage (`*lefrancpaysan*`) | **French** (auto or `TALER_MON_LANG=fr`) | +Generated HTML shows the installed release tag (e.g. `v1.8.0`) and links to Forgejo: -```bash -# force -TALER_MON_LANG=fr ./taler-monitoring.sh -d lefrancpaysan.ch urls -TALER_MON_LANG=en ./taler-monitoring.sh -d lefrancpaysan.ch urls -``` +`https://git.hacktivism.ch/hernani/taler-monitoring/src/tag/` -Host-agent env examples: -- `host-agent/env/francpaysan.env.example` → `TALER_MON_LANG=fr` -- `host-agent/env/stagepaysan.env.example` → `TALER_MON_LANG=fr` -- `host-agent/env/hacktivism.env.example` → `TALER_MON_LANG=en` +Resolved via `git describe --tags` / `VERSION`. + +## Public monitoring pages (v1.3.1+) + +Publish under `DEPLOY_WWW_ROOT` and Caddy `handle`s. Missing handles → merchant **JSON code 21**. +See koopa-admin-log issue **I-09** / `apply-monitoring-live.sh`. + +| Check | How | +|-------|-----| +| Phase | `./taler-monitoring.sh monpages` | +| CLI version | `./taler-monitoring.sh --ver` | + +## Language (v1.2.0+) + +| Method | Example | +|--------|---------| +| CLI | `./taler-monitoring.sh --lang fr …` | +| Env | `TALER_MON_LANG=en` or `fr` | +| Auto | `*lefrancpaysan*` → fr, else en | + +Full locale packs (fr-CH / de-CH): **v1.8.0** (planned). ## Pin a release ```bash git clone https://git.hacktivism.ch/hernani/taler-monitoring.git ~/src/taler-monitoring -cd ~/src/taler-monitoring && git checkout v1.1 +cd ~/src/taler-monitoring && git checkout v1.9.1 ``` - -Default host-agent tracks `origin/main`. Pin with `SUITE_GIT_REF=v1.1` in `~/.config/taler-monitoring/env`. diff --git a/android-test/GUI-AUTOMATION-NOTES.md b/android-test/GUI-AUTOMATION-NOTES.md index 18fe398..333179f 100644 --- a/android-test/GUI-AUTOMATION-NOTES.md +++ b/android-test/GUI-AUTOMATION-NOTES.md @@ -1,5 +1,10 @@ # GUI automation notes (Android wallet) + +Companion docs: +- CLI / wallet-cli: [`../CLI-AUTOMATION-NOTES.md`](../CLI-AUTOMATION-NOTES.md) +- Git / tags / suite tree: [`../VERSIONS.md`](../VERSIONS.md) + Canonical notes for **graphical** Android wallet tests against GOA / stage. --- @@ -87,7 +92,7 @@ Host-CLI-Settlement. - iOS / Dual-Platform → andere Repos/Flags, nicht diese Skripte. ```bash -cd scripts/taler-monitoring/android-test +cd ~/src/taler-monitoring/android-test # Minimal hybrid (install + deep-link + light taps) STACK=goa ./run-android-pay-smoke.sh @@ -111,7 +116,7 @@ GUI=1 ./run-android-build-and-smoke.sh ## Platform capability flags (planned + defaults) Automation can run on hosts that support **Android only**, **GUI tooling only** -(conceptual), or **both**. Flags keep that explicit for CI and laptops. +(conceptual), or **both**. Flags keep that explicit for CI and workstations. ### Proposed env flags @@ -147,7 +152,7 @@ with **`-no-window`** and still keep **graphics logic** in the guest: # or let smoke auto-start headless AVD when no phone is plugged in STACK=goa ./run-android-gui-smoke.sh -# laptop: show the emulator window +# ops workstation: show the emulator window WINDOWED=1 ./start-android-emulator.sh --wait # or: EMULATOR_HEADLESS=0 EMULATOR_GPU=host ./start-android-emulator.sh --wait ``` diff --git a/android-test/README.md b/android-test/README.md index 8ed2fb0..977ff0c 100644 --- a/android-test/README.md +++ b/android-test/README.md @@ -104,7 +104,7 @@ Canonical notes (minimal variant, **flags**, Linux defaults): ```bash # headless AVD only (default path for servers) ./start-android-emulator.sh --wait -# windowed (laptops) +# windowed (workstations with display) WINDOWED=1 ./start-android-emulator.sh --wait ``` diff --git a/android-test/run-goa-gui-chain.sh b/android-test/run-goa-gui-chain.sh index 1c20a63..5009b03 100755 --- a/android-test/run-goa-gui-chain.sh +++ b/android-test/run-goa-gui-chain.sh @@ -91,7 +91,7 @@ if [[ -z "${EXP_PW_FILE:-}" ]]; then fi # stage: try stagepaysan secret via ssh if missing if [[ ! -f "${EXP_PW_FILE:-}" && "$STACK" = "stage" ]]; then - for host in francpaysan-stage-user francpaysan-host; do + for host in ${INSIDE_SSH:+"$INSIDE_SSH"} ${FRANCPAYSAN_SSH:+"$FRANCPAYSAN_SSH"}; do tmp=$(mktemp) if ssh -o BatchMode=yes -o ConnectTimeout=10 "$host" \ 'tr -d "\n\r" /dev/null || true + +# Area ladder.* — withdraw/pay amount ladder (any EXPECT_CURRENCY / stack) +# Groups: ladder.plan / ladder.load / ladder.withdraw / ladder.pay / ladder.report +set_area ladder +set_group plan +SECTION_T0=$(date +%s) +now_ms() { python3 -c 'import time; print(int(time.time()*1000))'; } +elapsed_ms() { + # elapsed_ms START_MS + python3 -c 'import sys; print(int(sys.argv[1]) - int(sys.argv[2]))' "$(now_ms)" "$1" +} + +: "${LADDER_TIMEOUT_S:=3600}" +: "${LADDER_SETTLE_ROUNDS:=18}" +: "${LADDER_SETTLE_SLEEP:=2}" +: "${EXP_USER:=explorer}" +# EXP_PW_FILE / EXP_PW optional overrides; otherwise read_secret + SECRETS_ROOT / stage SSH +: "${EXP_PW_FILE:=}" +: "${EXP_PW:=}" +: "${CLI_JS:=}" +# GOA libeufin amount ceiling (hacktivism). Stage auto-replaces via bank /config. +: "${LADDER_MAX_AMOUNT:=4503599627370496}" +: "${LADDER_STEPS:=23}" +: "${LADDER_LOAD:=1}" +: "${LADDER_PAY:=1}" +: "${LADDER_STACK_AUTO:=1}" +# Withdraw mids = pay mids × scale (so wallet can afford the pay ladder) +: "${LADDER_WITHDRAW_SCALE:=1.5}" +# Floor for random mids (must be ≥ smallest exchange coin; GOA min denom ≈ 0.000001) +: "${LADDER_MIN_AMOUNT:=0.000001}" +: "${LADDER_CONFIRM_POLLS:=40}" +: "${LADDER_PAY_SETTLE_ROUNDS:=6}" +: "${MERCHANT_INSTANCE:=goa-demo-cp4zqk}" +# Public variable template for stage pays (optional; fixed templates used when amount maps) +: "${LADDER_PAY_TEMPLATE:=}" +: "${LADDER_PAY_TEMPLATE_INSTANCE:=fermes-des-collines}" +# Free-amount public template (hacktivism: goa-free on goa-demo-cp4zqk — any GOA amount) +: "${LADDER_FREE_TEMPLATE:=goa-free}" +: "${LADDER_FREE_TEMPLATE_INSTANCE:=}" +# 1 = create pays via public free template POST (amount in body); 0 = private orders only +: "${LADDER_USE_FREE_TEMPLATE:=1}" +# classic | max (max = highest withdrawable + highest payable; phase alias max-ladder) +: "${LADDER_MODE:=classic}" +# max-search: probe budget (0 = run until convergence / LADDER_TIMEOUT_S only) +: "${LADDER_MAX_PROBES:=32}" +: "${LADDER_MAX_TOL_REL:=0.001}" +# first N probes = random high (log-uniform); then log-bisect between lo/hi +: "${LADDER_HIGH_RUNGS:=6}" +# optional seed for reproducible max-search (empty = time-based) +: "${LADDER_MAX_SEED:=}" +# After withdraw, wait for spendable available>0 before pay (secs total; shows countdown) +: "${LADDER_PAY_WAIT_AVAILABLE_S:=180}" + +# Historic libeufin-ish absolute ceiling used as default LADDER_MAX_AMOUNT on GOA +LADDER_ABS_CEILING="4503599627370496" +GOA_LADDER_CEILING="${LADDER_ABS_CEILING}" # back-compat alias + +# Resolve wallet-cli .mjs (no hardcoded laptop path) +if [ -z "${CLI_JS}" ] || [ ! -f "${CLI_JS}" ]; then + CLI_JS=$(find_wallet_cli 2>/dev/null || true) +fi +if [ -z "${CLI_JS}" ] || [ ! -f "${CLI_JS}" ]; then + # PATH wrapper last resort + if command -v taler-wallet-cli >/dev/null 2>&1; then + CLI_JS="" + else + err prereq "taler-wallet-cli not found" \ + "problem: no CLI_JS / WALLET_CLI .mjs and taler-wallet-cli not on PATH — accept-uri/pay cannot run (payments stay stale). Install wallet-cli (e.g. ~/.local/bin/taler-wallet-cli.mjs) + taler-helper-sqlite3; set WALLET_CLI= in ~/.config/taler-monitoring/env" + exit 1 + fi +fi +# helper is required for real wallet DB (otherwise accept "works" empty / no coins) +if ! command -v taler-helper-sqlite3 >/dev/null 2>&1; then + err prereq "taler-helper-sqlite3 not on PATH" \ + "problem: wallet-cli needs taler-helper-sqlite3 for sqlite backend — put it on PATH next to wallet-cli" + exit 1 +fi + +CUR="${EXPECT_CURRENCY:-GOA}" +BANK="${BANK_PUBLIC%/}" +EX="${EXCHANGE_PUBLIC%/}/" +MER="${MERCHANT_PUBLIC%/}" +INST="${MERCHANT_INSTANCE}" + +# --- Stack-specific ladder maxima (TESTPAYSAN stage) --- +# Uses bank max_wire_transfer_amount and exchange min denom when still on GOA defaults. +apply_ladder_stack_defaults() { + [ "${LADDER_STACK_AUTO}" = "1" ] || return 0 + if [ "${CUR}" != "TESTPAYSAN" ]; then + case "${TALER_DOMAIN:-}" in + stage.lefrancpaysan.ch|stage.bank.lefrancpaysan.ch|stage.exchange.lefrancpaysan.ch|stage.monnaie.lefrancpaysan.ch) + CUR="TESTPAYSAN" + ;; + *) return 0 ;; + esac + fi + + local bank_cfg max_wire min_denom + bank_cfg=$(curl -sS -m 12 "${BANK}/config" 2>/dev/null || true) + max_wire=$(printf '%s' "$bank_cfg" | python3 -c ' +import json,sys +try: + d=json.load(sys.stdin) +except Exception: + print("") + raise SystemExit(0) +a=d.get("max_wire_transfer_amount") or "" +print(a.split(":",1)[-1] if a else "") +' 2>/dev/null || true) + + min_denom=$(curl -sS -m 25 -H 'Accept: application/json' "${EX%/}/keys" 2>/dev/null | python3 -c ' +import json,sys +try: + d=json.load(sys.stdin) +except Exception: + print("") + raise SystemExit(0) +den=d.get("denoms") or d.get("denominations") or [] +vals=[] +for x in den if isinstance(den,list) else []: + v=x.get("value") or x.get("amount") or "" + if not v: continue + try: vals.append(float(str(v).split(":")[-1])) + except Exception: pass +print(min(vals) if vals else "") +' 2>/dev/null || true) + + # Fallback maxima if public config unreachable + [ -n "$max_wire" ] || max_wire="2000" + [ -n "$min_denom" ] || min_denom="0.01" + + # Only rewrite when caller left GOA defaults (explicit LADDER_MAX_AMOUNT=… kept) + if [ -n "$max_wire" ] && { [ "${LADDER_MAX_AMOUNT}" = "${GOA_LADDER_CEILING}" ] || [ "${LADDER_MAX_AMOUNT}" = "${LADDER_ABS_CEILING}" ] || [ "${LADDER_MAX_AMOUNT}" = "4503599627370496" ]; }; then + LADDER_MAX_AMOUNT="$max_wire" + fi + if [ -n "$min_denom" ] && { [ "${LADDER_MIN_AMOUNT}" = "0.000001" ] || [ "${LADDER_MIN_AMOUNT}" = "0.00000001" ]; }; then + LADDER_MIN_AMOUNT="$min_denom" + fi + # Slightly fewer rungs on stage (full GOA 23 still ok if user set LADDER_STEPS) + if [ "${LADDER_STEPS}" = "23" ]; then + LADDER_STEPS=15 + fi + # Stage farmer shops: private goa-demo instance does not exist + if [ "${MERCHANT_INSTANCE}" = "goa-demo-cp4zqk" ]; then + MERCHANT_INSTANCE="${LADDER_PAY_TEMPLATE_INSTANCE:-fermes-des-collines}" + INST="$MERCHANT_INSTANCE" + fi + # Prefer public templates for pays when no merchant token (set later) + : "${E2E_USE_TEMPLATES:=1}" + export E2E_USE_TEMPLATES + export LADDER_MAX_AMOUNT LADDER_MIN_AMOUNT LADDER_STEPS MERCHANT_INSTANCE + info "ladder stack" "TESTPAYSAN · max=${CUR}:${LADDER_MAX_AMOUNT} min=${CUR}:${LADDER_MIN_AMOUNT} steps=${LADDER_STEPS} (from bank max_wire / keys denoms)" +} +apply_ladder_stack_defaults + +# Normalize mode (max-ladder phase / aliases) +case "${LADDER_MODE}" in + max|MAX|max-search|maxsearch|find-max|findmax) LADDER_MODE=max ;; + classic|CLASSIC|default|plan|"" ) LADDER_MODE=classic ;; + *) + warn ladder "unknown LADDER_MODE=${LADDER_MODE} — using classic" + LADDER_MODE=classic + ;; +esac +# max mode: withdraw + pay hunts (LADDER_PAY=0 still disables pay entirely) +: "${LADDER_FREE_TEMPLATE_INSTANCE:=${MERCHANT_INSTANCE}}" +export LADDER_MODE LADDER_MAX_PROBES LADDER_MAX_TOL_REL LADDER_HIGH_RUNGS LADDER_MAX_SEED LADDER_PAY +export LADDER_FREE_TEMPLATE LADDER_FREE_TEMPLATE_INSTANCE LADDER_USE_FREE_TEMPLATE LADDER_PAY_WAIT_AVAILABLE_S + +SCRATCH=$(mktemp -d) +WDB="$SCRATCH/wallet.sqlite3" +REPORT_DIR="${LADDER_REPORT_DIR:-$SCRATCH}" +mkdir -p "$REPORT_DIR" +TSV="$REPORT_DIR/ladder-results.tsv" +PAY_TSV="$REPORT_DIR/ladder-pay-results.tsv" +JSON="$REPORT_DIR/ladder-report.json" +LOAD_BEFORE="$REPORT_DIR/load-before.json" +LOAD_AFTER="$REPORT_DIR/load-after.json" +echo -e "rung\trange\tamount\tstatus\tms_mint\tms_accept\tms_confirm\tms_settle\tms_total\twid\tnote" >"$TSV" +echo -e "rung\trange\tamount\tstatus\tms_order\tms_handle\tms_settle\tms_total\toid\tnote" >"$PAY_TSV" + +ladder_over() { + local now + now=$(date +%s) + [ $((now - SECTION_T0)) -ge "$LADDER_TIMEOUT_S" ] +} + +wcli() { + # taler-helper-sqlite3 is Python ≥3.11; prefer Homebrew/local python on macOS + # so Apple /usr/bin/python3 3.9 does not FATAL the sqlite backend (no reservePub). + local _path="${PATH:-}" + case ":${_path}:" in + *:/opt/homebrew/bin:*) ;; + *) _path="/opt/homebrew/bin:/usr/local/bin:${_path}" ;; + esac + if [ -n "${CLI_JS:-}" ] && [ -f "$CLI_JS" ]; then + PATH="$_path" node "$CLI_JS" --wallet-db="$WDB" --no-throttle "$@" + else + PATH="$_path" taler-wallet-cli --wallet-db="$WDB" --no-throttle "$@" + fi +} + +# Explorer pool password: env → file override → stack secrets → SSH +resolve_explorer_pw() { + local pw="" f="" host="" + if [ -n "${EXP_PW:-}" ]; then + printf '%s' "$EXP_PW" + return 0 + fi + if [ -n "${EXP_PW_FILE:-}" ] && [ -f "$EXP_PW_FILE" ]; then + tr -d '\n\r' <"$EXP_PW_FILE" + return 0 + fi + + # Stage TESTPAYSAN — never use GOA koopa-admin-secrets explorer pw first + if [ "${CUR}" = "TESTPAYSAN" ] \ + || [[ "${TALER_DOMAIN:-}" == stage.*lefrancpaysan* ]] \ + || [[ "${TALER_DOMAIN:-}" == *stage.lefrancpaysan* ]]; then + for f in \ + ${FRANCPAYSAN_SECRETS:+"${FRANCPAYSAN_SECRETS}/stage/bank-explorer-password.txt"} \ + "${HOME}/.config/taler-landing/stage-bank-explorer-password.txt" + do + [ -n "$f" ] && [ -f "$f" ] || continue + tr -d '\n\r' <"$f" + return 0 + done + _remote_exp="${FRANCPAYSAN_REMOTE_BANK_EXPLORER_SECRET:-}" + [ -z "$_remote_exp" ] && [ -n "${FRANCPAYSAN_REMOTE_SECRETS_ROOT:-}" ] && \ + _remote_exp="${FRANCPAYSAN_REMOTE_SECRETS_ROOT}/bank/secrets/bank-explorer-password.txt" + for host in ${INSIDE_SSH:+"$INSIDE_SSH"} ${FRANCPAYSAN_SSH:+"$FRANCPAYSAN_SSH"}; do + [ -n "$host" ] && [ -n "$_remote_exp" ] || continue + pw=$(ssh -o BatchMode=yes -o ConnectTimeout=12 "$host" \ + "tr -d '\\n\\r' <${_remote_exp} 2>/dev/null || sudo tr -d '\\n\\r' <${_remote_exp} 2>/dev/null" \ + 2>/dev/null || true) + if [ -n "$pw" ]; then + printf '%s' "$pw" + return 0 + fi + done + return 1 + fi + + # GOA / default: SECRETS_ROOT / ~/.config / koopa SSH + if pw=$(read_secret "taler-bank/bank-explorer-password.txt" 2>/dev/null) && [ -n "$pw" ]; then + printf '%s' "$pw" + return 0 + fi + for f in \ + "${SECRETS_ROOT:+${SECRETS_ROOT}/taler-bank/bank-explorer-password.txt}" \ + "${HOME}/.config/taler-landing/bank-explorer-password.txt" + do + [ -n "$f" ] && [ -f "$f" ] || continue + tr -d '\n\r' <"$f" + return 0 + done + return 1 +} + +wallet_avail() { + wcli balance 2>/dev/null | python3 -c ' +import json,sys +t=sys.stdin.read() +dec=json.JSONDecoder() +i=t.find("{") +if i<0: + print("0"); raise SystemExit +try: + d,_=dec.raw_decode(t,i) +except Exception: + # last-resort: brace slice (may fail on trailing logs) + try: + d=json.loads(t[i:t.rfind("}")+1]) + except Exception: + print("0"); raise SystemExit +cur=sys.argv[1] +for b in d.get("balances") or []: + a=b.get("available") or "" + if a.startswith(cur+":"): + print(a.split(":",1)[1]); raise SystemExit +print("0") +' "$CUR" 2>/dev/null || echo "0" +} + +# pendingIncoming for same currency (same wallet DB as withdraw) — "0" if missing +wallet_pending_in() { + wcli balance 2>/dev/null | python3 -c ' +import json,sys +t=sys.stdin.read() +dec=json.JSONDecoder() +i=t.find("{") +if i<0: + print("0"); raise SystemExit +try: + d,_=dec.raw_decode(t,i) +except Exception: + try: + d=json.loads(t[i:t.rfind("}")+1]) + except Exception: + print("0"); raise SystemExit +cur=sys.argv[1] +for b in d.get("balances") or []: + a=b.get("pendingIncoming") or b.get("pending_incoming") or "" + if a.startswith(cur+":"): + print(a.split(":",1)[1]); raise SystemExit +print("0") +' "$CUR" 2>/dev/null || echo "0" +} + +# Build paired pay + withdraw ladders (same step count, shared random shape). +# Pay: [0] + log-uniform mids + [max-1] + [max] +# Wd: [0] + max(mid, mid×scale) mids + [max-1] + [max] (mids higher → enough to spend) +# Writes: $1 = withdraw list file, $2 = pay list file (space-separated CUR:amt lines as one line each) +build_ladder_pair() { + local wd_out="$1" pay_out="$2" + python3 - <<'PY' "$CUR" "${LADDER_MAX_AMOUNT}" "${LADDER_STEPS}" "${LADDER_MIN_AMOUNT}" "${LADDER_WITHDRAW_SCALE}" "$wd_out" "$pay_out" +import math, random, sys +from decimal import Decimal, ROUND_HALF_UP, ROUND_UP +from pathlib import Path + +cur = sys.argv[1] +max_amt = Decimal(sys.argv[2]) +steps = max(1, int(sys.argv[3])) +min_amt = Decimal(sys.argv[4]) +scale = Decimal(sys.argv[5]) +wd_path, pay_path = Path(sys.argv[6]), Path(sys.argv[7]) +if min_amt <= 0: + min_amt = Decimal("0.000001") +if min_amt >= max_amt: + min_amt = max_amt / Decimal(1000) +if scale < 1: + scale = Decimal(1) +max_m1 = max_amt - Decimal(1) if max_amt > 1 else max_amt + +def fmt(v: Decimal) -> str: + q = v.quantize(Decimal("0.00000001"), rounding=ROUND_HALF_UP) + if q == q.to_integral(): + return format(int(q), "d") + return format(q, "f").rstrip("0").rstrip(".") + +def quant(v: Decimal) -> Decimal: + if v >= 1: + return v.quantize(Decimal(1), rounding=ROUND_HALF_UP) + if v < min_amt: + return min_amt + return v.quantize(Decimal("0.00000001"), rounding=ROUND_UP) + +def amt(v: Decimal) -> str: + return "%s:%s" % (cur, fmt(v)) + +def make_mids(n_mid: int, hi_cap: Decimal): + if n_mid <= 0 or hi_cap <= min_amt: + return [] + lo, hi = float(min_amt), float(hi_cap) * 0.999999 + if hi <= lo: + hi = lo * 10 + cuts = sorted(math.exp(random.uniform(math.log(lo), math.log(hi))) for _ in range(n_mid)) + out, prev = [], Decimal(0) + for c in cuts: + v = quant(Decimal(str(c))) + if v <= prev: + step = min_amt if prev < 1 else max(prev * Decimal("1e-6"), Decimal(1)) + v = quant(prev + step) + if v >= hi_cap: + v = quant(hi_cap - (Decimal(1) if hi_cap > 1 else min_amt)) + if v <= prev or v >= hi_cap: + continue + out.append(v) + prev = v + return out + +# Build withdraw ladder first (full range), then pay mids = withdraw/scale +# so each pay mid is always cheaper than the matching withdraw mid. +if steps == 1: + wd_vals = [max_amt] +elif steps == 2: + wd_vals = [Decimal(0), max_amt] +else: + n_mid = max(0, steps - 3) + wd_vals = [Decimal(0)] + make_mids(n_mid, max_m1) + [max_m1, max_amt] + while len(wd_vals) > steps and len(wd_vals) > 3: + wd_vals.pop(len(wd_vals) // 2) + while len(wd_vals) < steps and len(wd_vals) >= 2: + i = max(1, len(wd_vals) - 2) + a, b = wd_vals[i - 1], wd_vals[i] + if a <= 0: + m = max(min_amt, b / 2 if b > 0 else min_amt) + else: + m = (a * b).sqrt() if a * b > 0 else (a + b) / 2 + m = quant(m) + if m <= a or m >= b: + break + wd_vals.insert(i, m) + wd_vals = wd_vals[:steps] + +pay_vals = [] +prev_p = Decimal(-1) +for i, w in enumerate(wd_vals): + is_first = i == 0 + is_last = i == len(wd_vals) - 1 + is_m1 = (not is_last) and w == max_m1 and i == len(wd_vals) - 2 + if is_first and w == 0: + pay_vals.append(Decimal(0)) + prev_p = Decimal(0) + continue + if is_last: + pay_vals.append(max_amt) + continue + if is_m1 or w == max_m1: + pay_vals.append(max_m1) + prev_p = max_m1 + continue + # pay mid = withdraw / scale (strictly less funding needed per step) + p = quant(w / scale) if scale > 0 else w + if p < min_amt and w >= min_amt: + p = min_amt + if p <= prev_p: + p = quant(prev_p + (min_amt if prev_p < 1 else Decimal(1))) + if p >= w: + # keep pay strictly below this withdraw rung when possible + p = quant(w - (Decimal(1) if w > 1 else min_amt)) if w > prev_p else prev_p + if p <= prev_p: + p = prev_p # flat ok only if stuck; still ≤ w + pay_vals.append(p) + prev_p = p + +assert len(pay_vals) == len(wd_vals) +# sanity: every non-pin pay mid ≤ matching withdraw mid +for i, (p, w) in enumerate(zip(pay_vals, wd_vals)): + if i == 0 or i >= len(wd_vals) - 2: + continue + if p > w: + pay_vals[i] = w + +wd_path.write_text(" ".join(amt(v) for v in wd_vals) + "\n") +pay_path.write_text(" ".join(amt(v) for v in pay_vals) + "\n") +print(" ".join(amt(v) for v in wd_vals)) +PY +} + +# shellcheck source=metrics.sh +source "$ROOT/metrics.sh" +# Payment module (classic + max-search payable) +# shellcheck source=ladder/lib_pay.sh +source "$ROOT/ladder/lib_pay.sh" +METRICS_DIR="$REPORT_DIR" +ALT_UNITS_FILE="${REPORT_DIR}/alt_unit_names.json" +export METRICS_DIR CUR WDB CLI_JS ALT_UNITS_FILE +# Ladder can disable host load without killing coin metrics +if [ "${LADDER_LOAD:-1}" = "0" ]; then + METRICS_LOAD=0 + export METRICS_LOAD +fi + +if [ "${LADDER_MODE}" = "max" ]; then + section "ladder · ${CUR} max-search (find highest withdrawable)" +else + section "ladder · ${CUR} classic (0 → random → max · ${LADDER_STEPS} steps)" +fi +info "bank" "$BANK" +info "exchange" "$EX" +info "currency" "$CUR" +info "ladder mode" "${LADDER_MODE}" +# alt_unit_names for human amounts (Kilo-GOA / Mega-GOA / … + base in parentheses) +if metrics_load_alt_units "${EX%/}/config"; then + info "alt_unit_names" "from ${EX%/}/config → $ALT_UNITS_FILE" +else + warn "alt_unit_names" "using built-in SI fallback ($ALT_UNITS_FILE)" +fi +info "budget" "${LADDER_TIMEOUT_S}s" +_max_m1=$(python3 -c 'import sys; from decimal import Decimal; m=Decimal(sys.argv[1]); print(m-1 if m>1 else m)' "${LADDER_MAX_AMOUNT}" 2>/dev/null || echo "${LADDER_MAX_AMOUNT}-1") +if [ "${LADDER_MODE}" = "max" ]; then + info "max-search" "probes≤${LADDER_MAX_PROBES} high_rand=${LADDER_HIGH_RUNGS} tol=${LADDER_MAX_TOL_REL} ceiling=${CUR}:${LADDER_MAX_AMOUNT}" +else + info "steps" "${LADDER_STEPS} (0 + random≥${LADDER_MIN_AMOUNT} + max-1=${CUR}:${_max_m1} + max=${CUR}:${LADDER_MAX_AMOUNT})" + info "withdraw_scale" "${LADDER_WITHDRAW_SCALE}× pay mids (fund pay ladder)" +fi +info "pay_phase" "$([ "${LADDER_PAY}" = "1" ] && echo enabled || echo disabled)" +info "currency/domain" "${CUR} · ${TALER_DOMAIN:-?} · bank=${BANK}" + +set_group load +section "ladder · load snapshot (before withdraws)" +metrics_report_load "$LOAD_BEFORE" "ladder-start" || true +# Fresh wallet per rung — baseline empty (or last-rung DB if re-used later) +metrics_report_coins "ladder-start" || true + +if ! EXP_PW=$(resolve_explorer_pw); then + err bank "explorer password missing" \ + "set EXP_PW / EXP_PW_FILE or SECRETS_ROOT=…/koopa/host-root (taler-bank/bank-explorer-password.txt)${SECRETS_ROOT:+ · SECRETS_ROOT=${SECRETS_ROOT}}" + secrets_hint 2>/dev/null || true + exit 1 +fi +if [ -n "${SECRETS_ROOT:-}" ]; then + info "explorer secret" "resolved (SECRETS_ROOT=${SECRETS_ROOT} · stage uses stagepaysan secret first when CUR=TESTPAYSAN)" +else + info "explorer secret" "resolved via EXP_PW / EXP_PW_FILE / stage SSH / koopa" +fi +if [ -n "${CLI_JS:-}" ] && [ -f "${CLI_JS}" ]; then + ok "wallet-cli" "$CLI_JS" +else + ok "wallet-cli" "PATH $(command -v taler-wallet-cli)" +fi +ok "wallet-helper" "$(command -v taler-helper-sqlite3)" +# prove wcli can start (avoids minting 32 probes then SKIP_ACCEPT command-not-found) +if ! wcli balance >"$SCRATCH/wcli-smoke.out" 2>&1; then + err prereq "wallet-cli smoke failed" \ + "problem: wcli balance failed — fix CLI_JS/PATH/helper. detail: $(tr '\n' ' ' <"$SCRATCH/wcli-smoke.out" | head -c 240)" + exit 1 +fi + +# --- auto-account --- +t0=$(now_ms) +if ! curl -sS -m 30 -o "$SCRATCH/auto-account.json" "${BANK}/intro/auto-account.json"; then + err bank "auto-account.json unreachable" + exit 1 +fi +ms_auto=$(elapsed_ms "$t0") +if ! python3 -c 'import json; d=json.load(open("'"$SCRATCH"'/auto-account.json")); assert d.get("ok") or d.get("username")' 2>/dev/null; then + err bank "auto-account create failed" "$(head -c 120 "$SCRATCH/auto-account.json" | tr '\n' ' ')" + exit 1 +fi +ACCT_USER=$(python3 -c 'import json; print(json.load(open("'"$SCRATCH"'/auto-account.json"))["username"])') +ok "auto-account ${ACCT_USER} (${ms_auto}ms) — personal ${CUR}:0; pool=explorer" +info "auto-account password" "(see $SCRATCH/auto-account.json — not logged)" + +# --- explorer token --- +t0=$(now_ms) +TOK=$(curl -sS -m 20 -u "${EXP_USER}:${EXP_PW}" \ + -H 'Content-Type: application/json' \ + -d '{"scope":"readwrite","duration":{"d_us":3600000000}}' \ + "${BANK}/accounts/${EXP_USER}/token" \ + | python3 -c 'import json,sys; print(json.load(sys.stdin)["access_token"])') +ms_tok=$(elapsed_ms "$t0") +[ -n "$TOK" ] || { err bank "explorer token failed"; exit 1; } +ok "explorer token (${ms_tok}ms)" + +# ONE cumulative wallet for all withdraws + pays (need balance to spend). +# force-select uses *last* reserve_pub from the current accept output. +wallet_prepare() { + local label="${1:-wallet}" + WDB="$SCRATCH/wallet-${label}.sqlite3" + export WDB + if [ ! -f "$WDB" ]; then + wcli exchanges add "$EX" >"$SCRATCH/ex-add-$label.out" 2>&1 || true + wcli exchanges update "$EX" >"$SCRATCH/ex-upd-$label.out" 2>&1 || true + wcli exchanges accept-tos "$EX" >"$SCRATCH/ex-tos-$label.out" 2>&1 || true + fi +} + +t0=$(now_ms) +rm -f "$SCRATCH/wallet-main.sqlite3" +wallet_prepare "main" +ms_tos=$(elapsed_ms "$t0") +ok "wallet exchange + ToS (${ms_tos}ms) — cumulative DB for withdraw+pay (no run-until-done)" + +: "${LADDER_MAX_RUNGS:=99}" +if [ "${LADDER_MODE}" = "max" ]; then + info "ladder mode" "max-search (highest withdrawable + highest payable ${CUR})" + info "max-search params" "probes≤${LADDER_MAX_PROBES} high_rand=${LADDER_HIGH_RUNGS} tol_rel=${LADDER_MAX_TOL_REL} ceiling=${CUR}:${LADDER_MAX_AMOUNT}" + info "max-search pay" "LADDER_PAY=${LADDER_PAY} free_template=${LADDER_FREE_TEMPLATE}@${LADDER_FREE_TEMPLATE_INSTANCE:-$INST}" + LADDER_LIST="" + PAY_LIST="" + LADDER_N=0 + set -- + printf '%s\n' "# max-search (amounts chosen adaptively)" >"$SCRATCH/ladder-plan.txt" + : >"$SCRATCH/ladder-wd-plan.txt" + : >"$SCRATCH/ladder-pay-plan.txt" +else + info "ladder mode" "classic (0 → random mids → max-1 → max)" + build_ladder_pair "$SCRATCH/ladder-wd-plan.txt" "$SCRATCH/ladder-pay-plan.txt" + LADDER_LIST=$(tr -d '\n' <"$SCRATCH/ladder-wd-plan.txt") + PAY_LIST=$(tr -d '\n' <"$SCRATCH/ladder-pay-plan.txt") + # shellcheck disable=SC2086 + set -- $LADDER_LIST + if [ "$#" -gt "$LADDER_MAX_RUNGS" ]; then + # shellcheck disable=SC2046 + set -- $(printf '%s\n' "$@" | head -n "$LADDER_MAX_RUNGS") + fi + LADDER_N=$# + info "withdraw plan" "$*" + info "withdraw plan (alt)" "$(format_amount_list_alt "$@")" + info "pay plan" "$PAY_LIST" + # shellcheck disable=SC2086 + info "pay plan (alt)" "$(format_amount_list_alt $PAY_LIST)" + printf '%s\n' "$@" >"$SCRATCH/ladder-plan.txt" + # shellcheck disable=SC2086 + printf '%s\n' $PAY_LIST >"$SCRATCH/ladder-pay-plan-lines.txt" 2>/dev/null || true +fi + +OK_N=0 +FAIL_N_L=0 +PAY_OK_N=0 +PAY_FAIL_N=0 +STOP_REASON="" +STOP_AMOUNT="" +MAX_BEST_AMT="" +MAX_LO="0" +MAX_HI="${LADDER_MAX_AMOUNT}" +declare -a RUNG_JSON=() + +# --------------------------------------------------------------------------- +# One withdraw rung. Caller sets: AMT, range_note, rung. +# Returns 0 = soft continue (ok/skip/ceiling); 1 = hard stop. +# Sets: status, AMT_NUM, WID, note, ms_*. +# --------------------------------------------------------------------------- +ladder_withdraw_rung() { + if ladder_over; then + STOP_REASON="timeout budget ${LADDER_TIMEOUT_S}s" + warn ladder "time budget exhausted" \ + "problem: LADDER_TIMEOUT_S=${LADDER_TIMEOUT_S}s reached after ${OK_N} ok rungs; remaining amounts not tried" + return 1 + fi + + # section header is printed by classic/max controller (avoids double banners) + tag=$(printf '%s' "$AMT" | tr '.:' '__') + # range_note set by caller (classic pins / max-search labels) + t_rung=$(now_ms) + ms_mint=0 ms_accept=0 ms_confirm=0 ms_settle=0 + note="" + status="FAIL" + WID="-" + FORCE_SEL_409_LOGGED=0 + + # keep cumulative main wallet + wallet_prepare "main" + + # mint from explorer pool + t0=$(now_ms) + code=$(curl -sS -m 30 -o "$SCRATCH/wd-$tag.json" -w '%{http_code}' \ + -H "Authorization: Bearer ${TOK}" \ + -H 'Content-Type: application/json' \ + -d "{\"amount\":\"${AMT}\"}" \ + "${BANK}/accounts/${EXP_USER}/withdrawals") + ms_mint=$(elapsed_ms "$t0") + WID=$(python3 -c 'import json;d=json.load(open("'"$SCRATCH"'/wd-'"$tag"'.json"));print(d.get("withdrawal_id") or "")' 2>/dev/null || true) + # libeufin often emits taler://withdraw/host:443/… — strip default ports (lib.sh) + URI=$(python3 -c 'import json;print(json.load(open("'"$SCRATCH"'/wd-'"$tag"'.json")).get("taler_withdraw_uri") or "")' 2>/dev/null || true) + URI=$(normalize_taler_uri "$URI") + if taler_uri_has_default_port "$URI"; then + warn bank "withdraw URI still has default port" "problem: $URI" + fi + # numeric amount (for zero / settle / ceiling special-cases) + AMT_NUM=$(python3 -c 'import sys; print(sys.argv[1].split(":",1)[-1])' "$AMT") + IS_ZERO=0 + python3 -c 'import sys; from decimal import Decimal; sys.exit(0 if Decimal(sys.argv[1])==0 else 1)' "$AMT_NUM" 2>/dev/null && IS_ZERO=1 + IS_MAX_PIN=0 + IS_MAX_M1_PIN=0 + if [ "$AMT_NUM" = "${LADDER_MAX_AMOUNT}" ]; then + IS_MAX_PIN=1 + range_note="pin:max" + elif [ "$AMT_NUM" = "$((LADDER_MAX_AMOUNT - 1))" ] 2>/dev/null || \ + [ "$AMT_NUM" = "$(python3 -c 'print(int("'"$LADDER_MAX_AMOUNT"'")-1)')" ]; then + IS_MAX_M1_PIN=1 + range_note="pin:max-1" + fi + + if [ "$code" != "200" ] && [ "$code" != "201" ] || [ -z "$WID" ] || [ -z "$URI" ]; then + # strip quotes so STOP_REASON never breaks later shell/python argv + note="mint HTTP $code $(head -c 100 "$SCRATCH/wd-$tag.json" 2>/dev/null | tr '\n\"' ' ')" + ms_total=$(elapsed_ms "$t_rung") + if [ "$IS_ZERO" = "1" ]; then + # Probe only: bank may reject GOA:0 — record and continue ladder + status="ZERO_REJECT" + note="zero-withdraw rejected (expected possible): $note" + warn bank "mint $AMT rejected" \ + "problem: bank will not create a GOA:0 withdrawal (zero amount probe). Ladder continues. detail: $note" + echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" + OK_N=$((OK_N + 1)) + return 0 + fi + # Absolute max is a ceiling probe — bank often returns SQL P0001/5110; do not abort. + if [ "$IS_MAX_PIN" = "1" ]; then + status="CEILING_REJECT" + note="absolute max rejected (ceiling probe; max-1 is the hard pin): $note" + warn bank "mint $AMT rejected (ceiling)" \ + "problem: bank rejects absolute LADDER_MAX_AMOUNT (often SQL P0001/5110). max-1 rung is the last expected success. Ladder continues to report. detail: $note" + echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" + return 0 + fi + # max-1 can also hit the same libeufin SQL ceiling (5110/P0001) on live GOA — + # treat as soft ceiling when mid rungs already passed; do not block the report. + if [ "$IS_MAX_M1_PIN" = "1" ] && echo "$note" | grep -qE '5110|P0001'; then + status="CEILING_REJECT" + note="max-1 rejected (same ceiling as absolute max): $note" + warn bank "mint $AMT rejected (max-1 ceiling)" \ + "problem: bank rejects max-1 with SQL P0001/5110 (pool ceiling). Mid-rung OK_BANK/OK still count. Ladder continues. detail: $note" + echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" + return 0 + fi + # max-search: any mint reject is "too high" / ceiling — never hard-stop the hunt + if [ "${LADDER_MODE}" = "max" ]; then + status="CEILING_REJECT" + note="max-search too-high mint: $note" + warn bank "mint $AMT rejected (max-search)" \ + "problem: mint failed while hunting max withdrawable — treat as upper bound. detail: $note" + echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" + return 0 + fi + # Classic mid/high random rungs: same 5110/P0001 pool ceiling as pins — soft, not FAIL_MINT. + if echo "$note" | grep -qE '5110|P0001'; then + status="CEILING_REJECT" + note="amount above live mint ceiling (5110/P0001): $note" + warn bank "mint $AMT rejected (ceiling)" \ + "problem: bank mint HTTP 500/5110 SQL P0001 on high amount (not only pin:max). Ladder continues; use max-ladder to bound hi. detail: $note" + echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" + return 0 + fi + err bank "mint $AMT failed" "$note" + status="FAIL_MINT" + STOP_REASON="$note" + STOP_AMOUNT="$AMT" + echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" + FAIL_N_L=$((FAIL_N_L + 1)) + return 1 + fi + ok "mint $AMT ($WID) ${ms_mint}ms" + + before=$(wallet_avail) + + # accept + t0=$(now_ms) + if wcli withdraw accept-uri --exchange "$EX" "$URI" >"$SCRATCH/accept-$tag.out" 2>&1; then + ms_accept=$(elapsed_ms "$t0") + ok "accept-uri $AMT ${ms_accept}ms" + else + ms_accept=$(elapsed_ms "$t0") + note="accept-uri failed: $(tail -c 200 "$SCRATCH/accept-$tag.out" | tr '\n' ' ')" + ms_total=$(elapsed_ms "$t_rung") + # Wallet 7006: no denominations for this amount (0, sub-denom dust, etc.) — warn & continue + if [ "$IS_ZERO" = "1" ] || grep -qE 'code: 7006|"code"[[:space:]]*:[[:space:]]*7006|No denominations could be selected' \ + "$SCRATCH/accept-$tag.out" 2>/dev/null; then + if [ "$IS_ZERO" = "1" ]; then + status="ZERO_SKIP" + note="zero-withdraw skip (7006 / no denoms): $note" + warn wallet "accept $AMT skipped" \ + "problem: wallet code 7006 — no coin denominations for GOA:0 (zero amount cannot be withdrawn as coins). Ladder continues. detail: $note" + else + status="SKIP_DENOM" + note="skip amount (wallet 7006 no denoms): $note" + warn wallet "accept $AMT skipped" \ + "problem: wallet code 7006 — no denominations match this amount (below smallest coin or not combinable). Ladder continues with next rung. detail: $note" + fi + echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" + return 0 + fi + if [ "${LADDER_MODE}" = "max" ]; then + status="SKIP_ACCEPT" + warn wallet "accept $AMT skipped (max-search)" "$note" + echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" + return 0 + fi + err wallet "accept $AMT" "$note" + status="FAIL_ACCEPT" + STOP_REASON="$note" + STOP_AMOUNT="$AMT" + echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" + FAIL_N_L=$((FAIL_N_L + 1)) + return 1 + fi + + # Confirm ASAP when bank status is selected. No run-until-done (hangs on macOS/wallet). + # Server-side auto-confirm only watches landing withdraw-watch.ids — ladder must confirm itself. + bank_st() { + curl -sS -m 8 "${BANK}/taler-integration/withdrawal-operation/${WID}" 2>/dev/null \ + | python3 -c 'import json,sys; print((json.load(sys.stdin).get("status") or "").strip())' 2>/dev/null || true + } + do_confirm() { + curl -sS -m 15 -o "$SCRATCH/conf-$tag.json" -w '%{http_code}' -X POST \ + -H "Authorization: Bearer ${TOK}" -H 'Content-Type: application/json' -d '{}' \ + "${BANK}/accounts/${EXP_USER}/withdrawals/${WID}/confirm" + } + # Collect reserve_pub candidates for *this* withdrawal (WID + amount). + # Cumulative wallets re-print old reserves in accept/tx dumps — never trust a single "last" blindly. + # Prints unique pubs one per line, preferred order first. + # + # Critical: wallet-cli dumps mix log lines + JSON. Never json.loads(rest_of_file) — + # trailing "Shutdown requested" lines make that always fail. Use JSONDecoder.raw_decode + # and regex on *both* accept and transactions output (reservePub lives in tx, not accept). + extract_rpubs_for_wid() { + # External module — never feed extract via bash heredoc (refactor tools + # lives in ladder/extract_rpubs.py). + local _ext="$ROOT/ladder/extract_rpubs.py" + if [ ! -f "$_ext" ]; then + warn bank "force-select extract missing" "problem: $_ext not found" + return 0 + fi + python3 "$_ext" "$WID" "$AMT" \ + "$SCRATCH/accept-$tag.out" "$SCRATCH/tx-$tag.json" "$SCRATCH/used-rpubs.txt" + } + + mark_rpub_used() { + local p="$1" + [ -n "$p" ] || return 0 + mkdir -p "$SCRATCH" 2>/dev/null || true + grep -qxF "$p" "$SCRATCH/used-rpubs.txt" 2>/dev/null || echo "$p" >>"$SCRATCH/used-rpubs.txt" + } + + force_select_if_needed() { + local st_now="$1" + [ "$st_now" = "pending" ] || [ -z "$st_now" ] || return 0 + local rpub epayto code_fs any=0 + # refresh tx dump each try (wallet may attach reserve late) + wcli transactions >"$SCRATCH/tx-$tag.json" 2>&1 || true + epayto=$(curl -sS -m 10 "${EX%/}/keys" 2>/dev/null | python3 -c ' +import json,sys +d=json.load(sys.stdin) +acc=d.get("accounts") or [] +for a in acc: + p=a.get("payto_uri") or a.get("payto_address") or "" + if "x-taler-bank" in p or "exchange" in p: + print(p); break +else: + if acc: print(acc[0].get("payto_uri") or "") +' 2>/dev/null || true) + if [ -z "$epayto" ]; then + warn bank "force-select skipped" "problem: exchange payto empty from /keys" + return 0 + fi + # Try candidates until bank leaves pending (200/204) or we exhaust + while IFS= read -r rpub; do + [ -n "$rpub" ] || continue + any=1 + code_fs=$(curl -sS -m 12 -o "$SCRATCH/force-sel-$tag.json" -w '%{http_code}' -X POST \ + -H 'Content-Type: application/json' \ + -d "{\"reserve_pub\":\"${rpub}\",\"selected_exchange\":\"${epayto}\"}" \ + "${BANK}/taler-integration/withdrawal-operation/${WID}" 2>/dev/null || echo "000") + if [ "$code_fs" = "200" ] || [ "$code_fs" = "204" ]; then + info "force-select" "HTTP ${code_fs} rpub=${rpub:0:12}… (ok for WID ${WID:0:8})" + mark_rpub_used "$rpub" + return 0 + fi + if [ "$code_fs" = "409" ]; then + # 5114 = this reserve already bound to another op — not "out of money" + info "force-select" "HTTP 409 rpub=${rpub:0:12}… (stale/used reserve — not balance; trying next)" + mark_rpub_used "$rpub" + continue + fi + info "force-select" "HTTP ${code_fs} rpub=${rpub:0:12}… body=$(tr '\n' ' ' <"$SCRATCH/force-sel-$tag.json" 2>/dev/null | head -c 120)" + # other errors: still try next candidate + done < <(extract_rpubs_for_wid) + if [ "$any" != "1" ]; then + warn bank "force-select skipped" \ + "problem: no reserve_pub for this withdraw (WID=${WID:0:8}…); wallet may not have selected yet" + fi + } + + # No run-until-done (hangs / banned). Read transactions only for reserve_pub candidates. + wcli transactions >"$SCRATCH/tx-$tag.json" 2>&1 || true + + t0=$(now_ms) + conf_ok=0 + st="" + # Poll bank status; force-select while pending; confirm as soon as selected + for i in $(seq 1 "${LADDER_CONFIRM_POLLS}"); do + ladder_over && break + st=$(bank_st) + case "$st" in + selected) + ccode=$(do_confirm) + if [ "$ccode" = "204" ] || [ "$ccode" = "200" ]; then + conf_ok=1 + info "confirm" "HTTP $ccode after selected (poll $i)" + else + note="confirm HTTP $ccode" + fi + break + ;; + confirmed) + conf_ok=1 + break + ;; + aborted) + note="withdrawal aborted by bank" + break + ;; + esac + # force-select while pending — try alternate rpubs on 5114 (not out-of-money) + if [ "$st" = "pending" ] || [ -z "$st" ]; then + if [ "$i" -eq 1 ] || [ "$i" -eq 2 ] || [ $((i % 3)) -eq 0 ]; then + force_select_if_needed "$st" + fi + fi + sleep 0.35 + done + ms_confirm=$(elapsed_ms "$t0") + st=$(printf '%s' "${st:-}" | tr -d '\r\n' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//') + if [ "$conf_ok" != "1" ]; then + note="${note:-confirm timeout last=${st:-empty}}" + # Soft: not confirmed — usually stale reserve_pub (5114), NOT empty pool balance + status="SKIP_CONFIRM" + warn bank "confirm $AMT skipped" \ + "problem: bank status='${st:-empty}' after ${LADDER_CONFIRM_POLLS} polls (want selected/confirmed). Usually reserve_pub mismatch (5114), not out-of-money — mint/accept already OK. detail: $note" + ms_total=$(elapsed_ms "$t_rung") + echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" + return 0 + fi + ok "confirm $AMT ${ms_confirm}ms (client, on selected)" + + # settle: poll wallet balance + bank transfer_done — never run-until-done. + # Prefer real coin delta (status OK). If bank transfer_done first, keep polling + # briefly for available coins (OK) before accepting OK_BANK (pay needs available). + t0=$(now_ms) + settled=0 + xfer="?" + bank_done=0 + if [ "$IS_ZERO" = "1" ]; then + settled=1 + note="zero-amount: no coin delta expected" + else + for r in $(seq 1 "$LADDER_SETTLE_ROUNDS"); do + ladder_over && break + after=$(wallet_avail) + if python3 -c " +from decimal import Decimal +import sys +sys.exit(0 if Decimal(sys.argv[1]) > Decimal(sys.argv[2]) else 1) +" "$after" "$before" 2>/dev/null; then + settled=1 + break + fi + xfer=$(curl -sS -m 10 "${BANK}/taler-integration/withdrawal-operation/${WID}" \ + | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d.get("transfer_done"), d.get("status"))' 2>/dev/null || echo "?") + if echo "$xfer" | grep -qi True; then + bank_done=1 + # nudge wallet to pick up wire (bounded; not run-until-done) + wcli transactions >"$SCRATCH/tx-settle-$tag.out" 2>&1 || true + if command -v timeout >/dev/null 2>&1 || command -v gtimeout >/dev/null 2>&1; then + _to=$(command -v gtimeout 2>/dev/null || command -v timeout) + "$_to" 8 wcli advanced run-pending >"$SCRATCH/run-pending-settle.out" 2>&1 || true + fi + # after bank done, still prefer coins; allow remaining rounds for available + note="bank transfer_done waiting coins avail=${after} $xfer" + fi + sleep "$LADDER_SETTLE_SLEEP" + done + # If bank done but still no coin delta after all rounds, OK_BANK (pay-wait may finish) + if [ "$settled" != "1" ] && [ "$bank_done" = "1" ]; then + note="bank transfer_done avail=$(wallet_avail) $xfer (no run-until-done)" + fi + fi + ms_settle=$(elapsed_ms "$t0") + after=$(wallet_avail) + ms_total=$(elapsed_ms "$t_rung") + if [ -z "${xfer:-}" ] || [ "$xfer" = "?" ]; then + xfer=$(curl -sS -m 10 "${BANK}/taler-integration/withdrawal-operation/${WID}" \ + | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d.get("transfer_done"), d.get("status"))' 2>/dev/null || echo "?") + fi + + if [ "$settled" = "1" ]; then + status="OK" + note="${note:-avail=${CUR}:${after}}" + ok "settle $AMT → ${CUR}:${after} (settle ${ms_settle}ms, rung ${ms_total}ms)" + OK_N=$((OK_N + 1)) + echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" + metrics_report_coins "ladder-r${rung}-after-${tag}" || true + metrics_record_flow withdrawn "$AMT" || true + elif echo "$xfer" | grep -qi True; then + status="OK_BANK" + note="bank transfer_done avail=${after} $xfer (no run-until-done)" + ok "settle $AMT bank transfer_done (wallet avail=${CUR}:${after}, ${ms_settle}ms)" + OK_N=$((OK_N + 1)) + echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" + metrics_report_coins "ladder-r${rung}-after-${tag}" || true + metrics_record_flow withdrawn "$AMT" || true + else + note="no coins / no transfer_done avail=${after} $xfer" + if [ "${LADDER_MODE}" = "max" ]; then + status="SKIP_SETTLE" + warn wallet "settle $AMT skipped (max-search)" \ + "problem: bank confirm ok but settle inconclusive — do not move max bounds. detail: $note" + echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" + return 0 + fi + err wallet "settle $AMT" "$note" + status="FAIL_SETTLE" + STOP_REASON="$note" + STOP_AMOUNT="$AMT" + FAIL_N_L=$((FAIL_N_L + 1)) + echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" + metrics_report_coins "ladder-r${rung}-fail-${tag}" || true + return 1 + fi + return 0 +} + +set_group withdraw + +# --- next amount for max-search (prints: AMT_NUM|range_note|action) --- +# action: probe | done +max_search_next() { + python3 - "$CUR" "$MAX_LO" "$MAX_HI" "$LADDER_MIN_AMOUNT" \ + "$LADDER_MAX_TOL_REL" "$rung" "$LADDER_HIGH_RUNGS" "$LADDER_MAX_SEED" \ + "${MAX_BEST_OK:-}" <<'PY' +import math, random, sys +from decimal import Decimal, ROUND_HALF_UP, ROUND_UP + +cur = sys.argv[1] +lo = Decimal(sys.argv[2] or "0") +hi = Decimal(sys.argv[3]) +min_amt = Decimal(sys.argv[4]) +tol = Decimal(sys.argv[5] or "0.001") +probe_i = int(sys.argv[6]) +high_n = max(1, int(sys.argv[7] or "4")) +seed = sys.argv[8].strip() +best = sys.argv[9].strip() + +if seed: + random.seed(int(seed) + probe_i) +else: + random.seed() + +if min_amt <= 0: + min_amt = Decimal("0.000001") +if hi <= min_amt: + print("0|max:done|done") + raise SystemExit(0) + +if lo > 0 and hi > lo: + gap = (hi - lo) / hi + abs_step = Decimal(1) if hi >= 1 else min_amt + if gap <= tol or (hi - lo) <= abs_step: + print("%s|max:done|done" % (best or str(lo))) + raise SystemExit(0) + +def fmt(v: Decimal) -> str: + if v >= 1: + q = v.quantize(Decimal(1), rounding=ROUND_HALF_UP) + return format(int(q), "d") + q = v.quantize(Decimal("0.00000001"), rounding=ROUND_UP) + s = format(q, "f").rstrip("0").rstrip(".") + return s or "0" + +def clamp(v: Decimal): + if v <= lo and lo > 0: + v = lo + (Decimal(1) if lo >= 1 else min_amt) + if v >= hi: + v = hi - (Decimal(1) if hi > 1 else min_amt) + if v < min_amt: + v = min_amt + if v <= 0: + v = min_amt + if v >= hi: + return None + return v + +if probe_i < high_n: + band_lo = max(lo if lo > 0 else min_amt, hi * Decimal("0.15")) + band_hi = hi * Decimal("0.999") + if band_hi <= band_lo: + band_lo = max(min_amt, hi * Decimal("0.5")) + band_hi = hi * Decimal("0.99") + flo, fhi = float(band_lo), float(band_hi) + if flo <= 0: + flo = float(min_amt) + if fhi <= flo: + fhi = flo * 2 + v = Decimal(str(math.exp(random.uniform(math.log(flo), math.log(fhi))))) + v = clamp(v) + if v is None: + print("%s|max:done|done" % (best or str(lo))) + raise SystemExit(0) + print("%s|max:high-rand|probe" % fmt(v)) + raise SystemExit(0) + +if lo <= 0: + flo = float(max(min_amt, hi * Decimal("0.05"))) + fhi = float(hi * Decimal("0.85")) + if fhi <= flo: + fhi = flo * 1.5 + v = Decimal(str(math.exp(random.uniform(math.log(flo), math.log(fhi))))) + label = "max:descent" +else: + v = (lo * hi).sqrt() + label = "max:bisect" +v = clamp(v) +if v is None: + print("%s|max:done|done" % (best or str(lo))) + raise SystemExit(0) +print("%s|%s|probe" % (fmt(v), label)) +PY +} + +if [ "${LADDER_MODE}" = "max" ]; then + section "ladder · phase A · max-search (find highest withdrawable ${CUR})" + info "max-search" "ceiling=${CUR}:${LADDER_MAX_AMOUNT} min=${CUR}:${LADDER_MIN_AMOUNT} probes≤${LADDER_MAX_PROBES} high_rand=${LADDER_HIGH_RUNGS} tol_rel=${LADDER_MAX_TOL_REL}" + info "max-search" "pay phase $([ "${LADDER_PAY}" = "1" ] && echo enabled || echo disabled)" + MAX_LO="0" + MAX_HI="${LADDER_MAX_AMOUNT}" + MAX_BEST_OK="" + MAX_BEST_AMT="" + rung=0 + # LADDER_MAX_PROBES=0 → no probe cap (stop on converge / timeout only) + while [ "${LADDER_MAX_PROBES}" = "0" ] || [ "$rung" -lt "${LADDER_MAX_PROBES}" ]; do + if ladder_over; then + STOP_REASON="timeout budget ${LADDER_TIMEOUT_S}s" + warn ladder "time budget exhausted" \ + "problem: LADDER_TIMEOUT_S=${LADDER_TIMEOUT_S}s during max-search after ${OK_N} ok; best=${MAX_BEST_AMT:-none} · $(format_amount_alt "${MAX_BEST_AMT:-${CUR}:0}")" + break + fi + _next=$(max_search_next) + _anum=$(printf '%s' "$_next" | cut -d'|' -f1) + range_note=$(printf '%s' "$_next" | cut -d'|' -f2) + _act=$(printf '%s' "$_next" | cut -d'|' -f3) + if [ "$_act" = "done" ]; then + info "max-search" "converged best=$(format_amount_alt "${MAX_BEST_AMT:-${CUR}:0}") · lo=$(format_amount_alt "${CUR}:${MAX_LO}") · hi=$(format_amount_alt "${CUR}:${MAX_HI}") · probes=${rung}" + break + fi + AMT="${CUR}:${_anum}" + if [ -n "${_LAST_MAX_AMT:-}" ] && [ "$AMT" = "$_LAST_MAX_AMT" ]; then + info "max-search" "repeat amount $(format_amount_alt "$AMT") — tightening hi" + MAX_HI=$(python3 -c 'from decimal import Decimal; a=Decimal("'"${_anum}"'"); print(a-(1 if a>=1 else Decimal("'"${LADDER_MIN_AMOUNT}"'")))') + continue + fi + _LAST_MAX_AMT="$AMT" + rung=$((rung + 1)) + section "ladder · max probe $rung $AMT · $(format_amount_alt "$AMT") [lo=$(format_amount_alt "${CUR}:${MAX_LO}") · hi=$(format_amount_alt "${CUR}:${MAX_HI}")]" + status="FAIL" + if ! ladder_withdraw_rung; then + break + fi + AMT_NUM=$(python3 -c 'import sys; print(sys.argv[1].split(":",1)[-1])' "$AMT") + case "$status" in + OK|OK_BANK) + MAX_LO="$AMT_NUM" + MAX_BEST_OK="$AMT_NUM" + MAX_BEST_AMT="$AMT" + info "max-search" "success bound lo=$(format_amount_alt "${CUR}:${MAX_LO}") (best so far)" + ;; + CEILING_REJECT|FAIL_MINT) + MAX_HI=$(python3 -c ' +from decimal import Decimal +a=Decimal("'"$AMT_NUM"'"); h=Decimal("'"$MAX_HI"'") +print(min(a,h)) +') + info "max-search" "too-high hi:=$(format_amount_alt "${CUR}:${MAX_HI}")" + ;; + SKIP_DENOM|ZERO_SKIP|ZERO_REJECT) + info "max-search" "denom-skip $(format_amount_alt "$AMT") (bounds unchanged)" + ;; + SKIP_CONFIRM|SKIP_SETTLE|SKIP_ACCEPT) + info "max-search" "inconclusive $status $(format_amount_alt "$AMT") (bounds unchanged)" + ;; + *) + info "max-search" "status=$status $(format_amount_alt "$AMT") (bounds unchanged)" + ;; + esac + if python3 -c 'from decimal import Decimal; import sys; sys.exit(0 if Decimal(sys.argv[1])>0 and Decimal(sys.argv[2])<=Decimal(sys.argv[1]) else 1)' "$MAX_LO" "$MAX_HI" 2>/dev/null; then + info "max-search" "bounds crossed lo=$(format_amount_alt "${CUR}:${MAX_LO}") · hi=$(format_amount_alt "${CUR}:${MAX_HI}") — stop" + break + fi + done + if [ -n "${MAX_BEST_AMT:-}" ]; then + ok "max-search best withdrawable $(format_amount_alt "$MAX_BEST_AMT") · lo=$(format_amount_alt "${CUR}:${MAX_LO}") · hi=$(format_amount_alt "${CUR}:${MAX_HI}") · probes=${rung}" + else + warn ladder "max-search no successful withdraw" \ + "problem: no OK/OK_BANK within ${rung} probes; hi=$(format_amount_alt "${CUR}:${MAX_HI}")" + fi + { + echo "# max-search mode (withdraw)" + echo "best_wd=${MAX_BEST_AMT:-}" + echo "best_wd_alt=$(format_amount_alt "${MAX_BEST_AMT:-${CUR}:0}")" + echo "lo=${MAX_LO}" + echo "lo_alt=$(format_amount_alt "${CUR}:${MAX_LO}")" + echo "hi=${MAX_HI}" + echo "hi_alt=$(format_amount_alt "${CUR}:${MAX_HI}")" + echo "probes=${rung}" + } >"$SCRATCH/ladder-plan.txt" + printf '%s\n' "${MAX_BEST_AMT:-${CUR}:0}" >"$SCRATCH/ladder-wd-plan.txt" + : >"$SCRATCH/ladder-pay-plan.txt" + # pay phase uses max-search (not classic PAY_LIST) + PAY_LIST="" +else + section "ladder · phase A · withdraw (${LADDER_N} rungs)" + rung=0 + for AMT in "$@"; do + rung=$((rung + 1)) + if ladder_over; then + STOP_REASON="timeout budget ${LADDER_TIMEOUT_S}s" + warn ladder "time budget exhausted" \ + "problem: LADDER_TIMEOUT_S=${LADDER_TIMEOUT_S}s reached after ${OK_N} ok rungs; remaining amounts not tried" + break + fi + if [ "$rung" -eq 1 ]; then + range_note="pin:0" + elif [ "$rung" -eq "$LADDER_N" ]; then + range_note="pin:max" + elif [ "$rung" -eq $((LADDER_N - 1)) ] && [ "$LADDER_N" -ge 3 ]; then + range_note="pin:max-1" + else + range_note="random" + fi + section "ladder · rung $rung $AMT · $(format_amount_alt "$AMT")" + if ! ladder_withdraw_rung; then + break + fi + done +fi + + +# --------------------------------------------------------------------------- +# Phase B — payment (ladder/lib_pay.sh) +# classic: planned pay list · max: highest payable ≤ available (goa-free) +# --------------------------------------------------------------------------- +PAY_OK_N=0 +PAY_FAIL_N=0 +MAX_PAY_BEST_AMT="" +MAX_PAY_LO="0" +MAX_PAY_HI="0" +ladder_run_pay_phase + +ms_phase=$(python3 -c 'import sys,time; print(int((time.time()-float(sys.argv[1]))*1000))' "$SECTION_T0") + +# --- report --- +set_group report +section "ladder · report" +info "auto-account" "$ACCT_USER" +info "ok_rungs" "$OK_N" +info "fail_rungs" "$FAIL_N_L" +info "pay_ok" "$PAY_OK_N" +info "pay_fail" "$PAY_FAIL_N" +info "phase_ms" "$ms_phase" +info "tsv" "$TSV" +info "pay_tsv" "$PAY_TSV" + +# speed summary via python — free-text stop reason via env (JSON " in bank errors +# used to break shell argv quoting → "syntax error near unexpected token '('") +export LADDER_REPORT_STOP_AMOUNT="${STOP_AMOUNT:-}" +export LADDER_REPORT_STOP_REASON="${STOP_REASON:-}" +export LADDER_REPORT_MODE="${LADDER_MODE:-classic}" +export LADDER_REPORT_MAX_BEST="${MAX_BEST_AMT:-}" +export LADDER_REPORT_MAX_PAY_BEST="${MAX_PAY_BEST_AMT:-}" +export LADDER_REPORT_MAX_PAY_LO="${MAX_PAY_LO:-}" +export LADDER_REPORT_MAX_PAY_HI="${MAX_PAY_HI:-}" +export LADDER_REPORT_MAX_PAY_BEST_ALT="$(format_amount_alt "${MAX_PAY_BEST_AMT:-${CUR}:0}" 2>/dev/null || true)" +export LADDER_REPORT_MAX_LO="${MAX_LO:-}" +export LADDER_REPORT_MAX_HI="${MAX_HI:-}" +# Human alt_unit_names (e.g. 8.15 Tera-GOA (GOA:…)) for report + final lines +export LADDER_REPORT_MAX_BEST_ALT="$(format_amount_alt "${MAX_BEST_AMT:-${CUR}:0}" 2>/dev/null || true)" +export LADDER_REPORT_MAX_LO_ALT="$(format_amount_alt "${CUR}:${MAX_LO:-0}" 2>/dev/null || true)" +export LADDER_REPORT_MAX_HI_ALT="$(format_amount_alt "${CUR}:${MAX_HI:-0}" 2>/dev/null || true)" +export LADDER_REPORT_ALT_UNITS="${ALT_UNITS_FILE:-}" +# Report must never abort the ladder (set -e): soft-fail print/JSON only. +python3 - "$TSV" "$PAY_TSV" "$JSON" "$OK_N" "$FAIL_N_L" "$PAY_OK_N" "$PAY_FAIL_N" "$ms_phase" "$ACCT_USER" "$CUR" <<'PY' || true +import csv, json, os, sys, statistics, traceback +from decimal import Decimal, InvalidOperation, ROUND_HALF_UP + +try: + tsv, pay_tsv, jpath = sys.argv[1:4] + ok_n, fail_n, pay_ok, pay_fail, phase_ms, acct, cur = sys.argv[4:11] + stop_amt = os.environ.get("LADDER_REPORT_STOP_AMOUNT") or None + stop_reason = os.environ.get("LADDER_REPORT_STOP_REASON") or None + mode = os.environ.get("LADDER_REPORT_MODE") or "classic" + max_best = os.environ.get("LADDER_REPORT_MAX_BEST") or None + max_lo = os.environ.get("LADDER_REPORT_MAX_LO") or None + max_hi = os.environ.get("LADDER_REPORT_MAX_HI") or None + max_best_alt = os.environ.get("LADDER_REPORT_MAX_BEST_ALT") or None + max_lo_alt = os.environ.get("LADDER_REPORT_MAX_LO_ALT") or None + max_hi_alt = os.environ.get("LADDER_REPORT_MAX_HI_ALT") or None + max_pay_best = os.environ.get("LADDER_REPORT_MAX_PAY_BEST") or None + max_pay_lo = os.environ.get("LADDER_REPORT_MAX_PAY_LO") or None + max_pay_hi = os.environ.get("LADDER_REPORT_MAX_PAY_HI") or None + max_pay_best_alt = os.environ.get("LADDER_REPORT_MAX_PAY_BEST_ALT") or None + alt_path = os.environ.get("LADDER_REPORT_ALT_UNITS") or "" + alt = {} + if alt_path: + try: + _loaded = json.load(open(alt_path)) + if isinstance(_loaded, dict): + alt = {str(k): str(v) for k, v in _loaded.items()} + except Exception: + alt = {} + if not isinstance(alt, dict) or not alt: + alt = {"0": cur or "GOA"} + elif "0" not in alt: + alt = dict(alt) + alt["0"] = cur or "GOA" + + def fmt_num(v): + try: + if v == v.to_integral(): + return format(int(v), "d") + s = format(v.quantize(Decimal("0.00000001"), rounding=ROUND_HALF_UP), "f") + return s.rstrip("0").rstrip(".") + except Exception: + return str(v) + + def format_alt(amt): + """Human alt_unit_names; never raises.""" + try: + if not amt: + return "-" + s = str(amt).strip() + try: + if ":" in s: + c, v = s.split(":", 1) + val = Decimal(v) + else: + c, val = cur, Decimal(s) + except (InvalidOperation, ValueError, TypeError): + return s + au = alt if isinstance(alt, dict) else {"0": cur or "GOA"} + base_name = au.get("0") or c or cur or "GOA" + base_s = "%s:%s" % (c, fmt_num(val)) + if val == 0: + return "0 %s (%s)" % (base_name, base_s) + scales = [] + for k, name in au.items(): + try: + scales.append((int(k), str(name))) + except Exception: + continue + scales.sort(key=lambda x: -x[0]) + absval = abs(val) + chosen = None + for sc, name in scales: + try: + unit = Decimal(10) ** sc + if unit <= 0: + continue + coeff = absval / unit + if coeff >= 1: + chosen = (sc, name, coeff if val >= 0 else -coeff) + break + except Exception: + continue + if chosen is None: + return "%s %s (%s)" % (fmt_num(val), base_name, base_s) + sc, name, coeff = chosen + if sc == 0: + return "%s %s" % (fmt_num(val), name) + return "%s %s (%s)" % (fmt_num(coeff), name, base_s) + except Exception: + return str(amt) if amt else "-" + + def as_amt(s): + try: + if s is None or s == "": + return None + s = str(s).strip() + if ":" in s: + return s + return "%s:%s" % (cur, s) + except Exception: + return None + + def load_rows(path): + rows = [] + try: + with open(path, newline="") as f: + for row in csv.DictReader(f, delimiter="\t"): + rows.append(row) + except Exception: + pass + return rows + + rows = load_rows(tsv) + prows = load_rows(pay_tsv) + + def nums(rs, key): + out = [] + for row in rs: + try: + out.append(int(row[key])) + except Exception: + pass + return out + + def stats(xs): + if not xs: + return {"n": 0} + return { + "n": len(xs), + "min_ms": min(xs), + "max_ms": max(xs), + "avg_ms": int(sum(xs) / len(xs)), + "p50_ms": int(statistics.median(xs)), + } + + _max_block = None + if mode == "max": + _best = as_amt(max_best) if max_best else None + _lo = as_amt(max_lo) if max_lo is not None and str(max_lo) != "" else None + _hi = as_amt(max_hi) if max_hi is not None and str(max_hi) != "" else None + _pb = as_amt(max_pay_best) if max_pay_best else None + _max_block = { + "best": _best, + "best_alt": max_best_alt or (format_alt(_best) if _best else None), + "lo": max_lo, + "lo_alt": max_lo_alt or (format_alt(_lo) if _lo else None), + "hi": max_hi, + "hi_alt": max_hi_alt or (format_alt(_hi) if _hi else None), + "best_pay": _pb, + "best_pay_alt": max_pay_best_alt or (format_alt(_pb) if _pb else None), + "pay_lo": max_pay_lo, + "pay_hi": max_pay_hi, + } + + report = { + "currency": cur, + "mode": mode, + "auto_account": acct, + "ok_rungs": int(ok_n), + "fail_rungs": int(fail_n), + "pay_ok": int(pay_ok), + "pay_fail": int(pay_fail), + "max_search": _max_block, + "stopped_at_amount": stop_amt or None, + "stop_reason": stop_reason or None, + "phase_ms": int(phase_ms), + "timing": { + "mint": stats(nums(rows, "ms_mint")), + "accept": stats(nums(rows, "ms_accept")), + "confirm": stats(nums(rows, "ms_confirm")), + "settle": stats(nums(rows, "ms_settle")), + "rung_total": stats(nums(rows, "ms_total")), + "pay_order": stats(nums(prows, "ms_order")), + "pay_handle": stats(nums(prows, "ms_handle")), + "pay_settle": stats(nums(prows, "ms_settle")), + "pay_total": stats(nums(prows, "ms_total")), + }, + "rungs": rows, + "pays": prows, + } + try: + json.dump(report, open(jpath, "w"), indent=2) + print("JSON", jpath) + except Exception as e: + print("JSON write error:", e) + + print("--- withdraw speed (ms) ---") + for k in ("mint", "accept", "confirm", "settle", "rung_total"): + v = report["timing"][k] + if v.get("n"): + print(" %s n=%s min=%s p50=%s avg=%s max=%s" % ( + k.ljust(12), v["n"], v["min_ms"], v["p50_ms"], v["avg_ms"], v["max_ms"])) + print("--- pay speed (ms) ---") + for k in ("pay_order", "pay_handle", "pay_settle", "pay_total"): + v = report["timing"][k] + if v.get("n"): + print(" %s n=%s min=%s p50=%s avg=%s max=%s" % ( + k.ljust(12), v["n"], v["min_ms"], v["p50_ms"], v["avg_ms"], v["max_ms"])) + print("--- withdraw rungs ---") + for row in rows: + try: + amt = row.get("amount") or "" + aalt = format_alt(amt) if amt else "" + print(" %2s %-18s %-14s total=%sms %s" % ( + row.get("rung"), amt, row.get("status"), row.get("ms_total"), aalt)) + except Exception as e: + print(" ? row-print-error %s" % (e,)) + print("--- pay rungs ---") + for row in prows: + try: + amt = row.get("amount") or "" + aalt = format_alt(amt) if amt else "" + print(" %2s %-18s %-14s total=%sms oid=%s %s" % ( + row.get("rung"), amt, row.get("status"), row.get("ms_total"), + row.get("oid"), aalt)) + except Exception as e: + print(" ? pay-row-print-error %s" % (e,)) + if mode == "max": + print("--- max-search ---") + try: + ba = (max_best_alt or format_alt(as_amt(max_best) or "")) if max_best else "-" + la = (max_lo_alt or format_alt(as_amt(max_lo) or "")) if max_lo is not None and str(max_lo) != "" else "-" + ha = (max_hi_alt or format_alt(as_amt(max_hi) or "")) if max_hi is not None and str(max_hi) != "" else "-" + print(" best=%s" % (max_best or "-")) + print(" best_alt=%s" % ba) + print(" lo=%s" % (max_lo if max_lo is not None else "-")) + print(" lo_alt=%s" % la) + print(" hi=%s" % (max_hi if max_hi is not None else "-")) + print(" hi_alt=%s" % ha) + print(" best_pay=%s" % (max_pay_best or "-")) + print(" best_pay_alt=%s" % (max_pay_best_alt or (format_alt(as_amt(max_pay_best) or "") if max_pay_best else "-"))) + except Exception as e: + print(" max-search print error: %s" % (e,)) + if stop_amt: + try: + print("STOPPED at %s · %s: %s" % (stop_amt, format_alt(stop_amt), stop_reason)) + except Exception: + print("STOPPED at %s: %s" % (stop_amt, stop_reason)) + else: + print("Completed without hard failure (or budget stop without fail).") +except Exception: + traceback.print_exc() +sys.exit(0) +PY + +wcli balance 2>&1 | tee "$REPORT_DIR/balance-final.out" | tail -20 || true + +set_group load +section "ladder · load snapshot (after withdraws)" +metrics_report_load "$LOAD_AFTER" "ladder-end" || true +if [ -f "$LOAD_BEFORE" ] && [ -f "$LOAD_AFTER" ]; then + section "metrics · ladder load delta" + metrics_print_load_delta "$LOAD_BEFORE" "$LOAD_AFTER" || true +fi +# Speed timings → metrics overall (min/p50/avg/max per phase) +if [ -f "$JSON" ]; then + python3 - "$JSON" "${METRICS_DIR}/perf-summary.json" <<'PY' 2>/dev/null || true +import json, sys +rep = json.load(open(sys.argv[1])) +out = {} +for k, v in (rep.get("timing") or {}).items(): + if isinstance(v, dict) and v.get("n"): + out[k] = v +json.dump(out, open(sys.argv[2], "w"), indent=2) +PY +fi +export METRICS_WITHDRAW_TSV="$TSV" +export METRICS_PAY_TSV="$PAY_TSV" +metrics_report_coins "ladder-end" || true +metrics_print_overall "ladder final" || true + +# Keep scratch if LADDER_REPORT_DIR set; else copy key files to /tmp +if [ -z "${LADDER_REPORT_DIR:-}" ]; then + KEEP="/tmp/amount-ladder-report-$(date +%Y%m%d-%H%M%S)" + mkdir -p "$KEEP" + cp -a "$TSV" "$PAY_TSV" "$JSON" "$SCRATCH/auto-account.json" "$SCRATCH/ladder-plan.txt" \ + "$SCRATCH/ladder-wd-plan.txt" "$SCRATCH/ladder-pay-plan.txt" \ + "$REPORT_DIR/balance-final.out" "$LOAD_BEFORE" "$LOAD_AFTER" "$KEEP/" 2>/dev/null || true + info "report_dir" "$KEEP" + echo "$KEEP" >"$SCRATCH/KEEP_PATH" +fi + +if [ "$FAIL_N_L" -gt 0 ]; then + blocker "ladder" "stopped at ${STOP_AMOUNT:-?} — ${STOP_REASON:-error}" + exit 1 +fi +if [ "$OK_N" -eq 0 ]; then + blocker "ladder" "no successful withdraw rungs" + exit 1 +fi +if [ "${LADDER_MODE}" = "max" ] && [ -n "${MAX_BEST_AMT:-}" ]; then + ok "ladder finished mode=max wd_best=$(format_amount_alt "$MAX_BEST_AMT") · pay_best=$(format_amount_alt "${MAX_PAY_BEST_AMT:-${CUR}:0}") · withdraw_ok=$OK_N pay_ok=$PAY_OK_N phase=${ms_phase}ms" +else + ok "ladder finished mode=${LADDER_MODE} withdraw_ok=$OK_N pay_ok=$PAY_OK_N phase=${ms_phase}ms" +fi +exit 0 diff --git a/check_apt_deploy.sh b/check_apt_deploy.sh index 5240e82..e7bc488 100755 --- a/check_apt_deploy.sh +++ b/check_apt_deploy.sh @@ -9,6 +9,9 @@ # koopa-taler-deploy-test-apt-src-trixie-upgrade # koopa-taler-deploy-test-apt-src-trixie-testing-upgrade # +# Emits a compact container board (board · key=value) for the HTML sticky/top +# overview: which pods are active and which fail (httpd / ldd / …). +# # Env: # APT_DEPLOY_CONTAINERS "name:suite:mode …" mode=fresh|upgrade (optional, default fresh) # APT_DEPLOY_SKIP=1 @@ -44,6 +47,58 @@ ok "podman" "$("$PODMAN_BIN" --version 2>/dev/null | head -1)" info "host" "$(hostname 2>/dev/null || echo unknown) · whoami=$(whoami)" fail_any=0 +# board rows for end overview (human table + machine board · lines) +BOARD_ROWS=() + +_short_name() { + # koopa-taler-deploy-test-apt-src-trixie-testing-upgrade → trixie-testing-upgrade + local n="$1" + n="${n#koopa-taler-deploy-test-apt-src-}" + printf '%s' "$n" +} + +# Record one container for the top-of-page board (HTML parses "board · key=value"). +board_record() { + local short="$1" name="$2" suite="$3" mode="$4" state="$5" + local merchant="$6" httpd="$7" ldd="$8" verdict="$9" note="${10:-}" + local kv + kv="short=${short} name=${name} suite=${suite} mode=${mode} state=${state}" + kv+=" merchant=${merchant} httpd=${httpd} ldd=${ldd} verdict=${verdict}" + # note: no spaces (HTML/parser splits on whitespace) + if [ -n "$note" ]; then + note_safe=$(printf '%s' "$note" | tr ' ' '_' | tr -d '|') + kv+=" note=${note_safe}" + fi + info "board" "$kv" + BOARD_ROWS+=("$short|$suite|$mode|$state|$merchant|$httpd|$ldd|$verdict|$note") +} + +print_board_table() { + section "aptdeploy · container overview (active / errors)" + info "board-table" "| short | suite | mode | podman | merchant | httpd | ldd | verdict | note |" + info "board-table" "|-------|-------|------|--------|----------|-------|-----|---------|------|" + local r short suite mode state merchant httpd ldd verdict note + local n_ok=0 n_err=0 n_miss=0 + for r in "${BOARD_ROWS[@]+"${BOARD_ROWS[@]}"}"; do + IFS='|' read -r short suite mode state merchant httpd ldd verdict note <<<"$r" + info "board-table" "| ${short} | ${suite} | ${mode} | ${state} | ${merchant} | ${httpd} | ${ldd} | ${verdict} | ${note} |" + case "$verdict" in + OK|ok) n_ok=$((n_ok + 1)) ;; + MISSING|missing) n_miss=$((n_miss + 1)) ;; + *) n_err=$((n_err + 1)) ;; + esac + done + info "board-summary" "containers=${#BOARD_ROWS[@]} ok=${n_ok} error=${n_err} missing=${n_miss}" + # one-line human skim (INFO/OK only — do not re-emit ERROR, details already counted) + for r in "${BOARD_ROWS[@]+"${BOARD_ROWS[@]}"}"; do + IFS='|' read -r short suite mode state merchant httpd ldd verdict note <<<"$r" + if [ "$verdict" = "OK" ]; then + ok "overview" "${short}: ${state} · merchant=${merchant} · httpd=${httpd} · ldd=${ldd}" + else + info "overview" "${short}: ${verdict} · state=${state} merchant=${merchant} httpd=${httpd} ldd=${ldd}${note:+ · ${note}}" + fi + done +} print_pkg_table() { local name=$1 @@ -58,16 +113,21 @@ print_pkg_table() { done ' 2>/dev/null || echo "| (query failed) | |") info "pkg-table" "$name" - # emit as plain lines so HTML log keeps the table while IFS= read -r line; do [ -n "$line" ] || continue info "pkg" "$line" done <<<"$rows" } +# merchant package version only (for board) +_merchant_ver() { + local name=$1 + "$PODMAN_BIN" exec "$name" dpkg-query -W -f='${Version}' taler-merchant 2>/dev/null || echo "?" +} + check_ldd_deep() { local name=$1 - local out + local out missing out=$("$PODMAN_BIN" exec "$name" bash -lc ' set +e bin=$(command -v taler-merchant-httpd) @@ -87,15 +147,48 @@ check_ldd_deep() { info "ldd" "$line" done <<<"$out" if echo "$out" | grep -q 'not found'; then - err "ldd" "shared library missing in $name" "see ldd lines above" + missing=$(echo "$out" | grep 'not found' | tr '\n' ' ' | head -c 280) + err "ldd" "shared library missing in $name" "${missing:-not found (see ldd INFO above)}" return 1 fi return 0 } +# Build a single-line diagnosis for err() so jump lists are self-contained +# (operators should not need to hunt nearby INFO lines). +_apt_systemd_diagnose() { + local out="$1" + local before after failed journal status_line result + before=$(echo "$out" | sed -n 's/^httpd_active: //p' | tail -1) + after=$(echo "$out" | sed -n 's/^after_start_httpd: //p' | tail -1) + failed=$(echo "$out" | sed -n 's/^failed_taler: //p' | grep -v '^none$' | tr '\n' ';' | sed 's/;$//') + [ -z "$failed" ] && failed=$(echo "$out" | grep -E '^\s*●\s+taler-' | head -3 | tr '\n' ';' | sed 's/;$//') + result=$(echo "$out" | sed -n 's/^httpd_Result: //p' | tail -1) + [ -z "$result" ] && result=$(echo "$out" | sed -n 's/^ *Result: //p' | head -1) + status_line=$(echo "$out" | sed -n 's/^httpd_status_line: //p' | tail -1) + journal=$(echo "$out" | sed -n 's/^httpd_journal: //p' | tr '\n' ' ' | head -c 220) + # fallback: first non-meta journal-ish line + if [ -z "$journal" ]; then + journal=$(echo "$out" | grep -E 'taler-merchant-httpd\[[0-9]+\]|Failed|error|Error|NRestarts|Main PID' | head -3 | tr '\n' ' ' | head -c 220) + fi + local parts=() + [ -n "$before" ] && parts+=("before=${before}") + [ -n "$after" ] && parts+=("after_start=${after}") + [ -n "$result" ] && parts+=("Result=${result}") + [ -n "$status_line" ] && parts+=("status=${status_line}") + [ -n "$failed" ] && [ "$failed" != "none" ] && parts+=("failed_units=${failed}") + [ -n "$journal" ] && parts+=("journal=${journal}") + if [ "${#parts[@]}" -eq 0 ]; then + printf '%s' "no status/journal captured (podman exec empty?)" + return + fi + local IFS=' · ' + printf '%s' "${parts[*]}" +} + check_systemd() { local name=$1 - local out active_target active_httpd enabled + local out active_httpd before_httpd detail out=$("$PODMAN_BIN" exec "$name" bash -lc ' set +e echo "system: $(systemctl is-system-running 2>&1)" @@ -103,31 +196,74 @@ check_systemd() { echo "target_active: $(systemctl is-active taler-merchant.target 2>&1)" echo "httpd_active: $(systemctl is-active taler-merchant-httpd.service 2>&1)" echo "httpd_enabled: $(systemctl is-enabled taler-merchant-httpd.service 2>&1)" - systemctl start taler-merchant.target 2>&1 | tail -5 - sleep 2 + # unit properties help when is-active is only "activating"/"failed" + echo "httpd_Result: $(systemctl show -p Result --value taler-merchant-httpd.service 2>&1)" + echo "httpd_SubState: $(systemctl show -p SubState --value taler-merchant-httpd.service 2>&1)" + echo "httpd_NRestarts: $(systemctl show -p NRestarts --value taler-merchant-httpd.service 2>&1)" + echo "httpd_ExecMainStatus: $(systemctl show -p ExecMainStatus --value taler-merchant-httpd.service 2>&1)" + echo "httpd_ExecMainCode: $(systemctl show -p ExecMainCode --value taler-merchant-httpd.service 2>&1)" + # one-line status (Active: failed / activating (start) …) + systemctl status taler-merchant-httpd.service --no-pager -l 2>&1 | head -12 | while IFS= read -r sl; do + echo "httpd_status: $sl" + done + # compact single-line for err detail + st1=$(systemctl status taler-merchant-httpd.service --no-pager -l 2>&1 | sed -n "s/^ *Active: //p" | head -1) + echo "httpd_status_line: ${st1:-?}" + systemctl start taler-merchant.target 2>&1 | tail -8 | while IFS= read -r sl; do + echo "start_target: $sl" + done + # wait longer: "activating" often clears only after dbinit + sleep 5 echo "after_start_target: $(systemctl is-active taler-merchant.target 2>&1)" echo "after_start_httpd: $(systemctl is-active taler-merchant-httpd.service 2>&1)" - systemctl --failed --no-legend 2>/dev/null | grep -i taler || echo "failed_taler: none" + echo "after_httpd_Result: $(systemctl show -p Result --value taler-merchant-httpd.service 2>&1)" + echo "after_httpd_SubState: $(systemctl show -p SubState --value taler-merchant-httpd.service 2>&1)" + # journal: last failure reasons (self-contained for ERROR jump list) + journalctl -u taler-merchant-httpd.service -n 12 --no-pager -o cat 2>/dev/null \ + | sed "/^$/d" | tail -8 | while IFS= read -r jl; do + echo "httpd_journal: $jl" + done + journalctl -u taler-merchant-dbinit.service -n 6 --no-pager -o cat 2>/dev/null \ + | sed "/^$/d" | tail -4 | while IFS= read -r jl; do + echo "dbinit_journal: $jl" + done + ft=$(systemctl --failed --no-legend 2>/dev/null | grep -i taler || true) + if [ -n "$ft" ]; then + echo "$ft" | while IFS= read -r fl; do echo "failed_taler: $fl"; done + else + echo "failed_taler: none" + fi ' 2>/dev/null || echo "systemd probe failed") while IFS= read -r line; do [ -n "$line" ] || continue info "systemd" "$line" done <<<"$out" + before_httpd=$(echo "$out" | sed -n 's/^httpd_active: //p' | tail -1) active_httpd=$(echo "$out" | sed -n 's/^after_start_httpd: //p' | tail -1) + # export for caller board + APT_LAST_HTTPD="${active_httpd:-?}" if [ "$active_httpd" = "active" ]; then ok "systemd httpd" "taler-merchant-httpd active after start taler-merchant.target" return 0 fi - # soft if target not shipped as enabled in container — still ERROR for deploy smoke - err "systemd httpd" "taler-merchant-httpd not active" "after_start_httpd=${active_httpd:-?} · see systemd lines" + detail=$(_apt_systemd_diagnose "$out") + # Prefer after_* Result if present + local after_result after_sub + after_result=$(echo "$out" | sed -n 's/^after_httpd_Result: //p' | tail -1) + after_sub=$(echo "$out" | sed -n 's/^after_httpd_SubState: //p' | tail -1) + [ -n "$after_result" ] && [ "$after_result" != "success" ] && detail="${detail} · after_Result=${after_result}" + [ -n "$after_sub" ] && detail="${detail} · after_SubState=${after_sub}" + # Human one-liner for jump list (no "see …" without content) + err "systemd httpd" \ + "taler-merchant-httpd not active (container=${name})" \ + "${detail}" return 1 } check_merchant_basics() { local name=$1 - local out code - # unix socket or curl localhost if httpd listens + local out out=$("$PODMAN_BIN" exec "$name" bash -lc ' set +e echo "=== binaries ===" @@ -135,7 +271,6 @@ check_merchant_basics() { echo "=== --version ===" taler-merchant-httpd --version 2>&1 echo "=== config probe ===" - # try common paths without TLS for u in \ http://127.0.0.1:9966/config \ http://127.0.0.1:8081/config \ @@ -162,26 +297,24 @@ check_merchant_basics() { done <<<"$out" if echo "$out" | grep -qi 'error while loading shared libraries'; then - err "httpd" "shared library error on --version" "see basics/ldd" + local lib_err + lib_err=$(echo "$out" | grep -i 'error while loading shared libraries' | head -1 | tr '\n' ' ' | head -c 240) + err "httpd" "shared library error on --version" "${lib_err:-see basics INFO lines}" return 1 fi - if ! echo "$out" | grep -q 'taler-merchant-httpd --version\|v\.\|taler-merchant-httpd'; then - # version line varies; success if exit was ok — check via separate exec - : - fi set +e - "$PODMAN_BIN" exec "$name" taler-merchant-httpd --version >/dev/null 2>&1 - local ec=$? + local ver_out ec + ver_out=$("$PODMAN_BIN" exec "$name" taler-merchant-httpd --version 2>&1) + ec=$? set -e if [ "$ec" -eq 0 ]; then ok "httpd --version" "exit 0" else - err "httpd --version" "exit $ec" + err "httpd --version" "exit $ec" "$(printf '%s' "$ver_out" | tr '\n' ' ' | head -c 240)" return 1 fi - # /config is nice-to-have if stack is fully configured if echo "$out" | grep -qE 'curl .* -> 200'; then ok "merchant /config" "HTTP 200 (local)" else @@ -192,7 +325,10 @@ check_merchant_basics() { check_one() { local name="$1" expect_suite="$2" mode="${3:-fresh}" - local st suite_line pkgs + local st short merchant="-" httpd="-" ldd="-" verdict="OK" note="" c_fail=0 + + short="$(_short_name "$name")" + APT_LAST_HTTPD="-" # group ids: aptdeploy.trixie-01 / aptdeploy.trixie-testing-upgrade-01 set_group "${expect_suite}${mode:+-$mode}" @@ -200,12 +336,14 @@ check_one() { if ! "$PODMAN_BIN" container exists "$name" 2>/dev/null; then err "container" "$name missing" "run ensure-apt-deploy-test-containers.sh" + board_record "$short" "$name" "$expect_suite" "$mode" "missing" "-" "-" "-" "MISSING" "ensure containers" fail_any=1 return fi st=$("$PODMAN_BIN" inspect -f '{{.State.Status}}' "$name" 2>/dev/null || echo unknown) if [ "$st" != "running" ]; then err "container" "$name not running" "status=$st" + board_record "$short" "$name" "$expect_suite" "$mode" "$st" "-" "-" "-" "ERROR" "not running" fail_any=1 return fi @@ -217,31 +355,48 @@ check_one() { ok "apt suite" "$expect_suite" else err "apt suite" "expected $expect_suite" "got: ${suite_line:-empty}" - fail_any=1 + c_fail=1 + note="suite mismatch" fi + merchant="$(_merchant_ver "$name")" print_pkg_table "$name" if ! check_merchant_basics "$name"; then - fail_any=1 - check_ldd_deep "$name" || fail_any=1 + c_fail=1 + if ! check_ldd_deep "$name"; then + ldd="FAIL" + c_fail=1 + else + ldd="OK" + fi else - # still show ldd summary as info when ok ldd_out=$("$PODMAN_BIN" exec "$name" bash -lc \ 'ldd "$(command -v taler-merchant-httpd)" 2>&1 | grep -E "libtalerutil|not found" || true' 2>/dev/null || true) if echo "$ldd_out" | grep -q 'not found'; then err "libtalerutil" "not found" "$ldd_out" check_ldd_deep "$name" || true - fail_any=1 + ldd="FAIL" + c_fail=1 else + ldd="OK" info "libtalerutil" "$(echo "$ldd_out" | tr '\n' ' ')" fi fi if ! check_systemd "$name"; then - fail_any=1 - check_ldd_deep "$name" || true + c_fail=1 + httpd="${APT_LAST_HTTPD:-FAIL}" + note="${note:+$note; }httpd=${httpd}" + # deeper ldd if httpd failed + if [ "$ldd" != "FAIL" ]; then + check_ldd_deep "$name" || { ldd="FAIL"; c_fail=1; note="${note:+$note; }ldd"; } + fi + else + httpd="active" fi + # normalize httpd cell + [ -z "$httpd" ] || [ "$httpd" = "-" ] && httpd="${APT_LAST_HTTPD:-?}" # Disk free inside this deploy-test container (+ host once, first container only) if [ "${_APT_DEPLOY_HOST_DISK_DONE:-0}" != "1" ]; then @@ -251,12 +406,39 @@ check_one() { fi set_group disk if ! mon_disk_check_podman "$name" "$PODMAN_BIN"; then - fail_any=1 + c_fail=1 + note="${note:+$note; }disk" fi + + if [ "$c_fail" -ne 0 ]; then + verdict="ERROR" + fail_any=1 + else + verdict="OK" + fi + board_record "$short" "$name" "$expect_suite" "$mode" "running" "$merchant" "$httpd" "$ldd" "$verdict" "$note" } +# --- inventory first (what exists before checks) --- +section "aptdeploy · inventory (podman)" +info "inventory" "expected tracks: trixie/trixie-testing × fresh/upgrade (4 containers)" +for entry in $LIST; do + cname=${entry%%:*} + rest=${entry#*:} + case "$rest" in + *:*) suite=${rest%%:*}; mode=${rest#*:} ;; + *) suite=$rest; mode=fresh ;; + esac + short="$(_short_name "$cname")" + if "$PODMAN_BIN" container exists "$cname" 2>/dev/null; then + st=$("$PODMAN_BIN" inspect -f '{{.State.Status}}' "$cname" 2>/dev/null || echo unknown) + ok "inventory" "${short}: ${st} (${suite}/${mode})" + else + warn "inventory" "${short}: missing (${suite}/${mode})" "will ERROR in check" + fi +done + for entry in $LIST; do - # name:suite or name:suite:mode cname=${entry%%:*} rest=${entry#*:} case "$rest" in @@ -272,6 +454,9 @@ for entry in $LIST; do check_one "$cname" "$suite" "$mode" done +# Compact matrix + overview ERROR lines (jump targets for HTML) +print_board_table + summary if [ "$fail_any" -ne 0 ]; then exit 1 diff --git a/check_devtesting.sh b/check_devtesting.sh new file mode 100755 index 0000000..899d51a --- /dev/null +++ b/check_devtesting.sh @@ -0,0 +1,337 @@ +#!/usr/bin/env bash +# check_devtesting.sh — fake-franken (CHF) via rusty.taler-ops.ch taler-devtesting +# +# Requires SSH to Host DEVTESTING_SSH (default: taler-rusty-devtesting → +# rusty.taler-ops.ch user devtesting, forced command taler-devtesting). +# +# Probes: +# 1) CLI reachable (fake-incoming --help) +# 2) geniban → Swiss IBAN (debit side — “enough money” is synthetic per rung) +# 3) baseline fake-incoming CHF:1.00 (plumbing) +# 4) optional amount ladder: DEVTESTING_LADDER=1 (default) — 23 rungs up to 50k CHF +# +# Env: +# DEVTESTING_SSH default taler-rusty-devtesting (or user@host) +# DEVTESTING_AMOUNT baseline amount (default CHF:1.00) +# DEVTESTING_CREDIT_PAYTO exchange IBAN payto +# DEVTESTING_EXCHANGE_IBAN fallback CH6808573105529100001 (TOPS CHF exchange) +# DEVTESTING_LADDER 1 (default) run amount ladder; 0 = baseline only +# DEVTESTING_LADDER_STEPS default 23 +# DEVTESTING_LADDER_MAX default 50000 (CHF major units) +# DEVTESTING_LADDER_MIN default 0.01 +# DEVTESTING_LADDER_STOP_ON_FAIL 1 (default) stop ladder on hard fail; 0 = continue +# DEVTESTING_SKIP=1 skip phase +# +set -euo pipefail +ROOT=$(cd "$(dirname "$0")" && pwd) +# shellcheck source=lib.sh +source "$ROOT/lib.sh" + +set_area devtesting +section "devtesting · fake-franken CHF (rusty taler-devtesting)" + +# Do NOT honor SKIP_SSH here: stage/tops profiles set SKIP_SSH=1 for container +# inside checks, but rusty.taler-ops.ch is a separate SSH hop (devtesting key). +if [ "${DEVTESTING_SKIP:-0}" = "1" ]; then + info "skip" "DEVTESTING_SKIP=1" + exit 0 +fi + +: "${DEVTESTING_SSH:=}" +: "${DEVTESTING_AMOUNT:=CHF:1.00}" +: "${DEVTESTING_EXCHANGE_IBAN:=CH6808573105529100001}" +: "${DEVTESTING_LADDER:=1}" +: "${DEVTESTING_LADDER_STEPS:=23}" +: "${DEVTESTING_LADDER_MAX:=50000}" +: "${DEVTESTING_LADDER_MIN:=0.01}" +: "${DEVTESTING_LADDER_STOP_ON_FAIL:=1}" +: "${SSH_CONNECT_TIMEOUT:=8}" +: "${SSH_CMD_TIMEOUT:=60}" + +ssh_dt() { + # Prefer Host alias; fall back to explicit user@host if config missing. + # Always -n: do not read local stdin (ladder while-read must not be drained by ssh). + local target="$DEVTESTING_SSH" + if ! ssh -G "$target" >/dev/null 2>&1; then + target="devtesting@rusty.taler-ops.ch" + fi + if command -v timeout >/dev/null 2>&1; then + timeout "${SSH_CMD_TIMEOUT}" ssh -n \ + -o BatchMode=yes \ + -o ConnectTimeout="${SSH_CONNECT_TIMEOUT}" \ + -o StrictHostKeyChecking=accept-new \ + "$target" "$@" + else + ssh -n \ + -o BatchMode=yes \ + -o ConnectTimeout="${SSH_CONNECT_TIMEOUT}" \ + -o StrictHostKeyChecking=accept-new \ + "$target" "$@" + fi +} + +# Run one fake-incoming. Sets globals: FI_RC FI_FLAT FI_STATUS +# FI_STATUS: ok_bounce | ok_in | ok_cli | fail_ssh | fail_creditor | fail_nexus | fail_other +run_fake_incoming() { + local amount="$1" subject="$2" credit="$3" debit="$4" + local fi_out + FI_RC=0 + FI_FLAT="" + FI_STATUS="fail_other" + fi_out=$(ssh_dt fake-incoming \ + --amount "${amount}" \ + --subject "${subject}" \ + --credit-payto "${credit}" \ + --debit-payto "${debit}" 2>&1) || FI_RC=$? + FI_FLAT=$(printf '%s' "$fi_out" | tr '\n' ' ') + + if printf '%s' "$FI_FLAT" | grep -qiE 'Permission denied|Connection refused|Could not resolve|No such command'; then + FI_STATUS=fail_ssh + return 1 + fi + if printf '%s' "$FI_FLAT" | grep -qiE 'Creditor must be the exchange'; then + FI_STATUS=fail_creditor + return 1 + fi + if printf '%s' "$FI_FLAT" | grep -qiE 'Missing amount|unable to read secrets|CalledProcessError'; then + if ! printf '%s' "$FI_FLAT" | grep -qE 'libeufin-nexus - IN .* CHF'; then + FI_STATUS=fail_nexus + return 1 + fi + fi + if printf '%s' "$FI_FLAT" | grep -qE 'libeufin-nexus - IN .* CHF:[0-9]'; then + if printf '%s' "$FI_FLAT" | grep -qiE 'bounced.*missing reserve public key'; then + FI_STATUS=ok_bounce + return 0 + fi + if printf '%s' "$FI_FLAT" | grep -qiE 'bounced'; then + FI_STATUS=ok_bounce + return 0 + fi + FI_STATUS=ok_in + return 0 + fi + if [ "$FI_RC" -eq 0 ] && printf '%s' "$FI_FLAT" | grep -qi 'Faking incoming CHF'; then + FI_STATUS=ok_cli + return 0 + fi + FI_STATUS=fail_other + return 1 +} + +set_group cli +info "ssh" "DEVTESTING_SSH=${DEVTESTING_SSH} baseline=${DEVTESTING_AMOUNT} ladder=${DEVTESTING_LADDER} steps=${DEVTESTING_LADDER_STEPS} max=CHF:${DEVTESTING_LADDER_MAX}" + +help_out=$(ssh_dt fake-incoming --help 2>&1) || true +if printf '%s' "$help_out" | grep -q 'fake-incoming'; then + ok "cli" "taler-devtesting fake-incoming reachable via ${DEVTESTING_SSH}" +else + fail "cli" "cannot run taler-devtesting on ${DEVTESTING_SSH}" \ + "$(printf '%s' "$help_out" | tr '\n' ' ' | head -c 200)" + exit 1 +fi + +set_group geniban +iban_out=$(ssh_dt geniban 2>&1) || true +iban=$(printf '%s\n' "$iban_out" | grep -E '^CH[0-9A-Z]+$' | tail -1 || true) +if [ -n "$iban" ]; then + ok "geniban" "synthetic IBAN $iban (debit / “withdraw” side for each rung)" +else + fail "geniban" "expected CH… IBAN" "$(printf '%s' "$iban_out" | tr '\n' ' ' | head -c 160)" + exit 1 +fi + +# Credit payto: rusty libeufin-nexus only accepts the *production* exchange IBAN +# configured there (DEVTESTING_EXCHANGE_IBAN). Do not use stage /keys first-account. +set_group credit-payto +credit_payto="${DEVTESTING_CREDIT_PAYTO:-}" +if [ -z "$credit_payto" ]; then + credit_payto="payto://iban/${DEVTESTING_EXCHANGE_IBAN}?receiver-name=Taler+Operations+AG" +fi +ok "credit-payto" "$credit_payto" + +debit_payto="payto://iban/${iban}?receiver-name=MonDevTest" +ts_base=$(date -u +%Y%m%d%H%M%S) + +# --- baseline plumbing probe --- +set_group baseline +subject="mon-fake-franken-${ts_base}-base-$$" +info "fake-incoming" "amount=${DEVTESTING_AMOUNT} subject=${subject}" +if run_fake_incoming "$DEVTESTING_AMOUNT" "$subject" "$credit_payto" "$debit_payto"; then + case "$FI_STATUS" in + ok_bounce) + ok "fake-incoming" "CHF IN accepted by nexus · bounce/missing reserve pub — plumbing OK" + info "note" "use real reserve pub as --subject for wirewatch credit (not required for mon probe)" + ;; + ok_in) ok "fake-incoming" "CHF IN recorded by nexus (no bounce line)" ;; + ok_cli) ok "fake-incoming" "CLI completed CHF fake-incoming" ;; + *) ok "fake-incoming" "status=${FI_STATUS}" ;; + esac +else + case "$FI_STATUS" in + fail_ssh) + fail "fake-incoming" "SSH/CLI failure" "$(printf '%s' "$FI_FLAT" | head -c 200)" + exit 1 + ;; + fail_creditor) + fail "fake-incoming" "wrong credit IBAN (not exchange)" "$(printf '%s' "$FI_FLAT" | head -c 220)" + exit 1 + ;; + fail_nexus) + fail "fake-incoming" "nexus rejected request" "$(printf '%s' "$FI_FLAT" | head -c 220)" + exit 1 + ;; + *) + fail "fake-incoming" "no CHF IN line from nexus" "rc=${FI_RC} $(printf '%s' "$FI_FLAT" | head -c 200)" + exit 1 + ;; + esac +fi + +# --- CHF amount ladder (fake-incoming rungs) --- +if [ "${DEVTESTING_LADDER}" = "1" ]; then + set_group ladder + section "devtesting · CHF fake-incoming ladder (→ ${DEVTESTING_LADDER_MAX})" + + LADDER_FILE=$(mktemp "${TMPDIR:-/tmp}/devtesting-ladder.XXXXXX") + python3 - <<'PY' "${DEVTESTING_LADDER_MAX}" "${DEVTESTING_LADDER_STEPS}" "${DEVTESTING_LADDER_MIN}" "$LADDER_FILE" +import math, sys +from decimal import Decimal, ROUND_HALF_UP +from pathlib import Path + +max_amt = Decimal(sys.argv[1]) +steps = max(1, int(sys.argv[2])) +min_amt = Decimal(sys.argv[3]) +out = Path(sys.argv[4]) +if min_amt <= 0: + min_amt = Decimal("0.01") +if min_amt >= max_amt: + min_amt = max_amt / Decimal(1000) +# CHF: 2 decimal places +q2 = Decimal("0.01") + +def quant(v: Decimal) -> Decimal: + if v < min_amt: + v = min_amt + return v.quantize(q2, rounding=ROUND_HALF_UP) + +def fmt(v: Decimal) -> str: + q = quant(v) + if q == q.to_integral(): + return str(int(q)) + return format(q, "f") + +if steps == 1: + amts = [quant(max_amt)] +elif steps == 2: + amts = [quant(min_amt), quant(max_amt)] +else: + # Exactly `steps` log-spaced rungs from min → max (includes both ends). + # Guarantees max rung = DEVTESTING_LADDER_MAX (“enough” volume at the top). + amts = [] + lo, hi = float(min_amt), float(max_amt) + if hi <= lo: + hi = lo * 10 + for i in range(steps): + if steps == 1: + v = max_amt + else: + t = i / (steps - 1) + v = Decimal(str(math.exp(math.log(lo) + t * (math.log(hi) - math.log(lo))))) + amts.append(quant(v)) + amts[0] = quant(min_amt) + amts[-1] = quant(max_amt) + # enforce strictly increasing + for i in range(1, len(amts)): + if amts[i] <= amts[i - 1]: + amts[i] = quant(amts[i - 1] + min_amt) + amts[-1] = quant(max_amt) + if amts[-1] <= amts[-2]: + # last-but-one squeezed; leave max exact and drop collision by nudging previous + amts[-2] = quant(max(amts[-3] + min_amt if len(amts) > 2 else min_amt, max_amt - Decimal(1))) + if amts[-2] >= amts[-1]: + amts[-2] = quant(amts[-1] - min_amt) + +lines = ["CHF:%s" % fmt(a) for a in amts] +out.write_text("\n".join(lines) + "\n") +PY + n_rungs_plan=$(grep -cE '^CHF:' "$LADDER_FILE" || echo 0) + ladder_sum=$(python3 -c ' +from decimal import Decimal +s=Decimal(0) +for ln in open("'"$LADDER_FILE"'"): + ln=ln.strip() + if not ln: continue + s += Decimal(ln.split(":",1)[-1]) +print(s) +' 2>/dev/null || echo "?") + + info "ladder plan" "steps=${n_rungs_plan} max=CHF:${DEVTESTING_LADDER_MAX} min=CHF:${DEVTESTING_LADDER_MIN} sum≈CHF:${ladder_sum} (synthetic debit geniban each rung)" + info "ladder funding" "each rung uses geniban debit payto — no real bank withdraw; volume covers max rung CHF:${DEVTESTING_LADDER_MAX}" + + LADDER_OK=0 + LADDER_FAIL=0 + LADDER_WARN=0 + rung=0 + # Load amounts into an array first — never `while read … done /dev/null 2>&1; then - if [ -n "${WSERVE_PID:-}" ] && [ -S "${WSOCK:-}" ]; then - perl -e 'alarm shift; exec @ARGV' 10 \ - node "$WALLET_CLI" --wallet-connection="$WSOCK" --no-throttle balance \ + local _node="${NODE_BIN:-$(command -v node 2>/dev/null || true)}" + [ -n "$_node" ] || { echo "(no node)" >"$out"; return 1; } + if [ -n "${WSERVE_PID:-}" ] && [ -S "${WSOCK:-}" ]; then + if command -v timeout >/dev/null 2>&1; then + timeout -k 2 10 "$_node" "$WALLET_CLI" --wallet-connection="$WSOCK" --no-throttle balance \ >"$out" 2>"$err" || true else - perl -e 'alarm shift; exec @ARGV' 10 \ - node "$WALLET_CLI" --wallet-db="$WDB" --no-throttle --skip-defaults balance \ + "$_node" "$WALLET_CLI" --wallet-connection="$WSOCK" --no-throttle balance \ >"$out" 2>"$err" || true fi else - if [ -n "${WSERVE_PID:-}" ] && [ -S "${WSOCK:-}" ]; then - node "$WALLET_CLI" --wallet-connection="$WSOCK" --no-throttle balance \ + if command -v timeout >/dev/null 2>&1; then + timeout -k 2 10 "$_node" "$WALLET_CLI" --wallet-db="$WDB" --no-throttle --skip-defaults balance \ >"$out" 2>"$err" || true else - node "$WALLET_CLI" --wallet-db="$WDB" --no-throttle --skip-defaults balance \ + "$_node" "$WALLET_CLI" --wallet-db="$WDB" --no-throttle --skip-defaults balance \ >"$out" 2>"$err" || true fi fi @@ -226,18 +226,20 @@ except Exception: print("")' 2>/dev/null || true) # --------------------------------------------------------------------------- # Variable amounts -# Withdraw = typical ATM notes only (Bankomat): 20 / 50 / 100 / 200 … +# Withdraw = typical bank cash-note amounts (CHF-style: was called ATM): 20 / 50 / 100 / 200 … # Pay = several smaller purchase amounts # Env: # E2E_WITHDRAW_VALUES="20 50 100" # bare numbers → prefixed with CUR # E2E_PAY_VALUES="0.01 0.05 0.5 1 2 5" # E2E_VARIABLE=0 # single fixed WITHDRAW_AMT / PAY_AMT -# E2E_ATM_MAX=N # try at most N ATM withdraws (budget) +# E2E_BANK_WITHDRAW_MAX=N # try at most N bank withdraws (budget) +# E2E_ATM_MAX=N # legacy alias of E2E_BANK_WITHDRAW_MAX # --------------------------------------------------------------------------- : "${E2E_VARIABLE:=1}" -: "${E2E_ATM_MAX:=5}" +: "${E2E_BANK_WITHDRAW_MAX:=${E2E_ATM_MAX:-5}}" +E2E_ATM_MAX="${E2E_ATM_MAX:-$E2E_BANK_WITHDRAW_MAX}" # legacy alias if [ "$E2E_REMOTE" = "1" ]; then - # remote: still ATM-shaped; TESTPAYSAN needs larger notes for shop pays + # remote: still bank cash-note shaped; TESTPAYSAN needs larger notes for shop pays if [ "$CUR" = "TESTPAYSAN" ]; then : "${E2E_WITHDRAW_VALUES:=20 50}" # farmer shop template face values (oeufs-6=5, fromage=8.5, jus=6) @@ -247,7 +249,7 @@ if [ "$E2E_REMOTE" = "1" ]; then : "${E2E_PAY_VALUES:=0.01 0.05 0.1 1}" fi else - # local GOA: classic ATM notes + 4200 for paivana paywall template + # local GOA: classic bank cash-note amounts + 4200 for paivana paywall template : "${E2E_WITHDRAW_VALUES:=20 50 100 200 4200}" : "${E2E_PAY_VALUES:=0.01 0.05 0.1 0.5 1 2 5 10}" fi @@ -284,7 +286,7 @@ if [ "$E2E_VARIABLE" != "1" ]; then fi fi -# Credit = sum of planned ATM withdraws + small buffer for fees +# Credit = sum of planned Bank withdraws + small buffer for fees CREDIT_AMT=$(python3 -c ' import sys cur=sys.argv[1] @@ -301,7 +303,7 @@ print("%s:%g" % (cur, s + buf)) WITHDRAW_AMT=$(printf '%s' "$WITHDRAW_LIST" | awk '{print $1}') PAY_AMT=$(printf '%s' "$PAY_LIST" | awk '{print $1}') -# More wall time when running ATM ladder +# More wall time when running Bank withdraw ladder if [ "$E2E_VARIABLE" = "1" ]; then n_w=$(printf '%s' "$WITHDRAW_LIST" | wc -w | tr -d ' ') n_p=$(printf '%s' "$PAY_LIST" | wc -w | tr -d ' ') @@ -309,9 +311,9 @@ if [ "$E2E_VARIABLE" = "1" ]; then if [ "${E2E_TIMEOUT}" -lt "$need" ]; then E2E_TIMEOUT=$need fi - # cap ATM attempts but always keep CUR:4200 if listed (paivana) + # cap bank withdraw attempts but always keep CUR:4200 if listed (paivana) WITHDRAW_LIST=$( - printf '%s' "$WITHDRAW_LIST" | tr ' ' '\n' | awk -v max="$E2E_ATM_MAX" -v big="${CUR}:4200" ' + printf '%s' "$WITHDRAW_LIST" | tr ' ' '\n' | awk -v max="$E2E_BANK_WITHDRAW_MAX" -v big="${CUR}:4200" ' NF==0 { next } $0==big { has=1; next } { small[++n]=$0 } @@ -376,13 +378,35 @@ WALLET_CLI=$(find_wallet_cli) || { blocker "prereq" "taler-wallet-cli not found (set WALLET_CLI=)" exit 1 } +# wallet-core spawns `taler-helper-sqlite3` via PATH (not an absolute path). +# Ensure operator wallet bindir + ~/.local/bin are first (systemd mon jobs). +export PATH="${WALLET_CLI_BINDIR:+$WALLET_CLI_BINDIR:}${HOME}/.local/bin:/usr/local/bin:/usr/bin:/bin:${PATH:-}" +NODE_BIN=$(command -v node 2>/dev/null || true) +if [ -z "$NODE_BIN" ] || [ ! -x "$NODE_BIN" ]; then + blocker "prereq" "node not found on PATH (install Node or set PATH; need node for wallet-cli)" + exit 1 +fi +if ! command -v taler-helper-sqlite3 >/dev/null 2>&1; then + for _h in \ + "${WALLET_CLI_BINDIR:+$WALLET_CLI_BINDIR/taler-helper-sqlite3}" \ + "${HOME}/.local/bin/taler-helper-sqlite3" \ + /usr/bin/taler-helper-sqlite3 + do + [ -n "$_h" ] && [ -x "$_h" ] && export PATH="$(dirname "$_h"):$PATH" && break + done +fi +if ! command -v taler-helper-sqlite3 >/dev/null 2>&1; then + blocker "prereq" "taler-helper-sqlite3 not on PATH (wallet accept-uri/pay will fail with spawn ENOENT) — install next to wallet-cli or set WALLET_CLI_BINDIR" + exit 1 +fi +ok "wallet-helper" "taler-helper-sqlite3=$(command -v taler-helper-sqlite3)" # Implementation version (--version) + wallet-core API ranges (version command) WALLET_IMPL_VER=$( - node "$WALLET_CLI" --version 2>/dev/null | head -1 | tr -d '\r' || true + "$NODE_BIN" "$WALLET_CLI" --version 2>/dev/null | head -1 | tr -d '\r' || true ) WALLET_CORE_VER=$( _wver_db=$(mktemp "${TMPDIR:-/tmp}/wver.XXXXXX.sqlite3" 2>/dev/null || echo "${TMPDIR:-/tmp}/wver-$$.sqlite3") - node "$WALLET_CLI" --wallet-db="$_wver_db" --no-throttle --skip-defaults version 2>/dev/null \ + "$NODE_BIN" "$WALLET_CLI" --wallet-db="$_wver_db" --no-throttle --skip-defaults version 2>/dev/null \ | python3 -c ' import json,re,sys raw=sys.stdin.read() @@ -413,7 +437,7 @@ else fi info "e2e mode" "$([ "$E2E_REMOTE" = "1" ] && echo "remote/public domain (no SSH)" || echo "local koopa stack")" info "currency" "$CUR" -info "ATM withdraw ladder" "$WITHDRAW_LIST (credit $CREDIT_AMT)" +info "Bank withdraw ladder" "$WITHDRAW_LIST (credit $CREDIT_AMT)" info "pay ladder" "$PAY_LIST" info "e2e budget" "${E2E_TIMEOUT}s" @@ -426,62 +450,61 @@ else warn "alt_unit_names" "using built-in SI fallback" fi -# Host + bank/exchange/merchant RAM/CPU/load (SSH koopa or koopa-external) +# Host + bank/exchange/merchant RAM/CPU/load (SSH via KOOPA_SSH from env) set_group load section "e2e · load snapshot (before withdraw/pay)" metrics_report_load "${METRICS_DIR}/load-before.json" "e2e-start" || true -# Local stack: koopa secrets. Remote: env first; TESTPAYSAN stage may load francpaysan host secrets. +# Local stack: koopa secrets. Remote: env first; TESTPAYSAN stage may load stagepaysan secrets via FRANCPAYSAN_SSH/INSIDE_SSH. set_group prereq if [ "$E2E_REMOTE" = "1" ]; then ADMIN_PASS="${E2E_BANK_ADMIN_PASS:-${BANK_ADMIN_PASS:-}}" MPW="${E2E_MERCHANT_TOKEN:-${MERCHANT_TOKEN:-}}" # Stage FrancPaysan: allow admin credit without committing passwords. - # Order: already-set env → local secrets tree → SSH francpaysan-host (ops laptop). + # Order: env → FRANCPAYSAN_SECRETS → SSH FRANCPAYSAN_SSH/INSIDE_SSH + remote secret paths. if [ -z "$ADMIN_PASS" ] && [ "${CUR:-${EXPECT_CURRENCY:-}}" = "TESTPAYSAN" ]; then _fp_try="" - for _fp_try in \ - "${FRANCPAYSAN_SECRETS:-}/stage/bank-admin-password.txt" \ - "${HOME}/francpaysan-secrets/stage/bank-admin-password.txt" \ - "${HOME}/src/francpaysan-secrets/stage/bank-admin-password.txt" \ - "${HOME}/taler/src/francpaysan-secrets/stage/bank-admin-password.txt" - do - if [ -n "$_fp_try" ] && [ -f "$_fp_try" ]; then + if [ -n "${FRANCPAYSAN_SECRETS:-}" ]; then + _fp_try="${FRANCPAYSAN_SECRETS}/stage/bank-admin-password.txt" + if [ -f "$_fp_try" ]; then ADMIN_PASS=$(tr -d '\n\r' <"$_fp_try") info "bank admin secret" "from $_fp_try" - break fi - done + fi if [ -z "$ADMIN_PASS" ] && command -v ssh >/dev/null 2>&1; then - for _fp_host in "${FRANCPAYSAN_SSH:-francpaysan-host}" francpaysan-host; do + _remote_bank_sec="${FRANCPAYSAN_REMOTE_BANK_ADMIN_SECRET:-}" + [ -z "$_remote_bank_sec" ] && [ -n "${FRANCPAYSAN_REMOTE_SECRETS_ROOT:-}" ] && \ + _remote_bank_sec="${FRANCPAYSAN_REMOTE_SECRETS_ROOT}/bank/secrets/bank-admin-password.txt" + for _fp_host in ${FRANCPAYSAN_SSH:+"$FRANCPAYSAN_SSH"} ${INSIDE_SSH:+"$INSIDE_SSH"}; do + [ -n "$_fp_host" ] && [ -n "$_remote_bank_sec" ] || continue ADMIN_PASS=$(ssh -o BatchMode=yes -o ConnectTimeout=8 "$_fp_host" \ - 'sudo cat /mnt/data/stagepaysan/bank/secrets/bank-admin-password.txt 2>/dev/null' \ + "sudo cat ${_remote_bank_sec} 2>/dev/null" \ 2>/dev/null | tr -d '\n\r' || true) if [ -n "$ADMIN_PASS" ]; then - info "bank admin secret" "from ${_fp_host}:/mnt/data/stagepaysan/bank/secrets/" + info "bank admin secret" "from ${_fp_host} (remote path via env)" break fi done fi - unset _fp_try _fp_host + unset _fp_try _fp_host _remote_bank_sec fi # Stage merchant: public templates need no token; optional default-instance token if [ -z "$MPW" ] && [ "${CUR:-${EXPECT_CURRENCY:-}}" = "TESTPAYSAN" ]; then - for _fp_try in \ - "${FRANCPAYSAN_SECRETS:-}/stage/default-instance-token.txt" \ - "${HOME}/francpaysan-secrets/stage/default-instance-token.txt" - do - if [ -n "$_fp_try" ] && [ -f "$_fp_try" ]; then - MPW=$(tr -d '\n\r' <"$_fp_try") - break - fi - done - if [ -z "$MPW" ] && command -v ssh >/dev/null 2>&1; then - MPW=$(ssh -o BatchMode=yes -o ConnectTimeout=8 "${FRANCPAYSAN_SSH:-francpaysan-host}" \ - 'sudo cat /mnt/data/stagepaysan/merchant/secrets/default-instance-token.txt 2>/dev/null' \ - 2>/dev/null | tr -d '\n\r' || true) + if [ -n "${FRANCPAYSAN_SECRETS:-}" ] && [ -f "${FRANCPAYSAN_SECRETS}/stage/default-instance-token.txt" ]; then + MPW=$(tr -d '\n\r' <"${FRANCPAYSAN_SECRETS}/stage/default-instance-token.txt") + fi + if [ -z "$MPW" ] && command -v ssh >/dev/null 2>&1; then + _remote_mer_sec="${FRANCPAYSAN_REMOTE_MERCHANT_TOKEN:-}" + [ -z "$_remote_mer_sec" ] && [ -n "${FRANCPAYSAN_REMOTE_SECRETS_ROOT:-}" ] && \ + _remote_mer_sec="${FRANCPAYSAN_REMOTE_SECRETS_ROOT}/merchant/secrets/default-instance-token.txt" + _fp_host="${FRANCPAYSAN_SSH:-${INSIDE_SSH:-}}" + if [ -n "$_fp_host" ] && [ -n "$_remote_mer_sec" ]; then + MPW=$(ssh -o BatchMode=yes -o ConnectTimeout=8 "$_fp_host" \ + "sudo cat ${_remote_mer_sec} 2>/dev/null" \ + 2>/dev/null | tr -d '\n\r' || true) + fi + unset _fp_host _remote_mer_sec fi - unset _fp_try fi else ADMIN_PASS="${E2E_BANK_ADMIN_PASS:-$(read_secret "taler-bank/bank-admin-password.txt" || true)}" @@ -576,7 +599,7 @@ start_wallet_serve() { stop_wallet_serve rm -f "$WSOCK" # serve holds the DB open; clients use --wallet-connection only - node "$WALLET_CLI" --wallet-db="$WDB" --no-throttle --skip-defaults \ + "$NODE_BIN" "$WALLET_CLI" --wallet-db="$WDB" --no-throttle --skip-defaults \ advanced serve --unix-path "$WSOCK" \ >"$SCRATCH/wallet-serve.log" 2>&1 & WSERVE_PID=$! @@ -593,6 +616,25 @@ start_wallet_serve() { return 1 } +# Run node wallet-cli with a hard wall clock. Prefer GNU/coreutils `timeout` +# (clear errors). Fall back to perl alarm only if timeout is missing. +# Always use absolute NODE_BIN + WALLET_CLI so systemd PATH quirks cannot +# turn into opaque "alarm shift; exec @ARGV" noise in accept-uri output. +_wcli_exec() { + local cap="$1" + shift + local -a cmd=( "$NODE_BIN" "$WALLET_CLI" "$@" ) + if command -v timeout >/dev/null 2>&1; then + timeout -k 2 "$cap" "${cmd[@]}" + elif command -v gtimeout >/dev/null 2>&1; then + gtimeout --kill-after=2 "$cap" "${cmd[@]}" + elif command -v perl >/dev/null 2>&1; then + perl -e 'alarm shift; exec @ARGV' "$cap" "${cmd[@]}" + else + "${cmd[@]}" + fi +} + # $1 = max seconds for this call (optional); rest = wallet-cli args # Prefer live serve socket so the shepherd stays up (alternative to run-until-done). wcli() { @@ -604,22 +646,15 @@ wcli() { e2e_over && return 124 local cap cap=$(e2e_left) - [ "$cap" -gt "$maxc" ] && cap=$maxc - [ "$cap" -lt 3 ] && return 124 + # e2e_left may print warnings on stdout — keep only first integer token + cap=$(printf '%s' "$cap" | awk '/^[0-9]+$/{print; exit} {for(i=1;i<=NF;i++) if($i+0==$i){print $i; exit}}') + [ -z "$cap" ] && cap=$maxc + [ "$cap" -gt "$maxc" ] 2>/dev/null && cap=$maxc + [ "$cap" -lt 3 ] 2>/dev/null && return 124 if [ -n "${WSERVE_PID:-}" ] && [ -S "${WSOCK:-}" ]; then - if command -v perl >/dev/null 2>&1; then - perl -e 'alarm shift; exec @ARGV' "$cap" \ - node "$WALLET_CLI" --wallet-connection="$WSOCK" --no-throttle "$@" - else - node "$WALLET_CLI" --wallet-connection="$WSOCK" --no-throttle "$@" - fi + _wcli_exec "$cap" --wallet-connection="$WSOCK" --no-throttle "$@" else - if command -v perl >/dev/null 2>&1; then - perl -e 'alarm shift; exec @ARGV' "$cap" \ - node "$WALLET_CLI" --wallet-db="$WDB" --no-throttle --skip-defaults "$@" - else - node "$WALLET_CLI" --wallet-db="$WDB" --no-throttle --skip-defaults "$@" - fi + _wcli_exec "$cap" --wallet-db="$WDB" --no-throttle --skip-defaults "$@" fi } @@ -628,9 +663,10 @@ wcli_pay() { local cap=$E2E_PAY_SECS local left left=$(e2e_left) - if [ "$left" -gt "$cap" ]; then + left=$(printf '%s' "$left" | awk '/^[0-9]+$/{print; exit} {for(i=1;i<=NF;i++) if($i+0==$i){print $i; exit}}') + if [ -n "$left" ] && [ "$left" -gt "$cap" ] 2>/dev/null; then cap=$E2E_PAY_SECS - elif [ "$left" -ge 8 ]; then + elif [ -n "$left" ] && [ "$left" -ge 8 ] 2>/dev/null; then cap=$left else # still try once with floor 12s so Alarm clock is not the blocker @@ -638,19 +674,9 @@ wcli_pay() { fi info "pay wallet cap" "${cap}s" if [ -n "${WSERVE_PID:-}" ] && [ -S "${WSOCK:-}" ]; then - if command -v perl >/dev/null 2>&1; then - perl -e 'alarm shift; exec @ARGV' "$cap" \ - node "$WALLET_CLI" --wallet-connection="$WSOCK" --no-throttle "$@" - else - node "$WALLET_CLI" --wallet-connection="$WSOCK" --no-throttle "$@" - fi + _wcli_exec "$cap" --wallet-connection="$WSOCK" --no-throttle "$@" else - if command -v perl >/dev/null 2>&1; then - perl -e 'alarm shift; exec @ARGV' "$cap" \ - node "$WALLET_CLI" --wallet-db="$WDB" --no-throttle --skip-defaults "$@" - else - node "$WALLET_CLI" --wallet-db="$WDB" --no-throttle --skip-defaults "$@" - fi + _wcli_exec "$cap" --wallet-db="$WDB" --no-throttle --skip-defaults "$@" fi } @@ -864,7 +890,7 @@ else warn "wallet accept ToS" "$(tail -c 80 "$SCRATCH/ex-tos.out" | tr '\n' ' ')" fi fi -# Empty wallet baseline before ATM / pay load +# Empty wallet baseline before bank withdraw / pay load metrics_report_coins "wallet-ready" || true wd_status() { @@ -902,15 +928,15 @@ fake_incoming_speedup() { fi } -# One ATM-style withdraw: create → accept → select → confirm → settle +# One bank withdraw: create → accept → select → confirm → settle # returns 0 on wallet balance increase, 1 on soft fail (ladder continues) e2e_one_withdraw() { WITHDRAW_AMT="$1" local tag tag=$(printf '%s' "$WITHDRAW_AMT" | tr '.:' '__') - section "e2e · ATM withdraw $WITHDRAW_AMT · $(format_amount_alt "$WITHDRAW_AMT")" - e2e_over && { warn "ATM withdraw" "budget exhausted — skip $WITHDRAW_AMT"; return 1; } - metrics_report_coins "before-ATM-${tag}" || true + section "e2e · Bank withdraw $WITHDRAW_AMT · $(format_amount_alt "$WITHDRAW_AMT")" + e2e_over && { warn "Bank withdraw" "budget exhausted — skip $WITHDRAW_AMT"; return 1; } + metrics_report_coins "before-bank-wd-${tag}" || true curl -sS -m 15 -o "$SCRATCH/wd-$tag.json" -H "Authorization: Bearer $UT" \ -H 'Content-Type: application/json' \ @@ -918,23 +944,29 @@ e2e_one_withdraw() { "$BANK/accounts/${USER}/withdrawals" WID=$(python3 -c 'import json;d=json.load(open("'"$SCRATCH"'/wd-'"$tag"'.json"));print(d.get("withdrawal_id") or d.get("id") or "")' 2>/dev/null || true) URI=$(python3 -c 'import json;d=json.load(open("'"$SCRATCH"'/wd-'"$tag"'.json"));print(d.get("taler_withdraw_uri") or "")' 2>/dev/null || true) - URI_CLEAN=$(python3 -c 'import json;u=json.load(open("'"$SCRATCH"'/wd-'"$tag"'.json")).get("taler_withdraw_uri") or "";print(u.replace(":443/","/"))' 2>/dev/null || true) + URI_CLEAN=$(normalize_taler_uri "$URI") if [ -z "$WID" ] || [ -z "$URI_CLEAN" ]; then - warn "ATM withdraw $WITHDRAW_AMT" "create failed — $(head -c 80 "$SCRATCH/wd-$tag.json" | tr '\n' ' ')" + warn "Bank withdraw $WITHDRAW_AMT" "create failed — $(head -c 80 "$SCRATCH/wd-$tag.json" | tr '\n' ' ')" return 1 fi - ok "ATM withdrawal created $WITHDRAW_AMT ($WID)" + ok "Bank withdrawal created $WITHDRAW_AMT ($WID)" cp "$SCRATCH/wd-$tag.json" "$SCRATCH/wd.json" - USE_URI="$URI" - [ -z "$USE_URI" ] && USE_URI="$URI_CLEAN" + # Always prefer stripped URI (libeufin often emits host:443) + USE_URI="${URI_CLEAN:-$URI}" if wcli withdraw accept-uri --exchange "${EXCHANGE_PUBLIC}/" "$USE_URI" >"$SCRATCH/accept-$tag.out" 2>&1; then ok "wallet accept $WITHDRAW_AMT" - elif [ "$USE_URI" != "$URI_CLEAN" ] && [ -n "$URI_CLEAN" ] \ - && wcli withdraw accept-uri --exchange "${EXCHANGE_PUBLIC}/" "$URI_CLEAN" >"$SCRATCH/accept-$tag.out" 2>&1; then - ok "wallet accept $WITHDRAW_AMT (no :443)" + elif [ -n "$URI" ] && [ "$USE_URI" != "$URI" ] \ + && wcli withdraw accept-uri --exchange "${EXCHANGE_PUBLIC}/" "$URI" >"$SCRATCH/accept-$tag.out" 2>&1; then + ok "wallet accept $WITHDRAW_AMT (raw URI fallback)" else - warn "ATM withdraw $WITHDRAW_AMT" "accept-uri failed — $(tail -c 100 "$SCRATCH/accept-$tag.out" | tr '\n' ' ')" + # Prefer the real Error/ENOENT line over noise (perl -e source, stack frames) + _acc_err=$( + grep -E 'Error:|ENOENT|spawn taler-helper|code:|An error occurred' \ + "$SCRATCH/accept-$tag.out" 2>/dev/null | head -3 | tr '\n' ' ' | head -c 200 + ) + [ -z "$_acc_err" ] && _acc_err=$(tail -c 160 "$SCRATCH/accept-$tag.out" 2>/dev/null | tr '\n' ' ') + warn "Bank withdraw $WITHDRAW_AMT" "accept-uri failed — ${_acc_err:-unknown}" return 1 fi cp "$SCRATCH/accept-$tag.out" "$SCRATCH/accept.out" @@ -976,7 +1008,7 @@ else: fi fi if [ "$st" != "selected" ] && [ "$st" != "confirmed" ]; then - warn "ATM withdraw $WITHDRAW_AMT" "not selected (status=${st:-?})" + warn "Bank withdraw $WITHDRAW_AMT" "not selected (status=${st:-?})" return 1 fi if [ "$st" != "confirmed" ]; then @@ -985,32 +1017,32 @@ else: "$BANK/accounts/${USER}/withdrawals/${WID}/confirm") case "$code" in 200|204) ok "bank confirm $WITHDRAW_AMT" ;; - *) warn "ATM withdraw $WITHDRAW_AMT" "confirm HTTP $code"; return 1 ;; + *) warn "Bank withdraw $WITHDRAW_AMT" "confirm HTTP $code"; return 1 ;; esac fi fake_incoming_speedup - # Short per-ATM settle: poll balance + bank transfer_done only (never run-until-done) + # Short per-withdraw settle: poll balance + bank transfer_done only (never run-until-done) local ok_bal=0 r av xfer for r in 1 2 3 4 5 6 8 10; do wcli_bal_snap "$SCRATCH/bal-$tag.out" || wcli balance >"$SCRATCH/bal-$tag.out" 2>&1 || true av=$(wallet_avail_num "$SCRATCH/bal-$tag.out") if python3 -c "import sys; sys.exit(0 if float(sys.argv[1]) > 0 else 1)" "$av" 2>/dev/null; then ok_bal=1 - ok "wallet funded after ATM $WITHDRAW_AMT" "avail=${CUR}:${av}" + ok "wallet funded after bank withdraw $WITHDRAW_AMT" "avail=${CUR}:${av}" info "balance" "$(fmt_bal "$SCRATCH/bal-$tag.out")" break fi xfer=$(curl -sS -m 8 "$BANK/taler-integration/withdrawal-operation/${WID}" 2>/dev/null \ | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d.get("transfer_done"), d.get("status"))' 2>/dev/null || echo "?") if echo "$xfer" | grep -qi True; then - ok "ATM $WITHDRAW_AMT bank transfer_done" "wallet avail=${CUR}:${av} ($xfer) — no run-until-done" + ok "Bank withdraw $WITHDRAW_AMT bank transfer_done" "wallet avail=${CUR}:${av} ($xfer) — no run-until-done" ok_bal=1 break fi sleep 2 done - metrics_report_coins "after-ATM-${tag}" || true + metrics_report_coins "after-bank-wd-${tag}" || true if [ "$ok_bal" = "1" ]; then metrics_record_flow withdrawn "$WITHDRAW_AMT" || true return 0 @@ -1018,10 +1050,10 @@ else: # Bank side often already confirmed — treat as timing lag, not hard fail st=$(wd_status) if [ "$st" = "confirmed" ]; then - warn "ATM withdraw $WITHDRAW_AMT" "bank confirmed; wallet not funded yet (settlement timing — will recheck later)" + warn "Bank withdraw $WITHDRAW_AMT" "bank confirmed; wallet not funded yet (settlement timing — will recheck later)" return 2 fi - warn "ATM withdraw $WITHDRAW_AMT" "no wallet balance yet (status=${st:-?})" + warn "Bank withdraw $WITHDRAW_AMT" "no wallet balance yet (status=${st:-?})" return 1 } @@ -1336,8 +1368,8 @@ sys.exit(1) } # --------------------------------------------------------------------------- -set_group atm -section "e2e · ATM withdraw ladder" +set_group bankwd +section "e2e · Bank withdraw ladder" # --------------------------------------------------------------------------- WITHDRAW_OK=0 WITHDRAW_OK_N=0 @@ -1345,7 +1377,7 @@ WITHDRAW_LAG_N=0 WITHDRAW_FAIL_N=0 WITHDRAW_REPORT="" for WITHDRAW_AMT in $WITHDRAW_LIST; do - e2e_over && { warn "ATM ladder" "time budget low — stopping more ATM withdraws (not a protocol error)"; break; } + e2e_over && { warn "Bank withdraw ladder" "time budget low — stopping more Bank withdraws (not a protocol error)"; break; } set +e e2e_one_withdraw "$WITHDRAW_AMT" wc=$? @@ -1366,26 +1398,26 @@ for WITHDRAW_AMT in $WITHDRAW_LIST; do ;; esac done -info "ATM withdraw summary" "$WITHDRAW_REPORT (ok=$WITHDRAW_OK_N lag=$WITHDRAW_LAG_N fail=$WITHDRAW_FAIL_N)" +info "Bank withdraw summary" "$WITHDRAW_REPORT (ok=$WITHDRAW_OK_N lag=$WITHDRAW_LAG_N fail=$WITHDRAW_FAIL_N)" set_group load -section "e2e · load snapshot (after ATM withdraws)" +section "e2e · load snapshot (after Bank withdraws)" metrics_report_load "${METRICS_DIR}/load-after-withdraw.json" "after-withdraw" || true cp -f "${METRICS_DIR}/load-after-withdraw.json" "${METRICS_DIR}/load-after.json" 2>/dev/null || true -metrics_report_coins "after-ATM-ladder" || true +metrics_report_coins "after-bank-wd-ladder" || true # Settlement catch-up: bank may have confirmed while wallet was still empty set_group settle section "e2e · wallet settlement (timing)" if ! wait_wallet_balance 0 "${E2E_SETTLE_ROUNDS}" "${E2E_SETTLE_SLEEP}"; then if [ "$WITHDRAW_LAG_N" -gt 0 ] || [ "$WITHDRAW_OK_N" -gt 0 ]; then - warn "settlement timing" "ATM path reached bank confirm (lag=$WITHDRAW_LAG_N ok=$WITHDRAW_OK_N) but wallet empty after wait — TIME lag, not protocol error" + warn "settlement timing" "bank withdraw path reached bank confirm (lag=$WITHDRAW_LAG_N ok=$WITHDRAW_OK_N) but wallet empty after wait — TIME lag, not protocol error" fi fi av_now=$(wallet_avail_num) if python3 -c "import sys; sys.exit(0 if float(sys.argv[1]) > 0 else 1)" "$av_now" 2>/dev/null; then WITHDRAW_OK=1 if [ "$WITHDRAW_OK_N" = "0" ]; then - warn "settlement timing" "coins arrived after ATM ladder (${CUR}:${av_now}) — earlier 'FAIL' was timing lag" + warn "settlement timing" "coins arrived after Bank withdraw ladder (${CUR}:${av_now}) — earlier 'FAIL' was timing lag" WITHDRAW_REPORT="${WITHDRAW_REPORT} → late-OK avail=${CUR}:${av_now}" fi ok "spendable balance for payments" "${CUR}:${av_now}" @@ -1404,7 +1436,7 @@ else WITHDRAW_OK=1 warn "settlement timing" "coins present after dig wait (${CUR}:${av_now}) — timing, not blocker" else - blocker "withdraw-settle" "no spendable ${CUR} after ATM ladder + settle wait ($WITHDRAW_LIST)" + blocker "withdraw-settle" "no spendable ${CUR} after Bank withdraw ladder + settle wait ($WITHDRAW_LIST)" section "e2e · report" info "WITHDRAW" "$WITHDRAW_REPORT — no coins after extended wait" info "PAY" "SKIPPED (no spendable balance after settle wait)" @@ -1597,7 +1629,7 @@ set_group paivana section "e2e · paivana paywall (template GOA:4200)" # --------------------------------------------------------------------------- # Public paywall: https://paivana.hacktivism.ch → 302 to merchant template "paivana". -# Requires spendable ≥ 4200 GOA (ATM ladder includes GOA:4200 by default). +# Requires spendable ≥ 4200 GOA (Bank withdraw ladder includes GOA:4200 by default). : "${PAIVANA_PUBLIC:=https://paivana.hacktivism.ch}" : "${E2E_PAIVANA:=1}" : "${E2E_PAIVANA_TEMPLATE:=paivana}" @@ -1631,11 +1663,11 @@ else ;; esac - # Ensure wallet can cover 4200 (extra ATM if ladder did not fund enough) + # Ensure wallet can cover 4200 (extra bank withdraw if ladder did not fund enough) pay_need=$(python3 -c 'import sys; print(float(sys.argv[1].split(":",1)[-1]))' "$E2E_PAIVANA_AMOUNT" 2>/dev/null || echo 4200) av_now=$(wallet_avail_num) if ! python3 -c "import sys; sys.exit(0 if float(sys.argv[1]) + 1e-9 >= float(sys.argv[2]) else 1)" "$av_now" "$pay_need" 2>/dev/null; then - info "paivana" "avail ${CUR}:${av_now} < ${pay_need} — ATM withdraw ${E2E_PAIVANA_AMOUNT}" + info "paivana" "avail ${CUR}:${av_now} < ${pay_need} — Bank withdraw ${E2E_PAIVANA_AMOUNT}" e2e_over && warn "paivana" "budget low before 4200 withdraw" set +e e2e_one_withdraw "$E2E_PAIVANA_AMOUNT" @@ -1688,7 +1720,7 @@ fi set_group report section "e2e · report" info "user" "$USER" -info "ATM withdraws" "$WITHDRAW_REPORT" +info "Bank withdraws" "$WITHDRAW_REPORT" info "payments" "$PAY_REPORT" info "shop" "${SHOP_REPORT:-(n/a)}" info "paivana" "${PAIVANA_REPORT:-(n/a)}" diff --git a/check_goa_ladder.sh b/check_goa_ladder.sh deleted file mode 100755 index d7c60ae..0000000 --- a/check_goa_ladder.sh +++ /dev/null @@ -1,1364 +0,0 @@ -#!/usr/bin/env bash -# check_goa_ladder.sh — withdraw/pay amount ladder (GOA + stage TESTPAYSAN) -# -# Flow (bank landings): -# 1) GET /intro/auto-account.json → personal *account-* (balance 0) -# 2) Mint pool withdrawals as explorer + confirm when selected -# 3) wallet-cli accept-uri only (no run-until-done) -# 4) bank confirm when selected; settle = balance + transfer_done -# -# Amounts: random strictly increasing; pins 0, max-1, max. -# Phase A: withdraw ladder (cumulative wallet). Phase B: pay ladder. -# -# Stacks: -# GOA — LADDER_MAX_AMOUNT = libeufin ceiling; explorer from koopa-admin-secrets -# TESTPAYSAN stage — auto max_wire from bank /config, min denom from /keys; -# explorer from stagepaysan secrets / SSH -# -# Usage: -# ./taler-monitoring.sh ladder -# ./taler-monitoring.sh -d stage.lefrancpaysan.ch ladder -# LADDER_MAX_AMOUNT=100 LADDER_STEPS=7 ./taler-monitoring.sh -d stage.lefrancpaysan.ch ladder -# -# Env (see body): LADDER_STEPS, LADDER_MAX_AMOUNT, LADDER_MIN_AMOUNT, LADDER_STACK_AUTO, -# LADDER_PAY, EXP_PW / EXP_PW_FILE, CLI_JS, MERCHANT_INSTANCE, … -set -euo pipefail -ROOT=$(cd "$(dirname "$0")" && pwd) -# shellcheck source=lib.sh -source "$ROOT/lib.sh" -# When invoked standalone (not via taler-monitoring.sh), still load secrets.env -load_monitoring_secrets_env 2>/dev/null || true - -# Area ladder.* — withdraw/pay amount ladder (GOA + TESTPAYSAN) -# Groups: ladder.plan / ladder.load / ladder.withdraw / ladder.pay / ladder.report -set_area ladder -set_group plan -SECTION_T0=$(date +%s) -now_ms() { python3 -c 'import time; print(int(time.time()*1000))'; } -elapsed_ms() { - # elapsed_ms START_MS - python3 -c 'import sys; print(int(sys.argv[1]) - int(sys.argv[2]))' "$(now_ms)" "$1" -} - -: "${LADDER_TIMEOUT_S:=3600}" -: "${LADDER_SETTLE_ROUNDS:=18}" -: "${LADDER_SETTLE_SLEEP:=2}" -: "${EXP_USER:=explorer}" -# EXP_PW_FILE / EXP_PW optional overrides; otherwise read_secret + SECRETS_ROOT / stage SSH -: "${EXP_PW_FILE:=}" -: "${EXP_PW:=}" -: "${CLI_JS:=}" -# GOA libeufin amount ceiling (hacktivism). Stage auto-replaces via bank /config. -: "${LADDER_MAX_AMOUNT:=4503599627370496}" -: "${LADDER_STEPS:=23}" -: "${LADDER_LOAD:=1}" -: "${LADDER_PAY:=1}" -: "${LADDER_STACK_AUTO:=1}" -# Withdraw mids = pay mids × scale (so wallet can afford the pay ladder) -: "${LADDER_WITHDRAW_SCALE:=1.5}" -# Floor for random mids (must be ≥ smallest exchange coin; GOA min denom ≈ 0.000001) -: "${LADDER_MIN_AMOUNT:=0.000001}" -: "${LADDER_CONFIRM_POLLS:=40}" -: "${LADDER_PAY_SETTLE_ROUNDS:=6}" -: "${MERCHANT_INSTANCE:=goa-demo-cp4zqk}" -# Public variable template for stage pays (optional; fixed templates used when amount maps) -: "${LADDER_PAY_TEMPLATE:=}" -: "${LADDER_PAY_TEMPLATE_INSTANCE:=fermes-des-collines}" - -GOA_LADDER_CEILING="4503599627370496" - -# Resolve wallet-cli .mjs (no hardcoded laptop path) -if [ -z "${CLI_JS}" ] || [ ! -f "${CLI_JS}" ]; then - CLI_JS=$(find_wallet_cli 2>/dev/null || true) -fi -if [ -z "${CLI_JS}" ] || [ ! -f "${CLI_JS}" ]; then - # allow PATH wrapper as last resort (wcli falls back to taler-wallet-cli) - CLI_JS="" -fi - -CUR="${EXPECT_CURRENCY:-GOA}" -BANK="${BANK_PUBLIC%/}" -EX="${EXCHANGE_PUBLIC%/}/" -MER="${MERCHANT_PUBLIC%/}" -INST="${MERCHANT_INSTANCE}" - -# --- Stack-specific ladder maxima (TESTPAYSAN stage) --- -# Uses bank max_wire_transfer_amount and exchange min denom when still on GOA defaults. -apply_ladder_stack_defaults() { - [ "${LADDER_STACK_AUTO}" = "1" ] || return 0 - if [ "${CUR}" != "TESTPAYSAN" ]; then - case "${TALER_DOMAIN:-}" in - stage.lefrancpaysan.ch|stage.bank.lefrancpaysan.ch|stage.exchange.lefrancpaysan.ch|stage.monnaie.lefrancpaysan.ch) - CUR="TESTPAYSAN" - ;; - *) return 0 ;; - esac - fi - - local bank_cfg max_wire min_denom - bank_cfg=$(curl -sS -m 12 "${BANK}/config" 2>/dev/null || true) - max_wire=$(printf '%s' "$bank_cfg" | python3 -c ' -import json,sys -try: - d=json.load(sys.stdin) -except Exception: - print("") - raise SystemExit(0) -a=d.get("max_wire_transfer_amount") or "" -print(a.split(":",1)[-1] if a else "") -' 2>/dev/null || true) - - min_denom=$(curl -sS -m 25 -H 'Accept: application/json' "${EX%/}/keys" 2>/dev/null | python3 -c ' -import json,sys -try: - d=json.load(sys.stdin) -except Exception: - print("") - raise SystemExit(0) -den=d.get("denoms") or d.get("denominations") or [] -vals=[] -for x in den if isinstance(den,list) else []: - v=x.get("value") or x.get("amount") or "" - if not v: continue - try: vals.append(float(str(v).split(":")[-1])) - except Exception: pass -print(min(vals) if vals else "") -' 2>/dev/null || true) - - # Fallback maxima if public config unreachable - [ -n "$max_wire" ] || max_wire="2000" - [ -n "$min_denom" ] || min_denom="0.01" - - # Only rewrite when caller left GOA defaults (explicit LADDER_MAX_AMOUNT=… kept) - if [ -n "$max_wire" ] && { [ "${LADDER_MAX_AMOUNT}" = "${GOA_LADDER_CEILING}" ] || [ "${LADDER_MAX_AMOUNT}" = "4503599627370496" ]; }; then - LADDER_MAX_AMOUNT="$max_wire" - fi - if [ -n "$min_denom" ] && { [ "${LADDER_MIN_AMOUNT}" = "0.000001" ] || [ "${LADDER_MIN_AMOUNT}" = "0.00000001" ]; }; then - LADDER_MIN_AMOUNT="$min_denom" - fi - # Slightly fewer rungs on stage (full GOA 23 still ok if user set LADDER_STEPS) - if [ "${LADDER_STEPS}" = "23" ]; then - LADDER_STEPS=15 - fi - # Stage farmer shops: private goa-demo instance does not exist - if [ "${MERCHANT_INSTANCE}" = "goa-demo-cp4zqk" ]; then - MERCHANT_INSTANCE="${LADDER_PAY_TEMPLATE_INSTANCE:-fermes-des-collines}" - INST="$MERCHANT_INSTANCE" - fi - # Prefer public templates for pays when no merchant token (set later) - : "${E2E_USE_TEMPLATES:=1}" - export E2E_USE_TEMPLATES - export LADDER_MAX_AMOUNT LADDER_MIN_AMOUNT LADDER_STEPS MERCHANT_INSTANCE - info "ladder stack" "TESTPAYSAN · max=${CUR}:${LADDER_MAX_AMOUNT} min=${CUR}:${LADDER_MIN_AMOUNT} steps=${LADDER_STEPS} (from bank max_wire / keys denoms)" -} -apply_ladder_stack_defaults -SCRATCH=$(mktemp -d) -WDB="$SCRATCH/wallet.sqlite3" -REPORT_DIR="${LADDER_REPORT_DIR:-$SCRATCH}" -mkdir -p "$REPORT_DIR" -TSV="$REPORT_DIR/ladder-results.tsv" -PAY_TSV="$REPORT_DIR/ladder-pay-results.tsv" -JSON="$REPORT_DIR/ladder-report.json" -LOAD_BEFORE="$REPORT_DIR/load-before.json" -LOAD_AFTER="$REPORT_DIR/load-after.json" -echo -e "rung\trange\tamount\tstatus\tms_mint\tms_accept\tms_confirm\tms_settle\tms_total\twid\tnote" >"$TSV" -echo -e "rung\trange\tamount\tstatus\tms_order\tms_handle\tms_settle\tms_total\toid\tnote" >"$PAY_TSV" - -ladder_over() { - local now - now=$(date +%s) - [ $((now - SECTION_T0)) -ge "$LADDER_TIMEOUT_S" ] -} - -wcli() { - if [ -n "${CLI_JS:-}" ] && [ -f "$CLI_JS" ]; then - node "$CLI_JS" --wallet-db="$WDB" --no-throttle "$@" - else - taler-wallet-cli --wallet-db="$WDB" --no-throttle "$@" - fi -} - -# Explorer pool password: env → file override → stack secrets → SSH -resolve_explorer_pw() { - local pw="" f="" host="" - if [ -n "${EXP_PW:-}" ]; then - printf '%s' "$EXP_PW" - return 0 - fi - if [ -n "${EXP_PW_FILE:-}" ] && [ -f "$EXP_PW_FILE" ]; then - tr -d '\n\r' <"$EXP_PW_FILE" - return 0 - fi - - # Stage TESTPAYSAN — never use GOA koopa-admin-secrets explorer pw first - if [ "${CUR}" = "TESTPAYSAN" ] \ - || [[ "${TALER_DOMAIN:-}" == stage.*lefrancpaysan* ]] \ - || [[ "${TALER_DOMAIN:-}" == *stage.lefrancpaysan* ]]; then - for f in \ - "${FRANCPAYSAN_SECRETS:-}/stage/bank-explorer-password.txt" \ - "${HOME}/francpaysan-secrets/stage/bank-explorer-password.txt" \ - "${HOME}/src/francpaysan-secrets/stage/bank-explorer-password.txt" \ - "${HOME}/taler/src/francpaysan-secrets/stage/bank-explorer-password.txt" \ - "${HOME}/.config/taler-landing/stage-bank-explorer-password.txt" - do - [ -n "$f" ] && [ -f "$f" ] || continue - tr -d '\n\r' <"$f" - return 0 - done - for host in \ - "${INSIDE_SSH:-}" \ - francpaysan-stage-user \ - francpaysan-host - do - [ -n "$host" ] || continue - pw=$(ssh -o BatchMode=yes -o ConnectTimeout=12 "$host" \ - 'tr -d "\n\r" /dev/null \ - || sudo tr -d "\n\r" /dev/null' \ - 2>/dev/null || true) - if [ -n "$pw" ]; then - printf '%s' "$pw" - return 0 - fi - done - return 1 - fi - - # GOA / default: SECRETS_ROOT / ~/.config / koopa SSH - if pw=$(read_secret "taler-bank/bank-explorer-password.txt" 2>/dev/null) && [ -n "$pw" ]; then - printf '%s' "$pw" - return 0 - fi - for f in \ - "${SECRETS_ROOT:+${SECRETS_ROOT}/taler-bank/bank-explorer-password.txt}" \ - "${HOME}/src/koopa/koopa-admin-secrets/koopa/host-root/taler-bank/bank-explorer-password.txt" \ - "${HOME}/.config/taler-landing/bank-explorer-password.txt" - do - [ -n "$f" ] && [ -f "$f" ] || continue - tr -d '\n\r' <"$f" - return 0 - done - return 1 -} - -wallet_avail() { - wcli balance 2>/dev/null | python3 -c ' -import json,sys -t=sys.stdin.read() -i=t.find("{") -if i<0: - print("0"); raise SystemExit -d=json.loads(t[i:t.rfind("}")+1]) -cur=sys.argv[1] -for b in d.get("balances") or []: - a=b.get("available") or "" - if a.startswith(cur+":"): - print(a.split(":",1)[1]); raise SystemExit -print("0") -' "$CUR" 2>/dev/null || echo "0" -} - -# Build paired pay + withdraw ladders (same step count, shared random shape). -# Pay: [0] + log-uniform mids + [max-1] + [max] -# Wd: [0] + max(mid, mid×scale) mids + [max-1] + [max] (mids higher → enough to spend) -# Writes: $1 = withdraw list file, $2 = pay list file (space-separated CUR:amt lines as one line each) -build_ladder_pair() { - local wd_out="$1" pay_out="$2" - python3 - <<'PY' "$CUR" "${LADDER_MAX_AMOUNT}" "${LADDER_STEPS}" "${LADDER_MIN_AMOUNT}" "${LADDER_WITHDRAW_SCALE}" "$wd_out" "$pay_out" -import math, random, sys -from decimal import Decimal, ROUND_HALF_UP, ROUND_UP -from pathlib import Path - -cur = sys.argv[1] -max_amt = Decimal(sys.argv[2]) -steps = max(1, int(sys.argv[3])) -min_amt = Decimal(sys.argv[4]) -scale = Decimal(sys.argv[5]) -wd_path, pay_path = Path(sys.argv[6]), Path(sys.argv[7]) -if min_amt <= 0: - min_amt = Decimal("0.000001") -if min_amt >= max_amt: - min_amt = max_amt / Decimal(1000) -if scale < 1: - scale = Decimal(1) -max_m1 = max_amt - Decimal(1) if max_amt > 1 else max_amt - -def fmt(v: Decimal) -> str: - q = v.quantize(Decimal("0.00000001"), rounding=ROUND_HALF_UP) - if q == q.to_integral(): - return format(int(q), "d") - return format(q, "f").rstrip("0").rstrip(".") - -def quant(v: Decimal) -> Decimal: - if v >= 1: - return v.quantize(Decimal(1), rounding=ROUND_HALF_UP) - if v < min_amt: - return min_amt - return v.quantize(Decimal("0.00000001"), rounding=ROUND_UP) - -def amt(v: Decimal) -> str: - return "%s:%s" % (cur, fmt(v)) - -def make_mids(n_mid: int, hi_cap: Decimal): - if n_mid <= 0 or hi_cap <= min_amt: - return [] - lo, hi = float(min_amt), float(hi_cap) * 0.999999 - if hi <= lo: - hi = lo * 10 - cuts = sorted(math.exp(random.uniform(math.log(lo), math.log(hi))) for _ in range(n_mid)) - out, prev = [], Decimal(0) - for c in cuts: - v = quant(Decimal(str(c))) - if v <= prev: - step = min_amt if prev < 1 else max(prev * Decimal("1e-6"), Decimal(1)) - v = quant(prev + step) - if v >= hi_cap: - v = quant(hi_cap - (Decimal(1) if hi_cap > 1 else min_amt)) - if v <= prev or v >= hi_cap: - continue - out.append(v) - prev = v - return out - -# Build withdraw ladder first (full range), then pay mids = withdraw/scale -# so each pay mid is always cheaper than the matching withdraw mid. -if steps == 1: - wd_vals = [max_amt] -elif steps == 2: - wd_vals = [Decimal(0), max_amt] -else: - n_mid = max(0, steps - 3) - wd_vals = [Decimal(0)] + make_mids(n_mid, max_m1) + [max_m1, max_amt] - while len(wd_vals) > steps and len(wd_vals) > 3: - wd_vals.pop(len(wd_vals) // 2) - while len(wd_vals) < steps and len(wd_vals) >= 2: - i = max(1, len(wd_vals) - 2) - a, b = wd_vals[i - 1], wd_vals[i] - if a <= 0: - m = max(min_amt, b / 2 if b > 0 else min_amt) - else: - m = (a * b).sqrt() if a * b > 0 else (a + b) / 2 - m = quant(m) - if m <= a or m >= b: - break - wd_vals.insert(i, m) - wd_vals = wd_vals[:steps] - -pay_vals = [] -prev_p = Decimal(-1) -for i, w in enumerate(wd_vals): - is_first = i == 0 - is_last = i == len(wd_vals) - 1 - is_m1 = (not is_last) and w == max_m1 and i == len(wd_vals) - 2 - if is_first and w == 0: - pay_vals.append(Decimal(0)) - prev_p = Decimal(0) - continue - if is_last: - pay_vals.append(max_amt) - continue - if is_m1 or w == max_m1: - pay_vals.append(max_m1) - prev_p = max_m1 - continue - # pay mid = withdraw / scale (strictly less funding needed per step) - p = quant(w / scale) if scale > 0 else w - if p < min_amt and w >= min_amt: - p = min_amt - if p <= prev_p: - p = quant(prev_p + (min_amt if prev_p < 1 else Decimal(1))) - if p >= w: - # keep pay strictly below this withdraw rung when possible - p = quant(w - (Decimal(1) if w > 1 else min_amt)) if w > prev_p else prev_p - if p <= prev_p: - p = prev_p # flat ok only if stuck; still ≤ w - pay_vals.append(p) - prev_p = p - -assert len(pay_vals) == len(wd_vals) -# sanity: every non-pin pay mid ≤ matching withdraw mid -for i, (p, w) in enumerate(zip(pay_vals, wd_vals)): - if i == 0 or i >= len(wd_vals) - 2: - continue - if p > w: - pay_vals[i] = w - -wd_path.write_text(" ".join(amt(v) for v in wd_vals) + "\n") -pay_path.write_text(" ".join(amt(v) for v in pay_vals) + "\n") -print(" ".join(amt(v) for v in wd_vals)) -PY -} - -# shellcheck source=metrics.sh -source "$ROOT/metrics.sh" -METRICS_DIR="$REPORT_DIR" -ALT_UNITS_FILE="${REPORT_DIR}/alt_unit_names.json" -export METRICS_DIR CUR WDB CLI_JS ALT_UNITS_FILE -# Ladder can disable host load without killing coin metrics -if [ "${LADDER_LOAD:-1}" = "0" ]; then - METRICS_LOAD=0 - export METRICS_LOAD -fi - -section "ladder · ${CUR} withdraw (0 → random → max · ${LADDER_STEPS} steps)" -info "bank" "$BANK" -info "exchange" "$EX" -info "currency" "$CUR" -# alt_unit_names for human amounts (Kilo-GOA / Mega-GOA / … + GOA in parentheses) -if metrics_load_alt_units "${EX%/}/config"; then - info "alt_unit_names" "from ${EX%/}/config → $ALT_UNITS_FILE" -else - warn "alt_unit_names" "using built-in SI fallback ($ALT_UNITS_FILE)" -fi -info "budget" "${LADDER_TIMEOUT_S}s" -_max_m1=$(python3 -c 'import sys; from decimal import Decimal; m=Decimal(sys.argv[1]); print(m-1 if m>1 else m)' "${LADDER_MAX_AMOUNT}" 2>/dev/null || echo "${LADDER_MAX_AMOUNT}-1") -info "steps" "${LADDER_STEPS} (0 + random≥${LADDER_MIN_AMOUNT} + max-1=${CUR}:${_max_m1} + max=${CUR}:${LADDER_MAX_AMOUNT})" -info "withdraw_scale" "${LADDER_WITHDRAW_SCALE}× pay mids (fund pay ladder)" -info "pay_phase" "$([ "${LADDER_PAY}" = "1" ] && echo enabled || echo disabled)" -info "currency/domain" "${CUR} · ${TALER_DOMAIN:-?} · bank=${BANK}" - -set_group load -section "ladder · load snapshot (before withdraws)" -metrics_report_load "$LOAD_BEFORE" "ladder-start" || true -# Fresh wallet per rung — baseline empty (or last-rung DB if re-used later) -metrics_report_coins "ladder-start" || true - -if ! EXP_PW=$(resolve_explorer_pw); then - err bank "explorer password missing" \ - "set EXP_PW / EXP_PW_FILE or SECRETS_ROOT=…/koopa/host-root (taler-bank/bank-explorer-password.txt)${SECRETS_ROOT:+ · SECRETS_ROOT=${SECRETS_ROOT}}" - secrets_hint 2>/dev/null || true - exit 1 -fi -if [ -n "${SECRETS_ROOT:-}" ]; then - info "explorer secret" "resolved (SECRETS_ROOT=${SECRETS_ROOT} · stage uses stagepaysan secret first when CUR=TESTPAYSAN)" -else - info "explorer secret" "resolved via EXP_PW / EXP_PW_FILE / stage SSH / koopa" -fi -if [ -n "${CLI_JS:-}" ] && [ -f "${CLI_JS}" ]; then - info "wallet-cli" "$CLI_JS" -else - info "wallet-cli" "PATH taler-wallet-cli ($(command -v taler-wallet-cli 2>/dev/null || echo missing))" -fi - -# --- auto-account --- -t0=$(now_ms) -if ! curl -sS -m 30 -o "$SCRATCH/auto-account.json" "${BANK}/intro/auto-account.json"; then - err bank "auto-account.json unreachable" - exit 1 -fi -ms_auto=$(elapsed_ms "$t0") -if ! python3 -c 'import json; d=json.load(open("'"$SCRATCH"'/auto-account.json")); assert d.get("ok") or d.get("username")' 2>/dev/null; then - err bank "auto-account create failed" "$(head -c 120 "$SCRATCH/auto-account.json" | tr '\n' ' ')" - exit 1 -fi -ACCT_USER=$(python3 -c 'import json; print(json.load(open("'"$SCRATCH"'/auto-account.json"))["username"])') -ok "auto-account ${ACCT_USER} (${ms_auto}ms) — personal ${CUR}:0; pool=explorer" -info "auto-account password" "(see $SCRATCH/auto-account.json — not logged)" - -# --- explorer token --- -t0=$(now_ms) -TOK=$(curl -sS -m 20 -u "${EXP_USER}:${EXP_PW}" \ - -H 'Content-Type: application/json' \ - -d '{"scope":"readwrite","duration":{"d_us":3600000000}}' \ - "${BANK}/accounts/${EXP_USER}/token" \ - | python3 -c 'import json,sys; print(json.load(sys.stdin)["access_token"])') -ms_tok=$(elapsed_ms "$t0") -[ -n "$TOK" ] || { err bank "explorer token failed"; exit 1; } -ok "explorer token (${ms_tok}ms)" - -# ONE cumulative wallet for all withdraws + pays (need balance to spend). -# force-select uses *last* reserve_pub from the current accept output. -wallet_prepare() { - local label="${1:-wallet}" - WDB="$SCRATCH/wallet-${label}.sqlite3" - export WDB - if [ ! -f "$WDB" ]; then - wcli exchanges add "$EX" >"$SCRATCH/ex-add-$label.out" 2>&1 || true - wcli exchanges update "$EX" >"$SCRATCH/ex-upd-$label.out" 2>&1 || true - wcli exchanges accept-tos "$EX" >"$SCRATCH/ex-tos-$label.out" 2>&1 || true - fi -} - -t0=$(now_ms) -rm -f "$SCRATCH/wallet-main.sqlite3" -wallet_prepare "main" -ms_tos=$(elapsed_ms "$t0") -ok "wallet exchange + ToS (${ms_tos}ms) — cumulative DB for withdraw+pay (no run-until-done)" - -build_ladder_pair "$SCRATCH/ladder-wd-plan.txt" "$SCRATCH/ladder-pay-plan.txt" -LADDER_LIST=$(tr -d '\n' <"$SCRATCH/ladder-wd-plan.txt") -PAY_LIST=$(tr -d '\n' <"$SCRATCH/ladder-pay-plan.txt") -: "${LADDER_MAX_RUNGS:=99}" -# shellcheck disable=SC2086 -set -- $LADDER_LIST -if [ "$#" -gt "$LADDER_MAX_RUNGS" ]; then - # shellcheck disable=SC2046 - set -- $(printf '%s\n' "$@" | head -n "$LADDER_MAX_RUNGS") -fi -LADDER_N=$# -info "withdraw plan" "$*" -info "withdraw plan (alt)" "$(format_amount_list_alt "$@")" -info "pay plan" "$PAY_LIST" -# shellcheck disable=SC2086 -info "pay plan (alt)" "$(format_amount_list_alt $PAY_LIST)" -printf '%s\n' "$@" >"$SCRATCH/ladder-plan.txt" -printf '%s\n' $PAY_LIST >"$SCRATCH/ladder-pay-plan-lines.txt" 2>/dev/null || true - -OK_N=0 -FAIL_N_L=0 -PAY_OK_N=0 -PAY_FAIL_N=0 -STOP_REASON="" -STOP_AMOUNT="" -declare -a RUNG_JSON=() - -set_group withdraw -section "ladder · phase A · withdraw (${LADDER_N} rungs)" -rung=0 -for AMT in "$@"; do - rung=$((rung + 1)) - if ladder_over; then - STOP_REASON="timeout budget ${LADDER_TIMEOUT_S}s" - warn ladder "time budget exhausted" \ - "problem: LADDER_TIMEOUT_S=${LADDER_TIMEOUT_S}s reached after ${OK_N} ok rungs; remaining amounts not tried" - break - fi - - section "ladder · rung $rung $AMT · $(format_amount_alt "$AMT")" - tag=$(printf '%s' "$AMT" | tr '.:' '__') - # TSV "range" column: fixed pins at ends, random mids (refined after AMT_NUM) - if [ "$rung" -eq 1 ]; then - range_note="pin:0" - elif [ "$rung" -eq "$LADDER_N" ]; then - range_note="pin:max" - elif [ "$rung" -eq $((LADDER_N - 1)) ] && [ "$LADDER_N" -ge 3 ]; then - range_note="pin:max-1" - else - range_note="random" - fi - t_rung=$(now_ms) - ms_mint=0 ms_accept=0 ms_confirm=0 ms_settle=0 - note="" - status="FAIL" - WID="-" - FORCE_SEL_409_LOGGED=0 - - # keep cumulative main wallet - wallet_prepare "main" - - # mint from explorer pool - t0=$(now_ms) - code=$(curl -sS -m 30 -o "$SCRATCH/wd-$tag.json" -w '%{http_code}' \ - -H "Authorization: Bearer ${TOK}" \ - -H 'Content-Type: application/json' \ - -d "{\"amount\":\"${AMT}\"}" \ - "${BANK}/accounts/${EXP_USER}/withdrawals") - ms_mint=$(elapsed_ms "$t0") - WID=$(python3 -c 'import json;d=json.load(open("'"$SCRATCH"'/wd-'"$tag"'.json"));print(d.get("withdrawal_id") or "")' 2>/dev/null || true) - URI=$(python3 -c 'import json;u=json.load(open("'"$SCRATCH"'/wd-'"$tag"'.json")).get("taler_withdraw_uri") or "";print(u.replace(":443/","/"))' 2>/dev/null || true) - # numeric amount (for zero / settle / ceiling special-cases) - AMT_NUM=$(python3 -c 'import sys; print(sys.argv[1].split(":",1)[-1])' "$AMT") - IS_ZERO=0 - python3 -c 'import sys; from decimal import Decimal; sys.exit(0 if Decimal(sys.argv[1])==0 else 1)' "$AMT_NUM" 2>/dev/null && IS_ZERO=1 - IS_MAX_PIN=0 - IS_MAX_M1_PIN=0 - if [ "$AMT_NUM" = "${LADDER_MAX_AMOUNT}" ]; then - IS_MAX_PIN=1 - range_note="pin:max" - elif [ "$AMT_NUM" = "$((LADDER_MAX_AMOUNT - 1))" ] 2>/dev/null || \ - [ "$AMT_NUM" = "$(python3 -c 'print(int("'"$LADDER_MAX_AMOUNT"'")-1)')" ]; then - IS_MAX_M1_PIN=1 - range_note="pin:max-1" - fi - - if [ "$code" != "200" ] && [ "$code" != "201" ] || [ -z "$WID" ] || [ -z "$URI" ]; then - # strip quotes so STOP_REASON never breaks later shell/python argv - note="mint HTTP $code $(head -c 100 "$SCRATCH/wd-$tag.json" 2>/dev/null | tr '\n\"' ' ')" - ms_total=$(elapsed_ms "$t_rung") - if [ "$IS_ZERO" = "1" ]; then - # Probe only: bank may reject GOA:0 — record and continue ladder - status="ZERO_REJECT" - note="zero-withdraw rejected (expected possible): $note" - warn bank "mint $AMT rejected" \ - "problem: bank will not create a GOA:0 withdrawal (zero amount probe). Ladder continues. detail: $note" - echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" - OK_N=$((OK_N + 1)) - continue - fi - # Absolute max is a ceiling probe — bank often returns SQL P0001/5110; do not abort. - if [ "$IS_MAX_PIN" = "1" ]; then - status="CEILING_REJECT" - note="absolute max rejected (ceiling probe; max-1 is the hard pin): $note" - warn bank "mint $AMT rejected (ceiling)" \ - "problem: bank rejects absolute LADDER_MAX_AMOUNT (often SQL P0001/5110). max-1 rung is the last expected success. Ladder continues to report. detail: $note" - echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" - continue - fi - err bank "mint $AMT failed" "$note" - status="FAIL_MINT" - STOP_REASON="$note" - STOP_AMOUNT="$AMT" - echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" - FAIL_N_L=$((FAIL_N_L + 1)) - break - fi - ok "mint $AMT ($WID) ${ms_mint}ms" - - before=$(wallet_avail) - - # accept - t0=$(now_ms) - if wcli withdraw accept-uri --exchange "$EX" "$URI" >"$SCRATCH/accept-$tag.out" 2>&1; then - ms_accept=$(elapsed_ms "$t0") - ok "accept-uri $AMT ${ms_accept}ms" - else - ms_accept=$(elapsed_ms "$t0") - note="accept-uri failed: $(tail -c 200 "$SCRATCH/accept-$tag.out" | tr '\n' ' ')" - ms_total=$(elapsed_ms "$t_rung") - # Wallet 7006: no denominations for this amount (0, sub-denom dust, etc.) — warn & continue - if [ "$IS_ZERO" = "1" ] || grep -qE 'code: 7006|"code"[[:space:]]*:[[:space:]]*7006|No denominations could be selected' \ - "$SCRATCH/accept-$tag.out" 2>/dev/null; then - if [ "$IS_ZERO" = "1" ]; then - status="ZERO_SKIP" - note="zero-withdraw skip (7006 / no denoms): $note" - warn wallet "accept $AMT skipped" \ - "problem: wallet code 7006 — no coin denominations for GOA:0 (zero amount cannot be withdrawn as coins). Ladder continues. detail: $note" - else - status="SKIP_DENOM" - note="skip amount (wallet 7006 no denoms): $note" - warn wallet "accept $AMT skipped" \ - "problem: wallet code 7006 — no denominations match this amount (below smallest coin or not combinable). Ladder continues with next rung. detail: $note" - fi - echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" - continue - fi - err wallet "accept $AMT" "$note" - status="FAIL_ACCEPT" - STOP_REASON="$note" - STOP_AMOUNT="$AMT" - echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" - FAIL_N_L=$((FAIL_N_L + 1)) - break - fi - - # Confirm ASAP when bank status is selected. No run-until-done (hangs on macOS/wallet). - # Server-side auto-confirm only watches landing withdraw-watch.ids — ladder must confirm itself. - bank_st() { - curl -sS -m 8 "${BANK}/taler-integration/withdrawal-operation/${WID}" 2>/dev/null \ - | python3 -c 'import json,sys; print((json.load(sys.stdin).get("status") or "").strip())' 2>/dev/null || true - } - do_confirm() { - curl -sS -m 15 -o "$SCRATCH/conf-$tag.json" -w '%{http_code}' -X POST \ - -H "Authorization: Bearer ${TOK}" -H 'Content-Type: application/json' -d '{}' \ - "${BANK}/accounts/${EXP_USER}/withdrawals/${WID}/confirm" - } - # Collect reserve_pub candidates for *this* withdrawal (WID + amount). - # Cumulative wallets re-print old reserves in accept/tx dumps — never trust a single "last" blindly. - # Prints unique pubs one per line, preferred order first. - extract_rpubs_for_wid() { - python3 - "$WID" "$AMT" "$SCRATCH/accept-$tag.out" "$SCRATCH/tx-$tag.json" "$SCRATCH/used-rpubs.txt" <<'PY' -import json, re, sys -from pathlib import Path - -wid = sys.argv[1] -amt = sys.argv[2] -paths = sys.argv[3:5] -used_path = sys.argv[5] -used = set() -if Path(used_path).is_file(): - used = {ln.strip() for ln in open(used_path) if ln.strip()} - -def walk_collect(o, bag, ctx=None): - ctx = dict(ctx or {}) - if isinstance(o, dict): - for k, v in o.items(): - kl = str(k).lower() - if kl in ("withdrawal_id", "withdraw_id", "wopid", "id") and isinstance(v, str): - ctx["id"] = v - if kl in ("amount", "rawamount", "instructedamount") and isinstance(v, str): - ctx["amount"] = v - if kl in ("taler_withdraw_uri", "talerwithdrawuri", "uri") and isinstance(v, str): - ctx["uri"] = v - if kl in ("reserve_pub", "reservepub") and isinstance(v, str) and len(v) >= 40: - bag.append((v, dict(ctx))) - walk_collect(v, bag, ctx) - elif isinstance(o, list): - for i in o: - walk_collect(i, bag, ctx) - -raw_pubs = [] # ordered as found -scored = [] # (score, pub) higher better -blob_all = "" -for p in paths: - try: - blob_all += open(p, errors="replace").read() + "\n" - except Exception: - pass - -# 1) full JSON objects in files -for p in paths: - try: - t = open(p, errors="replace").read() - except Exception: - continue - # whole-file JSON - for m in re.finditer(r"\{", t): - try: - o = json.loads(t[m.start():]) - except Exception: - continue - bag = [] - walk_collect(o, bag) - for pub, ctx in bag: - raw_pubs.append(pub) - score = 0 - cid = str(ctx.get("id") or "") - camt = str(ctx.get("amount") or "") - curi = str(ctx.get("uri") or "") - if wid and wid in cid: - score += 100 - if wid and wid in curi: - score += 80 - if amt and (camt == amt or camt.endswith(amt.split(":", 1)[-1])): - score += 40 - if pub in used: - score -= 200 - scored.append((score, pub)) - -# 2) regex fallback on accept file only (more current) -try: - acc = open(paths[0], errors="replace").read() -except Exception: - acc = "" -for pat in ( - r"\"reserve_pub\"\s*:\s*\"([A-Z0-9]{40,})\"", - r"\"reservePub\"\s*:\s*\"([A-Z0-9]{40,})\"", - r"reserve_pub[\"\s:=]+([A-Z0-9]{40,})", -): - for m in re.finditer(pat, acc, re.I): - pub = m.group(1) - raw_pubs.append(pub) - score = 10 - # proximity to WID in accept output - window = acc[max(0, m.start() - 400) : m.end() + 400] - if wid and wid in window: - score += 100 - if amt and amt in window: - score += 30 - if pub in used: - score -= 200 - scored.append((score, pub)) - -# prefer high score, then later occurrence -order = [] -seen = set() -for score, pub in sorted(scored, key=lambda x: (-x[0],), reverse=False): - # sort by score desc: use reverse sorted - pass -for score, pub in sorted(scored, key=lambda x: x[0], reverse=True): - if pub in seen or pub in used: - continue - seen.add(pub) - order.append(pub) -# append unused raw in reverse (newest-ish) -for pub in reversed(raw_pubs): - if pub in seen or pub in used: - continue - seen.add(pub) - order.append(pub) -for pub in order: - print(pub) -PY - } - - mark_rpub_used() { - local p="$1" - [ -n "$p" ] || return 0 - mkdir -p "$SCRATCH" 2>/dev/null || true - grep -qxF "$p" "$SCRATCH/used-rpubs.txt" 2>/dev/null || echo "$p" >>"$SCRATCH/used-rpubs.txt" - } - - force_select_if_needed() { - local st_now="$1" - [ "$st_now" = "pending" ] || [ -z "$st_now" ] || return 0 - local rpub epayto code_fs any=0 - # refresh tx dump each try (wallet may attach reserve late) - wcli transactions >"$SCRATCH/tx-$tag.json" 2>&1 || true - epayto=$(curl -sS -m 10 "${EX%/}/keys" 2>/dev/null | python3 -c ' -import json,sys -d=json.load(sys.stdin) -acc=d.get("accounts") or [] -for a in acc: - p=a.get("payto_uri") or a.get("payto_address") or "" - if "x-taler-bank" in p or "exchange" in p: - print(p); break -else: - if acc: print(acc[0].get("payto_uri") or "") -' 2>/dev/null || true) - if [ -z "$epayto" ]; then - warn bank "force-select skipped" "problem: exchange payto empty from /keys" - return 0 - fi - # Try candidates until bank leaves pending (200/204) or we exhaust - while IFS= read -r rpub; do - [ -n "$rpub" ] || continue - any=1 - code_fs=$(curl -sS -m 12 -o "$SCRATCH/force-sel-$tag.json" -w '%{http_code}' -X POST \ - -H 'Content-Type: application/json' \ - -d "{\"reserve_pub\":\"${rpub}\",\"selected_exchange\":\"${epayto}\"}" \ - "${BANK}/taler-integration/withdrawal-operation/${WID}" 2>/dev/null || echo "000") - if [ "$code_fs" = "200" ] || [ "$code_fs" = "204" ]; then - info "force-select" "HTTP ${code_fs} rpub=${rpub:0:12}… (ok for WID ${WID:0:8})" - mark_rpub_used "$rpub" - return 0 - fi - if [ "$code_fs" = "409" ]; then - # 5114 = this reserve already bound to another op — not "out of money" - info "force-select" "HTTP 409 rpub=${rpub:0:12}… (stale/used reserve — not balance; trying next)" - mark_rpub_used "$rpub" - continue - fi - info "force-select" "HTTP ${code_fs} rpub=${rpub:0:12}… body=$(tr '\n' ' ' <"$SCRATCH/force-sel-$tag.json" 2>/dev/null | head -c 120)" - # other errors: still try next candidate - done < <(extract_rpubs_for_wid) - if [ "$any" != "1" ]; then - warn bank "force-select skipped" \ - "problem: no reserve_pub for this withdraw (WID=${WID:0:8}…); wallet may not have selected yet" - fi - } - - # No run-until-done (hangs / banned). Read transactions only for reserve_pub candidates. - wcli transactions >"$SCRATCH/tx-$tag.json" 2>&1 || true - - t0=$(now_ms) - conf_ok=0 - st="" - # Poll bank status; force-select while pending; confirm as soon as selected - for i in $(seq 1 "${LADDER_CONFIRM_POLLS}"); do - ladder_over && break - st=$(bank_st) - case "$st" in - selected) - ccode=$(do_confirm) - if [ "$ccode" = "204" ] || [ "$ccode" = "200" ]; then - conf_ok=1 - info "confirm" "HTTP $ccode after selected (poll $i)" - else - note="confirm HTTP $ccode" - fi - break - ;; - confirmed) - conf_ok=1 - break - ;; - aborted) - note="withdrawal aborted by bank" - break - ;; - esac - # force-select while pending — try alternate rpubs on 5114 (not out-of-money) - if [ "$st" = "pending" ] || [ -z "$st" ]; then - if [ "$i" -eq 1 ] || [ "$i" -eq 2 ] || [ $((i % 3)) -eq 0 ]; then - force_select_if_needed "$st" - fi - fi - sleep 0.35 - done - ms_confirm=$(elapsed_ms "$t0") - st=$(printf '%s' "${st:-}" | tr -d '\r\n' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//') - if [ "$conf_ok" != "1" ]; then - note="${note:-confirm timeout last=${st:-empty}}" - # Soft: not confirmed — usually stale reserve_pub (5114), NOT empty pool balance - status="SKIP_CONFIRM" - warn bank "confirm $AMT skipped" \ - "problem: bank status='${st:-empty}' after ${LADDER_CONFIRM_POLLS} polls (want selected/confirmed). Usually reserve_pub mismatch (5114), not out-of-money — mint/accept already OK. detail: $note" - ms_total=$(elapsed_ms "$t_rung") - echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" - continue - fi - ok "confirm $AMT ${ms_confirm}ms (client, on selected)" - - # settle: poll wallet balance + bank transfer_done only — never run-until-done - t0=$(now_ms) - settled=0 - xfer="?" - if [ "$IS_ZERO" = "1" ]; then - settled=1 - note="zero-amount: no coin delta expected" - else - for r in $(seq 1 "$LADDER_SETTLE_ROUNDS"); do - ladder_over && break - after=$(wallet_avail) - if python3 -c " -from decimal import Decimal -import sys -sys.exit(0 if Decimal(sys.argv[1]) > Decimal(sys.argv[2]) else 1) -" "$after" "$before" 2>/dev/null; then - settled=1 - break - fi - xfer=$(curl -sS -m 10 "${BANK}/taler-integration/withdrawal-operation/${WID}" \ - | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d.get("transfer_done"), d.get("status"))' 2>/dev/null || echo "?") - # bank done is enough to leave settle without hanging on wallet - if echo "$xfer" | grep -qi True; then - note="bank transfer_done (no run-until-done) avail=${after} $xfer" - break - fi - sleep "$LADDER_SETTLE_SLEEP" - done - fi - ms_settle=$(elapsed_ms "$t0") - after=$(wallet_avail) - ms_total=$(elapsed_ms "$t_rung") - if [ -z "${xfer:-}" ] || [ "$xfer" = "?" ]; then - xfer=$(curl -sS -m 10 "${BANK}/taler-integration/withdrawal-operation/${WID}" \ - | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d.get("transfer_done"), d.get("status"))' 2>/dev/null || echo "?") - fi - - if [ "$settled" = "1" ]; then - status="OK" - note="${note:-avail=${CUR}:${after}}" - ok "settle $AMT → ${CUR}:${after} (settle ${ms_settle}ms, rung ${ms_total}ms)" - OK_N=$((OK_N + 1)) - echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" - metrics_report_coins "ladder-r${rung}-after-${tag}" || true - metrics_record_flow withdrawn "$AMT" || true - elif echo "$xfer" | grep -qi True; then - status="OK_BANK" - note="bank transfer_done avail=${after} $xfer (no run-until-done)" - ok "settle $AMT bank transfer_done (wallet avail=${CUR}:${after}, ${ms_settle}ms)" - OK_N=$((OK_N + 1)) - echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" - metrics_report_coins "ladder-r${rung}-after-${tag}" || true - metrics_record_flow withdrawn "$AMT" || true - else - note="no coins / no transfer_done avail=${after} $xfer" - err wallet "settle $AMT" "$note" - status="FAIL_SETTLE" - STOP_REASON="$note" - STOP_AMOUNT="$AMT" - FAIL_N_L=$((FAIL_N_L + 1)) - echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" - metrics_report_coins "ladder-r${rung}-fail-${tag}" || true - break - fi -done - -# --------------------------------------------------------------------------- -# Phase B — payment ladder (same shape: 0 → low → … → max-1 → max) -# --------------------------------------------------------------------------- -PAY_OK_N=0 -PAY_FAIL_N=0 -if [ "${LADDER_PAY}" = "1" ] && [ -n "${PAY_LIST:-}" ] && [ "$FAIL_N_L" -eq 0 ]; then - set_group pay - section "ladder · phase B · pay" - metrics_report_coins "before-pay-ladder" || true - # Merchant secret (same as e2e) — stage often has only public shop templates - MPW="${E2E_MERCHANT_TOKEN:-${MERCHANT_TOKEN:-}}" - if [ -z "$MPW" ]; then - MPW=$(read_secret "taler-merchant/merchant-${INST}-password.txt" 2>/dev/null || true) - fi - if [ -z "$MPW" ]; then - MPW=$(read_secret "taler-merchant/merchant-goa-demo-cp4zqk-password.txt" 2>/dev/null || true) - fi - if [ -z "$MPW" ] && [ "${CUR}" = "TESTPAYSAN" ]; then - for f in \ - "${FRANCPAYSAN_SECRETS:-}/stage/default-instance-token.txt" \ - "${HOME}/francpaysan-secrets/stage/default-instance-token.txt" \ - "${HOME}/src/francpaysan-secrets/stage/default-instance-token.txt" - do - [ -f "$f" ] || continue - MPW=$(tr -d '\n\r' <"$f") - break - done - if [ -z "$MPW" ]; then - for host in "${INSIDE_SSH:-}" francpaysan-stage-user francpaysan-host; do - [ -n "$host" ] || continue - MPW=$(ssh -o BatchMode=yes -o ConnectTimeout=12 "$host" \ - 'tr -d "\n\r" /dev/null || true' \ - 2>/dev/null || true) - [ -n "$MPW" ] && break - done - fi - fi - if [ -z "$MPW" ]; then - if [ "${CUR}" = "TESTPAYSAN" ]; then - # Stage: withdraw ladder is the main probe; pays need instance token or - # public templates for *exact* face values (not continuous ladder amounts). - info pay "no merchant token — skip pay ladder on TESTPAYSAN (withdraw maxima still covered)" - info pay "hint: set E2E_MERCHANT_TOKEN or use ./taler-monitoring.sh -d stage.lefrancpaysan.ch e2e for shop templates" - else - warn pay "no merchant token — skip pay ladder (set E2E_MERCHANT_TOKEN or secrets)" - fi - else - ok "merchant token" "instance ${INST}" - case "$MPW" in - secret-token:*) AUTH="Authorization: Bearer ${MPW}" ;; - *) AUTH="Authorization: Bearer secret-token:${MPW}" ;; - esac - wallet_prepare "main" - # shellcheck disable=SC2086 - set -- $PAY_LIST - PAY_N=$# - info "pay rungs" "$PAY_N · $*" - prung=0 - for PAMT in "$@"; do - prung=$((prung + 1)) - if ladder_over; then - warn pay "time budget exhausted" "after ${PAY_OK_N} ok pays" - break - fi - section "ladder · pay rung $prung $PAMT · $(format_amount_alt "$PAMT")" - ptag=$(printf '%s' "$PAMT" | tr '.:' '__') - if [ "$prung" -eq 1 ]; then - prange="pin:0" - elif [ "$prung" -eq "$PAY_N" ]; then - prange="pin:max" - elif [ "$prung" -eq $((PAY_N - 1)) ] && [ "$PAY_N" -ge 3 ]; then - prange="pin:max-1" - else - prange="random" - fi - PNUM=$(python3 -c 'import sys; print(sys.argv[1].split(":",1)[-1])' "$PAMT") - IS_PZERO=0 - python3 -c 'import sys; from decimal import Decimal; sys.exit(0 if Decimal(sys.argv[1])==0 else 1)' "$PNUM" 2>/dev/null && IS_PZERO=1 - IS_PMAX=0 - [ "$PNUM" = "${LADDER_MAX_AMOUNT}" ] && IS_PMAX=1 - t_pay=$(now_ms) - ms_order=0 ms_handle=0 ms_psettle=0 - pnote="" - pstatus="FAIL" - OID="-" - - metrics_report_coins "before-pay-${ptag}" || true - bal=$(wallet_avail) - - if [ "$IS_PZERO" = "1" ]; then - pstatus="ZERO_SKIP" - pnote="zero pay probe skipped" - warn pay "pay $PAMT skipped" "problem: zero amount order not useful. Ladder continues." - ms_total=$(elapsed_ms "$t_pay") - echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t${ms_handle}\t${ms_psettle}\t${ms_total}\t${OID}\t${pnote}" >>"$PAY_TSV" - PAY_OK_N=$((PAY_OK_N + 1)) - continue - fi - - # Soft: absolute max pay is a ceiling probe (unlikely affordable / merchant may reject) - if [ "$IS_PMAX" = "1" ]; then - # try once; on fail CEILING_REJECT - : - fi - - # Insufficient balance → soft skip for max pin only; hard fail for mid/max-1 - if ! python3 -c 'import sys; from decimal import Decimal; sys.exit(0 if Decimal(sys.argv[1])>=Decimal(sys.argv[2]) else 1)' "$bal" "$PNUM" 2>/dev/null; then - pnote="insufficient balance avail=${CUR}:${bal} need=${PAMT}" - ms_total=$(elapsed_ms "$t_pay") - if [ "$IS_PMAX" = "1" ]; then - pstatus="CEILING_SKIP" - warn pay "pay $PAMT skipped (ceiling)" "$pnote" - echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t0\t0\t0\t${ms_total}\t-\t${pnote}" >>"$PAY_TSV" - continue - fi - err pay "pay $PAMT" "$pnote" - pstatus="FAIL_BALANCE" - PAY_FAIL_N=$((PAY_FAIL_N + 1)) - FAIL_N_L=$((FAIL_N_L + 1)) - STOP_REASON="$pnote" - STOP_AMOUNT="$PAMT" - echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t0\t0\t0\t${ms_total}\t-\t${pnote}" >>"$PAY_TSV" - break - fi - - t0=$(now_ms) - SUM_JSON=$(python3 -c 'import json,sys; print(json.dumps(sys.argv[1]))' "ladder pay ${PAMT}" 2>/dev/null || echo '"ladder pay"') - curl -skS -m 20 -o "$SCRATCH/ord-$ptag.json" -X POST \ - -H "$AUTH" -H 'Content-Type: application/json' \ - -d "{\"order\":{\"summary\":${SUM_JSON},\"amount\":\"${PAMT}\",\"fulfillment_message\":\"ok\"},\"create_token\":true}" \ - "${MER}/instances/${INST}/private/orders" 2>"$SCRATCH/ord-$ptag.err" || true - ms_order=$(elapsed_ms "$t0") - OID=$(python3 -c 'import json,re,sys;t=open(sys.argv[1]).read() -try: print(json.loads(t).get("order_id") or "") -except Exception: - m=re.search(r"\"order_id\"\s*:\s*\"([^\"]+)\"",t); print(m.group(1) if m else "") -' "$SCRATCH/ord-$ptag.json" 2>/dev/null || true) - OTOK=$(python3 -c 'import json,re,sys;t=open(sys.argv[1]).read() -try: print(json.loads(t).get("token") or "") -except Exception: - m=re.search(r"\"token\"\s*:\s*\"([^\"]+)\"",t); print(m.group(1) if m else "") -' "$SCRATCH/ord-$ptag.json" 2>/dev/null || true) - if [ -z "$OID" ]; then - pnote="order create failed $(head -c 80 "$SCRATCH/ord-$ptag.json" 2>/dev/null | tr '\n\"' ' ')" - ms_total=$(elapsed_ms "$t_pay") - if [ "$IS_PMAX" = "1" ]; then - pstatus="CEILING_REJECT" - warn pay "pay $PAMT rejected (ceiling)" "$pnote" - echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t0\t0\t${ms_total}\t-\t${pnote}" >>"$PAY_TSV" - continue - fi - err pay "order $PAMT" "$pnote" - pstatus="FAIL_ORDER" - PAY_FAIL_N=$((PAY_FAIL_N + 1)) - FAIL_N_L=$((FAIL_N_L + 1)) - STOP_REASON="$pnote" - STOP_AMOUNT="$PAMT" - echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t0\t0\t${ms_total}\t-\t${pnote}" >>"$PAY_TSV" - break - fi - ok "order $OID ($PAMT) ${ms_order}ms" - - curl -skS -m 12 -o "$SCRATCH/ord-det-$ptag.json" -H "$AUTH" \ - "${MER}/instances/${INST}/private/orders/${OID}" 2>/dev/null || true - PAYURI=$(python3 -c 'import json,sys -try: print(json.load(open(sys.argv[1])).get("taler_pay_uri") or "") -except Exception: print("") -' "$SCRATCH/ord-det-$ptag.json" 2>/dev/null || true) - if [ -z "$PAYURI" ] && [ -n "$OTOK" ]; then - MH=$(python3 -c 'from urllib.parse import urlparse; print(urlparse("'"$MER"'").hostname or "taler.hacktivism.ch")' 2>/dev/null || echo "taler.hacktivism.ch") - PAYURI="taler://pay/${MH}/instances/${INST}/${OID}/?c=${OTOK}" - fi - PAYURI=$(printf '%s' "$PAYURI" | sed 's/:443\//\//g; s/:443?/?/g') - if [ -z "$PAYURI" ]; then - pnote="no pay URI for $OID" - ms_total=$(elapsed_ms "$t_pay") - err pay "uri $PAMT" "$pnote" - pstatus="FAIL_URI" - PAY_FAIL_N=$((PAY_FAIL_N + 1)) - FAIL_N_L=$((FAIL_N_L + 1)) - STOP_REASON="$pnote" - STOP_AMOUNT="$PAMT" - echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t0\t0\t${ms_total}\t${OID}\t${pnote}" >>"$PAY_TSV" - break - fi - - t0=$(now_ms) - if ! wcli handle-uri --yes "$PAYURI" >"$SCRATCH/pay-$ptag.out" 2>&1; then - ms_handle=$(elapsed_ms "$t0") - pnote="handle-uri failed $(tail -c 100 "$SCRATCH/pay-$ptag.out" | tr '\n\"' ' ')" - ms_total=$(elapsed_ms "$t_pay") - if [ "$IS_PMAX" = "1" ]; then - pstatus="CEILING_REJECT" - warn pay "pay $PAMT handle failed (ceiling)" "$pnote" - echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t${ms_handle}\t0\t${ms_total}\t${OID}\t${pnote}" >>"$PAY_TSV" - continue - fi - err pay "handle $PAMT" "$pnote" - pstatus="FAIL_HANDLE" - PAY_FAIL_N=$((PAY_FAIL_N + 1)) - FAIL_N_L=$((FAIL_N_L + 1)) - STOP_REASON="$pnote" - STOP_AMOUNT="$PAMT" - echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t${ms_handle}\t0\t${ms_total}\t${OID}\t${pnote}" >>"$PAY_TSV" - break - fi - ms_handle=$(elapsed_ms "$t0") - ok "handle-uri $PAMT ${ms_handle}ms" - - # Short settle polls: merchant order + wallet tx only — never run-until-done - t0=$(now_ms) - settled=0 - r=0 - while [ "$r" -lt "${LADDER_PAY_SETTLE_ROUNDS}" ]; do - r=$((r + 1)) - wcli transactions >"$SCRATCH/tx-$ptag.out" 2>&1 || true - curl -skS -m 8 -o "$SCRATCH/ord-paid-$ptag.json" -H "$AUTH" \ - "${MER}/instances/${INST}/private/orders/${OID}" 2>/dev/null || true - if grep -qiE 'payment|paid|Payment' "$SCRATCH/tx-$ptag.out" 2>/dev/null \ - || python3 -c 'import json,sys -d=json.load(open(sys.argv[1])) -sys.exit(0 if d.get("paid") is True or str(d.get("order_status","")).lower()=="paid" else 1) -' "$SCRATCH/ord-paid-$ptag.json" 2>/dev/null; then - settled=1 - break - fi - sleep 0.5 - done - ms_psettle=$(elapsed_ms "$t0") - ms_total=$(elapsed_ms "$t_pay") - after=$(wallet_avail) - if [ "$settled" = "1" ]; then - pstatus="OK" - pnote="avail=${CUR}:${after}" - ok "pay settled $PAMT → bal ${CUR}:${after} (total ${ms_total}ms)" - PAY_OK_N=$((PAY_OK_N + 1)) - echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t${ms_handle}\t${ms_psettle}\t${ms_total}\t${OID}\t${pnote}" >>"$PAY_TSV" - metrics_report_coins "after-pay-${ptag}" || true - metrics_record_flow spent "$PAMT" || true - else - pnote="not settled order=$OID avail=${CUR}:${after}" - if [ "$IS_PMAX" = "1" ]; then - pstatus="CEILING_REJECT" - warn pay "pay $PAMT not settled (ceiling)" "$pnote" - echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t${ms_handle}\t${ms_psettle}\t${ms_total}\t${OID}\t${pnote}" >>"$PAY_TSV" - continue - fi - err pay "settle $PAMT" "$pnote" - pstatus="FAIL_SETTLE" - PAY_FAIL_N=$((PAY_FAIL_N + 1)) - FAIL_N_L=$((FAIL_N_L + 1)) - STOP_REASON="$pnote" - STOP_AMOUNT="$PAMT" - echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t${ms_handle}\t${ms_psettle}\t${ms_total}\t${OID}\t${pnote}" >>"$PAY_TSV" - metrics_report_coins "after-pay-fail-${ptag}" || true - break - fi - done - metrics_report_coins "after-pay-ladder" || true - info "pay summary" "ok=$PAY_OK_N fail=$PAY_FAIL_N tsv=$PAY_TSV" - fi -elif [ "${LADDER_PAY}" = "1" ] && [ "$FAIL_N_L" -gt 0 ]; then - warn pay "skipped pay ladder" "withdraw phase already failed" -fi - -ms_phase=$(python3 -c 'import sys,time; print(int((time.time()-float(sys.argv[1]))*1000))' "$SECTION_T0") - -# --- report --- -set_group report -section "ladder · report" -info "auto-account" "$ACCT_USER" -info "ok_rungs" "$OK_N" -info "fail_rungs" "$FAIL_N_L" -info "pay_ok" "$PAY_OK_N" -info "pay_fail" "$PAY_FAIL_N" -info "phase_ms" "$ms_phase" -info "tsv" "$TSV" -info "pay_tsv" "$PAY_TSV" - -# speed summary via python — free-text stop reason via env (JSON " in bank errors -# used to break shell argv quoting → "syntax error near unexpected token '('") -export LADDER_REPORT_STOP_AMOUNT="${STOP_AMOUNT:-}" -export LADDER_REPORT_STOP_REASON="${STOP_REASON:-}" -python3 - "$TSV" "$PAY_TSV" "$JSON" "$OK_N" "$FAIL_N_L" "$PAY_OK_N" "$PAY_FAIL_N" "$ms_phase" "$ACCT_USER" "$CUR" <<'PY' -import csv, json, os, sys, statistics -tsv, pay_tsv, jpath = sys.argv[1:4] -ok_n, fail_n, pay_ok, pay_fail, phase_ms, acct, cur = sys.argv[4:11] -stop_amt = os.environ.get("LADDER_REPORT_STOP_AMOUNT") or None -stop_reason = os.environ.get("LADDER_REPORT_STOP_REASON") or None - -def load_rows(path): - rows = [] - try: - with open(path, newline="") as f: - for row in csv.DictReader(f, delimiter="\t"): - rows.append(row) - except Exception: - pass - return rows - -rows = load_rows(tsv) -prows = load_rows(pay_tsv) - -def nums(rs, key): - out = [] - for row in rs: - try: - out.append(int(row[key])) - except Exception: - pass - return out - -def stats(xs): - if not xs: - return {"n": 0} - return { - "n": len(xs), - "min_ms": min(xs), - "max_ms": max(xs), - "avg_ms": int(sum(xs) / len(xs)), - "p50_ms": int(statistics.median(xs)), - } - -report = { - "currency": cur, - "auto_account": acct, - "ok_rungs": int(ok_n), - "fail_rungs": int(fail_n), - "pay_ok": int(pay_ok), - "pay_fail": int(pay_fail), - "stopped_at_amount": stop_amt or None, - "stop_reason": stop_reason or None, - "phase_ms": int(phase_ms), - "timing": { - "mint": stats(nums(rows, "ms_mint")), - "accept": stats(nums(rows, "ms_accept")), - "confirm": stats(nums(rows, "ms_confirm")), - "settle": stats(nums(rows, "ms_settle")), - "rung_total": stats(nums(rows, "ms_total")), - "pay_order": stats(nums(prows, "ms_order")), - "pay_handle": stats(nums(prows, "ms_handle")), - "pay_settle": stats(nums(prows, "ms_settle")), - "pay_total": stats(nums(prows, "ms_total")), - }, - "rungs": rows, - "pays": prows, -} -json.dump(report, open(jpath, "w"), indent=2) -print("JSON", jpath) -print("--- withdraw speed (ms) ---") -for k in ("mint", "accept", "confirm", "settle", "rung_total"): - v = report["timing"][k] - if v.get("n"): - print(" %s n=%s min=%s p50=%s avg=%s max=%s" % (k.ljust(12), v["n"], v["min_ms"], v["p50_ms"], v["avg_ms"], v["max_ms"])) -print("--- pay speed (ms) ---") -for k in ("pay_order", "pay_handle", "pay_settle", "pay_total"): - v = report["timing"][k] - if v.get("n"): - print(" %s n=%s min=%s p50=%s avg=%s max=%s" % (k.ljust(12), v["n"], v["min_ms"], v["p50_ms"], v["avg_ms"], v["max_ms"])) -print("--- withdraw rungs ---") -for row in rows: - print(" %2s %-16s %-12s total=%sms" % (row.get("rung"), row.get("amount"), row.get("status"), row.get("ms_total"))) -print("--- pay rungs ---") -for row in prows: - print(" %2s %-16s %-12s total=%sms oid=%s" % (row.get("rung"), row.get("amount"), row.get("status"), row.get("ms_total"), row.get("oid"))) -if stop_amt: - print("STOPPED at %s: %s" % (stop_amt, stop_reason)) -else: - print("Completed without hard failure (or budget stop without fail).") -PY - -wcli balance 2>&1 | tee "$REPORT_DIR/balance-final.out" | tail -20 || true - -set_group load -section "ladder · load snapshot (after withdraws)" -metrics_report_load "$LOAD_AFTER" "ladder-end" || true -if [ -f "$LOAD_BEFORE" ] && [ -f "$LOAD_AFTER" ]; then - section "metrics · ladder load delta" - metrics_print_load_delta "$LOAD_BEFORE" "$LOAD_AFTER" || true -fi -# Speed timings → metrics overall (min/p50/avg/max per phase) -if [ -f "$JSON" ]; then - python3 - "$JSON" "${METRICS_DIR}/perf-summary.json" <<'PY' 2>/dev/null || true -import json, sys -rep = json.load(open(sys.argv[1])) -out = {} -for k, v in (rep.get("timing") or {}).items(): - if isinstance(v, dict) and v.get("n"): - out[k] = v -json.dump(out, open(sys.argv[2], "w"), indent=2) -PY -fi -export METRICS_WITHDRAW_TSV="$TSV" -export METRICS_PAY_TSV="$PAY_TSV" -metrics_report_coins "ladder-end" || true -metrics_print_overall "ladder final" || true - -# Keep scratch if LADDER_REPORT_DIR set; else copy key files to /tmp -if [ -z "${LADDER_REPORT_DIR:-}" ]; then - KEEP="/tmp/goa-ladder-report-$(date +%Y%m%d-%H%M%S)" - mkdir -p "$KEEP" - cp -a "$TSV" "$PAY_TSV" "$JSON" "$SCRATCH/auto-account.json" "$SCRATCH/ladder-plan.txt" \ - "$SCRATCH/ladder-wd-plan.txt" "$SCRATCH/ladder-pay-plan.txt" \ - "$REPORT_DIR/balance-final.out" "$LOAD_BEFORE" "$LOAD_AFTER" "$KEEP/" 2>/dev/null || true - info "report_dir" "$KEEP" - echo "$KEEP" >"$SCRATCH/KEEP_PATH" -fi - -if [ "$FAIL_N_L" -gt 0 ]; then - blocker "ladder" "stopped at ${STOP_AMOUNT:-?} — ${STOP_REASON:-error}" - exit 1 -fi -if [ "$OK_N" -eq 0 ]; then - blocker "ladder" "no successful withdraw rungs" - exit 1 -fi -ok "ladder finished withdraw_ok=$OK_N pay_ok=$PAY_OK_N phase=${ms_phase}ms" -exit 0 diff --git a/check_inside.sh b/check_inside.sh index 7f62880..38f145f 100755 --- a/check_inside.sh +++ b/check_inside.sh @@ -29,12 +29,14 @@ if [ -z "$PROFILE" ]; then fi fi -# Resolve host-podman vs ssh before first check ID +# Resolve host-podman vs ssh before first check ID. +# stage-lfp on the FP host itself uses local podman (INSIDE_PODMAN=1 / +# INSIDE_MODE=local-podman); laptop → stage still uses INSIDE_SSH. _use_local_podman=0 -if [ "$PROFILE" != "stage-lfp" ]; then - if [ "${INSIDE_PODMAN:-0}" = "1" ] || [ "${INSIDE_MODE:-}" = "local-podman" ]; then - _use_local_podman=1 - elif command -v podman >/dev/null 2>&1 \ +if [ "${INSIDE_PODMAN:-0}" = "1" ] || [ "${INSIDE_MODE:-}" = "local-podman" ]; then + _use_local_podman=1 +elif [ "$PROFILE" != "stage-lfp" ]; then + if command -v podman >/dev/null 2>&1 \ && podman ps --format '{{.Names}}' 2>/dev/null | grep -qE 'taler-hacktivism'; then _use_local_podman=1 fi @@ -52,10 +54,10 @@ section "inside · collect (${PROFILE} · access=${INSIDE_ACCESS})" info "flags" "INSIDE_ACCESS=${INSIDE_ACCESS} INSIDE_PODMAN=${INSIDE_PODMAN:-0} INSIDE_MODE=${INSIDE_MODE:-} LOCAL_STACK=${LOCAL_STACK:-0} SKIP_SSH=${SKIP_SSH:-0} KOOPA_SSH=${KOOPA_SSH:-} INSIDE_SSH=${INSIDE_SSH:-}" # --------------------------------------------------------------------------- -# stage-lfp: low-priv stagepaysan on francpaysan-host +# stage-lfp: stagepaysan podman (host-local or SSH INSIDE_SSH from laptop) # --------------------------------------------------------------------------- if [ "$PROFILE" = "stage-lfp" ]; then - SSH_HOST="${INSIDE_SSH:-francpaysan-stage-user}" + SSH_HOST="${INSIDE_SSH:-}" BANK_CTR="${INSIDE_BANK_CTR:-stage-lfp-bank}" EX_CTR="${INSIDE_EXCHANGE_CTR:-stage-lfp-exchange-ansible}" MER_CTR="${INSIDE_MERCHANT_CTR:-stage-lfp-merchant}" @@ -69,15 +71,23 @@ if [ "$PROFILE" = "stage-lfp" ]; then STAGE_SSH_T="${INSIDE_SSH_TIMEOUT:-${SSH_CMD_TIMEOUT:-24}}" if [ "${STAGE_SSH_T}" -lt 24 ] 2>/dev/null; then STAGE_SSH_T=24; fi - if ! mon_ssh_ok "$SSH_HOST"; then - err "ssh" "cannot reach ${SSH_HOST} (stagepaysan low-priv) — set INSIDE_SSH= or SKIP" + if [ "$_use_local_podman" = "1" ]; then + if ! command -v podman >/dev/null 2>&1; then + err "host" "INSIDE_PODMAN/host-podman but podman missing" + summary + exit 1 + fi + ok "host→container" "podman exec on this host (INSIDE_ACCESS=host-podman · no SSH)" + elif ! mon_ssh_ok "$SSH_HOST"; then + err "ssh" "cannot reach ${SSH_HOST:-?} (stagepaysan low-priv) — set INSIDE_SSH= or INSIDE_PODMAN=1" summary exit 1 + else + ok "ssh ${SSH_HOST}" "stagepaysan (podman, no sudo)" fi - ok "ssh ${SSH_HOST}" "stagepaysan (podman, no sudo)" - # Inject names/ports into remote (ssh bash -s does not inherit local env). - RAW=$( + # Inject names/ports (ssh bash -s does not inherit local env; local bash does). + _stage_lfp_script() { { printf 'BANK_CTR=%q; EX_CTR=%q; MER_CTR=%q\n' "$BANK_CTR" "$EX_CTR" "$MER_CTR" printf 'BANK_PORT=%q; EX_PORT=%q; MER_PORT=%q\n' "$BANK_PORT" "$EX_PORT" "$MER_PORT" @@ -173,11 +183,20 @@ fi echo DONE REMOTE - } | mon_ssh_bash "$SSH_HOST" "${STAGE_SSH_T}" || true - ) + } + } + if [ "$_use_local_podman" = "1" ]; then + RAW=$(_stage_lfp_script | bash || true) + else + RAW=$(_stage_lfp_script | mon_ssh_bash "$SSH_HOST" "${STAGE_SSH_T}" || true) + fi if [ -z "$RAW" ] || ! echo "$RAW" | grep -q '^E|'; then - err "ssh" "stage remote timed out or empty (cap ${STAGE_SSH_T}s · host=${SSH_HOST})" + if [ "$_use_local_podman" = "1" ]; then + err "host" "stage collect timed out or empty (local podman)" + else + err "ssh" "stage remote timed out or empty (cap ${STAGE_SSH_T}s · host=${SSH_HOST})" + fi summary exit 1 fi @@ -189,7 +208,7 @@ REMOTE IFS='|' read -r _ comp level key detail <<<"$line" case "$comp" in bank|exchange|merchant|caddy) _g="$comp" ;; - *) _g="ssh" ;; + *) _g="$INSIDE_ACCESS" ;; esac if [ "$_g" != "$_last_inside_grp" ]; then set_group "$_g" @@ -242,7 +261,29 @@ REMOTE # Host load as stagepaysan (no container RSS from koopa metrics) set_group load section "inside · load (stagepaysan host)" - LOAD_LINE=$(mon_ssh_bash "$SSH_HOST" 8 <<'EOF' || true + _load_py() { + python3 - <<'PY' +import os +la=os.getloadavg() +print("loadavg=%.2f,%.2f,%.2f" % la) +try: + with open("/proc/meminfo") as f: + d={} + for line in f: + k,v=line.split(":")[0], line.split(":")[1].strip().split()[0] + d[k]=int(v) + total=d.get("MemTotal",0)/1024/1024 + avail=d.get("MemAvailable",0)/1024/1024 + used=total-avail + print("mem_used=%.2fGiB avail=%.2fGiB total=%.2fGiB" % (used, avail, total)) +except Exception: + print("mem=?") +PY + } + if [ "$_use_local_podman" = "1" ]; then + LOAD_LINE=$(_load_py || true) + else + LOAD_LINE=$(mon_ssh_bash "$SSH_HOST" 8 <<'EOF' || true python3 - <<'PY' import os la=os.getloadavg() @@ -262,6 +303,7 @@ except Exception: PY EOF ) + fi if [ -n "$LOAD_LINE" ]; then info "stage host" "$(echo "$LOAD_LINE" | tr '\n' ' ')" else @@ -270,7 +312,8 @@ EOF set_group disk section "inside · disk free space (stage host + containers)" - _disk_raw=$(mon_ssh_bash "$SSH_HOST" "${STAGE_SSH_T:-24}" <<'DISK' || true + _disk_script() { + cat <<'DISK' set +e echo "###HOST###" df -Pk / /var /home /tmp /mnt/data 2>/dev/null || df -Pk @@ -280,9 +323,16 @@ for c in $(podman ps --format '{{.Names}}' 2>/dev/null); do podman exec "$c" df -Pk / /var /tmp 2>/dev/null || podman exec "$c" df -Pk 2>/dev/null done DISK -) + } + if [ "$_use_local_podman" = "1" ]; then + _disk_raw=$(_disk_script | bash || true) + _disk_label="host" + else + _disk_raw=$(_disk_script | mon_ssh_bash "$SSH_HOST" "${STAGE_SSH_T:-24}" || true) + _disk_label="ssh:${SSH_HOST}" + fi _host_df=$(printf '%s\n' "$_disk_raw" | sed -n '/^###HOST###$/,/^###CTRS###$/p' | sed '1d;$d') - mon_disk_check_remote_text "ssh:${SSH_HOST}" "$_host_df" || true + mon_disk_check_remote_text "$_disk_label" "$_host_df" || true _ctr=""; _buf="" while IFS= read -r _line || [ -n "$_line" ]; do case "$_line" in diff --git a/check_mail.sh b/check_mail.sh new file mode 100755 index 0000000..7f92433 --- /dev/null +++ b/check_mail.sh @@ -0,0 +1,292 @@ +#!/usr/bin/env bash +# check_mail.sh — outside-in mail (MX / SMTP / IMAP / SPF / DMARC) +# +# Catalog: mail-catalog.conf (MAIL_CATALOG=… to override) +# Covers firefly (taler.net, gnunet.org) and Anastasis/TSA +# (mail.anastasis.lu → pixel.taler-systems.com; anastasis.lu). +# +# Outside-only. Phase: mail +# +set -euo pipefail +ROOT=$(cd "$(dirname "$0")" && pwd) +# shellcheck source=lib.sh +source "$ROOT/lib.sh" + +set_area mail +section "mail · MX / SMTP / IMAP / SPF / DMARC (outside-in)" + +CATALOG="${MAIL_CATALOG:-$ROOT/mail-catalog.conf}" +PORT_TIMEOUT="${MAIL_PORT_TIMEOUT:-4}" +SMTP_TIMEOUT="${MAIL_SMTP_TIMEOUT:-10}" + +if [ ! -f "$CATALOG" ]; then + fail "catalog" "missing $CATALOG" + exit 1 +fi + +# dig or host fallback +dns_mx() { + local d="$1" + if command -v dig >/dev/null 2>&1; then + dig +short +time=3 +tries=2 MX "$d" 2>/dev/null | awk '{print tolower($2)}' | sed 's/\.$//' + else + host -t MX "$d" 2>/dev/null | awk -F'in mail is |has address ' '/mail is/{print tolower($NF)}' | sed 's/\.$//' + fi +} + +dns_a() { + local h="$1" + if command -v dig >/dev/null 2>&1; then + dig +short +time=3 +tries=2 A "$h" 2>/dev/null | grep -E '^[0-9.]+$' || true + dig +short +time=3 +tries=2 AAAA "$h" 2>/dev/null | grep -E ':' || true + else + getent ahosts "$h" 2>/dev/null | awk '{print $1}' | sort -u + fi +} + +dns_txt() { + local name="$1" + if command -v dig >/dev/null 2>&1; then + dig +short +time=3 +tries=2 TXT "$name" 2>/dev/null | tr -d '"' + else + host -t TXT "$name" 2>/dev/null | sed 's/.*"\(.*\)"/\1/' + fi +} + +tcp_open() { + local h="$1" p="$2" + if command -v timeout >/dev/null 2>&1; then + timeout "$PORT_TIMEOUT" bash -c "echo >/dev/tcp/${h}/${p}" 2>/dev/null + else + bash -c "echo >/dev/tcp/${h}/${p}" 2>/dev/null + fi +} + +# SMTP: read banner first, then EHLO (avoids "protocol synchronization") +smtp_probe() { + local host="$1" port="$2" + MAIL_HOST="$host" MAIL_PORT="$port" MAIL_TO="${SMTP_TIMEOUT}" python3 - <<'PY' 2>/dev/null +import os, socket, sys +host = os.environ["MAIL_HOST"] +port = int(os.environ["MAIL_PORT"]) +to = float(os.environ.get("MAIL_TO", "10")) +try: + s = socket.create_connection((host, port), to) + s.settimeout(to) + banner = s.recv(1024).decode("utf-8", "replace").strip().split("\n")[0] + if not banner.startswith("220"): + print(f"bad_banner={banner[:80]}") + sys.exit(1) + s.sendall(b"EHLO taler-monitoring.invalid\r\n") + data = b"" + while True: + chunk = s.recv(4096) + if not chunk: + break + data += chunk + if b"\n" in chunk and (data.endswith(b"\r\n") or len(data) > 8000): + # multi-line 250-… ends with 250 space + lines = data.decode("utf-8", "replace").splitlines() + if any(l.startswith("250 ") for l in lines): + break + if any(l.startswith("5") for l in lines[:3]): + break + text = data.decode("utf-8", "replace") + s.sendall(b"QUIT\r\n") + try: + s.recv(256) + except Exception: + pass + s.close() + starttls = "starttls" if "STARTTLS" in text.upper() else "no_starttls" + print(f"banner={banner[:60]} · {starttls}") + sys.exit(0) +except Exception as e: + print(f"err={e}") + sys.exit(1) +PY +} + +# IMAP unencrypted greeting on 143, or just TCP on 993 (TLS) +imap_probe() { + local host="$1" port="$2" + if [ "$port" = "993" ] || [ "$port" = "995" ]; then + if tcp_open "$host" "$port"; then + echo "tcp_open tls_port=$port" + return 0 + fi + return 1 + fi + MAIL_HOST="$host" MAIL_PORT="$port" MAIL_TO="${SMTP_TIMEOUT}" python3 - <<'PY' 2>/dev/null +import os, socket, sys +host = os.environ["MAIL_HOST"] +port = int(os.environ["MAIL_PORT"]) +to = float(os.environ.get("MAIL_TO", "10")) +try: + s = socket.create_connection((host, port), to) + s.settimeout(to) + banner = s.recv(1024).decode("utf-8", "replace").strip().split("\n")[0] + s.sendall(b"a001 LOGOUT\r\n") + try: + s.recv(256) + except Exception: + pass + s.close() + if banner.upper().startswith("* OK") or "IMAP" in banner.upper(): + print(f"banner={banner[:70]}") + sys.exit(0) + print(f"unexpected={banner[:70]}") + sys.exit(1) +except Exception as e: + print(f"err={e}") + sys.exit(1) +PY +} + +info "catalog" "$CATALOG" + +# Track probed hosts to avoid duplicate SMTP checks +declare -A HOST_DONE=() + +while IFS= read -r line || [ -n "$line" ]; do + line=${line%%#*} + line=$(echo "$line" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//') + [ -z "$line" ] && continue + # shellcheck disable=SC2086 + set -- $line + domain=$1 + expect_mx=${2:-} + mail_hosts=${3:-} + smtp_ports=${4:-25,465,587} + imap_ports=${5:-143,993} + label=${6:-$domain} + + set_group "$label" + info "domain" "$domain · expected_mx=$expect_mx · hosts=$mail_hosts" + + # --- MX --- + mapfile -t mx_hosts < <(dns_mx "$domain" | sed '/^$/d') + if [ "${#mx_hosts[@]}" -eq 0 ]; then + fail "mx" "$domain has no MX records" + else + ok "mx" "$domain MX → ${mx_hosts[*]}" + # expected MX match (suffix / exact) + if [ -n "$expect_mx" ]; then + matched=0 + IFS=',' read -ra want_list <<<"$expect_mx" + for w in "${want_list[@]}"; do + w=$(echo "$w" | tr '[:upper:]' '[:lower:]' | sed 's/\.$//') + for m in "${mx_hosts[@]}"; do + m=$(echo "$m" | tr '[:upper:]' '[:lower:]') + case "$m" in + "$w"|"$w".*|*."$w") matched=1; break ;; + esac + [ "$m" = "$w" ] && matched=1 + done + [ "$matched" = "1" ] && break + done + if [ "$matched" = "1" ]; then + ok "mx expected" "$domain MX matches catalog ($expect_mx)" + else + fail "mx expected" "$domain MX ${mx_hosts[*]} · want one of: $expect_mx" + fi + fi + fi + + # --- SPF / DMARC --- + spf=$(dns_txt "$domain" | tr '\n' ' ') + if echo "$spf" | grep -qi 'v=spf1'; then + ok "spf" "$domain has SPF" + else + warn "spf" "$domain no SPF TXT (v=spf1)" + fi + dmarc=$(dns_txt "_dmarc.$domain" | tr '\n' ' ') + if echo "$dmarc" | grep -qi 'v=dmarc1'; then + ok "dmarc" "$domain has DMARC" + else + warn "dmarc" "$domain no DMARC at _dmarc.$domain" + fi + + # --- mail hosts: DNS + ports + banners --- + IFS=',' read -ra hosts <<<"$mail_hosts" + for h in "${hosts[@]}"; do + h=$(echo "$h" | tr '[:upper:]' '[:lower:]' | sed 's/\.$//;s/^[[:space:]]*//;s/[[:space:]]*$//') + [ -z "$h" ] && continue + + addrs=$(dns_a "$h" | tr '\n' ' ') + if [ -z "${addrs// }" ]; then + fail "dns" "$h does not resolve (A/AAAA)" + continue + fi + ok "dns" "$h → $addrs" + + # skip full protocol re-probe if already done + if [ "${HOST_DONE[$h]:-}" = "1" ]; then + info "host" "$h already probed this run" + continue + fi + HOST_DONE[$h]=1 + + IFS=',' read -ra sports <<<"$smtp_ports" + for p in "${sports[@]}"; do + p=${p// /} + [ -z "$p" ] && continue + if ! tcp_open "$h" "$p"; then + # 587 optional on pixel + case "$p" in + 587) warn "smtp port" "$h:$p closed/filtered (submission)" ;; + *) fail "smtp port" "$h:$p closed/filtered" ;; + esac + continue + fi + case "$p" in + 465) + ok "smtp port" "$h:$p open (SMTPS)" + ;; + 25|587) + if detail=$(smtp_probe "$h" "$p"); then + ok "smtp" "$h:$p $detail" + else + # port open but banner failed — still ERROR for 25 + if [ "$p" = "25" ]; then + fail "smtp" "$h:$p open but SMTP handshake failed · $detail" + else + warn "smtp" "$h:$p open but handshake failed · $detail" + fi + fi + ;; + *) + ok "smtp port" "$h:$p open" + ;; + esac + done + + IFS=',' read -ra iports <<<"$imap_ports" + for p in "${iports[@]}"; do + p=${p// /} + [ -z "$p" ] && continue + if ! tcp_open "$h" "$p"; then + fail "imap port" "$h:$p closed/filtered" + continue + fi + case "$p" in + 993|995) + ok "imap port" "$h:$p open (TLS)" + ;; + 143|110) + if detail=$(imap_probe "$h" "$p"); then + ok "imap" "$h:$p $detail" + else + warn "imap" "$h:$p open but greeting failed · $detail" + fi + ;; + *) + ok "imap port" "$h:$p open" + ;; + esac + done + done +done <"$CATALOG" + +info "hint" "firefly = taler.net/gnunet.org MX; pixel = taler-systems.com (mail.anastasis.lu / mail.taler-systems.com)" +exit 0 diff --git a/check_mattermost.sh b/check_mattermost.sh new file mode 100755 index 0000000..a61bed7 --- /dev/null +++ b/check_mattermost.sh @@ -0,0 +1,255 @@ +#!/usr/bin/env bash +# check_mattermost.sh — outside-in checks for Taler Mattermost (chat) +# +# Default host: mattermost.taler.net +# Override: MATTERMOST_PUBLIC=https://mattermost.example.org +# MATTERMOST_HOST=mattermost.example.org +# +# Client compatibility (feature) — server min for current *official* clients: +# Android / iOS store apps → default ≥ 10.11.0 (Play/App Store + mobile docs 2026) +# Desktop (Win/Mac/Linux) → default ≥ 10.11.0 (current Desktop / ESR pairing) +# Vendor support floor → default ≥ 10.11.0 (ESR; older = unsupported) +# Web SPA → served by this server (always loads) but WARN if +# server is below support floor (security/EOL). +# +# Env: +# MATTERMOST_CLIENT_CHECK=1|0 master switch (default 1) +# MATTERMOST_MOBILE_CHECK=1|0 alias / legacy (if set 0 and CLIENT unset → off) +# MATTERMOST_MOBILE_MIN_SERVER default 10.11.0 +# MATTERMOST_DESKTOP_MIN_SERVER default 10.11.0 +# MATTERMOST_SUPPORT_MIN_SERVER default 10.11.0 (vendor-supported floor) +# +# Outside-only (no SSH). Phase name: mattermost +# +set -euo pipefail +ROOT=$(cd "$(dirname "$0")" && pwd) +# shellcheck source=lib.sh +source "$ROOT/lib.sh" + +set_area mattermost +section "mattermost · public chat (outside-in)" + +# Defaults: current official clients + vendor ESR floor (2026-07 docs). +: "${MATTERMOST_MOBILE_MIN_SERVER:=10.11.0}" +: "${MATTERMOST_DESKTOP_MIN_SERVER:=10.11.0}" +: "${MATTERMOST_SUPPORT_MIN_SERVER:=10.11.0}" +# Master switch: CLIENT_CHECK, else legacy MOBILE_CHECK, else on +if [ -n "${MATTERMOST_CLIENT_CHECK:-}" ]; then + : +elif [ -n "${MATTERMOST_MOBILE_CHECK:-}" ]; then + MATTERMOST_CLIENT_CHECK="${MATTERMOST_MOBILE_CHECK}" +else + MATTERMOST_CLIENT_CHECK=1 +fi +: "${MATTERMOST_MOBILE_CHECK:=${MATTERMOST_CLIENT_CHECK}}" + +if [ -n "${MATTERMOST_PUBLIC:-}" ]; then + BASE="${MATTERMOST_PUBLIC%/}" +elif [ -n "${MATTERMOST_HOST:-}" ]; then + BASE="https://${MATTERMOST_HOST}" +else + BASE="https://mattermost.taler.net" +fi +HOST=${BASE#https://} +HOST=${HOST#http://} +HOST=${HOST%%/*} + +info "target" "$BASE" +info "client policy" "mobile≥${MATTERMOST_MOBILE_MIN_SERVER} · desktop≥${MATTERMOST_DESKTOP_MIN_SERVER} · support≥${MATTERMOST_SUPPORT_MIN_SERVER} (CLIENT_CHECK=${MATTERMOST_CLIENT_CHECK})" + +tmp=$(mktemp -d) +trap 'rm -rf "$tmp"' EXIT + +# GET with follow redirects (Mattermost SPA often 302 → /) +mm_get() { + local url="$1" out="$2" + curl -skS -L --max-redirs 5 -m "${TIMEOUT:-12}" -o "$out" -w '%{http_code}' "$url" 2>/dev/null || echo 000 +} + +# --- DNS / HTTPS landing --- +set_group landing +code=$(mm_get "$BASE/" "$tmp/root.html") +if [ "$code" = "200" ] && head -c 512 "$tmp/root.html" | grep -qiE '/dev/null || echo 000) + +MM_SERVER_VER="" +if [ -f "$hdr_file" ]; then + # X-Version-Id: 9.5.3.8427860509.c8cabb… or 10.11.0.… + MM_SERVER_VER=$(grep -i '^x-version-id:' "$hdr_file" | head -1 \ + | sed 's/^[Xx]-[Vv]ersion-[Ii]d:[[:space:]]*//' | tr -d '\r' \ + | sed 's/^\([0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*\).*/\1/') +fi + +if [ "$code" != "200" ]; then + fail "system/ping" "$BASE/api/v4/system/ping → HTTP $code (want 200 JSON)" +else + if python3 - "$tmp/ping.json" <<'PY' 2>/dev/null +import json, sys +with open(sys.argv[1]) as f: + d = json.load(f) +sys.exit(0 if isinstance(d, dict) else 2) +PY + then + ok "system/ping" "$BASE/api/v4/system/ping → HTTP 200 JSON${MM_SERVER_VER:+ · server ${MM_SERVER_VER}}" + else + fail "system/ping" "$BASE/api/v4/system/ping → HTTP 200 but not valid JSON object" + fi +fi + +# --- Server version vs official client families (mobile / desktop / support / web) --- +set_group clients +mm_ver_ge() { + # $1=server $2=need → 0 if server >= need + python3 - "$1" "$2" <<'PY' +import sys + +def parse(v: str): + parts = [] + for p in v.strip().split(".")[:3]: + try: + parts.append(int(p)) + except ValueError: + parts.append(0) + while len(parts) < 3: + parts.append(0) + return tuple(parts) + +sys.exit(0 if parse(sys.argv[1]) >= parse(sys.argv[2]) else 1) +PY +} + +# One policy line: id|min_env_default|severity(error|warn)|human label +# severity: error → fail; warn → warn +_mm_client_policies() { + printf '%s\n' \ + "mobile|${MATTERMOST_MOBILE_MIN_SERVER}|error|Android/iOS store apps (Mattermost Mobile)" \ + "desktop|${MATTERMOST_DESKTOP_MIN_SERVER}|error|Desktop app (Windows/macOS/Linux official)" \ + "support|${MATTERMOST_SUPPORT_MIN_SERVER}|error|vendor-supported server floor (ESR/current)" \ + "web|${MATTERMOST_SUPPORT_MIN_SERVER}|warn|browser webapp (loads with server, but EOL server is insecure/unsupported)" +} + +if [ "${MATTERMOST_CLIENT_CHECK}" = "0" ]; then + info "client compat" "skipped (MATTERMOST_CLIENT_CHECK=0 / MATTERMOST_MOBILE_CHECK=0)" +elif [ -z "$MM_SERVER_VER" ]; then + warn "server version" "could not parse X-Version-Id from /api/v4/system/ping — cannot judge client compatibility" +else + info "server version" "X-Version-Id → ${MM_SERVER_VER}" + while IFS='|' read -r cid cmin csever clabel; do + [ -n "$cid" ] || continue + if mm_ver_ge "$MM_SERVER_VER" "$cmin"; then + ok "${cid} compat" "server ${MM_SERVER_VER} ≥ ${cmin} · ${clabel} OK" + else + detail="server ${MM_SERVER_VER} < ${cmin} · ${clabel}" + hint="upgrade Mattermost server · set MATTERMOST_${cid^^}_MIN_SERVER= or MATTERMOST_CLIENT_CHECK=0" + # Fix hint for support/web keys + case "$cid" in + mobile) hint="upgrade server · MATTERMOST_MOBILE_MIN_SERVER= / MATTERMOST_CLIENT_CHECK=0" ;; + desktop) hint="upgrade server · MATTERMOST_DESKTOP_MIN_SERVER= / MATTERMOST_CLIENT_CHECK=0" ;; + support) hint="server below vendor support floor · MATTERMOST_SUPPORT_MIN_SERVER= / MATTERMOST_CLIENT_CHECK=0" ;; + web) hint="web SPA is served by this old server (works) but platform is unsupported · upgrade recommended" ;; + esac + if [ "$csever" = "error" ]; then + fail "${cid} compat" "$detail" "$hint" + else + warn "${cid} compat" "$detail · $hint" + fi + fi + done < <(_mm_client_policies) +fi + +# Client config (optional): BuildDate / advertised Android/iOS min if server sets it +set_group client-config +code=$(mm_get "$BASE/api/v4/config/client?format=old" "$tmp/client.json") +if [ "$code" = "200" ] && [ -s "$tmp/client.json" ]; then + build=$(python3 - "$tmp/client.json" <<'PY' 2>/dev/null +import json, sys +with open(sys.argv[1]) as f: + d = json.load(f) +print(d.get("BuildDate") or d.get("Version") or "") +PY +) + and_min=$(python3 - "$tmp/client.json" <<'PY' 2>/dev/null +import json, sys +with open(sys.argv[1]) as f: + d = json.load(f) +print((d.get("AndroidMinVersion") or "").strip()) +PY +) + ios_min=$(python3 - "$tmp/client.json" <<'PY' 2>/dev/null +import json, sys +with open(sys.argv[1]) as f: + d = json.load(f) +print((d.get("IosMinVersion") or d.get("IOSMinVersion") or "").strip()) +PY +) + if [ -n "$build" ]; then + info "client config" "BuildDate/Version=${build}${and_min:+ · AndroidMinVersion=${and_min}}${ios_min:+ · IosMinVersion=${ios_min}}" + else + info "client config" "HTTP 200 (no BuildDate field)" + fi + if [ -n "$and_min" ] && [ "$and_min" != "0.0.0" ]; then + info "android min (server)" "server advertises AndroidMinVersion=${and_min}" + fi + if [ -n "$ios_min" ] && [ "$ios_min" != "0.0.0" ]; then + info "ios min (server)" "server advertises IosMinVersion=${ios_min}" + fi +else + info "client config" "HTTP ${code:-?} — optional" +fi + +# --- Login SPA (must be served for users) --- +set_group login +code=$(mm_get "$BASE/login" "$tmp/login.html") +if [ "$code" = "200" ] && head -c 512 "$tmp/login.html" | grep -qiE '/dev/null 2>&1; then + end=$(echo | openssl s_client -servername "$HOST" -connect "${HOST}:443" 2>/dev/null \ + | openssl x509 -noout -enddate 2>/dev/null | sed 's/notAfter=//') + if [ -n "$end" ]; then + # days remaining + end_epoch=$(date -d "$end" +%s 2>/dev/null || date -j -f "%b %d %T %Y %Z" "$end" +%s 2>/dev/null || echo 0) + now_epoch=$(date +%s) + if [ "$end_epoch" -gt 0 ]; then + days=$(( (end_epoch - now_epoch) / 86400 )) + if [ "$days" -lt 0 ]; then + fail "tls cert" "$HOST cert expired ($end)" + elif [ "$days" -lt 14 ]; then + warn "tls cert" "$HOST expires in ${days}d ($end)" + elif [ "$days" -lt 30 ]; then + warn "tls cert" "$HOST expires in ${days}d ($end)" + else + ok "tls cert" "$HOST valid · ${days}d left · $end" + fi + else + info "tls cert" "$HOST notAfter=$end" + fi + else + warn "tls cert" "$HOST could not read certificate" + fi +else + info "tls cert" "openssl not available — skip expiry check" +fi + +info "hint" "surface catalog also lists $HOST; client floors: mobile/desktop/support ≥${MATTERMOST_MOBILE_MIN_SERVER}/${MATTERMOST_DESKTOP_MIN_SERVER}/${MATTERMOST_SUPPORT_MIN_SERVER}" +# Non-zero when any fail/err so parent phase + mon pages mark ERROR +if [ "${FAIL_N:-0}" -gt 0 ]; then + exit 1 +fi +exit 0 diff --git a/check_monitoring_pages.sh b/check_monitoring_pages.sh new file mode 100755 index 0000000..4ca714a --- /dev/null +++ b/check_monitoring_pages.sh @@ -0,0 +1,673 @@ +#!/usr/bin/env bash +# check_monitoring_pages.sh — verify public monitoring HTML pages via FQDN +# +# Outside-in: curl https://// and require a real HTML monitoring page. +# Catches: merchant JSON code 21 (Caddy handle missing), WP 404, empty stubs, deploy skip. +# +# Policy (v1.7.6+): +# • monpages is obligatory → missing/wrong pages are ERROR (exit 1). +# • GOA / hacktivism: full inventory of suite monitoring sites (catalog + on-disk discovery). +# • FP: only that stack’s own monitoring hosts/paths (never GOA URLs). +# • Override soft mode only with MONPAGES_REQUIRE_PUBLIC=0 (emergency / staging). +# +# Env: +# MON_HOSTS space-separated FQDNs (default from TALER_DOMAIN) +# HTML_URL_OK path for OK page (default /monitoring/) +# HTML_URL_ERR err path; checked if present on disk or MONPAGES_CHECK_ERR=1 +# MONITORING_PAGE_URLS optional full URL list (overrides inventory construction) +# MONPAGES_EXTRA_URLS additional absolute URLs (space-separated) +# MONPAGES_INVENTORY auto|full|job (default auto) +# auto/full = stack inventory; job = only MON_HOSTS+paths +# MONPAGES_REQUIRE_PUBLIC default 1 (ERROR). 0 = WARN only (escape hatch). +# MONPAGES_STAGING_BASE / HTML_OUT / DEPLOY_WWW_ROOT trees for discovery +# TIMEOUT curl timeout (from lib.sh) +# +set -euo pipefail +ROOT=$(cd "$(dirname "$0")" && pwd) +# shellcheck source=lib.sh +source "$ROOT/lib.sh" + +set_area monpages +section "monpages · public monitoring HTML via FQDN (outside-in, obligatory)" + +# Obligatory: public pages missing → ERROR. Soft only if explicitly disabled. +: "${MONPAGES_REQUIRE_PUBLIC:=1}" +: "${MONPAGES_INVENTORY:=auto}" + +# Default hosts from domain (mirror run-host-report.sh) +if [ -z "${MON_HOSTS:-}" ]; then + case "${TALER_DOMAIN:-}" in + hacktivism.ch|koopa) + MON_HOSTS="bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch" + ;; + stage.lefrancpaysan.ch|stage.*lefrancpaysan*) + MON_HOSTS="stage.bank.lefrancpaysan.ch stage.exchange.lefrancpaysan.ch stage.monnaie.lefrancpaysan.ch" + ;; + lefrancpaysan.ch) + MON_HOSTS="bank.lefrancpaysan.ch exchange.lefrancpaysan.ch monnaie.lefrancpaysan.ch" + ;; + *) + MON_HOSTS="${TALER_DOMAIN:-}" + ;; + esac +fi + +HTML_URL_OK="${HTML_URL_OK:-/monitoring/}" +HTML_URL_ERR="${HTML_URL_ERR:-/monitoring_err/}" +# ensure leading + trailing slash +_norm_path() { + local p="$1" + case "$p" in + /*) ;; + *) p="/$p" ;; + esac + case "$p" in + */) ;; + *) p="${p}/" ;; + esac + printf '%s' "$p" +} +HTML_URL_OK="$(_norm_path "$HTML_URL_OK")" +HTML_URL_ERR="$(_norm_path "$HTML_URL_ERR")" + +# Stack family: goa | fp | other — FP never checks GOA and vice versa. +_monpages_family() { + case "${TALER_DOMAIN:-}" in + *lefrancpaysan*|*francpaysan*) printf 'fp' ;; + hacktivism.ch|koopa) printf 'goa' ;; + *) + # Infer from MON_HOSTS if domain ambiguous + case " ${MON_HOSTS:-} " in + *lefrancpaysan*) printf 'fp' ;; + *hacktivism*) printf 'goa' ;; + *) printf 'other' ;; + esac + ;; + esac +} + +_host_allowed() { + local h="$1" fam="$2" + case "$fam" in + goa) + case "$h" in *.hacktivism.ch) return 0 ;; *) return 1 ;; esac + ;; + fp) + case "$h" in *lefrancpaysan*) return 0 ;; *) return 1 ;; esac + ;; + *) return 0 ;; + esac +} + +# Public mon URL families (suite name "taler-monitoring" ≠ a public path): +# 1) /monitoring(+_err) — landing hosts (bare + slash) +# 2) /monitoring-max-ladder(+_err) — GOA max-search amount ladder (daily, hacktivism) +# 3) /taler-monitoring-surface(+_err) — taler.hacktivism.ch only +# 4) /taler-monitoring-aptdeploy(+_err) — taler.hacktivism.ch only +# No classic /monitoring-ladder. No /taler-monitoring page. + +_path_allowed() { + case "$1" in + monitoring|monitoring_err|\ + monitoring-max-ladder|monitoring-max-ladder_err|\ + taler-monitoring-surface|taler-monitoring-surface_err|\ + taler-monitoring-aptdeploy|taler-monitoring-aptdeploy_err) return 0 ;; + # explicit deny (never invent a mon page here) + monitoring-ladder|monitoring-ladder_err|\ + taler-monitoring|taler-monitoring-mail|taler-monitoring-mattermost) return 1 ;; + *) return 1 ;; + esac +} + +# Emit https://host/path/ for each …/host/path/index.html under root (max depth 3). +_discover_tree() { + local root="$1" fam="$2" + local f rel host path + [ -n "$root" ] && [ -d "$root" ] || return 0 + # layout: $root///index.html + while IFS= read -r -d '' f; do + rel="${f#"${root%/}"/}" + host="${rel%%/*}" + path="${rel#*/}" + path="${path%/index.html}" + [ -n "$host" ] && [ -n "$path" ] || continue + _host_allowed "$host" "$fam" || continue + _path_allowed "$path" || continue + printf 'https://%s/%s/\n' "$host" "$path" + done < <(find "${root%/}" -mindepth 3 -maxdepth 3 -type f -name index.html -print0 2>/dev/null) +} + +# Landing stacks (GOA 3 + FP stage 3) with /monitoring/ +_landing_allowed() { + printf '%s\n' \ + bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch \ + stage.bank.lefrancpaysan.ch stage.exchange.lefrancpaysan.ch stage.monnaie.lefrancpaysan.ch +} + +# Suite catalog of OK pages that must exist for this family. +_catalog_urls() { + local fam="$1" h + case "$fam" in + goa) + for h in bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch; do + printf 'https://%s/monitoring/\n' "$h" + printf 'https://%s/monitoring-max-ladder/\n' "$h" + done + # ecosystem surface (versions / software) + apt-src deploy mon page + printf 'https://taler.hacktivism.ch/taler-monitoring-surface/\n' + printf 'https://taler.hacktivism.ch/taler-monitoring-aptdeploy/\n' + ;; + fp) + for h in $MON_HOSTS; do + [ -z "$h" ] && continue + _host_allowed "$h" fp || continue + printf 'https://%s/monitoring/\n' "$h" + done + ;; + all|nine) + # GOA + FP stage landings + surface + aptdeploy (nine kept as alias for inventory) + while IFS= read -r h; do + [ -n "$h" ] || continue + printf 'https://%s/monitoring/\n' "$h" + case "$h" in + *.hacktivism.ch) + printf 'https://%s/monitoring-max-ladder/\n' "$h" + ;; + esac + done < <(_landing_allowed) + printf 'https://taler.hacktivism.ch/taler-monitoring-surface/\n' + printf 'https://taler.hacktivism.ch/taler-monitoring-aptdeploy/\n' + ;; + esac +} + +# Job-local URLs (this host-agent run’s path). +_job_urls() { + local h p + for h in $MON_HOSTS; do + [ -z "$h" ] && continue + printf 'https://%s%s\n' "$h" "$HTML_URL_OK" + if [ "${MONPAGES_CHECK_ERR:-0}" = "1" ]; then + printf 'https://%s%s\n' "$h" "$HTML_URL_ERR" + fi + done +} + +build_urls() { + local fam inv + fam="$(_monpages_family)" + inv="${MONPAGES_INVENTORY:-auto}" + + if [ -n "${MONITORING_PAGE_URLS:-}" ]; then + # shellcheck disable=SC2086 + printf '%s\n' $MONITORING_PAGE_URLS + else + case "$inv" in + job) + # Only this host-agent job’s HTML paths (MON_HOSTS + HTML_URL_OK/_ERR). + # Specialized timers (aptdeploy, surface) must not inherit GOA full catalog + # (no bank/exchange /monitoring bare-URL noise on the aptdeploy page). + _job_urls + ;; + full|auto|*) + # Always include this job’s paths (host-agent post-check). + _job_urls + # Stack inventory: GOA = landings + surface + aptdeploy; FP = only FP mon hosts. + # Full inventory belongs on the general stack run (run-hacktivism-monitoring). + if [ "$fam" = "goa" ] || [ "$fam" = "fp" ]; then + _catalog_urls "$fam" + # On-disk discovery (err pages only exist after failed runs; extras welcome) + [ -n "${DEPLOY_WWW_ROOT:-}" ] && _discover_tree "${DEPLOY_WWW_ROOT}" "$fam" + _discover_tree "${MONPAGES_STAGING_BASE:-${HTML_OUT:-${HTML_BASE:-$HOME/monitoring-sites-staging}}}" "$fam" + fi + ;; + esac + fi + if [ -n "${MONPAGES_EXTRA_URLS:-}" ]; then + # shellcheck disable=SC2086 + printf '%s\n' $MONPAGES_EXTRA_URLS + fi +} + +# Install stub only (no console log) — not a real mon page. +is_bootstrap_html() { + local f="$1" + grep -qiE 'Monitoring page \(bootstrap\)|pill">STAGE/dev/null \ + && ! grep -qE 'taler-mon:top|id="mon-console"|class="console"' "$f" 2>/dev/null +} + +# Stale mon HTML from pre-1.15.8 generators: box-drawing char-class regex in +# sticky filter JS (Python SyntaxWarning: invalid escape sequence '\-'). +# Never accept these as "current" public assets (v1.18.2+). +is_stale_generator_html() { + local f="$1" + # Old isLowValue() JS (literal box glyphs + \- class) + if grep -qE 'if \(/\^\[[╔╗╚╝║═]' "$f" 2>/dev/null; then + return 0 + fi + # Converter noise teed into old logs and re-rendered + if grep -qiE 'SyntaxWarning: invalid escape sequence' "$f" 2>/dev/null; then + return 0 + fi + if grep -qiE 'console_to_html\.py:[0-9]+: SyntaxWarning' "$f" 2>/dev/null; then + return 0 + fi + return 1 +} + +# True if body looks like our monitoring HTML (not merchant API / WP 404 / bootstrap). +is_monitoring_html() { + local f="$1" + if grep -qE '"code"[[:space:]]*:[[:space:]]*21' "$f" 2>/dev/null; then + return 1 + fi + if grep -qiE 'There is no endpoint defined for the URL' "$f" 2>/dev/null; then + return 1 + fi + if grep -qiE 'wp-content|wordpress' "$f" 2>/dev/null \ + && grep -qiE '404|not found|page introuvable' "$f" 2>/dev/null; then + return 1 + fi + # Bootstrap sticky is not enough — need suite console output + if is_bootstrap_html "$f"; then + return 1 + fi + if grep -qE 'sticky-bar|version-link|taler-monitoring|class="sticky|taler-mon:top' "$f" 2>/dev/null; then + return 0 + fi + if head -c 4096 "$f" | grep -qiE '/dev/null; then + return 0 + fi + if head -c 512 "$f" | grep -qiE '/dev/null \ + | grep -oE '[0-9]+' | sort -n | tail -1 || true) + printf '%s' "${m:-0}" +} + +# Mid-run: incomplete meta / redirect stub, or progress never reached 100% and no complete foot. +_page_mid_run() { + local f="$1" + local pct + if grep -qE 'name="taler-mon-page"[[:space:]]+content="incomplete"' "$f" 2>/dev/null \ + || grep -q 'taler-mon:bottom:incomplete' "$f" 2>/dev/null \ + || grep -qE 'http-equiv="refresh".*monitoring_err|mode-redirect' "$f" 2>/dev/null; then + return 0 + fi + # finished page always has complete foot or classic footer + if grep -q 'taler-mon:bottom:complete' "$f" 2>/dev/null \ + || grep -qE 'id="mon-footer"' "$f" 2>/dev/null; then + return 1 + fi + pct=$(_progress_max_pct "$f") + # max progress seen < 100% and no complete footer → still running + if [ "${pct:-0}" -gt 0 ] && [ "${pct:-0}" -lt 100 ]; then + if ! grep -qE 'Console-style render of taler-monitoring|Rendu console de taler-monitoring' "$f" 2>/dev/null; then + return 0 + fi + fi + return 1 +} + +# Validate top/bottom markers + meaningful body (v1.9.0). +# Sets _MON_CONTENT_MSG; returns 0=ok, 1=hard fail, 2=soft/mid-run bottom skip +# +# Head windows are large (v1.13.4+): sticky CSS grew past old 12–16 KiB limits and +# caused false TOP errors while publish races were also writing the file. +validate_mon_content() { + local f="$1" url="$2" + local top_ok=1 bot_ok=1 body_ok=1 mid=0 pct + local head_top="${MONPAGES_HEAD_TOP_BYTES:-49152}" + local head_meta="${MONPAGES_HEAD_META_BYTES:-65536}" + _MON_CONTENT_MSG="" + + # Tiny / empty body = likely mid-publish race (partial rsync or curl) + if [ ! -s "$f" ] || [ "$(wc -c <"$f" 2>/dev/null || echo 0)" -lt 200 ]; then + top_ok=0 + _MON_CONTENT_MSG="empty or truncated body (publish race?)" + # --- top markers (always required) --- + elif ! head -c "$head_top" "$f" | grep -qE 'sticky-bar|id="status-bar"|taler-mon:top'; then + # full-file fallback (small pages / unusual layout) + if ! grep -qE 'sticky-bar|id="status-bar"|taler-mon:top' "$f" 2>/dev/null; then + top_ok=0 + _MON_CONTENT_MSG="missing top marker (sticky-bar / status-bar / taler-mon:top)" + fi + elif ! head -c "$head_meta" "$f" | grep -qE 'data-generated-iso|name="generated"|version-link|taler-mon-page'; then + if ! grep -qE 'data-generated-iso|name="generated"|version-link|taler-mon-page' "$f" 2>/dev/null; then + top_ok=0 + _MON_CONTENT_MSG="missing top meta (generated / version-link / taler-mon-page)" + fi + fi + + # --- meaningful body: real checks, not only progress bars --- + if grep -qE 'class="line (ok|error|warn|blocker|info)"|┌ OK|┌ ERROR|┌ WARN|┌ BLOCKER|┌ INFO|class="box"' "$f" 2>/dev/null; then + body_ok=1 + elif grep -qE 'taler-mon:bottom:incomplete|mode-redirect|has failures' "$f" 2>/dev/null; then + body_ok=1 # redirect stub still has sticky + box + else + # only progress / empty? + if grep -qE '┌[[:space:]]*[0-9]+%┐|█|░' "$f" 2>/dev/null \ + && ! grep -qE '┌ OK|┌ ERROR|class="line ok"' "$f" 2>/dev/null; then + body_ok=0 + _MON_CONTENT_MSG="${_MON_CONTENT_MSG:+$_MON_CONTENT_MSG; }only progress bars / no check lines" + elif ! grep -qE 'console|host-agent|taler-monitoring|monitoring' "$f" 2>/dev/null; then + body_ok=0 + _MON_CONTENT_MSG="${_MON_CONTENT_MSG:+$_MON_CONTENT_MSG; }no meaningful suite content" + fi + fi + + # --- bottom markers --- + if grep -q 'taler-mon:bottom:complete' "$f" 2>/dev/null \ + || tail -c 8000 "$f" | grep -qE 'id="mon-footer"||Console-style render of taler-monitoring'; then + bot_ok=1 + elif grep -q 'taler-mon:bottom:incomplete' "$f" 2>/dev/null; then + bot_ok=1 # incomplete is an explicit bottom marker + mid=1 + else + bot_ok=0 + _MON_CONTENT_MSG="${_MON_CONTENT_MSG:+$_MON_CONTENT_MSG; }missing bottom marker (footer / taler-mon:bottom)" + fi + + if _page_mid_run "$f"; then + mid=1 + fi + pct=$(_progress_max_pct "$f") + + if [ "$top_ok" != "1" ]; then + # TOP missing is WARN-only (v1.13.8+): mid-run races and incomplete + # /monitoring stubs used to self-red the next HTML. Hard ERROR still for + # missing page / code 21 / non-suite body. Override: MONPAGES_TOP_STRICT=1. + _MON_CONTENT_MSG="TOP: $_MON_CONTENT_MSG" + if [ "${MONPAGES_TOP_STRICT:-0}" = "1" ]; then + return 1 + fi + return 2 + fi + if [ "$body_ok" != "1" ]; then + _MON_CONTENT_MSG="BODY: $_MON_CONTENT_MSG" + return 1 + fi + if [ "$bot_ok" != "1" ]; then + if [ "$mid" = "1" ]; then + _MON_CONTENT_MSG="BOTTOM soft (mid-run progress ${pct}%): $_MON_CONTENT_MSG" + return 2 + fi + _MON_CONTENT_MSG="BOTTOM: $_MON_CONTENT_MSG" + return 1 + fi + if [ "$mid" = "1" ] && [ "${pct:-0}" -gt 0 ] && [ "${pct:-0}" -lt 100 ]; then + _MON_CONTENT_MSG="mid-run progress ${pct}% (bottom markers relaxed)" + return 2 + fi + _MON_CONTENT_MSG="top+body+bottom markers OK" + return 0 +} + +_is_bare_url() { + local url="$1" + case " ${MONPAGES_BARE_URLS:-} " in + *" $url "*) return 0 ;; + esac + case "$url" in + */) return 1 ;; + *) return 0 ;; + esac +} + +_mon_fail_or_soft() { + local label="$1" detail="$2" + if [ "${MONPAGES_REQUIRE_PUBLIC:-1}" != "1" ]; then + warn "$label" "$detail (MONPAGES_REQUIRE_PUBLIC=0)" + return 0 + fi + fail "$label" "$detail" + return 1 +} + +# Fetch + validate one URL. Retries content failures (publish/rsync races). +# MONPAGES_PRE_PUBLISH=1 → content failures are WARN only (suite still regenerates HTML after). +# MONPAGES_FETCH_RETRIES (default 3), MONPAGES_FETCH_SLEEP_S (default 0.5). +check_one() { + local url="$1" + local body code hint soft=0 vc attempt max_try sleep_s pre_pub + body=$(mktemp) + if _is_bare_url "$url" && [ "${MONPAGES_BARE_STRICT:-0}" != "1" ]; then + soft=1 + fi + pre_pub=0 + [ "${MONPAGES_PRE_PUBLISH:-0}" = "1" ] && pre_pub=1 + max_try="${MONPAGES_FETCH_RETRIES:-3}" + sleep_s="${MONPAGES_FETCH_SLEEP_S:-0.5}" + [ "$max_try" -lt 1 ] && max_try=1 + + attempt=1 + while [ "$attempt" -le "$max_try" ]; do + : >"$body" + code=$(curl -skS -L --max-redirs 5 -m "${TIMEOUT}" -o "$body" -w '%{http_code}' "$url" 2>/dev/null || echo 000) + + if grep -qE '"code"[[:space:]]*:[[:space:]]*21' "$body" 2>/dev/null \ + || grep -qiE 'There is no endpoint defined for the URL' "$body" 2>/dev/null; then + # Bare without trailing slash often hits merchant until Caddy has redir * /path/ 302. + if [ "$soft" = "1" ]; then + warn "public mon page bare URL" \ + "$url -> HTTP $code code 21 (prefer trailing slash; Caddy: redir * /path/ 302; set MONPAGES_BARE_STRICT=1 to ERROR)" + rm -f "$body" + return 0 + fi + # hard 21: retry once more (rare brief outage) then fail + if [ "$attempt" -lt "$max_try" ]; then + attempt=$((attempt + 1)) + sleep "$sleep_s" 2>/dev/null || sleep 1 + continue + fi + _mon_fail_or_soft "public mon page missing" \ + "$url -> HTTP $code merchant/API JSON code 21 (Caddy handle/redir; use redir * /path/ 302 inside handle${APPLY_MONITORING_LIVE:+; apply: sudo $APPLY_MONITORING_LIVE})" || { rm -f "$body"; return 1; } + rm -f "$body" + return 0 + fi + + if [ "$code" != "200" ]; then + if [ "$soft" = "1" ]; then + warn "public mon page bare URL" "$url -> HTTP $code (prefer trailing slash form)" + rm -f "$body" + return 0 + fi + if [ "$attempt" -lt "$max_try" ]; then + attempt=$((attempt + 1)) + sleep "$sleep_s" 2>/dev/null || sleep 1 + continue + fi + _mon_fail_or_soft "public mon page" "$url -> HTTP $code (want 200 HTML)" || { rm -f "$body"; return 1; } + rm -f "$body" + return 0 + fi + + if is_bootstrap_html "$body"; then + if [ "$attempt" -lt "$max_try" ]; then + attempt=$((attempt + 1)) + sleep "$sleep_s" 2>/dev/null || sleep 1 + continue + fi + _mon_fail_or_soft "public mon page bootstrap stub" \ + "$url -> install bootstrap HTML still live (run host-agent; suite auto-update must replace stub)" || { rm -f "$body"; return 1; } + rm -f "$body" + return 0 + fi + + # Stale generator JS / SyntaxWarning in public body (pre-1.15.8 assets) + if is_stale_generator_html "$body"; then + if [ "$attempt" -lt "$max_try" ]; then + attempt=$((attempt + 1)) + sleep "$sleep_s" 2>/dev/null || sleep 1 + continue + fi + _mon_fail_or_soft "public mon page stale generator" \ + "$url -> old mon HTML (box-drawing JS regex and/or SyntaxWarning in body) — re-run host-agent on suite ≥v1.15.8 / v1.18.2; do not leave pre-update assets live" || { rm -f "$body"; return 1; } + rm -f "$body" + return 0 + fi + + if ! is_monitoring_html "$body"; then + if [ "$attempt" -lt "$max_try" ]; then + attempt=$((attempt + 1)) + sleep "$sleep_s" 2>/dev/null || sleep 1 + continue + fi + hint=$(head -c 120 "$body" | tr '\n' ' ' | tr -cd '[:print:] ') + _mon_fail_or_soft "public mon page not monitoring HTML" \ + "$url -> HTTP 200 body!=suite (${hint}...)" || { rm -f "$body"; return 1; } + rm -f "$body" + return 0 + fi + + # v1.9.0: top + bottom markers + meaningful content + set +e + validate_mon_content "$body" "$url" + vc=$? + set -e + case "$vc" in + 0) + ok "public mon page" "$url -> HTTP 200 · ${_MON_CONTENT_MSG}" + rm -f "$body" + return 0 + ;; + 2) + # soft: TOP incomplete / mid-run bottom — WARN, never hard fail + case "${_MON_CONTENT_MSG}" in + TOP:*) + warn "public mon page content" \ + "$url -> ${_MON_CONTENT_MSG} (WARN only; set MONPAGES_TOP_STRICT=1 to ERROR)" + ;; + *) + warn "public mon page mid-run" "$url -> ${_MON_CONTENT_MSG}" + ok "public mon page" "$url -> HTTP 200 suite HTML (mid-run bottom relaxed)" + ;; + esac + rm -f "$body" + return 0 + ;; + *) + # Content race (truncated HTML while another agent publishes): retry + if [ "$attempt" -lt "$max_try" ]; then + attempt=$((attempt + 1)) + sleep "$sleep_s" 2>/dev/null || sleep 1 + continue + fi + # TOP failures should already be soft (return 2); belt-and-braces: + case "${_MON_CONTENT_MSG}" in + TOP:*) + if [ "${MONPAGES_TOP_STRICT:-0}" != "1" ]; then + warn "public mon page content" \ + "$url -> ${_MON_CONTENT_MSG} (WARN only; set MONPAGES_TOP_STRICT=1 to ERROR)" + rm -f "$body" + return 0 + fi + ;; + esac + # Pre-publish inventory in host-agent PHASES: WARN only (post-check is hard) + if [ "$pre_pub" = "1" ] || [ "$soft" = "1" ]; then + warn "public mon page content" \ + "$url -> ${_MON_CONTENT_MSG}${pre_pub:+ (pre-publish; post-check is authoritative)}" + rm -f "$body" + return 0 + fi + _mon_fail_or_soft "public mon page content" \ + "$url -> ${_MON_CONTENT_MSG}" || { rm -f "$body"; return 1; } + rm -f "$body" + return 0 + ;; + esac + done + rm -f "$body" + return 0 +} + +URLS=() +while IFS= read -r line; do + [ -n "$line" ] || continue + # dedupe + skip=0 + for e in "${URLS[@]+"${URLS[@]}"}"; do + [ "$e" = "$line" ] && skip=1 && break + done + [ "$skip" = "1" ] && continue + URLS+=("$line") +done < <(build_urls | sort -u) + +# Bare URLs without trailing slash: checked by default. +# Merchant JSON code 21 is always ERROR (Caddy/proxy not serving mon path). +# Other bare failures: WARN unless MONPAGES_BARE_STRICT=1 (then ERROR). +# MONPAGES_CHECK_BARE=0 → skip bare entirely +# MONPAGES_BARE_STRICT=1 → non-21 bare failures are ERROR too +if [ "${MONPAGES_CHECK_BARE:-1}" = "1" ]; then + _extra=() + for u in "${URLS[@]}"; do + case "$u" in + */) _bare=${u%/}; [ -n "$_bare" ] && _extra+=("$_bare") ;; + esac + done + for u in "${_extra[@]+"${_extra[@]}"}"; do + skip=0 + for e in "${URLS[@]+"${URLS[@]}"}"; do + [ "$e" = "$u" ] && skip=1 && break + done + [ "$skip" = "1" ] && continue + URLS+=("$u") + MONPAGES_BARE_URLS="${MONPAGES_BARE_URLS:-} $u" + done + export MONPAGES_BARE_URLS +fi + +if [ "${#URLS[@]}" -eq 0 ]; then + fail "monpages" "no URLs to check (set MON_HOSTS or MONITORING_PAGE_URLS)" + exit 1 +fi + +_fam="$(_monpages_family)" +echo " family=${_fam} inventory=${MONPAGES_INVENTORY} require_public=${MONPAGES_REQUIRE_PUBLIC}" +echo " checking ${#URLS[@]} URL(s) via FQDN..." +ec=0 +for u in "${URLS[@]}"; do + check_one "$u" || ec=1 +done + +if [ "$ec" -ne 0 ]; then + echo " hint: public monitoring HTML not served (404 / merchant code 21)" + echo " publish staging HTML + configure reverse-proxy/Caddy handle for mon paths" + echo " (set APPLY_MONITORING_LIVE in env; sudo "$APPLY_MONITORING_LIVE")" + echo " bare code 21: Caddy redir inside handle must be: redir * /path/ 302 (not redir /path/ 302)" + if [ "$_fam" = "fp" ]; then + echo " FP: only FP mon hosts are checked — wire Infomaniak vhost for /monitoring*" + fi + if [ "$_fam" = "goa" ]; then + echo " GOA: bank/exchange/taler /monitoring/ + surface + aptdeploy (v1.9.4+ layout)" + fi + # Staging vs public diagnosis (host-agent sets MONPAGES_STAGING_BASE / HTML_OUT) + _stg="${MONPAGES_STAGING_BASE:-${HTML_OUT:-${HTML_BASE:-$HOME/monitoring-sites-staging}}}" + _ok_dir="${HTML_OK_DIR:-monitoring}" + if [ -n "${MON_HOSTS:-}" ] && [ -n "$_stg" ]; then + for _h in $MON_HOSTS; do + _f="$_stg/$_h/${_ok_dir}/index.html" + if [ -f "$_f" ]; then + echo " staging OK: $_f ($(wc -c <"$_f" | tr -d " ") bytes) — not published/served" + else + echo " staging missing: $_f" + fi + done + fi + _www="${DEPLOY_WWW_ROOT:-}" + if [ -e "$_www" ] && [ ! -w "$_www" ]; then + echo " DEPLOY_WWW_ROOT=$_www not writable by $(id -un 2>/dev/null || echo user)" + fi +fi + +exit "$ec" diff --git a/check_surface.sh b/check_surface.sh index 1452d3b..bbecb07 100755 --- a/check_surface.sh +++ b/check_surface.sh @@ -32,6 +32,10 @@ PORT_TIMEOUT="${SURFACE_PORT_TIMEOUT:-2}" HTTP_TIMEOUT="${SURFACE_HTTP_TIMEOUT:-12}" # CVE via OSV (public, no key). Disable: SURFACE_CVE=0 : "${SURFACE_CVE:=1}" +# nmap OS / service fingerprint (v1.8.0+). Disable: SURFACE_NMAP=0 +: "${SURFACE_NMAP:=1}" +# OS detection (-O) often needs root; falls back to -sV when denied +: "${SURFACE_NMAP_OS:=1}" # Extra ports when -d domain mode DOMAIN_EXTRA_PORTS="${SURFACE_DOMAIN_PORTS:-22,80,443,8443}" @@ -496,6 +500,109 @@ cert_expiry_check() { return 0 } +# nmap OS / service fingerprint (v1.8.0+). +# ERROR when fingerprint indicates an OS that is EOL / should be upgraded. +# Soft/info when nmap missing or unprivileged OS scan fails. +nmap_fingerprint_host() { + local h="$1" + local xml out osline accuracy eol=0 + [ "${SURFACE_NMAP:-1}" = "1" ] || return 0 + if ! command -v nmap >/dev/null 2>&1; then + warn "nmap" "$h · nmap not installed (apt install nmap) — skip OS fingerprint" + return 0 + fi + xml=$(mktemp) + out=$(mktemp) + # Prefer OS detection when allowed; always try service version light + if [ "${SURFACE_NMAP_OS:-1}" = "1" ] && { + nmap -Pn -n -T4 -O --osscan-guess --max-os-tries 1 --host-timeout 45s \ + -p 22,80,443,25,465,587,993 "$h" -oX "$xml" >"$out" 2>&1 + }; then + : + else + nmap -Pn -n -T4 -sV --version-light --host-timeout 35s \ + -p 22,80,443,25,465,587,993 "$h" -oX "$xml" >"$out" 2>&1 || true + fi + # Parse best OS match from XML + osline=$( + python3 - "$xml" <<'PY' 2>/dev/null || true +import sys, xml.etree.ElementTree as ET +path = sys.argv[1] +try: + root = ET.parse(path).getroot() +except Exception: + sys.exit(0) +best = None +best_acc = -1 +for osm in root.findall(".//osmatch"): + name = osm.get("name") or "" + try: + acc = int(osm.get("accuracy") or "0") + except ValueError: + acc = 0 + if name and acc >= best_acc: + best_acc = acc + best = name +if best: + print(f"{best_acc}|{best}") +# also surface product/version from service table +svcs = [] +for port in root.findall(".//port"): + state = (port.find("state").get("state") if port.find("state") is not None else "") + if state != "open": + continue + svc = port.find("service") + if svc is None: + continue + prod = svc.get("product") or svc.get("name") or "" + ver = svc.get("version") or "" + if prod: + svcs.append(f"{prod} {ver}".strip()) +if svcs: + print("SVC|" + "; ".join(svcs[:8])) +PY + ) + rm -f "$xml" "$out" + if [ -z "$osline" ]; then + info "nmap" "$h · no OS/service fingerprint (need root for -O, or host filtered)" + return 0 + fi + local osname="" svcinfo="" + while IFS= read -r line; do + case "$line" in + SVC\|*) svcinfo=${line#SVC|} ;; + *\|*) + accuracy=${line%%|*} + osname=${line#*|} + ok "nmap-os" "$h · OS guess ${accuracy}% · $osname" + ;; + esac + done <<<"$osline" + [ -n "$svcinfo" ] && info "nmap-svc" "$h · $svcinfo" + + # EOL / upgrade-required OS fingerprints → ERROR + case "$(printf '%s' "$osname" | tr 'A-Z' 'a-z')" in + *debian*6*|*debian*7*|*debian*8*|*debian*9*|*debian*10*|*debian*\"buster\"*|*debian*buster*) + eol=1 ;; + *ubuntu*12.04*|*ubuntu*14.04*|*ubuntu*16.04*|*ubuntu*18.04*|*ubuntu*20.04*) + eol=1 ;; + *centos*[5-7]*|*red\ hat\ enterprise*5*|*red\ hat\ enterprise*6*|*red\ hat\ enterprise*7*) + eol=1 ;; + *windows\ server\ 2008*|*windows\ server\ 2012*|*windows\ xp*|*windows\ 7*) + eol=1 ;; + *freebsd\ 1[0-2]*|*freebsd\ [0-9].*) + eol=1 ;; + esac + if [ "$eol" = "1" ]; then + err "nmap-upgrade" "$h OS fingerprint indicates EOL / upgrade required" "$osname" + return 1 + fi + if [ -n "$osname" ]; then + info "nmap-upgrade" "$h OS fingerprint not on EOL list · $osname" + fi + return 0 +} + scan_host() { local h="$1" local ports proto label expect @@ -621,6 +728,9 @@ scan_host() { fi fi + # nmap OS / service fingerprint (v1.8.0+) — ERROR on EOL OS + nmap_fingerprint_host "$h" || true + # Expected service must answer protocol (not just ping/port) # Note: ${array[*]:-x} is NOT valid default syntax (bash treats : as slice). local ports_txt="${open_ports[*]}" @@ -667,7 +777,7 @@ else fi info "inventory" "${#ORDERED_HOSTS[@]} hosts to probe (remote only · no SSH · no local podman on targets)" -info "flags" "SURFACE_CVE=${SURFACE_CVE:-1} PORT_TIMEOUT=${PORT_TIMEOUT:-2} HTTP_TIMEOUT=${HTTP_TIMEOUT:-12}" +info "flags" "SURFACE_CVE=${SURFACE_CVE:-1} SURFACE_NMAP=${SURFACE_NMAP:-1} SURFACE_NMAP_OS=${SURFACE_NMAP_OS:-1} PORT_TIMEOUT=${PORT_TIMEOUT:-2} HTTP_TIMEOUT=${HTTP_TIMEOUT:-12}" for h in "${ORDERED_HOSTS[@]}"; do [ -n "$h" ] || continue diff --git a/check_urls.sh b/check_urls.sh index 36520d4..f2557ff 100755 --- a/check_urls.sh +++ b/check_urls.sh @@ -259,7 +259,7 @@ set_group perf section "www · performance · public HTTPS latency (outside-in)" # Latency thresholds (ms), outside-in from this runner. -# ≥ PERF_WARN_MS → WARN. ≥ PERF_FAIL_MS → ERROR (suite fails). +# ≥ PERF_WARN_MS → WARN. ≥ PERF_FAIL_MS → suite fail (hard). PERF_WARN_MS="${PERF_WARN_MS:-8000}" PERF_FAIL_MS="${PERF_FAIL_MS:-20000}" PERF_TSV="$tmp/perf.tsv" @@ -303,14 +303,18 @@ check_perf() { esac } -# Bank first (wallet-critical paths before UI chrome) -check_perf "perf bank /taler-integration/config" "$BANK_PUBLIC/taler-integration/config" -check_perf "perf bank /config" "$BANK_PUBLIC/config" -if [ "${CHECK_LANDING:-1}" = "1" ]; then - check_perf "perf bank /intro/" "$BANK_PUBLIC/intro/" - check_perf "perf bank /intro/stats.json" "$BANK_PUBLIC/intro/stats.json" 200 +# Bank first (wallet-critical paths before UI chrome) — skip when CHECK_BANK=0 (e.g. mytops stage mon) +if [ "${CHECK_BANK:-1}" = "1" ]; then + check_perf "perf bank /taler-integration/config" "$BANK_PUBLIC/taler-integration/config" + check_perf "perf bank /config" "$BANK_PUBLIC/config" + if [ "${CHECK_LANDING:-1}" = "1" ]; then + check_perf "perf bank /intro/" "$BANK_PUBLIC/intro/" + check_perf "perf bank /intro/stats.json" "$BANK_PUBLIC/intro/stats.json" 200 + fi + check_perf "perf bank /webui/" "$BANK_PUBLIC/webui/" 200,301,302 +else + info "perf bank" "skipped (CHECK_BANK=0)" fi -check_perf "perf bank /webui/" "$BANK_PUBLIC/webui/" 200,301,302 # Exchange check_perf "perf exchange /config" "$EXCHANGE_PUBLIC/config" @@ -576,7 +580,8 @@ PY if [ "${CHECK_LANDING:-1}" = "1" ] || [ "${LOCAL_STACK:-0}" = "1" ]; then set_group stats section "www · landing stats.json (freshness + display fields · outside-in)" - info "stats policy" "STALE≥${STATS_STALE_SECS}s WARN · FAIL≥${STATS_FAIL_SECS}s ERROR (0=disable fail) · need generated_at_unix" + # Avoid the bare word ERROR in INFO text (older HTML converters treated it as a check failure / first-error). + info "stats policy" "STALE≥${STATS_STALE_SECS}s→WARN · FAIL≥${STATS_FAIL_SECS}s→suite-fail (0=off) · need generated_at_unix" check_landing_stats_json "bank" "$BANK_PUBLIC" || true check_landing_stats_json "exchange" "$EXCHANGE_PUBLIC" || true check_landing_stats_json "merchant" "$MERCHANT_PUBLIC" || true @@ -719,13 +724,17 @@ esac # --- bank --- set_group bank -if [ "${LOCAL_STACK:-1}" = "0" ]; then +if [ "${CHECK_BANK:-1}" != "1" ]; then + info "bank" "skipped (CHECK_BANK=0 — not in scope for this mon job)" + code="" +elif [ "${LOCAL_STACK:-1}" = "0" ]; then check_url_soft "bank /config" 200 "$BANK_PUBLIC/config" + code=$(http_body "$BANK_PUBLIC/config" "$tmp/bc.json") else check_url "bank /config" 200 "$BANK_PUBLIC/config" + code=$(http_body "$BANK_PUBLIC/config" "$tmp/bc.json") fi -code=$(http_body "$BANK_PUBLIC/config" "$tmp/bc.json") -if [ "$code" = "200" ]; then +if [ "${CHECK_BANK:-1}" = "1" ] && [ "$code" = "200" ]; then expect_currency "bank" "$tmp/bc.json" if json_has_alt_unit_names "$tmp/bc.json" >/tmp/alt-bank.$$ 2>&1; then ok "bank /config alt_unit_names" "$(tr '\n' '; ' /dev/null || true +import re, sys, json +from pathlib import Path +raw = Path(sys.argv[1]).read_text(errors="replace") +# collect taler:// from JSON fields and raw HTML/JS +uris = set(re.findall(r"taler://[A-Za-z0-9._~:/?#\[\]@!$&'()*+,;=%-]+", raw)) +# JSON-escaped sequences +uris.update(re.findall(r"taler:\\/\\/[A-Za-z0-9._~:/?#\[\]@!$&'()*+,;=%\\-]+", raw)) +uris = {u.replace("\\/", "/") for u in uris} +bad = [] +pat = re.compile(r"taler://\S+:(443|80)(/|$|\?|#)", re.I) +for u in sorted(uris): + if pat.search(u): + bad.append(u[:120]) +if bad: + print("BAD\t%d\t%s" % (len(bad), bad[0])) +else: + print("OK\t%d\t" % len(uris)) +PY +) + n_kind=${detail%%$'\t'*} + rest=${detail#*$'\t'} + n_count=${rest%%$'\t'*} + sample=${rest#*$'\t'} + if [ "$n_kind" = "BAD" ]; then + if [ "${LOCAL_STACK:-1}" = "1" ] || [ "${EXPECT_CURRENCY:-}" = "GOA" ]; then + fail "taler-uri ports · $label" "default :443/:80 still present · n=${n_count} e.g. ${sample}" + _port_scan_fail=$((_port_scan_fail + 1)) + else + warn "taler-uri ports · $label" "default port in URI · ${sample}" + fi + else + ok "taler-uri ports · $label" "no default ports · taler_uris=${n_count:-0} · $url" + _port_scan_ok=$((_port_scan_ok + 1)) + fi + rm -f "$f" +} +if [ "${CHECK_LANDING:-1}" = "1" ]; then + _port_scan_one "bank intro HTML" "${BANK_PUBLIC}/intro/" + _port_scan_one "bank demo-withdraw.json" "${BANK_PUBLIC}/intro/demo-withdraw.json" + _port_scan_one "bank auto-account.json" "${BANK_PUBLIC}/intro/auto-account.json" + _port_scan_one "exchange intro HTML" "${EXCHANGE_PUBLIC}/intro/" + _port_scan_one "merchant intro HTML" "${MERCHANT_PUBLIC}/intro/" + # stats may embed demo withdraw URI + _port_scan_one "bank stats.json" "${BANK_PUBLIC}/intro/stats.json" + if [ "$_port_scan_fail" -eq 0 ]; then + ok "taler-uri ports summary" "clean · ${_port_scan_ok} resource(s) checked (bank+exchange+merchant)" + fi +else + info "taler-uri ports" "skipped (CHECK_LANDING=0)" +fi + # Merchant shop assets — GOA: shop-pay.*; stage TESTPAYSAN: /assets/shop-ui.js + # shops.css + QR encoder (shop-ui modal needs /intro/qrcode.min.js or /qrcode.min.js). _ma=0 diff --git a/check_versions.sh b/check_versions.sh index 8fc7c66..3abc9ae 100755 --- a/check_versions.sh +++ b/check_versions.sh @@ -11,7 +11,7 @@ # TALER_APT_BASE=https://deb.taler.net/apt/debian # TALER_APT_INDEX=…/dists/trixie/main/binary-amd64/Packages # TALER_APT_TESTING_INDEX=…/dists/trixie-testing/… -# TALER_PKG_BEHIND=warn|error (default: core packages ERROR if behind, else warn) +# TALER_PKG_BEHIND=warn|error (default error as of v1.8.0 — any behind package is ERROR) # SKIP_SSH=1 skip container install checks set -euo pipefail ROOT=$(cd "$(dirname "$0")" && pwd) @@ -27,7 +27,7 @@ APT_BASE="${APT_BASE%/}" INDEX="${TALER_APT_INDEX:-${APT_BASE}/dists/${SUITE}/main/binary-amd64/Packages}" INRELEASE="${TALER_APT_INRELEASE:-${APT_BASE}/dists/${SUITE}/InRelease}" TESTING_INDEX="${TALER_APT_TESTING_INDEX:-${APT_BASE}/dists/${SUITE}-testing/main/binary-amd64/Packages}" -BEHIND_MODE="${TALER_PKG_BEHIND:-warn}" +BEHIND_MODE="${TALER_PKG_BEHIND:-error}" tmp=$(mktemp -d) trap 'rm -rf "$tmp"' EXIT @@ -178,7 +178,7 @@ if [ "${INSIDE_PODMAN:-0}" = "1" ] || [ "${INSIDE_MODE:-}" = "local-podman" ]; t elif [ "${INSIDE_PROFILE:-}" = "stage-lfp" ] \ || [ "${EXPECT_CURRENCY:-}" = "TESTPAYSAN" ] \ || [ -n "${INSIDE_SSH:-}" ]; then - _VERS_SSH_HOST="${INSIDE_SSH:-francpaysan-stage-user}" + _VERS_SSH_HOST="${INSIDE_SSH:-}" _VERS_BANK="${INSIDE_BANK_CTR:-stage-lfp-bank}" _VERS_EX="${INSIDE_EXCHANGE_CTR:-stage-lfp-exchange-ansible}" _VERS_MER="${INSIDE_MERCHANT_CTR:-stage-lfp-merchant}" diff --git a/domains.conf b/domains.conf index 1cc12af..1002e69 100644 --- a/domains.conf +++ b/domains.conf @@ -1,46 +1,53 @@ # taler-monitoring domain profiles # # When you add a stack, declare the three public endpoints here: -# name bank exchange merchant-backend currency local landing [canonical] +# name bank exchange merchant-backend currency local landing lang locale [canonical] # # - name: what -d / TALER_DOMAIN matches (aliases allowed as extra lines) # - bank / exchange / merchant: hostname or full https:// URL -# - currency: GOA | KUDOS | CHF | … (empty = report only) +# - currency: GOA | KUDOS | CHF | … (empty or "-" = report only) # - local: 1 = koopa SSH/inside/e2e stack; 0 = public-only # - landing: 1 = check /intro landings + assets; 0 = skip (no GOA-style landings) +# - lang: UI language for mon console/HTML (en|fr|de). Source of truth when +# TALER_MON_LANG_SET≠1. Workstation TALER_MON_LANG=en does not override. +# **de is supported but must not be used as a profile default** (no row +# sets lang=de unless an operator deliberately chooses it). +# - locale: BCP 47 regional locale for l10n defaults (not the same as UI lang). +# Use **fr-CH** (Suisse romande) for FrancPaysan; **de-CH** (Deutschschweiz) +# for other Swiss / default stacks. Accepts also ch-FR / ch-DE as aliases. # - canonical: optional TALER_DOMAIN label after alias match # -# CLI overrides after load: --bank --exchange --merchant --currency +# Legacy lines without locale still parse (locale → de-CH except FP heuristic). +# CLI: --lang en|fr|de · TALER_MON_LANG_SET=1 locks UI lang # Alternate file: TALER_DOMAINS_CONF=/path/to/domains.conf -# Local GOA (koopa) — landings matter -koopa bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch GOA 1 1 hacktivism.ch -hacktivism bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch GOA 1 1 hacktivism.ch -hacktivism.ch bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch GOA 1 1 +# Local GOA (koopa) — landings matter · UI en · locale de-CH +koopa bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch GOA 1 1 en de-CH hacktivism.ch +hacktivism bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch GOA 1 1 en de-CH hacktivism.ch +hacktivism.ch bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch GOA 1 1 en de-CH -# Public demo (KUDOS) — has intro-style pages; keep soft landing checks -taler.net bank.demo.taler.net exchange.demo.taler.net backend.demo.taler.net KUDOS 0 1 demo.taler.net -demo.taler.net bank.demo.taler.net exchange.demo.taler.net backend.demo.taler.net KUDOS 0 1 +# Public demo (KUDOS) +taler.net bank.demo.taler.net exchange.demo.taler.net backend.demo.taler.net KUDOS 0 1 en de-CH demo.taler.net +demo.taler.net bank.demo.taler.net exchange.demo.taler.net backend.demo.taler.net KUDOS 0 1 en de-CH -# Public test stack (TESTKUDOS) — no GOA-style landings -test.taler.net bank.test.taler.net exchange.test.taler.net backend.test.taler.net TESTKUDOS 0 0 +# Public test stack (TESTKUDOS) +test.taler.net bank.test.taler.net exchange.test.taler.net backend.test.taler.net TESTKUDOS 0 0 en de-CH -# TOPS CHF — multi-tenant merchant my.taler-ops.ch; no public GOA landings -taler-ops.ch bank.taler-ops.ch exchange.taler-ops.ch my.taler-ops.ch CHF 0 0 -my.taler-ops.ch bank.taler-ops.ch exchange.taler-ops.ch my.taler-ops.ch CHF 0 0 taler-ops.ch -stage.taler-ops.ch bank.stage.taler-ops.ch exchange.stage.taler-ops.ch my.stage.taler-ops.ch CHF 0 0 -my.stage.taler-ops.ch bank.stage.taler-ops.ch exchange.stage.taler-ops.ch my.stage.taler-ops.ch CHF 0 0 stage.taler-ops.ch +# TOPS CHF +taler-ops.ch bank.taler-ops.ch exchange.taler-ops.ch my.taler-ops.ch CHF 0 0 en de-CH +my.taler-ops.ch bank.taler-ops.ch exchange.taler-ops.ch my.taler-ops.ch CHF 0 0 en de-CH taler-ops.ch +stage.taler-ops.ch bank.stage.taler-ops.ch exchange.stage.taler-ops.ch stage.my.taler-ops.ch CHF 0 0 en de-CH +stage.my.taler-ops.ch bank.stage.taler-ops.ch exchange.stage.taler-ops.ch stage.my.taler-ops.ch CHF 0 0 en de-CH stage.taler-ops.ch +my.stage.taler-ops.ch bank.stage.taler-ops.ch exchange.stage.taler-ops.ch stage.my.taler-ops.ch CHF 0 0 en de-CH stage.taler-ops.ch -# Franc Paysan (Infomaniak) — merchant host is monnaie.* (not taler./backend.) -# Prod: no monetary launch yet; stage uses TESTPAYSAN (public HTTPS landings + shops) -lefrancpaysan.ch bank.lefrancpaysan.ch exchange.lefrancpaysan.ch monnaie.lefrancpaysan.ch - 0 0 -monnaie.lefrancpaysan.ch bank.lefrancpaysan.ch exchange.lefrancpaysan.ch monnaie.lefrancpaysan.ch - 0 0 lefrancpaysan.ch -bank.lefrancpaysan.ch bank.lefrancpaysan.ch exchange.lefrancpaysan.ch monnaie.lefrancpaysan.ch - 0 0 lefrancpaysan.ch -exchange.lefrancpaysan.ch bank.lefrancpaysan.ch exchange.lefrancpaysan.ch monnaie.lefrancpaysan.ch - 0 0 lefrancpaysan.ch +# Franc Paysan — UI fr · regional locale fr-CH (Suisse romande) +lefrancpaysan.ch bank.lefrancpaysan.ch exchange.lefrancpaysan.ch monnaie.lefrancpaysan.ch - 0 0 fr fr-CH +monnaie.lefrancpaysan.ch bank.lefrancpaysan.ch exchange.lefrancpaysan.ch monnaie.lefrancpaysan.ch - 0 0 fr fr-CH lefrancpaysan.ch +bank.lefrancpaysan.ch bank.lefrancpaysan.ch exchange.lefrancpaysan.ch monnaie.lefrancpaysan.ch - 0 0 fr fr-CH lefrancpaysan.ch +exchange.lefrancpaysan.ch bank.lefrancpaysan.ch exchange.lefrancpaysan.ch monnaie.lefrancpaysan.ch - 0 0 fr fr-CH lefrancpaysan.ch -# Stage TESTPAYSAN — landings on / + farmer shops under stage.monnaie…/shops/ -# name bank exchange merchant currency local landing [canonical] -stage.lefrancpaysan.ch stage.bank.lefrancpaysan.ch stage.exchange.lefrancpaysan.ch stage.monnaie.lefrancpaysan.ch TESTPAYSAN 0 1 -stage.bank.lefrancpaysan.ch stage.bank.lefrancpaysan.ch stage.exchange.lefrancpaysan.ch stage.monnaie.lefrancpaysan.ch TESTPAYSAN 0 1 stage.lefrancpaysan.ch -stage.exchange.lefrancpaysan.ch stage.bank.lefrancpaysan.ch stage.exchange.lefrancpaysan.ch stage.monnaie.lefrancpaysan.ch TESTPAYSAN 0 1 stage.lefrancpaysan.ch -stage.monnaie.lefrancpaysan.ch stage.bank.lefrancpaysan.ch stage.exchange.lefrancpaysan.ch stage.monnaie.lefrancpaysan.ch TESTPAYSAN 0 1 stage.lefrancpaysan.ch +# Stage TESTPAYSAN — UI fr · locale fr-CH +stage.lefrancpaysan.ch stage.bank.lefrancpaysan.ch stage.exchange.lefrancpaysan.ch stage.monnaie.lefrancpaysan.ch TESTPAYSAN 0 1 fr fr-CH +stage.bank.lefrancpaysan.ch stage.bank.lefrancpaysan.ch stage.exchange.lefrancpaysan.ch stage.monnaie.lefrancpaysan.ch TESTPAYSAN 0 1 fr fr-CH stage.lefrancpaysan.ch +stage.exchange.lefrancpaysan.ch stage.bank.lefrancpaysan.ch stage.exchange.lefrancpaysan.ch stage.monnaie.lefrancpaysan.ch TESTPAYSAN 0 1 fr fr-CH stage.lefrancpaysan.ch +stage.monnaie.lefrancpaysan.ch stage.bank.lefrancpaysan.ch stage.exchange.lefrancpaysan.ch stage.monnaie.lefrancpaysan.ch TESTPAYSAN 0 1 fr fr-CH stage.lefrancpaysan.ch diff --git a/host-agent/README.md b/host-agent/README.md index 0664f28..cf8969c 100644 --- a/host-agent/README.md +++ b/host-agent/README.md @@ -4,22 +4,42 @@ | Behaviour | Default | |-----------|---------| -| Suite refresh | `update-suite.sh` → `origin/main` | +| Suite refresh | **`update-suite.sh` every run** → `git fetch --tags` + hard reset to `origin/main` | +| Update strict | `SUITE_UPDATE_STRICT=1` (default): **abort** if fetch/reset fails — no silent old tree | | Wall clock | `RUN_TIMEOUT=600` | | Log | line-buffered `run-*.log` (continuous write) | -| HTML | always (first run / empty tree too) + commit URL | +| HTML | always (first run / empty tree too) + commit URL; **bootstrap install stubs are monpages ERROR** | | Timeout notice | end of log + top jump on `/monitoring_err/` | +**Rule:** public mon pages must show a **current suite tag/commit**, not install bootstrap or months-old trees. Timers exist so this keeps working without manual SSH. + Thin wrappers only set stack defaults: | Wrapper | Stack | Typical user | |---------|-------|----------------| | `run-hacktivism-monitoring.sh` | GOA `hacktivism.ch` | `hernani` @ koopa | | `run-fp-stage-monitoring.sh` | FP stage | `stagepaysan` | -| `run-fp-prod-monitoring.sh` | FP prod | `francpaysan` | -| `run-aptdeploy.sh` | CLI: ensure + **aptdeploy** (no HTML) | local if hostname=koopa, else **ssh koopa-external** | -| `run-aptdeploy-monitoring.sh` | **aptdeploy** → `/taler-monitoring-aptdeploy*` | `hernani` @ koopa (4h timer, taler only) | -| `run-surface-monitoring.sh` | **surface** → `/taler-monitoring-surface*` | `hernani` @ koopa (hourly, taler only) | +| `run-host-report.sh` + `taler-monitoring-mytops-stage.{service,timer}` | mytops **stage / betel** only | `taler-monitoring` @ betel (4h) | +| `run-aptdeploy.sh` | CLI: ensure + **aptdeploy** (no HTML) | local if hostname=koopa, else **ssh $DEPLOY_SSH** | +| `run-aptdeploy-monitoring.sh` | apt-src tests → `/taler-monitoring-aptdeploy*` | `hernani` @ koopa (4h) | +| `run-surface-monitoring.sh` | **surface** → only `/taler-monitoring-surface*` (ecosystem + mail + mattermost + versions) | `hernani` @ koopa (hourly) | +| `run-max-ladder-monitoring.sh` | GOA **max-search** ladder → `/monitoring-max-ladder/` on bank/exchange/taler | `hernani` @ koopa (**daily**) | + +Deprecated mail/mattermost wrappers: **`../meta/deprecated/`** (not installed). No classic **`/monitoring-ladder`** page. + +### GOA max-search ladder page (daily) + +| URL | Phase | Timer | +|-----|-------|-------| +| `https://{bank,exchange,taler}.hacktivism.ch/monitoring-max-ladder/` | `max-ladder` | `taler-monitoring-max-ladder.timer` (~06:30 + jitter) | + +```bash +systemctl --user enable --now taler-monitoring-max-ladder.timer +# first round now: +systemctl --user start taler-monitoring-max-ladder.service +``` + +Needs explorer secrets (`SECRETS_ROOT` / bank-explorer password). Wall ~3h (`RUN_TIMEOUT=10800`). ### apt-src deploy tests (hacktivism `/monitoring`) @@ -31,7 +51,7 @@ Thin wrappers only set stack defaults: | `koopa-taler-deploy-test-apt-src-trixie-testing-upgrade` | **upgrade** for testing suite | ```bash -# from laptop / clementine → triggers on koopa-external +# from ops workstation → triggers on koopa-external ./host-agent/run-aptdeploy.sh ./host-agent/run-aptdeploy.sh upgrade ./host-agent/run-aptdeploy.sh fresh @@ -44,27 +64,21 @@ Thin wrappers only set stack defaults: Checks: package table, `httpd --version`, libtalerutil/`ldd` on failure, `systemctl` for `taler-merchant.target`, basic local `/config` probe. No nginx/HTTPS in containers. -### surface + aptdeploy HTML (taler.hacktivism.ch only) +### Public HTML layout (v1.8.0 simplified) -| URL | Role | Timer | -|-----|------|-------| -| `…/taler-monitoring-surface/` | ecosystem surface scan | 1h | -| `…/taler-monitoring-surface_err/` | errors + jump list | ″ | -| `…/taler-monitoring-aptdeploy/` | apt-src container deploy tests | 4h | -| `…/taler-monitoring-aptdeploy_err/` | errors + jump list | ″ | +| URL | Role | +|-----|------| +| `https://{bank,exchange,merchant}/monitoring/` (+ `_err`) | **Landing-stack** report — GOA 3 + FP **stage** 3 (+ mytops stage as configured) | +| `https://taler.hacktivism.ch/taler-monitoring-surface/` (+ `_err`) | **Only** ecosystem page: software/versions in the Taler universe (nmap OS, mail firefly+anastasis, mattermost, …) | -Not on bank/exchange. Staging: -`~/monitoring-sites-staging/taler.hacktivism.ch/taler-monitoring-{surface,aptdeploy}*`. +No separate public pages for mail/mattermost. Staging under `~/monitoring-sites-staging/`. ```bash systemctl --user enable --now taler-monitoring-surface.timer -systemctl --user enable --now taler-monitoring-aptdeploy.timer systemctl --user start taler-monitoring-surface.service -systemctl --user start taler-monitoring-aptdeploy.service -# Caddy: handles only in taler.hacktivism.ch block (root) ``` -Env examples: **`env/*.env.example`**. FP docs: `francpaysan-admin-log/docs/taler-monitoring-host-agents.md`. +Env examples: **`env/*.env.example`**. FP stage: `run-fp-stage-monitoring.sh` (same `run-host-report.sh` pipeline). ## GOA (koopa) — host-podman @@ -74,7 +88,7 @@ Runs **as user `hernani`** (systemd user, linger). **Primary** mode: `podman exec` on this host into `taler-hacktivism-bank` / `taler-hacktivism-exchange-ansible` / `taler-hacktivism` (IDs `inside.host-*`, not `inside.ssh-*`). - From a **laptop**, leave `INSIDE_PODMAN` off → access mode **`ssh`** via `KOOPA_SSH` + From an **ops workstation**, leave `INSIDE_PODMAN` off → access mode **`ssh`** via `KOOPA_SSH` (`inside.ssh-*`). - Public **`urls` + `versions`** for the stack (`-d hacktivism.ch`). - Writes console HTML for all three sites under @@ -89,7 +103,7 @@ Caddy (host root) can serve that tree at `/monitoring` — you configure Caddy. Every host-agent / aptdeploy run: 1. **`update-suite.sh`**: `git fetch` + **`SUITE_UPDATE_MODE=reset`** (default) → `origin/main` - into `SUITE_DIR` (prefer `~/src/koopa-admin-log`). + into `SUITE_DIR` (prefer `~/src/taler-monitoring`). 2. **Re-exec** host-agent scripts from that tree so the running code matches `COMMIT_URL`. 3. Local overrides only in **`~/.config/taler-monitoring/env`** — never overwritten by git. 4. HTML footer = exact commit: @@ -109,8 +123,8 @@ mkdir -p ~/.config/taler-monitoring cp ~/src/taler-monitoring/host-agent/env.example \ ~/.config/taler-monitoring/env -# laptop: -./host-agent/install-host-agent.sh --remote koopa-external +# ops workstation: +./host-agent/install-host-agent.sh --remote $DEPLOY_SSH # or on koopa: ~/src/taler-monitoring/host-agent/install-host-agent.sh @@ -120,7 +134,7 @@ Units: | Unit | Role | |------|------| -| `taler-monitoring-hacktivism.path` | fires on software stamp + landing/caddy config changes | +| `taler-monitoring-hacktivism.path` | fires on software stamp (`touch-software-stamp.sh`) | | `taler-monitoring-hacktivism.timer` | every **4h** fallback | | `taler-monitoring-hacktivism.service` | oneshot → `/monitoring*` (urls inside versions) | | `taler-monitoring-surface.timer` | every **1h** | @@ -144,16 +158,16 @@ journalctl --user -u taler-monitoring-hacktivism.service -n 50 --no-pager ls ~/monitoring-sites-staging/bank.hacktivism.ch/monitoring_err/ ``` -## Relation to firecuda outside-only timer +## Relation to optional outside-only timer -| | host-agent (koopa) | firecuda site-gen | +| | host-agent (koopa) | outside-runner site-gen | |--|--------------------|-------------------| | Access | **podman on host** | public HTTPS only | | Phases | urls **inside** versions | urls versions | | Trigger | software path + 4h | *could* launchd 4h | | Primary for GOA | **yes** | **off** (optional only) | -**firecuda-external is not scheduled** right now (launchd disabled). Scripts under +**outside-runner is not scheduled** right now (launchd disabled). Scripts under `site-gen/` can still one-shot a public run if needed; see `site-gen/README.md`. No secrets required for default phases. `secrets.env` optional for e2e later. @@ -181,3 +195,14 @@ zypper in qrencode zbar Without them, monitoring still runs; QR group WARNs and skips encode/decode. Caddy `/var/www/monitoring-sites` is separate (you configured hernani write access). + + +## Exit codes (v1.7.1+) + +Host-agent jobs default to **`STRICT_EXIT=1`**: non-zero exit if suite checks fail, `DEPLOY_WWW_ROOT` is not writable, or public monpages (FQDN) fail. + +Set `STRICT_EXIT=0` only if you want timers to stay green while still logging errors (not recommended for production). + +## Monitoring pages (suite) + +Public sticky-bar HTML under `/monitoring*` and `/taler-monitoring-*` is part of **taler-monitoring** (not a separate product). See main [README — Monitoring pages](../README.md#monitoring-pages-part-of-the-suite). Phase **monpages** is obligatory (**ERROR** if missing): GOA checks the full suite inventory; FP only FP mon hosts. diff --git a/host-agent/ROOT-APPLY-MONITORING.md b/host-agent/ROOT-APPLY-MONITORING.md index 17dd386..7d38b35 100644 --- a/host-agent/ROOT-APPLY-MONITORING.md +++ b/host-agent/ROOT-APPLY-MONITORING.md @@ -1,5 +1,11 @@ # Root: make monitoring pages live on koopa (GOA) +## Detection (v1.3.1+) + +Suite phase **`monpages`** and the host-agent **post-check** curl each public +FQDN (`https://taler.hacktivism.ch/monitoring/` …). Merchant JSON **code 21** +or non-HTML is an **ERROR** until this apply is done. + ## Symptom if not applied Public URLs return **merchant/bank JSON** like: @@ -11,14 +17,22 @@ Public URLs return **merchant/bank JSON** like: or a **404 HTML** page from the app reverse_proxy. Cause: live `/etc/caddy/Caddyfile` has **no** `handle /monitoring*` (falls through to -API). Prepared config is only under **`/home/hernani/koopa-caddy/Caddyfile`**. +API). Prepared config is only under **`$KOOPA_CADDY_DIR/Caddyfile`**. ## One-shot as root (recommended) ```bash -sudo /home/hernani/koopa-caddy/apply-monitoring-live.sh +# absolute path required: as root, ~ is /root (not /home/hernani) +sudo $APPLY_MONITORING_LIVE +# or already root: +# $APPLY_MONITORING_LIVE ``` +**Caddy bare-path redir (v1.9.2+):** inside a `handle` block use +`redir * /path/ 302` — never `redir /path/ 302` alone (Caddy treats the first +token as matcher and sets `Location: 302`, so bare URLs fall through to the +merchant API as JSON **code 21**). + That script: 1. `rsync` staging HTML → `/var/www/monitoring-sites/` (incl. surface + aptdeploy) @@ -26,23 +40,29 @@ That script: 3. Installs prepared Caddyfile → `/etc/caddy/Caddyfile`, validates, reloads Caddy 4. Smoke-curls all monitoring URLs (must **not** be JSON `code:21`) -## Paths +## Paths (only three families — suite name ≠ URL) -| Path | Host | Source timer | -|------|------|----------------| -| `/monitoring/` + `/monitoring_err/` | bank + exchange + taler | `taler-monitoring-hacktivism.timer` (4h) | -| `/taler-monitoring-surface/` + `/_err/` | **taler.hacktivism.ch only** | `taler-monitoring-surface.timer` (1h) | -| `/taler-monitoring-aptdeploy/` + `/_err/` | **taler.hacktivism.ch only** | `taler-monitoring-aptdeploy.timer` (4h) | +There is **no** public page `/taler-monitoring`. Each family needs **bare and slash** +(`redir * /path/ 302` inside Caddy `handle`): + +| Family | Paths | Host | Source timer | +|--------|-------|------|----------------| +| **monitoring** | `/monitoring` · `/monitoring/` · `/monitoring_err` · `/monitoring_err/` | bank + exchange + taler (and other landings) | `taler-monitoring-hacktivism.timer` (4h) | +| **monitoring-max-ladder** | `/monitoring-max-ladder` · `/` · `_err` | bank + exchange + taler (**GOA** max-search only) | `taler-monitoring-max-ladder.timer` (**daily**) | +| **taler-monitoring-surface** | bare + slash + `_err` | **taler.hacktivism.ch only** | `taler-monitoring-surface.timer` (1h) | +| **taler-monitoring-aptdeploy** | bare + slash + `_err` | **taler.hacktivism.ch only** | `taler-monitoring-aptdeploy.timer` (4h) | + +No classic **`/monitoring-ladder`**. Max-ladder served via Caddy `handle /monitoring*` under `/var/www/monitoring-sites/{host}/`. HTML is generated as **hernani** into -`/home/hernani/monitoring-sites-staging/`. +`$DEPLOY_STAGING (or $HOME/monitoring-sites-staging)/`. ## Manual equivalent ```bash -rsync -a --delete /home/hernani/monitoring-sites-staging/ /var/www/monitoring-sites/ +rsync -a --delete $DEPLOY_STAGING (or $HOME/monitoring-sites-staging)/ /var/www/monitoring-sites/ chown -R hernani:caddy /var/www/monitoring-sites && chmod -R g+rwX /var/www/monitoring-sites -sudo /home/hernani/koopa-caddy/apply.sh # or apply-monitoring-live.sh +sudo $KOOPA_CADDY_DIR/apply.sh # or apply-monitoring-live.sh ``` Expect **200** (HTML with sticky bar). Not merchant JSON `code:21`. @@ -50,7 +70,7 @@ Expect **200** (HTML with sticky bar). Not merchant JSON `code:21`. ## After each monitoring run (if www is not hernani-writable) ```bash -rsync -a --delete /home/hernani/monitoring-sites-staging/ /var/www/monitoring-sites/ +rsync -a --delete $DEPLOY_STAGING (or $HOME/monitoring-sites-staging)/ /var/www/monitoring-sites/ chown -R caddy:caddy /var/www/monitoring-sites ``` diff --git a/host-agent/apply-monitoring-live.sh b/host-agent/apply-monitoring-live.sh new file mode 100755 index 0000000..dfc4798 --- /dev/null +++ b/host-agent/apply-monitoring-live.sh @@ -0,0 +1,94 @@ +#!/usr/bin/env bash +# ONE-SHOT as root: publish monitoring HTML + enable Caddy handles +# +# Suite copy (taler-monitoring host-agent). Live install path on koopa: +# ${KOOPA_CADDY_DIR}/apply-monitoring-live.sh +# Prefer absolute path ( ~ expands to /root when already root ): +# sudo ${KOOPA_CADDY_DIR}/apply-monitoring-live.sh +# ${KOOPA_CADDY_DIR}/apply-monitoring-live.sh # if already root +# +# After editing here: rsync to koopa ~/koopa-caddy/ before root apply. +set -euo pipefail +if [[ "$(id -u)" -ne 0 ]]; then + exec sudo -E "$0" "$@" +fi + +STAGING="${DEPLOY_STAGING:-${HTML_OUT:-$HOME/monitoring-sites-staging}}" +WWW="${DEPLOY_WWW_ROOT:?set DEPLOY_WWW_ROOT in env}" +SRC="${KOOPA_CADDY_DIR:?set KOOPA_CADDY_DIR in env}/Caddyfile" +DST=/etc/caddy/Caddyfile + +if [[ ! -f "$SRC" ]]; then + echo "ERROR: missing Caddyfile source: $SRC" >&2 + exit 1 +fi +if [[ ! -d "$STAGING" ]]; then + echo "ERROR: missing staging HTML: $STAGING" >&2 + exit 1 +fi + +echo "== 1) HTML staging → www ==" +mkdir -p "$WWW" +rsync -a --delete "$STAGING/" "$WWW/" +# allow hernani to update future runs without root +chown -R hernani:caddy "$WWW" +chmod -R g+rwX "$WWW" +find "$WWW" -type d -exec chmod g+s {} \; +# ACL default if available +if command -v setfacl >/dev/null 2>&1; then + setfacl -R -m u:hernani:rwx,g:caddy:rwx "$WWW" || true + setfacl -R -d -m u:hernani:rwx,g:caddy:rwx "$WWW" || true +fi +ls -la "$WWW" "$WWW/taler.hacktivism.ch" || true + +echo "== 2) Caddyfile (backup + install) ==" +ts=$(date +%Y%m%d-%H%M%S) +cp -a "$DST" "/etc/caddy/Caddyfile.bak-monitoring-${ts}" +cp -a "$SRC" "$DST" +chown root:caddy "$DST" 2>/dev/null || chown root:root "$DST" +chmod 644 "$DST" +caddy validate --config "$DST" +systemctl reload caddy +systemctl is-active caddy + +echo "== 3) smoke ==" +# bash: every continued line except the last must end with \ +urls=( + https://taler.hacktivism.ch/monitoring/ + https://bank.hacktivism.ch/monitoring/ + https://exchange.hacktivism.ch/monitoring/ + https://taler.hacktivism.ch/monitoring-max-ladder/ + https://bank.hacktivism.ch/monitoring-max-ladder/ + https://exchange.hacktivism.ch/monitoring-max-ladder/ + https://taler.hacktivism.ch/taler-monitoring-surface + https://taler.hacktivism.ch/taler-monitoring-surface/ + https://taler.hacktivism.ch/taler-monitoring-surface_err/ + https://taler.hacktivism.ch/taler-monitoring-aptdeploy/ + https://taler.hacktivism.ch/taler-monitoring-aptdeploy_err/ +) +smoke_fail=0 +for url in "${urls[@]}"; do + code=$(curl -sS -o /tmp/monchk -w '%{http_code}' -L --max-redirs 3 -m 12 "$url" || echo ERR) + ct=$(file -b /tmp/monchk 2>/dev/null | head -c 40) + echo " $code $url ($ct)" + if grep -qE '"code":[[:space:]]*21' /tmp/monchk 2>/dev/null; then + echo " STILL merchant JSON code 21 — Caddy handles not active?" + smoke_fail=1 + fi +done +# bare path must redirect (or land on HTML), not merchant JSON +bare_code=$(curl -sS -o /tmp/monbare -w '%{http_code}' --max-redirs 0 -m 12 \ + https://taler.hacktivism.ch/taler-monitoring-surface || echo ERR) +if grep -qE '"code":[[:space:]]*21' /tmp/monbare 2>/dev/null; then + echo "ERROR: bare /taler-monitoring-surface still merchant code 21 (HTTP $bare_code)" >&2 + smoke_fail=1 +elif [[ "$bare_code" != "302" && "$bare_code" != "301" && "$bare_code" != "200" ]]; then + echo "WARN: bare /taler-monitoring-surface HTTP $bare_code (want 302/301/200 HTML)" >&2 +fi +echo " bare_no_follow=$bare_code https://taler.hacktivism.ch/taler-monitoring-surface" + +if [[ "$smoke_fail" -ne 0 ]]; then + echo "ERROR: smoke failed" >&2 + exit 1 +fi +echo "OK apply-monitoring-live done" diff --git a/host-agent/ensure-apt-deploy-test-containers.sh b/host-agent/ensure-apt-deploy-test-containers.sh index 75fa3e4..5abd9c5 100755 --- a/host-agent/ensure-apt-deploy-test-containers.sh +++ b/host-agent/ensure-apt-deploy-test-containers.sh @@ -11,7 +11,7 @@ # koopa-taler-deploy-test-apt-src-trixie-upgrade # koopa-taler-deploy-test-apt-src-trixie-testing-upgrade # -# Run as hernani on koopa. From elsewhere use run-aptdeploy.sh (ssh koopa-external). +# Run as hernani on koopa. From elsewhere use run-aptdeploy.sh (ssh $DEPLOY_SSH or $APT_DEPLOY_SSH). # set -euo pipefail @@ -72,13 +72,14 @@ cleanup_legacy() { done < <("$PODMAN_BIN" images --format '{{.Repository}}:{{.Tag}}' 2>/dev/null | grep -E 'localhost/koopa-taler-.*-test-apt' || true) } -# Prefer single suite tree: if both ~/src/koopa-admin-log and ~/koopa-admin-log are -# full git clones, keep the newer (by HEAD commit time) and replace the older with a symlink. +# Prefer single suite tree: ~/src/taler-monitoring vs ~/taler-monitoring +# (legacy koopa-admin-log paths are not used). Keep the newer git clone; symlink the other. dedupe_suite_repos() { - local a="$HOME/src/koopa-admin-log" b="$HOME/koopa-admin-log" + local a="$HOME/src/taler-monitoring" b="$HOME/taler-monitoring" local ta tb [ "${APT_DEPLOY_DEDUPE_REPOS:-1}" = "1" ] || return 0 - if [ -d "$a/.git" ] && [ -d "$b/.git" ] && [ ! -L "$a" ] && [ ! -L "$b" ]; then + if [ -d "$a/.git" ] && [ -d "$b/.git" ] && [ ! -L "$a" ] && [ ! -L "$b" ] \ + && [ "$(readlink -f "$a" 2>/dev/null || echo "$a")" != "$(readlink -f "$b" 2>/dev/null || echo "$b")" ]; then ta=$(git -C "$a" log -1 --format=%ct 2>/dev/null || echo 0) tb=$(git -C "$b" log -1 --format=%ct 2>/dev/null || echo 0) echo "======== dedupe suite repos (both are git clones) ========" @@ -94,17 +95,11 @@ dedupe_suite_repos() { mkdir -p "$(dirname "$a")" ln -sfn "$b" "$a" fi - elif [ -d "$a/scripts/taler-monitoring" ] && [ -d "$b/scripts/taler-monitoring" ] \ - && [ ! -d "$a/.git" ] && [ -d "$b/.git" ] && [ ! -L "$a" ]; then - echo "remove non-git tree $a · symlink → $b" - rm -rf "$a" + elif [ -d "$a/.git" ] && [ ! -e "$b" ]; then + ln -sfn "$a" "$b" + elif [ -d "$b/.git" ] && [ ! -e "$a" ]; then mkdir -p "$(dirname "$a")" ln -sfn "$b" "$a" - elif [ -d "$b/scripts/taler-monitoring" ] && [ -d "$a/scripts/taler-monitoring" ] \ - && [ ! -d "$b/.git" ] && [ -d "$a/.git" ] && [ ! -L "$b" ]; then - echo "remove non-git tree $b · symlink → $a" - rm -rf "$b" - ln -sfn "$a" "$b" fi } diff --git a/host-agent/env.example b/host-agent/env.example index 2826f83..0915d39 100644 --- a/host-agent/env.example +++ b/host-agent/env.example @@ -4,7 +4,6 @@ # Full stack examples (same reporting generation for all): # env/hacktivism.env.example # env/stagepaysan.env.example -# env/francpaysan.env.example # # SUITE_GIT_URL=https://git.hacktivism.ch/hernani/taler-monitoring.git # SUITE_GIT_REF=main diff --git a/host-agent/env/francpaysan.env.example b/host-agent/env/francpaysan.env.example deleted file mode 100644 index 2f5019d..0000000 --- a/host-agent/env/francpaysan.env.example +++ /dev/null @@ -1,27 +0,0 @@ -# Copy to: ~/.config/taler-monitoring/env (user francpaysan) -# Never inside the git tree — update-suite never overwrites this file. -# -# Shared reporting generation (run-host-report.sh / run-fp-prod-monitoring.sh): -# RUN_TIMEOUT, line-buffered log, always HTML, commit link on git.hacktivism.ch - -TALER_DOMAIN=lefrancpaysan.ch -INSIDE_PODMAN=1 -INSIDE_MODE=local-podman -LOCAL_STACK=0 -CONTINUE_ON_ERROR=1 -PHASES="urls inside versions" -RUN_TIMEOUT=600 - -HTML_OUT=$HOME/monitoring-sites-staging -MON_HOSTS="bank.lefrancpaysan.ch exchange.lefrancpaysan.ch monnaie.lefrancpaysan.ch" - -SUITE_DIR=$HOME/src/taler-monitoring -SUITE_GIT_URL=https://git.hacktivism.ch/hernani/taler-monitoring.git -SUITE_GIT_REF=main -SUITE_UPDATE_MODE=reset -SOURCE_REPO_WEB=https://git.hacktivism.ch/hernani/taler-monitoring - -# DEPLOY_WWW_ROOT=/var/www/monitoring-sites - -# UI language for console + HTML sticky bar (en|fr) -TALER_MON_LANG=fr diff --git a/host-agent/env/mytops-stage.env.example b/host-agent/env/mytops-stage.env.example new file mode 100644 index 0000000..0da4f19 --- /dev/null +++ b/host-agent/env/mytops-stage.env.example @@ -0,0 +1,41 @@ +# Copy to: ~/.config/taler-monitoring/env (user taler-monitoring @ betel) +# mytops CHF stage — NOT lifeline / production +# +SUITE_GIT_URL=https://git.hacktivism.ch/hernani/taler-monitoring.git +SUITE_GIT_REF=main +SUITE_DIR=$HOME/src/taler-monitoring +TALER_DOMAIN=stage.taler-ops.ch +MON_HOSTS="stage.my.taler-ops.ch stage.taler-ops.ch" +HTML_OUT=$HOME/monitoring-sites-staging +DEPLOY_WWW_ROOT=/var/www/monitoring-sites +HTML_URL_OK=/monitoring/ +HTML_URL_ERR=/monitoring_err/ +HTML_OK_DIR=monitoring +HTML_ERR_DIR=monitoring_err +# urls + monpages + fake-franken CHF (rusty.taler-ops.ch) +PHASES="urls monpages devtesting" +SKIP_SSH=1 +LOCAL_STACK=0 +INSIDE_PODMAN=0 +CONTINUE_ON_ERROR=1 +# ladder needs wall time (23× fake-incoming SSH) +RUN_TIMEOUT=1800 +STRICT_EXIT=1 +MONPAGES_REQUIRE_PUBLIC=1 +MONPAGES_INVENTORY=job +TALER_MON_LANG=en +AGENT_LABEL=mytops-stage-host-agent +STATE_NAME=taler-monitoring-mytops-stage +SUITE_UPDATE_STRICT=1 +CHECK_BANK=0 +MERCHANT_REQUIRED=1 +CHECK_LANDING=0 +# rusty devtesting (SSH Host taler-rusty-devtesting → devtesting@rusty.taler-ops.ch) +DEVTESTING_SSH=taler-rusty-devtesting +DEVTESTING_AMOUNT=CHF:1.00 +# CHF fake-incoming amount ladder (23 rungs → 50k) +DEVTESTING_LADDER=1 +DEVTESTING_LADDER_STEPS=23 +DEVTESTING_LADDER_MAX=50000 +DEVTESTING_LADDER_MIN=0.01 +# DEVTESTING_SKIP=1 diff --git a/host-agent/env/stagepaysan.env.example b/host-agent/env/stagepaysan.env.example index f1510f2..abfcdc3 100644 --- a/host-agent/env/stagepaysan.env.example +++ b/host-agent/env/stagepaysan.env.example @@ -1,8 +1,10 @@ # Copy to: ~/.config/taler-monitoring/env (user stagepaysan) # Never inside the git tree — update-suite never overwrites this file. # -# Shared reporting generation (run-host-report.sh / run-fp-stage-monitoring.sh): -# RUN_TIMEOUT, line-buffered log, always HTML, commit link on git.hacktivism.ch +# Secrets: use francpaysan-secrets layout (operator tree or host mirror): +# $HOME/francpaysan-secrets/stage/bank-admin-password.txt +# $HOME/francpaysan-secrets/stage/default-instance-token.txt +# On stage host, symlink those to /mnt/data/stagepaysan/*/secrets/… TALER_DOMAIN=stage.lefrancpaysan.ch INSIDE_PODMAN=1 @@ -10,21 +12,32 @@ INSIDE_MODE=local-podman INSIDE_PROFILE=stage-lfp LOCAL_STACK=0 CONTINUE_ON_ERROR=1 -PHASES="urls inside versions" -RUN_TIMEOUT=600 +PHASES="urls inside versions monpages e2e" +RUN_TIMEOUT=1800 +E2E_TIMEOUT=900 +E2E_USE_TEMPLATES=1 HTML_OUT=$HOME/monitoring-sites-staging MON_HOSTS="stage.bank.lefrancpaysan.ch stage.exchange.lefrancpaysan.ch stage.monnaie.lefrancpaysan.ch" +DEPLOY_WWW_ROOT=/var/www/monitoring-sites -# Git clone of koopa-admin-log (repo root, not …/scripts/taler-monitoring) SUITE_DIR=$HOME/src/taler-monitoring SUITE_GIT_URL=https://git.hacktivism.ch/hernani/taler-monitoring.git SUITE_GIT_REF=main SUITE_UPDATE_MODE=reset SOURCE_REPO_WEB=https://git.hacktivism.ch/hernani/taler-monitoring -# Optional public docroot if this user can write it -# DEPLOY_WWW_ROOT=/var/www/monitoring-sites +FRANCPAYSAN_SECRETS=$HOME/francpaysan-secrets +WALLET_CLI=$HOME/.local/lib/taler-wallet-cli/node_modules/taler-wallet-cli/bin/taler-wallet-cli.mjs +WALLET_CLI_CANDIDATES="$HOME/.local/lib/taler-wallet-cli/node_modules/taler-wallet-cli/bin/taler-wallet-cli.mjs $HOME/.local/bin/taler-wallet-cli.mjs" + +INSIDE_BANK_CTR=stage-lfp-bank +INSIDE_EXCHANGE_CTR=stage-lfp-exchange-ansible +INSIDE_MERCHANT_CTR=stage-lfp-merchant -# UI language for console + HTML sticky bar (en|fr) TALER_MON_LANG=fr +MONPAGES_REQUIRE_PUBLIC=1 +MONPAGES_INVENTORY=job +STRICT_EXIT=1 +AGENT_LABEL=fp-stage-host-agent +STATE_NAME=taler-monitoring-stage-lfp diff --git a/host-agent/install-host-agent.sh b/host-agent/install-host-agent.sh index 5b64cf2..118ef7f 100755 --- a/host-agent/install-host-agent.sh +++ b/host-agent/install-host-agent.sh @@ -4,18 +4,21 @@ # # on koopa as hernani: # ~/src/taler-monitoring/host-agent/install-host-agent.sh # # or from laptop: -# ./install-host-agent.sh --remote koopa-external +# ./install-host-agent.sh --remote $DEPLOY_SSH +# +# Suite tree is ONLY ~/src/taler-monitoring (standalone repo). No koopa-admin-log. # set -euo pipefail ROOT=$(cd "$(dirname "$0")" && pwd) REMOTE="${1:-}" -[ "${1:-}" = "--remote" ] && REMOTE="${2:-koopa-external}" +[ "${1:-}" = "--remote" ] && REMOTE="${2:-${DEPLOY_SSH:-}}" + +SUITE_URL="${SUITE_GIT_URL:-https://git.hacktivism.ch/hernani/taler-monitoring.git}" install_local() { local mon home_unit mon=$(cd "$ROOT/.." && pwd) - # Prefer ~/koopa-admin-log on live host - if [ -d "$HOME/src/taler-monitoring" ]; then + if [ -d "$HOME/src/taler-monitoring/.git" ] || [ -d "$HOME/src/taler-monitoring" ]; then mon="$HOME/src/taler-monitoring" fi home_unit="$HOME/.config/systemd/user" @@ -31,75 +34,95 @@ install_local() { echo "created ~/.config/taler-monitoring/env" fi - # Prefer a real git clone for update-suite.sh - if [ ! -d "$HOME/src/koopa-admin-log/.git" ] && [ ! -d "$HOME/koopa-admin-log/.git" ]; then - if [ -d "$HOME/src/taler-monitoring" ] && [ ! -d "$HOME/koopa-admin-log/.git" ]; then - echo "note: $HOME/koopa-admin-log has no .git — update-suite will clone to ~/src/koopa-admin-log" + # Ensure suite git clone at canonical path + if [ ! -d "$HOME/src/taler-monitoring/.git" ]; then + if [ -d "$mon/.git" ] && [ "$mon" != "$HOME/src/taler-monitoring" ]; then + mkdir -p "$HOME/src" + ln -sfn "$mon" "$HOME/src/taler-monitoring" + echo "symlink $HOME/src/taler-monitoring → $mon" + else + echo "clone $SUITE_URL → $HOME/src/taler-monitoring" + git clone "$SUITE_URL" "$HOME/src/taler-monitoring" \ + || echo "WARN: clone failed (auth/network) — using tree at $mon if present" fi - git clone https://git.hacktivism.ch/hernani/taler-monitoring.git "$HOME/src/koopa-admin-log" 2>/dev/null \ - || echo "WARN: clone failed (auth/network) — will use existing tree if any" fi + mon="$HOME/src/taler-monitoring" + # Convenience link (update-suite.sh also creates this) + ln -sfn "$mon" "$HOME/taler-monitoring" 2>/dev/null || true # Ensure suite is executable chmod +x "$mon"/taler-monitoring.sh "$mon"/check_*.sh 2>/dev/null || true chmod +x "$mon/host-agent"/*.sh 2>/dev/null || true chmod +x "$mon/site-gen"/*.sh "$mon/site-gen/console_to_html.py" 2>/dev/null || true - # Units reference %h/koopa-admin-log — prefer git clone or symlink - if [ -d "$HOME/src/koopa-admin-log/.git" ] && [ ! -e "$HOME/koopa-admin-log" ]; then - ln -sfn "$HOME/src/koopa-admin-log" "$HOME/koopa-admin-log" - echo "symlink $HOME/koopa-admin-log → $HOME/src/koopa-admin-log" - elif [ ! -e "$HOME/koopa-admin-log" ]; then - admin_root=$(cd "$mon/../.." && pwd) - ln -sfn "$admin_root" "$HOME/koopa-admin-log" - echo "symlink $HOME/koopa-admin-log → $admin_root" - fi - install -m 644 "$ROOT/taler-monitoring-hacktivism.service" "$home_unit/" install -m 644 "$ROOT/taler-monitoring-hacktivism.path" "$home_unit/" install -m 644 "$ROOT/taler-monitoring-hacktivism.timer" "$home_unit/" install -m 644 "$ROOT/taler-monitoring-surface.service" "$home_unit/" install -m 644 "$ROOT/taler-monitoring-surface.timer" "$home_unit/" + # optional aptdeploy (not a public mon page layout — v1.8.0 simplified overview) install -m 644 "$ROOT/taler-monitoring-aptdeploy.service" "$home_unit/" install -m 644 "$ROOT/taler-monitoring-aptdeploy.timer" "$home_unit/" - - # Fix ExecStart if install path differs (units use %h/koopa-admin-log) + # GOA max-search amount ladder page (daily only — no classic /monitoring-ladder) + install -m 644 "$ROOT/taler-monitoring-max-ladder.service" "$home_unit/" + install -m 644 "$ROOT/taler-monitoring-max-ladder.timer" "$home_unit/" + chmod +x "$ROOT/run-max-ladder-monitoring.sh" 2>/dev/null || true systemctl --user daemon-reload systemctl --user enable --now taler-monitoring-hacktivism.path systemctl --user enable --now taler-monitoring-hacktivism.timer systemctl --user enable --now taler-monitoring-surface.timer systemctl --user enable --now taler-monitoring-aptdeploy.timer + systemctl --user enable --now taler-monitoring-max-ladder.timer + # drop classic ladder mon page if previously installed + systemctl --user disable --now taler-monitoring-ladder.timer 2>/dev/null || true + systemctl --user stop taler-monitoring-ladder.service 2>/dev/null || true + rm -f "$home_unit/taler-monitoring-ladder.service" \ + "$home_unit/taler-monitoring-ladder.timer" 2>/dev/null || true + # Remove leftover mail/mattermost units if present (moved to meta/deprecated/) + systemctl --user disable --now taler-monitoring-mattermost.timer 2>/dev/null || true + systemctl --user disable --now taler-monitoring-mail.timer 2>/dev/null || true + rm -f "$home_unit/taler-monitoring-mattermost.service" \ + "$home_unit/taler-monitoring-mattermost.timer" \ + "$home_unit/taler-monitoring-mail.service" \ + "$home_unit/taler-monitoring-mail.timer" 2>/dev/null || true + systemctl --user daemon-reload 2>/dev/null || true # initial stamp so path exists "$ROOT/touch-software-stamp.sh" - systemctl --user start taler-monitoring-hacktivism.service || true - systemctl --user start taler-monitoring-surface.service || true - systemctl --user start taler-monitoring-aptdeploy.service || true + # oneshot jobs: --no-block so install does not wait hours + systemctl --user start --no-block taler-monitoring-hacktivism.service || true + systemctl --user start --no-block taler-monitoring-surface.service || true + systemctl --user start --no-block taler-monitoring-aptdeploy.service || true + # first max-ladder round immediately (daily thereafter) + systemctl --user start --no-block taler-monitoring-max-ladder.service || true echo "--- status ---" systemctl --user status taler-monitoring-hacktivism.path --no-pager -l | head -15 systemctl --user status taler-monitoring-hacktivism.timer --no-pager -l | head -12 systemctl --user status taler-monitoring-surface.timer --no-pager -l | head -12 systemctl --user status taler-monitoring-aptdeploy.timer --no-pager -l | head -12 + systemctl --user status taler-monitoring-max-ladder.timer --no-pager -l | head -12 systemctl --user list-timers --all | grep taler-monitoring || true - echo "OK host-agent (hacktivism 4h + surface 1h + aptdeploy 4h)" - echo "manual: systemctl --user start taler-monitoring-hacktivism.service" - echo "manual surface: systemctl --user start taler-monitoring-surface.service" - echo "manual aptdeploy: systemctl --user start taler-monitoring-aptdeploy.service" + echo "OK host-agent:" + echo " · hacktivism 4h → /monitoring on bank/exchange/taler (landing stacks)" + echo " · max-ladder daily → /monitoring-max-ladder/ (GOA max-search, bank/exchange/taler)" + echo " · surface 1h → /taler-monitoring-surface* (ecosystem versions/software)" + echo " · aptdeploy 4h (optional tests)" + echo " · no classic /monitoring-ladder page" + echo " · mail/mattermost: not installed (meta/deprecated/; content in surface)" + echo "suite: $mon" + echo "manual max-ladder: systemctl --user start taler-monitoring-max-ladder.service" echo "after upgrades: $ROOT/touch-software-stamp.sh" } if [ -n "$REMOTE" ] && [ "$REMOTE" != "--remote" ]; then echo "install on $REMOTE …" + # ROOT = …/taler-monitoring/host-agent → suite root is parent rsync -az --delete \ --exclude secrets.env \ --exclude 'android-test/artifacts' \ - "$ROOT/../../" \ + --exclude '.git' \ + "$ROOT/../" \ "${REMOTE}:src/taler-monitoring/" - # ensure full tree for %h/koopa-admin-log if missing configs etc. - ssh -o BatchMode=yes -o ConnectTimeout=20 "$REMOTE" \ - 'test -d $HOME/koopa-admin-log || git clone https://git.hacktivism.ch/hernani/taler-monitoring.git $HOME/koopa-admin-log 2>/dev/null || true' - rsync -az "$ROOT/" "${REMOTE}:src/taler-monitoring/host-agent/" - rsync -az "$ROOT/../site-gen/" "${REMOTE}:src/taler-monitoring/site-gen/" 2>/dev/null || true ssh -o BatchMode=yes "$REMOTE" \ 'bash $HOME/src/taler-monitoring/host-agent/install-host-agent.sh' else diff --git a/host-agent/run-aptdeploy-monitoring.sh b/host-agent/run-aptdeploy-monitoring.sh index fe81257..bd7c46d 100755 --- a/host-agent/run-aptdeploy-monitoring.sh +++ b/host-agent/run-aptdeploy-monitoring.sh @@ -20,10 +20,10 @@ export SKIP_SSH=0 export CONTINUE_ON_ERROR="${CONTINUE_ON_ERROR:-1}" # four containers + checks; allow long wall export RUN_TIMEOUT="${RUN_TIMEOUT:-1800}" -# dedicated job: always aptdeploy only -export PHASES=aptdeploy -# Only merchant public front — not bank/exchange -export MON_HOSTS="${MON_HOSTS:-taler.hacktivism.ch}" +# dedicated job: always aptdeploy only (force — do not inherit main-timer env) +export PHASES="aptdeploy monpages" +# Public HTML only on taler.hacktivism.ch (not bank/exchange landings) +export MON_HOSTS="taler.hacktivism.ch" export HTML_OUT="${HTML_OUT:-$HOME/monitoring-sites-staging}" export DEPLOY_WWW_ROOT="${DEPLOY_WWW_ROOT:-/var/www/monitoring-sites}" @@ -32,6 +32,8 @@ export HTML_ERR_DIR="taler-monitoring-aptdeploy_err" export HTML_URL_OK="/taler-monitoring-aptdeploy/" export HTML_URL_ERR="/taler-monitoring-aptdeploy_err/" export PAGE_LABEL="taler-monitoring-aptdeploy" +# monpages: only this job’s paths on taler.hacktivism.ch +export MONPAGES_INVENTORY="job" # Ensure all four apt-src containers (fresh rebuild + upgrade track) if [ "${APT_DEPLOY_ENSURE:-1}" = "1" ] \ diff --git a/host-agent/run-aptdeploy.sh b/host-agent/run-aptdeploy.sh index 3286379..8c1f3cd 100755 --- a/host-agent/run-aptdeploy.sh +++ b/host-agent/run-aptdeploy.sh @@ -14,7 +14,7 @@ set -euo pipefail AGENT_DIR=$(cd "$(dirname "$0")" && pwd) BOOT_MON=$(cd "$AGENT_DIR/.." && pwd) MODE="${1:-all}" -APT_DEPLOY_SSH="${APT_DEPLOY_SSH:-koopa-external}" +APT_DEPLOY_SSH="${APT_DEPLOY_SSH:-${DEPLOY_SSH:-}}" OVERLAY_DIR="${XDG_CONFIG_HOME:-$HOME/.config}/taler-monitoring/suite-overlay" on_koopa() { @@ -114,7 +114,7 @@ run_local() { run_remote() { echo "======== aptdeploy remote via ssh ${APT_DEPLOY_SSH} mode=$MODE ========" if [ "${APT_DEPLOY_RSYNC:-1}" = "1" ]; then - echo "rsync mon suite → ${APT_DEPLOY_SSH}:src/src/taler-monitoring/" + echo "rsync mon suite → ${APT_DEPLOY_SSH}:src/taler-monitoring/" rsync -az \ --exclude secrets.env \ --exclude 'android-test/artifacts' \ @@ -122,7 +122,7 @@ run_remote() { --exclude 'android-test/out*' \ --exclude '__pycache__' \ "$BOOT_MON/" \ - "${APT_DEPLOY_SSH}:src/src/taler-monitoring/" \ + "${APT_DEPLOY_SSH}:src/taler-monitoring/" \ || echo "WARN: rsync failed" >&2 fi ssh -o BatchMode=yes -o ConnectTimeout=25 "$APT_DEPLOY_SSH" bash -s -- "$MODE" <<'EOS' diff --git a/host-agent/run-fp-prod-monitoring.sh b/host-agent/run-fp-prod-monitoring.sh deleted file mode 100755 index be876a8..0000000 --- a/host-agent/run-fp-prod-monitoring.sh +++ /dev/null @@ -1,20 +0,0 @@ -#!/usr/bin/env bash -# FrancPaysan PROD host-agent (user francpaysan on francpaysan-host). -# Shared reporting: run-host-report.sh (timeout, line-buffered log, first-run HTML). -set -uo pipefail -AGENT_DIR=$(cd "$(dirname "$0")" && pwd) - -export AGENT_LABEL="${AGENT_LABEL:-fp-prod-host-agent}" -export STATE_NAME="${STATE_NAME:-taler-monitoring-lfp-prod}" -export TALER_DOMAIN="${TALER_DOMAIN:-lefrancpaysan.ch}" -export INSIDE_PODMAN="${INSIDE_PODMAN:-1}" -export INSIDE_MODE="${INSIDE_MODE:-local-podman}" -export LOCAL_STACK="${LOCAL_STACK:-0}" -export CONTINUE_ON_ERROR="${CONTINUE_ON_ERROR:-1}" -export RUN_TIMEOUT="${RUN_TIMEOUT:-600}" -export PHASES="${PHASES:-urls inside versions}" -export MON_HOSTS="${MON_HOSTS:-bank.lefrancpaysan.ch exchange.lefrancpaysan.ch monnaie.lefrancpaysan.ch}" -export HTML_OUT="${HTML_OUT:-$HOME/monitoring-sites-staging}" -export DEPLOY_WWW_ROOT="${DEPLOY_WWW_ROOT:-/var/www/monitoring-sites}" - -exec bash "$AGENT_DIR/run-host-report.sh" "$@" diff --git a/host-agent/run-fp-stage-monitoring.sh b/host-agent/run-fp-stage-monitoring.sh index 70518f0..e7f95e8 100755 --- a/host-agent/run-fp-stage-monitoring.sh +++ b/host-agent/run-fp-stage-monitoring.sh @@ -1,7 +1,11 @@ #!/usr/bin/env bash -# FrancPaysan STAGE host-agent (user stagepaysan on francpaysan-host). +# FrancPaysan STAGE host-agent (user stagepaysan via francpaysan-stage-user). # Shared reporting: run-host-report.sh (timeout, line-buffered log, first-run HTML). set -uo pipefail + +# Language: en|fr (selectable). Default for this wrapper: fr +export TALER_MON_LANG="${TALER_MON_LANG:-fr}" +export TALER_MON_LANG_SET=1 AGENT_DIR=$(cd "$(dirname "$0")" && pwd) export AGENT_LABEL="${AGENT_LABEL:-fp-stage-host-agent}" @@ -12,8 +16,10 @@ export INSIDE_MODE="${INSIDE_MODE:-local-podman}" export INSIDE_PROFILE="${INSIDE_PROFILE:-stage-lfp}" export LOCAL_STACK="${LOCAL_STACK:-0}" export CONTINUE_ON_ERROR="${CONTINUE_ON_ERROR:-1}" -export RUN_TIMEOUT="${RUN_TIMEOUT:-600}" -export PHASES="${PHASES:-urls inside versions}" +# monpages obligatory (ERROR): only FP stage mon hosts below — never GOA inventory +export MONPAGES_REQUIRE_PUBLIC="${MONPAGES_REQUIRE_PUBLIC:-1}" +export RUN_TIMEOUT="${RUN_TIMEOUT:-1800}" +export PHASES="${PHASES:-urls inside versions monpages e2e}" export MON_HOSTS="${MON_HOSTS:-stage.bank.lefrancpaysan.ch stage.exchange.lefrancpaysan.ch stage.monnaie.lefrancpaysan.ch}" export HTML_OUT="${HTML_OUT:-$HOME/monitoring-sites-staging}" export DEPLOY_WWW_ROOT="${DEPLOY_WWW_ROOT:-/var/www/monitoring-sites}" diff --git a/host-agent/run-hacktivism-monitoring.sh b/host-agent/run-hacktivism-monitoring.sh index 8fd98fd..7ccf7c7 100755 --- a/host-agent/run-hacktivism-monitoring.sh +++ b/host-agent/run-hacktivism-monitoring.sh @@ -2,6 +2,10 @@ # GOA / hacktivism host-agent on koopa (hernani). # Stack defaults only — reporting pipeline is shared: run-host-report.sh set -uo pipefail + +# Language: en|fr (selectable). Default for this wrapper: en +export TALER_MON_LANG="${TALER_MON_LANG:-en}" +export TALER_MON_LANG_SET=1 AGENT_DIR=$(cd "$(dirname "$0")" && pwd) export AGENT_LABEL="${AGENT_LABEL:-hacktivism-host-agent}" @@ -12,12 +16,18 @@ export INSIDE_MODE="${INSIDE_MODE:-local-podman}" export LOCAL_STACK="${LOCAL_STACK:-1}" export CONTINUE_ON_ERROR="${CONTINUE_ON_ERROR:-1}" export RUN_TIMEOUT="${RUN_TIMEOUT:-600}" -# stack checks only — aptdeploy has its own HTML path + timer: -# run-aptdeploy-monitoring.sh → /taler-monitoring-aptdeploy* (taler.hacktivism.ch) -export PHASES="${PHASES:-urls inside versions}" +# stack checks + full monpages inventory (landings + max-ladder + surface + aptdeploy). +# Specialized pages also have their own timers; full outside-in inventory lives here. +# run-max-ladder-monitoring.sh → /monitoring-max-ladder* (daily max-search, GOA) +# run-aptdeploy-monitoring.sh → monpages job-only for /taler-monitoring-aptdeploy* +# run-surface-monitoring.sh → monpages job-only for /taler-monitoring-surface* +# No classic /monitoring-ladder page. +export PHASES="${PHASES:-urls inside versions monpages}" export MON_HOSTS="${MON_HOSTS:-bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch}" export HTML_OUT="${HTML_OUT:-$HOME/monitoring-sites-staging}" export DEPLOY_WWW_ROOT="${DEPLOY_WWW_ROOT:-/var/www/monitoring-sites}" export APT_DEPLOY_ENSURE=0 +# full GOA mon catalog (not job-only) +export MONPAGES_INVENTORY="${MONPAGES_INVENTORY:-auto}" exec bash "$AGENT_DIR/run-host-report.sh" "$@" diff --git a/host-agent/run-host-report.sh b/host-agent/run-host-report.sh index 16857c1..1f75caf 100755 --- a/host-agent/run-host-report.sh +++ b/host-agent/run-host-report.sh @@ -1,13 +1,15 @@ #!/usr/bin/env bash # run-host-report.sh — global reporting generation for all stacks -# (GOA/hacktivism, FP stage, FP prod, …). +# (GOA/hacktivism, FP stage, mytops stage, …). # # Shared behaviour (always): # 1) update-suite → latest git.hacktivism.ch (local ~/.config never overwritten) # 2) line-buffered run log (written continuously) # 3) RUN_TIMEOUT wall clock for taler-monitoring.sh (default 600s) # 4) always write HTML (first run / empty staging too) + Forgejo commit link -# 5) optional rsync to DEPLOY_WWW_ROOT if writable +# 5) rsync to DEPLOY_WWW_ROOT (ERROR if not writable — v1.3.1) +# 6) monpages post-check: public FQDN must serve monitoring HTML +# 7) exit code: STRICT_EXIT default 1 (v1.7.1) — unit fails if publish/public pages fail # # Configure via env file (not in git): # ${XDG_CONFIG_HOME:-$HOME/.config}/taler-monitoring/env @@ -16,7 +18,6 @@ # Thin wrappers only set stack defaults then exec this script: # run-hacktivism-monitoring.sh # run-fp-stage-monitoring.sh -# run-fp-prod-monitoring.sh # set -uo pipefail @@ -37,6 +38,9 @@ _wrap_keys=( RUN_TIMEOUT STATE_NAME AGENT_LABEL MON_HOSTS TALER_DOMAIN DEPLOY_WWW_ROOT INSIDE_PODMAN INSIDE_MODE LOCAL_STACK SKIP_SSH CONTINUE_ON_ERROR AUTH401_CONTINUE APT_DEPLOY_ENSURE SURFACE_SCOPE TALER_DOMAIN_FROM_CLI STRICT_EXIT SOURCE_SUITE_PATH + MONPAGES_INVENTORY MONPAGES_CHECK_BARE MONPAGES_BARE_STRICT MONPAGES_REQUIRE_PUBLIC + CHECK_BANK MERCHANT_REQUIRED CHECK_LANDING + WALLET_CLI CLI_JS PATH LADDER_MODE LADDER_PAY LADDER_PAY_WAIT_AVAILABLE_S ) for _k in "${_wrap_keys[@]}"; do if [ -n "${!_k+x}" ]; then @@ -72,16 +76,19 @@ export CONTINUE_ON_ERROR="${CONTINUE_ON_ERROR:-1}" export AUTH401_CONTINUE="${AUTH401_CONTINUE:-${CONTINUE_ON_ERROR}}" export RUN_TIMEOUT="${RUN_TIMEOUT:-600}" export TALER_DOMAIN="${TALER_DOMAIN:-hacktivism.ch}" -# UI language: en default; fr for FrancPaysan (override with TALER_MON_LANG) -if [ -z "${TALER_MON_LANG:-}" ]; then - case "$TALER_DOMAIN" in - *lefrancpaysan*|*francpaysan*) export TALER_MON_LANG=fr ;; +# UI language: --lang / TALER_MON_LANG_SET=1 locks. +# Otherwise domains.conf column `lang` → TALER_DOMAIN_LANG → TALER_MON_LANG +# (applied in suite apply_taler_domain / i18n_init). Leave SET=0 so -d can re-apply. +# Bare workstation TALER_MON_LANG=en must not override profile lang=fr. +if [ "${TALER_MON_LANG_SET:-0}" = "1" ]; then + case "${TALER_MON_LANG:-en}" in + fr|FR|fra|french) export TALER_MON_LANG=fr ;; *) export TALER_MON_LANG=en ;; esac fi export TALER_MON_LANG -PHASES="${PHASES:-urls inside versions}" +PHASES="${PHASES:-urls inside versions monpages}" HTML_BASE="${HTML_OUT:-$HOME/monitoring-sites-staging}" MON_HOSTS="${MON_HOSTS:-}" SUITE_PATH="${SOURCE_SUITE_PATH:-.}" @@ -96,6 +103,28 @@ HTML_ERR_DIR="${HTML_ERR_DIR:-monitoring_err}" HTML_URL_OK="${HTML_URL_OK:-/monitoring/}" HTML_URL_ERR="${HTML_URL_ERR:-/monitoring_err/}" PAGE_LABEL="${PAGE_LABEL:-monitoring}" +SUITE_VERSION="${SUITE_VERSION:-}" +SUITE_VERSION_URL="${SUITE_VERSION_URL:-}" +if [ -z "$SUITE_VERSION" ] && [ -d "${ROOT:-$AGENT_MON}/.git" ]; then + _vd="${ROOT:-$AGENT_MON}" + SUITE_VERSION=$(git -C "$_vd" describe --tags --exact-match 2>/dev/null || git -C "$_vd" describe --tags --abbrev=0 2>/dev/null || true) +fi +if [ -z "$SUITE_VERSION" ] && [ -f "${ROOT:-$AGENT_MON}/VERSION" ]; then + SUITE_VERSION="v$(tr -d '[:space:]' <"${ROOT:-$AGENT_MON}/VERSION" | sed 's/^v//')" +fi +[ -z "$SUITE_VERSION" ] && SUITE_VERSION="unknown" +case "$SUITE_VERSION" in unknown|v*) ;; *) SUITE_VERSION="v$SUITE_VERSION" ;; esac +# Prefer commit URL over /src/tag/… (tags lag push → sticky version 404 on Forgejo). +if [ -z "$SUITE_VERSION_URL" ]; then + _web="${SOURCE_REPO_WEB:-https://git.hacktivism.ch/hernani/taler-monitoring}" + if [ -n "${COMMIT:-}" ] && [ "$COMMIT" != "unknown" ]; then + SUITE_VERSION_URL="${_web}/src/commit/${COMMIT}" + elif [ "$SUITE_VERSION" != "unknown" ]; then + SUITE_VERSION_URL="${_web}/src/tag/${SUITE_VERSION}" + fi +fi +export SUITE_VERSION SUITE_VERSION_URL + # Sensible MON_HOSTS from domain if not set if [ -z "$MON_HOSTS" ]; then @@ -119,7 +148,32 @@ mkdir -p "$LOG_DIR" "$HTML_BASE" STAMP=$(date +%Y%m%d-%H%M%S) LOG="$LOG_DIR/run-${STAMP}.log" +UPD_LOG="$LOG_DIR/suite-update-${STAMP}.log" : >"$LOG" + +# --- suite auto-upgrade BEFORE the HTML log (git noise must not pollute mon pages) --- +OVERLAY_DIR="${XDG_CONFIG_HOME:-$HOME/.config}/taler-monitoring/suite-overlay" +if [ -f "$AGENT_MON/taler-monitoring.sh" ]; then + mkdir -p "$OVERLAY_DIR" + rsync -a \ + --exclude secrets.env --exclude 'android-test/artifacts' --exclude 'android-test/apks' \ + --exclude 'android-test/out*' --exclude '__pycache__' \ + "$AGENT_MON/" "$OVERLAY_DIR/" 2>/dev/null || true +fi + +# shellcheck source=update-suite.sh +if ! source "$AGENT_DIR/update-suite.sh" >"$UPD_LOG" 2>&1; then + echo "ERROR: suite auto-update failed — refusing to run with stale tree" >&2 + echo " detail: $UPD_LOG" >&2 + tail -n 30 "$UPD_LOG" >&2 || true + echo " fix git/network or set SUITE_UPDATE_STRICT=0 only as emergency" >&2 + exit 1 +fi +if [ -d "${SUITE_DIR:-}/host-agent" ]; then + AGENT_DIR="${SUITE_DIR}/host-agent" +fi + +# Now open the public HTML log (clean agent header + suite summary only) if command -v stdbuf >/dev/null 2>&1; then exec > >(stdbuf -oL -eL tee -a "$LOG") 2>&1 else @@ -131,20 +185,9 @@ echo "env_file=$ENV_FILE" echo "domain=$TALER_DOMAIN · phases=$PHASES · RUN_TIMEOUT=${RUN_TIMEOUT}s" echo "hosts=$MON_HOSTS" echo "flags INSIDE_PODMAN=${INSIDE_PODMAN:-} INSIDE_MODE=${INSIDE_MODE:-} LOCAL_STACK=${LOCAL_STACK:-} SKIP_SSH=${SKIP_SSH:-} INSIDE_PROFILE=${INSIDE_PROFILE:-}" - -# --- refresh from Forgejo; re-apply suite-overlay if Forgejo lacks needed phases --- -OVERLAY_DIR="${XDG_CONFIG_HOME:-$HOME/.config}/taler-monitoring/suite-overlay" -# preserve capable mon before reset wipes tracked files -if [ -f "$AGENT_MON/taler-monitoring.sh" ]; then - mkdir -p "$OVERLAY_DIR" - rsync -a \ - --exclude secrets.env --exclude 'android-test/artifacts' --exclude 'android-test/apks' \ - --exclude 'android-test/out*' --exclude '__pycache__' \ - "$AGENT_MON/" "$OVERLAY_DIR/" 2>/dev/null || true -fi - -# shellcheck source=update-suite.sh -source "$AGENT_DIR/update-suite.sh" +# One-line suite pin (no git M/checkout noise) +echo "suite ${SUITE_VERSION:-?} @ ${COMMIT_SHORT:-?} ${COMMIT_URL:-}" +echo " dir=${SUITE_DIR:-}" SUITE_MON="${SUITE_DIR}" ROOT="$SUITE_MON" @@ -186,16 +229,8 @@ elif [ "$_need_overlay" = "1" ]; then ROOT="$AGENT_MON" else echo "using suite mon @ ${COMMIT_SHORT:-?} · $ROOT" -fi -# Always restore host-agent + site-gen from pre-reset snapshot when present -# (systemd ExecStart points at koopa-admin-log which may be the reset suite tree). -if [ -d "$OVERLAY_DIR/host-agent" ]; then - mkdir -p "$SUITE_MON/host-agent" 2>/dev/null || true - rsync -a "$OVERLAY_DIR/host-agent/" "$SUITE_MON/host-agent/" 2>/dev/null || true -fi -if [ -d "$OVERLAY_DIR/site-gen" ]; then - mkdir -p "$SUITE_MON/site-gen" - rsync -a "$OVERLAY_DIR/site-gen/" "$SUITE_MON/site-gen/" 2>/dev/null || true + # Do NOT rsync suite-overlay over a fresh Forgejo tree — that reintroduced + # stale site-gen/host-agent (bootstrap sticky, old monpages) on stage/betel. fi unset _need_overlay @@ -241,9 +276,17 @@ if [ -x "${ROOT:-}/taler-monitoring.sh" ]; then fi done + # monpages inside PHASES runs *before* htmlify/publish — content failures would + # race with concurrent publishes or reflect last run's incomplete redirect stub. + # Soft-content there; post-check after rsync stays hard (authoritative). + case " $PHASES " in + *" monpages "*|*" pages "*) export MONPAGES_PRE_PUBLISH=1 ;; + *) unset MONPAGES_PRE_PUBLISH 2>/dev/null || true ;; + esac # shellcheck disable=SC2086 ./taler-monitoring.sh -d "$TALER_DOMAIN" $PHASES ec=$? + unset MONPAGES_PRE_PUBLISH 2>/dev/null || true fi # --- always generate HTML (first run / missing pages / every run) --- @@ -254,26 +297,38 @@ write_bootstrap_html() { local curl="${COMMIT_URL:-}" local clink="$cshort" local gen iso - gen=$(date -u +"%Y-%m-%d %H:%M:%SZ" 2>/dev/null || date) - iso=$(date -u +"%Y-%m-%dT%H:%M:%SZ" 2>/dev/null || date -Iseconds) + # Local wall clock (CEST/CET · MESZ/MEZ on CH/DE hosts), not UTC Z + if TZ=Europe/Zurich date +"%Y-%m-%d %H:%M:%S %Z" >/dev/null 2>&1; then + gen=$(TZ=Europe/Zurich date +"%Y-%m-%d %H:%M:%S %Z") + iso=$(TZ=Europe/Zurich date +"%Y-%m-%dT%H:%M:%S%z" | sed 's/\([+-][0-9][0-9]\)\([0-9][0-9]\)$/\1:\2/') + else + gen=$(date +"%Y-%m-%d %H:%M:%S %Z" 2>/dev/null || date) + iso=$(date -Iseconds 2>/dev/null || date) + fi if [ -n "$curl" ]; then clink="$cshort" fi - cat >"$out" <"$tmp" < + monitoring ${mode} · ${host} + @@ -281,8 +336,10 @@ main{padding:1.5rem}

Bootstrap page (converter missing or first install). See host-agent log.

$(tail -n 80 "$LOG" 2>/dev/null | sed 's/&/\&/g;s/
 
+
 
 EOF
+  mv -f "$tmp" "$out"
   echo "bootstrap html → $out"
 }
 
@@ -293,7 +350,29 @@ htmlify_host() {
   mkdir -p "$HTML_BASE/$host/${HTML_OK_DIR}" "$HTML_BASE/$host/${HTML_ERR_DIR}"
 
   if [ -n "$SITE_GEN" ] && [ -f "$SITE_GEN/console_to_html.py" ]; then
-    python3 "$SITE_GEN/console_to_html.py" \
+    # Never convert with a pre-1.15.8 generator (old assets → SyntaxWarning + bad sticky JS).
+    if grep -qE 'if \(/\^\[[╔╗╚╝║═]' "$SITE_GEN/console_to_html.py" 2>/dev/null; then
+      echo "ERROR: refuse stale site-gen $SITE_GEN/console_to_html.py (box-drawing JS regex)" >&2
+      return 1 2>/dev/null || exit 1
+    fi
+    # SyntaxWarning = generator regression: fail convert (do not publish old-style pages).
+    # Other stderr still goes to agent log file only, not the mon console stream.
+    _mon_html_py() {
+      local _out="" _i
+      for _i in "$@"; do
+        if [ "$_i" = "--out" ]; then _out=1; continue; fi
+        if [ "$_out" = "1" ]; then _out="$_i"; break; fi
+      done
+      PYTHONWARNINGS=error::SyntaxWarning \
+        python3 "$SITE_GEN/console_to_html.py" "$@" 2>>"${LOG_DIR:-/tmp}/console_to_html.err" || return 1
+      # Never leave pre-1.15.8 sticky JS in a just-written page (stale log text is filtered in converter).
+      if [ -n "$_out" ] && [ -f "$_out" ] && grep -qE 'if \(/\^\[[╔╗╚╝║═]' "$_out" 2>/dev/null; then
+        echo "ERROR: generated mon HTML still has stale box-drawing JS: $_out" >&2
+        return 1
+      fi
+      return 0
+    }
+    if ! _mon_html_py \
       --lang "${TALER_MON_LANG:-en}" \
       --log "$LOG" \
       --out "$mon_err" \
@@ -301,37 +380,55 @@ htmlify_host() {
       --mode err \
       --commit "${COMMIT:-}" \
       --commit-url "${COMMIT_URL:-}" \
+      --suite-version "${SUITE_VERSION:-unknown}" \
+      --suite-version-url "${SUITE_VERSION_URL:-}" \
       --suite-path "$SUITE_PATH" \
       --page-label "$PAGE_LABEL" \
       --path-err "$HTML_URL_ERR" \
       --link-other "$HTML_URL_OK"
+    then
+      echo "ERROR: mon HTML convert failed (err page) for $host — refuse publish of stale/broken assets" >&2
+      return 1 2>/dev/null || exit 1
+    fi
     if [ "$ec" -eq 0 ]; then
-      python3 "$SITE_GEN/console_to_html.py" \
-      --lang "${TALER_MON_LANG:-en}" \
+      if ! _mon_html_py \
+        --lang "${TALER_MON_LANG:-en}" \
         --log "$LOG" \
         --out "$mon" \
         --hostname "$host" \
         --mode ok \
         --commit "${COMMIT:-}" \
         --commit-url "${COMMIT_URL:-}" \
+        --suite-version "${SUITE_VERSION:-unknown}" \
+        --suite-version-url "${SUITE_VERSION_URL:-}" \
         --suite-path "$SUITE_PATH" \
         --page-label "$PAGE_LABEL" \
         --path-err "$HTML_URL_ERR" \
         --link-other ""
+      then
+        echo "ERROR: mon HTML convert failed (ok page) for $host" >&2
+        return 1 2>/dev/null || exit 1
+      fi
       rm -rf "$HTML_BASE/$host/${HTML_ERR_DIR}"
       echo "html $host → ${HTML_URL_OK} only (clean · ${COMMIT_SHORT:-?})"
     else
-      python3 "$SITE_GEN/console_to_html.py" \
-      --lang "${TALER_MON_LANG:-en}" \
+      if ! _mon_html_py \
+        --lang "${TALER_MON_LANG:-en}" \
         --log "$LOG" \
         --out "$mon" \
         --hostname "$host" \
         --mode redirect \
         --commit "${COMMIT:-}" \
         --commit-url "${COMMIT_URL:-}" \
+        --suite-version "${SUITE_VERSION:-unknown}" \
+        --suite-version-url "${SUITE_VERSION_URL:-}" \
         --suite-path "$SUITE_PATH" \
         --page-label "$PAGE_LABEL" \
         --path-err "$HTML_URL_ERR"
+      then
+        echo "ERROR: mon HTML convert failed (redirect page) for $host" >&2
+        return 1 2>/dev/null || exit 1
+      fi
       echo "html $host → ${HTML_URL_OK} stub + ${HTML_URL_ERR} (ec=$ec · ${COMMIT_SHORT:-?})"
     fi
   else
@@ -354,17 +451,76 @@ htmlify_host() {
 }
 for host in $MON_HOSTS; do
   htmlify_host "$host"
+  echo "public URL (expected): https://${host}${HTML_URL_OK}"
 done
 
-if [ -n "$DEPLOY_WWW" ] && [ -w "$DEPLOY_WWW" ] 2>/dev/null; then
-  rsync -a --delete "$HTML_BASE/" "$DEPLOY_WWW/" && \
-    echo "synced → $DEPLOY_WWW/"
+# Publish to live web root when possible (v1.3.1+: never silent-skip)
+if [ -n "$DEPLOY_WWW" ]; then
+  if [ -w "$DEPLOY_WWW" ] 2>/dev/null; then
+    # --delay-updates: write to temp names then rename (less partial-file exposure)
+    if rsync -a --delete --delay-updates "$HTML_BASE/" "$DEPLOY_WWW/"; then
+      echo "synced → $DEPLOY_WWW/ (delay-updates)"
+      # brief settle so concurrent monpages curls do not hit mid-rename FS views
+      sleep "${MONPAGES_PUBLISH_SETTLE_S:-0.3}" 2>/dev/null || sleep 1
+    else
+      echo "ERROR: rsync to DEPLOY_WWW_ROOT=$DEPLOY_WWW failed" >&2
+      ec=1
+    fi
+  elif [ -e "$DEPLOY_WWW" ]; then
+    echo "ERROR: DEPLOY_WWW_ROOT=$DEPLOY_WWW exists but is not writable by $(id -un)" >&2
+    echo "       public FQDN pages will stay missing/stale — fix ownership or run:" >&2
+    echo "       sudo ${APPLY_MONITORING_LIVE}" >&2
+    ec=1
+  else
+    echo "ERROR: DEPLOY_WWW_ROOT=$DEPLOY_WWW missing — public pages not published" >&2
+    echo "       create tree + Caddy handles: sudo ${APPLY_MONITORING_LIVE}" >&2
+    ec=1
+  fi
+else
+  echo "WARN: DEPLOY_WWW_ROOT empty — skipping live publish (staging only: $HTML_BASE)" >&2
+fi
+
+# Post-publish: monpages obligatory (v1.7.6+) — GOA full inventory / FP only FP hosts
+_monpages_failed=0
+if [ "${MONPAGES_POSTCHECK:-1}" = "1" ] && [ -x "$ROOT/check_monitoring_pages.sh" ]; then
+  echo "--- monpages post-check (public FQDN, obligatory) ---"
+  export MON_HOSTS HTML_URL_OK HTML_URL_ERR TALER_DOMAIN HTML_OUT HTML_BASE
+  export MONPAGES_STAGING_BASE="${HTML_BASE}"
+  export DEPLOY_WWW_ROOT="${DEPLOY_WWW:-${DEPLOY_WWW_ROOT:-}}"
+  export MONPAGES_REQUIRE_PUBLIC="${MONPAGES_REQUIRE_PUBLIC:-1}"
+  # auto: GOA = all suite mon sites; FP = only FP mon hosts; job paths always included
+  export MONPAGES_INVENTORY="${MONPAGES_INVENTORY:-auto}"
+  # Hard check (not pre-publish soft)
+  unset MONPAGES_PRE_PUBLISH 2>/dev/null || true
+  export MONPAGES_PRE_PUBLISH=0
+  if ! bash "$ROOT/check_monitoring_pages.sh"; then
+    echo "ERROR: public monitoring page(s) missing or not suite HTML via FQDN" >&2
+    echo "       staging HTML_OUT=${HTML_BASE} · DEPLOY_WWW=${DEPLOY_WWW:-}" >&2
+    ec=1
+    _monpages_failed=1
+  fi
+  unset MONPAGES_PRE_PUBLISH 2>/dev/null || true
+fi
+
+# If public check failed after first HTML write, rewrite pages so sticky bar shows monpages ERRORs
+if [ "$_monpages_failed" = "1" ]; then
+  echo "--- re-htmlify after monpages failure (include public outage in report) ---"
+  # append marker to log so converter sees ERROR lines if monpages was post-only
+  {
+    echo ""
+    echo "ERROR monpages: public FQDN monitoring HTML missing or merchant JSON code 21"
+    echo "ERROR monpages: fix with sudo ${APPLY_MONITORING_LIVE} (www + Caddy handles)"
+  } >>"$LOG"
+  for host in $MON_HOSTS; do
+    htmlify_host "$host"
+  done
 fi
 
 ls -1t "$LOG_DIR"/run-*.log 2>/dev/null | tail -n +30 | xargs rm -f 2>/dev/null || true
 
 echo "======== done ec=$ec · log=$LOG · ${COMMIT_URL:-} ========"
-if [ "${STRICT_EXIT:-0}" = "1" ]; then
+# v1.7.1: default STRICT_EXIT=1 so systemd/timer reflects publish/public failures
+if [ "${STRICT_EXIT:-1}" = "1" ]; then
   exit "$ec"
 fi
 exit 0
diff --git a/host-agent/run-max-ladder-monitoring.sh b/host-agent/run-max-ladder-monitoring.sh
new file mode 100755
index 0000000..813fedd
--- /dev/null
+++ b/host-agent/run-max-ladder-monitoring.sh
@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+# run-max-ladder-monitoring.sh — GOA max-search amount ladder → public HTML
+#
+# Public (landing hosts, same Caddy /monitoring* static root):
+#   https://{bank,exchange,taler}.hacktivism.ch/monitoring-max-ladder/
+#   https://{bank,exchange,taler}.hacktivism.ch/monitoring-max-ladder_err/
+#
+# Timer: once daily (taler-monitoring-max-ladder.timer). Long wall clock.
+#
+set -uo pipefail
+AGENT_DIR=$(cd "$(dirname "$0")" && pwd)
+
+export AGENT_LABEL="${AGENT_LABEL:-max-ladder-host-agent}"
+export STATE_NAME="${STATE_NAME:-taler-max-ladder-monitoring}"
+export TALER_DOMAIN="${TALER_DOMAIN:-hacktivism.ch}"
+export TALER_MON_LANG="${TALER_MON_LANG:-en}"
+export TALER_MON_LANG_SET=1
+export INSIDE_PODMAN="${INSIDE_PODMAN:-1}"
+export INSIDE_MODE="${INSIDE_MODE:-local-podman}"
+export LOCAL_STACK="${LOCAL_STACK:-1}"
+export SKIP_SSH="${SKIP_SSH:-0}"
+export CONTINUE_ON_ERROR="${CONTINUE_ON_ERROR:-1}"
+# max-search probes + pay hunt — force high wall clock (koopa env often has RUN_TIMEOUT=600)
+export RUN_TIMEOUT="${MAX_LADDER_RUN_TIMEOUT:-10800}"
+
+export PHASES="max-ladder monpages"
+export MON_HOSTS="${MON_HOSTS:-bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch}"
+export HTML_OUT="${HTML_OUT:-$HOME/monitoring-sites-staging}"
+export DEPLOY_WWW_ROOT="${DEPLOY_WWW_ROOT:-/var/www/monitoring-sites}"
+
+export HTML_OK_DIR="monitoring-max-ladder"
+export HTML_ERR_DIR="monitoring-max-ladder_err"
+export HTML_URL_OK="/monitoring-max-ladder/"
+export HTML_URL_ERR="/monitoring-max-ladder_err/"
+export PAGE_LABEL="monitoring-max-ladder"
+export MONPAGES_INVENTORY="job"
+
+export LADDER_MODE=max
+export LADDER_PAY="${LADDER_PAY:-1}"
+export LADDER_PAY_WAIT_AVAILABLE_S="${LADDER_PAY_WAIT_AVAILABLE_S:-180}"
+export APT_DEPLOY_ENSURE=0
+# Prefer host wallet-cli install used by mon (find_wallet_cli / WALLET_CLI)
+export PATH="${HOME}/.local/bin:${PATH:-}"
+if [ -z "${WALLET_CLI:-}" ] && [ -f "${HOME}/.local/bin/taler-wallet-cli.mjs" ]; then
+  export WALLET_CLI="${HOME}/.local/bin/taler-wallet-cli.mjs"
+fi
+
+exec bash "$AGENT_DIR/run-host-report.sh" "$@"
diff --git a/host-agent/run-surface-monitoring.sh b/host-agent/run-surface-monitoring.sh
index 1b7a8b8..d634a7f 100755
--- a/host-agent/run-surface-monitoring.sh
+++ b/host-agent/run-surface-monitoring.sh
@@ -1,27 +1,36 @@
 #!/usr/bin/env bash
-# run-surface-monitoring.sh — hourly remote-only ecosystem surface scan
-# Public only on taler.hacktivism.ch:
+# run-surface-monitoring.sh — ecosystem surface report (v1.8.0+ simplified)
+#
+# Public HTML only on taler.hacktivism.ch:
 #   https://taler.hacktivism.ch/taler-monitoring-surface/
 #   https://taler.hacktivism.ch/taler-monitoring-surface_err/
 #
+# This page covers remote ecosystem inventory: host/port probes, nmap OS
+# fingerprint, Mattermost, mail (firefly + mail.anastasis.lu/pixel), package/version signals.
+# There are NO separate public pages for mail / mattermost (folded here).
+# Apt-src deploy tests keep their own page + timer:
+#   run-aptdeploy-monitoring.sh → /taler-monitoring-aptdeploy(+_err)/
+#
+# Landing-stack reports stay on each of the 9 fronts as /monitoring(/_err).
+#
 set -uo pipefail
 AGENT_DIR=$(cd "$(dirname "$0")" && pwd)
 
 export AGENT_LABEL="${AGENT_LABEL:-surface-host-agent}"
 export STATE_NAME="${STATE_NAME:-taler-surface-monitoring}"
 export TALER_DOMAIN="${TALER_DOMAIN:-hacktivism.ch}"
-# surface is remote-only — force flags (ignore shared env)
-export INSIDE_PODMAN=0
-export LOCAL_STACK=0
-export SKIP_SSH=1
-export INSIDE_MODE="${INSIDE_MODE:-none}"
+# surface is primarily remote; versions may use host podman when available
+export INSIDE_PODMAN="${INSIDE_PODMAN:-1}"
+export LOCAL_STACK="${LOCAL_STACK:-0}"
+export SKIP_SSH="${SKIP_SSH:-1}"
+export INSIDE_MODE="${INSIDE_MODE:-local-podman}"
 export CONTINUE_ON_ERROR="${CONTINUE_ON_ERROR:-1}"
-# many hosts; allow up to 45 min wall (phase also has RUN_TIMEOUT)
 export RUN_TIMEOUT="${RUN_TIMEOUT:-2400}"
-# dedicated job: always surface only (never inherit PHASES from shared env)
-export PHASES=surface
-# Only merchant public front — not bank/exchange
-export MON_HOSTS="${MON_HOSTS:-taler.hacktivism.ch}"
+
+# All ecosystem checks → one surface HTML report (force — do not inherit main-timer env)
+export PHASES="surface mattermost mail versions monpages"
+
+export MON_HOSTS="taler.hacktivism.ch"
 export HTML_OUT="${HTML_OUT:-$HOME/monitoring-sites-staging}"
 export DEPLOY_WWW_ROOT="${DEPLOY_WWW_ROOT:-/var/www/monitoring-sites}"
 
@@ -31,11 +40,12 @@ export HTML_URL_OK="/taler-monitoring-surface/"
 export HTML_URL_ERR="/taler-monitoring-surface_err/"
 export PAGE_LABEL="taler-monitoring-surface"
 
-# ecosystem catalog by default (not domain-only)
 export SURFACE_SCOPE="${SURFACE_SCOPE:-ecosystem}"
 export TALER_DOMAIN_FROM_CLI=0
-
-# Do not run apt-deploy ensure for surface-only jobs
 export APT_DEPLOY_ENSURE=0
+# monpages job-only for surface paths on taler.hacktivism.ch
+export MONPAGES_INVENTORY="job"
+# package behind → ERROR (v1.8.0)
+export TALER_PKG_BEHIND="${TALER_PKG_BEHIND:-error}"
 
 exec bash "$AGENT_DIR/run-host-report.sh" "$@"
diff --git a/host-agent/taler-monitoring-hacktivism.path b/host-agent/taler-monitoring-hacktivism.path
index d1318fd..f585abf 100644
--- a/host-agent/taler-monitoring-hacktivism.path
+++ b/host-agent/taler-monitoring-hacktivism.path
@@ -4,13 +4,9 @@ Documentation=file:%h/src/taler-monitoring/host-agent/README.md
 
 [Path]
 # Explicit stamp after package/image upgrades (touch-software-stamp.sh)
+# Landing/caddy ops live outside this suite (host admin-log) — touch the stamp there.
 PathChanged=%h/.local/state/taler-hacktivism-monitoring/software.stamp
 PathModified=%h/.local/state/taler-hacktivism-monitoring/software.stamp
-# Landing + caddy config in admin-log (when checked out under ~)
-PathChanged=%h/koopa-admin-log/configs/bank-landing
-PathChanged=%h/koopa-admin-log/configs/exchange-landing
-PathChanged=%h/koopa-admin-log/configs/merchant-landing
-PathChanged=%h/koopa-admin-log/configs/caddy
 # Debounce: wait for quiet period before firing
 TriggerLimitIntervalSec=120
 TriggerLimitBurst=3
diff --git a/host-agent/taler-monitoring-max-ladder.service b/host-agent/taler-monitoring-max-ladder.service
new file mode 100644
index 0000000..9943edb
--- /dev/null
+++ b/host-agent/taler-monitoring-max-ladder.service
@@ -0,0 +1,19 @@
+[Unit]
+Description=Taler monitoring GOA max-search amount ladder → /monitoring-max-ladder/
+Documentation=file:%h/src/taler-monitoring/host-agent/README.md
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin
+Environment=CONTINUE_ON_ERROR=1
+Environment=RUN_TIMEOUT=10800
+EnvironmentFile=-%h/.config/taler-monitoring/env
+WorkingDirectory=%h/src/taler-monitoring
+ExecStart=%h/src/taler-monitoring/host-agent/run-max-ladder-monitoring.sh
+Nice=10
+TimeoutStartSec=4h
+
+[Install]
+WantedBy=default.target
diff --git a/host-agent/taler-monitoring-max-ladder.timer b/host-agent/taler-monitoring-max-ladder.timer
new file mode 100644
index 0000000..5b16361
--- /dev/null
+++ b/host-agent/taler-monitoring-max-ladder.timer
@@ -0,0 +1,14 @@
+[Unit]
+Description=Daily GOA max-search amount ladder mon page
+Documentation=file:%h/src/taler-monitoring/host-agent/README.md
+
+[Timer]
+# Once per day, after classic ladder window
+OnCalendar=*-*-* 06:30:00
+Persistent=true
+AccuracySec=15min
+RandomizedDelaySec=30min
+Unit=taler-monitoring-max-ladder.service
+
+[Install]
+WantedBy=timers.target
diff --git a/host-agent/taler-monitoring-mytops-stage.service b/host-agent/taler-monitoring-mytops-stage.service
new file mode 100644
index 0000000..0c07bfb
--- /dev/null
+++ b/host-agent/taler-monitoring-mytops-stage.service
@@ -0,0 +1,20 @@
+[Unit]
+Description=taler-monitoring mytops stage (betel) — auto-update suite + urls/monpages
+Documentation=file:%h/src/taler-monitoring/host-agent/README.md
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+# Env: ~/.config/taler-monitoring/env (SUITE_GIT_*, MON_HOSTS, PHASES=urls monpages, …)
+EnvironmentFile=-%h/.config/taler-monitoring/env
+Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin
+Environment=SUITE_UPDATE_STRICT=1
+WorkingDirectory=%h/src/taler-monitoring
+# run-host-report sources update-suite.sh first (strict fetch/reset to main)
+ExecStart=%h/src/taler-monitoring/host-agent/run-host-report.sh
+Nice=10
+TimeoutStartSec=20min
+
+[Install]
+WantedBy=default.target
diff --git a/host-agent/taler-monitoring-mytops-stage.timer b/host-agent/taler-monitoring-mytops-stage.timer
new file mode 100644
index 0000000..73a09fa
--- /dev/null
+++ b/host-agent/taler-monitoring-mytops-stage.timer
@@ -0,0 +1,11 @@
+[Unit]
+Description=taler-monitoring mytops stage (betel) every 4h — suite auto-upgrade + mon HTML
+
+[Timer]
+OnBootSec=5min
+OnUnitActiveSec=4h
+Persistent=true
+Unit=taler-monitoring-mytops-stage.service
+
+[Install]
+WantedBy=timers.target
diff --git a/host-agent/update-suite.sh b/host-agent/update-suite.sh
index 883079e..9c5f3ea 100755
--- a/host-agent/update-suite.sh
+++ b/host-agent/update-suite.sh
@@ -1,5 +1,18 @@
 #!/usr/bin/env bash
-# update-suite.sh — latest taler-monitoring from Forgejo (standalone repo root).
+# update-suite.sh — pull latest taler-monitoring from Forgejo (standalone repo root).
+#
+# Sourced by run-host-report.sh / wrappers. Every host-agent run must land on
+# current origin/$SUITE_GIT_REF (default main). Stale clones are not acceptable.
+#
+# Env:
+#   SUITE_GIT_URL          default https://git.hacktivism.ch/hernani/taler-monitoring.git
+#   SUITE_GIT_REF          default main
+#   SUITE_DIR              default ~/src/taler-monitoring
+#   SUITE_UPDATE_MODE       reset (default) | pull | keep
+#                          keep = no fetch/reset (local deploy / unpushed fixes)
+#   SUITE_UPDATE_STRICT     1 (default) = fetch/reset failure aborts the agent run
+#                          0 = warn and continue on old tree (emergency only)
+#
 set -uo pipefail
 CFG_DIR="${XDG_CONFIG_HOME:-$HOME/.config}/taler-monitoring"
 ENV_FILE="${TALER_MONITORING_ENV:-$CFG_DIR/env}"
@@ -9,37 +22,132 @@ fi
 SUITE_GIT_URL="${SUITE_GIT_URL:-https://git.hacktivism.ch/hernani/taler-monitoring.git}"
 SUITE_GIT_REF="${SUITE_GIT_REF:-main}"
 SUITE_UPDATE_MODE="${SUITE_UPDATE_MODE:-reset}"
+SUITE_UPDATE_STRICT="${SUITE_UPDATE_STRICT:-1}"
 if [ -z "${SUITE_DIR:-}" ]; then
   if [ -d "$HOME/src/taler-monitoring/.git" ]; then SUITE_DIR="$HOME/src/taler-monitoring"
   elif [ -d "$HOME/taler-monitoring/.git" ]; then SUITE_DIR="$HOME/taler-monitoring"
   else SUITE_DIR="$HOME/src/taler-monitoring"; fi
 fi
 export SUITE_DIR
+
+_upd_die() {
+  echo "ERROR suite-update: $*" >&2
+  if [ "${SUITE_UPDATE_STRICT}" = "1" ]; then
+    return 1 2>/dev/null || exit 1
+  fi
+  echo "WARN suite-update: continuing on existing tree (SUITE_UPDATE_STRICT=0)" >&2
+  return 0
+}
+
 mkdir -p "$(dirname "$SUITE_DIR")" "$CFG_DIR"
 if [ ! -d "$SUITE_DIR/.git" ]; then
   echo "clone $SUITE_GIT_URL → $SUITE_DIR"
-  git clone --branch "$SUITE_GIT_REF" "$SUITE_GIT_URL" "$SUITE_DIR" || git clone "$SUITE_GIT_URL" "$SUITE_DIR"
+  if ! git clone --branch "$SUITE_GIT_REF" "$SUITE_GIT_URL" "$SUITE_DIR" \
+    && ! git clone "$SUITE_GIT_URL" "$SUITE_DIR"; then
+    _upd_die "git clone failed"
+    return 1 2>/dev/null || exit 1
+  fi
 fi
-cd "$SUITE_DIR" || exit 1
+cd "$SUITE_DIR" || { _upd_die "cd $SUITE_DIR failed"; return 1 2>/dev/null || exit 1; }
 git remote set-url origin "$SUITE_GIT_URL" 2>/dev/null || true
-echo "fetch origin ($SUITE_GIT_REF) …"
-if ! git fetch --tags origin 2>&1; then
-  echo "WARN: git fetch failed — using existing tree" >&2
-  return 0 2>/dev/null || exit 0
-fi
+
+_before=$(git rev-parse --short=12 HEAD 2>/dev/null || echo none)
+_before_ver=$(git describe --tags --exact-match 2>/dev/null \
+  || git describe --tags --abbrev=0 2>/dev/null \
+  || echo unknown)
+
 case "$SUITE_UPDATE_MODE" in
-  reset)
-    git checkout -B "$SUITE_GIT_REF" "origin/$SUITE_GIT_REF" 2>/dev/null || git checkout -B "$SUITE_GIT_REF" FETCH_HEAD
-    git reset --hard "origin/$SUITE_GIT_REF" 2>/dev/null || git reset --hard FETCH_HEAD
+  keep|skip|off|none)
+    echo "suite update skipped (SUITE_UPDATE_MODE=$SUITE_UPDATE_MODE) — keep local tree ${_before} (${_before_ver})"
+    ;;
+  *)
+    echo "fetch origin ($SUITE_GIT_REF) + tags …"
+    if ! git fetch --tags --force --prune origin 2>&1; then
+      _upd_die "git fetch failed — suite would stay at ${_before} (${_before_ver})"
+      # fall through only if STRICT=0
+    else
+      case "$SUITE_UPDATE_MODE" in
+        reset)
+          if ! git checkout -B "$SUITE_GIT_REF" "origin/$SUITE_GIT_REF" 2>/dev/null \
+            && ! git checkout -B "$SUITE_GIT_REF" FETCH_HEAD; then
+            _upd_die "git checkout $SUITE_GIT_REF failed"
+          fi
+          if ! git reset --hard "origin/$SUITE_GIT_REF" 2>/dev/null \
+            && ! git reset --hard FETCH_HEAD; then
+            _upd_die "git reset --hard origin/$SUITE_GIT_REF failed"
+          fi
+          ;;
+        pull)
+          if ! git pull --ff-only origin "$SUITE_GIT_REF" 2>&1; then
+            _upd_die "git pull --ff-only failed"
+          fi
+          ;;
+        *)
+          if ! git pull --ff-only origin "$SUITE_GIT_REF" 2>&1; then
+            _upd_die "git pull --ff-only failed (mode=$SUITE_UPDATE_MODE)"
+          fi
+          ;;
+      esac
+    fi
     ;;
-  *) git pull --ff-only origin "$SUITE_GIT_REF" 2>&1 || true ;;
 esac
+
 COMMIT=$(git rev-parse HEAD)
 COMMIT_SHORT=$(git rev-parse --short=12 HEAD)
 SOURCE_REPO_WEB="${SOURCE_REPO_WEB:-https://git.hacktivism.ch/hernani/taler-monitoring}"
 COMMIT_URL="${SOURCE_REPO_WEB}/src/commit/${COMMIT}"
 export COMMIT COMMIT_SHORT COMMIT_URL SOURCE_REPO_WEB
+
+# Release tag for sticky bar (exact tag if HEAD is tagged, else nearest)
+SUITE_VERSION=""
+if command -v git >/dev/null 2>&1; then
+  SUITE_VERSION=$(git describe --tags --exact-match 2>/dev/null || true)
+  if [ -z "$SUITE_VERSION" ]; then
+    SUITE_VERSION=$(git describe --tags --abbrev=0 2>/dev/null || true)
+  fi
+fi
+if [ -z "$SUITE_VERSION" ] && [ -f "$SUITE_DIR/VERSION" ]; then
+  SUITE_VERSION="v$(tr -d '[:space:]' <"$SUITE_DIR/VERSION" | sed 's/^v//')"
+fi
+[ -z "$SUITE_VERSION" ] && SUITE_VERSION="unknown"
+case "$SUITE_VERSION" in
+  unknown|v*) ;;
+  *) SUITE_VERSION="v${SUITE_VERSION}" ;;
+esac
+# Sticky version badge: prefer commit URL (exact tree of this run).
+# Tag URLs (/src/tag/vX.Y.Z) 404 when the tag is not yet pushed to Forgejo;
+# commit URLs work as soon as main (or the tree) is on the remote.
+SUITE_VERSION_URL=""
+if [ -n "$SOURCE_REPO_WEB" ] && [ -n "${COMMIT:-}" ] && [ "$COMMIT" != "unknown" ]; then
+  SUITE_VERSION_URL="${SOURCE_REPO_WEB}/src/commit/${COMMIT}"
+elif [ -n "$SOURCE_REPO_WEB" ] && [ "$SUITE_VERSION" != "unknown" ]; then
+  SUITE_VERSION_URL="${SOURCE_REPO_WEB}/src/tag/${SUITE_VERSION}"
+fi
+export SUITE_VERSION SUITE_VERSION_URL
+
+if [ "$_before" != "$COMMIT_SHORT" ]; then
+  echo "  upgraded ${_before} (${_before_ver}) → ${COMMIT_SHORT} (${SUITE_VERSION})"
+else
+  echo "  already at ${COMMIT_SHORT} (${SUITE_VERSION})"
+fi
+echo "  version=$SUITE_VERSION ${SUITE_VERSION_URL:+· $SUITE_VERSION_URL}"
 echo "suite @ $COMMIT_SHORT  $COMMIT_URL"
 echo "  dir=$SUITE_DIR"
+
+# Sanity: must have suite entrypoint after update
+if [ ! -x "$SUITE_DIR/taler-monitoring.sh" ] && [ ! -f "$SUITE_DIR/taler-monitoring.sh" ]; then
+  _upd_die "taler-monitoring.sh missing after update"
+  return 1 2>/dev/null || exit 1
+fi
+
 ln -sfn "$SUITE_DIR" "$HOME/taler-monitoring" 2>/dev/null || true
 chmod +x taler-monitoring.sh check_*.sh host-agent/*.sh site-gen/*.sh site-gen/*.py 2>/dev/null || true
+# Drop bytecode so a stale site-gen never wins over the tree we just reset to.
+rm -rf "$SUITE_DIR/site-gen/__pycache__" 2>/dev/null || true
+# Refuse known-bad pre-1.15.8 generator (box-drawing JS regex → SyntaxWarning noise)
+if [ -f "$SUITE_DIR/site-gen/console_to_html.py" ] \
+  && grep -qE 'if \(/\^\[[╔╗╚╝║═]' "$SUITE_DIR/site-gen/console_to_html.py" 2>/dev/null; then
+  _upd_die "site-gen/console_to_html.py still has pre-1.15.8 box-drawing JS regex — refuse stale generator"
+  return 1 2>/dev/null || exit 1
+fi
+return 0 2>/dev/null || true
diff --git a/i18n.sh b/i18n.sh
index 45dd1b7..417a751 100644
--- a/i18n.sh
+++ b/i18n.sh
@@ -1,101 +1,202 @@
 #!/usr/bin/env bash
-# i18n.sh — language for taler-monitoring console + HTML chrome
+# i18n.sh — UI language + regional locale for taler-monitoring console + HTML
 #
-# TALER_MON_LANG=en|fr   (default en)
-# Auto: domains *lefrancpaysan* → fr unless TALER_MON_LANG is set explicitly.
+# UI language (mon badges / sticky chrome):
+#   TALER_MON_LANG=en|fr|de
+#   TALER_MON_LANG_SET=1  → explicit (--lang); do not override.
+#   Auto (SET≠1): TALER_DOMAIN_LANG from domains.conf `lang` column.
+#   de is supported but never a stock profile default (no domains.conf row uses lang=de).
+#
+# Regional locale (l10n defaults, BCP 47 — not the same as UI lang):
+#   TALER_MON_LOCALE / TALER_DOMAIN_LOCALE e.g. fr-CH, de-CH
+#   From domains.conf `locale` column (fr-CH for FP, de-CH for others).
+#   Aliases ch-FR / ch-DE accepted and normalized to fr-CH / de-CH.
+#
+# domains.conf is the source of truth for both lang and locale.
 
 : "${TALER_MON_LANG:=}"
 : "${TALER_MON_LANG_SET:=0}"
+: "${TALER_DOMAIN_LANG:=}"
+: "${TALER_DOMAIN_LOCALE:=}"
+: "${TALER_MON_LOCALE:=}"
 
 i18n_init() {
-  if [ -n "${TALER_MON_LANG:-}" ]; then
-    TALER_MON_LANG_SET=1
-  fi
-  if [ "${TALER_MON_LANG_SET}" != "1" ]; then
-    case "${TALER_DOMAIN:-}" in
-      *lefrancpaysan*|*francpaysan*)
-        TALER_MON_LANG=fr
-        ;;
-      *)
-        TALER_MON_LANG=en
-        ;;
-    esac
+  # Only TALER_MON_LANG_SET=1 is "explicit". Do not promote non-empty TALER_MON_LANG
+  # to SET=1 — env files often ship LANG=en for the default GOA workstation.
+  if [ "${TALER_MON_LANG_SET:-0}" != "1" ]; then
+    if [ -n "${TALER_DOMAIN_LANG:-}" ]; then
+      TALER_MON_LANG="$TALER_DOMAIN_LANG"
+    else
+      case "${TALER_DOMAIN:-}" in
+        *lefrancpaysan*|*francpaysan*)
+          TALER_MON_LANG=fr
+          ;;
+        *)
+          if [ -z "${TALER_MON_LANG:-}" ]; then
+            TALER_MON_LANG=en
+          fi
+          ;;
+      esac
+    fi
   fi
   case "${TALER_MON_LANG}" in
     fr|FR|fra|french) TALER_MON_LANG=fr ;;
+    de|DE|deu|ger|german|deutsch) TALER_MON_LANG=de ;;
     *) TALER_MON_LANG=en ;;
   esac
+
+  # Regional locale from profile (or FP/other fallback)
+  if [ -n "${TALER_DOMAIN_LOCALE:-}" ]; then
+    TALER_MON_LOCALE="$TALER_DOMAIN_LOCALE"
+  elif [ -z "${TALER_MON_LOCALE:-}" ]; then
+    case "${TALER_DOMAIN:-}" in
+      *lefrancpaysan*|*francpaysan*) TALER_MON_LOCALE=fr-CH ;;
+      *) TALER_MON_LOCALE=de-CH ;;
+    esac
+  fi
+  # normalize aliases
+  case "$(printf '%s' "$TALER_MON_LOCALE" | tr '[:upper:]' '[:lower:]' | tr '_' '-')" in
+    fr-ch|ch-fr) TALER_MON_LOCALE=fr-CH ;;
+    de-ch|ch-de) TALER_MON_LOCALE=de-CH ;;
+    en-ch|ch-en) TALER_MON_LOCALE=en-CH ;;
+  esac
+
   export TALER_MON_LANG
+  export TALER_MON_LANG_SET
+  export TALER_DOMAIN_LANG
+  export TALER_DOMAIN_LOCALE
+  export TALER_MON_LOCALE
 }
 
 i18n_tag() {
   local t="$1"
-  if [ "${TALER_MON_LANG:-en}" != "fr" ]; then
-    printf '%s' "$t"
-    return 0
-  fi
-  case "$t" in
-    OK) printf 'OK' ;;
-    ERROR) printf 'ERREUR' ;;
-    WARN) printf 'AVERT' ;;
-    INFO) printf 'INFO' ;;
-    BLOCKER) printf 'BLOCAGE' ;;
-    BLOCK) printf 'BLOC' ;;
-    SUMMARY) printf 'RESUME' ;;
-    PROG|PROGRESS) printf 'PROG' ;;
-    *) printf '%s' "$t" ;;
+  case "${TALER_MON_LANG:-en}" in
+    fr)
+      case "$t" in
+        OK) printf 'OK' ;;
+        ERROR) printf 'ERREUR' ;;
+        WARN) printf 'AVERT' ;;
+        INFO) printf 'INFO' ;;
+        BLOCKER) printf 'BLOCAGE' ;;
+        BLOCK) printf 'BLOC' ;;
+        SUMMARY) printf 'RESUME' ;;
+        PROG|PROGRESS) printf 'PROG' ;;
+        *) printf '%s' "$t" ;;
+      esac
+      ;;
+    de)
+      case "$t" in
+        OK) printf 'OK' ;;
+        ERROR) printf 'FEHLER' ;;
+        WARN) printf 'WARNUNG' ;;
+        INFO) printf 'INFO' ;;
+        BLOCKER) printf 'BLOCKER' ;;
+        BLOCK) printf 'BLOCK' ;;
+        SUMMARY) printf 'ZUSAMMENFASSUNG' ;;
+        PROG|PROGRESS) printf 'PROG' ;;
+        *) printf '%s' "$t" ;;
+      esac
+      ;;
+    *)
+      printf '%s' "$t"
+      ;;
   esac
 }
 
 i18n_text() {
   local s="$*"
-  if [ "${TALER_MON_LANG:-en}" != "fr" ] || [ -z "$s" ]; then
-    printf '%s' "$s"
-    return 0
-  fi
-  case "$s" in
-    "numbered checks this run:") printf 'controles numerotes de cette execution :' ;;
-    "failed — see list above"|"failed - see list above") printf 'echec — voir la liste ci-dessus' ;;
-    "BLOCKERS · pay/withdraw cannot finish") printf 'BLOCAGES · paiement/retrait impossible' ;;
-    "BLOCKERS (pay/withdraw cannot finish)") printf 'BLOCAGES (paiement/retrait impossible)' ;;
-    "ERRORS · failed checks") printf 'ERREURS · controles en echec' ;;
-    "ERRORS (failed checks)") printf 'ERREURS (controles en echec)' ;;
-    "pay/withdraw cannot finish") printf 'paiement/retrait impossible a terminer' ;;
-    "SUMMARY") printf 'RESUME' ;;
-    "totals:") printf 'totaux :' ;;
-    "container") printf 'conteneur' ;;
-    "disk") printf 'disque' ;;
-    "package") printf 'paquet' ;;
-    "version") printf 'version' ;;
-    "reachability") printf 'accessibilite' ;;
-    "server-header") printf 'en-tete Server' ;;
-    "inventory") printf 'inventaire' ;;
-    "flags") printf 'options' ;;
-    "target domain") printf 'domaine cible' ;;
-    "currency") printf 'devise' ;;
-    "phases") printf 'phases' ;;
-    "access") printf 'acces' ;;
+  [ -z "$s" ] && { printf '%s' "$s"; return 0; }
+  case "${TALER_MON_LANG:-en}" in
+    fr)
+      case "$s" in
+        "numbered checks this run:") printf 'controles numerotes de cette execution :' ;;
+        "numbered checks:") printf 'controles numerotes :' ;;
+        "(global done/total)") printf '(fait/total global)' ;;
+        "global checks:") printf 'controles globaux :' ;;
+        "failed — see list above"|"failed - see list above") printf 'echec — voir la liste ci-dessus' ;;
+        "BLOCKERS · pay/withdraw cannot finish") printf 'BLOCAGES · paiement/retrait impossible' ;;
+        "BLOCKERS (pay/withdraw cannot finish)") printf 'BLOCAGES (paiement/retrait impossible)' ;;
+        "ERRORS · failed checks") printf 'ERREURS · controles en echec' ;;
+        "ERRORS (failed checks)") printf 'ERREURS (controles en echec)' ;;
+        "pay/withdraw cannot finish") printf 'paiement/retrait impossible a terminer' ;;
+        "SUMMARY") printf 'RESUME' ;;
+        "totals:") printf 'totaux :' ;;
+        "all clear") printf 'tout est clair' ;;
+        "warnings only (no hard fail)") printf 'avertissements seulement (pas d'\''echec dur)' ;;
+        "monitoring") printf 'surveillance' ;;
+        "surface") printf 'surface' ;;
+        "aptdeploy") printf 'aptdeploy' ;;
+        "container") printf 'conteneur' ;;
+        "disk") printf 'disque' ;;
+        "package") printf 'paquet' ;;
+        "version") printf 'version' ;;
+        "reachability") printf 'accessibilite' ;;
+        "server-header") printf 'en-tete Server' ;;
+        "inventory") printf 'inventaire' ;;
+        "flags") printf 'options' ;;
+        "target domain") printf 'domaine cible' ;;
+        "currency") printf 'devise' ;;
+        "phases") printf 'phases' ;;
+        "access") printf 'acces' ;;
+        *)
+          local out="$s"
+          out=${out//shared library missing/bibliotheque partagee manquante}
+          out=${out//not active/non actif}
+          out=${out//does not resolve/ne resout pas}
+          out=${out//from Server header/depuis en-tete Server}
+          out=${out//public HTTPS/HTTPS public}
+          out=${out//no SSH/sans SSH}
+          out=${out//failed — see list above/echec — voir la liste ci-dessus}
+          printf '%s' "$out"
+          ;;
+      esac
+      ;;
+    de)
+      case "$s" in
+        "numbered checks this run:") printf 'Nummerierte Pruefungen dieser Ausfuehrung :' ;;
+        "numbered checks:") printf 'Nummerierte Pruefungen :' ;;
+        "(global done/total)") printf '(fertig/gesamt global)' ;;
+        "global checks:") printf 'Globale Pruefungen :' ;;
+        "failed — see list above"|"failed - see list above") printf 'fehlgeschlagen — siehe Liste oben' ;;
+        "BLOCKERS · pay/withdraw cannot finish") printf 'BLOCKER · Zahlung/Abhebung nicht moeglich' ;;
+        "BLOCKERS (pay/withdraw cannot finish)") printf 'BLOCKER (Zahlung/Abhebung nicht moeglich)' ;;
+        "ERRORS · failed checks") printf 'FEHLER · fehlgeschlagene Pruefungen' ;;
+        "ERRORS (failed checks)") printf 'FEHLER (fehlgeschlagene Pruefungen)' ;;
+        "pay/withdraw cannot finish") printf 'Zahlung/Abhebung kann nicht beendet werden' ;;
+        "SUMMARY") printf 'ZUSAMMENFASSUNG' ;;
+        "totals:") printf 'Summen :' ;;
+        "all clear") printf 'alles in Ordnung' ;;
+        "warnings only (no hard fail)") printf 'nur Warnungen (kein harter Fehler)' ;;
+        "monitoring") printf 'Ueberwachung' ;;
+        "surface") printf 'Surface' ;;
+        "aptdeploy") printf 'aptdeploy' ;;
+        "container") printf 'Container' ;;
+        "disk") printf 'Disk' ;;
+        "package") printf 'Paket' ;;
+        "version") printf 'Version' ;;
+        "reachability") printf 'Erreichbarkeit' ;;
+        "server-header") printf 'Server-Header' ;;
+        "inventory") printf 'Inventar' ;;
+        "flags") printf 'Flags' ;;
+        "target domain") printf 'Zieldomaene' ;;
+        "currency") printf 'Waehrung' ;;
+        "phases") printf 'Phasen' ;;
+        "access") printf 'Zugang' ;;
+        *)
+          local outd="$s"
+          outd=${outd//shared library missing/gemeinsame Bibliothek fehlt}
+          outd=${outd//not active/nicht aktiv}
+          outd=${outd//does not resolve/loest nicht auf}
+          outd=${outd//from Server header/aus Server-Header}
+          outd=${outd//public HTTPS/oeffentliches HTTPS}
+          outd=${outd//no SSH/ohne SSH}
+          outd=${outd//failed — see list above/fehlgeschlagen — siehe Liste oben}
+          printf '%s' "$outd"
+          ;;
+      esac
+      ;;
     *)
-      local out="$s"
-      out=${out//shared library missing/bibliotheque partagee manquante}
-      out=${out//shared library error/erreur de bibliotheque partagee}
-      out=${out//not active/non actif}
-      out=${out//after_start/apres_demarrage}
-      out=${out//catalogued but not reachable/catalogue mais inaccessible}
-      out=${out//does not resolve/ne resout pas}
-      out=${out//protocol confirmed/protocole confirme}
-      out=${out//catalogued service OK/service catalogue OK}
-      out=${out//from Server header/depuis en-tete Server}
-      out=${out//OSV clean/OSV sans vulnerabilite}
-      out=${out//actionable vuln/vuln. actionnable}
-      out=${out//header version · not Debian pkg/version en-tete · pas paquet Debian}
-      out=${out//RUN_TIMEOUT exceeded/RUN_TIMEOUT depasse}
-      out=${out//public HTTPS/HTTPS public}
-      out=${out//container status/etat du conteneur}
-      out=${out//remote-only/uniquement distant}
-      out=${out//no SSH/sans SSH}
-      out=${out//failed — see list above/echec — voir la liste ci-dessus}
-      printf '%s' "$out"
+      printf '%s' "$s"
       ;;
   esac
 }
diff --git a/ladder/README.md b/ladder/README.md
new file mode 100644
index 0000000..3c6e826
--- /dev/null
+++ b/ladder/README.md
@@ -0,0 +1,32 @@
+# Amount ladder modules
+
+**Entry:** `../check_amount_ladder.sh` via:
+
+```bash
+./taler-monitoring.sh ladder          # classic 23-rung withdraw + pay
+./taler-monitoring.sh max-ladder      # max-search withdraw + pay
+./taler-monitoring.sh goa-ladder      # legacy phase name = classic ladder
+```
+
+| Path | Role |
+|------|------|
+| `../check_amount_ladder.sh` | Orchestrator: classic / max **withdraw**, then pay phase |
+| `lib_pay.sh` | **Pay** module: free-amount template, classic pay list, max **payable** ≤ available |
+| `extract_rpubs.py` | Force-select: scrape `reservePub` from wallet dumps |
+
+There is **no** `check_goa_ladder.sh` — removed; use the phases above.
+
+## Modes
+
+- **classic** (default): 23-rung withdraw 0→mids→max-1→max, then matching pay plan (never overspend).
+- **max** (`max-ladder`): adaptive max withdraw, then adaptive max pay via public free template (hacktivism: `goa-free`).
+
+## Env (pay)
+
+| Variable | Default | Meaning |
+|----------|---------|---------|
+| `LADDER_PAY` | `1` | `0` = skip all pays |
+| `LADDER_USE_FREE_TEMPLATE` | `1` | public POST template with `{"amount":…}` |
+| `LADDER_FREE_TEMPLATE` | `goa-free` | template id (hacktivism) |
+| `LADDER_FREE_TEMPLATE_INSTANCE` | merchant instance | e.g. `goa-demo-cp4zqk` |
+| `LADDER_PAY_WAIT_AVAILABLE_S` | `90` | after withdraw, wait for spendable **available**>0; logs **withdraw board in start order** (OK/OK_BANK) with live `pendingIncoming`→`done` + timer (same wallet DB) |
diff --git a/ladder/extract_rpubs.py b/ladder/extract_rpubs.py
new file mode 100755
index 0000000..1c939e9
--- /dev/null
+++ b/ladder/extract_rpubs.py
@@ -0,0 +1,163 @@
+#!/usr/bin/env python3
+"""Extract reserve_pub candidates from wallet accept + transactions dumps.
+
+Used by check_amount_ladder.sh force-select. Kept as a file (not a bash
+heredoc) so shell refactors cannot turn Python ``continue`` into ``return``.
+
+Usage:
+  ladder_extract_rpubs.py WID AMT accept.out tx.json used-rpubs.txt
+Prints one reserve pub per line, preferred first.
+"""
+from __future__ import annotations
+
+import json
+import re
+import sys
+from pathlib import Path
+
+
+def main(argv: list[str]) -> int:
+    if len(argv) < 6:
+        print(
+            "usage: ladder_extract_rpubs.py WID AMT accept.out tx.json used-rpubs.txt",
+            file=sys.stderr,
+        )
+        return 2
+
+    wid = argv[1]
+    amt = argv[2]
+    paths = argv[3:5]
+    used_path = argv[5]
+    used: set[str] = set()
+    if Path(used_path).is_file():
+        used = {ln.strip() for ln in open(used_path) if ln.strip()}
+
+    amt_num = amt.split(":", 1)[-1] if amt else ""
+    dec = json.JSONDecoder()
+
+    def walk_collect(o, bag, ctx=None):
+        ctx = dict(ctx or {})
+        if isinstance(o, dict):
+            for k, v in o.items():
+                if not isinstance(v, str):
+                    continue
+                kl = str(k).lower().replace("-", "_")
+                if kl in (
+                    "withdrawal_id",
+                    "withdraw_id",
+                    "wopid",
+                    "id",
+                    "transactionid",
+                    "transaction_id",
+                ):
+                    ctx["id"] = v
+                elif kl in (
+                    "amount",
+                    "rawamount",
+                    "instructedamount",
+                    "amounteffective",
+                    "amountraw",
+                    "transferamount",
+                ):
+                    ctx["amount"] = v
+                elif kl in (
+                    "taler_withdraw_uri",
+                    "talerwithdrawuri",
+                    "uri",
+                    "bankconfirmationurl",
+                    "confirmtransferurl",
+                ):
+                    ctx["uri"] = v
+            for k, v in o.items():
+                kl = str(k).lower().replace("-", "_")
+                if kl in ("reserve_pub", "reservepub") and isinstance(v, str) and len(v) >= 40:
+                    bag.append((v, dict(ctx)))
+                elif not isinstance(v, (str, int, float, bool)) and v is not None:
+                    walk_collect(v, bag, ctx)
+        elif isinstance(o, list):
+            for i in o:
+                walk_collect(i, bag, ctx)
+
+    def score_ctx(pub, ctx, window=""):
+        score = 0
+        cid = str(ctx.get("id") or "")
+        camt = str(ctx.get("amount") or "")
+        curi = str(ctx.get("uri") or "")
+        if wid and wid in curi:
+            score += 120
+        if wid and wid in cid:
+            score += 100
+        if wid and wid in window:
+            score += 80
+        if amt and (camt == amt or camt.endswith(amt_num)):
+            score += 50
+        if amt and amt in window:
+            score += 30
+        if pub in used:
+            score -= 200
+        return score
+
+    def iter_json_objects(text: str):
+        i, n = 0, len(text)
+        while i < n:
+            if text[i] == "{":
+                try:
+                    o, end = dec.raw_decode(text, i)
+                    yield o
+                    i = end
+                    continue
+                except Exception:
+                    pass
+            i += 1
+
+    raw_pubs: list[str] = []
+    scored: list[tuple[int, str]] = []
+
+    for p in paths:
+        try:
+            t = open(p, errors="replace").read()
+        except Exception:
+            continue
+        for o in iter_json_objects(t):
+            bag = []
+            walk_collect(o, bag)
+            for pub, ctx in bag:
+                raw_pubs.append(pub)
+                scored.append((score_ctx(pub, ctx), pub))
+
+    rpub_pats = (
+        r"\"reservePub\"\s*:\s*\"([A-Z0-9]{40,})\"",
+        r"\"reserve_pub\"\s*:\s*\"([A-Z0-9]{40,})\"",
+        r"reserve[_ ]?pub[\"\s:=]+([A-Z0-9]{40,})",
+    )
+    for p in paths:
+        try:
+            blob = open(p, errors="replace").read()
+        except Exception:
+            continue
+        for pat in rpub_pats:
+            for m in re.finditer(pat, blob, re.I):
+                pub = m.group(1)
+                raw_pubs.append(pub)
+                window = blob[max(0, m.start() - 600) : m.end() + 600]
+                scored.append((score_ctx(pub, {}, window) + 10, pub))
+
+    order: list[str] = []
+    seen: set[str] = set()
+    for score, pub in sorted(scored, key=lambda x: x[0], reverse=True):
+        if pub in seen or pub in used:
+            continue
+        seen.add(pub)
+        order.append(pub)
+    for pub in reversed(raw_pubs):
+        if pub in seen or pub in used:
+            continue
+        seen.add(pub)
+        order.append(pub)
+    for pub in order:
+        print(pub)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main(sys.argv))
diff --git a/ladder/lib_pay.sh b/ladder/lib_pay.sh
new file mode 100644
index 0000000..01ba819
--- /dev/null
+++ b/ladder/lib_pay.sh
@@ -0,0 +1,559 @@
+# ladder/lib_pay.sh — payment side of amount ladder (sourced by check_amount_ladder.sh)
+# Expects: lib.sh + metrics.sh already sourced; CUR BANK MER INST EX SCRATCH WDB
+#           wcli wallet_avail format_amount_alt ladder_over now_ms elapsed_ms
+#           LADDER_* env, AUTH optional, PAY_TSV PAY_OK_N PAY_FAIL_N
+#
+# Public free-amount template (hacktivism): goa-free @ goa-demo-cp4zqk
+# Never spends more than wallet available.
+
+: "${LADDER_FREE_TEMPLATE:=goa-free}"
+: "${LADDER_FREE_TEMPLATE_INSTANCE:=}"
+: "${LADDER_USE_FREE_TEMPLATE:=1}"
+: "${LADDER_PAY_WAIT_AVAILABLE_S:=180}"
+: "${LADDER_PAY_SETTLE_ROUNDS:=6}"
+
+MAX_PAY_BEST_AMT="${MAX_PAY_BEST_AMT:-}"
+MAX_PAY_LO="${MAX_PAY_LO:-0}"
+MAX_PAY_HI="${MAX_PAY_HI:-0}"
+
+ladder_pay_resolve_merchant() {
+  # Sets AUTH (may be empty if free template only)
+  MPW="${E2E_MERCHANT_TOKEN:-${MERCHANT_TOKEN:-}}"
+  if [ -z "$MPW" ]; then
+    MPW=$(read_secret "taler-merchant/merchant-${INST}-password.txt" 2>/dev/null || true)
+  fi
+  if [ -z "$MPW" ]; then
+    MPW=$(read_secret "taler-merchant/merchant-goa-demo-cp4zqk-password.txt" 2>/dev/null || true)
+  fi
+  if [ -z "$MPW" ] && [ "${CUR}" = "TESTPAYSAN" ]; then
+    if [ -n "${FRANCPAYSAN_SECRETS:-}" ] && [ -f "${FRANCPAYSAN_SECRETS}/stage/default-instance-token.txt" ]; then
+      MPW=$(tr -d '\n\r' <"${FRANCPAYSAN_SECRETS}/stage/default-instance-token.txt")
+    fi
+  fi
+  AUTH=""
+  if [ -n "$MPW" ]; then
+    case "$MPW" in
+      secret-token:*) AUTH="Authorization: Bearer ${MPW}" ;;
+      *) AUTH="Authorization: Bearer secret-token:${MPW}" ;;
+    esac
+  fi
+  : "${LADDER_FREE_TEMPLATE_INSTANCE:=${INST}}"
+}
+
+# Create order for PAMT → OID OTOK PAYURI. Prefers public free template.
+ladder_create_order() {
+  local pamt="$1" ptag="$2"
+  local inst mh body tpl_https sum_json
+  OID=""; OTOK=""; PAYURI=""
+  inst="${LADDER_FREE_TEMPLATE_INSTANCE:-$INST}"
+  mh=$(python3 -c 'from urllib.parse import urlparse; print(urlparse("'"${MER}"'").hostname or "taler.hacktivism.ch")' 2>/dev/null || echo "taler.hacktivism.ch")
+
+  if [ "${LADDER_USE_FREE_TEMPLATE}" = "1" ] && [ -n "${LADDER_FREE_TEMPLATE:-}" ]; then
+    tpl_https="${MER}/instances/${inst}/templates/${LADDER_FREE_TEMPLATE}"
+    body=$(python3 -c 'import json,sys; print(json.dumps({"amount":sys.argv[1]}))' "$pamt" 2>/dev/null || printf '{"amount":"%s"}' "$pamt")
+    curl -skS -m 20 -o "$SCRATCH/ord-$ptag.json" -X POST \
+      -H 'Content-Type: application/json' \
+      -d "$body" \
+      "$tpl_https" 2>"$SCRATCH/ord-$ptag.err" || true
+    OID=$(python3 -c 'import json,re,sys;t=open(sys.argv[1]).read()
+try: print(json.loads(t).get("order_id") or "")
+except Exception:
+ m=re.search(r"\"order_id\"\s*:\s*\"([^\"]+)\"",t); print(m.group(1) if m else "")
+' "$SCRATCH/ord-$ptag.json" 2>/dev/null || true)
+    OTOK=$(python3 -c 'import json,re,sys;t=open(sys.argv[1]).read()
+try: print(json.loads(t).get("token") or "")
+except Exception:
+ m=re.search(r"\"token\"\s*:\s*\"([^\"]+)\"",t); print(m.group(1) if m else "")
+' "$SCRATCH/ord-$ptag.json" 2>/dev/null || true)
+    if [ -n "$OID" ] && [ -n "$OTOK" ]; then
+      PAYURI="taler://pay/${mh}/instances/${inst}/${OID}/?c=${OTOK}"
+      PAYURI=$(normalize_taler_uri "$PAYURI")
+      return 0
+    fi
+  fi
+
+  [ -n "${AUTH:-}" ] || return 1
+  sum_json=$(python3 -c 'import json,sys; print(json.dumps(sys.argv[1]))' "ladder pay ${pamt}" 2>/dev/null || echo '"ladder pay"')
+  curl -skS -m 20 -o "$SCRATCH/ord-$ptag.json" -X POST \
+    -H "$AUTH" -H 'Content-Type: application/json' \
+    -d "{\"order\":{\"summary\":${sum_json},\"amount\":\"${pamt}\",\"fulfillment_message\":\"ok\"},\"create_token\":true}" \
+    "${MER}/instances/${INST}/private/orders" 2>"$SCRATCH/ord-$ptag.err" || true
+  OID=$(python3 -c 'import json,re,sys;t=open(sys.argv[1]).read()
+try: print(json.loads(t).get("order_id") or "")
+except Exception:
+ m=re.search(r"\"order_id\"\s*:\s*\"([^\"]+)\"",t); print(m.group(1) if m else "")
+' "$SCRATCH/ord-$ptag.json" 2>/dev/null || true)
+  OTOK=$(python3 -c 'import json,re,sys;t=open(sys.argv[1]).read()
+try: print(json.loads(t).get("token") or "")
+except Exception:
+ m=re.search(r"\"token\"\s*:\s*\"([^\"]+)\"",t); print(m.group(1) if m else "")
+' "$SCRATCH/ord-$ptag.json" 2>/dev/null || true)
+  [ -n "$OID" ] || return 1
+  curl -skS -m 12 -o "$SCRATCH/ord-det-$ptag.json" -H "$AUTH" \
+    "${MER}/instances/${INST}/private/orders/${OID}" 2>/dev/null || true
+  PAYURI=$(python3 -c 'import json,sys
+try: print(json.load(open(sys.argv[1])).get("taler_pay_uri") or "")
+except Exception: print("")
+' "$SCRATCH/ord-det-$ptag.json" 2>/dev/null || true)
+  if [ -z "$PAYURI" ] && [ -n "$OTOK" ]; then
+    PAYURI="taler://pay/${mh}/instances/${INST}/${OID}/?c=${OTOK}"
+  fi
+  PAYURI=$(normalize_taler_uri "$PAYURI")
+  [ -n "$PAYURI" ]
+}
+
+# One pay attempt; never overspends. Always soft (no hard ladder stop).
+ladder_pay_attempt() {
+  local pamt="$1" prange="$2" prung="$3"
+  local ptag pnum bal t_pay t0 ms_order ms_handle ms_psettle after settled r pstatus pnote
+  ptag=$(printf '%s' "$pamt" | tr '.:' '__')
+  pnum=$(python3 -c 'import sys; print(sys.argv[1].split(":",1)[-1])' "$pamt")
+  t_pay=$(now_ms)
+  ms_order=0; ms_handle=0; ms_psettle=0
+  pnote=""; pstatus="FAIL"; OID="-"
+
+  bal=$(wallet_avail)
+  if ! python3 -c 'import sys; from decimal import Decimal; sys.exit(0 if Decimal(sys.argv[1])>0 and Decimal(sys.argv[1])>=Decimal(sys.argv[2]) else 1)' "$bal" "$pnum" 2>/dev/null; then
+    pstatus="SKIP_BALANCE"
+    pnote="skip: would spend more than available avail=${CUR}:${bal} need=${pamt}"
+    warn pay "pay $pamt skipped (balance)" \
+      "problem: never overspend. $(format_amount_alt "${CUR}:${bal}") free vs $(format_amount_alt "$pamt")"
+    echo -e "${prung}\t${prange}\t${pamt}\t${pstatus}\t0\t0\t0\t$(elapsed_ms "$t_pay")\t-\t${pnote}" >>"$PAY_TSV"
+    return 0
+  fi
+
+  t0=$(now_ms)
+  if ! ladder_create_order "$pamt" "$ptag"; then
+    ms_order=$(elapsed_ms "$t0")
+    pstatus="FAIL_ORDER"
+    pnote="order create failed $(head -c 100 "$SCRATCH/ord-$ptag.json" 2>/dev/null | tr '\n\"' '  ')"
+    warn pay "order $pamt" "$pnote"
+    echo -e "${prung}\t${prange}\t${pamt}\t${pstatus}\t${ms_order}\t0\t0\t$(elapsed_ms "$t_pay")\t-\t${pnote}" >>"$PAY_TSV"
+    return 0
+  fi
+  ms_order=$(elapsed_ms "$t0")
+  ok "order $OID ($pamt) ${ms_order}ms · $(format_amount_alt "$pamt")"
+
+  t0=$(now_ms)
+  if ! wcli handle-uri --yes "$PAYURI" >"$SCRATCH/pay-$ptag.out" 2>&1; then
+    ms_handle=$(elapsed_ms "$t0")
+    pstatus="FAIL_HANDLE"
+    pnote="handle-uri failed $(tail -c 100 "$SCRATCH/pay-$ptag.out" | tr '\n\"' '  ')"
+    warn pay "handle $pamt" "$pnote"
+    echo -e "${prung}\t${prange}\t${pamt}\t${pstatus}\t${ms_order}\t${ms_handle}\t0\t$(elapsed_ms "$t_pay")\t${OID}\t${pnote}" >>"$PAY_TSV"
+    return 0
+  fi
+  ms_handle=$(elapsed_ms "$t0")
+  ok "handle-uri $pamt ${ms_handle}ms"
+
+  t0=$(now_ms)
+  settled=0
+  r=0
+  while [ "$r" -lt "${LADDER_PAY_SETTLE_ROUNDS}" ]; do
+    r=$((r + 1))
+    wcli transactions >"$SCRATCH/tx-pay-$ptag.out" 2>&1 || true
+    if [ -n "${AUTH:-}" ]; then
+      curl -skS -m 8 -o "$SCRATCH/ord-paid-$ptag.json" -H "$AUTH" \
+        "${MER}/instances/${INST}/private/orders/${OID}" 2>/dev/null || true
+    fi
+    if grep -qiE 'payment|paid|Payment' "$SCRATCH/tx-pay-$ptag.out" 2>/dev/null \
+      || python3 -c 'import json,sys
+try:
+ d=json.load(open(sys.argv[1]))
+ sys.exit(0 if d.get("paid") is True or str(d.get("order_status","")).lower()=="paid" else 1)
+except Exception:
+ sys.exit(1)
+' "$SCRATCH/ord-paid-$ptag.json" 2>/dev/null; then
+      settled=1
+      break
+    fi
+    sleep 0.5
+  done
+  ms_psettle=$(elapsed_ms "$t0")
+  after=$(wallet_avail)
+  if [ "$settled" = "1" ]; then
+    pstatus="OK"
+    pnote="avail=${CUR}:${after}"
+    ok "pay settled $pamt · $(format_amount_alt "$pamt") → bal ${CUR}:${after}"
+    PAY_OK_N=$((PAY_OK_N + 1))
+    echo -e "${prung}\t${prange}\t${pamt}\t${pstatus}\t${ms_order}\t${ms_handle}\t${ms_psettle}\t$(elapsed_ms "$t_pay")\t${OID}\t${pnote}" >>"$PAY_TSV"
+    metrics_report_coins "after-pay-${ptag}" || true
+    metrics_record_flow spent "$pamt" || true
+  else
+    pstatus="FAIL_SETTLE"
+    pnote="not settled order=$OID avail=${CUR}:${after}"
+    warn pay "settle $pamt" "$pnote"
+    echo -e "${prung}\t${prange}\t${pamt}\t${pstatus}\t${ms_order}\t${ms_handle}\t${ms_psettle}\t$(elapsed_ms "$t_pay")\t${OID}\t${pnote}" >>"$PAY_TSV"
+  fi
+  return 0
+}
+
+# Print withdraw board: OK/OK_BANK rungs in *start order* + wallet tx state.
+# Uses TSV (ladder order) + transactions dump. Call after refreshing tx dump.
+# Args: left_secs available_num pendingIncoming_num
+ladder_print_wd_board() {
+  local left="${1:-?}" bal="${2:-0}" pend="${3:-0}"
+  local tsv="${TSV:-}" txf="${SCRATCH:-/tmp}/tx-wait-pay.out"
+  local board
+  board=$(
+    python3 - "$tsv" "$txf" "${CUR:-GOA}" "$left" "$bal" "$pend" <<'PY' 2>/dev/null || true
+import json, sys
+from pathlib import Path
+
+tsv, txf, cur, left, bal, pend = sys.argv[1:7]
+dec = json.JSONDecoder()
+
+def num(a: str) -> str:
+    a = (a or "").strip()
+    if ":" in a:
+        return a.split(":", 1)[1]
+    return a
+
+def iter_json(text: str):
+    i, n = 0, len(text)
+    while i < n:
+        if text[i] == "{":
+            try:
+                o, end = dec.raw_decode(text, i)
+                yield o
+                i = end
+                continue
+            except Exception:
+                pass
+        i += 1
+
+# TSV rows in start order (OK / OK_BANK only — successful bank path)
+rows = []
+if tsv and Path(tsv).is_file():
+    for ln in open(tsv, errors="replace"):
+        if not ln.strip() or ln.startswith("rung"):
+            continue
+        p = ln.rstrip("\n").split("\t")
+        if len(p) < 10:
+            continue
+        status = p[3]
+        if status not in ("OK", "OK_BANK"):
+            continue
+        rows.append({
+            "rung": p[0],
+            "amt": p[2],
+            "bank": status,
+            "wid": (p[9] if len(p) > 9 else "")[:12],
+        })
+
+# Wallet withdrawal txs (chronological if timestamp present)
+wds = []
+if txf and Path(txf).is_file():
+    try:
+        text = open(txf, errors="replace").read()
+    except Exception:
+        text = ""
+    for o in iter_json(text):
+        txs = o.get("transactions") if isinstance(o, dict) else None
+        if not isinstance(txs, list):
+            continue
+        for t in txs:
+            if not isinstance(t, dict):
+                continue
+            typ = str(t.get("type") or t.get("Type") or "").lower()
+            if "withdraw" not in typ:
+                continue
+            amt = t.get("amountRaw") or t.get("amountEffective") or t.get("amount") or ""
+            st = (
+                t.get("txState")
+                or t.get("status")
+                or t.get("pendingMajor")
+                or t.get("txMajorState")
+                or ""
+            )
+            st = str(st).lower()
+            det = t.get("withdrawalDetails") or t.get("withdrawal_details") or {}
+            if not isinstance(det, dict):
+                det = {}
+            conf = det.get("confirmed")
+            if conf is True and not st:
+                st = "done"
+            elif conf is False and not st:
+                st = "pending"
+            # pending* / dialog / kyc → still incoming; done/abort final
+            if any(x in st for x in ("done", "abort", "fail", "delete")):
+                phase = "done" if "done" in st else st
+            elif st in ("", "none", "null"):
+                phase = "unknown"
+            else:
+                phase = "pendingIncoming"
+            ts = 0
+            for k in ("timestamp", "timestamp_ms", "age"):
+                v = t.get(k)
+                if isinstance(v, dict):
+                    v = v.get("t_s") or v.get("t_ms") or v.get("t_usec")
+                try:
+                    ts = int(v or 0)
+                except Exception:
+                    ts = 0
+                if ts:
+                    break
+            wds.append({"amt": str(amt), "n": num(str(amt)), "phase": phase, "st": st or "?", "ts": ts})
+
+wds.sort(key=lambda x: x["ts"])
+used = [False] * len(wds)
+
+lines = [
+    f"withdraw board (start order) · timer {left}s left · available={cur}:{bal} · pendingIncoming={cur}:{pend}"
+]
+if not rows:
+    lines.append("  (no OK/OK_BANK rows in TSV yet)")
+else:
+    for i, r in enumerate(rows, 1):
+        n = num(r["amt"])
+        match = None
+        for j, w in enumerate(wds):
+            if used[j]:
+                continue
+            if w["n"] == n or w["amt"] == r["amt"] or w["amt"].endswith(":" + n):
+                used[j] = True
+                match = w
+                break
+        if match is None:
+            wlabel = "wallet=? (no tx match)"
+        elif match["phase"] == "pendingIncoming":
+            wlabel = f"wallet=pendingIncoming ({match['st']})"
+        elif match["phase"] == "done":
+            wlabel = f"wallet=done ({match['st']})"
+        else:
+            wlabel = f"wallet={match['phase']} ({match['st']})"
+        wid = r["wid"] or "-"
+        lines.append(
+            f"  #{i:<3} rung={r['rung']:<4} {r['amt']:<22} bank={r['bank']:<8} {wlabel}  wid={wid}"
+        )
+    pending_n = sum(1 for w in wds if w["phase"] == "pendingIncoming")
+    done_n = sum(1 for w in wds if w["phase"] == "done")
+    lines.append(
+        f"  summary: tsv_ok={len(rows)} · wallet_wd={len(wds)} (pendingIncoming={pending_n} done={done_n})"
+    )
+
+print("\n".join(lines))
+PY
+  )
+  if [ -n "$board" ]; then
+    while IFS= read -r line || [ -n "$line" ]; do
+      [ -n "$line" ] && info pay "$line"
+    done <<<"$board"
+  else
+    info pay "timer ${left}s left · available=${CUR}:${bal} · pendingIncoming=${CUR}:${pend} · same wallet (board n/a)"
+  fi
+}
+
+# Wait for spendable coins on the *same* wallet DB as withdraw (WDB).
+# After OK_BANK, available is often still 0 while pendingIncoming > 0 — pay needs available.
+# Shows withdraw board in start order + countdown (never looks hung).
+ladder_wait_available() {
+  local wait_s="${1:-$LADDER_PAY_WAIT_AVAILABLE_S}"
+  local end now left bal pend tick=0
+  end=$(( $(date +%s) + wait_s ))
+  bal=$(wallet_avail)
+  pend=$(wallet_pending_in 2>/dev/null || echo "0")
+  if python3 -c 'import sys; from decimal import Decimal; sys.exit(0 if Decimal(sys.argv[1])>0 else 1)' "$bal" 2>/dev/null; then
+    printf '%s' "$bal"
+    return 0
+  fi
+  info pay "same wallet as withdraw (db=$(basename "${WDB:-wallet}")) · available=${CUR}:${bal} · pendingIncoming=${CUR}:${pend}"
+  info pay "waiting up to ${wait_s}s for spendable available>0 — board lists OK/OK_BANK withdraws in start order"
+  # initial board before loop
+  wcli transactions >"$SCRATCH/tx-wait-pay.out" 2>&1 || true
+  ladder_print_wd_board "$wait_s" "$bal" "$pend"
+  while python3 -c 'import sys; from decimal import Decimal; sys.exit(0 if Decimal(sys.argv[1])<=0 else 1)' "$bal" 2>/dev/null; do
+    now=$(date +%s)
+    left=$(( end - now ))
+    if [ "$left" -le 0 ]; then
+      wcli transactions >"$SCRATCH/tx-wait-pay.out" 2>&1 || true
+      ladder_print_wd_board "0" "$bal" "$pend"
+      info pay "timer 0s left — stop waiting (available still ${CUR}:0)"
+      break
+    fi
+    # nudge wallet (same DB); do not run-until-done (can hang)
+    wcli transactions >"$SCRATCH/tx-wait-pay.out" 2>&1 || true
+    if command -v timeout >/dev/null 2>&1 || command -v gtimeout >/dev/null 2>&1; then
+      local _to
+      _to=$(command -v gtimeout 2>/dev/null || command -v timeout)
+      "$_to" 8 wcli advanced run-pending >"$SCRATCH/run-pending.out" 2>&1 || true
+    fi
+    # progress board every ~5s (tick every 2s → every 3rd)
+    if [ $((tick % 3)) -eq 0 ]; then
+      ladder_print_wd_board "$left" "$bal" "$pend"
+    fi
+    sleep 2
+    tick=$((tick + 1))
+    bal=$(wallet_avail)
+    pend=$(wallet_pending_in 2>/dev/null || echo "0")
+  done
+  if python3 -c 'import sys; from decimal import Decimal; sys.exit(0 if Decimal(sys.argv[1])>0 else 1)' "$bal" 2>/dev/null; then
+    wcli transactions >"$SCRATCH/tx-wait-pay.out" 2>&1 || true
+    ladder_print_wd_board "ready" "$bal" "$pend"
+    ok pay "spendable coins ready available=${CUR}:${bal} · $(format_amount_alt "${CUR}:${bal}") (same wallet)"
+  else
+    warn pay "no spendable coins after ${wait_s}s" \
+      "problem: available still ${CUR}:0 (pendingIncoming=${CUR}:${pend}). Board above shows which withdraws still pendingIncoming. Pay will SKIP_BALANCE / skip max-pay until wirewatch+wallet finish. Same wallet as withdraw."
+  fi
+  printf '%s' "$bal"
+}
+
+# Classic planned pays (PAY_LIST)
+ladder_run_pay_classic() {
+  # shellcheck disable=SC2086
+  set -- $PAY_LIST
+  local pay_n=$# prung=0 PAMT prange PNUM
+  if [ "$pay_n" -eq 0 ]; then
+    info pay "empty pay plan — skip"
+    return 0
+  fi
+  info "pay rungs" "$pay_n · $*"
+  for PAMT in "$@"; do
+    prung=$((prung + 1))
+    ladder_over && break
+    if [ "$prung" -eq 1 ]; then
+      prange="pin:0"
+    elif [ "$prung" -eq "$pay_n" ]; then
+      prange="pin:max"
+    elif [ "$prung" -eq $((pay_n - 1)) ] && [ "$pay_n" -ge 3 ]; then
+      prange="pin:max-1"
+    else
+      prange="random"
+    fi
+    section "ladder · pay rung $prung  $PAMT · $(format_amount_alt "$PAMT")"
+    PNUM=$(python3 -c 'import sys; print(sys.argv[1].split(":",1)[-1])' "$PAMT")
+    if python3 -c 'import sys; from decimal import Decimal; sys.exit(0 if Decimal(sys.argv[1])==0 else 1)' "$PNUM" 2>/dev/null; then
+      echo -e "${prung}\t${prange}\t${PAMT}\tZERO_SKIP\t0\t0\t0\t0\t-\tzero" >>"$PAY_TSV"
+      PAY_OK_N=$((PAY_OK_N + 1))
+      continue
+    fi
+    ladder_pay_attempt "$PAMT" "$prange" "$prung"
+  done
+}
+
+# Max-search: highest payable ≤ available (uses max_search_next from withdraw module)
+ladder_run_pay_max_search() {
+  local bal prung _next _anum prange _act _last _slo _shi
+  section "ladder · phase B · max-search pay (highest payable ≤ available)"
+  bal=$(ladder_wait_available "$LADDER_PAY_WAIT_AVAILABLE_S")
+  MAX_PAY_LO="0"
+  MAX_PAY_HI="$bal"
+  MAX_PAY_BEST_AMT=""
+  if ! python3 -c 'import sys; from decimal import Decimal; sys.exit(0 if Decimal(sys.argv[1])>0 else 1)' "$MAX_PAY_HI" 2>/dev/null; then
+    warn pay "max-search pay skipped" \
+      "problem: wallet available is ${CUR}:0 — need spendable coins (pendingIncoming is not enough)"
+    return 0
+  fi
+  info "max-search pay" "hi=available $(format_amount_alt "${CUR}:${MAX_PAY_HI}") · template=${LADDER_FREE_TEMPLATE}@${LADDER_FREE_TEMPLATE_INSTANCE}"
+  info "pay budget" "never order more than $(format_amount_alt "${CUR}:${MAX_PAY_HI}")"
+  prung=0
+  _last=""
+  while [ "${LADDER_MAX_PROBES}" = "0" ] || [ "$prung" -lt "${LADDER_MAX_PROBES}" ]; do
+    ladder_over && break
+    # reuse withdraw amount picker with pay bounds
+    _slo=${MAX_LO:-0}; _shi=${MAX_HI:-0}
+    MAX_LO=$MAX_PAY_LO
+    MAX_HI=$MAX_PAY_HI
+    if ! type max_search_next >/dev/null 2>&1; then
+      warn pay "max_search_next missing — cannot run pay max-search"
+      MAX_LO=$_slo; MAX_HI=$_shi
+      return 0
+    fi
+    _next=$(max_search_next)
+    MAX_LO=$_slo; MAX_HI=$_shi
+    _anum=$(printf '%s' "$_next" | cut -d'|' -f1)
+    prange=$(printf '%s' "$_next" | cut -d'|' -f2 | sed 's/^max:/pay:/')
+    _act=$(printf '%s' "$_next" | cut -d'|' -f3)
+    if [ "$_act" = "done" ]; then
+      info "max-search pay" "converged best=$(format_amount_alt "${MAX_PAY_BEST_AMT:-${CUR}:0}") · lo=$(format_amount_alt "${CUR}:${MAX_PAY_LO}") · hi=$(format_amount_alt "${CUR}:${MAX_PAY_HI}")"
+      break
+    fi
+    if ! python3 -c 'import sys; from decimal import Decimal; sys.exit(0 if Decimal(sys.argv[1])<=Decimal(sys.argv[2]) else 1)' "$_anum" "$MAX_PAY_HI" 2>/dev/null; then
+      _anum=$MAX_PAY_HI
+    fi
+    if ! python3 -c 'import sys; from decimal import Decimal; sys.exit(0 if Decimal(sys.argv[1])>0 else 1)' "$_anum" 2>/dev/null; then
+      break
+    fi
+    PAMT="${CUR}:${_anum}"
+    if [ -n "$_last" ] && [ "$PAMT" = "$_last" ]; then
+      MAX_PAY_HI=$(python3 -c 'from decimal import Decimal; a=Decimal("'"$_anum"'"); print(max(Decimal(0), a-(1 if a>=1 else Decimal("0.000001"))))')
+      continue
+    fi
+    _last=$PAMT
+    prung=$((prung + 1))
+    section "ladder · max pay probe $prung  $PAMT · $(format_amount_alt "$PAMT")  [lo=$(format_amount_alt "${CUR}:${MAX_PAY_LO}") · hi=$(format_amount_alt "${CUR}:${MAX_PAY_HI}")]"
+    ladder_pay_attempt "$PAMT" "$prange" "$prung"
+    _last_st=$(tail -1 "$PAY_TSV" 2>/dev/null | cut -f4)
+    case "$_last_st" in
+      OK)
+        MAX_PAY_LO=$_anum
+        MAX_PAY_BEST_AMT=$PAMT
+        info "max-search pay" "success bound lo=$(format_amount_alt "${CUR}:${MAX_PAY_LO}")"
+        ;;
+      SKIP_BALANCE|FAIL_ORDER|FAIL_HANDLE|FAIL_SETTLE)
+        MAX_PAY_HI=$(python3 -c 'from decimal import Decimal; a=Decimal("'"$_anum"'"); h=Decimal("'"$MAX_PAY_HI"'"); print(min(a,h))')
+        info "max-search pay" "too-high/fail hi:=$(format_amount_alt "${CUR}:${MAX_PAY_HI}") status=${_last_st}"
+        ;;
+    esac
+    if python3 -c 'from decimal import Decimal; import sys; sys.exit(0 if Decimal(sys.argv[1])>0 and Decimal(sys.argv[2])<=Decimal(sys.argv[1]) else 1)' "$MAX_PAY_LO" "$MAX_PAY_HI" 2>/dev/null; then
+      info "max-search pay" "bounds crossed — stop"
+      break
+    fi
+  done
+  if [ -n "${MAX_PAY_BEST_AMT:-}" ]; then
+    ok "max-search best payable $(format_amount_alt "$MAX_PAY_BEST_AMT") · lo=$(format_amount_alt "${CUR}:${MAX_PAY_LO}") · hi=$(format_amount_alt "${CUR}:${MAX_PAY_HI}")"
+  else
+    warn pay "max-search no successful pay" "available was ${CUR}:$(wallet_avail)"
+  fi
+  {
+    echo "best_pay=${MAX_PAY_BEST_AMT:-}"
+    echo "best_pay_alt=$(format_amount_alt "${MAX_PAY_BEST_AMT:-${CUR}:0}" 2>/dev/null || true)"
+    echo "pay_lo=${MAX_PAY_LO}"
+    echo "pay_hi=${MAX_PAY_HI}"
+  } >>"${SCRATCH}/ladder-plan.txt" 2>/dev/null || true
+  printf '%s\n' "${MAX_PAY_BEST_AMT:-}" >"${SCRATCH}/ladder-pay-plan.txt" 2>/dev/null || true
+}
+
+# Entry: glue from check_amount_ladder.sh after withdraw phase
+ladder_run_pay_phase() {
+  PAY_OK_N=0
+  PAY_FAIL_N=0
+  if [ "${LADDER_PAY}" != "1" ]; then
+    info pay "LADDER_PAY=0 — pay phase skipped"
+    return 0
+  fi
+  if [ "${FAIL_N_L:-0}" -gt 0 ]; then
+    warn pay "skipped pay ladder" "withdraw phase already failed"
+    return 0
+  fi
+
+  set_group pay
+  section "ladder · phase B · pay"
+  metrics_report_coins "before-pay-ladder" || true
+  ladder_pay_resolve_merchant
+
+  if [ -z "${AUTH:-}" ] && ! { [ "${LADDER_USE_FREE_TEMPLATE}" = "1" ] && [ -n "${LADDER_FREE_TEMPLATE:-}" ]; }; then
+    warn pay "no merchant token and free template off — skip pay"
+    return 0
+  fi
+  if [ -n "${AUTH:-}" ]; then
+    ok "merchant token" "instance ${INST}"
+  else
+    info pay "public free template ${LADDER_FREE_TEMPLATE}@${LADDER_FREE_TEMPLATE_INSTANCE} (no merchant token)"
+  fi
+
+  # Same cumulative wallet as withdraw (WDB=wallet-main.sqlite3)
+  wallet_prepare "main"
+  info pay "using same wallet as withdraw · WDB=${WDB:-?}"
+  local bal0
+  bal0=$(ladder_wait_available "$LADDER_PAY_WAIT_AVAILABLE_S")
+  info "pay budget" "wallet available ${CUR}:${bal0} · $(format_amount_alt "${CUR}:${bal0}") — never order more than this"
+
+  if [ "${LADDER_MODE}" = "max" ]; then
+    ladder_run_pay_max_search
+  else
+    ladder_run_pay_classic
+  fi
+  metrics_report_coins "after-pay-ladder" || true
+  info "pay summary" "ok=$PAY_OK_N fail=$PAY_FAIL_N tsv=$PAY_TSV"
+}
diff --git a/lib.sh b/lib.sh
index 27bbca0..9187afd 100755
--- a/lib.sh
+++ b/lib.sh
@@ -1,5 +1,22 @@
 # shellcheck shell=bash
 # Shared helpers for taler-monitoring (laptop or koopa).
+#
+# Machine paths/SSH hosts: only from env (~/.config/taler-monitoring/env or
+# TALER_MONITORING_ENV=…), installed via taler-monitoring-env — never hardcode.
+
+# Load host env once (before any := defaults)
+if [ "${_TALER_MON_ENV_LOADED:-0}" != "1" ]; then
+  _mon_cfg="${TALER_MONITORING_ENV:-${XDG_CONFIG_HOME:-$HOME/.config}/taler-monitoring/env}"
+  if [ -f "$_mon_cfg" ]; then
+    # shellcheck disable=SC1090
+    set -a
+    # shellcheck disable=SC1091
+    . "$_mon_cfg"
+    set +a
+  fi
+  unset _mon_cfg
+  export _TALER_MON_ENV_LOADED=1
+fi
 
 # Default stack = GOA / hacktivism (overridden by TALER_DOMAIN / --domain)
 # i18n (en default; fr for FrancPaysan — see i18n.sh / TALER_MON_LANG)
@@ -22,16 +39,32 @@ fi
 : "${EXCHANGE_LOCAL:=http://127.0.0.1:9011}"
 : "${MERCHANT_LOCAL:=https://127.0.0.1:9010}"
 : "${LANDING_LOCAL:=http://127.0.0.1:9013}"
-: "${KOOPA_SSH:=koopa}"
-# When LAN Host "koopa" is unreachable, try WAN DNAT (see ~/.ssh/config Host koopa-external).
-: "${KOOPA_SSH_FALLBACKS:=koopa-external}"
+# Hostnames/paths: set via ~/.config/taler-monitoring/env (no hardcoded machine values).
+: "${KOOPA_SSH:=}"
+: "${KOOPA_SSH_FALLBACKS:=}"
+: "${INSIDE_SSH:=}"
+: "${FRANCPAYSAN_SSH:=}"
+: "${DEVTESTING_SSH:=}"
+: "${DEPLOY_SSH:=}"
+: "${FIRECUDA_SSH:=}"
+: "${HTML_OUT:=${HOME}/monitoring-sites-staging}"
+: "${DEPLOY_STAGING:=${HTML_OUT}}"
+: "${DEPLOY_WWW_ROOT:=}"
+: "${APPLY_MONITORING_LIVE:=}"
+: "${KOOPA_CADDY_DIR:=}"
+: "${FRANCPAYSAN_SECRETS:=}"
+: "${FRANCPAYSAN_REMOTE_SECRETS_ROOT:=}"
+: "${FRANCPAYSAN_REMOTE_BANK_ADMIN_SECRET:=}"
+: "${FRANCPAYSAN_REMOTE_BANK_EXPLORER_SECRET:=}"
+: "${FRANCPAYSAN_REMOTE_MERCHANT_TOKEN:=}"
+: "${WALLET_CLI:=}"
 : "${MERCHANT_INSTANCE:=goa-demo-cp4zqk}"
 # Merchant portal /private API: instance ids must be lowercase or you get odd 401s.
 # (Display names may use capitals; path segment /instances/{id}/ must not.)
 MERCHANT_INSTANCE=$(printf '%s' "${MERCHANT_INSTANCE}" | tr '[:upper:]' '[:lower:]')
-: "${WITHDRAW_AMT:=GOA:20}"   # single-shot fallback; e2e ladder uses ATM notes
+: "${WITHDRAW_AMT:=GOA:20}"   # single-shot fallback; e2e ladder uses bank cash-note amounts
 : "${PAY_AMT:=GOA:0.01}"
-: "${CREDIT_AMT:=GOA:4700}"  # covers ATM ladder 20+50+100+200+4200 (paivana)
+: "${CREDIT_AMT:=GOA:4700}"  # covers bank withdraw ladder 20+50+100+200+4200 (paivana)
 : "${TIMEOUT:=12}"
 : "${E2E_TIMEOUT:=55}"       # whole e2e budget; skip rest when exceeded (e2e raises as needed)
 : "${E2E_PAY_SECS:=22}"      # dedicated seconds for pay handle-uri (avoid Alarm clock)
@@ -63,7 +96,7 @@ MERCHANT_PUBLIC=${MERCHANT_PUBLIC%/}
 # ---------------------------------------------------------------------------
 # Domain profiles → bank / exchange / merchant base URLs
 #
-# Single place to declare a stack: scripts/taler-monitoring/domains.conf
+# Single place to declare a stack: domains.conf (suite root)
 # (or TALER_DOMAINS_CONF). Each profile names the three public endpoints.
 #
 # CLI still wins after profile load:
@@ -133,13 +166,53 @@ set_taler_stack() {
   fi
 }
 
+# Normalize domains.conf UI lang token → en|fr|de (empty if not a lang token).
+# Note: de is valid for --lang / explicit profile, but no stock profile defaults to de.
+_domain_lang_token() {
+  case "${1:-}" in
+    en|EN|eng|english) printf 'en' ;;
+    fr|FR|fra|french) printf 'fr' ;;
+    de|DE|deu|ger|german|deutsch) printf 'de' ;;
+    *) printf '' ;;
+  esac
+}
+
+# Normalize regional locale → BCP 47 (fr-CH, de-CH, en-CH, …). Empty if unknown.
+# Accepts fr-CH, fr_CH, ch-FR / de-CH, ch-DE (user-friendly aliases).
+_domain_locale_token() {
+  local t="${1:-}"
+  t=$(printf '%s' "$t" | tr '[:upper:]' '[:lower:]' | tr '_' '-')
+  case "$t" in
+    fr-ch|ch-fr|fr_ch) printf 'fr-CH' ;;
+    de-ch|ch-de|de_ch) printf 'de-CH' ;;
+    en-ch|ch-en|en_ch) printf 'en-CH' ;;
+    en-us|en-gb|fr-fr|de-de|it-ch|rm-ch)
+      # pass through other xx-YY
+      printf '%s' "$t" | awk -F- '{printf "%s-%s", tolower($1), toupper($2)}'
+      ;;
+    *)
+      # bare xx-YY pattern
+      if printf '%s' "$t" | grep -qE '^[a-z]{2}-[a-z]{2}$'; then
+        printf '%s' "$t" | awk -F- '{printf "%s-%s", tolower($1), toupper($2)}'
+      else
+        printf ''
+      fi
+      ;;
+  esac
+}
+
 # Load first matching profile from domains.conf.
 # Fields (whitespace-separated; # comments; blank lines ignored):
-#   name  bank  exchange  merchant  currency  local[0|1]  landing[0|1]  [canonical_domain]
+#   name  bank  exchange  merchant  currency  local  landing  lang  locale  [canonical]
+# Legacy:
+#   … landing  lang  [canonical]           → locale defaulted
+#   … landing  [canonical]                 → lang=en, locale defaulted
+# Sets TALER_DOMAIN_LANG + TALER_DOMAIN_LOCALE from the profile.
 # Returns 0 if found, 1 if not.
 load_domain_profile() {
   local want="$1" conf="${TALER_DOMAINS_CONF:-}"
   local line name bank exchange merchant currency local_stack landing canon
+  local profile_lang profile_locale f8 f9 f10
   [ -n "$want" ] || return 1
   [ -n "$conf" ] && [ -f "$conf" ] || return 1
 
@@ -156,12 +229,55 @@ load_domain_profile() {
     currency="$5"
     local_stack="${6:-0}"
     landing="${7:-}"
-    canon="${8:-$name}"
+    f8="${8:-}"
+    f9="${9:-}"
+    f10="${10:-}"
+    canon="$name"
+    profile_lang=""
+    profile_locale=""
     # Backward compat: old 7th field was canonical domain (contains a dot)
     if [ -n "$landing" ] && [[ "$landing" == *.* && "$landing" != "0" && "$landing" != "1" ]]; then
       canon="$landing"
       landing=""
+      f8="${7:-}"
+      f9="${8:-}"
+      f10="${9:-}"
     fi
+    if [ -n "$f8" ]; then
+      profile_lang=$(_domain_lang_token "$f8")
+      if [ -n "$profile_lang" ]; then
+        # modern: lang [locale] [canonical]
+        profile_locale=$(_domain_locale_token "$f9")
+        if [ -n "$profile_locale" ]; then
+          canon="${f10:-$name}"
+        else
+          # lang + canonical (no locale column)
+          if [ -n "$f9" ] && [ -z "$(_domain_lang_token "$f9")" ]; then
+            canon="$f9"
+          else
+            canon="${f9:-$name}"
+          fi
+          profile_locale=$(_domain_locale_token "$f10")
+        fi
+      else
+        # legacy: field 8 = canonical only
+        canon="$f8"
+        profile_lang=$(_domain_lang_token "$f9")
+        profile_locale=$(_domain_locale_token "${f10:-$f9}")
+        [ -z "$profile_lang" ] && profile_lang=en
+      fi
+    fi
+    [ -z "$profile_lang" ] && profile_lang=en
+    if [ -z "$profile_locale" ]; then
+      # default regional locale: FP → fr-CH, else de-CH
+      case "$name$canon$bank$merchant" in
+        *lefrancpaysan*|*francpaysan*) profile_locale=fr-CH ;;
+        *) profile_locale=de-CH ;;
+      esac
+    fi
+    TALER_DOMAIN_LANG="$profile_lang"
+    TALER_DOMAIN_LOCALE="$profile_locale"
+    export TALER_DOMAIN_LANG TALER_DOMAIN_LOCALE
     set_taler_stack "$bank" "$exchange" "$merchant" "$currency" "$local_stack" "$landing" "$canon"
     return 0
   done <"$conf"
@@ -235,13 +351,13 @@ apply_taler_domain() {
         : "${E2E_TEMPLATE_MAP:=5=oeufs-6 4.5=pain-seigle 6=jus-pomme 8.5=fromage-chevre 12=miel-printemps 25=panier-legumes}"
         : "${E2E_PAY_VALUES:=5 8.5 6}"
         : "${E2E_WITHDRAW_VALUES:=20 50}"
-        # Inside checks as low-priv stagepaysan (podman, no sudo) — not koopa
+        # Inside checks as low-priv stagepaysan (podman, no sudo) — not koopa.
+        # INSIDE_SSH / container names: set in host env (no hardcoded SSH host).
         INSIDE_PROFILE="${INSIDE_PROFILE:-stage-lfp}"
-        INSIDE_SSH="${INSIDE_SSH:-francpaysan-stage-user}"
         INSIDE_BANK_CTR="${INSIDE_BANK_CTR:-stage-lfp-bank}"
         INSIDE_EXCHANGE_CTR="${INSIDE_EXCHANGE_CTR:-stage-lfp-exchange-ansible}"
         INSIDE_MERCHANT_CTR="${INSIDE_MERCHANT_CTR:-stage-lfp-merchant}"
-        # Host pasta ports (127.0.0.1 on francpaysan-host)
+        # Host pasta ports (127.0.0.1 on the FP host (INSIDE_SSH))
         INSIDE_BANK_PORT="${INSIDE_BANK_PORT:-9032}"
         INSIDE_EXCHANGE_PORT="${INSIDE_EXCHANGE_PORT:-9031}"
         INSIDE_MERCHANT_PORT="${INSIDE_MERCHANT_PORT:-9030}"
@@ -323,13 +439,24 @@ SSH_BASE_OPTS=(
 # Hard wall-clock timeout so ssh/curl never block the monitoring run forever.
 with_timeout() {
   local secs="$1"; shift
+  local rc
   if command -v gtimeout >/dev/null 2>&1; then
     gtimeout --kill-after=2 "$secs" "$@"
     return $?
   fi
   if command -v timeout >/dev/null 2>&1; then
-    timeout -k 2 "$secs" "$@" 2>/dev/null || timeout --kill-after=2 "$secs" "$@"
-    return $?
+    # Prefer GNU -k; if that flag form is unsupported, fall back once.
+    # Do NOT use `cmd || cmd` on the child — a phase exit 1 would re-run it.
+    set +e
+    timeout -k 2 "$secs" "$@" 2>/dev/null
+    rc=$?
+    set -e
+    # 125 = timeout(1) itself failed (bad options); 126/127 = not executable
+    if [ "$rc" -eq 125 ] || [ "$rc" -eq 126 ] || [ "$rc" -eq 127 ]; then
+      timeout --kill-after=2 "$secs" "$@"
+      return $?
+    fi
+    return "$rc"
   fi
   # Portable: perl alarm + process group kill
   perl -e '
@@ -370,7 +497,7 @@ mon_ssh_ok() {
     ssh "${SSH_BASE_OPTS[@]}" "$host" true >/dev/null 2>&1
 }
 
-# Pick a working SSH host: KOOPA_SSH first, then KOOPA_SSH_FALLBACKS (koopa-external).
+# Pick a working SSH host: KOOPA_SSH first, then KOOPA_SSH_FALLBACKS (from env).
 # Sets KOOPA_SSH to the first host that answers. 0 = ok, 1 = none.
 KOOPA_SSH_RESOLVED=0
 resolve_koopa_ssh() {
@@ -483,7 +610,13 @@ TEST_GROUP=""
 TEST_N=0
 LAST_TID=""
 # Global / progress: may continue across check_*.sh processes via TALER_MON_STATE
+# GLOBAL_*_N = whole-run status totals (multi-phase summary at parent end)
 GLOBAL_N="${GLOBAL_N:-0}"
+GLOBAL_PASS_N="${GLOBAL_PASS_N:-0}"
+GLOBAL_FAIL_N="${GLOBAL_FAIL_N:-0}"
+GLOBAL_WARN_N="${GLOBAL_WARN_N:-0}"
+GLOBAL_INFO_N="${GLOBAL_INFO_N:-0}"
+GLOBAL_BLOCK_N="${GLOBAL_BLOCK_N:-0}"
 LAST_GLOBAL=""
 PROGRESS_DONE="${PROGRESS_DONE:-0}"
 PROGRESS_TOTAL="${PROGRESS_TOTAL:-0}"
@@ -493,14 +626,28 @@ _mon_state_load() {
   [ -n "${TALER_MON_STATE:-}" ] && [ -f "${TALER_MON_STATE}" ] || return 0
   # shellcheck disable=SC1090
   . "${TALER_MON_STATE}"
+  : "${GLOBAL_N:=0}"
+  : "${GLOBAL_PASS_N:=0}"
+  : "${GLOBAL_FAIL_N:=0}"
+  : "${GLOBAL_WARN_N:=0}"
+  : "${GLOBAL_INFO_N:=0}"
+  : "${GLOBAL_BLOCK_N:=0}"
+  : "${PROGRESS_DONE:=0}"
+  : "${PROGRESS_TOTAL:=0}"
+  : "${PROGRESS_LAST_SHOWN:=0}"
 }
 _mon_state_save() {
   [ -n "${TALER_MON_STATE:-}" ] || return 0
   {
-    printf 'GLOBAL_N=%s\n' "$GLOBAL_N"
-    printf 'PROGRESS_DONE=%s\n' "$PROGRESS_DONE"
-    printf 'PROGRESS_TOTAL=%s\n' "$PROGRESS_TOTAL"
-    printf 'PROGRESS_LAST_SHOWN=%s\n' "$PROGRESS_LAST_SHOWN"
+    printf 'GLOBAL_N=%s\n' "${GLOBAL_N:-0}"
+    printf 'GLOBAL_PASS_N=%s\n' "${GLOBAL_PASS_N:-0}"
+    printf 'GLOBAL_FAIL_N=%s\n' "${GLOBAL_FAIL_N:-0}"
+    printf 'GLOBAL_WARN_N=%s\n' "${GLOBAL_WARN_N:-0}"
+    printf 'GLOBAL_INFO_N=%s\n' "${GLOBAL_INFO_N:-0}"
+    printf 'GLOBAL_BLOCK_N=%s\n' "${GLOBAL_BLOCK_N:-0}"
+    printf 'PROGRESS_DONE=%s\n' "${PROGRESS_DONE:-0}"
+    printf 'PROGRESS_TOTAL=%s\n' "${PROGRESS_TOTAL:-0}"
+    printf 'PROGRESS_LAST_SHOWN=%s\n' "${PROGRESS_LAST_SHOWN:-0}"
   } >"${TALER_MON_STATE}"
 }
 _mon_state_load
@@ -531,7 +678,7 @@ set_area() {
 }
 # Start a logical sub-group. Short stable names:
 #   exchange bank merchant perf stats landing paivana
-#   prereq wallet atm settle pay shop dig load report
+#   prereq wallet bankwd settle pay shop dig load report
 #   ssh caddy outside inside compare withdraw
 # Counter continues if the same group is re-entered later in the area.
 # Prints a compact group chip so log + issue text map cleanly (www.bank-03).
@@ -563,8 +710,9 @@ set_progress_total() {
   # Optional: expected number of numbered checks in this run.
   # PROGRESS_TOTAL=0 → no percent, only "done=N".
   # Does not reset GLOBAL_N / PROGRESS_DONE if already mid-run (state file).
+  # May shrink when parent re-fits after a short phase (total = done + remaining est),
+  # but never below PROGRESS_DONE.
   local n="${1:-0}"
-  # Never shrink below already completed work (re-estimate mid-run safe)
   if [ "${PROGRESS_DONE:-0}" -gt 0 ] && [ "$n" -gt 0 ] && [ "$n" -lt "$PROGRESS_DONE" ]; then
     n=$PROGRESS_DONE
   fi
@@ -579,20 +727,38 @@ add_progress_total() {
   PROGRESS_TOTAL=$((PROGRESS_TOTAL + n))
   _mon_state_save
 }
-# When estimate was short, grow total so we never print done>total (e.g. 224/205).
-# Keeps ~10% headroom so the bar is not stuck at 100% while checks continue.
+# When the global estimate was short, grow total just enough so done≤total.
+# Never shrink. Mid-run headroom is small and monotonic (stable N in done/N).
 _progress_rebalance() {
   [ "${PROGRESS_TOTAL:-0}" -gt 0 ] || return 0
   [ "${PROGRESS_DONE:-0}" -gt 0 ] || return 0
   if [ "$PROGRESS_DONE" -gt "$PROGRESS_TOTAL" ]; then
-    local head=$((PROGRESS_DONE / 10))
-    [ "$head" -lt 12 ] && head=12
+    # + max(8, 5% of done) so we do not snap to 100% then jump again next check
+    local head=$((PROGRESS_DONE / 20))
+    [ "$head" -lt 8 ] && head=8
     PROGRESS_TOTAL=$((PROGRESS_DONE + head))
+    _mon_state_save
+  fi
+}
+# Snap total to actual done — only at whole-run end (not after each phase).
+progress_finish() {
+  [ "${PROGRESS_OFF:-0}" = "1" ] && return 0
+  [ "${PROGRESS_DONE:-0}" -gt 0 ] || return 0
+  if [ "${PROGRESS_TOTAL:-0}" -ne "$PROGRESS_DONE" ]; then
+    PROGRESS_TOTAL=$PROGRESS_DONE
+    _mon_state_save
+  fi
+  _progress_bar_line "$PROGRESS_DONE" "$PROGRESS_TOTAL"
+  if [ "${GLOBAL_N:-0}" -gt 0 ]; then
+    printf -- '%s %s %d/%d (#001…#%03d)%s\n' \
+      "$D" "$(i18n_text 'global checks:')" \
+      "$PROGRESS_DONE" "$PROGRESS_TOTAL" "$GLOBAL_N" "$N"
   fi
 }
 _progress_bar_line() {
-  # Format (grows left → right; badge = percent, not "PROG"):
+  # Format (grows left → right; badge = percent):
   #   ┌  42% ┐ ████████░░░░░░░░░░░░░░░░  31/74
+  # done/total = global numbered checks so far / global expected total
   local done="$1" total="$2" width=28 pct=0 filled empty i bar pct_lab
   # Defensive: never display done > total
   if [ "$total" -gt 0 ] && [ "$done" -gt "$total" ]; then
@@ -625,7 +791,7 @@ _progress_bar_line() {
     while [ "$i" -lt "$empty" ]; do bar="${bar}░"; i=$((i + 1)); done
     pct_lab=$(printf '%3d%%' "$pct")
     if [ "${BOX:-0}" = "1" ]; then
-      # badge = percent (clearer than "PROG"); bar grows ░→█ left to right
+      # badge = percent; bar grows ░→█ left to right; N/M = global check count
       printf '%s┌%s┐%s %s%s%s  %s%d/%d%s\n' \
         "$BG_PROG" "$pct_lab" "$N" \
         "$C" "$bar" "$N" \
@@ -636,10 +802,10 @@ _progress_bar_line() {
     fi
   else
     if [ "${BOX:-0}" = "1" ]; then
-      printf '%s┌  …  ┐%s %sdone=%d%s (set PROGRESS_TOTAL= for %%)\n' \
+      printf '%s┌  …  ┐%s %s%d/?%s (set PROGRESS_TOTAL= for global %%)\n' \
         "$BG_PROG" "$N" "$D" "$done" "$N"
     else
-      printf -- '%sdone=%d%s (set PROGRESS_TOTAL= for %%)\n' "$D" "$done" "$N"
+      printf -- '%s%d/?%s (set PROGRESS_TOTAL= for global %%)\n' "$D" "$done" "$N"
     fi
   fi
 }
@@ -720,6 +886,8 @@ ok() {
   _take_tid
   _msg_line "$BG_OK" "OK" "$G" "$label" "$detail"
   PASS_N=$((PASS_N + 1))
+  GLOBAL_PASS_N=$((GLOBAL_PASS_N + 1))
+  _mon_state_save
 }
 # component-scoped error: err bank "what failed" "why / HTTP / path"
 err() {
@@ -727,7 +895,9 @@ err() {
   _take_tid
   _msg_line "$BG_ERR" "ERROR" "$R" "${comp}: ${msg}" "$detail"
   FAIL_N=$((FAIL_N + 1))
+  GLOBAL_FAIL_N=$((GLOBAL_FAIL_N + 1))
   ERRORS+=("${LAST_TID:+$LAST_TID }[$comp] $msg${detail:+ · $detail}")
+  _mon_state_save
 }
 # fail "what failed" ["why / HTTP code / path"]
 fail() {
@@ -735,7 +905,9 @@ fail() {
   _take_tid
   _msg_line "$BG_ERR" "ERROR" "$R" "$label" "$detail"
   FAIL_N=$((FAIL_N + 1))
+  GLOBAL_FAIL_N=$((GLOBAL_FAIL_N + 1))
   ERRORS+=("${LAST_TID:+$LAST_TID }$label${detail:+ · $detail}")
+  _mon_state_save
 }
 warn() {
   # warn "what is soft-bad" ["why still ok to continue"]
@@ -756,6 +928,8 @@ warn() {
   fi
   _msg_line "$BG_WARN" "WARN" "$Y" "$head" "$detail"
   WARN_N=$((WARN_N + 1))
+  GLOBAL_WARN_N=$((GLOBAL_WARN_N + 1))
+  _mon_state_save
 }
 info() {
   # info "topic" ["concrete fact / value / next step"]
@@ -763,6 +937,8 @@ info() {
   _take_tid
   _msg_line "$BG_INFO" "INFO" "$C" "$label" "$detail"
   INFO_N=$((INFO_N + 1))
+  GLOBAL_INFO_N=$((GLOBAL_INFO_N + 1))
+  _mon_state_save
 }
 blocker() {
   # Hard stop on pay/withdraw path: blocker "step" "why it cannot continue"
@@ -771,7 +947,10 @@ blocker() {
   _msg_line "$BG_BLK" "BLOCKER" "$M" "${step}" "$msg"
   BLOCKERS+=("${LAST_TID:+$LAST_TID }[$step] $msg")
   FAIL_N=$((FAIL_N + 1))
+  GLOBAL_FAIL_N=$((GLOBAL_FAIL_N + 1))
+  GLOBAL_BLOCK_N=$((GLOBAL_BLOCK_N + 1))
   ERRORS+=("BLOCKER ${LAST_TID:+$LAST_TID }[$step] $msg")
+  _mon_state_save
 }
 section() {
   # Boxed section header (3 lines when colour on)
@@ -797,20 +976,156 @@ section() {
   fi
 }
 
+# Print SUMMARY totals box for given counts (phase-local or global).
+# Each severity row is colour-distinct: badge + full-line bg tint + bold count (v1.14.0).
+# $1=pass $2=fail $3=warn $4=info $5=block
+_summary_totals_box() {
+  local pass_n="$1" fail_n="$2" warn_n="$3" info_n="$4" blk_n="$5"
+  local hdr_bg hdr_fg _sum_lbl
+  _sum_lbl=$(i18n_tag SUMMARY)
+  # Header colour follows overall verdict
+  if [ "$fail_n" -gt 0 ] || [ "$blk_n" -gt 0 ]; then
+    hdr_bg="${BG_ERR}"
+    hdr_fg="${R}"
+  elif [ "$warn_n" -gt 0 ]; then
+    hdr_bg="${BG_WARN}"
+    hdr_fg="${Y}"
+  else
+    hdr_bg="${BG_OK}"
+    hdr_fg="${G}"
+  fi
+
+  if [ "${BOX:-0}" = "1" ]; then
+    # Dim full-line tints (bold fg on soft bg) — distinct per severity
+    local ROW_OK=$'\e[1;32;48;5;22m'    # bold green on dark green
+    local ROW_ERR=$'\e[1;97;48;5;52m'   # bold white on dark red
+    local ROW_WARN=$'\e[1;30;48;5;178m' # bold black on gold
+    local ROW_INFO=$'\e[1;97;48;5;24m'  # bold white on dark blue
+    local ROW_BLK=$'\e[1;97;48;5;53m'   # bold white on dark magenta
+    local ROW_TOT=$'\e[1;37;48;5;236m'  # bold white on dark gray
+
+    # Include SUMMARY on all three frame lines so HTML classify maps them to sum-header
+    # (phase section() boxes use the same ╔═║╚ glyphs but must not get sum-* paint).
+    printf -- '\n%s╔════════ %s ════════════════════╗%s\n' "$hdr_bg" "$_sum_lbl" "$N"
+    printf -- '%s║  %-34s  ║%s\n' "$hdr_bg" "$_sum_lbl" "$N"
+    printf -- '%s╚════════ %s ════════════════════╝%s\n' "$hdr_bg" "$_sum_lbl" "$N"
+
+    # $1=row_bg $2=badge_bg $3=tag $4=n $5=label_fg — full-width-ish row
+    _sum_row() {
+      local row_bg="$1" badge="$2" tag="$3" n="$4" fg="$5" _tg pad
+      _tg=$(i18n_tag "$tag")
+      pad=$(printf '%-5s' "$_tg")
+      printf -- '  %s┌ %s┐%s %s %s%6d%s %s%-12s%s\n' \
+        "$badge" "$pad" "$N" \
+        "$row_bg" "$B" "$n" "$N" \
+        "$fg$B" "$_tg" "$N"
+    }
+    _sum_row "$ROW_OK" "$BG_OK" "OK" "$pass_n" "$G"
+    [ "$fail_n" -gt 0 ] && _sum_row "$ROW_ERR" "$BG_ERR" "ERROR" "$fail_n" "$R"
+    [ "$warn_n" -gt 0 ] && _sum_row "$ROW_WARN" "$BG_WARN" "WARN" "$warn_n" "$Y"
+    [ "$info_n" -gt 0 ] && _sum_row "$ROW_INFO" "$BG_INFO" "INFO" "$info_n" "$C"
+    [ "$blk_n" -gt 0 ] && _sum_row "$ROW_BLK" "$BG_BLK" "BLOCK" "$blk_n" "$M"
+
+    # totals rollup on tinted bar
+    printf -- '  %s %s %s %s%d OK%s' "$ROW_TOT" "$B" "$(i18n_text 'totals:')" "$G$B" "$pass_n" "$N$ROW_TOT"
+    [ "$fail_n" -gt 0 ] && printf -- ' %s·%s %s%d ERROR%s' "$D" "$N$ROW_TOT" "$R$B" "$fail_n" "$N$ROW_TOT"
+    [ "$warn_n" -gt 0 ] && printf -- ' %s·%s %s%d WARN%s' "$D" "$N$ROW_TOT" "$Y$B" "$warn_n" "$N$ROW_TOT"
+    [ "$info_n" -gt 0 ] && printf -- ' %s·%s %s%d INFO%s' "$D" "$N$ROW_TOT" "$C$B" "$info_n" "$N$ROW_TOT"
+    [ "$blk_n" -gt 0 ] && printf -- ' %s·%s %s%d BLOCKER%s' "$D" "$N$ROW_TOT" "$M$B" "$blk_n" "$N$ROW_TOT"
+    printf -- ' %s\n' "$N"
+  else
+    printf -- '\n[ %s ]\n' "$_sum_lbl"
+    printf -- '  [ %-5s ] %s%4d%s\n' "$(i18n_tag OK)" "$G$B" "$pass_n" "$N"
+    [ "$fail_n" -gt 0 ] && printf -- '  [ %-5s ] %s%4d%s\n' "$(i18n_tag ERROR)" "$R$B" "$fail_n" "$N"
+    [ "$warn_n" -gt 0 ] && printf -- '  [ %-5s ] %s%4d%s\n' "$(i18n_tag WARN)" "$Y$B" "$warn_n" "$N"
+    [ "$info_n" -gt 0 ] && printf -- '  [ %-5s ] %s%4d%s\n' "$(i18n_tag INFO)" "$C$B" "$info_n" "$N"
+    [ "$blk_n" -gt 0 ] && printf -- '  [ %-5s ] %s%4d%s\n' "$(i18n_tag BLOCK)" "$M$B" "$blk_n" "$N"
+    printf -- '  %s%s%s %s%d OK%s' "$B" "$(i18n_text 'totals:')" "$N" "$G$B" "$pass_n" "$N"
+    [ "$fail_n" -gt 0 ] && printf -- ' · %s%d ERROR%s' "$R$B" "$fail_n" "$N"
+    [ "$warn_n" -gt 0 ] && printf -- ' · %s%d WARN%s' "$Y$B" "$warn_n" "$N"
+    [ "$info_n" -gt 0 ] && printf -- ' · %s%d INFO%s' "$C$B" "$info_n" "$N"
+    [ "$blk_n" -gt 0 ] && printf -- ' · %s%d BLOCKER%s' "$M$B" "$blk_n" "$N"
+    printf '\n'
+  fi
+  # overall verdict — bold label, filled badge (i18n for FR console/HTML)
+  local _ok_msg _warn_msg _fail_msg
+  _ok_msg=$(i18n_text 'all clear')
+  _warn_msg=$(i18n_text 'warnings only (no hard fail)')
+  if [ "$fail_n" -eq 0 ] && [ "$blk_n" -eq 0 ] && [ "$warn_n" -eq 0 ]; then
+    if [ "${BOX:-0}" = "1" ]; then
+      printf -- '  %s┌ OK    ┐%s %s%s%s%s\n' "$BG_OK" "$N" "$G" "$B" "$_ok_msg" "$N"
+    else
+      printf -- '  %s[ OK    ] %s%s%s\n' "$G" "$B" "$_ok_msg" "$N"
+    fi
+  elif [ "$fail_n" -eq 0 ] && [ "$blk_n" -eq 0 ]; then
+    if [ "${BOX:-0}" = "1" ]; then
+      printf -- '  %s┌ WARN  ┐%s %s%s%s%s\n' "$BG_WARN" "$N" "$Y" "$B" "$_warn_msg" "$N"
+    else
+      printf -- '  %s[ WARN  ] %s%s%s\n' "$Y" "$B" "$_warn_msg" "$N"
+    fi
+  else
+    _fail_msg=$(i18n_text 'failed — see list above')
+    if [ "${BOX:-0}" = "1" ]; then
+      printf -- '  %s┌ ERROR ┐%s %s%s%s%s\n' "$BG_ERR" "$N" "$R" "$B" "$_fail_msg" "$N"
+    else
+      printf -- '  %s[ ERROR ] %s%s%s\n' "$R" "$B" "$_fail_msg" "$N"
+    fi
+  fi
+}
+
+# Whole-run SUMMARY from TALER_MON_STATE (parent after all phases).
+# Multi-phase: phases never print SUMMARY totals; only this does.
+summary_global() {
+  _mon_state_load
+  local pass_n="${GLOBAL_PASS_N:-0}"
+  local fail_n="${GLOBAL_FAIL_N:-0}"
+  local warn_n="${GLOBAL_WARN_N:-0}"
+  local info_n="${GLOBAL_INFO_N:-0}"
+  local blk_n="${GLOBAL_BLOCK_N:-0}"
+  # Progress bar + numbered checks already printed by progress_finish when present.
+  if [ "${GLOBAL_N:-0}" -gt 0 ]; then
+    if [ "${PROGRESS_TOTAL:-0}" -gt 0 ]; then
+      printf -- '%s %s #001…#%03d · %d/%d %s%s\n' \
+        "$D" "$(i18n_text 'numbered checks:')" "$GLOBAL_N" \
+        "${PROGRESS_DONE:-0}" "$PROGRESS_TOTAL" \
+        "$(i18n_text '(global done/total)')" "$N"
+    else
+      printf -- '%s %s #001…#%03d%s\n' \
+        "$D" "$(i18n_text 'numbered checks this run:')" "$GLOBAL_N" "$N"
+    fi
+  fi
+  _summary_totals_box "$pass_n" "$fail_n" "$warn_n" "$info_n" "$blk_n"
+  [ "$fail_n" -eq 0 ]
+}
+
 summary() {
   local blk_n=${#BLOCKERS[@]}
-  echo ""
-  # final progress line (no _take_tid — plain printf)
-  if [ "${PROGRESS_OFF:-0}" != "1" ] && [ "$PROGRESS_DONE" -gt 0 ]; then
-    # Snap total to actual so last line is exact N/N 100%
-    if [ "$PROGRESS_TOTAL" -gt 0 ] && [ "$PROGRESS_DONE" -ne "$PROGRESS_TOTAL" ]; then
-      PROGRESS_TOTAL=$PROGRESS_DONE
-      _mon_state_save
-    fi
-    _progress_bar_line "$PROGRESS_DONE" "$PROGRESS_TOTAL"
+  # Multi-phase runs: never print phase SUMMARY / progress standings (v1.13.10+).
+  # Parent prints one global SUMMARY via summary_global after progress_finish.
+  # Solo check_*.sh (TALER_MON_MULTI_PHASE unset/0) still full summary.
+  # Phase still lists ERRORS/BLOCKERS so mid-run failures stay visible.
+  local _full=1
+  if [ "${TALER_MON_MULTI_PHASE:-0}" = "1" ]; then
+    _full=0
   fi
-  if [ "$GLOBAL_N" -gt 0 ]; then
-    printf -- '%s %s #001…#%03d%s\n' "$D" "$(i18n_text 'numbered checks this run:')" "$GLOBAL_N" "$N"
+
+  echo ""
+  if [ "$_full" = "1" ]; then
+    # Progress snapshot — whole-run snap also in progress_finish.
+    if [ "${PROGRESS_OFF:-0}" != "1" ] && [ "${PROGRESS_DONE:-0}" -gt 0 ]; then
+      _progress_rebalance
+      _progress_bar_line "$PROGRESS_DONE" "$PROGRESS_TOTAL"
+    fi
+    if [ "$GLOBAL_N" -gt 0 ]; then
+      if [ "${PROGRESS_TOTAL:-0}" -gt 0 ]; then
+        printf -- '%s %s #001…#%03d · %d/%d %s%s\n' \
+          "$D" "$(i18n_text 'numbered checks:')" "$GLOBAL_N" \
+          "$PROGRESS_DONE" "$PROGRESS_TOTAL" \
+          "$(i18n_text '(global done/total)')" "$N"
+      else
+        printf -- '%s %s #001…#%03d%s\n' "$D" "$(i18n_text 'numbered checks this run:')" "$GLOBAL_N" "$N"
+      fi
+    fi
   fi
   if [ "$blk_n" -gt 0 ]; then
     if [ "${BOX:-0}" = "1" ]; then
@@ -836,54 +1151,13 @@ summary() {
     done
   fi
 
-  # Coloured totals — same severity badges as check lines (always; NO_COLOR blanks ANSI)
-  if [ "${BOX:-0}" = "1" ]; then
-    printf -- '\n%s┌ %s ┐%s\n' "$BG_SEC" "$(i18n_tag SUMMARY)" "$N"
-    _sum_badge() { # $1=bg $2=tag $3=fg $4=n
-      local _tg
-      _tg=$(i18n_tag "$2")
-      printf -- '  %s┌ %-5s┐%s %s%4d%s\n' "$1" "$_tg" "$N" "$3" "$4" "$N"
-    }
-    _sum_badge "$BG_OK" "OK" "$G" "$PASS_N"
-    [ "$FAIL_N" -gt 0 ] && _sum_badge "$BG_ERR" "ERROR" "$R" "$FAIL_N"
-    [ "$WARN_N" -gt 0 ] && _sum_badge "$BG_WARN" "WARN" "$Y" "$WARN_N"
-    [ "$INFO_N" -gt 0 ] && _sum_badge "$BG_INFO" "INFO" "$C" "$INFO_N"
-    [ "$blk_n" -gt 0 ] && _sum_badge "$BG_BLK" "BLOCK" "$M" "$blk_n"
-  else
-    printf -- '\n[ %s ]\n' "$(i18n_tag SUMMARY)"
-    printf -- '  [ %-5s ] %s%4d%s\n' "$(i18n_tag OK)" "$G" "$PASS_N" "$N"
-    [ "$FAIL_N" -gt 0 ] && printf -- '  [ %-5s ] %s%4d%s\n' "$(i18n_tag ERROR)" "$R" "$FAIL_N" "$N"
-    [ "$WARN_N" -gt 0 ] && printf -- '  [ %-5s ] %s%4d%s\n' "$(i18n_tag WARN)" "$Y" "$WARN_N" "$N"
-    [ "$INFO_N" -gt 0 ] && printf -- '  [ %-5s ] %s%4d%s\n' "$(i18n_tag INFO)" "$C" "$INFO_N" "$N"
-    [ "$blk_n" -gt 0 ] && printf -- '  [ %-5s ] %s%4d%s\n' "$(i18n_tag BLOCK)" "$M" "$blk_n" "$N"
-  fi
-  # one-line rollup (bold label, severity-coloured counts)
-  printf -- '  %stotals:%s %s%d OK%s' "$B" "$N" "$G" "$PASS_N" "$N"
-  [ "$FAIL_N" -gt 0 ] && printf -- ' · %s%d ERROR%s' "$R" "$FAIL_N" "$N"
-  [ "$WARN_N" -gt 0 ] && printf -- ' · %s%d WARN%s' "$Y" "$WARN_N" "$N"
-  [ "$INFO_N" -gt 0 ] && printf -- ' · %s%d INFO%s' "$C" "$INFO_N" "$N"
-  [ "$blk_n" -gt 0 ] && printf -- ' · %s%d BLOCKER%s' "$M" "$blk_n" "$N"
-  printf '\n'
-  # overall verdict
-  if [ "$FAIL_N" -eq 0 ] && [ "$blk_n" -eq 0 ] && [ "$WARN_N" -eq 0 ]; then
-    if [ "${BOX:-0}" = "1" ]; then
-      printf -- '  %s┌ OK    ┐%s %sall clear%s\n' "$BG_OK" "$N" "$G" "$N"
-    else
-      printf -- '  %s[ OK    ] all clear%s\n' "$G" "$N"
-    fi
-  elif [ "$FAIL_N" -eq 0 ] && [ "$blk_n" -eq 0 ]; then
-    if [ "${BOX:-0}" = "1" ]; then
-      printf -- '  %s┌ WARN  ┐%s %swarnings only (no hard fail)%s\n' "$BG_WARN" "$N" "$Y" "$N"
-    else
-      printf -- '  %s[ WARN  ] warnings only (no hard fail)%s\n' "$Y" "$N"
-    fi
-  else
-    if [ "${BOX:-0}" = "1" ]; then
-      printf -- '  %s┌ ERROR ┐%s %s"$(i18n_text "failed — see list above")"%s\n' "$BG_ERR" "$N" "$R" "$N"
-    else
-      printf -- '  %s[ ERROR ] "$(i18n_text "failed — see list above")"%s\n' "$R" "$N"
-    fi
+  # Coloured totals — only at full phase summary (solo script / single-phase)
+  if [ "$_full" != "1" ]; then
+    # Still fail the phase exit when this phase had errors (set -e + trailing summary)
+    [ "$FAIL_N" -eq 0 ]
+    return
   fi
+  _summary_totals_box "$PASS_N" "$FAIL_N" "$WARN_N" "$INFO_N" "$blk_n"
   [ "$FAIL_N" -eq 0 ]
 }
 
@@ -1152,26 +1426,8 @@ if [ -z "$SECRETS_ROOT" ]; then
     SECRETS_ROOT="${KOOPA_ADMIN_SECRETS}/koopa/host-root"
   fi
 fi
-if [ -z "$SECRETS_ROOT" ]; then
-  _mon_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-  _admin_log="$(cd "${_mon_dir}/../.." && pwd)"
-  for d in \
-    "${_admin_log}/../koopa-admin-secrets/koopa/host-root" \
-    "${_admin_log}/../../koopa-admin-secrets/koopa/host-root" \
-    "${HOME}/taler/src/koopa-admin-secrets/koopa/host-root" \
-    "${HOME}/src/koopa/koopa-admin-secrets/koopa/host-root" \
-    "${HOME}/koopa-admin-secrets/koopa/host-root" \
-    "/Users/newkamek/src/koopa/koopa-admin-secrets/koopa/host-root" \
-    "$(cd "${_admin_log}/../.." 2>/dev/null && pwd)/koopa-admin-secrets/koopa/host-root"
-  do
-    # resolve .. without requiring the dir to exist first for cd
-    if [ -d "$d/taler-bank" ]; then
-      SECRETS_ROOT=$(cd "$d" && pwd)
-      break
-    fi
-  done
-  unset _mon_dir _admin_log d
-fi
+# No further home-path probes — set SECRETS_ROOT or KOOPA_ADMIN_SECRETS in env.
+
 
 read_secret() {
   local rel="$1" base val=""
@@ -1214,9 +1470,9 @@ read_secret() {
 secrets_hint() {
   cat </dev/null || printf '%s' "$u"
+}
+
+# True (exit 0) if URI still has default port in taler:// authority.
+taler_uri_has_default_port() {
+  # host:443 or host:80 before next path/query/end
+  python3 -c "import re,sys; u=sys.argv[1] if len(sys.argv)>1 else ''; sys.exit(0 if re.search(r'taler://\\S+:(443|80)(/|$|\\?|#)', u, re.I) else 1)" "${1-}" 2>/dev/null
+}
+
 find_wallet_cli() {
   # Must return a path suitable for: node "$WALLET_CLI" …
   # Prefer *.mjs / bundled entry; plain /usr/bin/taler-wallet-cli is often a
@@ -1256,16 +1559,20 @@ find_wallet_cli() {
     echo "$WALLET_CLI"; return 0
   fi
   local c cand
-  for c in \
+  # WALLET_CLI / WALLET_CLI_CANDIDATES from env; then portable package paths.
+  # Prefer local ts-core / mon-env builds (schema-compatible) over stale system
+  # wrappers (e.g. /usr/local 1.3.x vs DB schema 29 from 1.6.x).
+  # shellcheck disable=SC2086
+  for c in ${WALLET_CLI_CANDIDATES:-} \
     "${HOME}/.local/bin/taler-wallet-cli.mjs" \
-    "${HOME}/taler/src/taler-typescript-core/packages/taler-wallet-cli/bin/taler-wallet-cli.mjs" \
-    "${HOME}/src/taler-typescript-core/packages/taler-wallet-cli/bin/taler-wallet-cli.mjs" \
-    "${HOME}/taler/opt/taler-wallet-cli/usr/lib/taler-wallet-cli/node_modules/taler-wallet-cli/bin/taler-wallet-cli.mjs" \
-    /home/hernani/taler/opt/taler-wallet-cli/usr/lib/taler-wallet-cli/node_modules/taler-wallet-cli/bin/taler-wallet-cli.mjs \
-    /home/hernani/taler/src/taler-typescript-core/packages/taler-wallet-cli/bin/taler-wallet-cli.mjs \
-    /Users/newkamek/src/taler/taler-typescript-core/packages/taler-wallet-cli/bin/taler-wallet-cli.mjs \
+    "${HOME}/src/taler/taler-typescript-core/packages/taler-wallet-cli/bin/taler-wallet-cli.mjs" \
+    "${HOME}/src/taler/taler-typescript-core/packages/taler-wallet-cli/dist/taler-wallet-cli.mjs" \
+    "${HOME}/src/taler/taler-typescript-core/packages/taler-wallet-cli/dist/taler-wallet-cli-local.mjs" \
+    "${HOME}/src/taler/taler-typescript-core/packages/taler-wallet-cli/dist/taler-wallet-cli-bundled.cjs" \
     /usr/lib/taler-wallet-cli/node_modules/taler-wallet-cli/bin/taler-wallet-cli.mjs \
-    /usr/share/taler-wallet-cli/bin/taler-wallet-cli.mjs
+    /usr/share/taler-wallet-cli/bin/taler-wallet-cli.mjs \
+    /usr/local/lib/taler-wallet-cli/node_modules/taler-wallet-cli/bin/taler-wallet-cli.mjs \
+    /usr/local/lib/taler-wallet-cli/node_modules/taler-wallet-cli/dist/taler-wallet-cli-bundled.cjs
   do
     [ -n "$c" ] && [ -f "$c" ] && { echo "$c"; return 0; }
   done
@@ -1281,6 +1588,14 @@ find_wallet_cli() {
       c=$(readlink -f "$cand" 2>/dev/null || true)
       [ -n "$c" ] && [ -f "$c" ] && { echo "$c"; return 0; }
     fi
+    # macOS /usr/local wrapper often runs an older bundle; prefer its bundled.cjs only
+    # when no newer candidate above matched (already returned).
+    for c in \
+      /usr/local/lib/taler-wallet-cli/node_modules/taler-wallet-cli/dist/taler-wallet-cli-bundled.cjs \
+      /usr/local/lib/node_modules/taler-wallet-cli/dist/taler-wallet-cli-bundled.cjs
+    do
+      [ -f "$c" ] && { echo "$c"; return 0; }
+    done
   fi
   return 1
 }
diff --git a/mail-catalog.conf b/mail-catalog.conf
new file mode 100644
index 0000000..f489234
--- /dev/null
+++ b/mail-catalog.conf
@@ -0,0 +1,20 @@
+# mail-catalog.conf — domains and mail hosts for phase `mail` (v1.8.0+)
+#
+# Format (whitespace-separated; # comments):
+#   domain   expected_mx (comma)   mail_hosts (comma)   smtp_ports   imap_ports   label
+#
+# Mail planes (outside-in only — no host login):
+#   • firefly.gnunet.org     — taler.net + gnunet.org
+#   • mail.anastasis.lu      — Anastasis / TSA (A → pixel.taler-systems.com)
+#     (legacy catalog host anastasis.taler-systems.com does NOT resolve)
+#
+# Override path: MAIL_CATALOG=/path/to/file
+
+# --- firefly.gnunet.org ---
+taler.net                     firefly.gnunet.org              firefly.gnunet.org                         25,465,587  143,993  firefly-taler
+gnunet.org                    firefly.gnunet.org              firefly.gnunet.org                         25,465,587  143,993  firefly-gnunet
+
+# --- Anastasis / TSA (MX mail.anastasis.lu → pixel) ---
+# Port 587 often closed/filtered on this plane → WARN only (see check_mail.sh).
+taler-systems.com             mail.anastasis.lu               mail.anastasis.lu,pixel.taler-systems.com  25,465,587  143,993  tsa-anastasis
+anastasis.lu                  mail.anastasis.lu               mail.anastasis.lu                         25,465,587  143,993  anastasis-lu
diff --git a/meta/README.md b/meta/README.md
new file mode 100644
index 0000000..0d9d45d
--- /dev/null
+++ b/meta/README.md
@@ -0,0 +1,21 @@
+# meta/ — optional & deprecated tooling
+
+Not required for the normal **host-agent** monitoring pipeline
+(`host-agent/run-*.sh` + `site-gen/console_to_html.py`).
+
+| Subdir | Contents |
+|--------|----------|
+| **`deprecated/`** | Pre-v1.8 mail/mattermost host-agent wrappers + user systemd units (folded into **surface**) |
+| **`firecuda/`** | Outside-runner install (launchd) + pull helpers — optional; prefer koopa host-agent |
+| **`site-gen-legacy/`** | Multi-host `generate-monitoring-sites` / `deploy-monitoring-sites` (older laptop/outside path) |
+
+## Production path (keep using)
+
+```text
+host-agent/run-host-report.sh
+host-agent/run-{hacktivism,surface,aptdeploy,fp-*}.sh
+host-agent/update-suite.sh
+site-gen/console_to_html.py
+```
+
+Machine-local paths/SSH: **`taler-monitoring-env`** → `~/.config/taler-monitoring/env`.
diff --git a/meta/deprecated/run-mail-monitoring.sh b/meta/deprecated/run-mail-monitoring.sh
new file mode 100755
index 0000000..cd13132
--- /dev/null
+++ b/meta/deprecated/run-mail-monitoring.sh
@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+# DEPRECATED (v1.8.0): mail is part of taler-monitoring-surface.
+# Kept as a thin alias so old timers/docs do not break.
+#
+# Prefer: run-surface-monitoring.sh
+# Public page: https://taler.hacktivism.ch/taler-monitoring-surface/
+#
+set -uo pipefail
+AGENT_DIR=$(cd "$(dirname "$0")" && pwd)
+echo "NOTE: run-mail-monitoring.sh is deprecated (v1.8.0) — folding into surface" >&2
+export PHASES="${PHASES:-mail surface monpages}"
+# force surface HTML paths (no /taler-monitoring-mail*)
+export HTML_OK_DIR="taler-monitoring-surface"
+export HTML_ERR_DIR="taler-monitoring-surface_err"
+export HTML_URL_OK="/taler-monitoring-surface/"
+export HTML_URL_ERR="/taler-monitoring-surface_err/"
+export PAGE_LABEL="taler-monitoring-surface"
+export MON_HOSTS="${MON_HOSTS:-taler.hacktivism.ch}"
+export MONPAGES_INVENTORY=job
+exec bash "$AGENT_DIR/run-surface-monitoring.sh" "$@"
diff --git a/meta/deprecated/run-mattermost-monitoring.sh b/meta/deprecated/run-mattermost-monitoring.sh
new file mode 100755
index 0000000..991f68c
--- /dev/null
+++ b/meta/deprecated/run-mattermost-monitoring.sh
@@ -0,0 +1,19 @@
+#!/usr/bin/env bash
+# DEPRECATED (v1.8.0): mattermost is part of taler-monitoring-surface.
+# Kept as a thin alias so old timers/docs do not break.
+#
+# Prefer: run-surface-monitoring.sh
+# Public page: https://taler.hacktivism.ch/taler-monitoring-surface/
+#
+set -uo pipefail
+AGENT_DIR=$(cd "$(dirname "$0")" && pwd)
+echo "NOTE: run-mattermost-monitoring.sh is deprecated (v1.8.0) — folding into surface" >&2
+export PHASES="${PHASES:-mattermost surface monpages}"
+export HTML_OK_DIR="taler-monitoring-surface"
+export HTML_ERR_DIR="taler-monitoring-surface_err"
+export HTML_URL_OK="/taler-monitoring-surface/"
+export HTML_URL_ERR="/taler-monitoring-surface_err/"
+export PAGE_LABEL="taler-monitoring-surface"
+export MON_HOSTS="${MON_HOSTS:-taler.hacktivism.ch}"
+export MONPAGES_INVENTORY=job
+exec bash "$AGENT_DIR/run-surface-monitoring.sh" "$@"
diff --git a/meta/deprecated/taler-monitoring-mail.service b/meta/deprecated/taler-monitoring-mail.service
new file mode 100644
index 0000000..852c7f9
--- /dev/null
+++ b/meta/deprecated/taler-monitoring-mail.service
@@ -0,0 +1,18 @@
+[Unit]
+Description=Taler monitoring mail (firefly/pixel MX SMTP IMAP)
+Documentation=file:%h/src/taler-monitoring/host-agent/README.md
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin
+Environment=CONTINUE_ON_ERROR=1
+Environment=RUN_TIMEOUT=600
+WorkingDirectory=%h/src/taler-monitoring
+ExecStart=%h/src/taler-monitoring/meta/deprecated/run-mail-monitoring.sh
+Nice=10
+TimeoutStartSec=20min
+
+[Install]
+WantedBy=default.target
diff --git a/meta/deprecated/taler-monitoring-mail.timer b/meta/deprecated/taler-monitoring-mail.timer
new file mode 100644
index 0000000..a0f03c8
--- /dev/null
+++ b/meta/deprecated/taler-monitoring-mail.timer
@@ -0,0 +1,14 @@
+[Unit]
+Description=Periodic taler-monitoring mail (MX/SMTP/IMAP)
+Documentation=file:%h/src/taler-monitoring/host-agent/README.md
+
+[Timer]
+OnBootSec=30min
+OnUnitActiveSec=4h
+AccuracySec=5min
+Persistent=true
+RandomizedDelaySec=10min
+Unit=taler-monitoring-mail.service
+
+[Install]
+WantedBy=timers.target
diff --git a/meta/deprecated/taler-monitoring-mattermost.service b/meta/deprecated/taler-monitoring-mattermost.service
new file mode 100644
index 0000000..0119036
--- /dev/null
+++ b/meta/deprecated/taler-monitoring-mattermost.service
@@ -0,0 +1,18 @@
+[Unit]
+Description=Taler monitoring Mattermost (mattermost.taler.net outside-in)
+Documentation=file:%h/src/taler-monitoring/host-agent/README.md
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin
+Environment=CONTINUE_ON_ERROR=1
+Environment=RUN_TIMEOUT=300
+WorkingDirectory=%h/src/taler-monitoring
+ExecStart=%h/src/taler-monitoring/meta/deprecated/run-mattermost-monitoring.sh
+Nice=10
+TimeoutStartSec=15min
+
+[Install]
+WantedBy=default.target
diff --git a/meta/deprecated/taler-monitoring-mattermost.timer b/meta/deprecated/taler-monitoring-mattermost.timer
new file mode 100644
index 0000000..3b077b0
--- /dev/null
+++ b/meta/deprecated/taler-monitoring-mattermost.timer
@@ -0,0 +1,14 @@
+[Unit]
+Description=Periodic taler-monitoring Mattermost (chat health)
+Documentation=file:%h/src/taler-monitoring/host-agent/README.md
+
+[Timer]
+OnBootSec=25min
+OnUnitActiveSec=1h
+AccuracySec=2min
+Persistent=true
+RandomizedDelaySec=5min
+Unit=taler-monitoring-mattermost.service
+
+[Install]
+WantedBy=timers.target
diff --git a/site-gen/com.hacktivism.taler-monitoring-sites.plist b/meta/firecuda/com.hacktivism.taler-monitoring-sites.plist
similarity index 100%
rename from site-gen/com.hacktivism.taler-monitoring-sites.plist
rename to meta/firecuda/com.hacktivism.taler-monitoring-sites.plist
diff --git a/site-gen/install-firecuda-timer.sh b/meta/firecuda/install-firecuda-timer.sh
similarity index 75%
rename from site-gen/install-firecuda-timer.sh
rename to meta/firecuda/install-firecuda-timer.sh
index 55052e9..be02be5 100755
--- a/site-gen/install-firecuda-timer.sh
+++ b/meta/firecuda/install-firecuda-timer.sh
@@ -11,8 +11,12 @@
 set -euo pipefail
 
 ROOT=$(cd "$(dirname "$0")" && pwd)
-MON_ROOT=$(cd "$ROOT/.." && pwd)
-FIRECUDA="${FIRECUDA_SSH:-firecuda-external}"
+MON_ROOT=$(cd "$ROOT/../.." && pwd)
+FIRECUDA="${FIRECUDA_SSH:-}"
+if [ -z "$FIRECUDA" ]; then
+  echo "Set FIRECUDA_SSH to your outside-runner SSH host alias" >&2
+  exit 2
+fi
 REMOTE_BASE="${FIRECUDA_INSTALL_DIR:-taler-monitoring-site-gen}"
 RUN_NOW=0
 [ "${1:-}" = "--run-now" ] && RUN_NOW=1
@@ -39,8 +43,8 @@ SKIP_SSH=1
 CONTINUE_ON_ERROR=1
 PHASES_ERR=urls versions
 PHASES_OK=urls versions
-DEPLOY_SSH=koopa-external
-DEPLOY_STAGING=/home/hernani/monitoring-sites-staging
+DEPLOY_SSH=${DEPLOY_SSH:-}
+DEPLOY_STAGING=${DEPLOY_STAGING:-$HOME/monitoring-sites-staging}
 WORK_ROOT=
 MONITORING_ROOT=..
 SOURCE_REPO_WEB=https://git.hacktivism.ch/hernani/taler-monitoring
@@ -56,12 +60,14 @@ sed -i.bak \"s|^WORK_ROOT=.*|WORK_ROOT=\$HOME/var/taler-monitoring-sites-work|\"
 ssh -o BatchMode=yes "$FIRECUDA" bash -s < "\$PLIST_DST"
-chmod +x "\$SITE_GEN"/*.sh "\$SITE_GEN"/console_to_html.py 2>/dev/null || true
-chmod +x "\$SITE_GEN/run-on-firecuda.sh"
+sed -e "s|__SITE_GEN__|\${FIRECUDA_META}|g" -e "s|__HOME__|\${HOME_DIR}|g" "\$PLIST_SRC" > "\$PLIST_DST"
+chmod +x "\$FIRECUDA_META"/*.sh "\$SITE_GEN"/console_to_html.py 2>/dev/null || true
+chmod +x "\$HOME_DIR/$REMOTE_BASE/meta/site-gen-legacy"/*.sh 2>/dev/null || true
+chmod +x "\$FIRECUDA_META/run-on-firecuda.sh"
 launchctl bootout "gui/\$(id -u)/com.hacktivism.taler-monitoring-sites" 2>/dev/null || true
 launchctl unload "\$PLIST_DST" 2>/dev/null || true
 launchctl bootstrap "gui/\$(id -u)" "\$PLIST_DST" 2>/dev/null || launchctl load "\$PLIST_DST"
@@ -72,8 +78,8 @@ REMOTE
 
 if [ "$RUN_NOW" = "1" ]; then
   echo "run once now…"
-  ssh -o BatchMode=yes "$FIRECUDA" "\"\$HOME/$REMOTE_BASE/site-gen/run-on-firecuda.sh\"" || true
+  ssh -o BatchMode=yes "$FIRECUDA" "\"\$HOME/$REMOTE_BASE/meta/firecuda/run-on-firecuda.sh\"" || true
 fi
 
 echo "done. logs: $FIRECUDA:~/Library/Logs/taler-monitoring-sites/"
-echo "manual: ssh $FIRECUDA '~/taler-monitoring-site-gen/site-gen/run-on-firecuda.sh'"
+echo "manual: ssh $FIRECUDA '~/taler-monitoring-site-gen/meta/firecuda/run-on-firecuda.sh'"
diff --git a/site-gen/pull-firecuda-to-koopa.sh b/meta/firecuda/pull-firecuda-to-koopa.sh
similarity index 58%
rename from site-gen/pull-firecuda-to-koopa.sh
rename to meta/firecuda/pull-firecuda-to-koopa.sh
index b1b10ec..0b7dd4e 100755
--- a/site-gen/pull-firecuda-to-koopa.sh
+++ b/meta/firecuda/pull-firecuda-to-koopa.sh
@@ -2,14 +2,18 @@
 # pull-firecuda-to-koopa.sh — copy HTML from firecuda work dir → koopa staging
 # Run on laptop (hernani) where both SSH aliases work.
 set -euo pipefail
-FIRECUDA="${FIRECUDA_SSH:-firecuda-external}"
-KOOPA="${DEPLOY_SSH:-koopa-external}"
+FIRECUDA="${FIRECUDA_SSH:-}"
+if [ -z "$FIRECUDA" ]; then
+  echo "Set FIRECUDA_SSH to your outside-runner SSH host alias" >&2
+  exit 2
+fi
+KOOPA="${DEPLOY_SSH:?set DEPLOY_SSH in env}"
 REMOTE_HTML="${FIRECUDA_HTML:-var/taler-monitoring-sites-work/html/}"
-STAGE="${DEPLOY_STAGING:-/home/hernani/monitoring-sites-staging}"
+STAGE="${DEPLOY_STAGING:-${DEPLOY_STAGING:-$HOME/monitoring-sites-staging}}"
 
 echo "firecuda:$REMOTE_HTML → $KOOPA:$STAGE"
 ssh -o BatchMode=yes -o ConnectTimeout=20 "$KOOPA" "mkdir -p '$STAGE'"
 rsync -az --delete \
   "${FIRECUDA}:${REMOTE_HTML}" \
   "${KOOPA}:${STAGE}/"
-echo "OK staged. Root: rsync to /var/www/monitoring-sites (see ROOT-ON-KOOPA.md)"
+echo "OK staged. Root: rsync to  (see ROOT-ON-KOOPA.md)"
diff --git a/site-gen/run-on-firecuda.sh b/meta/firecuda/run-on-firecuda.sh
similarity index 53%
rename from site-gen/run-on-firecuda.sh
rename to meta/firecuda/run-on-firecuda.sh
index 6f77de6..f6038d2 100755
--- a/site-gen/run-on-firecuda.sh
+++ b/meta/firecuda/run-on-firecuda.sh
@@ -1,24 +1,26 @@
 #!/usr/bin/env bash
-# run-on-firecuda.sh — entrypoint for launchd/cron on firecuda-external.
+# run-on-firecuda.sh — entrypoint for launchd/cron on outside-runner.
 # Outside-only monitoring (SKIP_SSH=1), all 9 sites, then stage HTML to koopa.
 #
-# Installed path on firecuda (default):
-#   ~/taler-monitoring-site-gen/site-gen/run-on-firecuda.sh
+# Lives in meta/firecuda/ (optional). Prefer koopa host-agent for production mon.
 #
 set -uo pipefail
 
 export PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:${PATH:-}"
 
-ROOT=$(cd "$(dirname "$0")" && pwd)
+HERE=$(cd "$(dirname "$0")" && pwd)
+SUITE_ROOT=$(cd "$HERE/../.." && pwd)
+LEGACY="$SUITE_ROOT/meta/site-gen-legacy"
+SITE_GEN="$SUITE_ROOT/site-gen"
+
 LOG_DIR="${FIRECUDA_LOG_DIR:-$HOME/Library/Logs/taler-monitoring-sites}"
 mkdir -p "$LOG_DIR"
 STAMP=$(date +%Y%m%d-%H%M%S)
 LOG="$LOG_DIR/run-${STAMP}.log"
 exec >>"$LOG" 2>&1
 
-echo "======== $(date -Iseconds 2>/dev/null || date) firecuda monitoring sites ========"
+echo "======== $(date -Iseconds 2>/dev/null || date) outside-runner monitoring sites ========"
 
-# Always: public DNS only, no SSH checks into stacks
 export SKIP_SSH=1
 export CONTINUE_ON_ERROR=1
 export AUTH401_CONTINUE=1
@@ -26,33 +28,31 @@ export RUNNER_SSH=
 export PHASES_ERR="${PHASES_ERR:-urls versions}"
 export PHASES_OK="${PHASES_OK:-urls versions}"
 export WORK_ROOT="${WORK_ROOT:-$HOME/var/taler-monitoring-sites-work}"
-export MONITORING_ROOT="${MONITORING_ROOT:-$ROOT/..}"
-export DEPLOY_SSH="${DEPLOY_SSH:-koopa-external}"
-export DEPLOY_STAGING="${DEPLOY_STAGING:-/home/hernani/monitoring-sites-staging}"
+export MONITORING_ROOT="${MONITORING_ROOT:-$SUITE_ROOT}"
+export DEPLOY_SSH="${DEPLOY_SSH:-}"
+export DEPLOY_STAGING="${DEPLOY_STAGING:-$HOME/monitoring-sites-staging}"
+export SITE_GEN_DIR="$SITE_GEN"
 
-# Prefer full 9-site list from settings or defaults inside generate script
 unset ONLY_HOSTS
 
-cd "$ROOT" || exit 1
-chmod +x generate-monitoring-sites.sh deploy-monitoring-sites.sh console_to_html.py 2>/dev/null || true
+cd "$LEGACY" || exit 1
+chmod +x generate-monitoring-sites.sh deploy-monitoring-sites.sh 2>/dev/null || true
+chmod +x "$SITE_GEN/console_to_html.py" 2>/dev/null || true
 
 ./generate-monitoring-sites.sh
 ec=$?
 
-# HTML stays on firecuda under WORK_ROOT/html.
-# Optional push to koopa if this host can SSH there (often only from laptop):
 if [ -n "${DEPLOY_SSH:-}" ] && [ "${DEPLOY_FROM_FIRECUDA:-0}" = "1" ]; then
   if [ -x ./deploy-monitoring-sites.sh ]; then
     ./deploy-monitoring-sites.sh || echo "WARN: deploy staging failed"
   fi
 else
   echo "note: HTML at ${WORK_ROOT}/html — pull from laptop:"
-  echo "  rsync -az firecuda-external:var/taler-monitoring-sites-work/html/ \\"
-  echo "    koopa-external:monitoring-sites-staging/"
-  echo "  # or: ./pull-firecuda-to-koopa.sh"
+  echo "  rsync -az outside-runner:var/taler-monitoring-sites-work/html/ \\"
+  echo "    \${DEPLOY_SSH}:monitoring-sites-staging/"
+  echo "  # or: $HERE/pull-firecuda-to-koopa.sh"
 fi
 
-# keep last 20 logs
 ls -1t "$LOG_DIR"/run-*.log 2>/dev/null | tail -n +21 | xargs rm -f 2>/dev/null || true
 
 echo "======== done ec=$ec ========"
diff --git a/site-gen/deploy-monitoring-sites.sh b/meta/site-gen-legacy/deploy-monitoring-sites.sh
similarity index 78%
rename from site-gen/deploy-monitoring-sites.sh
rename to meta/site-gen-legacy/deploy-monitoring-sites.sh
index c5718e1..b13e64b 100755
--- a/site-gen/deploy-monitoring-sites.sh
+++ b/meta/site-gen-legacy/deploy-monitoring-sites.sh
@@ -3,9 +3,12 @@
 # Does NOT reload Caddy (needs root — see ROOT-ON-KOOPA.md).
 set -uo pipefail
 
-ROOT=$(cd "$(dirname "$0")" && pwd)
-SETTINGS="${SITE_GEN_SETTINGS:-$ROOT/settings.conf}"
-[ -f "$SETTINGS" ] || SETTINGS="$ROOT/settings.conf.example"
+HERE=$(cd "$(dirname "$0")" && pwd)
+SUITE_ROOT=$(cd "$HERE/../.." && pwd)
+SITE_GEN="${SITE_GEN_DIR:-$SUITE_ROOT/site-gen}"
+ROOT="$HERE"
+SETTINGS="${SITE_GEN_SETTINGS:-$SITE_GEN/settings.conf}"
+[ -f "$SETTINGS" ] || SETTINGS="$SITE_GEN/settings.conf.example"
 # shellcheck disable=SC1090
 set -a
 source <(grep -v '^\s*#' "$SETTINGS" | grep -v '^\s*$' | sed 's/\r$//')
@@ -13,8 +16,8 @@ set +a
 
 WORK_ROOT="${WORK_ROOT:-/tmp/taler-monitoring-sites-work}"
 HTML_DIR="$WORK_ROOT/html"
-DEPLOY_SSH="${DEPLOY_SSH:-koopa-external}"
-DEPLOY_WWW_ROOT="${DEPLOY_WWW_ROOT:-/var/www/monitoring-sites}"
+DEPLOY_SSH="${DEPLOY_SSH:-}"
+DEPLOY_WWW_ROOT="${DEPLOY_WWW_ROOT:-}"
 
 if [ ! -d "$HTML_DIR" ]; then
   echo "no HTML_DIR $HTML_DIR — run generate-monitoring-sites.sh first" >&2
@@ -23,7 +26,7 @@ fi
 
 echo "deploy $HTML_DIR → ${DEPLOY_SSH}:${DEPLOY_WWW_ROOT}/"
 # Prefer writing to a hernani-owned staging dir if /var/www not writable
-STAGE_REMOTE="${DEPLOY_STAGING:-/home/hernani/monitoring-sites-staging}"
+STAGE_REMOTE="${DEPLOY_STAGING:-${DEPLOY_STAGING:-$HOME/monitoring-sites-staging}}"
 
 ssh -o BatchMode=yes -o ConnectTimeout=20 "$DEPLOY_SSH" \
   "mkdir -p '$STAGE_REMOTE' '$DEPLOY_WWW_ROOT' 2>/dev/null || mkdir -p '$STAGE_REMOTE'"
diff --git a/site-gen/generate-monitoring-sites.sh b/meta/site-gen-legacy/generate-monitoring-sites.sh
similarity index 94%
rename from site-gen/generate-monitoring-sites.sh
rename to meta/site-gen-legacy/generate-monitoring-sites.sh
index f9e344e..b4a991b 100755
--- a/site-gen/generate-monitoring-sites.sh
+++ b/meta/site-gen-legacy/generate-monitoring-sites.sh
@@ -18,9 +18,13 @@ export PYTHONUNBUFFERED=1
 : "${RUN_TIMEOUT:=600}"
 export RUN_TIMEOUT
 
-ROOT=$(cd "$(dirname "$0")" && pwd)
-SETTINGS="${SITE_GEN_SETTINGS:-$ROOT/settings.conf}"
-EXAMPLE="$ROOT/settings.conf.example"
+HERE=$(cd "$(dirname "$0")" && pwd)
+SUITE_ROOT=$(cd "$HERE/../.." && pwd)
+SITE_GEN="${SITE_GEN_DIR:-$SUITE_ROOT/site-gen}"
+ROOT="$HERE"
+SETTINGS="${SITE_GEN_SETTINGS:-$SITE_GEN/settings.conf}"
+EXAMPLE="${SITE_GEN}/settings.conf.example"
+[ -f "$EXAMPLE" ] || EXAMPLE="$HERE/settings.conf.example"
 
 if [ ! -f "$SETTINGS" ]; then
   if [ -f "$EXAMPLE" ]; then
@@ -80,7 +84,7 @@ while [ $# -gt 0 ]; do
   esac
 done
 
-MONITORING_ROOT=$(cd "$ROOT/${MONITORING_ROOT:-..}" && pwd)
+MONITORING_ROOT=$(cd "${MONITORING_ROOT:-$SUITE_ROOT}" && pwd)
 WORK_ROOT="${WORK_ROOT:-/tmp/taler-monitoring-sites-work}"
 LOG_DIR="$WORK_ROOT/logs"
 HTML_DIR="$WORK_ROOT/html"
@@ -174,7 +178,7 @@ run_mon() {
 htmlify() {
   local host="$1" mode="$2" log="$3" out="$4" other="$5"
   mkdir -p "$(dirname "$out")"
-  if [ ! -f "$ROOT/console_to_html.py" ]; then
+  if [ ! -f "$SITE_GEN/console_to_html.py" ]; then
     echo "WARN: console_to_html.py missing — bootstrap $out"
     {
       echo "$host $mode"
@@ -185,7 +189,7 @@ htmlify() {
     } >"$out"
     return 0
   fi
-  python3 "$ROOT/console_to_html.py" \
+  python3 "$SITE_GEN/console_to_html.py" \
     --log "$log" \
     --out "$out" \
     --hostname "$host" \
@@ -269,5 +273,5 @@ echo "HTML under $HTML_DIR"
 echo "commit $COMMIT_SHORT  $COMMIT_URL"
 echo "RUN_TIMEOUT=${RUN_TIMEOUT}s (global; 0=unlimited)"
 echo "Deploy (as hernani, may need sudo for /var/www):"
-echo "  ./deploy-monitoring-sites.sh"
+echo "  $HERE/deploy-monitoring-sites.sh"
 echo "Root on koopa: see ROOT-ON-KOOPA.md"
diff --git a/metrics.sh b/metrics.sh
index 4c440ea..5e6ac42 100644
--- a/metrics.sh
+++ b/metrics.sh
@@ -70,11 +70,15 @@ path = sys.argv[2] if len(sys.argv) > 2 else ""
 alt = {}
 if path:
     try:
-        alt = json.load(open(path))
+        _loaded = json.load(open(path))
+        if isinstance(_loaded, dict):
+            alt = {str(k): str(v) for k, v in _loaded.items()}
     except Exception:
         alt = {}
-if not alt:
+if not isinstance(alt, dict) or not alt:
     alt = {"0": "GOA"}
+elif "0" not in alt:
+    alt["0"] = "GOA"
 
 def parse(s):
     if ":" in s:
@@ -513,7 +517,7 @@ metrics_taler_load() {
     _m_bank="${INSIDE_BANK_CTR:-stage-lfp-bank}"
     _m_mer="${INSIDE_MERCHANT_CTR:-stage-lfp-merchant}"
     _m_ex="${INSIDE_EXCHANGE_CTR:-stage-lfp-exchange-ansible}"
-    _m_ssh="${INSIDE_SSH:-francpaysan-stage-user}"
+    _m_ssh="${INSIDE_SSH:-}"
   fi
   remote_py=$(cat < secrets.env > SECRETS_ROOT files > SSH koopa
 
@@ -30,8 +30,8 @@
 # Default phases for -d stage.lefrancpaysan.ch: urls + inside + versions + e2e
 # inside uses francpaysan-stage-user (stagepaysan); stats from public HTTPS.
 # Admin credit (required for ATM withdraw ladder on stage):
-# E2E_BANK_ADMIN_PASS=…   # or auto-load via SSH francpaysan-host
-# FRANCPAYSAN_SSH=francpaysan-host
+# E2E_BANK_ADMIN_PASS=…   # or auto-load via SSH 
+# FRANCPAYSAN_SSH=
 # FRANCPAYSAN_SECRETS=$HOME/francpaysan-secrets
 # Public shop pays use E2E_USE_TEMPLATES=1 (default on TESTPAYSAN) — no merchant token.
 
@@ -69,6 +69,15 @@
 # AUTH401_ADMIN_USER=admin
 # FRANCPAYSAN_SECRETS=$HOME/francpaysan-secrets
 
+# --- Devtesting / fake-franken CHF (rusty.taler-ops.ch) ---
+# Phase: devtesting | franken | fake-franken
+# SSH Host taler-rusty-devtesting (user devtesting, forced taler-devtesting CLI).
+# DEVTESTING_SSH=taler-rusty-devtesting
+# DEVTESTING_AMOUNT=CHF:1.00
+# DEVTESTING_CREDIT_PAYTO=   # optional; auto from EXCHANGE_PUBLIC/keys or TOPS IBAN
+# DEVTESTING_EXCHANGE_IBAN=CH6808573105529100001
+# DEVTESTING_SKIP=1          # skip phase
+
 # --- Ladder (GOA withdraw ladder · explorer pool) ---
 # Prefer SECRETS_ROOT / KOOPA_ADMIN_SECRETS (bank-explorer-password.txt).
 # Overrides only if needed:
diff --git a/site-gen/README.md b/site-gen/README.md
index 8d51e10..6d95bbf 100644
--- a/site-gen/README.md
+++ b/site-gen/README.md
@@ -1,81 +1,20 @@
-# monitoring site-gen
+# site-gen — HTML converter + Caddy notes
 
-Build **console-style HTML** from `taler-monitoring` runs and stage them for
-**host Caddy** (`/monitoring`, `/monitoring_err`).
+**Required for host-agent:** `console_to_html.py` (log → sticky HTML).
 
-Same **global reporting defaults** as host-agents (`run-host-report.sh`):
-
-- **`RUN_TIMEOUT=600`** on each suite run  
-- **line-buffered** logs (continuous write)  
-- **always** write HTML (first run / empty logs too)  
-- Forgejo **commit link** in the page header  
-
-Applies to all nine fronts (GOA + FP stage + FP prod) in the default `SITES` list.
-
-| Host | Role |
+| File | Role |
 |------|------|
-| **firecuda-external** | optional runner (public DNS) |
-| **koopa-external** | deploy staging + Caddy (root service) |
-| **host-agents** | primary: GOA + FP on-box (`../host-agent/`) |
+| `console_to_html.py` | Convert suite console log → public mon HTML |
+| `caddy-monitoring-handles.snippet` | Caddy `handle` / bare redir notes |
+| `ROOT-ON-KOOPA.md` | Publish staging → `/var/www` (paths from env) |
+| `settings.conf.example` | Optional settings for **legacy** multi-host generator |
 
-No secrets in `settings.conf` — only hosts, paths, site list.
+## Optional / moved
 
-## firecuda-external (outside-only) — **disabled for now**
-
-**firecuda-external could** run public-only monitoring (`SKIP_SSH=1`, phases
-`urls versions`) on a schedule (launchd every 4h) and write HTML under
-`~/var/taler-monitoring-sites-work/html/`. Scripts for that still live here
-(`install-firecuda-timer.sh`, `run-on-firecuda.sh`, plist template).
-
-**Current policy: do not run a real launchd job on firecuda.**  
-Primary GOA monitoring is **koopa host-agent** (`host-agent/`, user `hernani`,
-local `podman exec`). Re-enable firecuda only if you explicitly want a second,
-outside-only view.
+Legacy multi-host generate/deploy and firecuda launchd tooling live under
+**`../meta/`** (see `meta/README.md`).
 
 ```bash
-# optional one-shot (no timer):
-# ssh firecuda-external '~/taler-monitoring-site-gen/site-gen/run-on-firecuda.sh'
-
-# if a timer was installed earlier, ensure it is off:
-# ssh firecuda-external 'launchctl list | grep taler-monitoring || echo off'
-```
-
-Disabled plists may remain as `*.plist.disabled-*` under `~/Library/LaunchAgents/`.
-
-## Quick start (laptop)
-
-```bash
-cd site-gen
-cp settings.conf.example settings.conf   # optional
-chmod +x generate-monitoring-sites.sh deploy-monitoring-sites.sh console_to_html.py
-
-ONLY_HOSTS="bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch" \
-  SKIP_SSH=1 RUNNER_SSH= \
-  ./generate-monitoring-sites.sh
-
-./deploy-monitoring-sites.sh
-```
-
-Then on koopa **as root**: see **[ROOT-ON-KOOPA.md](./ROOT-ON-KOOPA.md)**.
-
-## URL rules
-
-| Path | When |
-|------|------|
-| `/monitoring_err/` | always: full console log + error list on top (CONTINUE_ON_ERROR run) |
-| `/monitoring/` | **clean** full log if last strict run exit 0; else **stub** linking to `/monitoring_err/` |
-
-Footer links **Forgejo commit** of the admin-log tree used for that run.
-
-## Layout
-
-```text
-site-gen/
-  settings.conf.example
-  generate-monitoring-sites.sh
-  deploy-monitoring-sites.sh
-  console_to_html.py
-  caddy-monitoring-handles.snippet
-  ROOT-ON-KOOPA.md
-  README.md
+# normal publish is host-agent on the mon host, not these scripts
+../host-agent/run-hacktivism-monitoring.sh   # etc.
 ```
diff --git a/site-gen/ROOT-ON-KOOPA.md b/site-gen/ROOT-ON-KOOPA.md
index 604b997..c0e4c1c 100644
--- a/site-gen/ROOT-ON-KOOPA.md
+++ b/site-gen/ROOT-ON-KOOPA.md
@@ -3,31 +3,33 @@
 Host Caddy runs as **root/systemd** (`caddy` user). Static monitoring HTML is
 **not** put into Taler containers — only host paths + Caddy `handle`.
 
-After `generate-monitoring-sites.sh` + `deploy-monitoring-sites.sh` (as hernani),
-files live in:
+After host-agent (or optional `meta/site-gen-legacy/generate-monitoring-sites.sh`) + deploy, files live in:
 
 ```text
-/home/hernani/monitoring-sites-staging//monitoring/index.html
-/home/hernani/monitoring-sites-staging//monitoring_err/index.html
+$DEPLOY_STAGING (or $HOME/monitoring-sites-staging)//monitoring/index.html
+$DEPLOY_STAGING (or $HOME/monitoring-sites-staging)//monitoring_err/index.html
 ```
 
-## One-shot as root (copy + Caddy)
+## One-shot as root (preferred)
 
 ```bash
-# on koopa (koopa-external), as root:
+# absolute path — as root, ~ is /root (script not found under ~/koopa-caddy/…)
+sudo $APPLY_MONITORING_LIVE
+# same script is versioned in the suite:
+#   host-agent/apply-monitoring-live.sh  → install/sync to $KOOPA_CADDY_DIR/
+```
 
-# 1) web root
-install -d -o caddy -g caddy -m 755 /var/www/monitoring-sites
-rsync -a --delete /home/hernani/monitoring-sites-staging/ /var/www/monitoring-sites/
-chown -R caddy:caddy /var/www/monitoring-sites
+That rsyncs staging → `/var/www/monitoring-sites`, installs the prepared
+Caddyfile, reloads Caddy, and smokes mon URLs (including bare surface).
 
-# 2) Caddyfile — merge handles from admin-log mirror, then:
-#    (either edit /etc/caddy/Caddyfile by hand using snippet below,
-#     or copy full mirror after review)
-#
-#    install -m 644 /home/hernani/koopa-admin-log/configs/caddy/Caddyfile /etc/caddy/Caddyfile
-#    # or: ~/koopa-caddy/apply.sh  after syncing Caddyfile into ~/koopa-caddy/
+## Manual equivalent
 
+```bash
+install -d -o hernani -g caddy -m 755 /var/www/monitoring-sites
+rsync -a --delete $DEPLOY_STAGING (or $HOME/monitoring-sites-staging)/ /var/www/monitoring-sites/
+chown -R hernani:caddy /var/www/monitoring-sites
+
+install -m 644 $KOOPA_CADDY_DIR/Caddyfile /etc/caddy/Caddyfile
 caddy validate --config /etc/caddy/Caddyfile
 systemctl reload caddy
 systemctl is-active caddy
@@ -35,23 +37,10 @@ systemctl is-active caddy
 
 ## Snippet per hacktivism site (inside each `*.hacktivism.ch { }` block)
 
-Place **before** the catch-all `reverse_proxy` (same idea as `/intro*`):
+Place **before** the catch-all `reverse_proxy`. Canonical copy:
+`site-gen/caddy-monitoring-handles.snippet`.
 
-```caddy
-    # Public taler-monitoring console HTML (static; host only)
-    handle /monitoring_err* {
-        root * /var/www/monitoring-sites/{host}
-        rewrite * /monitoring_err{uri}
-        # uri is /monitoring_err or /monitoring_err/ → serve directory index
-        file_server
-    }
-    handle /monitoring* {
-        root * /var/www/monitoring-sites/{host}
-        file_server
-    }
-```
-
-Simpler layout (recommended): path on disk matches URL under host root:
+Disk layout (path on disk matches URL under host root):
 
 ```text
 /var/www/monitoring-sites/bank.hacktivism.ch/monitoring/index.html
@@ -59,34 +48,50 @@ Simpler layout (recommended): path on disk matches URL under host root:
 ```
 
 ```caddy
-    handle_path /monitoring_err/* {
-        root * /var/www/monitoring-sites/{host}/monitoring_err
+    # bare → slash (MUST use redir * /path/ — see footgun below)
+    handle /monitoring {
+        redir * /monitoring/ 302
+    }
+    handle /monitoring* {
+        root * /var/www/monitoring-sites/{host}
         file_server
     }
     handle /monitoring_err {
-        redir /monitoring_err/ 302
+        redir * /monitoring_err/ 302
     }
-    handle_path /monitoring/* {
-        root * /var/www/monitoring-sites/{host}/monitoring
+    handle /monitoring_err* {
+        root * /var/www/monitoring-sites/{host}
         file_server
     }
-    handle /monitoring {
-        redir /monitoring/ 302
-    }
 ```
 
-Canonical snippet is also in `caddy-monitoring-handles.snippet` and applied
-in `configs/caddy/Caddyfile` for the three GOA hosts.
+### Caddy redir footgun (merchant code 21)
+
+```caddy
+# WRONG — parsed as matcher=/monitoring/ to="302" → Location: 302
+handle /monitoring {
+    redir /monitoring/ 302
+}
+
+# RIGHT
+handle /monitoring {
+    redir * /monitoring/ 302
+}
+```
+
+Symptom: `https://…/monitoring` or `…/taler-monitoring-surface` returns merchant
+JSON `{"code":21,…}` while the slash form serves HTML.
 
 ## Check
 
 ```bash
 curl -sS -o /dev/null -w '%{http_code}\n' https://bank.hacktivism.ch/monitoring/
-curl -sS -o /dev/null -w '%{http_code}\n' https://bank.hacktivism.ch/monitoring_err/
-curl -sS https://bank.hacktivism.ch/monitoring_err/ | head
+curl -sS -D- -o /dev/null https://taler.hacktivism.ch/taler-monitoring-surface | head -15
+# bare: expect 302 Location: /taler-monitoring-surface/  — not JSON code 21
+curl -sS -o /dev/null -w '%{http_code}\n' https://taler.hacktivism.ch/taler-monitoring-surface/
 ```
 
 ## Not needed as root
 
-- Running `taler-monitoring` (use **firecuda-external** / laptop)
+- Running `taler-monitoring` (use **optional outside runner** / laptop)
 - Writing into podman Taler containers
diff --git a/site-gen/caddy-monitoring-handles.snippet b/site-gen/caddy-monitoring-handles.snippet
index 1c09531..9ab43a7 100644
--- a/site-gen/caddy-monitoring-handles.snippet
+++ b/site-gen/caddy-monitoring-handles.snippet
@@ -1,57 +1,76 @@
-# Host Caddy on koopa (root service). Paste BEFORE catch-all reverse_proxy.
+# Host Caddy (root service). Place BEFORE catch-all reverse_proxy in each site block.
+#
+# CRITICAL: bare paths (no trailing slash) must redirect to slash, or the request
+# falls through to merchant/bank/exchange API → Taler JSON code 21.
+#
+# CORRECT (site-level named matcher — preferred):
+#   @mon_bare path /monitoring /monitoring_err \
+#       /monitoring-max-ladder /monitoring-max-ladder_err \
+#       /taler-monitoring-surface /taler-monitoring-surface_err \
+#       /taler-monitoring-aptdeploy /taler-monitoring-aptdeploy_err
+#   redir @mon_bare {path}/ 302
+#
+# ALSO OK inside handle (matcher must be `*`):
+#   handle /taler-monitoring-surface {
+#       redir * /taler-monitoring-surface/ 302
+#   }
+#
+# WRONG (Caddy footgun — causes merchant code 21 on bare URLs):
+#   redir /taler-monitoring-surface/ 302
+# → parsed as matcher=/taler-monitoring-surface/  to="302"
+# → Location: 302, or no-op then fallthrough to reverse_proxy
+#
+# root must be the *host directory* (…/monitoring-sites/{host}), and the
+# file_server matcher a path *prefix* (`/monitoring*`), so
+# /monitoring/index.html maps to {root}/monitoring/index.html.
 #
 # Disk layout:
 #   /var/www/monitoring-sites/{host}/monitoring/index.html
-#   /var/www/monitoring-sites/{host}/monitoring_err/index.html
-#   /var/www/monitoring-sites/taler.hacktivism.ch/taler-monitoring-surface*/index.html
-#   /var/www/monitoring-sites/taler.hacktivism.ch/taler-monitoring-aptdeploy*/index.html
+#   /var/www/monitoring-sites/taler.hacktivism.ch/taler-monitoring-*/index.html
 #
-# /monitoring*                    → bank + exchange + taler.hacktivism.ch
-# /taler-monitoring-surface*      → taler.hacktivism.ch only
-# /taler-monitoring-aptdeploy*    → taler.hacktivism.ch only
+# Live apply on koopa (absolute path — ~ as root is /root):
+#   sudo ${APPLY_MONITORING_LIVE}
+# Runtime-only (no sudo; hernani admin API):
+#   curl -g -X POST http://[::1]:2019/load -H 'Content-Type: text/caddyfile' \
+#     --data-binary @${KOOPA_CADDY_DIR}/Caddyfile
 
 # --- only inside taler.hacktivism.ch { ... } ---
-    handle /taler-monitoring-surface_err {
-        redir /taler-monitoring-surface_err/ 302
-    }
-    handle /taler-monitoring-surface_err/ {
-        root * /var/www/monitoring-sites/taler.hacktivism.ch/taler-monitoring-surface_err
+    @mon_bare path /monitoring /monitoring_err \
+        /monitoring-max-ladder /monitoring-max-ladder_err \
+        /taler-monitoring-surface /taler-monitoring-surface_err \
+        /taler-monitoring-aptdeploy /taler-monitoring-aptdeploy_err \
+        /taler-monitoring-mail /taler-monitoring-mail_err \
+        /taler-monitoring-mattermost /taler-monitoring-mattermost_err
+    redir @mon_bare {path}/ 302
+
+    handle /taler-monitoring-surface_err* {
+        root * /var/www/monitoring-sites/taler.hacktivism.ch
         file_server
     }
-    handle /taler-monitoring-surface {
-        redir /taler-monitoring-surface/ 302
-    }
-    handle /taler-monitoring-surface/ {
-        root * /var/www/monitoring-sites/taler.hacktivism.ch/taler-monitoring-surface
+    handle /taler-monitoring-surface* {
+        root * /var/www/monitoring-sites/taler.hacktivism.ch
         file_server
     }
-    handle /taler-monitoring-aptdeploy_err {
-        redir /taler-monitoring-aptdeploy_err/ 302
-    }
-    handle /taler-monitoring-aptdeploy_err/ {
-        root * /var/www/monitoring-sites/taler.hacktivism.ch/taler-monitoring-aptdeploy_err
+    handle /taler-monitoring-aptdeploy_err* {
+        root * /var/www/monitoring-sites/taler.hacktivism.ch
         file_server
     }
-    handle /taler-monitoring-aptdeploy {
-        redir /taler-monitoring-aptdeploy/ 302
-    }
-    handle /taler-monitoring-aptdeploy/ {
-        root * /var/www/monitoring-sites/taler.hacktivism.ch/taler-monitoring-aptdeploy
+    handle /taler-monitoring-aptdeploy* {
+        root * /var/www/monitoring-sites/taler.hacktivism.ch
         file_server
     }
 
-# --- bank + exchange + taler (each site block) ---
-    handle /monitoring_err {
-        redir /monitoring_err/ 302
-    }
-    handle /monitoring_err/ {
-        root * /var/www/monitoring-sites/{host}/monitoring_err
+# --- bank + exchange + taler (each site block; set root host dir) ---
+# /monitoring* already covers /monitoring-max-ladder*
+    @mon_bare path /monitoring /monitoring_err \
+        /monitoring-max-ladder /monitoring-max-ladder_err
+    redir @mon_bare {path}/ 302
+
+    handle /monitoring_err* {
+        root * /var/www/monitoring-sites/{host}
         file_server
     }
-    handle /monitoring {
-        redir /monitoring/ 302
-    }
-    handle /monitoring/ {
-        root * /var/www/monitoring-sites/{host}/monitoring
+    handle /monitoring* {
+        root * /var/www/monitoring-sites/{host}
         file_server
     }
diff --git a/site-gen/console_to_html.py b/site-gen/console_to_html.py
index 4c4927f..6141c0e 100755
--- a/site-gen/console_to_html.py
+++ b/site-gen/console_to_html.py
@@ -4,23 +4,209 @@ from __future__ import annotations
 
 import argparse
 import html
+import os
 import re
+import sys
 from datetime import datetime, timezone
 from pathlib import Path
 
-def ui_lang(explicit: str = "") -> str:
-    """en (default) or fr. Env: TALER_MON_LANG; auto fr for lefrancpaysan hosts."""
-    import os
-    lang = (explicit or os.environ.get("TALER_MON_LANG") or "").strip().lower()
-    if lang in ("fr", "fra", "french"):
+# Display timestamps in Central European time (CEST/CET · MESZ/MEZ), same as `date` on CH/DE hosts.
+try:
+    from zoneinfo import ZoneInfo
+
+    MON_TZ = ZoneInfo("Europe/Zurich")
+except Exception:  # pragma: no cover
+    MON_TZ = timezone.utc
+
+
+def format_generated_now() -> tuple[str, str]:
+    """Return (display_local, iso_with_offset) for sticky bar +