monitoring: boxed badges + area.group test IDs

scripts/taler-monitoring/README.md

@@ -1,24 +1,26 @@
 # taler-monitoring
 
-Report for the **GOA** stack with clear severity tags and **per-area test IDs**:
+Report for the **GOA** stack with boxed severity badges and **grouped test IDs**
+`area.group-NN` (e.g. `www.exchange-01`, `e2e.pay-03`):
 
 | Tag | Meaning |
 |-----|---------|
-| `[OK]` | check passed |
+| `┌ OK ┐` | check passed |
-| `[INFO]` | inside status (container, ports, log noise) |
+| `┌ INFO ┐` | status / note |
-| `[WARN]` | degraded but maybe not fatal |
+| `┌ WARN ┐` | degraded but maybe not fatal |
-| `[ERROR]` | component problem |
+| `┌ ERROR ┐` | component problem |
-| `[BLOCKER]` | **withdraw/pay path cannot complete** because of this |
+| `┌ BLOCKER ┐` | **withdraw/pay path cannot complete** because of this |
 
-IDs: **`www-001`**, **`inside-001`**, **`versions-001`**, **`sanity-001`**, **`server-001`**, **`e2e-001`** …  
+IDs reset per **group** inside an area — so “too many www tests” become
+`www.exchange-*`, `www.bank-*`, `www.landing-*`, etc.  
 Catalog: **[TESTS.md](./TESTS.md)**.
 
 ```text
-[OK]    www-001  exchange /config https://exchange…/config
+┌ OK    ┐ www.exchange-01  exchange /config · HTTP 200
-[BLOCKER] e2e-015  prereq: merchant HTTP 502
+┌ BLOCKER┐ e2e.prereq-02   merchant HTTP 502
 ```
 
-End of each phase: totals + list of **BLOCKERS** and **ERRORS**.
+End of each phase: totals + list of **BLOCKERS** and **ERRORS** (ids included).
 
 ## Commands
 

scripts/taler-monitoring/TESTS.md

@@ -1,217 +1,132 @@
-# taler-monitoring — test IDs by area
+# taler-monitoring — grouped test IDs
 
-Every check line is numbered **per area** as `AREA-NNN` (zero-padded):
+Every check line is numbered **`area.group-NN`** (two-digit group counter):
-
-| Area | Phase script | Meaning |
-|------|--------------|---------|
-| **www** | `check_urls.sh` | public HTTPS (outside-in) |
-| **inside** | `check_inside.sh` | containers / processes on koopa |
-| **versions** | `check_versions.sh` | deb.taler.net available + packages vs trixie |
-| **sanity** | `check_sanity.sh` | public + server per component |
-| **server** | `check_server.sh` | SSH host ports / processes |
-| **e2e** | `check_e2e.sh` | withdraw + pay cycle |
-
-Format in output:
 
 ```text
-[OK]    www-001  exchange /config https://exchange…/config
+┌ OK    ┐ www.exchange-01  exchange /config · HTTP 200
-[ERROR] e2e-012  bank-auth: admin token failed
+┌ ERROR ┐ e2e.pay-03       order create failed · HTTP 502
-[BLOCKER] e2e-015  prereq: merchant HTTP 502
+┌ BLOCKER┐ e2e.prereq-02   merchant secret missing
 ```
 
-IDs are assigned **in run order** within the area (`set_area` resets the counter). Optional soft checks still consume a number when they WARN.
+| Area | Phase script | Groups (examples) |
+|------|--------------|-------------------|
+| **www** | `check_urls.sh` | `exchange` `perf` `stats` `bank` `merchant` `paivana` `landing` |
+| **inside** | `check_inside.sh` | `ssh` `bank` `exchange` `merchant` `caddy` `load` |
+| **versions** | `check_versions.sh` | `outside` `inside` `compare` |
+| **sanity** | `check_sanity.sh` | `bank` `exchange` `merchant` |
+| **server** | `check_server.sh` | (flat `server-NN` or host groups) |
+| **e2e** | `check_e2e.sh` | `prereq` `load` `bank` `wallet` `atm` `settle` `pay` `shop` `paivana` `dig` `report` |
+| **ladder** | `check_goa_ladder.sh` | `plan` `load` `withdraw` `pay` `report` |
+
+**Why groups:** flat `www-001`…`www-080` was hard to talk about.  
+`www.bank-04` / `e2e.atm-02` pin the failure to a logical block.
+
+Usage in scripts:
+
+```bash
+set_area www
+set_group exchange   # → www.exchange-01, www.exchange-02, …
+set_group bank       # counter resets → www.bank-01
+```
+
+`set_group` also prints a small group chip (`┌ www.exchange ┐`) so logs show section boundaries.  
+Re-entering the same group later (e.g. exchange legal after perf) **continues** NN — `www.exchange-03`, not a second `-01`.  
+Without `set_group`, IDs stay flat: `area-01`, `area-02`, …
+
+Optional soft checks still consume a number when they WARN.  
+Numbering follows **executed** checks (early skip may shift later NN inside the same group).
 
 ---
 
 ## www — public URLs (`./taler-monitoring.sh urls`)
 
-| ID | Check |
+| Group | Checks (typical) |
-|----|--------|
+|-------|------------------|
-| www-… | exchange `/config`, currency, **alt_unit_names** |
+| **www.exchange-** | `/config`, currency, **alt_unit_names**; `/keys` (+ alt soft); `/intro/`, `/`; **`/terms`**, **`/privacy`**, `/terms/` |
-| www-… | exchange `/keys` (+ alt_unit_names soft) |
+| **www.perf-** | outside-in RTT: bank/exchange/merchant `/config`, keys, webui, intro — ms; WARN ≥ `PERF_WARN_MS` (8000); ERROR ≥ `PERF_FAIL_MS` (20000) |
-| www-… | exchange `/intro/`, `/` (302→intro) |
+| **www.stats-** | `/intro/stats.json` loadavg / container RSS / probe ms (public; SSH fallback) |
-| www-… | **exchange `/terms`** body (not empty / not API error) |
+| **www.bank-** | `/config`, currency, alt_unit_names; integration/webui/intro; **auto-account.json**; `/terms`, `/privacy` |
-| www-… | **exchange `/privacy`** body |
+| **www.merchant-** | `/config` currency + currencies alt_unit_names; listed exchanges alt; webui/intro; **`/terms`**, **`/privacy`** |
-| www-… | exchange `/terms/` (200 or redirect) |
+| **www.paivana-** | local GOA paywall front (redirect to template) |
-| www-… | bank `/config`, currency, **alt_unit_names** |
+| **www.landing-** | every own-stack link on bank/merchant/exchange intros; static assets; demo-withdraw / cross-links |
-| www-… | bank integration / webui / intro / `/` |
-| www-… | **bank `/terms`** body |
-| www-… | **bank `/privacy`** (or `/intro/privacy.html` fallback) |
-| www-… | merchant `/config`, currency, currencies alt_unit_names |
-| www-… | each merchant `exchanges[]` `/config` alt_unit_names |
-| www-… | merchant `/intro/`, `/webui/`, `/` |
-| www-… | **merchant `/terms`** body (dual-currency notice) |
-| www-… | **merchant `/privacy`** body (must not be `not configured`) |
-| www-… | merchant `/terms/` redirect |
-| www-… | **landing (aggregated per site)**: one line for bank/merchant/exchange (`/intro` + assets + own/external link counts); failures listed as samples; soft external/shop assets compact |
-| www-… | cross-links: one line (4 intros) on local stack |
-| www-… | bank demo-withdraw + shop assets: one line (+ soft shop-pay if missing) |
-| www-… | **bank `/intro/demo-withdraw.json`** → `taler://withdraw/HOST:PORT/taler-integration/…` + integration op HTTP 200 |
-| www-… | **landing exposed links** (bank / merchant / exchange): parse each `/intro/` HTML, probe every own-stack `https://` + root-relative `href`/`src`/`content`, soft-check external stores/docs |
-| www-… | landing static: `qrcode.min.js`, `og-goa-shop.png`, `qr-logo.png`, shop-pay.js/css |
-| www-… | cross-links between bank ↔ merchant ↔ exchange intros (local stack) |
-| www-… | **bank `/intro/demo-withdraw.json`** → `taler://withdraw/HOST/taler-integration/…` (no default `:443`/`:80`; non-default port OK) + integration op HTTP 200 |
-| www-… | bank `/intro/auto-account.json` (earlier) → same withdraw shape, **no payto_uri**, login at `/webui/` |
-| www-… | **performance** (outside-in RTT): bank/exchange/merchant `/config`, keys, webui, intro — ms; WARN ≥ `PERF_WARN_MS` (8000); ERROR ≥ `PERF_FAIL_MS` (20000) |
-| www-… | **landing load stats**: `/intro/stats.json` → loadavg, container RSS, in-container probe ms (public; SSH container cat fallback); warn if stale/missing |
 
-**Legal docs rule:** HTTP 200, non-empty body, not plain `not configured`, not merchant API JSON `code:21`. On local stack, optional content needle (terms/privacy/FADP/GOA…).
+**Legal docs rule:** HTTP 200, non-empty body, not plain `not configured`, not merchant API JSON `code:21`. Local stack may require content needle (terms/privacy/FADP/GOA…).
 
-**Performance rule:** Measured from the **monitoring runner** (public URLs via Caddy), not container loopback. HTTP must match expect (usually 200); **latency (ms) is on every perf OK/WARN/ERROR line**, plus a **perf summary** (n / min / p50 / avg / max). Slow ≥ `PERF_WARN_MS` (default 8000) → WARN; ≥ `PERF_FAIL_MS` (default 20000) → ERROR.
+**Performance rule:** Measured from the **monitoring runner** (public via Caddy), not container loopback. Latency (ms) on every perf line + **perf summary** (n / min / p50 / avg / max).
 
-**Landing links rule:** Own-stack (bank/exchange/taler.\* + page host) must be HTTP 200 (or redirect→200). External (App Store, Play, F-Droid, wallet.taler.net, docs/git.taler.net, …) soft WARN if down. Auto-account wallet link must be `taler://withdraw/HOST/taler-integration/…` (default ports stripped for mobile wallets), never payto.
+**Landing links rule:** Own-stack must be HTTP 200 (or redirect→200). External stores/docs soft WARN. Auto-account wallet link must be `taler://withdraw/HOST/taler-integration/…` (default ports stripped), never payto.
 
-**alt_unit_names rule:** wallet codec requires a non-empty map including scale key `"0"`. For multi-currency merchant, also follow every entry in `exchanges[]` and check that exchange’s public `/config`.
+**alt_unit_names rule:** non-empty map including scale key `"0"`. Multi-currency merchant: follow each `exchanges[]` public `/config`.
-
-(IDs after a failed early check may shift if later soft checks are skipped when body missing — numbering follows **executed** checks.)
 
 ---
 
 ## inside — koopa SSH (`./taler-monitoring.sh inside`)
 
-| ID | Check (typical order) |
+| Group | Checks |
-|----|------------------------|
+|-------|--------|
-| inside-001 | ssh koopa (or **koopa-external** fallback) |
+| **inside.ssh-** | ssh reachability / remote collect |
-| inside-002+ | per-component emit: container, ports, libeufin/httpd, postgres, local `/config`/`/keys`, wirewatch, DNS pin, caddy |
+| **inside.bank-** | container, libeufin, postgres, local `/config`, nginx, DNS pins |
-| inside-… | **load / memory**: host loadavg + RAM; bank/exchange/merchant podman CPU/mem/block + process RSS by role + DB sizes |
+| **inside.exchange-** | container, httpd, wirewatch, aggregator, transfer, local keys, DNS |
+| **inside.merchant-** | container, httpd, wirewatch, depositcheck, DNS |
+| **inside.caddy-** | host reverse-proxy process |
+| **inside.load-** | host loadavg + RAM; per-container RSS/CPU |
 
-Remote lines `E|comp|LEVEL|key|detail` each become one numbered result.
+Remote lines `E|comp|LEVEL|key|detail` each become one numbered result under that component group.
 
-SSH: `KOOPA_SSH` (default `koopa`), then `KOOPA_SSH_FALLBACKS` (default `koopa-external`) when LAN is unreachable.
+SSH: `KOOPA_SSH` (default `koopa`), then `KOOPA_SSH_FALLBACKS` when LAN is unreachable.
 
 ---
 
-## sanity — bank · exchange · merchant (`./taler-monitoring.sh sanity`)
+## e2e — withdraw + pay (`./taler-monitoring.sh e2e`)
 
-| ID | Section |
+| Group | Checks |
-|----|---------|
+|-------|--------|
-| sanity-001… | bank public + server |
+| **e2e.prereq-** | wallet-cli, currency, budgets, secrets, public reachability |
-| sanity-… | exchange public + server |
+| **e2e.load-** | host/container load snapshots (before / after ATM / after pay) |
-| sanity-… | merchant public + server |
+| **e2e.bank-** | account, credit, withdraw ops |
+| **e2e.wallet-** | exchange + ToS, wallet setup |
+| **e2e.atm-** | ATM withdraw ladder rungs |
+| **e2e.settle-** | wallet settlement wait / spendable balance |
+| **e2e.pay-** | variable payments |
+| **e2e.shop-** | GOA shop product catalog pays |
+| **e2e.paivana-** | template paywall pay |
+| **e2e.dig-** | coins-missing dig (inside + withdrawal-operation) |
+| **e2e.report-** | final tallies |
 
-Sequential through the whole script (one `set_area sanity`).
+When filing an issue, quote the full id + label, e.g.  
+`e2e.atm-03  ATM withdraw 5 GOA · bank confirmed, wallet empty`.
 
 ---
 
-## versions — packages vs deb.taler.net (`./taler-monitoring.sh versions`)
+## versions / sanity / ladder / server
 
-### Outside (runner / public network — no SSH)
+| Area.group | Meaning |
-
+|------------|---------|
-| ID (order) | Check |
+| **versions.outside-** | deb.taler.net suite index |
-|------------|--------|
+| **versions.inside-** | packages in containers |
-| versions-… | DNS `deb.taler.net` |
+| **versions.compare-** | installed vs suite |
-| versions-… | HTTPS portal + apt base URL |
+| **sanity.bank-** / **.exchange-** / **.merchant-** | public + optional server-side per component |
-| versions-… | suite `InRelease` / `Release` |
+| **ladder.plan-** / **.load-** / **.withdraw-** / **.pay-** / **.report-** | GOA amount ladder |
-| versions-… | suite `Packages` + `Packages.gz` |
+| **server-** | SSH host ports / processes (flat unless grouped later) |
-| versions-… | sample pool `.deb` fetchable (Range 200/206) |
-| versions-… | suite offers `taler-exchange`, `taler-merchant`, `libeufin-bank` |
-| versions-… | optional `trixie-testing` Packages |
-| versions-… | TLS verify (soft) |
-
-### Inside (SSH koopa containers)
-
-| ID | Check |
-|----|--------|
-| versions-… | ssh koopa |
-| versions-… | each container → `InRelease` (pasta can reach apt repo) |
-| versions-… | each container lists `deb.taler.net` in apt sources |
-| versions-… | each installed `taler*` / `libeufin*` / `libtaler*` / `libdonau*` vs suite version |
-| versions-… | core packages installed (`taler-exchange`, `libeufin-bank`, `taler-merchant`) |
-
-Outside always runs. Inside skipped with `SKIP_SSH=1` (still reports outside results).
-
-Compare rules:
-
-- **match** suite → OK  
-- **ahead** of suite (often testing/dev) → INFO  
-- **behind** suite → ERROR for core packages, WARN otherwise (`TALER_PKG_BEHIND=error` forces ERROR)
-
-Default suite: **trixie** (`TALER_APT_SUITE`, `TALER_APT_BASE=https://deb.taler.net/apt/debian`).
-
-Without SSH (`SKIP_SSH=1` or remote domain): still runs outside-in repo checks; skips container install compare.
 
 ---
 
-## server — SSH ports (`./taler-monitoring.sh server`)
+## Visual format (colour on)
-
-| ID | Check |
-|----|--------|
-| server-001 | ssh |
-| server-002+ | containers, local pasta ports, processes, caddy |
-
----
-
-## e2e — payment path (`./taler-monitoring.sh e2e`)
-
-Local GOA also: **ATM includes GOA:4200**, then **paivana** (HTTP 302 on `PAIVANA_PUBLIC` + public template pay `paivana` / GOA:4200 on `goa-shop`). Disable with `E2E_PAIVANA=0`.
-
-| ID | Step (approx.) |
-|----|----------------|
-| e2e-001 | budget info |
-| e2e-002 | wallet-cli present |
-| e2e-003 | mode / currency info |
-| e2e-004… | secrets, reachability gates |
-| e2e-… | account, credit, withdraw, confirm, coins, order, pay ladder |
-| e2e-… | **load snapshots** at e2e-start, after-withdraw, after-pay, after-shop, e2e-end (host loadavg/RAM + container RSS/CPU/DB) |
-| e2e-… | **coins** before/after each ATM withdraw, settle, pay, shop: count in circulation / spent, amount + denoms using **`alt_unit_names`** (e.g. `5 Kilo-GOA (GOA:5000)`), Δ vs previous snap |
-| e2e-… | **GOA shop products** — full catalog list; **random pick of 2** (override `E2E_SHOP_PICK_N`) |
-| e2e-… | balances, dig on failure, load delta + overall metrics |
-
-Shop product pays use instance `goa-shop` (default) and catalog `E2E_SHOP_PRODUCTS`
-(`id|Product name|amount` lines). Each e2e run **shuffles** the catalog and pays
-only `E2E_SHOP_PICK_N` products (**default 2**). Flow matches the landing popup
-(public POST `/templates/{id}`, not private orders). Report labels use product name.
-
-Blockers keep the same ID prefix: `[BLOCKER] e2e-0NN step: message`.
-
----
-
-## ladder — GOA withdraw ranges (`./taler-monitoring.sh ladder`)
-
-| Step | What |
-|------|------|
-| ladder-001 | `GET /intro/auto-account.json` (personal account, GOA:0) |
-| ladder-… | **load snapshot** before withdraws (host + bank/exchange/merchant mem/CPU) |
-| ladder-… | **Phase A withdraw** (23): mint explorer pool — 0 → random → max−1 → max into **one cumulative wallet** (mids × `LADDER_WITHDRAW_SCALE`, default 1.5) |
-| ladder-… | wallet-cli accept-uri + explorer confirm when `selected`; settle; coins snap |
-| ladder-… | **Phase B pay** (23): same shape 0 → random → max−1 → max via merchant orders + `handle-uri --yes` |
-| ladder-… | **load snapshot** after + delta + overall timing (withdraw + pay) |
-| report | withdraw TSV + pay TSV + JSON |
-
-Default amounts (`build_ladder_pair`, strictly increasing):
 
 ```text
-withdraw: [0] + random mids + [max−1] + [max]
+╔══════════════════════════════╗
-pay:      [0] + (withdraw_mid / scale) + [max−1] + [max]   # always ≤ matching withdraw mid
+║ www · public URLs · …        ║
+╚══════════════════════════════╝
+┌ www.exchange ┐
+┌ OK    ┐ www.exchange-01  exchange /config · HTTP 200
+┌ www.bank ┐
+┌ OK    ┐ www.bank-01  bank /config · HTTP 200
+┌ ERROR ┐ www.bank-04  bank /intro/auto-account.json · invalid withdraw
+
+┌ ERRORS · failed checks ┐
+  • www.bank-04 bank /intro/auto-account.json · invalid withdraw
+totals: 40 OK, 1 ERROR, 2 WARN, 5 INFO
 ```
 
-TSV `range`: `pin:0` | `random` | `pin:max-1` | `pin:max`.
+`NO_COLOR=1` or `CLICOLOR=0` disables boxes/colours (ASCII `[ OK ]` + `-- group --` headers).  
-
+Always use `printf --` friendly plain headers when colour is off (leading `---` is not a printf option).
-Soft: absolute **max** mint/pay may `CEILING_REJECT` / skip (WARN). **max-1** is a hard pin.  
-`LADDER_PAY=0` skips phase B.
-
-| Env | Default | Meaning |
-|-----|---------|---------|
-| `LADDER_STEPS` | `23` | total rungs (0 + mids + max) |
-| `LADDER_MAX_AMOUNT` | `4503599627370496` | absolute last pin |
-| `LADDER_TIMEOUT_S` | `3600` | large rungs need time |
-| `LADDER_MAX_RUNGS` | `99` | cap list length after build |
-
-```bash
-./taler-monitoring.sh ladder
-LADDER_STEPS=10 ./taler-monitoring.sh ladder
-LADDER_REPORT_DIR=/tmp/my-ladder ./taler-monitoring.sh ladder
-```
-
----
-
-## Run one area
-
-```bash
-./taler-monitoring.sh urls      # www only
-./taler-monitoring.sh inside    # inside only
-./taler-monitoring.sh versions  # deb.taler.net + package drift
-./taler-monitoring.sh e2e       # e2e only
-./taler-monitoring.sh ladder    # GOA amount ladder + timings
-./taler-monitoring.sh -d taler.net urls
-```

scripts/taler-monitoring/check_e2e.sh

@@ -182,6 +182,7 @@ e2e_skip_rest() {
 
 # Area e2e-### — withdraw + pay cycle
 set_area e2e
+set_group prereq
 section "e2e · prerequisites"
 info "e2e budget" "${E2E_TIMEOUT}s (set E2E_TIMEOUT= to change)"
 
@@ -382,10 +383,12 @@ else
 fi
 
 # Host + bank/exchange/merchant RAM/CPU/load (SSH koopa or koopa-external)
+set_group load
 section "e2e · load snapshot (before withdraw/pay)"
 metrics_report_load "${METRICS_DIR}/load-before.json" "e2e-start" || true
 
 # Local stack: koopa secrets. Remote: only explicit env (never leak local passwords to foreign banks).
+set_group prereq
 if [ "$E2E_REMOTE" = "1" ]; then
   ADMIN_PASS="${E2E_BANK_ADMIN_PASS:-}"
   MPW="${E2E_MERCHANT_TOKEN:-${MERCHANT_TOKEN:-}}"
@@ -496,6 +499,7 @@ wcli_pay() {
 
 # When coins never become available: run inside + targeted bank/exchange dig
 dig_when_no_coins() {
+  set_group dig
   section "e2e · coins missing — inside dig"
   info "reason" "wallet empty after withdraw path (often wirewatch timing) — probing bank + exchange"
   if [ -n "${WID:-}" ]; then
@@ -580,6 +584,7 @@ BAL_BEFORE=$(fmt_bal "$SCRATCH/bal-before.out")
 info "BALANCE before" "$BAL_BEFORE"
 
 # ---------------------------------------------------------------------------
+set_group bank
 section "e2e · bank (account · credit · withdraw)"
 # ---------------------------------------------------------------------------
 AT=""
@@ -678,6 +683,7 @@ else
 fi
 
 # ---------------------------------------------------------------------------
+set_group wallet
 section "e2e · wallet setup (exchange + ToS once)"
 # ---------------------------------------------------------------------------
 if ! wcli exchanges add "${EXCHANGE_PUBLIC}/" >"$SCRATCH/ex-add.out" 2>&1; then
@@ -1079,6 +1085,7 @@ sys.exit(0 if d.get("paid") is True or str(d.get("order_status","")).lower()=="p
 }
 
 # ---------------------------------------------------------------------------
+set_group atm
 section "e2e · ATM withdraw ladder"
 # ---------------------------------------------------------------------------
 WITHDRAW_OK=0
@@ -1109,12 +1116,14 @@ for WITHDRAW_AMT in $WITHDRAW_LIST; do
   esac
 done
 info "ATM withdraw summary" "$WITHDRAW_REPORT (ok=$WITHDRAW_OK_N lag=$WITHDRAW_LAG_N fail=$WITHDRAW_FAIL_N)"
+set_group load
 section "e2e · load snapshot (after ATM withdraws)"
 metrics_report_load "${METRICS_DIR}/load-after-withdraw.json" "after-withdraw" || true
 cp -f "${METRICS_DIR}/load-after-withdraw.json" "${METRICS_DIR}/load-after.json" 2>/dev/null || true
 metrics_report_coins "after-ATM-ladder" || true
 
 # Settlement catch-up: bank may have confirmed while wallet was still empty
+set_group settle
 section "e2e · wallet settlement (timing)"
 if ! wait_wallet_balance 0 "${E2E_SETTLE_ROUNDS}" "${E2E_SETTLE_SLEEP}"; then
   if [ "$WITHDRAW_LAG_N" -gt 0 ] || [ "$WITHDRAW_OK_N" -gt 0 ]; then
@@ -1154,6 +1163,7 @@ else
 fi
 
 # ---------------------------------------------------------------------------
+set_group pay
 section "e2e · variable payments (if balance allows)"
 # ---------------------------------------------------------------------------
 PAY_OK=0
@@ -1193,6 +1203,7 @@ for PAY_AMT in $PAY_LIST; do
   fi
 done
 info "pay summary" "$PAY_REPORT (ok=$PAY_OK_N fail=$PAY_FAIL_N skip=$PAY_SKIP_N)"
+set_group load
 section "e2e · load snapshot (after payments)"
 metrics_report_load "${METRICS_DIR}/load-after-pay.json" "after-pay" || true
 cp -f "${METRICS_DIR}/load-after-pay.json" "${METRICS_DIR}/load-after.json" 2>/dev/null || true
@@ -1207,6 +1218,7 @@ if [ "$PAY_OK" != "1" ]; then
 fi
 
 # ---------------------------------------------------------------------------
+set_group shop
 section "e2e · GOA shop products (merchant landing catalog)"
 # ---------------------------------------------------------------------------
 # Public template POST + wallet pay — full catalog list, random pick of N products.
@@ -1297,6 +1309,7 @@ EOF
 fi
 
 # ---------------------------------------------------------------------------
+set_group paivana
 section "e2e · paivana paywall (template GOA:4200)"
 # ---------------------------------------------------------------------------
 # Public paywall: https://paivana.hacktivism.ch → 302 to merchant template "paivana".
@@ -1388,6 +1401,7 @@ else
   info "paivana summary" "${PAIVANA_REPORT:-?} · instance ${E2E_PAIVANA_INSTANCE}"
 fi
 
+set_group report
 section "e2e · report"
 info "user" "$USER"
 info "ATM withdraws" "$WITHDRAW_REPORT"

scripts/taler-monitoring/check_goa_ladder.sh

@@ -29,7 +29,10 @@ ROOT=$(cd "$(dirname "$0")" && pwd)
 # shellcheck source=lib.sh
 source "$ROOT/lib.sh"
 
+# Area ladder.* — GOA withdraw/pay ladder
+# Groups: ladder.plan / ladder.load / ladder.withdraw / ladder.pay / ladder.report
 set_area ladder
+set_group plan
 SECTION_T0=$(date +%s)
 now_ms() { python3 -c 'import time; print(int(time.time()*1000))'; }
 elapsed_ms() {
@@ -261,6 +264,7 @@ info "steps" "${LADDER_STEPS} (0 + random≥${LADDER_MIN_AMOUNT} + max-1=${CUR}:
 info "withdraw_scale" "${LADDER_WITHDRAW_SCALE}× pay mids (fund pay ladder)"
 info "pay_phase" "$([ "${LADDER_PAY}" = "1" ] && echo enabled || echo disabled)"
 
+set_group load
 section "ladder · load snapshot (before withdraws)"
 metrics_report_load "$LOAD_BEFORE" "ladder-start" || true
 # Fresh wallet per rung — baseline empty (or last-rung DB if re-used later)
@@ -344,6 +348,7 @@ STOP_REASON=""
 STOP_AMOUNT=""
 declare -a RUNG_JSON=()
 
+set_group withdraw
 section "ladder · phase A · withdraw (${LADDER_N} rungs)"
 rung=0
 for AMT in "$@"; do
@@ -787,6 +792,7 @@ done
 PAY_OK_N=0
 PAY_FAIL_N=0
 if [ "${LADDER_PAY}" = "1" ] && [ -n "${PAY_LIST:-}" ] && [ "$FAIL_N_L" -eq 0 ]; then
+  set_group pay
   section "ladder · phase B · pay"
   metrics_report_coins "before-pay-ladder" || true
   # Merchant secret (same as e2e)
@@ -1024,6 +1030,7 @@ fi
 ms_phase=$(python3 -c 'import sys,time; print(int((time.time()-float(sys.argv[1]))*1000))' "$SECTION_T0")
 
 # --- report ---
+set_group report
 section "ladder · report"
 info "auto-account" "$ACCT_USER"
 info "ok_rungs" "$OK_N"
@@ -1128,6 +1135,7 @@ PY
 
 wcli balance 2>&1 | tee "$REPORT_DIR/balance-final.out" | tail -20 || true
 
+set_group load
 section "ladder · load snapshot (after withdraws)"
 metrics_report_load "$LOAD_AFTER" "ladder-end" || true
 if [ -f "$LOAD_BEFORE" ] && [ -f "$LOAD_AFTER" ]; then

scripts/taler-monitoring/check_inside.sh

@@ -7,8 +7,10 @@ source "$ROOT/lib.sh"
 # shellcheck source=metrics.sh
 source "$ROOT/metrics.sh"
 
-# Area inside-### — container / process state on koopa (SSH)
+# Area inside.* — container / process state on koopa (SSH)
+# Groups: inside.ssh / inside.bank / inside.exchange / inside.merchant / inside.caddy
 set_area inside
+set_group ssh
 section "inside · collect from koopa"
 
 if [ "${SKIP_SSH}" = "1" ]; then
@@ -128,10 +130,20 @@ if [ -z "$RAW" ] || ! echo "$RAW" | grep -q '^E|'; then
   exit 1
 fi
 
+_last_inside_grp=""
 while IFS= read -r line; do
   case "$line" in
     E\|*)
       IFS='|' read -r _ comp level key detail <<<"$line"
+      # Group IDs by component so issues map cleanly: inside.bank-02, inside.exchange-04
+      case "$comp" in
+        bank|exchange|merchant|caddy) _g="$comp" ;;
+        *) _g="ssh" ;;
+      esac
+      if [ "$_g" != "$_last_inside_grp" ]; then
+        set_group "$_g"
+        _last_inside_grp="$_g"
+      fi
       case "$level" in
         OK)    ok "[$comp] $key${detail:+ ($detail)}" ;;
         ERROR) err "$comp" "$key" "$detail" ;;
@@ -143,6 +155,7 @@ while IFS= read -r line; do
 done <<<"$RAW"
 
 # Host loadavg + RAM + per-container RSS/CPU (same probe as e2e/ladder)
+set_group load
 section "inside · load / memory"
 METRICS_DIR="${METRICS_DIR:-$(mktemp -d)}"
 export METRICS_DIR

scripts/taler-monitoring/check_sanity.sh

@@ -23,10 +23,12 @@ json_currency() {
   python3 -c 'import json,sys; d=json.load(open(sys.argv[1])); print(d.get("currency") or "")' "$1" 2>/dev/null || true
 }
 
-# Area sanity-### — public + optional server-side per component
+# Area sanity.* — public + optional server-side per component
+# Groups: sanity.bank / sanity.exchange / sanity.merchant
 set_area sanity
 
 # ---------------------------------------------------------------------------
+set_group bank
 section "sanity · bank"
 # ---------------------------------------------------------------------------
 expect_code "bank public /config" 200 "$BANK_PUBLIC/config"
@@ -98,6 +100,7 @@ else
 fi
 
 # ---------------------------------------------------------------------------
+set_group exchange
 section "sanity · exchange"
 # ---------------------------------------------------------------------------
 expect_code "exchange public /config" 200 "$EXCHANGE_PUBLIC/config"
@@ -193,6 +196,7 @@ else
 fi
 
 # ---------------------------------------------------------------------------
+set_group merchant
 section "sanity · merchant"
 # ---------------------------------------------------------------------------
 expect_code "merchant public /config" 200 "$MERCHANT_PUBLIC/config"

scripts/taler-monitoring/check_server.sh

@@ -5,8 +5,9 @@ ROOT=$(cd "$(dirname "$0")" && pwd)
 # shellcheck source=lib.sh
 source "$ROOT/lib.sh"
 
-# Area server-### — host/container ports via SSH
+# Area server.* — host/container ports via SSH
 set_area server
+set_group ssh
 section "server · ssh ${KOOPA_SSH}"
 
 if ! koopa_ssh_ok; then

scripts/taler-monitoring/check_urls.sh

@@ -5,7 +5,9 @@ ROOT=$(cd "$(dirname "$0")" && pwd)
 # shellcheck source=lib.sh
 source "$ROOT/lib.sh"
 
-# Area www-### — public HTTPS (outside-in)
+# Area www.* — public HTTPS (outside-in)
+# Groups: www.exchange / www.perf / www.stats / www.bank / www.merchant /
+#         www.paivana / www.landing
 set_area www
 section "www · public URLs · ${TALER_DOMAIN:-?} (outside-in, no SSH)"
 
@@ -169,7 +171,8 @@ sys.exit(0)
 PY
 }
 
-# --- exchange (core; always required) ---  www-001 …
+# --- exchange (core; always required) ---  www.exchange-NN
+set_group exchange
 check_url "exchange /config" 200 "$EXCHANGE_PUBLIC/config"
 code=$(http_body "$EXCHANGE_PUBLIC/config" "$tmp/ec.json")
 if [ "$code" = "200" ]; then
@@ -219,6 +222,7 @@ fi
 # Performance — outside-in public HTTPS latency (this runner, not loopback)
 # Same spirit as bank landing-stats public probes; measured here from outside.
 # ---------------------------------------------------------------------------
+set_group perf
 section "www · performance · public HTTPS latency (outside-in)"
 
 # Latency thresholds (ms), outside-in from this runner.
@@ -392,6 +396,7 @@ PY
 }
 
 if [ "${CHECK_LANDING:-1}" = "1" ] || [ "${LOCAL_STACK:-0}" = "1" ]; then
+  set_group stats
   section "www · performance · landing load stats (stats.json · in-container probes)"
   report_landing_load_stats "bank" "$BANK_PUBLIC"
   report_landing_load_stats "exchange" "$EXCHANGE_PUBLIC"
@@ -470,7 +475,8 @@ fi
 info "perf note" "measured from this host (outside-in); thresholds PERF_WARN_MS=${PERF_WARN_MS} PERF_FAIL_MS=${PERF_FAIL_MS}"
 
 
-# Terms + privacy (legal docs)
+# Exchange terms + privacy (same www.exchange group — issues map to exchange)
+set_group exchange
 check_legal_doc "exchange /terms" "$EXCHANGE_PUBLIC/terms" "terms|GOA|exploration|FADP|revDSG|privacy"
 check_legal_doc "exchange /privacy" "$EXCHANGE_PUBLIC/privacy" "privacy|FADP|revDSG|data|GOA|exploration"
 # trailing slash: 200 or redirect to bare path
@@ -485,6 +491,7 @@ case "$code" in
 esac
 
 # --- bank ---
+set_group bank
 if [ "${LOCAL_STACK:-1}" = "0" ]; then
   check_url_soft "bank /config" 200 "$BANK_PUBLIC/config"
 else
@@ -545,6 +552,7 @@ else
 fi
 
 # --- merchant ---
+set_group merchant
 if [ "${LOCAL_STACK:-1}" = "0" ]; then
   check_url_soft "merchant /config" 200 "$MERCHANT_PUBLIC/config"
 else
@@ -610,6 +618,7 @@ esac
 # Paivana paywall (local GOA stack) — public front only; pay path is e2e
 # ---------------------------------------------------------------------------
 if [ "${LOCAL_STACK:-1}" = "1" ] && [ "${E2E_PAIVANA:-1}" != "0" ]; then
+  set_group paivana
   section "www · paivana paywall"
   : "${PAIVANA_PUBLIC:=https://paivana.hacktivism.ch}"
   PAIVANA_PUBLIC="${PAIVANA_PUBLIC%/}"
@@ -646,6 +655,7 @@ if [ "${CHECK_LANDING:-1}" != "1" ]; then
   exit 0
 fi
 
+set_group landing
 section "www · landing exposed links · bank / merchant / exchange"
 
 # Probe one URL: print code to stdout (200 after following redirects counts as 200).

scripts/taler-monitoring/check_versions.sh

@@ -19,6 +19,7 @@ ROOT=$(cd "$(dirname "$0")" && pwd)
 source "$ROOT/lib.sh"
 
 set_area versions
+set_group outside
 
 SUITE="${TALER_APT_SUITE:-trixie}"
 APT_BASE="${TALER_APT_BASE:-https://deb.taler.net/apt/debian}"
@@ -166,6 +167,7 @@ fi
 # ---------------------------------------------------------------------------
 # 2) INSIDE — containers can reach deb.taler.net (pasta / install path)
 # ---------------------------------------------------------------------------
+set_group inside
 section "versions · inside · containers → deb.taler.net"
 
 if [ "${SKIP_SSH}" = "1" ]; then
@@ -266,6 +268,7 @@ ok "collected installed packages" "${n_inst} rows"
 # ---------------------------------------------------------------------------
 # 3) compare installed vs trixie (and note testing)
 # ---------------------------------------------------------------------------
+set_group compare
 section "versions · compare installed vs ${SUITE}"
 
 export BEHIND_MODE HAVE_TESTING

scripts/taler-monitoring/lib.sh

@@ -352,11 +352,12 @@ koopa_ssh_python() {
   with_timeout "$t" ssh "${SSH_BASE_OPTS[@]}" "${KOOPA_SSH}" python3 -
 }
 
-# ANSI colours — badge tags (fg+bg) + body (label bold, detail dim).
+# ANSI colours — boxed badges (fg+bg) + body (label bold, detail dim).
-# Off: NO_COLOR=1 or CLICOLOR=0. Badges need a terminal that supports SGR bg.
+# Off: NO_COLOR=1 or CLICOLOR=0.
 if [ "${NO_COLOR:-0}" = "1" ] || [ "${CLICOLOR:-1}" = "0" ]; then
   G= R= Y= C= M= D= W= B= N=
   BG_OK= BG_INFO= BG_WARN= BG_ERR= BG_BLK= BG_SEC=
+  BOX=0
 else
   G=$'\e[1;32m'          # green text (totals)
   R=$'\e[1;31m'          # red text
@@ -367,13 +368,14 @@ else
   W=$'\e[1;37m'          # bold white label
   B=$'\e[1m'             # bold section
   N=$'\e[0m'             # reset
-  # Badge: bright text on solid background (space-padded in _msg_line)
+  # Badge fill: bright text on solid background
   BG_OK=$'\e[1;30;42m'   # black on green
   BG_INFO=$'\e[1;30;46m' # black on cyan
   BG_WARN=$'\e[1;30;43m' # black on yellow
   BG_ERR=$'\e[1;37;41m'  # white on red
   BG_BLK=$'\e[1;37;45m'  # white on magenta
   BG_SEC=$'\e[1;37;44m'  # white on blue (section)
+  BOX=1
 fi
 
 PASS_N=0
@@ -383,17 +385,68 @@ INFO_N=0
 BLOCKERS=()   # human-readable payment/withdraw blockers
 ERRORS=()     # all ERROR lines (component scope)
 
-# Test IDs by area: www-001, e2e-001, inside-001, …
+# Grouped test IDs: area.group-NN  (e.g. www.exchange-01, e2e.pay-03)
-# Usage: set_area www   then each ok/fail/warn/info/blocker/err auto-numbers.
+#   set_area www              # phase: www | e2e | inside | versions | …
+#   set_group exchange        # resets counter + prints group box → www.exchange-01
+#   set_group bank            # → www.bank-01
+# Without set_group: area-01, area-02, … (flat within area)
 TEST_AREA=""
+TEST_GROUP=""
 TEST_N=0
-# Last issued id (www-001); set by _take_tid — not via $(…) so TEST_N persists.
 LAST_TID=""
+# Per-group counters so re-entering www.exchange after perf continues NN
+# (shell vars TEST_CNT_<area>_<group>).
+_tid_key() {
+  # $1=area $2=group → safe identifier
+  printf 'TEST_CNT_%s_%s' "$1" "$2" | tr -c 'A-Za-z0-9_' '_'
+}
+_tid_save() {
+  [ -z "${TEST_AREA:-}" ] || [ -z "${TEST_GROUP:-}" ] && return 0
+  local k
+  k=$(_tid_key "$TEST_AREA" "$TEST_GROUP")
+  eval "$k=\$TEST_N"
+}
+_tid_load() {
+  local k
+  k=$(_tid_key "$TEST_AREA" "$1")
+  eval "TEST_N=\${$k:-0}"
+}
 set_area() {
+  # leave previous group counter saved
+  _tid_save
   TEST_AREA="$1"
+  TEST_GROUP=""
   TEST_N=0
   LAST_TID=""
 }
+# Start a logical sub-group. Short stable names:
+#   exchange bank merchant perf stats landing paivana
+#   prereq wallet atm settle pay shop dig load report
+#   ssh caddy outside inside compare withdraw
+# Counter continues if the same group is re-entered later in the area.
+# Prints a compact group chip so log + issue text map cleanly (www.bank-03).
+set_group() {
+  local g="$1"
+  # no-op if same group already active (e.g. inside emit loop)
+  if [ "$g" = "${TEST_GROUP:-}" ] && [ -n "$g" ]; then
+    return 0
+  fi
+  _tid_save
+  TEST_GROUP="$g"
+  LAST_TID=""
+  if [ -z "${TEST_AREA:-}" ] || [ -z "$g" ]; then
+    TEST_N=0
+    return 0
+  fi
+  _tid_load "$g"
+  local tag="${TEST_AREA}.${g}"
+  if [ "${BOX:-0}" = "1" ]; then
+    # cyan-outline group chip: ┌ www.exchange ┐
+    printf '%s%s┌ %s ┐%s\n' "$D" "$C" "$tag" "$N"
+  else
+    printf -- '-- %s --\n' "$tag"
+  fi
+}
 # Assign next id into LAST_TID (must not run in a subshell).
 _take_tid() {
   LAST_TID=""
@@ -401,27 +454,40 @@ _take_tid() {
     return
   fi
   TEST_N=$((TEST_N + 1))
-  LAST_TID=$(printf '%s-%03d' "$TEST_AREA" "$TEST_N")
+  if [ -n "${TEST_GROUP:-}" ]; then
+    LAST_TID=$(printf '%s.%s-%02d' "$TEST_AREA" "$TEST_GROUP" "$TEST_N")
+    _tid_save
+  else
+    LAST_TID=$(printf '%s-%02d' "$TEST_AREA" "$TEST_N")
+  fi
 }
-# Dim test id: "www-001  " or empty
+# Dim test id: "www.exchange-01  " or empty
 _fmt_tid() {
   if [ -n "${LAST_TID:-}" ]; then
     printf '%s%s%s  ' "$D" "$LAST_TID" "$N"
   fi
 }
-# One concrete line: [TAG] tid  label  ·  detail
+# Boxed badge cell: ┌ OK     ┐ with filled bg (tag left-padded, width 7 for BLOCKER)
-# $1=badge style (bg+fg)  $2=tag text  $3=label colour  $4=label  $5=detail
+# $1=badge style  $2=tag text
+_fmt_badge() {
+  local badge="$1" tag="$2" inner
+  inner=$(printf -- '%-7s' "$tag")
+  if [ "${BOX:-0}" = "1" ]; then
+    printf '%s┌ %s┐%s' "$badge" "$inner" "$N"
+  else
+    printf '%s[ %s]%s' "$badge" "$inner" "$N"
+  fi
+}
+# One concrete line: ┌ OK    ┐ tid  label  ·  detail
+# $1=badge style  $2=tag text  $3=label colour  $4=label  $5=detail
 _msg_line() {
   local badge="$1" tag="$2" lcol="$3" label="$4" detail="${5:-}"
-  # Fixed-width badge cell: " OK     " / " BLOCKER" (9 chars, space-padded)
-  local tcell
-  tcell=$(printf ' %-7s ' "$tag")
   if [ -n "$detail" ]; then
-    printf '%s%s%s %s%s%s%s %s·%s %s%s%s\n' \
+    printf -- '%s %s%s%s%s %s·%s %s%s%s\n' \
-      "$badge" "$tcell" "$N" "$(_fmt_tid)" "$lcol" "$label" "$N" "$D" "$N" "$D" "$detail" "$N"
+      "$(_fmt_badge "$badge" "$tag")" "$(_fmt_tid)" "$lcol" "$label" "$N" "$D" "$N" "$D" "$detail" "$N"
   else
-    printf '%s%s%s %s%s%s%s\n' \
+    printf -- '%s %s%s%s%s\n' \
-      "$badge" "$tcell" "$N" "$(_fmt_tid)" "$lcol" "$label" "$N"
+      "$(_fmt_badge "$badge" "$tag")" "$(_fmt_tid)" "$lcol" "$label" "$N"
   fi
 }
 
@@ -484,33 +550,58 @@ blocker() {
   ERRORS+=("BLOCKER ${LAST_TID:+$LAST_TID }[$step] $msg")
 }
 section() {
-  # Blue badge + bold title
+  # Boxed section header (3 lines when colour on)
-  local title=" $* "
+  local title="$*"
-  printf '\n%s%s%s %s%s%s\n' "$BG_SEC" " == " "$N" "$B" "$title" "$N"
+  local w=${#title}
+  [ "$w" -lt 24 ] && w=24
+  [ "$w" -gt 56 ] && w=56
+  local pad line i
+  pad=$(printf -- "%-${w}s" "$title")
+  if [ "${BOX:-0}" = "1" ]; then
+    line=""
+    i=0
+    while [ "$i" -lt $((w + 2)) ]; do
+      line="${line}═"
+      i=$((i + 1))
+    done
+    printf -- '\n%s╔%s╗%s\n' "$BG_SEC" "$line" "$N"
+    printf -- '%s║%s %s%s%s %s║%s\n' "$BG_SEC" "$N" "$B" "$pad" "$N" "$BG_SEC" "$N"
+    printf -- '%s╚%s╝%s\n' "$BG_SEC" "$line" "$N"
+  else
+    printf -- '\n== %s ==\n' "$title"
+  fi
 }
 
 summary() {
   echo ""
   if [ "${#BLOCKERS[@]}" -gt 0 ]; then
-    printf '%s BLOCKERS %s %s(pay/withdraw cannot finish)%s\n' "$BG_BLK" "$N" "$D" "$N"
+    if [ "${BOX:-0}" = "1" ]; then
+      printf -- '%s┌ BLOCKERS · pay/withdraw cannot finish ┐%s\n' "$BG_BLK" "$N"
+    else
+      printf -- '--- BLOCKERS (pay/withdraw cannot finish) ---\n'
+    fi
     local b
     for b in "${BLOCKERS[@]}"; do
-      printf '%s  •%s %s%s%s\n' "$M" "$N" "$W" "$b" "$N"
+      printf -- '%s  •%s %s%s%s\n' "$M" "$N" "$W" "$b" "$N"
     done
   fi
   if [ "${#ERRORS[@]}" -gt 0 ] && [ "${#BLOCKERS[@]}" -lt "${#ERRORS[@]}" ]; then
-    printf '%s ERRORS %s %s(failed checks)%s\n' "$BG_ERR" "$N" "$D" "$N"
+    if [ "${BOX:-0}" = "1" ]; then
+      printf -- '%s┌ ERRORS · failed checks ┐%s\n' "$BG_ERR" "$N"
+    else
+      printf -- '--- ERRORS (failed checks) ---\n'
+    fi
     local e
     for e in "${ERRORS[@]}"; do
       case "$e" in BLOCKER*) continue ;; esac
-      printf '%s  •%s %s%s%s\n' "$R" "$N" "$W" "$e" "$N"
+      printf -- '%s  •%s %s%s%s\n' "$R" "$N" "$W" "$e" "$N"
     done
   fi
-  printf '%stotals:%s %s%d OK%s' "$D" "$N" "$G" "$PASS_N" "$N"
+  printf -- '%stotals:%s %s%d OK%s' "$D" "$N" "$G" "$PASS_N" "$N"
-  [ "$FAIL_N" -gt 0 ] && printf ', %s%d ERROR%s' "$R" "$FAIL_N" "$N"
+  [ "$FAIL_N" -gt 0 ] && printf -- ', %s%d ERROR%s' "$R" "$FAIL_N" "$N"
-  [ "$WARN_N" -gt 0 ] && printf ', %s%d WARN%s' "$Y" "$WARN_N" "$N"
+  [ "$WARN_N" -gt 0 ] && printf -- ', %s%d WARN%s' "$Y" "$WARN_N" "$N"
-  [ "$INFO_N" -gt 0 ] && printf ', %s%d INFO%s' "$C" "$INFO_N" "$N"
+  [ "$INFO_N" -gt 0 ] && printf -- ', %s%d INFO%s' "$C" "$INFO_N" "$N"
-  [ "${#BLOCKERS[@]}" -gt 0 ] && printf ', %s%d BLOCKER%s' "$M" "${#BLOCKERS[@]}" "$N"
+  [ "${#BLOCKERS[@]}" -gt 0 ] && printf -- ', %s%d BLOCKER%s' "$M" "${#BLOCKERS[@]}" "$N"
   printf '\n'
   [ "$FAIL_N" -eq 0 ]
 }