monitoring: boxed badges + area.group test IDs

This commit is contained in:
Hernâni Marques 2026-07-17 15:55:50 +02:00
parent f8affeb651
commit e1cc6d7141
No known key found for this signature in database
GPG key ID: CB5738652768F7E9
10 changed files with 286 additions and 225 deletions

View file

@ -1,24 +1,26 @@
# taler-monitoring
Report for the **GOA** stack with clear severity tags and **per-area test IDs**:
Report for the **GOA** stack with boxed severity badges and **grouped test IDs**
`area.group-NN` (e.g. `www.exchange-01`, `e2e.pay-03`):
| Tag | Meaning |
|-----|---------|
| `[OK]` | check passed |
| `[INFO]` | inside status (container, ports, log noise) |
| `[WARN]` | degraded but maybe not fatal |
| `[ERROR]` | component problem |
| `[BLOCKER]` | **withdraw/pay path cannot complete** because of this |
| `┌ OK ┐` | check passed |
| `┌ INFO ┐` | status / note |
| `┌ WARN ┐` | degraded but maybe not fatal |
| `┌ ERROR ┐` | component problem |
| `┌ BLOCKER ┐` | **withdraw/pay path cannot complete** because of this |
IDs: **`www-001`**, **`inside-001`**, **`versions-001`**, **`sanity-001`**, **`server-001`**, **`e2e-001`** …
IDs reset per **group** inside an area — so “too many www tests” become
`www.exchange-*`, `www.bank-*`, `www.landing-*`, etc.
Catalog: **[TESTS.md](./TESTS.md)**.
```text
[OK] www-001 exchange /config https://exchange…/config
[BLOCKER] e2e-015 prereq: merchant HTTP 502
┌ OK ┐ www.exchange-01 exchange /config · HTTP 200
┌ BLOCKER┐ e2e.prereq-02 merchant HTTP 502
```
End of each phase: totals + list of **BLOCKERS** and **ERRORS**.
End of each phase: totals + list of **BLOCKERS** and **ERRORS** (ids included).
## Commands

View file

@ -1,217 +1,132 @@
# taler-monitoring — test IDs by area
# taler-monitoring — grouped test IDs
Every check line is numbered **per area** as `AREA-NNN` (zero-padded):
| Area | Phase script | Meaning |
|------|--------------|---------|
| **www** | `check_urls.sh` | public HTTPS (outside-in) |
| **inside** | `check_inside.sh` | containers / processes on koopa |
| **versions** | `check_versions.sh` | deb.taler.net available + packages vs trixie |
| **sanity** | `check_sanity.sh` | public + server per component |
| **server** | `check_server.sh` | SSH host ports / processes |
| **e2e** | `check_e2e.sh` | withdraw + pay cycle |
Format in output:
Every check line is numbered **`area.group-NN`** (two-digit group counter):
```text
[OK] www-001 exchange /config https://exchange…/config
[ERROR] e2e-012 bank-auth: admin token failed
[BLOCKER] e2e-015 prereq: merchant HTTP 502
┌ OK ┐ www.exchange-01 exchange /config · HTTP 200
┌ ERROR ┐ e2e.pay-03 order create failed · HTTP 502
┌ BLOCKER┐ e2e.prereq-02 merchant secret missing
```
IDs are assigned **in run order** within the area (`set_area` resets the counter). Optional soft checks still consume a number when they WARN.
| Area | Phase script | Groups (examples) |
|------|--------------|-------------------|
| **www** | `check_urls.sh` | `exchange` `perf` `stats` `bank` `merchant` `paivana` `landing` |
| **inside** | `check_inside.sh` | `ssh` `bank` `exchange` `merchant` `caddy` `load` |
| **versions** | `check_versions.sh` | `outside` `inside` `compare` |
| **sanity** | `check_sanity.sh` | `bank` `exchange` `merchant` |
| **server** | `check_server.sh` | (flat `server-NN` or host groups) |
| **e2e** | `check_e2e.sh` | `prereq` `load` `bank` `wallet` `atm` `settle` `pay` `shop` `paivana` `dig` `report` |
| **ladder** | `check_goa_ladder.sh` | `plan` `load` `withdraw` `pay` `report` |
**Why groups:** flat `www-001``www-080` was hard to talk about.
`www.bank-04` / `e2e.atm-02` pin the failure to a logical block.
Usage in scripts:
```bash
set_area www
set_group exchange # → www.exchange-01, www.exchange-02, …
set_group bank # counter resets → www.bank-01
```
`set_group` also prints a small group chip (`┌ www.exchange ┐`) so logs show section boundaries.
Re-entering the same group later (e.g. exchange legal after perf) **continues** NN — `www.exchange-03`, not a second `-01`.
Without `set_group`, IDs stay flat: `area-01`, `area-02`, …
Optional soft checks still consume a number when they WARN.
Numbering follows **executed** checks (early skip may shift later NN inside the same group).
---
## www — public URLs (`./taler-monitoring.sh urls`)
| ID | Check |
|----|--------|
| www-… | exchange `/config`, currency, **alt_unit_names** |
| www-… | exchange `/keys` (+ alt_unit_names soft) |
| www-… | exchange `/intro/`, `/` (302→intro) |
| www-… | **exchange `/terms`** body (not empty / not API error) |
| www-… | **exchange `/privacy`** body |
| www-… | exchange `/terms/` (200 or redirect) |
| www-… | bank `/config`, currency, **alt_unit_names** |
| www-… | bank integration / webui / intro / `/` |
| www-… | **bank `/terms`** body |
| www-… | **bank `/privacy`** (or `/intro/privacy.html` fallback) |
| www-… | merchant `/config`, currency, currencies alt_unit_names |
| www-… | each merchant `exchanges[]` `/config` alt_unit_names |
| www-… | merchant `/intro/`, `/webui/`, `/` |
| www-… | **merchant `/terms`** body (dual-currency notice) |
| www-… | **merchant `/privacy`** body (must not be `not configured`) |
| www-… | merchant `/terms/` redirect |
| www-… | **landing (aggregated per site)**: one line for bank/merchant/exchange (`/intro` + assets + own/external link counts); failures listed as samples; soft external/shop assets compact |
| www-… | cross-links: one line (4 intros) on local stack |
| www-… | bank demo-withdraw + shop assets: one line (+ soft shop-pay if missing) |
| www-… | **bank `/intro/demo-withdraw.json`**`taler://withdraw/HOST:PORT/taler-integration/…` + integration op HTTP 200 |
| www-… | **landing exposed links** (bank / merchant / exchange): parse each `/intro/` HTML, probe every own-stack `https://` + root-relative `href`/`src`/`content`, soft-check external stores/docs |
| www-… | landing static: `qrcode.min.js`, `og-goa-shop.png`, `qr-logo.png`, shop-pay.js/css |
| www-… | cross-links between bank ↔ merchant ↔ exchange intros (local stack) |
| www-… | **bank `/intro/demo-withdraw.json`**`taler://withdraw/HOST/taler-integration/…` (no default `:443`/`:80`; non-default port OK) + integration op HTTP 200 |
| www-… | bank `/intro/auto-account.json` (earlier) → same withdraw shape, **no payto_uri**, login at `/webui/` |
| www-… | **performance** (outside-in RTT): bank/exchange/merchant `/config`, keys, webui, intro — ms; WARN ≥ `PERF_WARN_MS` (8000); ERROR ≥ `PERF_FAIL_MS` (20000) |
| www-… | **landing load stats**: `/intro/stats.json` → loadavg, container RSS, in-container probe ms (public; SSH container cat fallback); warn if stale/missing |
| Group | Checks (typical) |
|-------|------------------|
| **www.exchange-** | `/config`, currency, **alt_unit_names**; `/keys` (+ alt soft); `/intro/`, `/`; **`/terms`**, **`/privacy`**, `/terms/` |
| **www.perf-** | outside-in RTT: bank/exchange/merchant `/config`, keys, webui, intro — ms; WARN ≥ `PERF_WARN_MS` (8000); ERROR ≥ `PERF_FAIL_MS` (20000) |
| **www.stats-** | `/intro/stats.json` loadavg / container RSS / probe ms (public; SSH fallback) |
| **www.bank-** | `/config`, currency, alt_unit_names; integration/webui/intro; **auto-account.json**; `/terms`, `/privacy` |
| **www.merchant-** | `/config` currency + currencies alt_unit_names; listed exchanges alt; webui/intro; **`/terms`**, **`/privacy`** |
| **www.paivana-** | local GOA paywall front (redirect to template) |
| **www.landing-** | every own-stack link on bank/merchant/exchange intros; static assets; demo-withdraw / cross-links |
**Legal docs rule:** HTTP 200, non-empty body, not plain `not configured`, not merchant API JSON `code:21`. On local stack, optional content needle (terms/privacy/FADP/GOA…).
**Legal docs rule:** HTTP 200, non-empty body, not plain `not configured`, not merchant API JSON `code:21`. Local stack may require content needle (terms/privacy/FADP/GOA…).
**Performance rule:** Measured from the **monitoring runner** (public URLs via Caddy), not container loopback. HTTP must match expect (usually 200); **latency (ms) is on every perf OK/WARN/ERROR line**, plus a **perf summary** (n / min / p50 / avg / max). Slow ≥ `PERF_WARN_MS` (default 8000) → WARN; ≥ `PERF_FAIL_MS` (default 20000) → ERROR.
**Performance rule:** Measured from the **monitoring runner** (public via Caddy), not container loopback. Latency (ms) on every perf line + **perf summary** (n / min / p50 / avg / max).
**Landing links rule:** Own-stack (bank/exchange/taler.\* + page host) must be HTTP 200 (or redirect→200). External (App Store, Play, F-Droid, wallet.taler.net, docs/git.taler.net, …) soft WARN if down. Auto-account wallet link must be `taler://withdraw/HOST/taler-integration/…` (default ports stripped for mobile wallets), never payto.
**Landing links rule:** Own-stack must be HTTP 200 (or redirect→200). External stores/docs soft WARN. Auto-account wallet link must be `taler://withdraw/HOST/taler-integration/…` (default ports stripped), never payto.
**alt_unit_names rule:** wallet codec requires a non-empty map including scale key `"0"`. For multi-currency merchant, also follow every entry in `exchanges[]` and check that exchanges public `/config`.
(IDs after a failed early check may shift if later soft checks are skipped when body missing — numbering follows **executed** checks.)
**alt_unit_names rule:** non-empty map including scale key `"0"`. Multi-currency merchant: follow each `exchanges[]` public `/config`.
---
## inside — koopa SSH (`./taler-monitoring.sh inside`)
| ID | Check (typical order) |
|----|------------------------|
| inside-001 | ssh koopa (or **koopa-external** fallback) |
| inside-002+ | per-component emit: container, ports, libeufin/httpd, postgres, local `/config`/`/keys`, wirewatch, DNS pin, caddy |
| inside-… | **load / memory**: host loadavg + RAM; bank/exchange/merchant podman CPU/mem/block + process RSS by role + DB sizes |
| Group | Checks |
|-------|--------|
| **inside.ssh-** | ssh reachability / remote collect |
| **inside.bank-** | container, libeufin, postgres, local `/config`, nginx, DNS pins |
| **inside.exchange-** | container, httpd, wirewatch, aggregator, transfer, local keys, DNS |
| **inside.merchant-** | container, httpd, wirewatch, depositcheck, DNS |
| **inside.caddy-** | host reverse-proxy process |
| **inside.load-** | host loadavg + RAM; per-container RSS/CPU |
Remote lines `E|comp|LEVEL|key|detail` each become one numbered result.
Remote lines `E|comp|LEVEL|key|detail` each become one numbered result under that component group.
SSH: `KOOPA_SSH` (default `koopa`), then `KOOPA_SSH_FALLBACKS` (default `koopa-external`) when LAN is unreachable.
SSH: `KOOPA_SSH` (default `koopa`), then `KOOPA_SSH_FALLBACKS` when LAN is unreachable.
---
## sanity — bank · exchange · merchant (`./taler-monitoring.sh sanity`)
## e2e — withdraw + pay (`./taler-monitoring.sh e2e`)
| ID | Section |
|----|---------|
| sanity-001… | bank public + server |
| sanity-… | exchange public + server |
| sanity-… | merchant public + server |
| Group | Checks |
|-------|--------|
| **e2e.prereq-** | wallet-cli, currency, budgets, secrets, public reachability |
| **e2e.load-** | host/container load snapshots (before / after ATM / after pay) |
| **e2e.bank-** | account, credit, withdraw ops |
| **e2e.wallet-** | exchange + ToS, wallet setup |
| **e2e.atm-** | ATM withdraw ladder rungs |
| **e2e.settle-** | wallet settlement wait / spendable balance |
| **e2e.pay-** | variable payments |
| **e2e.shop-** | GOA shop product catalog pays |
| **e2e.paivana-** | template paywall pay |
| **e2e.dig-** | coins-missing dig (inside + withdrawal-operation) |
| **e2e.report-** | final tallies |
Sequential through the whole script (one `set_area sanity`).
When filing an issue, quote the full id + label, e.g.
`e2e.atm-03 ATM withdraw 5 GOA · bank confirmed, wallet empty`.
---
## versions — packages vs deb.taler.net (`./taler-monitoring.sh versions`)
## versions / sanity / ladder / server
### Outside (runner / public network — no SSH)
| ID (order) | Check |
|------------|--------|
| versions-… | DNS `deb.taler.net` |
| versions-… | HTTPS portal + apt base URL |
| versions-… | suite `InRelease` / `Release` |
| versions-… | suite `Packages` + `Packages.gz` |
| versions-… | sample pool `.deb` fetchable (Range 200/206) |
| versions-… | suite offers `taler-exchange`, `taler-merchant`, `libeufin-bank` |
| versions-… | optional `trixie-testing` Packages |
| versions-… | TLS verify (soft) |
### Inside (SSH koopa containers)
| ID | Check |
|----|--------|
| versions-… | ssh koopa |
| versions-… | each container → `InRelease` (pasta can reach apt repo) |
| versions-… | each container lists `deb.taler.net` in apt sources |
| versions-… | each installed `taler*` / `libeufin*` / `libtaler*` / `libdonau*` vs suite version |
| versions-… | core packages installed (`taler-exchange`, `libeufin-bank`, `taler-merchant`) |
Outside always runs. Inside skipped with `SKIP_SSH=1` (still reports outside results).
Compare rules:
- **match** suite → OK
- **ahead** of suite (often testing/dev) → INFO
- **behind** suite → ERROR for core packages, WARN otherwise (`TALER_PKG_BEHIND=error` forces ERROR)
Default suite: **trixie** (`TALER_APT_SUITE`, `TALER_APT_BASE=https://deb.taler.net/apt/debian`).
Without SSH (`SKIP_SSH=1` or remote domain): still runs outside-in repo checks; skips container install compare.
| Area.group | Meaning |
|------------|---------|
| **versions.outside-** | deb.taler.net suite index |
| **versions.inside-** | packages in containers |
| **versions.compare-** | installed vs suite |
| **sanity.bank-** / **.exchange-** / **.merchant-** | public + optional server-side per component |
| **ladder.plan-** / **.load-** / **.withdraw-** / **.pay-** / **.report-** | GOA amount ladder |
| **server-** | SSH host ports / processes (flat unless grouped later) |
---
## server — SSH ports (`./taler-monitoring.sh server`)
| ID | Check |
|----|--------|
| server-001 | ssh |
| server-002+ | containers, local pasta ports, processes, caddy |
---
## e2e — payment path (`./taler-monitoring.sh e2e`)
Local GOA also: **ATM includes GOA:4200**, then **paivana** (HTTP 302 on `PAIVANA_PUBLIC` + public template pay `paivana` / GOA:4200 on `goa-shop`). Disable with `E2E_PAIVANA=0`.
| ID | Step (approx.) |
|----|----------------|
| e2e-001 | budget info |
| e2e-002 | wallet-cli present |
| e2e-003 | mode / currency info |
| e2e-004… | secrets, reachability gates |
| e2e-… | account, credit, withdraw, confirm, coins, order, pay ladder |
| e2e-… | **load snapshots** at e2e-start, after-withdraw, after-pay, after-shop, e2e-end (host loadavg/RAM + container RSS/CPU/DB) |
| e2e-… | **coins** before/after each ATM withdraw, settle, pay, shop: count in circulation / spent, amount + denoms using **`alt_unit_names`** (e.g. `5 Kilo-GOA (GOA:5000)`), Δ vs previous snap |
| e2e-… | **GOA shop products** — full catalog list; **random pick of 2** (override `E2E_SHOP_PICK_N`) |
| e2e-… | balances, dig on failure, load delta + overall metrics |
Shop product pays use instance `goa-shop` (default) and catalog `E2E_SHOP_PRODUCTS`
(`id|Product name|amount` lines). Each e2e run **shuffles** the catalog and pays
only `E2E_SHOP_PICK_N` products (**default 2**). Flow matches the landing popup
(public POST `/templates/{id}`, not private orders). Report labels use product name.
Blockers keep the same ID prefix: `[BLOCKER] e2e-0NN step: message`.
---
## ladder — GOA withdraw ranges (`./taler-monitoring.sh ladder`)
| Step | What |
|------|------|
| ladder-001 | `GET /intro/auto-account.json` (personal account, GOA:0) |
| ladder-… | **load snapshot** before withdraws (host + bank/exchange/merchant mem/CPU) |
| ladder-… | **Phase A withdraw** (23): mint explorer pool — 0 → random → max1 → max into **one cumulative wallet** (mids × `LADDER_WITHDRAW_SCALE`, default 1.5) |
| ladder-… | wallet-cli accept-uri + explorer confirm when `selected`; settle; coins snap |
| ladder-… | **Phase B pay** (23): same shape 0 → random → max1 → max via merchant orders + `handle-uri --yes` |
| ladder-… | **load snapshot** after + delta + overall timing (withdraw + pay) |
| report | withdraw TSV + pay TSV + JSON |
Default amounts (`build_ladder_pair`, strictly increasing):
## Visual format (colour on)
```text
withdraw: [0] + random mids + [max1] + [max]
pay: [0] + (withdraw_mid / scale) + [max1] + [max] # always ≤ matching withdraw mid
╔══════════════════════════════╗
║ www · public URLs · … ║
╚══════════════════════════════╝
┌ www.exchange ┐
┌ OK ┐ www.exchange-01 exchange /config · HTTP 200
┌ www.bank ┐
┌ OK ┐ www.bank-01 bank /config · HTTP 200
┌ ERROR ┐ www.bank-04 bank /intro/auto-account.json · invalid withdraw
┌ ERRORS · failed checks ┐
• www.bank-04 bank /intro/auto-account.json · invalid withdraw
totals: 40 OK, 1 ERROR, 2 WARN, 5 INFO
```
TSV `range`: `pin:0` | `random` | `pin:max-1` | `pin:max`.
Soft: absolute **max** mint/pay may `CEILING_REJECT` / skip (WARN). **max-1** is a hard pin.
`LADDER_PAY=0` skips phase B.
| Env | Default | Meaning |
|-----|---------|---------|
| `LADDER_STEPS` | `23` | total rungs (0 + mids + max) |
| `LADDER_MAX_AMOUNT` | `4503599627370496` | absolute last pin |
| `LADDER_TIMEOUT_S` | `3600` | large rungs need time |
| `LADDER_MAX_RUNGS` | `99` | cap list length after build |
```bash
./taler-monitoring.sh ladder
LADDER_STEPS=10 ./taler-monitoring.sh ladder
LADDER_REPORT_DIR=/tmp/my-ladder ./taler-monitoring.sh ladder
```
---
## Run one area
```bash
./taler-monitoring.sh urls # www only
./taler-monitoring.sh inside # inside only
./taler-monitoring.sh versions # deb.taler.net + package drift
./taler-monitoring.sh e2e # e2e only
./taler-monitoring.sh ladder # GOA amount ladder + timings
./taler-monitoring.sh -d taler.net urls
```
`NO_COLOR=1` or `CLICOLOR=0` disables boxes/colours (ASCII `[ OK ]` + `-- group --` headers).
Always use `printf --` friendly plain headers when colour is off (leading `---` is not a printf option).

View file

@ -182,6 +182,7 @@ e2e_skip_rest() {
# Area e2e-### — withdraw + pay cycle
set_area e2e
set_group prereq
section "e2e · prerequisites"
info "e2e budget" "${E2E_TIMEOUT}s (set E2E_TIMEOUT= to change)"
@ -382,10 +383,12 @@ else
fi
# Host + bank/exchange/merchant RAM/CPU/load (SSH koopa or koopa-external)
set_group load
section "e2e · load snapshot (before withdraw/pay)"
metrics_report_load "${METRICS_DIR}/load-before.json" "e2e-start" || true
# Local stack: koopa secrets. Remote: only explicit env (never leak local passwords to foreign banks).
set_group prereq
if [ "$E2E_REMOTE" = "1" ]; then
ADMIN_PASS="${E2E_BANK_ADMIN_PASS:-}"
MPW="${E2E_MERCHANT_TOKEN:-${MERCHANT_TOKEN:-}}"
@ -496,6 +499,7 @@ wcli_pay() {
# When coins never become available: run inside + targeted bank/exchange dig
dig_when_no_coins() {
set_group dig
section "e2e · coins missing — inside dig"
info "reason" "wallet empty after withdraw path (often wirewatch timing) — probing bank + exchange"
if [ -n "${WID:-}" ]; then
@ -580,6 +584,7 @@ BAL_BEFORE=$(fmt_bal "$SCRATCH/bal-before.out")
info "BALANCE before" "$BAL_BEFORE"
# ---------------------------------------------------------------------------
set_group bank
section "e2e · bank (account · credit · withdraw)"
# ---------------------------------------------------------------------------
AT=""
@ -678,6 +683,7 @@ else
fi
# ---------------------------------------------------------------------------
set_group wallet
section "e2e · wallet setup (exchange + ToS once)"
# ---------------------------------------------------------------------------
if ! wcli exchanges add "${EXCHANGE_PUBLIC}/" >"$SCRATCH/ex-add.out" 2>&1; then
@ -1079,6 +1085,7 @@ sys.exit(0 if d.get("paid") is True or str(d.get("order_status","")).lower()=="p
}
# ---------------------------------------------------------------------------
set_group atm
section "e2e · ATM withdraw ladder"
# ---------------------------------------------------------------------------
WITHDRAW_OK=0
@ -1109,12 +1116,14 @@ for WITHDRAW_AMT in $WITHDRAW_LIST; do
esac
done
info "ATM withdraw summary" "$WITHDRAW_REPORT (ok=$WITHDRAW_OK_N lag=$WITHDRAW_LAG_N fail=$WITHDRAW_FAIL_N)"
set_group load
section "e2e · load snapshot (after ATM withdraws)"
metrics_report_load "${METRICS_DIR}/load-after-withdraw.json" "after-withdraw" || true
cp -f "${METRICS_DIR}/load-after-withdraw.json" "${METRICS_DIR}/load-after.json" 2>/dev/null || true
metrics_report_coins "after-ATM-ladder" || true
# Settlement catch-up: bank may have confirmed while wallet was still empty
set_group settle
section "e2e · wallet settlement (timing)"
if ! wait_wallet_balance 0 "${E2E_SETTLE_ROUNDS}" "${E2E_SETTLE_SLEEP}"; then
if [ "$WITHDRAW_LAG_N" -gt 0 ] || [ "$WITHDRAW_OK_N" -gt 0 ]; then
@ -1154,6 +1163,7 @@ else
fi
# ---------------------------------------------------------------------------
set_group pay
section "e2e · variable payments (if balance allows)"
# ---------------------------------------------------------------------------
PAY_OK=0
@ -1193,6 +1203,7 @@ for PAY_AMT in $PAY_LIST; do
fi
done
info "pay summary" "$PAY_REPORT (ok=$PAY_OK_N fail=$PAY_FAIL_N skip=$PAY_SKIP_N)"
set_group load
section "e2e · load snapshot (after payments)"
metrics_report_load "${METRICS_DIR}/load-after-pay.json" "after-pay" || true
cp -f "${METRICS_DIR}/load-after-pay.json" "${METRICS_DIR}/load-after.json" 2>/dev/null || true
@ -1207,6 +1218,7 @@ if [ "$PAY_OK" != "1" ]; then
fi
# ---------------------------------------------------------------------------
set_group shop
section "e2e · GOA shop products (merchant landing catalog)"
# ---------------------------------------------------------------------------
# Public template POST + wallet pay — full catalog list, random pick of N products.
@ -1297,6 +1309,7 @@ EOF
fi
# ---------------------------------------------------------------------------
set_group paivana
section "e2e · paivana paywall (template GOA:4200)"
# ---------------------------------------------------------------------------
# Public paywall: https://paivana.hacktivism.ch → 302 to merchant template "paivana".
@ -1388,6 +1401,7 @@ else
info "paivana summary" "${PAIVANA_REPORT:-?} · instance ${E2E_PAIVANA_INSTANCE}"
fi
set_group report
section "e2e · report"
info "user" "$USER"
info "ATM withdraws" "$WITHDRAW_REPORT"

View file

@ -29,7 +29,10 @@ ROOT=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=lib.sh
source "$ROOT/lib.sh"
# Area ladder.* — GOA withdraw/pay ladder
# Groups: ladder.plan / ladder.load / ladder.withdraw / ladder.pay / ladder.report
set_area ladder
set_group plan
SECTION_T0=$(date +%s)
now_ms() { python3 -c 'import time; print(int(time.time()*1000))'; }
elapsed_ms() {
@ -261,6 +264,7 @@ info "steps" "${LADDER_STEPS} (0 + random≥${LADDER_MIN_AMOUNT} + max-1=${CUR}:
info "withdraw_scale" "${LADDER_WITHDRAW_SCALE}× pay mids (fund pay ladder)"
info "pay_phase" "$([ "${LADDER_PAY}" = "1" ] && echo enabled || echo disabled)"
set_group load
section "ladder · load snapshot (before withdraws)"
metrics_report_load "$LOAD_BEFORE" "ladder-start" || true
# Fresh wallet per rung — baseline empty (or last-rung DB if re-used later)
@ -344,6 +348,7 @@ STOP_REASON=""
STOP_AMOUNT=""
declare -a RUNG_JSON=()
set_group withdraw
section "ladder · phase A · withdraw (${LADDER_N} rungs)"
rung=0
for AMT in "$@"; do
@ -787,6 +792,7 @@ done
PAY_OK_N=0
PAY_FAIL_N=0
if [ "${LADDER_PAY}" = "1" ] && [ -n "${PAY_LIST:-}" ] && [ "$FAIL_N_L" -eq 0 ]; then
set_group pay
section "ladder · phase B · pay"
metrics_report_coins "before-pay-ladder" || true
# Merchant secret (same as e2e)
@ -1024,6 +1030,7 @@ fi
ms_phase=$(python3 -c 'import sys,time; print(int((time.time()-float(sys.argv[1]))*1000))' "$SECTION_T0")
# --- report ---
set_group report
section "ladder · report"
info "auto-account" "$ACCT_USER"
info "ok_rungs" "$OK_N"
@ -1128,6 +1135,7 @@ PY
wcli balance 2>&1 | tee "$REPORT_DIR/balance-final.out" | tail -20 || true
set_group load
section "ladder · load snapshot (after withdraws)"
metrics_report_load "$LOAD_AFTER" "ladder-end" || true
if [ -f "$LOAD_BEFORE" ] && [ -f "$LOAD_AFTER" ]; then

View file

@ -7,8 +7,10 @@ source "$ROOT/lib.sh"
# shellcheck source=metrics.sh
source "$ROOT/metrics.sh"
# Area inside-### — container / process state on koopa (SSH)
# Area inside.* — container / process state on koopa (SSH)
# Groups: inside.ssh / inside.bank / inside.exchange / inside.merchant / inside.caddy
set_area inside
set_group ssh
section "inside · collect from koopa"
if [ "${SKIP_SSH}" = "1" ]; then
@ -128,10 +130,20 @@ if [ -z "$RAW" ] || ! echo "$RAW" | grep -q '^E|'; then
exit 1
fi
_last_inside_grp=""
while IFS= read -r line; do
case "$line" in
E\|*)
IFS='|' read -r _ comp level key detail <<<"$line"
# Group IDs by component so issues map cleanly: inside.bank-02, inside.exchange-04
case "$comp" in
bank|exchange|merchant|caddy) _g="$comp" ;;
*) _g="ssh" ;;
esac
if [ "$_g" != "$_last_inside_grp" ]; then
set_group "$_g"
_last_inside_grp="$_g"
fi
case "$level" in
OK) ok "[$comp] $key${detail:+ ($detail)}" ;;
ERROR) err "$comp" "$key" "$detail" ;;
@ -143,6 +155,7 @@ while IFS= read -r line; do
done <<<"$RAW"
# Host loadavg + RAM + per-container RSS/CPU (same probe as e2e/ladder)
set_group load
section "inside · load / memory"
METRICS_DIR="${METRICS_DIR:-$(mktemp -d)}"
export METRICS_DIR

View file

@ -23,10 +23,12 @@ json_currency() {
python3 -c 'import json,sys; d=json.load(open(sys.argv[1])); print(d.get("currency") or "")' "$1" 2>/dev/null || true
}
# Area sanity-### — public + optional server-side per component
# Area sanity.* — public + optional server-side per component
# Groups: sanity.bank / sanity.exchange / sanity.merchant
set_area sanity
# ---------------------------------------------------------------------------
set_group bank
section "sanity · bank"
# ---------------------------------------------------------------------------
expect_code "bank public /config" 200 "$BANK_PUBLIC/config"
@ -98,6 +100,7 @@ else
fi
# ---------------------------------------------------------------------------
set_group exchange
section "sanity · exchange"
# ---------------------------------------------------------------------------
expect_code "exchange public /config" 200 "$EXCHANGE_PUBLIC/config"
@ -193,6 +196,7 @@ else
fi
# ---------------------------------------------------------------------------
set_group merchant
section "sanity · merchant"
# ---------------------------------------------------------------------------
expect_code "merchant public /config" 200 "$MERCHANT_PUBLIC/config"

View file

@ -5,8 +5,9 @@ ROOT=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=lib.sh
source "$ROOT/lib.sh"
# Area server-### — host/container ports via SSH
# Area server.* — host/container ports via SSH
set_area server
set_group ssh
section "server · ssh ${KOOPA_SSH}"
if ! koopa_ssh_ok; then

View file

@ -5,7 +5,9 @@ ROOT=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=lib.sh
source "$ROOT/lib.sh"
# Area www-### — public HTTPS (outside-in)
# Area www.* — public HTTPS (outside-in)
# Groups: www.exchange / www.perf / www.stats / www.bank / www.merchant /
# www.paivana / www.landing
set_area www
section "www · public URLs · ${TALER_DOMAIN:-?} (outside-in, no SSH)"
@ -169,7 +171,8 @@ sys.exit(0)
PY
}
# --- exchange (core; always required) --- www-001 …
# --- exchange (core; always required) --- www.exchange-NN
set_group exchange
check_url "exchange /config" 200 "$EXCHANGE_PUBLIC/config"
code=$(http_body "$EXCHANGE_PUBLIC/config" "$tmp/ec.json")
if [ "$code" = "200" ]; then
@ -219,6 +222,7 @@ fi
# Performance — outside-in public HTTPS latency (this runner, not loopback)
# Same spirit as bank landing-stats public probes; measured here from outside.
# ---------------------------------------------------------------------------
set_group perf
section "www · performance · public HTTPS latency (outside-in)"
# Latency thresholds (ms), outside-in from this runner.
@ -392,6 +396,7 @@ PY
}
if [ "${CHECK_LANDING:-1}" = "1" ] || [ "${LOCAL_STACK:-0}" = "1" ]; then
set_group stats
section "www · performance · landing load stats (stats.json · in-container probes)"
report_landing_load_stats "bank" "$BANK_PUBLIC"
report_landing_load_stats "exchange" "$EXCHANGE_PUBLIC"
@ -470,7 +475,8 @@ fi
info "perf note" "measured from this host (outside-in); thresholds PERF_WARN_MS=${PERF_WARN_MS} PERF_FAIL_MS=${PERF_FAIL_MS}"
# Terms + privacy (legal docs)
# Exchange terms + privacy (same www.exchange group — issues map to exchange)
set_group exchange
check_legal_doc "exchange /terms" "$EXCHANGE_PUBLIC/terms" "terms|GOA|exploration|FADP|revDSG|privacy"
check_legal_doc "exchange /privacy" "$EXCHANGE_PUBLIC/privacy" "privacy|FADP|revDSG|data|GOA|exploration"
# trailing slash: 200 or redirect to bare path
@ -485,6 +491,7 @@ case "$code" in
esac
# --- bank ---
set_group bank
if [ "${LOCAL_STACK:-1}" = "0" ]; then
check_url_soft "bank /config" 200 "$BANK_PUBLIC/config"
else
@ -545,6 +552,7 @@ else
fi
# --- merchant ---
set_group merchant
if [ "${LOCAL_STACK:-1}" = "0" ]; then
check_url_soft "merchant /config" 200 "$MERCHANT_PUBLIC/config"
else
@ -610,6 +618,7 @@ esac
# Paivana paywall (local GOA stack) — public front only; pay path is e2e
# ---------------------------------------------------------------------------
if [ "${LOCAL_STACK:-1}" = "1" ] && [ "${E2E_PAIVANA:-1}" != "0" ]; then
set_group paivana
section "www · paivana paywall"
: "${PAIVANA_PUBLIC:=https://paivana.hacktivism.ch}"
PAIVANA_PUBLIC="${PAIVANA_PUBLIC%/}"
@ -646,6 +655,7 @@ if [ "${CHECK_LANDING:-1}" != "1" ]; then
exit 0
fi
set_group landing
section "www · landing exposed links · bank / merchant / exchange"
# Probe one URL: print code to stdout (200 after following redirects counts as 200).

View file

@ -19,6 +19,7 @@ ROOT=$(cd "$(dirname "$0")" && pwd)
source "$ROOT/lib.sh"
set_area versions
set_group outside
SUITE="${TALER_APT_SUITE:-trixie}"
APT_BASE="${TALER_APT_BASE:-https://deb.taler.net/apt/debian}"
@ -166,6 +167,7 @@ fi
# ---------------------------------------------------------------------------
# 2) INSIDE — containers can reach deb.taler.net (pasta / install path)
# ---------------------------------------------------------------------------
set_group inside
section "versions · inside · containers → deb.taler.net"
if [ "${SKIP_SSH}" = "1" ]; then
@ -266,6 +268,7 @@ ok "collected installed packages" "${n_inst} rows"
# ---------------------------------------------------------------------------
# 3) compare installed vs trixie (and note testing)
# ---------------------------------------------------------------------------
set_group compare
section "versions · compare installed vs ${SUITE}"
export BEHIND_MODE HAVE_TESTING

View file

@ -352,11 +352,12 @@ koopa_ssh_python() {
with_timeout "$t" ssh "${SSH_BASE_OPTS[@]}" "${KOOPA_SSH}" python3 -
}
# ANSI colours — badge tags (fg+bg) + body (label bold, detail dim).
# Off: NO_COLOR=1 or CLICOLOR=0. Badges need a terminal that supports SGR bg.
# ANSI colours — boxed badges (fg+bg) + body (label bold, detail dim).
# Off: NO_COLOR=1 or CLICOLOR=0.
if [ "${NO_COLOR:-0}" = "1" ] || [ "${CLICOLOR:-1}" = "0" ]; then
G= R= Y= C= M= D= W= B= N=
BG_OK= BG_INFO= BG_WARN= BG_ERR= BG_BLK= BG_SEC=
BOX=0
else
G=$'\e[1;32m' # green text (totals)
R=$'\e[1;31m' # red text
@ -367,13 +368,14 @@ else
W=$'\e[1;37m' # bold white label
B=$'\e[1m' # bold section
N=$'\e[0m' # reset
# Badge: bright text on solid background (space-padded in _msg_line)
# Badge fill: bright text on solid background
BG_OK=$'\e[1;30;42m' # black on green
BG_INFO=$'\e[1;30;46m' # black on cyan
BG_WARN=$'\e[1;30;43m' # black on yellow
BG_ERR=$'\e[1;37;41m' # white on red
BG_BLK=$'\e[1;37;45m' # white on magenta
BG_SEC=$'\e[1;37;44m' # white on blue (section)
BOX=1
fi
PASS_N=0
@ -383,17 +385,68 @@ INFO_N=0
BLOCKERS=() # human-readable payment/withdraw blockers
ERRORS=() # all ERROR lines (component scope)
# Test IDs by area: www-001, e2e-001, inside-001, …
# Usage: set_area www then each ok/fail/warn/info/blocker/err auto-numbers.
# Grouped test IDs: area.group-NN (e.g. www.exchange-01, e2e.pay-03)
# set_area www # phase: www | e2e | inside | versions | …
# set_group exchange # resets counter + prints group box → www.exchange-01
# set_group bank # → www.bank-01
# Without set_group: area-01, area-02, … (flat within area)
TEST_AREA=""
TEST_GROUP=""
TEST_N=0
# Last issued id (www-001); set by _take_tid — not via $(…) so TEST_N persists.
LAST_TID=""
# Per-group counters so re-entering www.exchange after perf continues NN
# (shell vars TEST_CNT_<area>_<group>).
_tid_key() {
# $1=area $2=group → safe identifier
printf 'TEST_CNT_%s_%s' "$1" "$2" | tr -c 'A-Za-z0-9_' '_'
}
_tid_save() {
[ -z "${TEST_AREA:-}" ] || [ -z "${TEST_GROUP:-}" ] && return 0
local k
k=$(_tid_key "$TEST_AREA" "$TEST_GROUP")
eval "$k=\$TEST_N"
}
_tid_load() {
local k
k=$(_tid_key "$TEST_AREA" "$1")
eval "TEST_N=\${$k:-0}"
}
set_area() {
# leave previous group counter saved
_tid_save
TEST_AREA="$1"
TEST_GROUP=""
TEST_N=0
LAST_TID=""
}
# Start a logical sub-group. Short stable names:
# exchange bank merchant perf stats landing paivana
# prereq wallet atm settle pay shop dig load report
# ssh caddy outside inside compare withdraw
# Counter continues if the same group is re-entered later in the area.
# Prints a compact group chip so log + issue text map cleanly (www.bank-03).
set_group() {
local g="$1"
# no-op if same group already active (e.g. inside emit loop)
if [ "$g" = "${TEST_GROUP:-}" ] && [ -n "$g" ]; then
return 0
fi
_tid_save
TEST_GROUP="$g"
LAST_TID=""
if [ -z "${TEST_AREA:-}" ] || [ -z "$g" ]; then
TEST_N=0
return 0
fi
_tid_load "$g"
local tag="${TEST_AREA}.${g}"
if [ "${BOX:-0}" = "1" ]; then
# cyan-outline group chip: ┌ www.exchange ┐
printf '%s%s┌ %s ┐%s\n' "$D" "$C" "$tag" "$N"
else
printf -- '-- %s --\n' "$tag"
fi
}
# Assign next id into LAST_TID (must not run in a subshell).
_take_tid() {
LAST_TID=""
@ -401,27 +454,40 @@ _take_tid() {
return
fi
TEST_N=$((TEST_N + 1))
LAST_TID=$(printf '%s-%03d' "$TEST_AREA" "$TEST_N")
if [ -n "${TEST_GROUP:-}" ]; then
LAST_TID=$(printf '%s.%s-%02d' "$TEST_AREA" "$TEST_GROUP" "$TEST_N")
_tid_save
else
LAST_TID=$(printf '%s-%02d' "$TEST_AREA" "$TEST_N")
fi
}
# Dim test id: "www-001 " or empty
# Dim test id: "www.exchange-01 " or empty
_fmt_tid() {
if [ -n "${LAST_TID:-}" ]; then
printf '%s%s%s ' "$D" "$LAST_TID" "$N"
fi
}
# One concrete line: [TAG] tid label · detail
# $1=badge style (bg+fg) $2=tag text $3=label colour $4=label $5=detail
# Boxed badge cell: ┌ OK ┐ with filled bg (tag left-padded, width 7 for BLOCKER)
# $1=badge style $2=tag text
_fmt_badge() {
local badge="$1" tag="$2" inner
inner=$(printf -- '%-7s' "$tag")
if [ "${BOX:-0}" = "1" ]; then
printf '%s┌ %s┐%s' "$badge" "$inner" "$N"
else
printf '%s[ %s]%s' "$badge" "$inner" "$N"
fi
}
# One concrete line: ┌ OK ┐ tid label · detail
# $1=badge style $2=tag text $3=label colour $4=label $5=detail
_msg_line() {
local badge="$1" tag="$2" lcol="$3" label="$4" detail="${5:-}"
# Fixed-width badge cell: " OK " / " BLOCKER" (9 chars, space-padded)
local tcell
tcell=$(printf ' %-7s ' "$tag")
if [ -n "$detail" ]; then
printf '%s%s%s %s%s%s%s %s·%s %s%s%s\n' \
"$badge" "$tcell" "$N" "$(_fmt_tid)" "$lcol" "$label" "$N" "$D" "$N" "$D" "$detail" "$N"
printf -- '%s %s%s%s%s %s·%s %s%s%s\n' \
"$(_fmt_badge "$badge" "$tag")" "$(_fmt_tid)" "$lcol" "$label" "$N" "$D" "$N" "$D" "$detail" "$N"
else
printf '%s%s%s %s%s%s%s\n' \
"$badge" "$tcell" "$N" "$(_fmt_tid)" "$lcol" "$label" "$N"
printf -- '%s %s%s%s%s\n' \
"$(_fmt_badge "$badge" "$tag")" "$(_fmt_tid)" "$lcol" "$label" "$N"
fi
}
@ -484,33 +550,58 @@ blocker() {
ERRORS+=("BLOCKER ${LAST_TID:+$LAST_TID }[$step] $msg")
}
section() {
# Blue badge + bold title
# Boxed section header (3 lines when colour on)
local title="$*"
printf '\n%s%s%s %s%s%s\n' "$BG_SEC" " == " "$N" "$B" "$title" "$N"
local w=${#title}
[ "$w" -lt 24 ] && w=24
[ "$w" -gt 56 ] && w=56
local pad line i
pad=$(printf -- "%-${w}s" "$title")
if [ "${BOX:-0}" = "1" ]; then
line=""
i=0
while [ "$i" -lt $((w + 2)) ]; do
line="${line}"
i=$((i + 1))
done
printf -- '\n%s╔%s╗%s\n' "$BG_SEC" "$line" "$N"
printf -- '%s║%s %s%s%s %s║%s\n' "$BG_SEC" "$N" "$B" "$pad" "$N" "$BG_SEC" "$N"
printf -- '%s╚%s╝%s\n' "$BG_SEC" "$line" "$N"
else
printf -- '\n== %s ==\n' "$title"
fi
}
summary() {
echo ""
if [ "${#BLOCKERS[@]}" -gt 0 ]; then
printf '%s BLOCKERS %s %s(pay/withdraw cannot finish)%s\n' "$BG_BLK" "$N" "$D" "$N"
if [ "${BOX:-0}" = "1" ]; then
printf -- '%s┌ BLOCKERS · pay/withdraw cannot finish ┐%s\n' "$BG_BLK" "$N"
else
printf -- '--- BLOCKERS (pay/withdraw cannot finish) ---\n'
fi
local b
for b in "${BLOCKERS[@]}"; do
printf '%s •%s %s%s%s\n' "$M" "$N" "$W" "$b" "$N"
printf -- '%s •%s %s%s%s\n' "$M" "$N" "$W" "$b" "$N"
done
fi
if [ "${#ERRORS[@]}" -gt 0 ] && [ "${#BLOCKERS[@]}" -lt "${#ERRORS[@]}" ]; then
printf '%s ERRORS %s %s(failed checks)%s\n' "$BG_ERR" "$N" "$D" "$N"
if [ "${BOX:-0}" = "1" ]; then
printf -- '%s┌ ERRORS · failed checks ┐%s\n' "$BG_ERR" "$N"
else
printf -- '--- ERRORS (failed checks) ---\n'
fi
local e
for e in "${ERRORS[@]}"; do
case "$e" in BLOCKER*) continue ;; esac
printf '%s •%s %s%s%s\n' "$R" "$N" "$W" "$e" "$N"
printf -- '%s •%s %s%s%s\n' "$R" "$N" "$W" "$e" "$N"
done
fi
printf '%stotals:%s %s%d OK%s' "$D" "$N" "$G" "$PASS_N" "$N"
[ "$FAIL_N" -gt 0 ] && printf ', %s%d ERROR%s' "$R" "$FAIL_N" "$N"
[ "$WARN_N" -gt 0 ] && printf ', %s%d WARN%s' "$Y" "$WARN_N" "$N"
[ "$INFO_N" -gt 0 ] && printf ', %s%d INFO%s' "$C" "$INFO_N" "$N"
[ "${#BLOCKERS[@]}" -gt 0 ] && printf ', %s%d BLOCKER%s' "$M" "${#BLOCKERS[@]}" "$N"
printf -- '%stotals:%s %s%d OK%s' "$D" "$N" "$G" "$PASS_N" "$N"
[ "$FAIL_N" -gt 0 ] && printf -- ', %s%d ERROR%s' "$R" "$FAIL_N" "$N"
[ "$WARN_N" -gt 0 ] && printf -- ', %s%d WARN%s' "$Y" "$WARN_N" "$N"
[ "$INFO_N" -gt 0 ] && printf -- ', %s%d INFO%s' "$C" "$INFO_N" "$N"
[ "${#BLOCKERS[@]}" -gt 0 ] && printf -- ', %s%d BLOCKER%s' "$M" "${#BLOCKERS[@]}" "$N"
printf '\n'
[ "$FAIL_N" -eq 0 ]
}