taler-monitoring/TESTS.md

taler-monitoring — grouped test IDs

Every check line has a global run number and a grouped id:

┌ OK    ┐ #001  www.exchange-01  exchange /config · HTTP 200
┌ WARN  ┐ #002  www.stats-02     soft age  · yellow (never red)
┌ ERROR ┐ #003  e2e.pay-03       order create failed · HTTP 502
┌ BLOCKER┐ #004  e2e.prereq-02   merchant secret missing
┌ PROG  ┐ ████░░░░  25%  4/16

• #NNN — monotonic for the whole monitoring run
• area.group-NN — logical block (resets per group, continues if re-entered)

Area	Phase script	Groups (examples)
www	check_urls.sh	exchange perf stats bank merchant paivana landing qr
inside	check_inside.sh	ssh bank exchange merchant caddy load
versions	check_versions.sh	outside inside compare
aptdeploy	check_apt_deploy.sh	trixie trixie-testing (koopa podman apt-src merchant smoke)
surface	check_surface.sh	remote-only ecosystem / -d domain inventory (not in default/all/full)
monpages	check_monitoring_pages.sh	public HTML via FQDN — ERROR if missing or markers fail (top+bottom+body); surface required on GOA (v1.9.0)
surface	check_surface.sh	ecosystem inventory + nmap OS fingerprint; mail/mattermost content reported via surface page
sanity	check_sanity.sh	bank exchange merchant
server	check_server.sh	(flat server-NN or host groups)
e2e	check_e2e.sh	prereq load bank wallet atm settle pay shop paivana dig report
ladder	check_goa_ladder.sh	plan load withdraw pay report

Why groups: flat www-001…www-080 was hard to talk about.
www.bank-04 / e2e.atm-02 pin the failure to a logical block.

Usage in scripts:

set_area www
set_group exchange   # → www.exchange-01, www.exchange-02, …
set_group bank       # counter resets → www.bank-01

set_group also prints a small group chip (┌ www.exchange ┐) so logs show section boundaries.
Re-entering the same group later (e.g. exchange legal after perf) continues NN — www.exchange-03, not a second -01.
Without set_group, IDs stay flat: area-01, area-02, …

Optional soft checks still consume a number when they WARN.
Numbering follows executed checks (early skip may shift later NN inside the same group).

---

www — public URLs (./taler-monitoring.sh urls)

Group	Checks (typical)
www.exchange-	/config, currency, alt_unit_names; /keys (+ alt soft); /intro/, /; /terms, /privacy, /terms/
www.perf-	outside-in RTT: bank/exchange/merchant /config, keys, webui, intro — ms; WARN ≥ PERF_WARN_MS (8000); ERROR ≥ PERF_FAIL_MS (20000)
www.stats-	/intro/stats.json reachable; timestamps (generated_at_unix + human/ISO); freshness vs wall clock (STATS_STALE_SECS WARN, STATS_FAIL_SECS ERROR); display fields (bank-shape withdraws/accounts, or exchange-db / merchant-db keys); optional performance block; shared-feed equal gen_unix when bank collector is published to all three
www.bank-	/config, currency, alt_unit_names; integration/webui/intro; auto-account.json (required GOA/local; skip/soft off-GOA e.g. TESTPAYSAN); /terms, /privacy
www.merchant-	/config currency + currencies alt_unit_names; listed exchanges alt; webui/intro; /terms, /privacy
www.webui-	SPA fingerprints: /webui/version.txt, /webui/version-overlay.txt, index.html / index.js; optional pin via EXPECT_WEBUI_VERSION / EXPECT_WEBUI_OVERLAY / WEBUI_OVERLAY_DENY
www.paivana-	local GOA paywall front (redirect to template)
www.landing-	own-stack intro links; static assets (qrcode.min.js hard; og-goa-shop.png hard only GOA/local); demo-withdraw.json GOA-only; shop-pay soft; stage merchant shop: shop-ui.js + shops.css + qrcode.min.js; cross-links local
www.qr-	QR payloads: harvest taler:// / payto:// / app data-qr-url from landings + mint JSON; form check; qrencode → zbarimg exact roundtrip; optional static QR images. Needs qrencode + zbar-tools. Skip: QR_CHECK=0

QR rule: taler://withdraw/HOST/taler-integration/UUID (no default :443/:80); taler://pay/ / pay-template/; payto://… shape OK; decoded PNG must equal payload. Form errors on withdraw/pay/payto are ERROR on local/GOA.

Legal docs rule: HTTP 200, non-empty body, not plain not configured, not merchant API JSON code:21. Local stack may require content needle (terms/privacy/FADP/GOA…).

Performance rule: Measured from the monitoring runner (public via Caddy), not container loopback. Latency (ms) on every perf line + perf summary (n / min / p50 / avg / max).

Landing links rule: Own-stack must be HTTP 200 (or redirect→200). External stores/docs soft WARN. Auto-account wallet link must be taler://withdraw/HOST/taler-integration/… (default ports stripped), never payto.

alt_unit_names rule: non-empty map including scale key "0". Multi-currency merchant: follow each exchanges[] public /config.

---

inside — koopa SSH (./taler-monitoring.sh inside)

Group	Checks
inside.host-	host-podman: reachability via podman exec on this machine (GOA host / INSIDE_PODMAN=1) — not SSH
inside.ssh-	ssh: from laptop/remote — SSH to KOOPA_SSH / INSIDE_SSH, then podman there
inside.bank-	container, libeufin, postgres, local /config, nginx, DNS pins
inside.exchange-	container, httpd, wirewatch, aggregator, transfer, local keys, DNS
inside.merchant-	container, httpd, wirewatch, depositcheck, DNS
inside.caddy-	host reverse-proxy process
inside.load-	host loadavg + RAM; per-container RSS/CPU
inside.disk- / aptdeploy.disk- / versions.disk- / server.disk-	free space via df: WARN if use ≥ DISK_WARN_USED_PCT (85), ERROR if full or use ≥ DISK_ERR_USED_PCT (95); host + containers

Remote lines E|comp|LEVEL|key|detail each become one numbered result under that component group.

SSH: KOOPA_SSH (default koopa), then KOOPA_SSH_FALLBACKS when LAN is unreachable.

---

e2e — withdraw + pay (./taler-monitoring.sh e2e)

Group	Checks
e2e.prereq-	wallet-cli, currency, budgets, secrets, public reachability
e2e.load-	host/container load snapshots (before / after ATM / after pay)
e2e.bank-	account, credit, withdraw ops
e2e.wallet-	exchange + ToS, wallet setup
e2e.atm-	ATM withdraw ladder rungs
e2e.settle-	wallet settlement wait / spendable balance
e2e.pay-	variable payments
e2e.shop-	GOA shop product catalog pays
e2e.paivana-	template paywall pay
e2e.dig-	coins-missing dig (inside + withdrawal-operation)
e2e.report-	final tallies

When filing an issue, quote the full id + label, e.g.
e2e.atm-03 ATM withdraw 5 GOA · bank confirmed, wallet empty.

---

versions / sanity / ladder / server / auth401

Area.group	Meaning
versions.outside-	deb.taler.net suite index
versions.host-	host-podman: packages via local podman exec (INSIDE_PODMAN=1)
versions.ssh-	ssh: packages via KOOPA_SSH / INSIDE_SSH from laptop/remote
versions.compare-	installed vs suite
aptdeploy.trixie-fresh-	fresh rebuild container …-apt-src-trixie
aptdeploy.trixie-testing-fresh-	fresh rebuild …-trixie-testing
aptdeploy.trixie-upgrade-	upgrade-track …-trixie-upgrade (install once, then mytops-style upgrade)
aptdeploy.trixie-testing-upgrade-	upgrade-track …-trixie-testing-upgrade

Standalone: host-agent/run-aptdeploy.sh (on koopa local; else ssh $DEPLOY_SSH / $APT_DEPLOY_SSH).

surface (remote-only public inventory)

./taler-monitoring.sh surface                      # taler.net / gnunet / taler-systems / mattermost / …
./taler-monitoring.sh -d hacktivism.ch surface     # all things under that domain

• Catalog: surface-catalog.conf (extend as hosts are found)
• DNS, optional ICMP, TCP ports, then protocol checks (HTTPS/HTTP/SSH) before ERROR
• TLS cert expiry; Server-header / /config version hints; else WARN: version unknown
• OSV CVE query when package+version known → ERROR on hits
• Never SSH / never podman on the targets | sanity.bank- / .exchange- / .merchant- | public + optional server-side per component | | ladder.plan- / .load- / .withdraw- / .pay- / .report- | amount ladder (GOA ceiling or stage TESTPAYSAN max_wire) | | server- | SSH host ports / processes (flat unless grouped later) | | auth401.* | merchant 401 / case matrix (WebUI APIs): signup POST /instances MIX id, login Basic user low vs MIX/UPPER, full case matrix, password bad/empty, pwchange POST private/auth + re-login, admin-create POST /management/instances (if admin token), bearer, durable mon401, webui SPA toLowerCase probe (check_auth401.sh) |

SPA pin (after selfbuild-webui)

EXPECT_WEBUI_VERSION=1.6.11 \
EXPECT_WEBUI_OVERLAY=selfbuild-v1.6.11 \
WEBUI_OVERLAY_DENY='c778af04|master-11340' \
  ./taler-monitoring.sh -d hacktivism.ch urls

Skip SPA block: CHECK_WEBUI_SPA=0.

Full load (maximum phases)

Alias	Expands to (local GOA)
all	urls inside versions sanity e2e
full	urls inside versions sanity server e2e ladder auth401

cd ~/src/taler-monitoring
# secrets.env with bank admin + merchant tokens (see secrets.env.example)
./taler-monitoring.sh -d hacktivism.ch full
# same, explicit:
./taler-monitoring.sh -d hacktivism.ch \
  urls inside versions sanity server e2e ladder auth401

---

Visual format (colour on)

╔══════════════════════════════╗
║ www · public URLs · …        ║
╚══════════════════════════════╝
┌ www.exchange ┐
┌ OK    ┐ www.exchange-01  exchange /config · HTTP 200
┌ www.bank ┐
┌ OK    ┐ www.bank-01  bank /config · HTTP 200
┌ ERROR ┐ www.bank-04  bank /intro/auto-account.json · invalid withdraw

┌ ERRORS · failed checks ┐
  • www.bank-04 bank /intro/auto-account.json · invalid withdraw
totals: 40 OK, 1 ERROR, 2 WARN, 5 INFO

NO_COLOR=1 or CLICOLOR=0 disables boxes/colours (ASCII [ OK ] + -- group -- headers).
Always use printf -- friendly plain headers when colour is off (leading --- is not a printf option).

devtesting · fake-franken (v1.13+)

ID prefix	What
devtesting.cli-	SSH to rusty · taler-devtesting CLI
devtesting.geniban-	synthetic CH IBAN
devtesting.credit-payto-	exchange credit IBAN for nexus
devtesting.fake-incoming- / baseline-	CHF IN via nexus; bounce missing reserve pub = plumbing OK
devtesting.ladder-	CHF fake-incoming amount ladder (default 23 rungs → CHF:50000; synthetic geniban debit each rung)