taler-monitoring/TESTS.md

# taler-monitoring — grouped test IDs

Every check line has a **global** run number and a **grouped** id:

```text
┌ OK    ┐ #001  www.exchange-01  exchange /config · HTTP 200
┌ WARN  ┐ #002  www.stats-02     soft age  · yellow (never red)
┌ ERROR ┐ #003  e2e.pay-03       order create failed · HTTP 502
┌ BLOCKER┐ #004  e2e.prereq-02   merchant secret missing
┌ PROG  ┐ ████░░░░  25%  4/16
```

- `#NNN` — monotonic for the whole monitoring run  
- `area.group-NN` — logical block (resets per group, continues if re-entered)

| Area | Phase script | Groups (examples) |
|------|--------------|-------------------|
| **www** | `check_urls.sh` | `exchange` `perf` `stats` `bank` `merchant` `paivana` `landing` `qr` |
| **inside** | `check_inside.sh` | `ssh` `bank` `exchange` `merchant` `caddy` `load` |
| **versions** | `check_versions.sh` | `outside` `inside` `compare` |
| **aptdeploy** | `check_apt_deploy.sh` | `trixie` `trixie-testing` (koopa podman apt-src merchant smoke) |
| **surface** | `check_surface.sh` | **remote-only** ecosystem / `-d` domain inventory (not in default/all/full) |
| **monpages** | `check_monitoring_pages.sh` | public HTML via FQDN — **ERROR** if missing or markers fail (top+bottom+body); surface required on GOA (v1.9.0) |
| **surface** | `check_surface.sh` | ecosystem inventory + **nmap** OS fingerprint; mail/mattermost content reported via surface page |
| **sanity** | `check_sanity.sh` | `bank` `exchange` `merchant` |
| **server** | `check_server.sh` | (flat `server-NN` or host groups) |
| **e2e** | `check_e2e.sh` | `prereq` `load` `bank` `wallet` `atm` `settle` `pay` `shop` `paivana` `dig` `report` |
| **ladder** | `check_goa_ladder.sh` | `plan` `load` `withdraw` `pay` `report` |

**Why groups:** flat `www-001`…`www-080` was hard to talk about.  
`www.bank-04` / `e2e.atm-02` pin the failure to a logical block.

Usage in scripts:

```bash
set_area www
set_group exchange   # → www.exchange-01, www.exchange-02, …
set_group bank       # counter resets → www.bank-01
```

`set_group` also prints a small group chip (`┌ www.exchange ┐`) so logs show section boundaries.  
Re-entering the same group later (e.g. exchange legal after perf) **continues** NN — `www.exchange-03`, not a second `-01`.  
Without `set_group`, IDs stay flat: `area-01`, `area-02`, …

Optional soft checks still consume a number when they WARN.  
Numbering follows **executed** checks (early skip may shift later NN inside the same group).

---

## www — public URLs (`./taler-monitoring.sh urls`)

| Group | Checks (typical) |
|-------|------------------|
| **www.exchange-** | `/config`, currency, **alt_unit_names**; `/keys` (+ alt soft); `/intro/`, `/`; **`/terms`**, **`/privacy`**, `/terms/` |
| **www.perf-** | outside-in RTT: bank/exchange/merchant `/config`, keys, webui, intro — ms; WARN ≥ `PERF_WARN_MS` (8000); ERROR ≥ `PERF_FAIL_MS` (20000) |
| **www.stats-** | `/intro/stats.json` **reachable**; **timestamps** (`generated_at_unix` + human/ISO); **freshness** vs wall clock (`STATS_STALE_SECS` WARN, `STATS_FAIL_SECS` ERROR); **display fields** (bank-shape withdraws/accounts, or exchange-db / merchant-db keys); optional performance block; shared-feed equal `gen_unix` when bank collector is published to all three |
| **www.bank-** | `/config`, currency, alt_unit_names; integration/webui/intro; **auto-account.json** (required GOA/local; skip/soft off-GOA e.g. TESTPAYSAN); `/terms`, `/privacy` |
| **www.merchant-** | `/config` currency + currencies alt_unit_names; listed exchanges alt; webui/intro; **`/terms`**, **`/privacy`** |
| **www.webui-** | SPA fingerprints: **`/webui/version.txt`**, **`/webui/version-overlay.txt`**, `index.html` / `index.js`; optional pin via `EXPECT_WEBUI_VERSION` / `EXPECT_WEBUI_OVERLAY` / `WEBUI_OVERLAY_DENY` |
| **www.paivana-** | local GOA paywall front (redirect to template) |
| **www.landing-** | own-stack intro links; static assets (`qrcode.min.js` hard; `og-goa-shop.png` hard only GOA/local); **demo-withdraw.json** GOA-only; shop-pay soft; stage merchant shop: `shop-ui.js` + `shops.css` + `qrcode.min.js`; cross-links local |
| **www.qr-** | QR payloads: harvest `taler://` / `payto://` / app `data-qr-url` from landings + mint JSON; **form** check; **qrencode → zbarimg** exact roundtrip; optional static QR images. Needs `qrencode` + `zbar-tools`. Skip: `QR_CHECK=0` |

**QR rule:** `taler://withdraw/HOST/taler-integration/UUID` (no default `:443`/`:80`); `taler://pay/` / `pay-template/`; `payto://…` shape OK; decoded PNG must equal payload. Form errors on withdraw/pay/payto are **ERROR** on local/GOA.

**Legal docs rule:** HTTP 200, non-empty body, not plain `not configured`, not merchant API JSON `code:21`. Local stack may require content needle (terms/privacy/FADP/GOA…).

**Performance rule:** Measured from the **monitoring runner** (public via Caddy), not container loopback. Latency (ms) on every perf line + **perf summary** (n / min / p50 / avg / max).

**Landing links rule:** Own-stack must be HTTP 200 (or redirect→200). External stores/docs soft WARN. Auto-account wallet link must be `taler://withdraw/HOST/taler-integration/…` (default ports stripped), never payto.

**alt_unit_names rule:** non-empty map including scale key `"0"`. Multi-currency merchant: follow each `exchanges[]` public `/config`.

---

## inside — koopa SSH (`./taler-monitoring.sh inside`)

| Group | Checks |
|-------|--------|
| **inside.host-** | **host-podman**: reachability via `podman exec` on this machine (GOA host / `INSIDE_PODMAN=1`) — not SSH |
| **inside.ssh-** | **ssh**: from laptop/remote — SSH to `KOOPA_SSH` / `INSIDE_SSH`, then podman there |
| **inside.bank-** | container, libeufin, postgres, local `/config`, nginx, DNS pins |
| **inside.exchange-** | container, httpd, wirewatch, aggregator, transfer, local keys, DNS |
| **inside.merchant-** | container, httpd, wirewatch, depositcheck, DNS |
| **inside.caddy-** | host reverse-proxy process |
| **inside.load-** | host loadavg + RAM; per-container RSS/CPU |
| **inside.disk-** / **aptdeploy.disk-** / **versions.disk-** / **server.disk-** | free space via `df`: **WARN** if use ≥ `DISK_WARN_USED_PCT` (85), **ERROR** if full or use ≥ `DISK_ERR_USED_PCT` (95); host + containers |

Remote lines `E|comp|LEVEL|key|detail` each become one numbered result under that component group.

SSH: `KOOPA_SSH` (default `koopa`), then `KOOPA_SSH_FALLBACKS` when LAN is unreachable.

---

## e2e — withdraw + pay (`./taler-monitoring.sh e2e`)

| Group | Checks |
|-------|--------|
| **e2e.prereq-** | wallet-cli, currency, budgets, secrets, public reachability |
| **e2e.load-** | host/container load snapshots (before / after ATM / after pay) |
| **e2e.bank-** | account, credit, withdraw ops |
| **e2e.wallet-** | exchange + ToS, wallet setup |
| **e2e.atm-** | ATM withdraw ladder rungs |
| **e2e.settle-** | wallet settlement wait / spendable balance |
| **e2e.pay-** | variable payments |
| **e2e.shop-** | GOA shop product catalog pays |
| **e2e.paivana-** | template paywall pay |
| **e2e.dig-** | coins-missing dig (inside + withdrawal-operation) |
| **e2e.report-** | final tallies |

When filing an issue, quote the full id + label, e.g.  
`e2e.atm-03  ATM withdraw 5 GOA · bank confirmed, wallet empty`.

---

## versions / sanity / ladder / server / auth401

| Area.group | Meaning |
|------------|---------|
| **versions.outside-** | deb.taler.net suite index |
| **versions.host-** | host-podman: packages via local `podman exec` (`INSIDE_PODMAN=1`) |
| **versions.ssh-** | ssh: packages via `KOOPA_SSH` / `INSIDE_SSH` from laptop/remote |
| **versions.compare-** | installed vs suite |
| **aptdeploy.trixie-fresh-** | fresh rebuild container `…-apt-src-trixie` |
| **aptdeploy.trixie-testing-fresh-** | fresh rebuild `…-trixie-testing` |
| **aptdeploy.trixie-upgrade-** | upgrade-track `…-trixie-upgrade` (install once, then mytops-style upgrade) |
| **aptdeploy.trixie-testing-upgrade-** | upgrade-track `…-trixie-testing-upgrade` |

Standalone: `host-agent/run-aptdeploy.sh` (on koopa local; else ssh `$DEPLOY_SSH` / `$APT_DEPLOY_SSH`).

### surface (remote-only public inventory)

```bash
./taler-monitoring.sh surface                      # taler.net / gnunet / taler-systems / mattermost / …
./taler-monitoring.sh -d hacktivism.ch surface     # all things under that domain
```

- Catalog: `surface-catalog.conf` (extend as hosts are found)
- DNS, optional ICMP, TCP ports, then **protocol** checks (HTTPS/HTTP/SSH) before ERROR
- TLS cert expiry; Server-header / `/config` version hints; else **WARN: version unknown**
- OSV CVE query when package+version known → **ERROR** on hits
- Never SSH / never podman on the *targets*
| **sanity.bank-** / **.exchange-** / **.merchant-** | public + optional server-side per component |
| **ladder.plan-** / **.load-** / **.withdraw-** / **.pay-** / **.report-** | amount ladder (GOA ceiling or stage TESTPAYSAN `max_wire`) |
| **server-** | SSH host ports / processes (flat unless grouped later) |
| **auth401.*** | merchant 401 / case matrix (WebUI APIs): **signup** `POST /instances` MIX id, **login** Basic user low vs MIX/UPPER, full **case** matrix, **password** bad/empty, **pwchange** `POST private/auth` + re-login, **admin-create** `POST /management/instances` (if admin token), **bearer**, **durable mon401**, **webui** SPA `toLowerCase` probe (`check_auth401.sh`) |

### SPA pin (after selfbuild-webui)

```bash
EXPECT_WEBUI_VERSION=1.6.11 \
EXPECT_WEBUI_OVERLAY=selfbuild-v1.6.11 \
WEBUI_OVERLAY_DENY='c778af04|master-11340' \
  ./taler-monitoring.sh -d hacktivism.ch urls
```

Skip SPA block: `CHECK_WEBUI_SPA=0`.

### Full load (maximum phases)

| Alias | Expands to (local GOA) |
|-------|-------------------------|
| **`all`** | `urls inside versions sanity e2e` |
| **`full`** | `urls inside versions sanity server e2e ladder auth401` |

```bash
cd ~/src/taler-monitoring
# secrets.env with bank admin + merchant tokens (see secrets.env.example)
./taler-monitoring.sh -d hacktivism.ch full
# same, explicit:
./taler-monitoring.sh -d hacktivism.ch \
  urls inside versions sanity server e2e ladder auth401
```

---

## Visual format (colour on)

```text
╔══════════════════════════════╗
║ www · public URLs · …        ║
╚══════════════════════════════╝
┌ www.exchange ┐
┌ OK    ┐ www.exchange-01  exchange /config · HTTP 200
┌ www.bank ┐
┌ OK    ┐ www.bank-01  bank /config · HTTP 200
┌ ERROR ┐ www.bank-04  bank /intro/auto-account.json · invalid withdraw

┌ ERRORS · failed checks ┐
  • www.bank-04 bank /intro/auto-account.json · invalid withdraw
totals: 40 OK, 1 ERROR, 2 WARN, 5 INFO
```

`NO_COLOR=1` or `CLICOLOR=0` disables boxes/colours (ASCII `[ OK ]` + `-- group --` headers).  
Always use `printf --` friendly plain headers when colour is off (leading `---` is not a printf option).


## devtesting · fake-franken (v1.13+)

| ID prefix | What |
|-----------|------|
| **devtesting.cli-** | SSH to rusty · `taler-devtesting` CLI |
| **devtesting.geniban-** | synthetic CH IBAN |
| **devtesting.credit-payto-** | exchange credit IBAN for nexus |
| **devtesting.fake-incoming-** | CHF IN via nexus; bounce missing reserve pub = plumbing OK |