# taler-monitoring — test IDs by area

Every check line is numbered **per area** as `AREA-NNN` (zero-padded):

| Area | Phase script | Meaning |
|------|--------------|---------|
| **www** | `check_urls.sh` | public HTTPS (outside-in) |
| **inside** | `check_inside.sh` | containers / processes on koopa |
| **versions** | `check_versions.sh` | deb.taler.net available + packages vs trixie |
| **sanity** | `check_sanity.sh` | public + server per component |
| **server** | `check_server.sh` | SSH host ports / processes |
| **e2e** | `check_e2e.sh` | withdraw + pay cycle |

Format in output:

```text
[OK]    www-001  exchange /config https://exchange…/config
[ERROR] e2e-012  bank-auth: admin token failed
[BLOCKER] e2e-015  prereq: merchant HTTP 502
```

IDs are assigned **in run order** within the area (`set_area` resets the counter). Optional soft checks still consume a number when they WARN.

---

## www — public URLs (`./taler-monitoring.sh urls`)

| ID | Check |
|----|--------|
| www-… | exchange `/config`, currency, **alt_unit_names** |
| www-… | exchange `/keys` (+ alt_unit_names soft) |
| www-… | exchange `/intro/`, `/` (302→intro) |
| www-… | **exchange `/terms`** body (not empty / not API error) |
| www-… | **exchange `/privacy`** body |
| www-… | exchange `/terms/` (200 or redirect) |
| www-… | bank `/config`, currency, **alt_unit_names** |
| www-… | bank integration / webui / intro / `/` |
| www-… | **bank `/terms`** body |
| www-… | **bank `/privacy`** (or `/intro/privacy.html` fallback) |
| www-… | merchant `/config`, currency, currencies alt_unit_names |
| www-… | each merchant `exchanges[]` `/config` alt_unit_names |
| www-… | merchant `/intro/`, `/webui/`, `/` |
| www-… | **merchant `/terms`** body (dual-currency notice) |
| www-… | **merchant `/privacy`** body (must not be `not configured`) |
| www-… | merchant `/terms/` redirect |
| www-… | **landing (aggregated per site)**: one line for bank/merchant/exchange (`/intro` + assets + own/external link counts); failures listed as samples; soft external/shop assets compact |
| www-… | cross-links: one line (4 intros) on local stack |
| www-… | bank demo-withdraw + shop assets: one line (+ soft shop-pay if missing) |
| www-… | **bank `/intro/demo-withdraw.json`** → `taler://withdraw/HOST:PORT/taler-integration/…` + integration op HTTP 200 |
| www-… | **landing exposed links** (bank / merchant / exchange): parse each `/intro/` HTML, probe every own-stack `https://` + root-relative `href`/`src`/`content`, soft-check external stores/docs |
| www-… | landing static: `qrcode.min.js`, `og-goa-shop.png`, `qr-logo.png`, shop-pay.js/css |
| www-… | cross-links between bank ↔ merchant ↔ exchange intros (local stack) |
| www-… | **bank `/intro/demo-withdraw.json`** → `taler://withdraw/HOST/taler-integration/…` (no default `:443`/`:80`; non-default port OK) + integration op HTTP 200 |
| www-… | bank `/intro/auto-account.json` (earlier) → same withdraw shape, **no payto_uri**, login at `/webui/` |
| www-… | **performance** (outside-in RTT): bank/exchange/merchant `/config`, keys, webui, intro — ms; WARN ≥ `PERF_WARN_MS` (8000); ERROR ≥ `PERF_FAIL_MS` (20000) |
| www-… | **landing load stats**: `/intro/stats.json` → loadavg, container RSS, in-container probe ms (public; SSH container cat fallback); warn if stale/missing |

**Legal docs rule:** HTTP 200, non-empty body, not plain `not configured`, not merchant API JSON `code:21`. On local stack, optional content needle (terms/privacy/FADP/GOA…).

**Performance rule:** Measured from the **monitoring runner** (public URLs via Caddy), not container loopback. HTTP must match expect (usually 200); **latency (ms) is on every perf OK/WARN/ERROR line**, plus a **perf summary** (n / min / p50 / avg / max). Slow ≥ `PERF_WARN_MS` (default 8000) → WARN; ≥ `PERF_FAIL_MS` (default 20000) → ERROR.

**Landing links rule:** Own-stack (bank/exchange/taler.\* + page host) must be HTTP 200 (or redirect→200). External (App Store, Play, F-Droid, wallet.taler.net, docs/git.taler.net, …) soft WARN if down. Auto-account wallet link must be `taler://withdraw/HOST/taler-integration/…` (default ports stripped for mobile wallets), never payto.

**alt_unit_names rule:** wallet codec requires a non-empty map including scale key `"0"`. For multi-currency merchant, also follow every entry in `exchanges[]` and check that exchange’s public `/config`.

(IDs after a failed early check may shift if later soft checks are skipped when body missing — numbering follows **executed** checks.)

---

## inside — koopa SSH (`./taler-monitoring.sh inside`)

| ID | Check (typical order) |
|----|------------------------|
| inside-001 | ssh koopa (or **koopa-external** fallback) |
| inside-002+ | per-component emit: container, ports, libeufin/httpd, postgres, local `/config`/`/keys`, wirewatch, DNS pin, caddy |
| inside-… | **load / memory**: host loadavg + RAM; bank/exchange/merchant podman CPU/mem/block + process RSS by role + DB sizes |

Remote lines `E|comp|LEVEL|key|detail` each become one numbered result.

SSH: `KOOPA_SSH` (default `koopa`), then `KOOPA_SSH_FALLBACKS` (default `koopa-external`) when LAN is unreachable.

---

## sanity — bank · exchange · merchant (`./taler-monitoring.sh sanity`)

| ID | Section |
|----|---------|
| sanity-001… | bank public + server |
| sanity-… | exchange public + server |
| sanity-… | merchant public + server |

Sequential through the whole script (one `set_area sanity`).

---

## versions — packages vs deb.taler.net (`./taler-monitoring.sh versions`)

### Outside (runner / public network — no SSH)

| ID (order) | Check |
|------------|--------|
| versions-… | DNS `deb.taler.net` |
| versions-… | HTTPS portal + apt base URL |
| versions-… | suite `InRelease` / `Release` |
| versions-… | suite `Packages` + `Packages.gz` |
| versions-… | sample pool `.deb` fetchable (Range 200/206) |
| versions-… | suite offers `taler-exchange`, `taler-merchant`, `libeufin-bank` |
| versions-… | optional `trixie-testing` Packages |
| versions-… | TLS verify (soft) |

### Inside (SSH koopa containers)

| ID | Check |
|----|--------|
| versions-… | ssh koopa |
| versions-… | each container → `InRelease` (pasta can reach apt repo) |
| versions-… | each container lists `deb.taler.net` in apt sources |
| versions-… | each installed `taler*` / `libeufin*` / `libtaler*` / `libdonau*` vs suite version |
| versions-… | core packages installed (`taler-exchange`, `libeufin-bank`, `taler-merchant`) |

Outside always runs. Inside skipped with `SKIP_SSH=1` (still reports outside results).

Compare rules:

- **match** suite → OK  
- **ahead** of suite (often testing/dev) → INFO  
- **behind** suite → ERROR for core packages, WARN otherwise (`TALER_PKG_BEHIND=error` forces ERROR)

Default suite: **trixie** (`TALER_APT_SUITE`, `TALER_APT_BASE=https://deb.taler.net/apt/debian`).

Without SSH (`SKIP_SSH=1` or remote domain): still runs outside-in repo checks; skips container install compare.

---

## server — SSH ports (`./taler-monitoring.sh server`)

| ID | Check |
|----|--------|
| server-001 | ssh |
| server-002+ | containers, local pasta ports, processes, caddy |

---

## e2e — payment path (`./taler-monitoring.sh e2e`)

Local GOA also: **ATM includes GOA:4200**, then **paivana** (HTTP 302 on `PAIVANA_PUBLIC` + public template pay `paivana` / GOA:4200 on `goa-shop`). Disable with `E2E_PAIVANA=0`.

| ID | Step (approx.) |
|----|----------------|
| e2e-001 | budget info |
| e2e-002 | wallet-cli present |
| e2e-003 | mode / currency info |
| e2e-004… | secrets, reachability gates |
| e2e-… | account, credit, withdraw, confirm, coins, order, pay ladder |
| e2e-… | **load snapshots** at e2e-start, after-withdraw, after-pay, after-shop, e2e-end (host loadavg/RAM + container RSS/CPU/DB) |
| e2e-… | **coins** before/after each ATM withdraw, settle, pay, shop: count in circulation / spent, amount + denoms using **`alt_unit_names`** (e.g. `5 Kilo-GOA (GOA:5000)`), Δ vs previous snap |
| e2e-… | **GOA shop products** — full catalog list; **random pick of 2** (override `E2E_SHOP_PICK_N`) |
| e2e-… | balances, dig on failure, load delta + overall metrics |

Shop product pays use instance `goa-shop` (default) and catalog `E2E_SHOP_PRODUCTS`
(`id|Product name|amount` lines). Each e2e run **shuffles** the catalog and pays
only `E2E_SHOP_PICK_N` products (**default 2**). Flow matches the landing popup
(public POST `/templates/{id}`, not private orders). Report labels use product name.

Blockers keep the same ID prefix: `[BLOCKER] e2e-0NN step: message`.

---

## ladder — GOA withdraw ranges (`./taler-monitoring.sh ladder`)

| Step | What |
|------|------|
| ladder-001 | `GET /intro/auto-account.json` (personal account, GOA:0) |
| ladder-… | **load snapshot** before withdraws (host + bank/exchange/merchant mem/CPU) |
| ladder-… | **Phase A withdraw** (23): mint explorer pool — 0 → random → max−1 → max into **one cumulative wallet** (mids × `LADDER_WITHDRAW_SCALE`, default 1.5) |
| ladder-… | wallet-cli accept-uri + explorer confirm when `selected`; settle; coins snap |
| ladder-… | **Phase B pay** (23): same shape 0 → random → max−1 → max via merchant orders + `handle-uri --yes` |
| ladder-… | **load snapshot** after + delta + overall timing (withdraw + pay) |
| report | withdraw TSV + pay TSV + JSON |

Default amounts (`build_ladder_pair`, strictly increasing):

```text
withdraw: [0] + random mids + [max−1] + [max]
pay:      [0] + (withdraw_mid / scale) + [max−1] + [max]   # always ≤ matching withdraw mid
```

TSV `range`: `pin:0` | `random` | `pin:max-1` | `pin:max`.

Soft: absolute **max** mint/pay may `CEILING_REJECT` / skip (WARN). **max-1** is a hard pin.  
`LADDER_PAY=0` skips phase B.

| Env | Default | Meaning |
|-----|---------|---------|
| `LADDER_STEPS` | `23` | total rungs (0 + mids + max) |
| `LADDER_MAX_AMOUNT` | `4503599627370496` | absolute last pin |
| `LADDER_TIMEOUT_S` | `3600` | large rungs need time |
| `LADDER_MAX_RUNGS` | `99` | cap list length after build |

```bash
./taler-monitoring.sh ladder
LADDER_STEPS=10 ./taler-monitoring.sh ladder
LADDER_REPORT_DIR=/tmp/my-ladder ./taler-monitoring.sh ladder
```

---

## Run one area

```bash
./taler-monitoring.sh urls      # www only
./taler-monitoring.sh inside    # inside only
./taler-monitoring.sh versions  # deb.taler.net + package drift
./taler-monitoring.sh e2e       # e2e only
./taler-monitoring.sh ladder    # GOA amount ladder + timings
./taler-monitoring.sh -d taler.net urls
```