# taler-monitoring

Checks for **GNU Taler** stacks (bank, exchange, merchant; optional surface inventory).

| Mode | What |
|------|------|
| **Outside-only** | Public HTTPS from anywhere — no SSH, no host-agent. Phases such as `urls`, `monpages`, `surface`, … |
| **Inside / local** | Where you operate the stack: podman/SSH (`inside`, `versions`, `e2e`, `ladder`, …) |
| **Host-agent + HTML** | Optional: timers run the suite and publish sticky-bar HTML (see simplified page layout below) |

Outside-only example: `./taler-monitoring.sh -d demo.taler.net urls`  
Surface (ecosystem + mail + mattermost + versions): `./taler-monitoring.sh surface mattermost mail versions`  
Full local GOA example: `./taler-monitoring.sh -d hacktivism.ch urls inside versions`

Repo: https://git.hacktivism.ch/hernani/taler-monitoring · tree: `~/src/taler-monitoring`  
Version: `./taler-monitoring.sh --ver` (or `--version` / `-V`)


## Monitoring pages — public URL families (only these three)

**`taler-monitoring` is the suite name (git / CLI), not a public URL path.**  
There is **no** mon page at `/taler-monitoring` or `/taler-monitoring/`.

Exactly **three** first-class public path families (each must work **bare and with trailing slash**; reverse-proxy should `redir` bare → slash). Error companions use `…_err` when the last run failed:

| Family | Paths (bare + slash) | Where | Purpose |
|--------|----------------------|--------|---------|
| **monitoring** | `/monitoring` · `/monitoring/` · optional `/monitoring_err` · `/monitoring_err/` | Per **landing** host (GOA bank/exchange/taler · FP · mytops stage, …) | Stack report for that host |
| **taler-monitoring-surface** | `/taler-monitoring-surface` · `/…/` · optional `…_err` | **only** `taler.hacktivism.ch` | Ecosystem surface (ports, nmap, mail, mattermost, versions) |
| **taler-monitoring-aptdeploy** | `/taler-monitoring-aptdeploy` · `/…/` · optional `…_err` | **only** `taler.hacktivism.ch` | apt-src merchant deploy tests (podman on koopa) |

**Not** public mon pages: `/taler-monitoring-mail*`, `/taler-monitoring-mattermost*`, bare `/taler-monitoring` — mail/mm are folded into **surface**.

Related:

- **Phase `monpages`:** obligatory **ERROR** if a listed page is missing, bootstrap stub, not suite HTML, or **content markers fail**. Bare paths that fall through to merchant **code 21** are ERROR (not “OK because slash works”).
- **Publish:** staging `~/monitoring-sites-staging` → `DEPLOY_WWW_ROOT` + reverse-proxy handles for **only** the three families above.

```bash
./taler-monitoring.sh monpages
./taler-monitoring.sh -d hacktivism.ch monpages
```


Host packages (koopa / runners): **[DEPENDENCIES.md](./DEPENDENCIES.md)**.  
Host-agents (GOA + FP; shared report pipeline): **[host-agent/README.md](./host-agent/README.md)**.  
Public HTML site-gen (same timeout/log/HTML defaults): **[site-gen/README.md](./site-gen/README.md)**.  
FP stage host-agent: `host-agent/run-fp-stage-monitoring.sh` + `host-agent/env/stagepaysan.env.example` (see `host-agent/README.md`).

Report for the **GOA** stack with boxed severity badges and **grouped test IDs**
`area.group-NN` (e.g. `www.exchange-01`, `e2e.pay-03`):

| Tag | Meaning |
|-----|---------|
| `┌ OK ┐` | check passed |
| `┌ INFO ┐` | status / note |
| `┌ WARN ┐` | degraded but maybe not fatal |
| `┌ ERROR ┐` | component problem |
| `┌ BLOCKER ┐` | **withdraw/pay path cannot complete** because of this |

IDs reset per **group** inside an area — so “too many www tests” become
`www.exchange-*`, `www.bank-*`, `www.landing-*`, etc.  
Catalog: **[TESTS.md](./TESTS.md)**.  
Git / tags / suite tree: **[VERSIONS.md](./VERSIONS.md)**.  
Optional / deprecated tooling: **[meta/](./meta/)** (firecuda, legacy site-gen, old mail/mm units).
  
GUI Android notes: **[android-test/GUI-AUTOMATION-NOTES.md](./android-test/GUI-AUTOMATION-NOTES.md)**.  
CLI / wallet-cli recurring issues: **[CLI-AUTOMATION-NOTES.md](./CLI-AUTOMATION-NOTES.md)**.

```text
┌ OK    ┐ www.exchange-01  exchange /config · HTTP 200
┌ BLOCKER┐ e2e.prereq-02   merchant HTTP 502
```

End of each phase: totals + list of **BLOCKERS** and **ERRORS** (ids included).

## Secrets (local GOA e2e)

**Do not commit passwords.** Suite tree: `~/src/taler-monitoring` · host ops (optional):
sibling `~/src/koopa/koopa-admin-secrets`.

### Local file (recommended on laptop)

```bash
cd ~/src/taler-monitoring
cp secrets.env.example secrets.env   # gitignored
$EDITOR secrets.env
./taler-monitoring.sh e2e
```

See **[secrets.env.example](./secrets.env.example)** for every variable that
may appear (admin pass, merchant token, SECRETS_ROOT, progress knobs).

### Search order

1. Already-exported env (`E2E_BANK_ADMIN_PASS`, …)
2. `secrets.env` next to this suite (`MONITORING_SECRETS_ENV=` override)
3. `SECRETS_ROOT` / `KOOPA_ADMIN_SECRETS` file tree
4. Sibling `../koopa-admin-secrets/koopa/host-root`
5. `$HOME/src/koopa/koopa-admin-secrets/koopa/host-root`
6. `$HOME/.config/taler-landing/bank-admin-password.txt`
7. SSH koopa `/root/…` or bank container

### Line IDs + progress

Each check prints **global** and **grouped** ids:

```text
┌ OK    ┐ #014  www.exchange-03  exchange /config · HTTP 200
┌ WARN  ┐ #015  www.stats-02     stats age soft  · yellow badge (not red)
┌  42% ┐ ████████░░░░░░░░░░░░░░░░░░░░  14/55
```

- `#014` — run-wide counter (whole `./taler-monitoring.sh` invocation)  
- `www.exchange-03` — area.group-NN  
- **Colours (left badge + label):**
  - **OK** green · **INFO** blue/cyan · **WARN** yellow (never red)  
  - **ERROR** red · **BLOCKER** magenta  
- Progress: badge shows **percent** (not “PROG”); bar fills **left → right**;
  `done/total` on the right. Estimate auto-grows if short. Override:
  `PROGRESS_TOTAL=N` / `PROGRESS_OFF=1`
- **QR (urls):** harvest `taler://` / `payto://` (+ app store `data-qr-url`) from landings and
  `demo-withdraw.json` / `auto-account.json` → validate form → `qrencode` PNG → `zbarimg`
  decode must match. Requires packages **`qrencode`** + **`zbar-tools`**.

## Commands

```bash
# Local GOA stack (may use SSH for inside/e2e)
./taler-monitoring.sh              # urls + inside + versions + e2e
./taler-monitoring.sh inside       # containers on koopa
./taler-monitoring.sh versions     # deb.taler.net + package versions vs trixie
./taler-monitoring.sh sanity
./taler-monitoring.sh e2e

# FrancPaysan STAGE (TESTPAYSAN) — like hacktivism breadth:
#   default phases: urls + inside + versions + e2e
./taler-monitoring.sh -d stage.lefrancpaysan.ch
./taler-monitoring.sh -d stage.lefrancpaysan.ch inside   # stagepaysan only
# inside: low-priv SSH francpaysan-stage-user (User stagepaysan) → podman
#   containers stage-lfp-{bank,exchange-ansible,merchant} · ports 9032/9031/9030
# stats: outside-in public …/intro/stats.json (age via generated_at_unix)
# Admin credit for e2e: E2E_BANK_ADMIN_PASS=… or stage host secrets
#   (bank-admin-password.txt under /mnt/data/stagepaysan/bank/secrets/)
# Override: INSIDE_SSH=… INSIDE_PROFILE=stage-lfp

# Other domains — public HTTPS only (pass e2e/inside explicitly)
./taler-monitoring.sh -d taler.net
./taler-monitoring.sh --domain taler-ops.ch
./taler-monitoring.sh -d my.taler-ops.ch urls
./taler-monitoring.sh -d demo.taler.net urls
./taler-monitoring.sh taler.net    # bare domain = same as -d

# Explicit endpoints (any mix; override profile or skip -d)
./taler-monitoring.sh --bank https://bank.demo.taler.net \
  --exchange https://exchange.demo.taler.net \
  --merchant https://backend.demo.taler.net --currency KUDOS urls

# Landing stats.json freshness (outside-in; age via generated_at_unix)
STATS_STALE_SECS=900 STATS_FAIL_SECS=3600 ./taler-monitoring.sh -d stage.lefrancpaysan.ch urls

# --- Full load (GOA / hacktivism) ---
# all  = urls + inside + versions + sanity + e2e
# full = all + server + ladder + auth401  (long; needs secrets.env)
./taler-monitoring.sh -d hacktivism.ch full
./taler-monitoring.sh -d hacktivism.ch urls inside versions sanity server e2e ladder auth401

# SPA pin after selfbuild-webui.sh (container):
EXPECT_WEBUI_VERSION=1.6.11 EXPECT_WEBUI_OVERLAY=selfbuild-v1.6.11 \
  WEBUI_OVERLAY_DENY='c778af04|master-11340' \
  ./taler-monitoring.sh -d hacktivism.ch urls

# Only 401 matrix (may create throwaway instance):
./taler-monitoring.sh -d hacktivism.ch auth401

# Stage FrancPaysan — durable mon401 account (password in francpaysan-secrets):
./taler-monitoring.sh -d stage.monnaie.lefrancpaysan.ch auth401
# secrets: ~/francpaysan-secrets/stage/instances/mon401.password
#           ~/francpaysan-secrets/stage/default-instance-token.txt
# force throwaway create instead: AUTH401_PREFER_DURABLE=0 AUTH401_CREATE=1 …
```

### Landing `stats.json` (www.stats)

Public `GET …/intro/stats.json` must expose timestamps so monitoring can tell
whether the collector is still updating:

| Field | Role |
|-------|------|
| `generated_at_unix` | epoch seconds — **preferred** for age = now − unix |
| `generated_at` | ISO-8601 (fallback if unix missing) |
| `generated_at_human` | UI “updated” / footer string |

Also checks display payloads (bank-shape `withdraws` / accounts, or exchange-db /
merchant-db counters). Soft WARN if age > `STATS_STALE_SECS` (default 900);
ERROR if age > `STATS_FAIL_SECS` (default 3600; set `0` to disable).
Stage shared feed: same `gen_unix` on bank + exchange + monnaies.

## Domain profiles (`domains.conf`)

Each monitored stack must say **what is bank, exchange, and merchant-backend**.

Edit **[domains.conf](./domains.conf)** (or set `TALER_DOMAINS_CONF=`):

```text
# name   bank   exchange   merchant-backend   currency   local   [canonical]
taler-ops.ch  bank.taler-ops.ch  exchange.taler-ops.ch  my.taler-ops.ch  CHF  0
```

| Field | Meaning |
|-------|---------|
| **name** | `-d NAME` / `TALER_DOMAIN` match (add alias lines as needed) |
| **bank** | libeufin/bank public host or `https://…` |
| **exchange** | exchange public host or URL |
| **merchant** | merchant **backend** (SPA/API), e.g. `my.taler-ops.ch` |
| **currency** | `GOA` / `KUDOS` / `CHF` / … |
| **local** | `1` = koopa SSH stack; `0` = public only |
| **landing** | `1` = check `/intro` landings + assets; `0` = skip (TOPS / mytops) |
| **canonical** | optional label after alias (e.g. `my.taler-ops.ch` → `taler-ops.ch`) |

Then:

```bash
./taler-monitoring.sh -d taler-ops.ch urls
```

Overrides (always win after profile load):

```bash
./taler-monitoring.sh -d taler-ops.ch \
  --merchant https://my.taler-ops.ch \
  --exchange https://exchange.taler-ops.ch \
  --bank https://bank.taler-ops.ch urls
```

Helpers in `lib.sh`: `set_taler_stack`, `load_domain_profile`, `apply_taler_domain`.

### Built-in profiles (see `domains.conf`)

| Domain / alias | Bank | Exchange | Merchant | Currency |
|----------------|------|----------|----------|----------|
| `hacktivism.ch` / `koopa` | bank.hacktivism.ch | exchange.hacktivism.ch | taler.hacktivism.ch | GOA |
| `taler.net` / `demo.taler.net` | bank.demo.taler.net | exchange.demo.taler.net | backend.demo.taler.net | KUDOS |
| `test.taler.net` | bank.test… | exchange.test… | backend.test… | **TESTKUDOS** · no landings |
| `lefrancpaysan.ch` | bank.… | exchange.… | **monnaie.…** (merchant) | report-only · no landings |
| **`stage.lefrancpaysan.ch`** | stage.bank.… | stage.exchange.… | **stage.monnaie.…** | **TESTPAYSAN** · landings · **inside via stagepaysan** |
| `taler-ops.ch` / `my.taler-ops.ch` | bank.taler-ops.ch | exchange.taler-ops.ch | **my.taler-ops.ch** | CHF · **no landings** |
| `stage.taler-ops.ch` | bank.stage… | exchange.stage… | **my.stage…** | CHF · **no landings** |
| unknown | bank.DOMAIN | exchange.DOMAIN | my.DOMAIN then probe | any · no landings |

**Landings:** GOA, stage LFP (`TESTPAYSAN`), and optionally demo use public `/intro` pages. **taler-ops.ch does not** — `CHECK_LANDING=0` skips intro crawl.

**SSH:**
| Stack | Host alias | User | Scope |
|-------|------------|------|--------|
| koopa / hacktivism | `koopa` | hernani | full host + containers |
| stage LFP | `francpaysan-stage-user` | **stagepaysan** | own podman only (no sudo / no prod data) |

Other domains: never SSH. Optional **e2e** aborts cleanly on login/KYC.

### E2E amounts (variable)

| | Local (koopa) | Remote |
|--|---------------|--------|
| **ATM withdraw** | 20 · 50 · 100 · 200 · **4200** (paivana) | 10 · 20 · 50 |
| **Pay ladder** | 0.01 … 10 | 0.01 … 1 |
| **Paivana** | HTTP paywall + template pay **GOA:4200** (`goa-shop` / `paivana`) | skipped |

```bash
./taler-monitoring.sh e2e
./taler-monitoring.sh -d taler.net urls e2e
# Amount ladder (withdraw 0 → … → max, then pay): same script for GOA + stage
./taler-monitoring.sh ladder                              # GOA / current domain
./taler-monitoring.sh -d stage.lefrancpaysan.ch ladder    # TESTPAYSAN maxima
# Stage auto-reads bank max_wire_transfer_amount (e.g. 2000) and min denom (0.01).
# Explorer password: francpaysan-stage-user …/bank-explorer-password.txt (or EXP_PW=).
# Override: LADDER_MAX_AMOUNT=100 LADDER_STEPS=7 LADDER_PAY=0 …
# customize e2e:
E2E_WITHDRAW_VALUES="20 50 100" E2E_PAY_VALUES="0.05 1 5" ./taler-monitoring.sh e2e
E2E_VARIABLE=0 WITHDRAW_AMT=GOA:50 PAY_AMT=GOA:1 ./taler-monitoring.sh e2e   # single fixed
# GOA shop catalog (local hacktivism): full list in E2E_SHOP_PRODUCTS; each run
#   shuffles and pays E2E_SHOP_PICK_N products (default 2).
#   E2E_SHOP_PRODUCTS lines: id|Product name|GOA:amount
#   E2E_SHOP_PICK_N=2
#   (landing QR = taler://pay-template/…/{id}; popup = live taler://pay after POST templates/{id})
# Paivana paywall (local GOA only):
#   E2E_PAIVANA=1 (default)  PAIVANA_PUBLIC=https://paivana.hacktivism.ch
#   E2E_PAIVANA_TEMPLATE=paivana  E2E_PAIVANA_AMOUNT=GOA:4200  E2E_PAIVANA_INSTANCE=goa-shop
#   E2E_PAIVANA=0            # skip
# remote secrets:
#   E2E_BANK_ADMIN_PASS=…  E2E_MERCHANT_TOKEN=…
```

## Phases

| Phase | What |
|-------|------|
| **inside** | SSH koopa: processes, postgres, local /config,/keys, wirewatch, recent log ERRORs |
| **versions** | `deb.taler.net` reachable (InRelease/Packages/pool `.deb`); containers can reach it + have apt source; installed Taler packages vs **trixie** |
| **sanity** | bank · exchange · merchant sections (public + server) |
| **e2e** | account → credit → withdraw → wallet → confirm → order → pay |
| **ladder** | GOA withdraw: fixed **0** + low bands + **random high ranges** + fixed **max** |

```bash
# package suite (default trixie on deb.taler.net)
TALER_APT_SUITE=trixie ./taler-monitoring.sh versions
TALER_PKG_BEHIND=error ./taler-monitoring.sh versions   # any behind = ERROR
```

E2E maps failures to blockers, e.g.:

- `bank-confirm HTTP 409` → wallet did not select exchange  
- `no GOA balance` → wirewatch / transfer  
- `create order failed` → merchant auth/instance  
- `Alarm clock` during pay → usually missing `handle-uri --yes` or wrong pay URI (must be `…/instances/{inst}/{oid}/?c={token}` from merchant `taler_pay_uri`)
- `insufficient balance` → withdraw incomplete  

## Performance (urls phase)

Outside-in HTTPS RTT for bank / exchange / merchant critical paths. Each probe prints
`HTTP … · N ms` on the `[OK]`/`[WARN]`/`[ERROR]` line; end of the block prints
`perf summary` (n / min / p50 / avg / max).

| Env | Default | Meaning |
|-----|---------|---------|
| `PERF_WARN_MS` | `8000` | WARN if latency ≥ this |
| `PERF_FAIL_MS` | `20000` | ERROR if latency ≥ this |
| `PERF_CURL_TIMEOUT` | `25` | curl max seconds per probe |

```bash
PERF_WARN_MS=3000 PERF_FAIL_MS=10000 ./taler-monitoring.sh urls
```

## Wallet coins (e2e · ladder)

At every relevant withdraw/pay step the wallet is snapped via `advanced dump-coins`.
Amounts use exchange **`alt_unit_names`** (Kilo-GOA, Mega-GOA, …) with the base
**GOA value in parentheses**:

```text
[INFO]  coins after-ATM — amount_circ=2 Kilo-GOA (GOA:2000)
[INFO]  coins after-ATM — denoms_in_circ 1 Kilo-GOA (GOA:1000)×2 (=2 Kilo-GOA (GOA:2000))
== ladder · rung 12  GOA:1000000 · 1 Mega-GOA (GOA:1000000) ==
```

Also: coin counts (in circulation / spent), status histogram, Δ vs previous snap.
History TSV: `$METRICS_DIR/coins-history.tsv`. Map loaded from
`${EXCHANGE_PUBLIC}/config` → `currency_specification.alt_unit_names`.

### Final statistics dashboard (e2e · ladder)

At the end of **e2e** and **ladder**, a boxed **FINAL STATISTICS** block prints:

| Section | Content |
|---------|---------|
| **Coins** | in circulation / spent counts + amounts + denoms (alt names) |
| **Money flow** | withdrawn events/amount, spent (paid), net (wd − spent) |
| **Tendency** | coin/amount trend over snaps (↑↓→), slope, last history rows |
| **Performance** | min / p50 / avg / max per phase; first-half vs second-half latency |
| **Load** | host loadavg + mem before/after; per-container RSS/CPU; tendencies |

Successful withdraw/pay steps append to `flow-withdrawn.txt` / `flow-spent.txt` (ladder also uses its TSVs).

## Load / memory (inside · e2e · ladder)

Host **loadavg**, **RAM used/avail**, and per-container **RSS / CPU / block I/O**
(plus postgres DB sizes and role RSS: libeufin, exchange, merchant, …) are
snapshotted via SSH whenever the stack is under load:

| Phase | When |
|-------|------|
| **inside** | end of container status |
| **e2e** | before work · after ATM withdraws · after payments · after shop · e2e-end (+ delta) |
| **ladder** | before first rung · after last rung (+ delta + timing overall) |

```bash
# LAN down? force WAN DNAT host
KOOPA_SSH=koopa-external ./taler-monitoring.sh inside
# or rely on automatic fallback (KOOPA_SSH_FALLBACKS from env)

METRICS_LOAD=0 ./taler-monitoring.sh e2e   # skip host probes
LADDER_LOAD=0 ./taler-monitoring.sh ladder
```

| Env | Default | Meaning |
|-----|---------|---------|
| `KOOPA_SSH` | `koopa` | primary SSH host |
| `KOOPA_SSH_FALLBACKS` | *(from env)* | tried if primary fails |
| `METRICS_LOAD` | `1` | `0` = skip all host/container load probes |
| `LADDER_LOAD` | `1` | `0` = skip load in ladder only |
| `LADDER_PAY` | `1` | `0` = skip payment ladder after withdraws |
| `LADDER_WITHDRAW_SCALE` | `1.5` | withdraw mids / pay mids ratio (funding headroom) |
| `LADDER_STEPS` | `23` | rungs for both withdraw and pay (0 + mids + max−1 + max) |
| `METRICS_LOAD_SSH_TIMEOUT` | `90` | seconds for remote load python |

## Needs

- SSH via `KOOPA_SSH` / `KOOPA_SSH_FALLBACKS` (env) (inside / load / sanity server bits)
- secrets under `koopa-admin-secrets/...` for e2e
- `taler-wallet-cli` for e2e

## Languages (i18n) — **v1.2+**

Default **English**. Select language with `--lang en|fr` or `TALER_MON_LANG`.
FrancPaysan stacks auto-select **French** (`*lefrancpaysan*`). See `VERSIONS.md` and `i18n.sh`.

## Public monitoring pages via FQDN (v1.3.1+ / obligatory v1.7.6+ / layout v1.8.0+)

Phase **`monpages`** is **obligatory**: missing or non-suite HTML is **ERROR**.
Soft mode only with `MONPAGES_REQUIRE_PUBLIC=0` (escape hatch).

| Stack | What monpages expects (v1.9.0) |
|-------|--------------------------------|
| **GOA** | 3× `/monitoring/` + **surface** (existence **and** top/bottom/content markers) |
| **FrancPaysan** | Stack mon hosts `/monitoring/` with same content rules |
| **Nine landings (model)** | Each landing front: `/monitoring` + `/monitoring_err` (err when a failed run exists) |

```bash
./taler-monitoring.sh -d hacktivism.ch monpages
./taler-monitoring.sh -d lefrancpaysan.ch monpages
```

If pages are missing publicly: `sudo $APPLY_MONITORING_LIVE` on koopa
(FP: Infomaniak vhost for `/monitoring*`).

## Devtesting · fake-franken CHF (v1.13+)

Phase **`devtesting`** (aliases: `franken`, `fake-franken`) exercises the
Taler Operations **rusty** host (`taler-devtesting` forced command):

```bash
./taler-monitoring.sh -d stage.taler-ops.ch --currency CHF devtesting
```

Needs SSH to `devtesting@rusty.taler-ops.ch` (Host `taler-rusty-devtesting`).
Probes: CLI help, `geniban` (CH IBAN / synthetic debit funding), baseline `fake-incoming`,
optional **amount ladder** (`DEVTESTING_LADDER=1`, default **23** rungs up to **CHF:50000**).
Each rung reuses geniban debit payto so volume covers the max rung without a real bank withdraw.
`fake-incoming` CHF to exchange credit
IBAN. A nexus bounce *missing reserve public key* is **OK** for the plumbing
probe (no real wallet reserve required).

## Version on monitoring pages (v1.3+ / sticky bar)

The sticky bar shows the suite **tag** (e.g. `v1.11.0`) and links to that tag on Forgejo.

**v1.12.0:** Run header (**monitoring env context**: env_file, suite pin, domain/hosts/flags) is a collapsed `<details>` block above the console log — expand to inspect agent context.

**v1.11.0:** Click the sticky **error** or **warning** count to filter the console log to only those lines (other log lines hide; sticky bar, stats, and top overviews stay). Same via `#filter-error` / `#filter-warn` (or `#first-error` / `#first-warn`). Click again or **Show all** to clear.