e7a3cd558f
Public layout: only taler-monitoring-surface(+_err) for ecosystem software/versions; nine landing stacks keep /monitoring(+_err). Fold mail (firefly.gnunet.org, anastasis.taler-systems.com) and mattermost into the surface job; deprecate separate mon pages/timers. nmap OS fingerprinting; package-behind defaults to ERROR.
195 lines
11 KiB
Markdown
195 lines
11 KiB
Markdown
# taler-monitoring — grouped test IDs
|
||
|
||
Every check line has a **global** run number and a **grouped** id:
|
||
|
||
```text
|
||
┌ OK ┐ #001 www.exchange-01 exchange /config · HTTP 200
|
||
┌ WARN ┐ #002 www.stats-02 soft age · yellow (never red)
|
||
┌ ERROR ┐ #003 e2e.pay-03 order create failed · HTTP 502
|
||
┌ BLOCKER┐ #004 e2e.prereq-02 merchant secret missing
|
||
┌ PROG ┐ ████░░░░ 25% 4/16
|
||
```
|
||
|
||
- `#NNN` — monotonic for the whole monitoring run
|
||
- `area.group-NN` — logical block (resets per group, continues if re-entered)
|
||
|
||
| Area | Phase script | Groups (examples) |
|
||
|------|--------------|-------------------|
|
||
| **www** | `check_urls.sh` | `exchange` `perf` `stats` `bank` `merchant` `paivana` `landing` `qr` |
|
||
| **inside** | `check_inside.sh` | `ssh` `bank` `exchange` `merchant` `caddy` `load` |
|
||
| **versions** | `check_versions.sh` | `outside` `inside` `compare` |
|
||
| **aptdeploy** | `check_apt_deploy.sh` | `trixie` `trixie-testing` (koopa podman apt-src merchant smoke) |
|
||
| **surface** | `check_surface.sh` | **remote-only** ecosystem / `-d` domain inventory (not in default/all/full) |
|
||
| **monpages** | `check_monitoring_pages.sh` | public HTML via FQDN — **obligatory ERROR**; GOA: 3×`/monitoring/` + surface; FP: stack mon only (v1.8.0) |
|
||
| **surface** | `check_surface.sh` | ecosystem inventory + **nmap** OS fingerprint; mail/mattermost content reported via surface page |
|
||
| **sanity** | `check_sanity.sh` | `bank` `exchange` `merchant` |
|
||
| **server** | `check_server.sh` | (flat `server-NN` or host groups) |
|
||
| **e2e** | `check_e2e.sh` | `prereq` `load` `bank` `wallet` `atm` `settle` `pay` `shop` `paivana` `dig` `report` |
|
||
| **ladder** | `check_goa_ladder.sh` | `plan` `load` `withdraw` `pay` `report` |
|
||
|
||
**Why groups:** flat `www-001`…`www-080` was hard to talk about.
|
||
`www.bank-04` / `e2e.atm-02` pin the failure to a logical block.
|
||
|
||
Usage in scripts:
|
||
|
||
```bash
|
||
set_area www
|
||
set_group exchange # → www.exchange-01, www.exchange-02, …
|
||
set_group bank # counter resets → www.bank-01
|
||
```
|
||
|
||
`set_group` also prints a small group chip (`┌ www.exchange ┐`) so logs show section boundaries.
|
||
Re-entering the same group later (e.g. exchange legal after perf) **continues** NN — `www.exchange-03`, not a second `-01`.
|
||
Without `set_group`, IDs stay flat: `area-01`, `area-02`, …
|
||
|
||
Optional soft checks still consume a number when they WARN.
|
||
Numbering follows **executed** checks (early skip may shift later NN inside the same group).
|
||
|
||
---
|
||
|
||
## www — public URLs (`./taler-monitoring.sh urls`)
|
||
|
||
| Group | Checks (typical) |
|
||
|-------|------------------|
|
||
| **www.exchange-** | `/config`, currency, **alt_unit_names**; `/keys` (+ alt soft); `/intro/`, `/`; **`/terms`**, **`/privacy`**, `/terms/` |
|
||
| **www.perf-** | outside-in RTT: bank/exchange/merchant `/config`, keys, webui, intro — ms; WARN ≥ `PERF_WARN_MS` (8000); ERROR ≥ `PERF_FAIL_MS` (20000) |
|
||
| **www.stats-** | `/intro/stats.json` **reachable**; **timestamps** (`generated_at_unix` + human/ISO); **freshness** vs wall clock (`STATS_STALE_SECS` WARN, `STATS_FAIL_SECS` ERROR); **display fields** (bank-shape withdraws/accounts, or exchange-db / merchant-db keys); optional performance block; shared-feed equal `gen_unix` when bank collector is published to all three |
|
||
| **www.bank-** | `/config`, currency, alt_unit_names; integration/webui/intro; **auto-account.json** (required GOA/local; skip/soft off-GOA e.g. TESTPAYSAN); `/terms`, `/privacy` |
|
||
| **www.merchant-** | `/config` currency + currencies alt_unit_names; listed exchanges alt; webui/intro; **`/terms`**, **`/privacy`** |
|
||
| **www.webui-** | SPA fingerprints: **`/webui/version.txt`**, **`/webui/version-overlay.txt`**, `index.html` / `index.js`; optional pin via `EXPECT_WEBUI_VERSION` / `EXPECT_WEBUI_OVERLAY` / `WEBUI_OVERLAY_DENY` |
|
||
| **www.paivana-** | local GOA paywall front (redirect to template) |
|
||
| **www.landing-** | own-stack intro links; static assets (`qrcode.min.js` hard; `og-goa-shop.png` hard only GOA/local); **demo-withdraw.json** GOA-only; shop-pay soft; stage merchant shop: `shop-ui.js` + `shops.css` + `qrcode.min.js`; cross-links local |
|
||
| **www.qr-** | QR payloads: harvest `taler://` / `payto://` / app `data-qr-url` from landings + mint JSON; **form** check; **qrencode → zbarimg** exact roundtrip; optional static QR images. Needs `qrencode` + `zbar-tools`. Skip: `QR_CHECK=0` |
|
||
|
||
**QR rule:** `taler://withdraw/HOST/taler-integration/UUID` (no default `:443`/`:80`); `taler://pay/` / `pay-template/`; `payto://…` shape OK; decoded PNG must equal payload. Form errors on withdraw/pay/payto are **ERROR** on local/GOA.
|
||
|
||
**Legal docs rule:** HTTP 200, non-empty body, not plain `not configured`, not merchant API JSON `code:21`. Local stack may require content needle (terms/privacy/FADP/GOA…).
|
||
|
||
**Performance rule:** Measured from the **monitoring runner** (public via Caddy), not container loopback. Latency (ms) on every perf line + **perf summary** (n / min / p50 / avg / max).
|
||
|
||
**Landing links rule:** Own-stack must be HTTP 200 (or redirect→200). External stores/docs soft WARN. Auto-account wallet link must be `taler://withdraw/HOST/taler-integration/…` (default ports stripped), never payto.
|
||
|
||
**alt_unit_names rule:** non-empty map including scale key `"0"`. Multi-currency merchant: follow each `exchanges[]` public `/config`.
|
||
|
||
---
|
||
|
||
## inside — koopa SSH (`./taler-monitoring.sh inside`)
|
||
|
||
| Group | Checks |
|
||
|-------|--------|
|
||
| **inside.host-** | **host-podman**: reachability via `podman exec` on this machine (GOA host / `INSIDE_PODMAN=1`) — not SSH |
|
||
| **inside.ssh-** | **ssh**: from laptop/remote — SSH to `KOOPA_SSH` / `INSIDE_SSH`, then podman there |
|
||
| **inside.bank-** | container, libeufin, postgres, local `/config`, nginx, DNS pins |
|
||
| **inside.exchange-** | container, httpd, wirewatch, aggregator, transfer, local keys, DNS |
|
||
| **inside.merchant-** | container, httpd, wirewatch, depositcheck, DNS |
|
||
| **inside.caddy-** | host reverse-proxy process |
|
||
| **inside.load-** | host loadavg + RAM; per-container RSS/CPU |
|
||
| **inside.disk-** / **aptdeploy.disk-** / **versions.disk-** / **server.disk-** | free space via `df`: **WARN** if use ≥ `DISK_WARN_USED_PCT` (85), **ERROR** if full or use ≥ `DISK_ERR_USED_PCT` (95); host + containers |
|
||
|
||
Remote lines `E|comp|LEVEL|key|detail` each become one numbered result under that component group.
|
||
|
||
SSH: `KOOPA_SSH` (default `koopa`), then `KOOPA_SSH_FALLBACKS` when LAN is unreachable.
|
||
|
||
---
|
||
|
||
## e2e — withdraw + pay (`./taler-monitoring.sh e2e`)
|
||
|
||
| Group | Checks |
|
||
|-------|--------|
|
||
| **e2e.prereq-** | wallet-cli, currency, budgets, secrets, public reachability |
|
||
| **e2e.load-** | host/container load snapshots (before / after ATM / after pay) |
|
||
| **e2e.bank-** | account, credit, withdraw ops |
|
||
| **e2e.wallet-** | exchange + ToS, wallet setup |
|
||
| **e2e.atm-** | ATM withdraw ladder rungs |
|
||
| **e2e.settle-** | wallet settlement wait / spendable balance |
|
||
| **e2e.pay-** | variable payments |
|
||
| **e2e.shop-** | GOA shop product catalog pays |
|
||
| **e2e.paivana-** | template paywall pay |
|
||
| **e2e.dig-** | coins-missing dig (inside + withdrawal-operation) |
|
||
| **e2e.report-** | final tallies |
|
||
|
||
When filing an issue, quote the full id + label, e.g.
|
||
`e2e.atm-03 ATM withdraw 5 GOA · bank confirmed, wallet empty`.
|
||
|
||
---
|
||
|
||
## versions / sanity / ladder / server / auth401
|
||
|
||
| Area.group | Meaning |
|
||
|------------|---------|
|
||
| **versions.outside-** | deb.taler.net suite index |
|
||
| **versions.host-** | host-podman: packages via local `podman exec` (`INSIDE_PODMAN=1`) |
|
||
| **versions.ssh-** | ssh: packages via `KOOPA_SSH` / `INSIDE_SSH` from laptop/remote |
|
||
| **versions.compare-** | installed vs suite |
|
||
| **aptdeploy.trixie-fresh-** | fresh rebuild container `…-apt-src-trixie` |
|
||
| **aptdeploy.trixie-testing-fresh-** | fresh rebuild `…-trixie-testing` |
|
||
| **aptdeploy.trixie-upgrade-** | upgrade-track `…-trixie-upgrade` (install once, then mytops-style upgrade) |
|
||
| **aptdeploy.trixie-testing-upgrade-** | upgrade-track `…-trixie-testing-upgrade` |
|
||
|
||
Standalone: `host-agent/run-aptdeploy.sh` (on koopa local; else `ssh koopa-external`).
|
||
|
||
### surface (remote-only public inventory)
|
||
|
||
```bash
|
||
./taler-monitoring.sh surface # taler.net / gnunet / taler-systems / mattermost / …
|
||
./taler-monitoring.sh -d hacktivism.ch surface # all things under that domain
|
||
```
|
||
|
||
- Catalog: `surface-catalog.conf` (extend as hosts are found)
|
||
- DNS, optional ICMP, TCP ports, then **protocol** checks (HTTPS/HTTP/SSH) before ERROR
|
||
- TLS cert expiry; Server-header / `/config` version hints; else **WARN: version unknown**
|
||
- OSV CVE query when package+version known → **ERROR** on hits
|
||
- Never SSH / never podman on the *targets*
|
||
| **sanity.bank-** / **.exchange-** / **.merchant-** | public + optional server-side per component |
|
||
| **ladder.plan-** / **.load-** / **.withdraw-** / **.pay-** / **.report-** | amount ladder (GOA ceiling or stage TESTPAYSAN `max_wire`) |
|
||
| **server-** | SSH host ports / processes (flat unless grouped later) |
|
||
| **auth401.*** | merchant 401 / case matrix (WebUI APIs): **signup** `POST /instances` MIX id, **login** Basic user low vs MIX/UPPER, full **case** matrix, **password** bad/empty, **pwchange** `POST private/auth` + re-login, **admin-create** `POST /management/instances` (if admin token), **bearer**, **durable mon401**, **webui** SPA `toLowerCase` probe (`check_auth401.sh`) |
|
||
|
||
### SPA pin (after selfbuild-webui)
|
||
|
||
```bash
|
||
EXPECT_WEBUI_VERSION=1.6.11 \
|
||
EXPECT_WEBUI_OVERLAY=selfbuild-v1.6.11 \
|
||
WEBUI_OVERLAY_DENY='c778af04|master-11340' \
|
||
./taler-monitoring.sh -d hacktivism.ch urls
|
||
```
|
||
|
||
Skip SPA block: `CHECK_WEBUI_SPA=0`.
|
||
|
||
### Full load (maximum phases)
|
||
|
||
| Alias | Expands to (local GOA) |
|
||
|-------|-------------------------|
|
||
| **`all`** | `urls inside versions sanity e2e` |
|
||
| **`full`** | `urls inside versions sanity server e2e ladder auth401` |
|
||
|
||
```bash
|
||
cd ~/src/taler-monitoring
|
||
# secrets.env with bank admin + merchant tokens (see secrets.env.example)
|
||
./taler-monitoring.sh -d hacktivism.ch full
|
||
# same, explicit:
|
||
./taler-monitoring.sh -d hacktivism.ch \
|
||
urls inside versions sanity server e2e ladder auth401
|
||
```
|
||
|
||
---
|
||
|
||
## Visual format (colour on)
|
||
|
||
```text
|
||
╔══════════════════════════════╗
|
||
║ www · public URLs · … ║
|
||
╚══════════════════════════════╝
|
||
┌ www.exchange ┐
|
||
┌ OK ┐ www.exchange-01 exchange /config · HTTP 200
|
||
┌ www.bank ┐
|
||
┌ OK ┐ www.bank-01 bank /config · HTTP 200
|
||
┌ ERROR ┐ www.bank-04 bank /intro/auto-account.json · invalid withdraw
|
||
|
||
┌ ERRORS · failed checks ┐
|
||
• www.bank-04 bank /intro/auto-account.json · invalid withdraw
|
||
totals: 40 OK, 1 ERROR, 2 WARN, 5 INFO
|
||
```
|
||
|
||
`NO_COLOR=1` or `CLICOLOR=0` disables boxes/colours (ASCII `[ OK ]` + `-- group --` headers).
|
||
Always use `printf --` friendly plain headers when colour is off (leading `---` is not a printf option).
|