monitoring: boxed badges + area.group test IDs
This commit is contained in:
Hernâni Marques
parent
f8affeb651
commit
e1cc6d7141
10 changed files with 286 additions and 225 deletions
|
|
@ -1,217 +1,132 @@
|
|||
# taler-monitoring — test IDs by area
|
||||
# taler-monitoring — grouped test IDs
|
||||
|
||||
Every check line is numbered **per area** as `AREA-NNN` (zero-padded):
|
||||
|
||||
| Area | Phase script | Meaning |
|
||||
|------|--------------|---------|
|
||||
| **www** | `check_urls.sh` | public HTTPS (outside-in) |
|
||||
| **inside** | `check_inside.sh` | containers / processes on koopa |
|
||||
| **versions** | `check_versions.sh` | deb.taler.net available + packages vs trixie |
|
||||
| **sanity** | `check_sanity.sh` | public + server per component |
|
||||
| **server** | `check_server.sh` | SSH host ports / processes |
|
||||
| **e2e** | `check_e2e.sh` | withdraw + pay cycle |
|
||||
|
||||
Format in output:
|
||||
Every check line is numbered **`area.group-NN`** (two-digit group counter):
|
||||
|
||||
```text
|
||||
[OK] www-001 exchange /config https://exchange…/config
|
||||
[ERROR] e2e-012 bank-auth: admin token failed
|
||||
[BLOCKER] e2e-015 prereq: merchant HTTP 502
|
||||
┌ OK ┐ www.exchange-01 exchange /config · HTTP 200
|
||||
┌ ERROR ┐ e2e.pay-03 order create failed · HTTP 502
|
||||
┌ BLOCKER┐ e2e.prereq-02 merchant secret missing
|
||||
```
|
||||
|
||||
IDs are assigned **in run order** within the area (`set_area` resets the counter). Optional soft checks still consume a number when they WARN.
|
||||
| Area | Phase script | Groups (examples) |
|
||||
|------|--------------|-------------------|
|
||||
| **www** | `check_urls.sh` | `exchange` `perf` `stats` `bank` `merchant` `paivana` `landing` |
|
||||
| **inside** | `check_inside.sh` | `ssh` `bank` `exchange` `merchant` `caddy` `load` |
|
||||
| **versions** | `check_versions.sh` | `outside` `inside` `compare` |
|
||||
| **sanity** | `check_sanity.sh` | `bank` `exchange` `merchant` |
|
||||
| **server** | `check_server.sh` | (flat `server-NN` or host groups) |
|
||||
| **e2e** | `check_e2e.sh` | `prereq` `load` `bank` `wallet` `atm` `settle` `pay` `shop` `paivana` `dig` `report` |
|
||||
| **ladder** | `check_goa_ladder.sh` | `plan` `load` `withdraw` `pay` `report` |
|
||||
|
||||
**Why groups:** flat `www-001`…`www-080` was hard to talk about.
|
||||
`www.bank-04` / `e2e.atm-02` pin the failure to a logical block.
|
||||
|
||||
Usage in scripts:
|
||||
|
||||
```bash
|
||||
set_area www
|
||||
set_group exchange # → www.exchange-01, www.exchange-02, …
|
||||
set_group bank # counter resets → www.bank-01
|
||||
```
|
||||
|
||||
`set_group` also prints a small group chip (`┌ www.exchange ┐`) so logs show section boundaries.
|
||||
Re-entering the same group later (e.g. exchange legal after perf) **continues** NN — `www.exchange-03`, not a second `-01`.
|
||||
Without `set_group`, IDs stay flat: `area-01`, `area-02`, …
|
||||
|
||||
Optional soft checks still consume a number when they WARN.
|
||||
Numbering follows **executed** checks (early skip may shift later NN inside the same group).
|
||||
|
||||
---
|
||||
|
||||
## www — public URLs (`./taler-monitoring.sh urls`)
|
||||
|
||||
| ID | Check |
|
||||
|----|--------|
|
||||
| www-… | exchange `/config`, currency, **alt_unit_names** |
|
||||
| www-… | exchange `/keys` (+ alt_unit_names soft) |
|
||||
| www-… | exchange `/intro/`, `/` (302→intro) |
|
||||
| www-… | **exchange `/terms`** body (not empty / not API error) |
|
||||
| www-… | **exchange `/privacy`** body |
|
||||
| www-… | exchange `/terms/` (200 or redirect) |
|
||||
| www-… | bank `/config`, currency, **alt_unit_names** |
|
||||
| www-… | bank integration / webui / intro / `/` |
|
||||
| www-… | **bank `/terms`** body |
|
||||
| www-… | **bank `/privacy`** (or `/intro/privacy.html` fallback) |
|
||||
| www-… | merchant `/config`, currency, currencies alt_unit_names |
|
||||
| www-… | each merchant `exchanges[]` `/config` alt_unit_names |
|
||||
| www-… | merchant `/intro/`, `/webui/`, `/` |
|
||||
| www-… | **merchant `/terms`** body (dual-currency notice) |
|
||||
| www-… | **merchant `/privacy`** body (must not be `not configured`) |
|
||||
| www-… | merchant `/terms/` redirect |
|
||||
| www-… | **landing (aggregated per site)**: one line for bank/merchant/exchange (`/intro` + assets + own/external link counts); failures listed as samples; soft external/shop assets compact |
|
||||
| www-… | cross-links: one line (4 intros) on local stack |
|
||||
| www-… | bank demo-withdraw + shop assets: one line (+ soft shop-pay if missing) |
|
||||
| www-… | **bank `/intro/demo-withdraw.json`** → `taler://withdraw/HOST:PORT/taler-integration/…` + integration op HTTP 200 |
|
||||
| www-… | **landing exposed links** (bank / merchant / exchange): parse each `/intro/` HTML, probe every own-stack `https://` + root-relative `href`/`src`/`content`, soft-check external stores/docs |
|
||||
| www-… | landing static: `qrcode.min.js`, `og-goa-shop.png`, `qr-logo.png`, shop-pay.js/css |
|
||||
| www-… | cross-links between bank ↔ merchant ↔ exchange intros (local stack) |
|
||||
| www-… | **bank `/intro/demo-withdraw.json`** → `taler://withdraw/HOST/taler-integration/…` (no default `:443`/`:80`; non-default port OK) + integration op HTTP 200 |
|
||||
| www-… | bank `/intro/auto-account.json` (earlier) → same withdraw shape, **no payto_uri**, login at `/webui/` |
|
||||
| www-… | **performance** (outside-in RTT): bank/exchange/merchant `/config`, keys, webui, intro — ms; WARN ≥ `PERF_WARN_MS` (8000); ERROR ≥ `PERF_FAIL_MS` (20000) |
|
||||
| www-… | **landing load stats**: `/intro/stats.json` → loadavg, container RSS, in-container probe ms (public; SSH container cat fallback); warn if stale/missing |
|
||||
| Group | Checks (typical) |
|
||||
|-------|------------------|
|
||||
| **www.exchange-** | `/config`, currency, **alt_unit_names**; `/keys` (+ alt soft); `/intro/`, `/`; **`/terms`**, **`/privacy`**, `/terms/` |
|
||||
| **www.perf-** | outside-in RTT: bank/exchange/merchant `/config`, keys, webui, intro — ms; WARN ≥ `PERF_WARN_MS` (8000); ERROR ≥ `PERF_FAIL_MS` (20000) |
|
||||
| **www.stats-** | `/intro/stats.json` loadavg / container RSS / probe ms (public; SSH fallback) |
|
||||
| **www.bank-** | `/config`, currency, alt_unit_names; integration/webui/intro; **auto-account.json**; `/terms`, `/privacy` |
|
||||
| **www.merchant-** | `/config` currency + currencies alt_unit_names; listed exchanges alt; webui/intro; **`/terms`**, **`/privacy`** |
|
||||
| **www.paivana-** | local GOA paywall front (redirect to template) |
|
||||
| **www.landing-** | every own-stack link on bank/merchant/exchange intros; static assets; demo-withdraw / cross-links |
|
||||
|
||||
**Legal docs rule:** HTTP 200, non-empty body, not plain `not configured`, not merchant API JSON `code:21`. On local stack, optional content needle (terms/privacy/FADP/GOA…).
|
||||
**Legal docs rule:** HTTP 200, non-empty body, not plain `not configured`, not merchant API JSON `code:21`. Local stack may require content needle (terms/privacy/FADP/GOA…).
|
||||
|
||||
**Performance rule:** Measured from the **monitoring runner** (public URLs via Caddy), not container loopback. HTTP must match expect (usually 200); **latency (ms) is on every perf OK/WARN/ERROR line**, plus a **perf summary** (n / min / p50 / avg / max). Slow ≥ `PERF_WARN_MS` (default 8000) → WARN; ≥ `PERF_FAIL_MS` (default 20000) → ERROR.
|
||||
**Performance rule:** Measured from the **monitoring runner** (public via Caddy), not container loopback. Latency (ms) on every perf line + **perf summary** (n / min / p50 / avg / max).
|
||||
|
||||
**Landing links rule:** Own-stack (bank/exchange/taler.\* + page host) must be HTTP 200 (or redirect→200). External (App Store, Play, F-Droid, wallet.taler.net, docs/git.taler.net, …) soft WARN if down. Auto-account wallet link must be `taler://withdraw/HOST/taler-integration/…` (default ports stripped for mobile wallets), never payto.
|
||||
**Landing links rule:** Own-stack must be HTTP 200 (or redirect→200). External stores/docs soft WARN. Auto-account wallet link must be `taler://withdraw/HOST/taler-integration/…` (default ports stripped), never payto.
|
||||
|
||||
**alt_unit_names rule:** wallet codec requires a non-empty map including scale key `"0"`. For multi-currency merchant, also follow every entry in `exchanges[]` and check that exchange’s public `/config`.
|
||||
|
||||
(IDs after a failed early check may shift if later soft checks are skipped when body missing — numbering follows **executed** checks.)
|
||||
**alt_unit_names rule:** non-empty map including scale key `"0"`. Multi-currency merchant: follow each `exchanges[]` public `/config`.
|
||||
|
||||
---
|
||||
|
||||
## inside — koopa SSH (`./taler-monitoring.sh inside`)
|
||||
|
||||
| ID | Check (typical order) |
|
||||
|----|------------------------|
|
||||
| inside-001 | ssh koopa (or **koopa-external** fallback) |
|
||||
| inside-002+ | per-component emit: container, ports, libeufin/httpd, postgres, local `/config`/`/keys`, wirewatch, DNS pin, caddy |
|
||||
| inside-… | **load / memory**: host loadavg + RAM; bank/exchange/merchant podman CPU/mem/block + process RSS by role + DB sizes |
|
||||
| Group | Checks |
|
||||
|-------|--------|
|
||||
| **inside.ssh-** | ssh reachability / remote collect |
|
||||
| **inside.bank-** | container, libeufin, postgres, local `/config`, nginx, DNS pins |
|
||||
| **inside.exchange-** | container, httpd, wirewatch, aggregator, transfer, local keys, DNS |
|
||||
| **inside.merchant-** | container, httpd, wirewatch, depositcheck, DNS |
|
||||
| **inside.caddy-** | host reverse-proxy process |
|
||||
| **inside.load-** | host loadavg + RAM; per-container RSS/CPU |
|
||||
|
||||
Remote lines `E|comp|LEVEL|key|detail` each become one numbered result.
|
||||
Remote lines `E|comp|LEVEL|key|detail` each become one numbered result under that component group.
|
||||
|
||||
SSH: `KOOPA_SSH` (default `koopa`), then `KOOPA_SSH_FALLBACKS` (default `koopa-external`) when LAN is unreachable.
|
||||
SSH: `KOOPA_SSH` (default `koopa`), then `KOOPA_SSH_FALLBACKS` when LAN is unreachable.
|
||||
|
||||
---
|
||||
|
||||
## sanity — bank · exchange · merchant (`./taler-monitoring.sh sanity`)
|
||||
## e2e — withdraw + pay (`./taler-monitoring.sh e2e`)
|
||||
|
||||
| ID | Section |
|
||||
|----|---------|
|
||||
| sanity-001… | bank public + server |
|
||||
| sanity-… | exchange public + server |
|
||||
| sanity-… | merchant public + server |
|
||||
| Group | Checks |
|
||||
|-------|--------|
|
||||
| **e2e.prereq-** | wallet-cli, currency, budgets, secrets, public reachability |
|
||||
| **e2e.load-** | host/container load snapshots (before / after ATM / after pay) |
|
||||
| **e2e.bank-** | account, credit, withdraw ops |
|
||||
| **e2e.wallet-** | exchange + ToS, wallet setup |
|
||||
| **e2e.atm-** | ATM withdraw ladder rungs |
|
||||
| **e2e.settle-** | wallet settlement wait / spendable balance |
|
||||
| **e2e.pay-** | variable payments |
|
||||
| **e2e.shop-** | GOA shop product catalog pays |
|
||||
| **e2e.paivana-** | template paywall pay |
|
||||
| **e2e.dig-** | coins-missing dig (inside + withdrawal-operation) |
|
||||
| **e2e.report-** | final tallies |
|
||||
|
||||
Sequential through the whole script (one `set_area sanity`).
|
||||
When filing an issue, quote the full id + label, e.g.
|
||||
`e2e.atm-03 ATM withdraw 5 GOA · bank confirmed, wallet empty`.
|
||||
|
||||
---
|
||||
|
||||
## versions — packages vs deb.taler.net (`./taler-monitoring.sh versions`)
|
||||
## versions / sanity / ladder / server
|
||||
|
||||
### Outside (runner / public network — no SSH)
|
||||
|
||||
| ID (order) | Check |
|
||||
|------------|--------|
|
||||
| versions-… | DNS `deb.taler.net` |
|
||||
| versions-… | HTTPS portal + apt base URL |
|
||||
| versions-… | suite `InRelease` / `Release` |
|
||||
| versions-… | suite `Packages` + `Packages.gz` |
|
||||
| versions-… | sample pool `.deb` fetchable (Range 200/206) |
|
||||
| versions-… | suite offers `taler-exchange`, `taler-merchant`, `libeufin-bank` |
|
||||
| versions-… | optional `trixie-testing` Packages |
|
||||
| versions-… | TLS verify (soft) |
|
||||
|
||||
### Inside (SSH koopa containers)
|
||||
|
||||
| ID | Check |
|
||||
|----|--------|
|
||||
| versions-… | ssh koopa |
|
||||
| versions-… | each container → `InRelease` (pasta can reach apt repo) |
|
||||
| versions-… | each container lists `deb.taler.net` in apt sources |
|
||||
| versions-… | each installed `taler*` / `libeufin*` / `libtaler*` / `libdonau*` vs suite version |
|
||||
| versions-… | core packages installed (`taler-exchange`, `libeufin-bank`, `taler-merchant`) |
|
||||
|
||||
Outside always runs. Inside skipped with `SKIP_SSH=1` (still reports outside results).
|
||||
|
||||
Compare rules:
|
||||
|
||||
- **match** suite → OK
|
||||
- **ahead** of suite (often testing/dev) → INFO
|
||||
- **behind** suite → ERROR for core packages, WARN otherwise (`TALER_PKG_BEHIND=error` forces ERROR)
|
||||
|
||||
Default suite: **trixie** (`TALER_APT_SUITE`, `TALER_APT_BASE=https://deb.taler.net/apt/debian`).
|
||||
|
||||
Without SSH (`SKIP_SSH=1` or remote domain): still runs outside-in repo checks; skips container install compare.
|
||||
| Area.group | Meaning |
|
||||
|------------|---------|
|
||||
| **versions.outside-** | deb.taler.net suite index |
|
||||
| **versions.inside-** | packages in containers |
|
||||
| **versions.compare-** | installed vs suite |
|
||||
| **sanity.bank-** / **.exchange-** / **.merchant-** | public + optional server-side per component |
|
||||
| **ladder.plan-** / **.load-** / **.withdraw-** / **.pay-** / **.report-** | GOA amount ladder |
|
||||
| **server-** | SSH host ports / processes (flat unless grouped later) |
|
||||
|
||||
---
|
||||
|
||||
## server — SSH ports (`./taler-monitoring.sh server`)
|
||||
|
||||
| ID | Check |
|
||||
|----|--------|
|
||||
| server-001 | ssh |
|
||||
| server-002+ | containers, local pasta ports, processes, caddy |
|
||||
|
||||
---
|
||||
|
||||
## e2e — payment path (`./taler-monitoring.sh e2e`)
|
||||
|
||||
Local GOA also: **ATM includes GOA:4200**, then **paivana** (HTTP 302 on `PAIVANA_PUBLIC` + public template pay `paivana` / GOA:4200 on `goa-shop`). Disable with `E2E_PAIVANA=0`.
|
||||
|
||||
| ID | Step (approx.) |
|
||||
|----|----------------|
|
||||
| e2e-001 | budget info |
|
||||
| e2e-002 | wallet-cli present |
|
||||
| e2e-003 | mode / currency info |
|
||||
| e2e-004… | secrets, reachability gates |
|
||||
| e2e-… | account, credit, withdraw, confirm, coins, order, pay ladder |
|
||||
| e2e-… | **load snapshots** at e2e-start, after-withdraw, after-pay, after-shop, e2e-end (host loadavg/RAM + container RSS/CPU/DB) |
|
||||
| e2e-… | **coins** before/after each ATM withdraw, settle, pay, shop: count in circulation / spent, amount + denoms using **`alt_unit_names`** (e.g. `5 Kilo-GOA (GOA:5000)`), Δ vs previous snap |
|
||||
| e2e-… | **GOA shop products** — full catalog list; **random pick of 2** (override `E2E_SHOP_PICK_N`) |
|
||||
| e2e-… | balances, dig on failure, load delta + overall metrics |
|
||||
|
||||
Shop product pays use instance `goa-shop` (default) and catalog `E2E_SHOP_PRODUCTS`
|
||||
(`id|Product name|amount` lines). Each e2e run **shuffles** the catalog and pays
|
||||
only `E2E_SHOP_PICK_N` products (**default 2**). Flow matches the landing popup
|
||||
(public POST `/templates/{id}`, not private orders). Report labels use product name.
|
||||
|
||||
Blockers keep the same ID prefix: `[BLOCKER] e2e-0NN step: message`.
|
||||
|
||||
---
|
||||
|
||||
## ladder — GOA withdraw ranges (`./taler-monitoring.sh ladder`)
|
||||
|
||||
| Step | What |
|
||||
|------|------|
|
||||
| ladder-001 | `GET /intro/auto-account.json` (personal account, GOA:0) |
|
||||
| ladder-… | **load snapshot** before withdraws (host + bank/exchange/merchant mem/CPU) |
|
||||
| ladder-… | **Phase A withdraw** (23): mint explorer pool — 0 → random → max−1 → max into **one cumulative wallet** (mids × `LADDER_WITHDRAW_SCALE`, default 1.5) |
|
||||
| ladder-… | wallet-cli accept-uri + explorer confirm when `selected`; settle; coins snap |
|
||||
| ladder-… | **Phase B pay** (23): same shape 0 → random → max−1 → max via merchant orders + `handle-uri --yes` |
|
||||
| ladder-… | **load snapshot** after + delta + overall timing (withdraw + pay) |
|
||||
| report | withdraw TSV + pay TSV + JSON |
|
||||
|
||||
Default amounts (`build_ladder_pair`, strictly increasing):
|
||||
## Visual format (colour on)
|
||||
|
||||
```text
|
||||
withdraw: [0] + random mids + [max−1] + [max]
|
||||
pay: [0] + (withdraw_mid / scale) + [max−1] + [max] # always ≤ matching withdraw mid
|
||||
╔══════════════════════════════╗
|
||||
║ www · public URLs · … ║
|
||||
╚══════════════════════════════╝
|
||||
┌ www.exchange ┐
|
||||
┌ OK ┐ www.exchange-01 exchange /config · HTTP 200
|
||||
┌ www.bank ┐
|
||||
┌ OK ┐ www.bank-01 bank /config · HTTP 200
|
||||
┌ ERROR ┐ www.bank-04 bank /intro/auto-account.json · invalid withdraw
|
||||
|
||||
┌ ERRORS · failed checks ┐
|
||||
• www.bank-04 bank /intro/auto-account.json · invalid withdraw
|
||||
totals: 40 OK, 1 ERROR, 2 WARN, 5 INFO
|
||||
```
|
||||
|
||||
TSV `range`: `pin:0` | `random` | `pin:max-1` | `pin:max`.
|
||||
|
||||
Soft: absolute **max** mint/pay may `CEILING_REJECT` / skip (WARN). **max-1** is a hard pin.
|
||||
`LADDER_PAY=0` skips phase B.
|
||||
|
||||
| Env | Default | Meaning |
|
||||
|-----|---------|---------|
|
||||
| `LADDER_STEPS` | `23` | total rungs (0 + mids + max) |
|
||||
| `LADDER_MAX_AMOUNT` | `4503599627370496` | absolute last pin |
|
||||
| `LADDER_TIMEOUT_S` | `3600` | large rungs need time |
|
||||
| `LADDER_MAX_RUNGS` | `99` | cap list length after build |
|
||||
|
||||
```bash
|
||||
./taler-monitoring.sh ladder
|
||||
LADDER_STEPS=10 ./taler-monitoring.sh ladder
|
||||
LADDER_REPORT_DIR=/tmp/my-ladder ./taler-monitoring.sh ladder
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Run one area
|
||||
|
||||
```bash
|
||||
./taler-monitoring.sh urls # www only
|
||||
./taler-monitoring.sh inside # inside only
|
||||
./taler-monitoring.sh versions # deb.taler.net + package drift
|
||||
./taler-monitoring.sh e2e # e2e only
|
||||
./taler-monitoring.sh ladder # GOA amount ladder + timings
|
||||
./taler-monitoring.sh -d taler.net urls
|
||||
```
|
||||
`NO_COLOR=1` or `CLICOLOR=0` disables boxes/colours (ASCII `[ OK ]` + `-- group --` headers).
|
||||
Always use `printf --` friendly plain headers when colour is off (leading `---` is not a printf option).
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue