hernani/koopa-admin-log
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(monitoring): load, coins, pay ladder, alt units, SSH fallbacks

Browse source
This commit is contained in:
Hernâni Marques 2026-07-17 03:04:54 +02:00
parent 765c528313
commit 51785031d6
No known key found for this signature in database
3 changed files with 76 additions and 13 deletions
Download patch file Download diff file Expand all files Collapse all files

50
scripts/taler-monitoring/README.md
View file

@ -110,8 +110,56 @@ Outside-in HTTPS RTT for bank / exchange / merchant critical paths. Each probe p
PERF_WARN_MS=3000 PERF_FAIL_MS=10000 ./taler-monitoring.sh urls
```
## Wallet coins (e2e · ladder)
At every relevant withdraw/pay step the wallet is snapped via `advanced dump-coins`.
Amounts use exchange **`alt_unit_names`** (Kilo-GOA, Mega-GOA, …) with the base
**GOA value in parentheses**:
```text
[INFO] coins after-ATM — amount_circ=2 Kilo-GOA (GOA:2000)
[INFO] coins after-ATM — denoms_in_circ 1 Kilo-GOA (GOA:1000)×2 (=2 Kilo-GOA (GOA:2000))
== ladder · rung 12 GOA:1000000 · 1 Mega-GOA (GOA:1000000) ==
```
Also: coin counts (in circulation / spent), status histogram, Δ vs previous snap.
History TSV: `$METRICS_DIR/coins-history.tsv`. Map loaded from
`${EXCHANGE_PUBLIC}/config` → `currency_specification.alt_unit_names`.
## Load / memory (inside · e2e · ladder)
Host **loadavg**, **RAM used/avail**, and per-container **RSS / CPU / block I/O**
(plus postgres DB sizes and role RSS: libeufin, exchange, merchant, …) are
snapshotted via SSH whenever the stack is under load:
| Phase | When |
|-------|------|
| **inside** | end of container status |
| **e2e** | before work · after ATM withdraws · after payments · after shop · e2e-end (+ delta) |
| **ladder** | before first rung · after last rung (+ delta + timing overall) |
```bash
# LAN down? force WAN DNAT host
KOOPA_SSH=koopa-external ./taler-monitoring.sh inside
# or rely on automatic fallback (default KOOPA_SSH_FALLBACKS=koopa-external)
METRICS_LOAD=0 ./taler-monitoring.sh e2e # skip host probes
LADDER_LOAD=0 ./taler-monitoring.sh ladder
```
| Env | Default | Meaning |
|-----|---------|---------|
| `KOOPA_SSH` | `koopa` | primary SSH host |
| `KOOPA_SSH_FALLBACKS` | `koopa-external` | tried if primary fails |
| `METRICS_LOAD` | `1` | `0` = skip all host/container load probes |
| `LADDER_LOAD` | `1` | `0` = skip load in ladder only |
| `LADDER_PAY` | `1` | `0` = skip payment ladder after withdraws |
| `LADDER_WITHDRAW_SCALE` | `1.5` | withdraw mids / pay mids ratio (funding headroom) |
| `LADDER_STEPS` | `23` | rungs for both withdraw and pay (0 + mids + max−1 + max) |
| `METRICS_LOAD_SSH_TIMEOUT` | `90` | seconds for remote load python |
## Needs
- SSH `koopa` (inside/sanity server bits)
- SSH `koopa` or `koopa-external` (inside / load / sanity server bits)
- secrets under `koopa-admin-secrets/...` for e2e
- `taler-wallet-cli` for e2e

31
scripts/taler-monitoring/TESTS.md
View file

@ -66,11 +66,14 @@ IDs are assigned **in run order** within the area (`set_area` resets the counter
| ID | Check (typical order) |
|----|------------------------|
| inside-001 | ssh koopa |
| inside-001 | ssh koopa (or **koopa-external** fallback) |
| inside-002+ | per-component emit: container, ports, libeufin/httpd, postgres, local `/config`/`/keys`, wirewatch, DNS pin, caddy |
| inside-… | **load / memory**: host loadavg + RAM; bank/exchange/merchant podman CPU/mem/block + process RSS by role + DB sizes |
Remote lines `E|comp|LEVEL|key|detail` each become one numbered result.
SSH: `KOOPA_SSH` (default `koopa`), then `KOOPA_SSH_FALLBACKS` (default `koopa-external`) when LAN is unreachable.
---
## sanity — bank · exchange · merchant (`./taler-monitoring.sh sanity`)
@ -142,8 +145,10 @@ Without SSH (`SKIP_SSH=1` or remote domain): still runs outside-in repo checks;
| e2e-003 | mode / currency info |
| e2e-004… | secrets, reachability gates |
| e2e-… | account, credit, withdraw, confirm, coins, order, pay ladder |
| e2e-… | **load snapshots** at e2e-start, after-withdraw, after-pay, after-shop, e2e-end (host loadavg/RAM + container RSS/CPU/DB) |
| e2e-… | **coins** before/after each ATM withdraw, settle, pay, shop: count in circulation / spent, amount + denoms using **`alt_unit_names`** (e.g. `5 Kilo-GOA (GOA:5000)`), Δ vs previous snap |
| e2e-… | **GOA shop products** — full catalog list; **random pick of 2** (override `E2E_SHOP_PICK_N`) |
| e2e-… | balances, dig on failure |
| e2e-… | balances, dig on failure, load delta + overall metrics |
Shop product pays use instance `goa-shop` (default) and catalog `E2E_SHOP_PRODUCTS`
(`id|Product name|amount` lines). Each e2e run **shuffles** the catalog and pays
@ -159,20 +164,24 @@ Blockers keep the same ID prefix: `[BLOCKER] e2e-0NN step: message`.
| Step | What |
|------|------|
| ladder-001 | `GET /intro/auto-account.json` (personal account, GOA:0) |
| ladder-… | Mint explorer pool withdrawals: **zero**, then **random** in ranges (strictly increasing), then **pin at bank max** |
| ladder-… | wallet-cli accept-uri + explorer confirm when `selected` |
| ladder-… | Settle coins; **stop on first hard failure** |
| report | TSV + JSON with **ms_mint / ms_accept / ms_confirm / ms_settle / ms_total** |
| ladder-… | **load snapshot** before withdraws (host + bank/exchange/merchant mem/CPU) |
| ladder-… | **Phase A withdraw** (23): mint explorer pool — 0 → random → max−1 → max into **one cumulative wallet** (mids × `LADDER_WITHDRAW_SCALE`, default 1.5) |
| ladder-… | wallet-cli accept-uri + explorer confirm when `selected`; settle; coins snap |
| ladder-… | **Phase B pay** (23): same shape 0 → random → max−1 → max via merchant orders + `handle-uri --yes` |
| ladder-… | **load snapshot** after + delta + overall timing (withdraw + pay) |
| report | withdraw TSV + pay TSV + JSON |
Default path (`build_ladder`, strictly increasing):
Default amounts (`build_ladder_pair`, strictly increasing):
```text
[fixed] GOA:0
[random] (LADDER_STEPS − 2) log-uniform amounts in (ε, max)
[fixed] GOA:4503599627370496 (LADDER_MAX_AMOUNT)
withdraw: [0] + random mids + [max−1] + [max]
pay: [0] + (withdraw_mid / scale) + [max−1] + [max] # always ≤ matching withdraw mid
```
TSV column `range`: `pin:0` | `random` | `pin:max` (no legacy `LADDER_RANGES`).
TSV `range`: `pin:0` | `random` | `pin:max-1` | `pin:max`.
Soft: absolute **max** mint/pay may `CEILING_REJECT` / skip (WARN). **max-1** is a hard pin.
`LADDER_PAY=0` skips phase B.
| Env | Default | Meaning |
|-----|---------|---------|

8
scripts/taler-monitoring/taler-monitoring.sh
View file

@ -52,6 +52,8 @@ Env (same meaning):
TALER_DOMAIN BANK_PUBLIC EXCHANGE_PUBLIC MERCHANT_PUBLIC EXPECT_CURRENCY
SKIP_SSH=1 NO_COLOR=1
PERF_WARN_MS PERF_FAIL_MS (urls latency; default 8000 / 20000)
KOOPA_SSH KOOPA_SSH_FALLBACKS (default koopa → koopa-external)
METRICS_LOAD=0 skip host/container RAM/CPU probes (e2e/ladder/inside)
EOF
}
@ -135,7 +137,7 @@ export EXPECT_CURRENCY SKIP_SSH LOCAL_STACK TALER_DOMAIN_PROBE
export WITHDRAW_AMT PAY_AMT CREDIT_AMT MERCHANT_INSTANCE
export E2E_FAKE_INCOMING E2E_REMOTE E2E_VARIABLE E2E_ATM_MAX
export E2E_WITHDRAW_VALUES E2E_PAY_VALUES
# Ladder: 0 + random mids + max (see check_goa_ladder.sh build_ladder). No LADDER_RANGES.
# Ladder: withdraw then pay — 0 + random mids + max-1 + max (see check_goa_ladder.sh).
# Defaults so set -u export is safe when vars were never set by caller.
: "${LADDER_STEPS:=23}"
: "${LADDER_MIN_AMOUNT:=0.000001}"
@ -151,10 +153,14 @@ export E2E_WITHDRAW_VALUES E2E_PAY_VALUES
: "${LADDER_HIGH_FROM:=1000000}"
: "${LADDER_HIGH_RUNGS:=12}"
: "${LADDER_LOAD:=1}"
: "${LADDER_PAY:=1}"
: "${LADDER_WITHDRAW_SCALE:=1.5}"
: "${LADDER_PAY_SETTLE_ROUNDS:=6}"
export LADDER_STEPS LADDER_MIN_AMOUNT LADDER_CONFIRM_POLLS LADDER_MAX_RUNGS LADDER_TIMEOUT_S LADDER_REPORT_DIR
export LADDER_SETTLE_ROUNDS LADDER_SETTLE_SLEEP EXP_PW_FILE EXP_USER
export LADDER_MAX_AMOUNT LADDER_INCLUDE_ZERO LADDER_INCLUDE_MAX
export LADDER_HIGH_FROM LADDER_HIGH_RUNGS LADDER_LOAD
export LADDER_PAY LADDER_WITHDRAW_SCALE LADDER_PAY_SETTLE_ROUNDS
export TALER_DOMAIN_APPLIED=1
# Default phases