diff --git a/scripts/taler-monitoring/README.md b/scripts/taler-monitoring/README.md index d8408c8..98214d6 100644 --- a/scripts/taler-monitoring/README.md +++ b/scripts/taler-monitoring/README.md @@ -110,8 +110,56 @@ Outside-in HTTPS RTT for bank / exchange / merchant critical paths. Each probe p PERF_WARN_MS=3000 PERF_FAIL_MS=10000 ./taler-monitoring.sh urls ``` +## Wallet coins (e2e · ladder) + +At every relevant withdraw/pay step the wallet is snapped via `advanced dump-coins`. +Amounts use exchange **`alt_unit_names`** (Kilo-GOA, Mega-GOA, …) with the base +**GOA value in parentheses**: + +```text +[INFO] coins after-ATM — amount_circ=2 Kilo-GOA (GOA:2000) +[INFO] coins after-ATM — denoms_in_circ 1 Kilo-GOA (GOA:1000)×2 (=2 Kilo-GOA (GOA:2000)) +== ladder · rung 12 GOA:1000000 · 1 Mega-GOA (GOA:1000000) == +``` + +Also: coin counts (in circulation / spent), status histogram, Δ vs previous snap. +History TSV: `$METRICS_DIR/coins-history.tsv`. Map loaded from +`${EXCHANGE_PUBLIC}/config` → `currency_specification.alt_unit_names`. + +## Load / memory (inside · e2e · ladder) + +Host **loadavg**, **RAM used/avail**, and per-container **RSS / CPU / block I/O** +(plus postgres DB sizes and role RSS: libeufin, exchange, merchant, …) are +snapshotted via SSH whenever the stack is under load: + +| Phase | When | +|-------|------| +| **inside** | end of container status | +| **e2e** | before work · after ATM withdraws · after payments · after shop · e2e-end (+ delta) | +| **ladder** | before first rung · after last rung (+ delta + timing overall) | + +```bash +# LAN down? force WAN DNAT host +KOOPA_SSH=koopa-external ./taler-monitoring.sh inside +# or rely on automatic fallback (default KOOPA_SSH_FALLBACKS=koopa-external) + +METRICS_LOAD=0 ./taler-monitoring.sh e2e # skip host probes +LADDER_LOAD=0 ./taler-monitoring.sh ladder +``` + +| Env | Default | Meaning | +|-----|---------|---------| +| `KOOPA_SSH` | `koopa` | primary SSH host | +| `KOOPA_SSH_FALLBACKS` | `koopa-external` | tried if primary fails | +| `METRICS_LOAD` | `1` | `0` = skip all host/container load probes | +| `LADDER_LOAD` | `1` | `0` = skip load in ladder only | +| `LADDER_PAY` | `1` | `0` = skip payment ladder after withdraws | +| `LADDER_WITHDRAW_SCALE` | `1.5` | withdraw mids / pay mids ratio (funding headroom) | +| `LADDER_STEPS` | `23` | rungs for both withdraw and pay (0 + mids + max−1 + max) | +| `METRICS_LOAD_SSH_TIMEOUT` | `90` | seconds for remote load python | + ## Needs -- SSH `koopa` (inside/sanity server bits) +- SSH `koopa` or `koopa-external` (inside / load / sanity server bits) - secrets under `koopa-admin-secrets/...` for e2e - `taler-wallet-cli` for e2e diff --git a/scripts/taler-monitoring/TESTS.md b/scripts/taler-monitoring/TESTS.md index c9dbe09..ad1e6d5 100644 --- a/scripts/taler-monitoring/TESTS.md +++ b/scripts/taler-monitoring/TESTS.md @@ -66,11 +66,14 @@ IDs are assigned **in run order** within the area (`set_area` resets the counter | ID | Check (typical order) | |----|------------------------| -| inside-001 | ssh koopa | +| inside-001 | ssh koopa (or **koopa-external** fallback) | | inside-002+ | per-component emit: container, ports, libeufin/httpd, postgres, local `/config`/`/keys`, wirewatch, DNS pin, caddy | +| inside-… | **load / memory**: host loadavg + RAM; bank/exchange/merchant podman CPU/mem/block + process RSS by role + DB sizes | Remote lines `E|comp|LEVEL|key|detail` each become one numbered result. +SSH: `KOOPA_SSH` (default `koopa`), then `KOOPA_SSH_FALLBACKS` (default `koopa-external`) when LAN is unreachable. + --- ## sanity — bank · exchange · merchant (`./taler-monitoring.sh sanity`) @@ -142,8 +145,10 @@ Without SSH (`SKIP_SSH=1` or remote domain): still runs outside-in repo checks; | e2e-003 | mode / currency info | | e2e-004… | secrets, reachability gates | | e2e-… | account, credit, withdraw, confirm, coins, order, pay ladder | +| e2e-… | **load snapshots** at e2e-start, after-withdraw, after-pay, after-shop, e2e-end (host loadavg/RAM + container RSS/CPU/DB) | +| e2e-… | **coins** before/after each ATM withdraw, settle, pay, shop: count in circulation / spent, amount + denoms using **`alt_unit_names`** (e.g. `5 Kilo-GOA (GOA:5000)`), Δ vs previous snap | | e2e-… | **GOA shop products** — full catalog list; **random pick of 2** (override `E2E_SHOP_PICK_N`) | -| e2e-… | balances, dig on failure | +| e2e-… | balances, dig on failure, load delta + overall metrics | Shop product pays use instance `goa-shop` (default) and catalog `E2E_SHOP_PRODUCTS` (`id|Product name|amount` lines). Each e2e run **shuffles** the catalog and pays @@ -159,20 +164,24 @@ Blockers keep the same ID prefix: `[BLOCKER] e2e-0NN step: message`. | Step | What | |------|------| | ladder-001 | `GET /intro/auto-account.json` (personal account, GOA:0) | -| ladder-… | Mint explorer pool withdrawals: **zero**, then **random** in ranges (strictly increasing), then **pin at bank max** | -| ladder-… | wallet-cli accept-uri + explorer confirm when `selected` | -| ladder-… | Settle coins; **stop on first hard failure** | -| report | TSV + JSON with **ms_mint / ms_accept / ms_confirm / ms_settle / ms_total** | +| ladder-… | **load snapshot** before withdraws (host + bank/exchange/merchant mem/CPU) | +| ladder-… | **Phase A withdraw** (23): mint explorer pool — 0 → random → max−1 → max into **one cumulative wallet** (mids × `LADDER_WITHDRAW_SCALE`, default 1.5) | +| ladder-… | wallet-cli accept-uri + explorer confirm when `selected`; settle; coins snap | +| ladder-… | **Phase B pay** (23): same shape 0 → random → max−1 → max via merchant orders + `handle-uri --yes` | +| ladder-… | **load snapshot** after + delta + overall timing (withdraw + pay) | +| report | withdraw TSV + pay TSV + JSON | -Default path (`build_ladder`, strictly increasing): +Default amounts (`build_ladder_pair`, strictly increasing): ```text -[fixed] GOA:0 -[random] (LADDER_STEPS − 2) log-uniform amounts in (ε, max) -[fixed] GOA:4503599627370496 (LADDER_MAX_AMOUNT) +withdraw: [0] + random mids + [max−1] + [max] +pay: [0] + (withdraw_mid / scale) + [max−1] + [max] # always ≤ matching withdraw mid ``` -TSV column `range`: `pin:0` | `random` | `pin:max` (no legacy `LADDER_RANGES`). +TSV `range`: `pin:0` | `random` | `pin:max-1` | `pin:max`. + +Soft: absolute **max** mint/pay may `CEILING_REJECT` / skip (WARN). **max-1** is a hard pin. +`LADDER_PAY=0` skips phase B. | Env | Default | Meaning | |-----|---------|---------| diff --git a/scripts/taler-monitoring/taler-monitoring.sh b/scripts/taler-monitoring/taler-monitoring.sh index 9be5740..ff6586e 100755 --- a/scripts/taler-monitoring/taler-monitoring.sh +++ b/scripts/taler-monitoring/taler-monitoring.sh @@ -52,6 +52,8 @@ Env (same meaning): TALER_DOMAIN BANK_PUBLIC EXCHANGE_PUBLIC MERCHANT_PUBLIC EXPECT_CURRENCY SKIP_SSH=1 NO_COLOR=1 PERF_WARN_MS PERF_FAIL_MS (urls latency; default 8000 / 20000) + KOOPA_SSH KOOPA_SSH_FALLBACKS (default koopa → koopa-external) + METRICS_LOAD=0 skip host/container RAM/CPU probes (e2e/ladder/inside) EOF } @@ -135,7 +137,7 @@ export EXPECT_CURRENCY SKIP_SSH LOCAL_STACK TALER_DOMAIN_PROBE export WITHDRAW_AMT PAY_AMT CREDIT_AMT MERCHANT_INSTANCE export E2E_FAKE_INCOMING E2E_REMOTE E2E_VARIABLE E2E_ATM_MAX export E2E_WITHDRAW_VALUES E2E_PAY_VALUES -# Ladder: 0 + random mids + max (see check_goa_ladder.sh build_ladder). No LADDER_RANGES. +# Ladder: withdraw then pay — 0 + random mids + max-1 + max (see check_goa_ladder.sh). # Defaults so set -u export is safe when vars were never set by caller. : "${LADDER_STEPS:=23}" : "${LADDER_MIN_AMOUNT:=0.000001}" @@ -151,10 +153,14 @@ export E2E_WITHDRAW_VALUES E2E_PAY_VALUES : "${LADDER_HIGH_FROM:=1000000}" : "${LADDER_HIGH_RUNGS:=12}" : "${LADDER_LOAD:=1}" +: "${LADDER_PAY:=1}" +: "${LADDER_WITHDRAW_SCALE:=1.5}" +: "${LADDER_PAY_SETTLE_ROUNDS:=6}" export LADDER_STEPS LADDER_MIN_AMOUNT LADDER_CONFIRM_POLLS LADDER_MAX_RUNGS LADDER_TIMEOUT_S LADDER_REPORT_DIR export LADDER_SETTLE_ROUNDS LADDER_SETTLE_SLEEP EXP_PW_FILE EXP_USER export LADDER_MAX_AMOUNT LADDER_INCLUDE_ZERO LADDER_INCLUDE_MAX export LADDER_HIGH_FROM LADDER_HIGH_RUNGS LADDER_LOAD +export LADDER_PAY LADDER_WITHDRAW_SCALE LADDER_PAY_SETTLE_ROUNDS export TALER_DOMAIN_APPLIED=1 # Default phases