b65018cd0b
Print a boxed FINAL STATISTICS block after e2e/ladder (coins, flow, tendencies, perf, load) and match force-select reserve_pub to the current withdrawal so cumulative ladder wallets do not reuse stale pubs.
179 lines
7.3 KiB
Markdown
179 lines
7.3 KiB
Markdown
# taler-monitoring
|
||
|
||
Report for the **GOA** stack with clear severity tags and **per-area test IDs**:
|
||
|
||
| Tag | Meaning |
|
||
|-----|---------|
|
||
| `[OK]` | check passed |
|
||
| `[INFO]` | inside status (container, ports, log noise) |
|
||
| `[WARN]` | degraded but maybe not fatal |
|
||
| `[ERROR]` | component problem |
|
||
| `[BLOCKER]` | **withdraw/pay path cannot complete** because of this |
|
||
|
||
IDs: **`www-001`**, **`inside-001`**, **`versions-001`**, **`sanity-001`**, **`server-001`**, **`e2e-001`** …
|
||
Catalog: **[TESTS.md](./TESTS.md)**.
|
||
|
||
```text
|
||
[OK] www-001 exchange /config https://exchange…/config
|
||
[BLOCKER] e2e-015 prereq: merchant HTTP 502
|
||
```
|
||
|
||
End of each phase: totals + list of **BLOCKERS** and **ERRORS**.
|
||
|
||
## Commands
|
||
|
||
```bash
|
||
# Local GOA stack (may use SSH for inside/e2e)
|
||
./taler-monitoring.sh # urls + inside + versions + e2e
|
||
./taler-monitoring.sh inside # containers on koopa
|
||
./taler-monitoring.sh versions # deb.taler.net + package versions vs trixie
|
||
./taler-monitoring.sh sanity
|
||
./taler-monitoring.sh e2e
|
||
|
||
# Other domains — public HTTPS only, never SSH
|
||
./taler-monitoring.sh -d taler.net
|
||
./taler-monitoring.sh --domain taler-ops.ch
|
||
./taler-monitoring.sh -d demo.taler.net urls
|
||
./taler-monitoring.sh taler.net # bare domain = same as -d
|
||
```
|
||
|
||
| Domain | Bank | Exchange | Merchant | Currency |
|
||
|--------|------|----------|----------|----------|
|
||
| `hacktivism.ch` (default) | bank.hacktivism.ch | exchange.hacktivism.ch | taler.hacktivism.ch | GOA |
|
||
| `taler.net` / `demo.taler.net` | bank.demo.taler.net | exchange.demo.taler.net | backend.demo.taler.net | KUDOS |
|
||
| `taler-ops.ch` | bank.* (probe) | exchange.taler-ops.ch | backend.* (probe) | CHF |
|
||
| other | bank.DOMAIN | exchange.DOMAIN | backend/taler/merchant (probe) | any |
|
||
|
||
**SSH only for koopa** (`hacktivism.ch` / `-d koopa`).
|
||
|
||
Other domains: never SSH. Optional **e2e** aborts cleanly on login/KYC.
|
||
|
||
### E2E amounts (variable)
|
||
|
||
| | Local (koopa) | Remote |
|
||
|--|---------------|--------|
|
||
| **ATM withdraw** | 20 · 50 · 100 · 200 | 10 · 20 · 50 |
|
||
| **Pay ladder** | 0.01 … 10 | 0.01 … 1 |
|
||
|
||
```bash
|
||
./taler-monitoring.sh e2e
|
||
./taler-monitoring.sh -d taler.net urls e2e
|
||
# customize:
|
||
E2E_WITHDRAW_VALUES="20 50 100" E2E_PAY_VALUES="0.05 1 5" ./taler-monitoring.sh e2e
|
||
E2E_VARIABLE=0 WITHDRAW_AMT=GOA:50 PAY_AMT=GOA:1 ./taler-monitoring.sh e2e # single fixed
|
||
# GOA shop catalog (local hacktivism): full list in E2E_SHOP_PRODUCTS; each run
|
||
# shuffles and pays E2E_SHOP_PICK_N products (default 2).
|
||
# E2E_SHOP_PRODUCTS lines: id|Product name|GOA:amount
|
||
# E2E_SHOP_PICK_N=2
|
||
# (landing QR = taler://pay-template/…/{id}; popup = live taler://pay after POST templates/{id})
|
||
# remote secrets:
|
||
# E2E_BANK_ADMIN_PASS=… E2E_MERCHANT_TOKEN=…
|
||
```
|
||
|
||
## Phases
|
||
|
||
| Phase | What |
|
||
|-------|------|
|
||
| **inside** | SSH koopa: processes, postgres, local /config,/keys, wirewatch, recent log ERRORs |
|
||
| **versions** | `deb.taler.net` reachable (InRelease/Packages/pool `.deb`); containers can reach it + have apt source; installed Taler packages vs **trixie** |
|
||
| **sanity** | bank · exchange · merchant sections (public + server) |
|
||
| **e2e** | account → credit → withdraw → wallet → confirm → order → pay |
|
||
| **ladder** | GOA withdraw: fixed **0** + low bands + **random high ranges** + fixed **max** |
|
||
|
||
```bash
|
||
# package suite (default trixie on deb.taler.net)
|
||
TALER_APT_SUITE=trixie ./taler-monitoring.sh versions
|
||
TALER_PKG_BEHIND=error ./taler-monitoring.sh versions # any behind = ERROR
|
||
```
|
||
|
||
E2E maps failures to blockers, e.g.:
|
||
|
||
- `bank-confirm HTTP 409` → wallet did not select exchange
|
||
- `no GOA balance` → wirewatch / transfer
|
||
- `create order failed` → merchant auth/instance
|
||
- `Alarm clock` during pay → usually missing `handle-uri --yes` or wrong pay URI (must be `…/instances/{inst}/{oid}/?c={token}` from merchant `taler_pay_uri`)
|
||
- `insufficient balance` → withdraw incomplete
|
||
|
||
## Performance (urls phase)
|
||
|
||
Outside-in HTTPS RTT for bank / exchange / merchant critical paths. Each probe prints
|
||
`HTTP … · N ms` on the `[OK]`/`[WARN]`/`[ERROR]` line; end of the block prints
|
||
`perf summary` (n / min / p50 / avg / max).
|
||
|
||
| Env | Default | Meaning |
|
||
|-----|---------|---------|
|
||
| `PERF_WARN_MS` | `8000` | WARN if latency ≥ this |
|
||
| `PERF_FAIL_MS` | `20000` | ERROR if latency ≥ this |
|
||
| `PERF_CURL_TIMEOUT` | `25` | curl max seconds per probe |
|
||
|
||
```bash
|
||
PERF_WARN_MS=3000 PERF_FAIL_MS=10000 ./taler-monitoring.sh urls
|
||
```
|
||
|
||
## Wallet coins (e2e · ladder)
|
||
|
||
At every relevant withdraw/pay step the wallet is snapped via `advanced dump-coins`.
|
||
Amounts use exchange **`alt_unit_names`** (Kilo-GOA, Mega-GOA, …) with the base
|
||
**GOA value in parentheses**:
|
||
|
||
```text
|
||
[INFO] coins after-ATM — amount_circ=2 Kilo-GOA (GOA:2000)
|
||
[INFO] coins after-ATM — denoms_in_circ 1 Kilo-GOA (GOA:1000)×2 (=2 Kilo-GOA (GOA:2000))
|
||
== ladder · rung 12 GOA:1000000 · 1 Mega-GOA (GOA:1000000) ==
|
||
```
|
||
|
||
Also: coin counts (in circulation / spent), status histogram, Δ vs previous snap.
|
||
History TSV: `$METRICS_DIR/coins-history.tsv`. Map loaded from
|
||
`${EXCHANGE_PUBLIC}/config` → `currency_specification.alt_unit_names`.
|
||
|
||
### Final statistics dashboard (e2e · ladder)
|
||
|
||
At the end of **e2e** and **ladder**, a boxed **FINAL STATISTICS** block prints:
|
||
|
||
| Section | Content |
|
||
|---------|---------|
|
||
| **Coins** | in circulation / spent counts + amounts + denoms (alt names) |
|
||
| **Money flow** | withdrawn events/amount, spent (paid), net (wd − spent) |
|
||
| **Tendency** | coin/amount trend over snaps (↑↓→), slope, last history rows |
|
||
| **Performance** | min / p50 / avg / max per phase; first-half vs second-half latency |
|
||
| **Load** | host loadavg + mem before/after; per-container RSS/CPU; tendencies |
|
||
|
||
Successful withdraw/pay steps append to `flow-withdrawn.txt` / `flow-spent.txt` (ladder also uses its TSVs).
|
||
|
||
## Load / memory (inside · e2e · ladder)
|
||
|
||
Host **loadavg**, **RAM used/avail**, and per-container **RSS / CPU / block I/O**
|
||
(plus postgres DB sizes and role RSS: libeufin, exchange, merchant, …) are
|
||
snapshotted via SSH whenever the stack is under load:
|
||
|
||
| Phase | When |
|
||
|-------|------|
|
||
| **inside** | end of container status |
|
||
| **e2e** | before work · after ATM withdraws · after payments · after shop · e2e-end (+ delta) |
|
||
| **ladder** | before first rung · after last rung (+ delta + timing overall) |
|
||
|
||
```bash
|
||
# LAN down? force WAN DNAT host
|
||
KOOPA_SSH=koopa-external ./taler-monitoring.sh inside
|
||
# or rely on automatic fallback (default KOOPA_SSH_FALLBACKS=koopa-external)
|
||
|
||
METRICS_LOAD=0 ./taler-monitoring.sh e2e # skip host probes
|
||
LADDER_LOAD=0 ./taler-monitoring.sh ladder
|
||
```
|
||
|
||
| Env | Default | Meaning |
|
||
|-----|---------|---------|
|
||
| `KOOPA_SSH` | `koopa` | primary SSH host |
|
||
| `KOOPA_SSH_FALLBACKS` | `koopa-external` | tried if primary fails |
|
||
| `METRICS_LOAD` | `1` | `0` = skip all host/container load probes |
|
||
| `LADDER_LOAD` | `1` | `0` = skip load in ladder only |
|
||
| `LADDER_PAY` | `1` | `0` = skip payment ladder after withdraws |
|
||
| `LADDER_WITHDRAW_SCALE` | `1.5` | withdraw mids / pay mids ratio (funding headroom) |
|
||
| `LADDER_STEPS` | `23` | rungs for both withdraw and pay (0 + mids + max−1 + max) |
|
||
| `METRICS_LOAD_SSH_TIMEOUT` | `90` | seconds for remote load python |
|
||
|
||
## Needs
|
||
|
||
- SSH `koopa` or `koopa-external` (inside / load / sanity server bits)
|
||
- secrets under `koopa-admin-secrets/...` for e2e
|
||
- `taler-wallet-cli` for e2e
|