hernani/koopa-admin-log
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
koopa-admin-log/scripts/taler-monitoring/README.md
Hernâni Marques b65018cd0b
monitoring: final stats dashboard and ladder reserve matching 
Print a boxed FINAL STATISTICS block after e2e/ladder (coins, flow,
tendencies, perf, load) and match force-select reserve_pub to the current
withdrawal so cumulative ladder wallets do not reuse stale pubs.
2026-07-17 08:28:00 +02:00

7.3 KiB
Raw Blame History

taler-monitoring

Report for the GOA stack with clear severity tags and per-area test IDs:

Tag Meaning
[OK] check passed
[INFO] inside status (container, ports, log noise)
[WARN] degraded but maybe not fatal
[ERROR] component problem
[BLOCKER] withdraw/pay path cannot complete because of this

IDs: www-001, inside-001, versions-001, sanity-001, server-001, e2e-001 …
Catalog: TESTS.md.

[OK]    www-001  exchange /config https://exchange…/config
[BLOCKER] e2e-015  prereq: merchant HTTP 502

End of each phase: totals + list of BLOCKERS and ERRORS.

Commands

# Local GOA stack (may use SSH for inside/e2e)
./taler-monitoring.sh              # urls + inside + versions + e2e
./taler-monitoring.sh inside       # containers on koopa
./taler-monitoring.sh versions     # deb.taler.net + package versions vs trixie
./taler-monitoring.sh sanity
./taler-monitoring.sh e2e

# Other domains — public HTTPS only, never SSH
./taler-monitoring.sh -d taler.net
./taler-monitoring.sh --domain taler-ops.ch
./taler-monitoring.sh -d demo.taler.net urls
./taler-monitoring.sh taler.net    # bare domain = same as -d
Domain Bank Exchange Merchant Currency
hacktivism.ch (default) bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch GOA
taler.net / demo.taler.net bank.demo.taler.net exchange.demo.taler.net backend.demo.taler.net KUDOS
taler-ops.ch bank.* (probe) exchange.taler-ops.ch backend.* (probe) CHF
other bank.DOMAIN exchange.DOMAIN backend/taler/merchant (probe) any

SSH only for koopa (hacktivism.ch / -d koopa).

Other domains: never SSH. Optional e2e aborts cleanly on login/KYC.

E2E amounts (variable)

Local (koopa) Remote
ATM withdraw 20 · 50 · 100 · 200 10 · 20 · 50
Pay ladder 0.01 … 10 0.01 … 1 
./taler-monitoring.sh e2e
./taler-monitoring.sh -d taler.net urls e2e
# customize:
E2E_WITHDRAW_VALUES="20 50 100" E2E_PAY_VALUES="0.05 1 5" ./taler-monitoring.sh e2e
E2E_VARIABLE=0 WITHDRAW_AMT=GOA:50 PAY_AMT=GOA:1 ./taler-monitoring.sh e2e   # single fixed
# GOA shop catalog (local hacktivism): full list in E2E_SHOP_PRODUCTS; each run
#   shuffles and pays E2E_SHOP_PICK_N products (default 2).
#   E2E_SHOP_PRODUCTS lines: id|Product name|GOA:amount
#   E2E_SHOP_PICK_N=2
#   (landing QR = taler://pay-template/…/{id}; popup = live taler://pay after POST templates/{id})
# remote secrets:
#   E2E_BANK_ADMIN_PASS=…  E2E_MERCHANT_TOKEN=…

Phases

Phase What
inside SSH koopa: processes, postgres, local /config,/keys, wirewatch, recent log ERRORs
versions deb.taler.net reachable (InRelease/Packages/pool .deb); containers can reach it + have apt source; installed Taler packages vs trixie
sanity bank · exchange · merchant sections (public + server)
e2e account → credit → withdraw → wallet → confirm → order → pay
ladder GOA withdraw: fixed 0 + low bands + random high ranges + fixed max 
# package suite (default trixie on deb.taler.net)
TALER_APT_SUITE=trixie ./taler-monitoring.sh versions
TALER_PKG_BEHIND=error ./taler-monitoring.sh versions   # any behind = ERROR

E2E maps failures to blockers, e.g.:

  • bank-confirm HTTP 409 → wallet did not select exchange
  • no GOA balance → wirewatch / transfer
  • create order failed → merchant auth/instance
  • Alarm clock during pay → usually missing handle-uri --yes or wrong pay URI (must be …/instances/{inst}/{oid}/?c={token} from merchant taler_pay_uri)
  • insufficient balance → withdraw incomplete

Performance (urls phase)

Outside-in HTTPS RTT for bank / exchange / merchant critical paths. Each probe prints HTTP … · N ms on the [OK]/[WARN]/[ERROR] line; end of the block prints perf summary (n / min / p50 / avg / max).

Env Default Meaning
PERF_WARN_MS 8000 WARN if latency ≥ this
PERF_FAIL_MS 20000 ERROR if latency ≥ this
PERF_CURL_TIMEOUT 25 curl max seconds per probe 
PERF_WARN_MS=3000 PERF_FAIL_MS=10000 ./taler-monitoring.sh urls

Wallet coins (e2e · ladder)

At every relevant withdraw/pay step the wallet is snapped via advanced dump-coins. Amounts use exchange alt_unit_names (Kilo-GOA, Mega-GOA, …) with the base GOA value in parentheses:

[INFO]  coins after-ATM — amount_circ=2 Kilo-GOA (GOA:2000)
[INFO]  coins after-ATM — denoms_in_circ 1 Kilo-GOA (GOA:1000)×2 (=2 Kilo-GOA (GOA:2000))
== ladder · rung 12  GOA:1000000 · 1 Mega-GOA (GOA:1000000) ==

Also: coin counts (in circulation / spent), status histogram, Δ vs previous snap. History TSV: $METRICS_DIR/coins-history.tsv. Map loaded from ${EXCHANGE_PUBLIC}/config → currency_specification.alt_unit_names.

Final statistics dashboard (e2e · ladder)

At the end of e2e and ladder, a boxed FINAL STATISTICS block prints:

Section Content
Coins in circulation / spent counts + amounts + denoms (alt names)
Money flow withdrawn events/amount, spent (paid), net (wd − spent)
Tendency coin/amount trend over snaps (↑↓→), slope, last history rows
Performance min / p50 / avg / max per phase; first-half vs second-half latency
Load host loadavg + mem before/after; per-container RSS/CPU; tendencies

Successful withdraw/pay steps append to flow-withdrawn.txt / flow-spent.txt (ladder also uses its TSVs).

Load / memory (inside · e2e · ladder)

Host loadavg, RAM used/avail, and per-container RSS / CPU / block I/O (plus postgres DB sizes and role RSS: libeufin, exchange, merchant, …) are snapshotted via SSH whenever the stack is under load:

Phase When
inside end of container status
e2e before work · after ATM withdraws · after payments · after shop · e2e-end (+ delta)
ladder before first rung · after last rung (+ delta + timing overall) 
# LAN down? force WAN DNAT host
KOOPA_SSH=koopa-external ./taler-monitoring.sh inside
# or rely on automatic fallback (default KOOPA_SSH_FALLBACKS=koopa-external)

METRICS_LOAD=0 ./taler-monitoring.sh e2e   # skip host probes
LADDER_LOAD=0 ./taler-monitoring.sh ladder
Env Default Meaning
KOOPA_SSH koopa primary SSH host
KOOPA_SSH_FALLBACKS koopa-external tried if primary fails
METRICS_LOAD 1 0 = skip all host/container load probes
LADDER_LOAD 1 0 = skip load in ladder only
LADDER_PAY 1 0 = skip payment ladder after withdraws
LADDER_WITHDRAW_SCALE 1.5 withdraw mids / pay mids ratio (funding headroom)
LADDER_STEPS 23 rungs for both withdraw and pay (0 + mids + max−1 + max)
METRICS_LOAD_SSH_TIMEOUT 90 seconds for remote load python

Needs

  • SSH koopa or koopa-external (inside / load / sanity server bits)
  • secrets under koopa-admin-secrets/... for e2e
  • taler-wallet-cli for e2e