koopa-admin-log/scripts/taler-monitoring

taler-monitoring

Report for the GOA stack with boxed severity badges and grouped test IDs area.group-NN (e.g. www.exchange-01, e2e.pay-03):

Tag	Meaning
┌ OK ┐	check passed
┌ INFO ┐	status / note
┌ WARN ┐	degraded but maybe not fatal
┌ ERROR ┐	component problem
┌ BLOCKER ┐	withdraw/pay path cannot complete because of this

IDs reset per group inside an area — so “too many www tests” become www.exchange-*, www.bank-*, www.landing-*, etc.
Catalog: TESTS.md.

┌ OK    ┐ www.exchange-01  exchange /config · HTTP 200
┌ BLOCKER┐ e2e.prereq-02   merchant HTTP 502

End of each phase: totals + list of BLOCKERS and ERRORS (ids included).

Secrets (local GOA e2e)

check_e2e.sh loads bank/merchant passwords from koopa-admin-secrets (not this repo). Search order:

1. SECRETS_ROOT → …/taler-bank/bank-admin-password.txt
2. KOOPA_ADMIN_SECRETS → $KOOPA_ADMIN_SECRETS/koopa/host-root/…
3. Sibling of this checkout: ../koopa-admin-secrets/koopa/host-root
4. $HOME/taler/src/koopa-admin-secrets/koopa/host-root (common laptop layout)
5. $HOME/.config/taler-landing/bank-admin-password.txt
6. SSH koopa: /root/bank-admin-password.txt or podman bank container

If you keep admin-log at ~/koopa-admin-log but secrets under ~/taler/src/…:

export SECRETS_ROOT="$HOME/taler/src/koopa-admin-secrets/koopa/host-root"
# or: export KOOPA_ADMIN_SECRETS="$HOME/taler/src/koopa-admin-secrets"
./taler-monitoring.sh e2e

Or one-shot: E2E_BANK_ADMIN_PASS=… E2E_MERCHANT_TOKEN=… ./taler-monitoring.sh e2e

Commands

# Local GOA stack (may use SSH for inside/e2e)
./taler-monitoring.sh              # urls + inside + versions + e2e
./taler-monitoring.sh inside       # containers on koopa
./taler-monitoring.sh versions     # deb.taler.net + package versions vs trixie
./taler-monitoring.sh sanity
./taler-monitoring.sh e2e


# Other domains — public HTTPS only, never SSH
./taler-monitoring.sh -d taler.net
./taler-monitoring.sh --domain taler-ops.ch
./taler-monitoring.sh -d my.taler-ops.ch urls
./taler-monitoring.sh -d demo.taler.net urls
./taler-monitoring.sh taler.net    # bare domain = same as -d

# Explicit endpoints (any mix; override profile or skip -d)
./taler-monitoring.sh --bank https://bank.demo.taler.net \
  --exchange https://exchange.demo.taler.net \
  --merchant https://backend.demo.taler.net --currency KUDOS urls

# Landing stats.json freshness (outside-in; age via generated_at_unix)
STATS_STALE_SECS=900 STATS_FAIL_SECS=3600 ./taler-monitoring.sh -d stage.lefrancpaysan.ch urls

Landing stats.json (www.stats)

Public GET …/intro/stats.json must expose timestamps so monitoring can tell whether the collector is still updating:

Field	Role
generated_at_unix	epoch seconds — preferred for age = now − unix
generated_at	ISO-8601 (fallback if unix missing)
generated_at_human	UI “updated” / footer string

Also checks display payloads (bank-shape withdraws / accounts, or exchange-db / merchant-db counters). Soft WARN if age > STATS_STALE_SECS (default 900); ERROR if age > STATS_FAIL_SECS (default 3600; set 0 to disable). Stage shared feed: same gen_unix on bank + exchange + monnaies.

Domain profiles (domains.conf)

Each monitored stack must say what is bank, exchange, and merchant-backend.

Edit domains.conf (or set TALER_DOMAINS_CONF=):

# name   bank   exchange   merchant-backend   currency   local   [canonical]
taler-ops.ch  bank.taler-ops.ch  exchange.taler-ops.ch  my.taler-ops.ch  CHF  0

Field	Meaning
name	-d NAME / TALER_DOMAIN match (add alias lines as needed)
bank	libeufin/bank public host or https://…
exchange	exchange public host or URL
merchant	merchant backend (SPA/API), e.g. my.taler-ops.ch
currency	GOA / KUDOS / CHF / …
local	1 = koopa SSH stack; 0 = public only
landing	1 = check /intro landings + assets; 0 = skip (TOPS / mytops)
canonical	optional label after alias (e.g. my.taler-ops.ch → taler-ops.ch)

Then:

./taler-monitoring.sh -d taler-ops.ch urls

Overrides (always win after profile load):

./taler-monitoring.sh -d taler-ops.ch \
  --merchant https://my.taler-ops.ch \
  --exchange https://exchange.taler-ops.ch \
  --bank https://bank.taler-ops.ch urls

Helpers in lib.sh: set_taler_stack, load_domain_profile, apply_taler_domain.

Built-in profiles (see domains.conf)

Domain / alias	Bank	Exchange	Merchant	Currency
hacktivism.ch / koopa	bank.hacktivism.ch	exchange.hacktivism.ch	taler.hacktivism.ch	GOA
taler.net / demo.taler.net	bank.demo.taler.net	exchange.demo.taler.net	backend.demo.taler.net	KUDOS
test.taler.net	bank.test…	exchange.test…	backend.test…	TESTKUDOS · no landings
lefrancpaysan.ch	bank.…	exchange.…	monnaie.… (merchant)	report-only · no landings
taler-ops.ch / my.taler-ops.ch	bank.taler-ops.ch	exchange.taler-ops.ch	my.taler-ops.ch	CHF · no landings
stage.taler-ops.ch	bank.stage…	exchange.stage…	my.stage…	CHF · no landings
unknown	bank.DOMAIN	exchange.DOMAIN	my.DOMAIN then probe	any · no landings

Landings: only GOA (and optionally demo) use public /intro pages. taler-ops.ch does not — CHECK_LANDING=0 in the profile skips intro crawl, shop assets, and demo-withdraw checks.

SSH only for koopa (hacktivism.ch / -d koopa).

Other domains: never SSH. Optional e2e aborts cleanly on login/KYC.

E2E amounts (variable)

	Local (koopa)	Remote
ATM withdraw	20 · 50 · 100 · 200 · 4200 (paivana)	10 · 20 · 50
Pay ladder	0.01 … 10	0.01 … 1
Paivana	HTTP paywall + template pay GOA:4200 (goa-shop / paivana)	skipped

./taler-monitoring.sh e2e
./taler-monitoring.sh -d taler.net urls e2e
# customize:
E2E_WITHDRAW_VALUES="20 50 100" E2E_PAY_VALUES="0.05 1 5" ./taler-monitoring.sh e2e
E2E_VARIABLE=0 WITHDRAW_AMT=GOA:50 PAY_AMT=GOA:1 ./taler-monitoring.sh e2e   # single fixed
# GOA shop catalog (local hacktivism): full list in E2E_SHOP_PRODUCTS; each run
#   shuffles and pays E2E_SHOP_PICK_N products (default 2).
#   E2E_SHOP_PRODUCTS lines: id|Product name|GOA:amount
#   E2E_SHOP_PICK_N=2
#   (landing QR = taler://pay-template/…/{id}; popup = live taler://pay after POST templates/{id})
# Paivana paywall (local GOA only):
#   E2E_PAIVANA=1 (default)  PAIVANA_PUBLIC=https://paivana.hacktivism.ch
#   E2E_PAIVANA_TEMPLATE=paivana  E2E_PAIVANA_AMOUNT=GOA:4200  E2E_PAIVANA_INSTANCE=goa-shop
#   E2E_PAIVANA=0            # skip
# remote secrets:
#   E2E_BANK_ADMIN_PASS=…  E2E_MERCHANT_TOKEN=…

Phases

Phase	What
inside	SSH koopa: processes, postgres, local /config,/keys, wirewatch, recent log ERRORs
versions	deb.taler.net reachable (InRelease/Packages/pool .deb); containers can reach it + have apt source; installed Taler packages vs trixie
sanity	bank · exchange · merchant sections (public + server)
e2e	account → credit → withdraw → wallet → confirm → order → pay
ladder	GOA withdraw: fixed 0 + low bands + random high ranges + fixed max

# package suite (default trixie on deb.taler.net)
TALER_APT_SUITE=trixie ./taler-monitoring.sh versions
TALER_PKG_BEHIND=error ./taler-monitoring.sh versions   # any behind = ERROR

E2E maps failures to blockers, e.g.:

• bank-confirm HTTP 409 → wallet did not select exchange
• no GOA balance → wirewatch / transfer
• create order failed → merchant auth/instance
• Alarm clock during pay → usually missing handle-uri --yes or wrong pay URI (must be …/instances/{inst}/{oid}/?c={token} from merchant taler_pay_uri)
• insufficient balance → withdraw incomplete

Performance (urls phase)

Outside-in HTTPS RTT for bank / exchange / merchant critical paths. Each probe prints HTTP … · N ms on the [OK]/[WARN]/[ERROR] line; end of the block prints perf summary (n / min / p50 / avg / max).

Env	Default	Meaning
PERF_WARN_MS	8000	WARN if latency ≥ this
PERF_FAIL_MS	20000	ERROR if latency ≥ this
PERF_CURL_TIMEOUT	25	curl max seconds per probe

PERF_WARN_MS=3000 PERF_FAIL_MS=10000 ./taler-monitoring.sh urls

Wallet coins (e2e · ladder)

At every relevant withdraw/pay step the wallet is snapped via advanced dump-coins. Amounts use exchange alt_unit_names (Kilo-GOA, Mega-GOA, …) with the base GOA value in parentheses:

[INFO]  coins after-ATM — amount_circ=2 Kilo-GOA (GOA:2000)
[INFO]  coins after-ATM — denoms_in_circ 1 Kilo-GOA (GOA:1000)×2 (=2 Kilo-GOA (GOA:2000))
== ladder · rung 12  GOA:1000000 · 1 Mega-GOA (GOA:1000000) ==

Also: coin counts (in circulation / spent), status histogram, Δ vs previous snap. History TSV: $METRICS_DIR/coins-history.tsv. Map loaded from ${EXCHANGE_PUBLIC}/config → currency_specification.alt_unit_names.

Final statistics dashboard (e2e · ladder)

At the end of e2e and ladder, a boxed FINAL STATISTICS block prints:

Section	Content
Coins	in circulation / spent counts + amounts + denoms (alt names)
Money flow	withdrawn events/amount, spent (paid), net (wd − spent)
Tendency	coin/amount trend over snaps (↑↓→), slope, last history rows
Performance	min / p50 / avg / max per phase; first-half vs second-half latency
Load	host loadavg + mem before/after; per-container RSS/CPU; tendencies

Successful withdraw/pay steps append to flow-withdrawn.txt / flow-spent.txt (ladder also uses its TSVs).

Load / memory (inside · e2e · ladder)

Host loadavg, RAM used/avail, and per-container RSS / CPU / block I/O (plus postgres DB sizes and role RSS: libeufin, exchange, merchant, …) are snapshotted via SSH whenever the stack is under load:

Phase	When
inside	end of container status
e2e	before work · after ATM withdraws · after payments · after shop · e2e-end (+ delta)
ladder	before first rung · after last rung (+ delta + timing overall)

# LAN down? force WAN DNAT host
KOOPA_SSH=koopa-external ./taler-monitoring.sh inside
# or rely on automatic fallback (default KOOPA_SSH_FALLBACKS=koopa-external)

METRICS_LOAD=0 ./taler-monitoring.sh e2e   # skip host probes
LADDER_LOAD=0 ./taler-monitoring.sh ladder

Env	Default	Meaning
KOOPA_SSH	koopa	primary SSH host
KOOPA_SSH_FALLBACKS	koopa-external	tried if primary fails
METRICS_LOAD	1	0 = skip all host/container load probes
LADDER_LOAD	1	0 = skip load in ladder only
LADDER_PAY	1	0 = skip payment ladder after withdraws
LADDER_WITHDRAW_SCALE	1.5	withdraw mids / pay mids ratio (funding headroom)
LADDER_STEPS	23	rungs for both withdraw and pay (0 + mids + max−1 + max)
METRICS_LOAD_SSH_TIMEOUT	90	seconds for remote load python

Needs

• SSH koopa or koopa-external (inside / load / sanity server bits)
• secrets under koopa-admin-secrets/... for e2e
• taler-wallet-cli for e2e