taler-monitoring/README.md

taler-monitoring

Checks for GNU Taler stacks (bank, exchange, merchant; optional surface inventory).

Mode	What
Outside-only	Public HTTPS from anywhere — no SSH, no host-agent. Phases such as urls, monpages, surface, …
Inside / local	Where you operate the stack: podman/SSH (inside, versions, e2e, ladder, …)
Host-agent + HTML	Optional: timers run the suite and publish sticky-bar HTML (see simplified page layout below)

Outside-only example: ./taler-monitoring.sh -d demo.taler.net urls
Surface (ecosystem + mail + mattermost + versions): ./taler-monitoring.sh surface mattermost mail versions
Full local GOA example: ./taler-monitoring.sh -d hacktivism.ch urls inside versions

Repo: https://git.hacktivism.ch/hernani/taler-monitoring · tree: ~/src/taler-monitoring
Version: ./taler-monitoring.sh --ver (or --version / -V)

Monitoring pages — public URL families (only these three)

taler-monitoring is the suite name (git / CLI), not a public URL path.
There is no mon page at /taler-monitoring or /taler-monitoring/.

Exactly three first-class public path families (each must work bare and with trailing slash; reverse-proxy should redir bare → slash). Error companions use …_err when the last run failed:

Family	Paths (bare + slash)	Where	Purpose
monitoring	/monitoring · /monitoring/ · optional /monitoring_err · /monitoring_err/	Per landing host (GOA bank/exchange/taler · FP · mytops stage, …)	Stack report for that host
taler-monitoring-surface	/taler-monitoring-surface · /…/ · optional …_err	only taler.hacktivism.ch	Ecosystem surface (ports, nmap, mail, mattermost, versions)
taler-monitoring-aptdeploy	/taler-monitoring-aptdeploy · /…/ · optional …_err	only taler.hacktivism.ch	apt-src merchant deploy tests (podman on koopa)

Not public mon pages: /taler-monitoring-mail*, /taler-monitoring-mattermost*, bare /taler-monitoring — mail/mm are folded into surface.

Related:

• Phase monpages: obligatory ERROR if a listed page is missing, bootstrap stub, not suite HTML, or content markers fail. Bare paths that fall through to merchant code 21 are ERROR (not “OK because slash works”).
• Publish: staging ~/monitoring-sites-staging → DEPLOY_WWW_ROOT + reverse-proxy handles for only the three families above.

./taler-monitoring.sh monpages
./taler-monitoring.sh -d hacktivism.ch monpages

Host packages (koopa / runners): DEPENDENCIES.md.
Host-agents (GOA + FP; shared report pipeline): host-agent/README.md.
Public HTML site-gen (same timeout/log/HTML defaults): site-gen/README.md.
FP stage host-agent: host-agent/run-fp-stage-monitoring.sh + host-agent/env/stagepaysan.env.example (see host-agent/README.md).

Report for the GOA stack with boxed severity badges and grouped test IDs area.group-NN (e.g. www.exchange-01, e2e.pay-03):

Tag	Meaning
┌ OK ┐	check passed
┌ INFO ┐	status / note
┌ WARN ┐	degraded but maybe not fatal
┌ ERROR ┐	component problem
┌ BLOCKER ┐	withdraw/pay path cannot complete because of this

IDs reset per group inside an area — so “too many www tests” become www.exchange-*, www.bank-*, www.landing-*, etc.
Catalog: TESTS.md.
Git / tags / suite tree: VERSIONS.md.
Optional / deprecated tooling: meta/ (firecuda, legacy site-gen, old mail/mm units).

GUI Android notes: android-test/GUI-AUTOMATION-NOTES.md.
CLI / wallet-cli recurring issues: CLI-AUTOMATION-NOTES.md.

┌ OK    ┐ www.exchange-01  exchange /config · HTTP 200
┌ BLOCKER┐ e2e.prereq-02   merchant HTTP 502

End of each phase: totals + list of BLOCKERS and ERRORS (ids included).

Secrets (local GOA e2e)

Do not commit passwords. Suite tree: ~/src/taler-monitoring · host ops (optional): sibling ~/src/koopa/koopa-admin-secrets.

Local file (recommended on laptop)

cd ~/src/taler-monitoring
cp secrets.env.example secrets.env   # gitignored
$EDITOR secrets.env
./taler-monitoring.sh e2e

See secrets.env.example for every variable that may appear (admin pass, merchant token, SECRETS_ROOT, progress knobs).

Search order

1. Already-exported env (E2E_BANK_ADMIN_PASS, …)
2. secrets.env next to this suite (MONITORING_SECRETS_ENV= override)
3. SECRETS_ROOT / KOOPA_ADMIN_SECRETS file tree
4. Sibling ../koopa-admin-secrets/koopa/host-root
5. $HOME/src/koopa/koopa-admin-secrets/koopa/host-root
6. $HOME/.config/taler-landing/bank-admin-password.txt
7. SSH koopa /root/… or bank container

Line IDs + progress

Each check prints global and grouped ids:

┌ OK    ┐ #014  www.exchange-03  exchange /config · HTTP 200
┌ WARN  ┐ #015  www.stats-02     stats age soft  · yellow badge (not red)
┌  42% ┐ ████████░░░░░░░░░░░░░░░░░░░░  14/55

• #014 — run-wide counter (whole ./taler-monitoring.sh invocation)
• www.exchange-03 — area.group-NN
• Colours (left badge + label):
  • OK green · INFO blue/cyan · WARN yellow (never red)
  • ERROR red · BLOCKER magenta
• Progress: badge shows percent (not “PROG”); bar fills left → right; done/total on the right. Estimate auto-grows if short. Override: PROGRESS_TOTAL=N / PROGRESS_OFF=1
• QR (urls): harvest taler:// / payto:// (+ app store data-qr-url) from landings and demo-withdraw.json / auto-account.json → validate form → qrencode PNG → zbarimg decode must match. Requires packages qrencode + zbar-tools.

Commands

# Local GOA stack (may use SSH for inside/e2e)
./taler-monitoring.sh              # urls + inside + versions + e2e
./taler-monitoring.sh inside       # containers on koopa
./taler-monitoring.sh versions     # deb.taler.net + package versions vs trixie
./taler-monitoring.sh sanity
./taler-monitoring.sh e2e

# FrancPaysan STAGE (TESTPAYSAN) — like hacktivism breadth:
#   default phases: urls + inside + versions + e2e
./taler-monitoring.sh -d stage.lefrancpaysan.ch
./taler-monitoring.sh -d stage.lefrancpaysan.ch inside   # stagepaysan only
# inside: low-priv SSH francpaysan-stage-user (User stagepaysan) → podman
#   containers stage-lfp-{bank,exchange-ansible,merchant} · ports 9032/9031/9030
# stats: outside-in public …/intro/stats.json (age via generated_at_unix)
# Admin credit for e2e: E2E_BANK_ADMIN_PASS=… or stage host secrets
#   (bank-admin-password.txt under /mnt/data/stagepaysan/bank/secrets/)
# Override: INSIDE_SSH=… INSIDE_PROFILE=stage-lfp

# Other domains — public HTTPS only (pass e2e/inside explicitly)
./taler-monitoring.sh -d taler.net
./taler-monitoring.sh --domain taler-ops.ch
./taler-monitoring.sh -d my.taler-ops.ch urls
./taler-monitoring.sh -d demo.taler.net urls
./taler-monitoring.sh taler.net    # bare domain = same as -d

# Explicit endpoints (any mix; override profile or skip -d)
./taler-monitoring.sh --bank https://bank.demo.taler.net \
  --exchange https://exchange.demo.taler.net \
  --merchant https://backend.demo.taler.net --currency KUDOS urls

# Landing stats.json freshness (outside-in; age via generated_at_unix)
STATS_STALE_SECS=900 STATS_FAIL_SECS=3600 ./taler-monitoring.sh -d stage.lefrancpaysan.ch urls

# --- Full load (GOA / hacktivism) ---
# all  = urls + inside + versions + sanity + e2e
# full = all + server + ladder + auth401  (long; needs secrets.env)
./taler-monitoring.sh -d hacktivism.ch full
./taler-monitoring.sh -d hacktivism.ch urls inside versions sanity server e2e ladder auth401

# SPA pin after selfbuild-webui.sh (container):
EXPECT_WEBUI_VERSION=1.6.11 EXPECT_WEBUI_OVERLAY=selfbuild-v1.6.11 \
  WEBUI_OVERLAY_DENY='c778af04|master-11340' \
  ./taler-monitoring.sh -d hacktivism.ch urls

# Only 401 matrix (may create throwaway instance):
./taler-monitoring.sh -d hacktivism.ch auth401

# Stage FrancPaysan — durable mon401 account (password in francpaysan-secrets):
./taler-monitoring.sh -d stage.monnaie.lefrancpaysan.ch auth401
# secrets: ~/francpaysan-secrets/stage/instances/mon401.password
#           ~/francpaysan-secrets/stage/default-instance-token.txt
# force throwaway create instead: AUTH401_PREFER_DURABLE=0 AUTH401_CREATE=1 …

Landing stats.json (www.stats)

Public GET …/intro/stats.json must expose timestamps so monitoring can tell whether the collector is still updating:

Field	Role
generated_at_unix	epoch seconds — preferred for age = now − unix
generated_at	ISO-8601 (fallback if unix missing)
generated_at_human	UI “updated” / footer string

Also checks display payloads (bank-shape withdraws / accounts, or exchange-db / merchant-db counters). Soft WARN if age > STATS_STALE_SECS (default 900); ERROR if age > STATS_FAIL_SECS (default 3600; set 0 to disable). Stage shared feed: same gen_unix on bank + exchange + monnaies.

Domain profiles (domains.conf)

Each monitored stack must say what is bank, exchange, and merchant-backend.

Edit domains.conf (or set TALER_DOMAINS_CONF=):

# name  bank  exchange  merchant  currency  local  landing  lang  locale  [canonical]
taler-ops.ch  bank.…  exchange.…  my.…  CHF  0  0  en  de-CH
stage.lefrancpaysan.ch  stage.bank.…  stage.exchange.…  stage.monnaie.…  TESTPAYSAN  0  1  fr  fr-CH

Field	Meaning
name	-d NAME / TALER_DOMAIN match (add alias lines as needed)
bank	libeufin/bank public host or https://…
exchange	exchange public host or URL
merchant	merchant backend (SPA/API), e.g. my.taler-ops.ch
currency	GOA / KUDOS / CHF / …
local	1 = koopa SSH stack; 0 = public only
landing	1 = check /intro landings + assets; 0 = skip (TOPS / mytops)
lang	UI language en / fr / de when TALER_MON_LANG_SET≠1. Stock profiles use en or fr only — de is never a default (opt-in via --lang de or an explicit profile edit).
locale	Regional l10n (BCP 47): fr-CH for FrancPaysan (Suisse romande); de-CH for other Swiss stacks. Aliases ch-FR / ch-DE accepted. Not the same as UI lang.
canonical	optional label after alias (e.g. my.taler-ops.ch → taler-ops.ch)

Workstation TALER_MON_LANG=en does not override a profile with lang=fr. Force English on FP only with --lang en / TALER_MON_LANG_SET=1.

Then:

./taler-monitoring.sh -d taler-ops.ch urls

Overrides (always win after profile load):

./taler-monitoring.sh -d taler-ops.ch \
  --merchant https://my.taler-ops.ch \
  --exchange https://exchange.taler-ops.ch \
  --bank https://bank.taler-ops.ch urls

Helpers in lib.sh: set_taler_stack, load_domain_profile, apply_taler_domain.

Built-in profiles (see domains.conf)

Domain / alias	Bank	Exchange	Merchant	Currency
hacktivism.ch / koopa	bank.hacktivism.ch	exchange.hacktivism.ch	taler.hacktivism.ch	GOA
taler.net / demo.taler.net	bank.demo.taler.net	exchange.demo.taler.net	backend.demo.taler.net	KUDOS
test.taler.net	bank.test…	exchange.test…	backend.test…	TESTKUDOS · no landings
lefrancpaysan.ch	bank.…	exchange.…	monnaie.… (merchant)	report-only · no landings
stage.lefrancpaysan.ch	stage.bank.…	stage.exchange.…	stage.monnaie.…	TESTPAYSAN · landings · inside via stagepaysan
taler-ops.ch / my.taler-ops.ch	bank.taler-ops.ch	exchange.taler-ops.ch	my.taler-ops.ch	CHF · no landings
stage.taler-ops.ch	bank.stage…	exchange.stage…	my.stage…	CHF · no landings
unknown	bank.DOMAIN	exchange.DOMAIN	my.DOMAIN then probe	any · no landings

Landings: GOA, stage LFP (TESTPAYSAN), and optionally demo use public /intro pages. taler-ops.ch does not — CHECK_LANDING=0 skips intro crawl.

SSH:

Stack	Host alias	User	Scope
koopa / hacktivism	koopa	hernani	full host + containers
stage LFP	francpaysan-stage-user	stagepaysan	own podman only (no sudo / no prod data)

Other domains: never SSH. Optional e2e aborts cleanly on login/KYC.

E2E amounts (variable)

	Local (koopa)	Remote
Bank withdraw	20 · 50 · 100 · 200 · 4200 (paivana)	10 · 20 · 50
Pay ladder	0.01 … 10	0.01 … 1
Paivana	HTTP paywall + template pay GOA:4200 (goa-shop / paivana)	skipped

./taler-monitoring.sh e2e
./taler-monitoring.sh -d taler.net urls e2e
# Amount ladder (withdraw 0 → … → max, then pay): same script for GOA + stage
./taler-monitoring.sh ladder                              # GOA / current domain
./taler-monitoring.sh -d stage.lefrancpaysan.ch ladder    # TESTPAYSAN maxima
# Stage auto-reads bank max_wire_transfer_amount (e.g. 2000) and min denom (0.01).
# Explorer password: francpaysan-stage-user …/bank-explorer-password.txt (or EXP_PW=).
# Override: LADDER_MAX_AMOUNT=100 LADDER_STEPS=7 LADDER_PAY=0 …
# customize e2e:
E2E_WITHDRAW_VALUES="20 50 100" E2E_PAY_VALUES="0.05 1 5" ./taler-monitoring.sh e2e
E2E_VARIABLE=0 WITHDRAW_AMT=GOA:50 PAY_AMT=GOA:1 ./taler-monitoring.sh e2e   # single fixed
# GOA shop catalog (local hacktivism): full list in E2E_SHOP_PRODUCTS; each run
#   shuffles and pays E2E_SHOP_PICK_N products (default 2).
#   E2E_SHOP_PRODUCTS lines: id|Product name|GOA:amount
#   E2E_SHOP_PICK_N=2
#   (landing QR = taler://pay-template/…/{id}; popup = live taler://pay after POST templates/{id})
# Paivana paywall (local GOA only):
#   E2E_PAIVANA=1 (default)  PAIVANA_PUBLIC=https://paivana.hacktivism.ch
#   E2E_PAIVANA_TEMPLATE=paivana  E2E_PAIVANA_AMOUNT=GOA:4200  E2E_PAIVANA_INSTANCE=goa-shop
#   E2E_PAIVANA=0            # skip
# remote secrets:
#   E2E_BANK_ADMIN_PASS=…  E2E_MERCHANT_TOKEN=…

Phases

Phase	What
inside	SSH koopa: processes, postgres, local /config,/keys, wirewatch, recent log ERRORs
versions	deb.taler.net reachable (InRelease/Packages/pool .deb); containers can reach it + have apt source; installed Taler packages vs trixie
sanity	bank · exchange · merchant sections (public + server)
e2e	account → credit → withdraw → wallet → confirm → order → pay
ladder	GOA withdraw: fixed 0 + low bands + random high ranges + fixed max

# package suite (default trixie on deb.taler.net)
TALER_APT_SUITE=trixie ./taler-monitoring.sh versions
TALER_PKG_BEHIND=error ./taler-monitoring.sh versions   # any behind = ERROR

E2E maps failures to blockers, e.g.:

• bank-confirm HTTP 409 → wallet did not select exchange
• no GOA balance → wirewatch / transfer
• create order failed → merchant auth/instance
• Alarm clock during pay → usually missing handle-uri --yes or wrong pay URI (must be …/instances/{inst}/{oid}/?c={token} from merchant taler_pay_uri)
• insufficient balance → withdraw incomplete

Performance (urls phase)

Outside-in HTTPS RTT for bank / exchange / merchant critical paths. Each probe prints HTTP … · N ms on the [OK]/[WARN]/[ERROR] line; end of the block prints perf summary (n / min / p50 / avg / max).

Env	Default	Meaning
PERF_WARN_MS	8000	WARN if latency ≥ this
PERF_FAIL_MS	20000	ERROR if latency ≥ this
PERF_CURL_TIMEOUT	25	curl max seconds per probe

PERF_WARN_MS=3000 PERF_FAIL_MS=10000 ./taler-monitoring.sh urls

Wallet coins (e2e · ladder)

At every relevant withdraw/pay step the wallet is snapped via advanced dump-coins. Amounts use exchange alt_unit_names (Kilo-GOA, Mega-GOA, …) with the base GOA value in parentheses:

[INFO]  coins after-bank-wd — amount_circ=2 Kilo-GOA (GOA:2000)
[INFO]  coins after-bank-wd — denoms_in_circ 1 Kilo-GOA (GOA:1000)×2 (=2 Kilo-GOA (GOA:2000))
== ladder · rung 12  GOA:1000000 · 1 Mega-GOA (GOA:1000000) ==

Also: coin counts (in circulation / spent), status histogram, Δ vs previous snap. History TSV: $METRICS_DIR/coins-history.tsv. Map loaded from ${EXCHANGE_PUBLIC}/config → currency_specification.alt_unit_names.

Final statistics dashboard (e2e · ladder)

At the end of e2e and ladder, a boxed FINAL STATISTICS block prints:

Section	Content
Coins	in circulation / spent counts + amounts + denoms (alt names)
Money flow	withdrawn events/amount, spent (paid), net (wd − spent)
Tendency	coin/amount trend over snaps (↑↓→), slope, last history rows
Performance	min / p50 / avg / max per phase; first-half vs second-half latency
Load	host loadavg + mem before/after; per-container RSS/CPU; tendencies

Successful withdraw/pay steps append to flow-withdrawn.txt / flow-spent.txt (ladder also uses its TSVs).

Load / memory (inside · e2e · ladder)

Host loadavg, RAM used/avail, and per-container RSS / CPU / block I/O (plus postgres DB sizes and role RSS: libeufin, exchange, merchant, …) are snapshotted via SSH whenever the stack is under load:

Phase	When
inside	end of container status
e2e	before work · after bank withdraws · after payments · after shop · e2e-end (+ delta)
ladder	before first rung · after last rung (+ delta + timing overall)

# LAN down? force WAN DNAT host
KOOPA_SSH=koopa-external ./taler-monitoring.sh inside
# or rely on automatic fallback (KOOPA_SSH_FALLBACKS from env)

METRICS_LOAD=0 ./taler-monitoring.sh e2e   # skip host probes
LADDER_LOAD=0 ./taler-monitoring.sh ladder

Env	Default	Meaning
KOOPA_SSH	koopa	primary SSH host
KOOPA_SSH_FALLBACKS	(from env)	tried if primary fails
METRICS_LOAD	1	0 = skip all host/container load probes
LADDER_LOAD	1	0 = skip load in ladder only
LADDER_PAY	1	0 = skip payment ladder after withdraws
LADDER_WITHDRAW_SCALE	1.5	withdraw mids / pay mids ratio (funding headroom)
LADDER_STEPS	23	rungs for both withdraw and pay (0 + mids + max−1 + max)
METRICS_LOAD_SSH_TIMEOUT	90	seconds for remote load python

Needs

• SSH via KOOPA_SSH / KOOPA_SSH_FALLBACKS (env) (inside / load / sanity server bits)
• secrets under koopa-admin-secrets/... for e2e
• taler-wallet-cli for e2e

Languages (i18n) — v1.2+

Default English. Select language with --lang en|fr or TALER_MON_LANG. FrancPaysan stacks auto-select French (*lefrancpaysan*). See VERSIONS.md and i18n.sh.

Public monitoring pages via FQDN (v1.3.1+ / obligatory v1.7.6+ / layout v1.8.0+)

Phase monpages is obligatory: missing or non-suite HTML is ERROR. Soft mode only with MONPAGES_REQUIRE_PUBLIC=0 (escape hatch).

Stack	What monpages expects (v1.9.0)
GOA	3× /monitoring/ + surface (existence and top/bottom/content markers)
FrancPaysan	Stack mon hosts /monitoring/ with same content rules
Nine landings (model)	Each landing front: /monitoring + /monitoring_err (err when a failed run exists)

./taler-monitoring.sh -d hacktivism.ch monpages
./taler-monitoring.sh -d lefrancpaysan.ch monpages

If pages are missing publicly: sudo $APPLY_MONITORING_LIVE on koopa (FP: Infomaniak vhost for /monitoring*).

Devtesting · fake-franken CHF (v1.13+)

Phase devtesting (aliases: franken, fake-franken) exercises the Taler Operations rusty host (taler-devtesting forced command):

./taler-monitoring.sh -d stage.taler-ops.ch --currency CHF devtesting

Needs SSH to devtesting@rusty.taler-ops.ch (Host taler-rusty-devtesting). Probes: CLI help, geniban (CH IBAN / synthetic debit funding), baseline fake-incoming, optional amount ladder (DEVTESTING_LADDER=1, default 23 rungs up to CHF:50000). Each rung reuses geniban debit payto so volume covers the max rung without a real bank withdraw. fake-incoming CHF to exchange credit IBAN. A nexus bounce missing reserve public key is OK for the plumbing probe (no real wallet reserve required).

Version on monitoring pages (v1.3+ / sticky bar)

The sticky bar shows the suite tag (e.g. v1.11.0) and links to that tag on Forgejo.

v1.12.0: Run header (monitoring env context: env_file, suite pin, domain/hosts/flags) is a collapsed <details> block above the console log — expand to inspect agent context.

v1.11.0: Click the sticky error or warning count to filter the console log to only those lines (other log lines hide; sticky bar, stats, and top overviews stay). Same via #filter-error / #filter-warn (or #first-error / #first-warn). Click again or Show all to clear.