monitoring/merge: origin ladder+metrics + stage domains/profiles

2026-07-17 15:46:50 +02:00 · 2026-07-17 15:46:50 +02:00 · d6f4f649c1
commit d6f4f649c1
parent 94c02058b3 78bbd80d1f
86 changed files with 8199 additions and 256 deletions
--- a/scripts/taler-monitoring/README.md
+++ b/scripts/taler-monitoring/README.md
@ -135,6 +135,7 @@ E2E_VARIABLE=0 WITHDRAW_AMT=GOA:50 PAY_AMT=GOA:1 ./taler-monitoring.sh e2e   # s
 | **versions** | `deb.taler.net` reachable (InRelease/Packages/pool `.deb`); containers can reach it + have apt source; installed Taler packages vs **trixie** |
 | **sanity** | bank · exchange · merchant sections (public + server) |
 | **e2e** | account → credit → withdraw → wallet → confirm → order → pay |
+| **ladder** | GOA withdraw: fixed **0** + low bands + **random high ranges** + fixed **max** |

 ```bash
 # package suite (default trixie on deb.taler.net)
@ -150,8 +151,86 @@ E2E maps failures to blockers, e.g.:
 - `Alarm clock` during pay → usually missing `handle-uri --yes` or wrong pay URI (must be `…/instances/{inst}/{oid}/?c={token}` from merchant `taler_pay_uri`)
 - `insufficient balance` → withdraw incomplete  

+## Performance (urls phase)
+
+Outside-in HTTPS RTT for bank / exchange / merchant critical paths. Each probe prints
+`HTTP … · N ms` on the `[OK]`/`[WARN]`/`[ERROR]` line; end of the block prints
+`perf summary` (n / min / p50 / avg / max).
+
+| Env | Default | Meaning |
+|-----|---------|---------|
+| `PERF_WARN_MS` | `8000` | WARN if latency ≥ this |
+| `PERF_FAIL_MS` | `20000` | ERROR if latency ≥ this |
+| `PERF_CURL_TIMEOUT` | `25` | curl max seconds per probe |
+
+```bash
+PERF_WARN_MS=3000 PERF_FAIL_MS=10000 ./taler-monitoring.sh urls
+```
+
+## Wallet coins (e2e · ladder)
+
+At every relevant withdraw/pay step the wallet is snapped via `advanced dump-coins`.
+Amounts use exchange **`alt_unit_names`** (Kilo-GOA, Mega-GOA, …) with the base
+**GOA value in parentheses**:
+
+```text
+[INFO]  coins after-ATM — amount_circ=2 Kilo-GOA (GOA:2000)
+[INFO]  coins after-ATM — denoms_in_circ 1 Kilo-GOA (GOA:1000)×2 (=2 Kilo-GOA (GOA:2000))
+== ladder · rung 12  GOA:1000000 · 1 Mega-GOA (GOA:1000000) ==
+```
+
+Also: coin counts (in circulation / spent), status histogram, Δ vs previous snap.
+History TSV: `$METRICS_DIR/coins-history.tsv`. Map loaded from
+`${EXCHANGE_PUBLIC}/config` → `currency_specification.alt_unit_names`.
+
+### Final statistics dashboard (e2e · ladder)
+
+At the end of **e2e** and **ladder**, a boxed **FINAL STATISTICS** block prints:
+
+| Section | Content |
+|---------|---------|
+| **Coins** | in circulation / spent counts + amounts + denoms (alt names) |
+| **Money flow** | withdrawn events/amount, spent (paid), net (wd − spent) |
+| **Tendency** | coin/amount trend over snaps (↑↓→), slope, last history rows |
+| **Performance** | min / p50 / avg / max per phase; first-half vs second-half latency |
+| **Load** | host loadavg + mem before/after; per-container RSS/CPU; tendencies |
+
+Successful withdraw/pay steps append to `flow-withdrawn.txt` / `flow-spent.txt` (ladder also uses its TSVs).
+
+## Load / memory (inside · e2e · ladder)
+
+Host **loadavg**, **RAM used/avail**, and per-container **RSS / CPU / block I/O**
+(plus postgres DB sizes and role RSS: libeufin, exchange, merchant, …) are
+snapshotted via SSH whenever the stack is under load:
+
+| Phase | When |
+|-------|------|
+| **inside** | end of container status |
+| **e2e** | before work · after ATM withdraws · after payments · after shop · e2e-end (+ delta) |
+| **ladder** | before first rung · after last rung (+ delta + timing overall) |
+
+```bash
+# LAN down? force WAN DNAT host
+KOOPA_SSH=koopa-external ./taler-monitoring.sh inside
+# or rely on automatic fallback (default KOOPA_SSH_FALLBACKS=koopa-external)
+
+METRICS_LOAD=0 ./taler-monitoring.sh e2e   # skip host probes
+LADDER_LOAD=0 ./taler-monitoring.sh ladder
+```
+
+| Env | Default | Meaning |
+|-----|---------|---------|
+| `KOOPA_SSH` | `koopa` | primary SSH host |
+| `KOOPA_SSH_FALLBACKS` | `koopa-external` | tried if primary fails |
+| `METRICS_LOAD` | `1` | `0` = skip all host/container load probes |
+| `LADDER_LOAD` | `1` | `0` = skip load in ladder only |
+| `LADDER_PAY` | `1` | `0` = skip payment ladder after withdraws |
+| `LADDER_WITHDRAW_SCALE` | `1.5` | withdraw mids / pay mids ratio (funding headroom) |
+| `LADDER_STEPS` | `23` | rungs for both withdraw and pay (0 + mids + max−1 + max) |
+| `METRICS_LOAD_SSH_TIMEOUT` | `90` | seconds for remote load python |
+
 ## Needs

- SSH `koopa` (inside/sanity server bits)
+- SSH `koopa` or `koopa-external` (inside / load / sanity server bits)
 - secrets under `koopa-admin-secrets/...` for e2e
 - `taler-wallet-cli` for e2e
--- a/scripts/taler-monitoring/TESTS.md
+++ b/scripts/taler-monitoring/TESTS.md
@ -47,15 +47,19 @@ IDs are assigned **in run order** within the area (`set_area` resets the counter
 | www-… | cross-links: one line (4 intros) on local stack |
 | www-… | bank demo-withdraw + shop assets: one line (+ soft shop-pay if missing) |
 | www-… | **bank `/intro/demo-withdraw.json`** → `taler://withdraw/HOST:PORT/taler-integration/…` + integration op HTTP 200 |
+| www-… | **landing exposed links** (bank / merchant / exchange): parse each `/intro/` HTML, probe every own-stack `https://` + root-relative `href`/`src`/`content`, soft-check external stores/docs |
+| www-… | landing static: `qrcode.min.js`, `og-goa-shop.png`, `qr-logo.png`, shop-pay.js/css |
+| www-… | cross-links between bank ↔ merchant ↔ exchange intros (local stack) |
+| www-… | **bank `/intro/demo-withdraw.json`** → `taler://withdraw/HOST/taler-integration/…` (no default `:443`/`:80`; non-default port OK) + integration op HTTP 200 |
 | www-… | bank `/intro/auto-account.json` (earlier) → same withdraw shape, **no payto_uri**, login at `/webui/` |
 | www-… | **performance** (outside-in RTT): bank/exchange/merchant `/config`, keys, webui, intro — ms; WARN ≥ `PERF_WARN_MS` (8000); ERROR ≥ `PERF_FAIL_MS` (20000) |
 | www-… | **landing load stats**: `/intro/stats.json` → loadavg, container RSS, in-container probe ms (public; SSH container cat fallback); warn if stale/missing |

 **Legal docs rule:** HTTP 200, non-empty body, not plain `not configured`, not merchant API JSON `code:21`. On local stack, optional content needle (terms/privacy/FADP/GOA…).

-**Performance rule:** Measured from the **monitoring runner** (public URLs via Caddy), not container loopback. HTTP must match expect (usually 200); latency is reported on the OK line. Slow ≥ `PERF_WARN_MS` → WARN only (no ERROR on slowness alone).
+**Performance rule:** Measured from the **monitoring runner** (public URLs via Caddy), not container loopback. HTTP must match expect (usually 200); **latency (ms) is on every perf OK/WARN/ERROR line**, plus a **perf summary** (n / min / p50 / avg / max). Slow ≥ `PERF_WARN_MS` (default 8000) → WARN; ≥ `PERF_FAIL_MS` (default 20000) → ERROR.

-**Landing links rule:** Own-stack (bank/exchange/taler.\* + page host) must be HTTP 200 (or redirect→200). External (App Store, Play, F-Droid, wallet.taler.net, docs/git.taler.net, …) soft WARN if down. Auto-account wallet link must be `taler://withdraw/…:port/taler-integration/…`, never payto.
+**Landing links rule:** Own-stack (bank/exchange/taler.\* + page host) must be HTTP 200 (or redirect→200). External (App Store, Play, F-Droid, wallet.taler.net, docs/git.taler.net, …) soft WARN if down. Auto-account wallet link must be `taler://withdraw/HOST/taler-integration/…` (default ports stripped for mobile wallets), never payto.

 **alt_unit_names rule:** wallet codec requires a non-empty map including scale key `"0"`. For multi-currency merchant, also follow every entry in `exchanges[]` and check that exchange’s public `/config`.

@ -67,11 +71,14 @@ IDs are assigned **in run order** within the area (`set_area` resets the counter

 | ID | Check (typical order) |
 |----|------------------------|
-| inside-001 | ssh koopa |
+| inside-001 | ssh koopa (or **koopa-external** fallback) |
 | inside-002+ | per-component emit: container, ports, libeufin/httpd, postgres, local `/config`/`/keys`, wirewatch, DNS pin, caddy |
+| inside-… | **load / memory**: host loadavg + RAM; bank/exchange/merchant podman CPU/mem/block + process RSS by role + DB sizes |

 Remote lines `E|comp|LEVEL|key|detail` each become one numbered result.

+SSH: `KOOPA_SSH` (default `koopa`), then `KOOPA_SSH_FALLBACKS` (default `koopa-external`) when LAN is unreachable.
+
 ---

 ## sanity — bank · exchange · merchant (`./taler-monitoring.sh sanity`)
@ -145,8 +152,10 @@ Local GOA also: **ATM includes GOA:4200**, then **paivana** (HTTP 302 on `PAIVAN
 | e2e-003 | mode / currency info |
 | e2e-004… | secrets, reachability gates |
 | e2e-… | account, credit, withdraw, confirm, coins, order, pay ladder |
+| e2e-… | **load snapshots** at e2e-start, after-withdraw, after-pay, after-shop, e2e-end (host loadavg/RAM + container RSS/CPU/DB) |
+| e2e-… | **coins** before/after each ATM withdraw, settle, pay, shop: count in circulation / spent, amount + denoms using **`alt_unit_names`** (e.g. `5 Kilo-GOA (GOA:5000)`), Δ vs previous snap |
 | e2e-… | **GOA shop products** — full catalog list; **random pick of 2** (override `E2E_SHOP_PICK_N`) |
-| e2e-… | balances, dig on failure |
+| e2e-… | balances, dig on failure, load delta + overall metrics |

 Shop product pays use instance `goa-shop` (default) and catalog `E2E_SHOP_PRODUCTS`
 (`id|Product name|amount` lines). Each e2e run **shuffles** the catalog and pays
@ -157,6 +166,45 @@ Blockers keep the same ID prefix: `[BLOCKER] e2e-0NN step: message`.

 ---

+## ladder — GOA withdraw ranges (`./taler-monitoring.sh ladder`)
+
+| Step | What |
+|------|------|
+| ladder-001 | `GET /intro/auto-account.json` (personal account, GOA:0) |
+| ladder-… | **load snapshot** before withdraws (host + bank/exchange/merchant mem/CPU) |
+| ladder-… | **Phase A withdraw** (23): mint explorer pool — 0 → random → max−1 → max into **one cumulative wallet** (mids × `LADDER_WITHDRAW_SCALE`, default 1.5) |
+| ladder-… | wallet-cli accept-uri + explorer confirm when `selected`; settle; coins snap |
+| ladder-… | **Phase B pay** (23): same shape 0 → random → max−1 → max via merchant orders + `handle-uri --yes` |
+| ladder-… | **load snapshot** after + delta + overall timing (withdraw + pay) |
+| report | withdraw TSV + pay TSV + JSON |
+
+Default amounts (`build_ladder_pair`, strictly increasing):
+
+```text
+withdraw: [0] + random mids + [max−1] + [max]
+pay:      [0] + (withdraw_mid / scale) + [max−1] + [max]   # always ≤ matching withdraw mid
+```
+
+TSV `range`: `pin:0` | `random` | `pin:max-1` | `pin:max`.
+
+Soft: absolute **max** mint/pay may `CEILING_REJECT` / skip (WARN). **max-1** is a hard pin.  
+`LADDER_PAY=0` skips phase B.
+
+| Env | Default | Meaning |
+|-----|---------|---------|
+| `LADDER_STEPS` | `23` | total rungs (0 + mids + max) |
+| `LADDER_MAX_AMOUNT` | `4503599627370496` | absolute last pin |
+| `LADDER_TIMEOUT_S` | `3600` | large rungs need time |
+| `LADDER_MAX_RUNGS` | `99` | cap list length after build |
+
+```bash
+./taler-monitoring.sh ladder
+LADDER_STEPS=10 ./taler-monitoring.sh ladder
+LADDER_REPORT_DIR=/tmp/my-ladder ./taler-monitoring.sh ladder
+```
+
+---
+
 ## Run one area

 ```bash
@ -164,5 +212,6 @@ Blockers keep the same ID prefix: `[BLOCKER] e2e-0NN step: message`.
 ./taler-monitoring.sh inside    # inside only
 ./taler-monitoring.sh versions  # deb.taler.net + package drift
 ./taler-monitoring.sh e2e       # e2e only
+./taler-monitoring.sh ladder    # GOA amount ladder + timings
 ./taler-monitoring.sh -d taler.net urls
 ```
--- a/scripts/taler-monitoring/check_e2e.sh
+++ b/scripts/taler-monitoring/check_e2e.sh
@ -5,6 +5,8 @@ set -euo pipefail
 ROOT=$(cd "$(dirname "$0")" && pwd)
 # shellcheck source=lib.sh
 source "$ROOT/lib.sh"
+# shellcheck source=metrics.sh
+source "$ROOT/metrics.sh"

 : "${E2E_TIMEOUT:=180}"  # ATM ladder + wirewatch lag needs room (was 55 — too tight)
 : "${E2E_PAY_SECS:=22}"  # hard reserve for handle-uri + settle (do not starve pay)
@ -150,6 +152,15 @@ e2e_finish() {
  fi
  E2E_REPORTED=1
  print_balances
+  # Final coin inventory + host/container load + statistics dashboard
+  if [ -n "${METRICS_DIR:-}" ]; then
+    section "metrics · e2e coins final"
+    metrics_report_coins "e2e-end" || true
+    if [ "${METRICS_LOAD:-1}" != "0" ]; then
+      metrics_report_load "${METRICS_DIR}/load-after.json" "e2e-end" || true
+    fi
+    metrics_print_overall "e2e final" || true
+  fi
  summary || true
  return "$code"
 }
@ -288,6 +299,9 @@ BANK_HOST=$(python3 -c 'from urllib.parse import urlparse; print(urlparse("'"$BA

 SCRATCH=$(mktemp -d)
 WDB="$SCRATCH/wallet.sqlite3"
+METRICS_DIR="${METRICS_DIR:-$SCRATCH/metrics}"
+mkdir -p "$METRICS_DIR"
+export METRICS_DIR CUR="${CUR:-${EXPECT_CURRENCY:-GOA}}" WDB
 export PATH="${HOME}/.local/bin:${HOME}/taler/opt/taler-wallet-cli/usr/bin:/opt/homebrew/bin:/usr/local/bin:$PATH"
 # Always dump balances on any exit (timeout, blocker, signal, success)
 trap 'ec=$?; e2e_finish "$ec"; rm -rf "$SCRATCH"; exit "$ec"' EXIT
@ -358,6 +372,19 @@ info "ATM withdraw ladder" "$WITHDRAW_LIST  (credit $CREDIT_AMT)"
 info "pay ladder" "$PAY_LIST"
 info "e2e budget" "${E2E_TIMEOUT}s"

+# alt_unit_names for human amounts (Kilo-GOA … + base GOA in parentheses)
+ALT_UNITS_FILE="${METRICS_DIR}/alt_unit_names.json"
+export ALT_UNITS_FILE
+if metrics_load_alt_units "${EXCHANGE_PUBLIC}/config"; then
+  info "alt_unit_names" "from ${EXCHANGE_PUBLIC}/config"
+else
+  warn "alt_unit_names" "using built-in SI fallback"
+fi
+
+# Host + bank/exchange/merchant RAM/CPU/load (SSH koopa or koopa-external)
+section "e2e · load snapshot (before withdraw/pay)"
+metrics_report_load "${METRICS_DIR}/load-before.json" "e2e-start" || true
+
 # Local stack: koopa secrets. Remote: only explicit env (never leak local passwords to foreign banks).
 if [ "$E2E_REMOTE" = "1" ]; then
  ADMIN_PASS="${E2E_BANK_ADMIN_PASS:-}"
@ -672,6 +699,8 @@ else
    warn "wallet accept ToS" "$(tail -c 80 "$SCRATCH/ex-tos.out" | tr '\n' ' ')"
  fi
 fi
+# Empty wallet baseline before ATM / pay load
+metrics_report_coins "wallet-ready" || true

 wd_status() {
  curl -sS -m 8 -o "$SCRATCH/wd-st.json" -w '' \
@ -714,8 +743,9 @@ e2e_one_withdraw() {
  WITHDRAW_AMT="$1"
  local tag
  tag=$(printf '%s' "$WITHDRAW_AMT" | tr '.:' '__')
-  section "e2e · ATM withdraw $WITHDRAW_AMT"
+  section "e2e · ATM withdraw $WITHDRAW_AMT · $(format_amount_alt "$WITHDRAW_AMT")"
  e2e_over && { warn "ATM withdraw" "budget exhausted — skip $WITHDRAW_AMT"; return 1; }
+  metrics_report_coins "before-ATM-${tag}" || true

  curl -sS -m 15 -o "$SCRATCH/wd-$tag.json" -H "Authorization: Bearer $UT" \
    -H 'Content-Type: application/json' \
@ -808,7 +838,9 @@ else:
    fi
    sleep 2
  done
+  metrics_report_coins "after-ATM-${tag}" || true
  if [ "$ok_bal" = "1" ]; then
+    metrics_record_flow withdrawn "$WITHDRAW_AMT" || true
    return 0
  fi
  # Bank side often already confirmed — treat as timing lag, not hard fail
@ -832,8 +864,9 @@ e2e_one_pay() {
  fi
  # shell-safe tag for files
  tag=$(printf '%s' "$tag" | tr -c 'A-Za-z0-9._-' '_')
-  section "e2e · pay $PAY_AMT · $PAY_SUM"
+  section "e2e · pay $PAY_AMT · $(format_amount_alt "$PAY_AMT") · $PAY_SUM"
  e2e_over && { warn "pay" "budget exhausted — skip $PAY_AMT ($PAY_SUM)"; return 1; }
+  metrics_report_coins "before-pay-${tag}" || true
  if [ -z "${MPW:-}" ]; then
    warn "pay $PAY_AMT" "no merchant token"
    return 1
@ -906,9 +939,12 @@ d=json.load(open(sys.argv[1]))
 sys.exit(0 if d.get("paid") is True or str(d.get("order_status","")).lower()=="paid" else 1)
 ' "$SCRATCH/ord-paid-$tag.json" 2>/dev/null; then
      ok "payment settled $PAY_AMT ($PAY_SUM · order $OID)"
+      metrics_report_coins "after-pay-${tag}" || true
+      metrics_record_flow spent "$PAY_AMT" || true
      return 0
    fi
  done
+  metrics_report_coins "after-pay-fail-${tag}" || true
  warn "pay $PAY_AMT ($PAY_SUM)" "not settled for order $OID"
  return 1
 }
@ -945,8 +981,9 @@ e2e_one_pay_public_template() {
  local pamt="$3"
  local tag
  tag=$(printf 'shop_%s' "$pid" | tr -c 'A-Za-z0-9._-' '_')
-  section "e2e · shop product · $pname ($pid) · $pamt"
+  section "e2e · shop product · $pname ($pid) · $pamt · $(format_amount_alt "$pamt")"
  e2e_over && { warn "shop $pname" "budget exhausted — skip"; return 1; }
+  metrics_report_coins "before-shop-${tag}" || true

  local MH
  MH=$(python3 -c 'from urllib.parse import urlparse; print(urlparse("'"$MERCHANT_PUBLIC"'").hostname or "taler.hacktivism.ch")' 2>/dev/null || echo "taler.hacktivism.ch")
@ -1023,15 +1060,20 @@ d=json.load(open(sys.argv[1]))
 sys.exit(0 if d.get("paid") is True or str(d.get("order_status","")).lower()=="paid" else 1)
 ' "$SCRATCH/ord-paid-$tag.json" 2>/dev/null; then
        ok "shop $pname" "payment settled ($pamt · order $OID)"
+        metrics_report_coins "after-shop-${tag}" || true
+        metrics_record_flow spent "$pamt" || true
        return 0
      fi
    fi
    if grep -qiE 'payment|paid|Payment' "$SCRATCH/tx-$tag.out" 2>/dev/null \
      || grep -qiE 'done|paid|success|Payment' "$SCRATCH/pay-$tag.out" 2>/dev/null; then
      ok "shop $pname" "payment settled via wallet tx ($pamt · order $OID)"
+      metrics_report_coins "after-shop-${tag}" || true
+      metrics_record_flow spent "$pamt" || true
      return 0
    fi
  done
+  metrics_report_coins "after-shop-fail-${tag}" || true
  warn "shop $pname ($pid)" "not settled for order $OID"
  return 1
 }
@ -1067,6 +1109,10 @@ for WITHDRAW_AMT in $WITHDRAW_LIST; do
  esac
 done
 info "ATM withdraw summary" "$WITHDRAW_REPORT (ok=$WITHDRAW_OK_N lag=$WITHDRAW_LAG_N fail=$WITHDRAW_FAIL_N)"
+section "e2e · load snapshot (after ATM withdraws)"
+metrics_report_load "${METRICS_DIR}/load-after-withdraw.json" "after-withdraw" || true
+cp -f "${METRICS_DIR}/load-after-withdraw.json" "${METRICS_DIR}/load-after.json" 2>/dev/null || true
+metrics_report_coins "after-ATM-ladder" || true

 # Settlement catch-up: bank may have confirmed while wallet was still empty
 section "e2e · wallet settlement (timing)"
@ -1083,6 +1129,7 @@ if python3 -c "import sys; sys.exit(0 if float(sys.argv[1]) > 0 else 1)" "$av_no
    WITHDRAW_REPORT="${WITHDRAW_REPORT} → late-OK avail=${CUR}:${av_now}"
  fi
  ok "spendable balance for payments" "${CUR}:${av_now}"
+  metrics_report_coins "after-settle" || true
 else
  if [ "$WITHDRAW_OK" != "1" ]; then
    warn "withdraw" "still no spendable ${CUR} after settle wait"
@ -1146,6 +1193,10 @@ for PAY_AMT in $PAY_LIST; do
  fi
 done
 info "pay summary" "$PAY_REPORT (ok=$PAY_OK_N fail=$PAY_FAIL_N skip=$PAY_SKIP_N)"
+section "e2e · load snapshot (after payments)"
+metrics_report_load "${METRICS_DIR}/load-after-pay.json" "after-pay" || true
+cp -f "${METRICS_DIR}/load-after-pay.json" "${METRICS_DIR}/load-after.json" 2>/dev/null || true
+metrics_report_coins "after-pay-ladder" || true
 if [ "$PAY_OK" != "1" ]; then
  av_now=$(wallet_avail_num)
  if python3 -c "import sys; sys.exit(0 if float(sys.argv[1]) > 0 else 1)" "$av_now" 2>/dev/null; then
@ -1239,6 +1290,9 @@ EOF
    if [ "$SHOP_OK" != "1" ] && [ "$SHOP_FAIL_N" -gt 0 ]; then
      warn "goa-shop" "no sample product payment succeeded ($SHOP_REPORT)"
    fi
+    section "e2e · load snapshot (after shop pays)"
+    metrics_report_load "${METRICS_DIR}/load-after-shop.json" "after-shop" || true
+    cp -f "${METRICS_DIR}/load-after-shop.json" "${METRICS_DIR}/load-after.json" 2>/dev/null || true
  fi
 fi

--- a/scripts/taler-monitoring/check_goa_ladder.sh
+++ b/scripts/taler-monitoring/check_goa_ladder.sh
--- a/scripts/taler-monitoring/check_inside.sh
+++ b/scripts/taler-monitoring/check_inside.sh
@ -4,6 +4,8 @@ set -euo pipefail
 ROOT=$(cd "$(dirname "$0")" && pwd)
 # shellcheck source=lib.sh
 source "$ROOT/lib.sh"
+# shellcheck source=metrics.sh
+source "$ROOT/metrics.sh"

 # Area inside-### — container / process state on koopa (SSH)
 set_area inside
@ -35,8 +37,10 @@ hasp() { podman exec "$1" pgrep -f "$2" >/dev/null 2>&1; }
 BANK=$(podman ps --format '{{.Names}}' 2>/dev/null | grep -iE 'hacktivism-bank|taler-bank' | head -1)
 [ -z "$BANK" ] && BANK=$(podman ps --format '{{.Names}}' 2>/dev/null | grep -i bank | head -1)
 EX=$(podman ps --format '{{.Names}}' 2>/dev/null | grep -i exchange | head -1)
+# Exact merchant container name first — never fall through to *-bank
 MER=$(podman ps --format '{{.Names}}' 2>/dev/null | grep -E '^taler-hacktivism$' | head -1)
-[ -z "$MER" ] && MER=$(podman ps --format '{{.Names}}' 2>/dev/null | grep -iE 'merchant|hacktivism' | grep -viE 'bank|exchange' | head -1)
+[ -z "$MER" ] && MER=$(podman ps --format '{{.Names}}' 2>/dev/null | grep -i merchant | grep -viE 'bank|exchange' | head -1)
+[ -z "$MER" ] && MER=$(podman ps --format '{{.Names}}' 2>/dev/null | grep -E 'hacktivism' | grep -viE 'bank|exchange' | head -1)

 # Domain resolve inside container (wirewatch needs bank.hacktivism.ch → real IP)
 # emit: comp LEVEL dns "host → ip" or fail
@ -138,4 +142,10 @@ while IFS= read -r line; do
  esac
 done <<<"$RAW"

+# Host loadavg + RAM + per-container RSS/CPU (same probe as e2e/ladder)
+section "inside · load / memory"
+METRICS_DIR="${METRICS_DIR:-$(mktemp -d)}"
+export METRICS_DIR
+metrics_report_load "${METRICS_DIR}/load-inside.json" "inside" || true
+
 summary
--- a/scripts/taler-monitoring/check_urls.sh
+++ b/scripts/taler-monitoring/check_urls.sh
@ -98,6 +98,77 @@ check_legal_doc() {
  rm -f "$f"
 }

+# Shared shape for bank mint JSON (demo-withdraw + auto-account).
+# Args: $1=json file  $2=auto|demo
+# stdout (ok): one detail line; for demo, optional "\twithdrawal_id" suffix.
+# stderr (fail): short reason. Exit 0/1.
+validate_bank_withdraw_json() {
+  python3 - "$1" "$2" <<'PY'
+import json, re, sys
+from urllib.parse import urlparse
+
+def check_withdraw_uri(wuri: str):
+    """HOST or HOST:non-default-port; default :443/:80 must be stripped (wallet fix)."""
+    m = re.match(
+        r"^taler://withdraw/([^/]+)/taler-integration/([0-9a-fA-F-]+)$",
+        wuri or "",
+    )
+    if not m:
+        print(
+            "need taler://withdraw/HOST/taler-integration/ID:",
+            (wuri or "")[:120],
+            file=sys.stderr,
+        )
+        return None
+    host = m.group(1)
+    if not host or host.startswith(":"):
+        print("bad withdraw host:", host, file=sys.stderr)
+        return None
+    if ":" in host:
+        h, _, p = host.rpartition(":")
+        if not h or not p.isdigit():
+            print("bad withdraw host:port:", host, file=sys.stderr)
+            return None
+        if p in ("443", "80"):
+            print("withdraw must strip default port :%s:" % p, host, file=sys.stderr)
+            return None
+    return host, m.group(2), wuri
+
+path, mode = sys.argv[1], sys.argv[2]
+d = json.load(open(path))
+wuri = d.get("taler_withdraw_uri") or (
+    d.get("qr_payload") if mode == "auto" else ""
+) or ""
+
+if mode == "auto":
+    if not d.get("ok"):
+        print("ok!=true", file=sys.stderr)
+        sys.exit(1)
+    if d.get("payto_uri"):
+        print("payto_uri must not be present", file=sys.stderr)
+        sys.exit(1)
+    if not check_withdraw_uri(wuri):
+        sys.exit(1)
+    webui = d.get("login_url") or d.get("webui") or d.get("account_url") or ""
+    u = urlparse(webui)
+    if u.scheme not in ("http", "https") or "webui" not in (u.path or ""):
+        print("login webui missing:", webui[:80], file=sys.stderr)
+        sys.exit(1)
+    print("%s · %s" % (d.get("username", ""), wuri[:72]))
+elif mode == "demo":
+    if not d.get("ok", True) and "taler_withdraw_uri" not in d:
+        print("not ok", file=sys.stderr)
+        sys.exit(1)
+    if not check_withdraw_uri(wuri):
+        sys.exit(1)
+    print("%s\t%s" % (wuri[:80], d.get("withdrawal_id") or ""))
+else:
+    print("bad mode:", mode, file=sys.stderr)
+    sys.exit(2)
+sys.exit(0)
+PY
+}
+
 # --- exchange (core; always required) ---  www-001 …
 check_url "exchange /config" 200 "$EXCHANGE_PUBLIC/config"
 code=$(http_body "$EXCHANGE_PUBLIC/config" "$tmp/ec.json")
@ -154,6 +225,8 @@ section "www · performance · public HTTPS latency (outside-in)"
 # ≥ PERF_WARN_MS → WARN. ≥ PERF_FAIL_MS → ERROR (suite fails).
 PERF_WARN_MS="${PERF_WARN_MS:-8000}"
 PERF_FAIL_MS="${PERF_FAIL_MS:-20000}"
+PERF_TSV="$tmp/perf.tsv"
+: >"$PERF_TSV"

 # Measure one URL: require HTTP expect (default 200), report time_total in ms.
 # $1=label $2=url $3=optional expected codes (default 200)
@ -169,6 +242,8 @@ check_perf() {
    if (ms>0 && ms<1) ms=1
    printf "%d", int(ms+0.5)
  }')
+  # Always record sample for rollup (label, ms, http, url)
+  printf '%s\t%s\t%s\t%s\n' "$label" "$ms" "$code" "$url" >>"$PERF_TSV"
  case ",$expect," in
    *",$code,"*)
      if [ "$ms" -ge "${PERF_FAIL_MS}" ] 2>/dev/null; then
@ -323,6 +398,76 @@ if [ "${CHECK_LANDING:-1}" = "1" ] || [ "${LOCAL_STACK:-0}" = "1" ]; then
  report_landing_load_stats "merchant" "$MERCHANT_PUBLIC"
  info "perf landing-stats note" "from /intro/stats.json (public); SSH container fallback if needed"
 fi
+# Rollup + optional JSON for metrics_print_overall
+if [ -s "$PERF_TSV" ]; then
+  PERF_JSON="$tmp/perf-summary.json"
+  perf_line=$(python3 - "$PERF_TSV" "$PERF_JSON" "$PERF_WARN_MS" "$PERF_FAIL_MS" <<'PY'
+import json, sys
+rows = []
+for line in open(sys.argv[1]):
+    parts = line.rstrip("\n").split("\t")
+    if len(parts) < 3:
+        continue
+    label, ms_s, code = parts[0], parts[1], parts[2]
+    try:
+        ms = int(float(ms_s))
+    except Exception:
+        continue
+    rows.append({"label": label, "ms": ms, "http": code})
+vals = sorted(r["ms"] for r in rows)
+n = len(vals)
+if n == 0:
+    print("n=0")
+    json.dump({"n": 0}, open(sys.argv[2], "w"))
+    raise SystemExit(0)
+def pct(p):
+    if n == 1:
+        return vals[0]
+    i = min(n - 1, max(0, int(round((p / 100.0) * (n - 1)))))
+    return vals[i]
+avg = int(round(sum(vals) / n))
+warn_ms = int(sys.argv[3])
+fail_ms = int(sys.argv[4])
+slow = [r for r in rows if r["ms"] >= warn_ms]
+rep = {
+    "n": n,
+    "min_ms": vals[0],
+    "p50_ms": pct(50),
+    "avg_ms": avg,
+    "max_ms": vals[-1],
+    "warn_ms": warn_ms,
+    "fail_ms": fail_ms,
+    "slow": [{"label": r["label"], "ms": r["ms"]} for r in slow],
+    "samples": rows,
+}
+# metrics_print_overall expects named buckets with n/min/p50/avg/max
+out = {
+    "www_public_https": {
+        "n": n,
+        "min_ms": vals[0],
+        "p50_ms": pct(50),
+        "avg_ms": avg,
+        "max_ms": vals[-1],
+    },
+    **{r["label"].replace(" ", "_"): {"n": 1, "min_ms": r["ms"], "p50_ms": r["ms"], "avg_ms": r["ms"], "max_ms": r["ms"]} for r in rows},
+}
+json.dump(out, open(sys.argv[2], "w"), indent=2)
+extra = ""
+if slow:
+    extra = "  slow: " + ", ".join("%s=%dms" % (r["label"].replace("perf ", ""), r["ms"]) for r in slow)
+print(
+    "n=%d  min=%dms  p50=%dms  avg=%dms  max=%dms  (warn≥%dms fail≥%dms)%s"
+    % (n, vals[0], pct(50), avg, vals[-1], warn_ms, fail_ms, extra)
+)
+PY
+  )
+  info "perf summary" "$perf_line"
+  # Keep a copy if METRICS_DIR is set (e2e/ladder overall stats)
+  if [ -n "${METRICS_DIR:-}" ] && [ -d "${METRICS_DIR}" ]; then
+    cp -f "$PERF_JSON" "${METRICS_DIR}/perf-summary.json" 2>/dev/null || true
+  fi
+fi
+info "perf note" "measured from this host (outside-in); thresholds PERF_WARN_MS=${PERF_WARN_MS} PERF_FAIL_MS=${PERF_FAIL_MS}"


 # Terms + privacy (legal docs)
@ -361,48 +506,25 @@ if [ "$code" = "200" ]; then
    check_url_soft "bank /taler-integration/config" 200 "$BANK_PUBLIC/taler-integration/config"
  fi
  check_url_soft "bank /webui/" 200 "$BANK_PUBLIC/webui/"
-  if [ "${CHECK_LANDING:-1}" = "1" ]; then
-    check_url_soft "bank /intro/" 200 "$BANK_PUBLIC/intro/"
-    check_url_soft "bank /" 302,301,200 "$BANK_PUBLIC/"
-    # Auto-account: credentials + shared-pool taler://withdraw (like step 2)
-    aa_code=$(http_body "$BANK_PUBLIC/intro/auto-account.json" "$tmp/aa.json")
-    case "$aa_code" in
-      200)
-        if python3 - "$tmp/aa.json" <<'PY'
-import json, re, sys
-from urllib.parse import urlparse
-d = json.load(open(sys.argv[1]))
-if not d.get("ok"):
-    print("ok!=true"); sys.exit(1)
-if "payto_uri" in d and d.get("payto_uri"):
-    print("payto_uri must not be present"); sys.exit(1)
-wuri = d.get("taler_withdraw_uri") or d.get("qr_payload") or ""
-wm = re.match(r"^taler://withdraw/([^/]+)/taler-integration/([0-9a-fA-F-]+)$", wuri)
-if not wm:
-    print("need taler://withdraw/HOST:PORT/taler-integration/ID:", wuri[:120]); sys.exit(1)
-if ":" not in wm.group(1):
-    print("withdraw missing port:", wm.group(1)); sys.exit(1)
-webui = d.get("login_url") or d.get("webui") or d.get("account_url") or ""
-u = urlparse(webui)
-if u.scheme not in ("http", "https") or "webui" not in (u.path or ""):
-    print("login webui missing:", webui[:80]); sys.exit(1)
-print("user=%s withdraw=%s login=%s" % (d.get("username"), wm.group(1), webui))
-sys.exit(0)
-PY
-        then
-          ok "bank /intro/auto-account.json" "$(python3 -c 'import json;d=json.load(open("'"$tmp/aa.json"'"));print(d.get("username",""),"·",(d.get("taler_withdraw_uri") or "")[:72])' 2>/dev/null || true)"
-        else
-          fail "bank /intro/auto-account.json" "invalid withdraw/login (HTTP body bad)"
-        fi
-        ;;
-      405|501|404|502|503|000)
-        fail "bank /intro/auto-account.json" "HTTP $aa_code (want 200; 405/501 = broken)"
-        ;;
-      *)
-        fail "bank /intro/auto-account.json" "HTTP $aa_code want 200"
-        ;;
-    esac
-  fi
+  check_url_soft "bank /intro/" 200 "$BANK_PUBLIC/intro/"
+  check_url_soft "bank /" 302,301,200 "$BANK_PUBLIC/"
+  # Auto-account: credentials + shared-pool taler://withdraw (like step 2)
+  aa_code=$(http_body "$BANK_PUBLIC/intro/auto-account.json" "$tmp/aa.json")
+  case "$aa_code" in
+    200)
+      if aa_detail=$(validate_bank_withdraw_json "$tmp/aa.json" auto 2>"$tmp/aa-val"); then
+        ok "bank /intro/auto-account.json" "$aa_detail"
+      else
+        fail "bank /intro/auto-account.json" "invalid withdraw/login ($(tr '\n' ' ' <"$tmp/aa-val" | sed 's/[[:space:]]*$//'))"
+      fi
+      ;;
+    405|501|404|502|503|000)
+      fail "bank /intro/auto-account.json" "HTTP $aa_code (want 200; 405/501 = broken)"
+      ;;
+    *)
+      fail "bank /intro/auto-account.json" "HTTP $aa_code want 200"
+      ;;
+  esac
 fi

 # Bank legal docs (landing nginx via Caddy /terms* /privacy* or /intro/*)
@ -802,23 +924,11 @@ if [ "${LOCAL_STACK:-1}" = "1" ] || [ -n "${BANK_PUBLIC:-}" ]; then
  dw_code=$(http_body "$BANK_PUBLIC/intro/demo-withdraw.json" "$tmp/dw.json")
  case "$dw_code" in
    200)
-      if python3 - "$tmp/dw.json" <<'PY'
-import json, re, sys
-d = json.load(open(sys.argv[1]))
-if not d.get("ok", True) and "taler_withdraw_uri" not in d:
-    print("not ok"); sys.exit(1)
-u = d.get("taler_withdraw_uri") or ""
-m = re.match(r"^taler://withdraw/([^/]+)/taler-integration/([0-9a-fA-F-]+)$", u)
-if not m:
-    print("bad uri:", u[:120]); sys.exit(1)
-if ":" not in m.group(1):
-    print("missing port:", m.group(1)); sys.exit(1)
-print(u[:88])
-sys.exit(0)
-PY
-      then
-        _ba_ok=$((_ba_ok + 1))
-        wid=$(python3 -c 'import json;print(json.load(open("'"$tmp/dw.json"'")).get("withdrawal_id",""))' 2>/dev/null || true)
+      if dw_out=$(validate_bank_withdraw_json "$tmp/dw.json" demo 2>"$tmp/dw-val"); then
+        dw_detail=${dw_out%%$'\t'*}
+        wid=${dw_out#*$'\t'}
+        [ "$wid" = "$dw_out" ] && wid=
+        ok "bank /intro/demo-withdraw.json" "$dw_detail"
        if [ -n "$wid" ]; then
          if _landing_probe "$BANK_PUBLIC/taler-integration/withdrawal-operation/${wid}"; then
            _ba_ok=$((_ba_ok + 1))
@ -829,6 +939,7 @@ PY
        ok "landing bank withdraw/shop" "demo-withdraw + shop assets ok (${_ba_ok} checks)"
      else
        fail "landing bank demo-withdraw" "invalid taler://withdraw shape"
+        fail "bank /intro/demo-withdraw.json" "invalid taler://withdraw ($(tr '\n' ' ' <"$tmp/dw-val" | sed 's/[[:space:]]*$//'))"
      fi
      ;;
    405|501|404|502|503|000)
--- a/scripts/taler-monitoring/lib.sh
+++ b/scripts/taler-monitoring/lib.sh
@ -11,6 +11,8 @@
 : "${MERCHANT_LOCAL:=https://127.0.0.1:9010}"
 : "${LANDING_LOCAL:=http://127.0.0.1:9013}"
 : "${KOOPA_SSH:=koopa}"
+# When LAN Host "koopa" is unreachable, try WAN DNAT (see ~/.ssh/config Host koopa-external).
+: "${KOOPA_SSH_FALLBACKS:=koopa-external}"
 : "${MERCHANT_INSTANCE:=goa-demo-cp4zqk}"
 : "${WITHDRAW_AMT:=GOA:20}"   # single-shot fallback; e2e ladder uses ATM notes
 : "${PAY_AMT:=GOA:0.01}"
@ -289,11 +291,42 @@ with_timeout() {
  ' "$secs" "$@"
 }

-# Probe: 0 if koopa SSH works quickly
+# Pick a working SSH host: KOOPA_SSH first, then KOOPA_SSH_FALLBACKS (koopa-external).
+# Sets KOOPA_SSH to the first host that answers. 0 = ok, 1 = none.
+KOOPA_SSH_RESOLVED=0
+resolve_koopa_ssh() {
+  [ "${SKIP_SSH:-0}" = "1" ] && return 1
+  if [ "${KOOPA_SSH_RESOLVED}" = "1" ]; then
+    return 0
+  fi
+  local cands=() c f seen=" "
+  cands+=("${KOOPA_SSH}")
+  # shellcheck disable=SC2086
+  for f in ${KOOPA_SSH_FALLBACKS}; do
+    case "$seen" in *" $f "*) continue ;; esac
+    cands+=("$f")
+    seen="$seen$f "
+  done
+  for c in "${cands[@]}"; do
+    if with_timeout $((SSH_CONNECT_TIMEOUT + 5)) \
+      ssh "${SSH_BASE_OPTS[@]}" "$c" 'echo ok' >/dev/null 2>&1; then
+      if [ "$c" != "${KOOPA_SSH}" ]; then
+        # surface once so operators know we used WAN jump
+        printf '[INFO]  SSH host %s unreachable — using %s\n' "${KOOPA_SSH}" "$c" >&2 || true
+      fi
+      KOOPA_SSH="$c"
+      KOOPA_SSH_RESOLVED=1
+      export KOOPA_SSH
+      return 0
+    fi
+  done
+  return 1
+}
+
+# Probe: 0 if any koopa SSH host works quickly
 koopa_ssh_ok() {
  [ "${SKIP_SSH}" = "1" ] && return 1
-  with_timeout $((SSH_CONNECT_TIMEOUT + 3)) \
-    ssh "${SSH_BASE_OPTS[@]}" "${KOOPA_SSH}" 'echo ok' >/dev/null 2>&1
+  resolve_koopa_ssh
 }

 # Run remote bash -s with optional stdin script; hard-capped
@ -302,17 +335,25 @@ koopa_ssh_ok() {
 koopa_ssh_run() {
  local t="${1:-$SSH_CMD_TIMEOUT}"
  shift
+  resolve_koopa_ssh || return 1
  with_timeout "$t" ssh "${SSH_BASE_OPTS[@]}" "${KOOPA_SSH}" "$@"
 }

 koopa_ssh_bash() {
  local t="${1:-$SSH_CMD_TIMEOUT}"
+  resolve_koopa_ssh || return 1
  with_timeout "$t" ssh "${SSH_BASE_OPTS[@]}" "${KOOPA_SSH}" 'bash -s'
 }

+# stdin → remote python3 - (for metrics load probe)
+koopa_ssh_python() {
+  local t="${1:-60}"
+  resolve_koopa_ssh || return 1
+  with_timeout "$t" ssh "${SSH_BASE_OPTS[@]}" "${KOOPA_SSH}" python3 -
+}
+
 # ANSI colours for tags (green OK / yellow WARN / red ERROR·BLOCKER / cyan INFO).
-# Default: always on so pasted ```bash``` logs keep visible tags when the terminal
-# supports colour. Disable: NO_COLOR=1. Force off for dumb pipes: CLICOLOR=0.
+# Default: always on so pasted logs keep visible tags. Disable: NO_COLOR=1 or CLICOLOR=0.
 if [ "${NO_COLOR:-0}" = "1" ] || [ "${CLICOLOR:-1}" = "0" ]; then
  G= R= Y= C= N= B=
 else
@ -359,6 +400,9 @@ _fmt_tid() {
 }

 ok() {
+  # Forms (same idea as info/warn):
+  #   ok "what passed"
+  #   ok "what passed" "detail / ms / bytes / …"
  local label="$1" detail="${2:-}"
  _take_tid
  if [ -n "$detail" ]; then
@ -385,9 +429,28 @@ fail() {
  ERRORS+=("${LAST_TID:+$LAST_TID }$label${detail:+ — $detail}")
 }
 warn() {
-  local label="$1" detail="${2:-}"
+  # Forms (same idea as err; always try to state the problem):
+  #   warn "problem"
+  #   warn "problem" "why / context"
+  #   warn component "problem" "why / context"
+  local a1="${1:-}" a2="${2:-}" a3="${3:-}"
+  local head detail
  _take_tid
-  printf '%s[WARN]%s  %s%s%s\n' "$Y" "$N" "$(_fmt_tid)" "$label" "${detail:+ — $detail}"
+  if [ -n "$a3" ]; then
+    head="${a1}: ${a2}"
+    detail="$a3"
+  elif [ -n "$a2" ]; then
+    head="$a1"
+    detail="$a2"
+  else
+    head="$a1"
+    detail=""
+  fi
+  if [ -n "$detail" ]; then
+    printf '%s[WARN]%s  %s%s — %s\n' "$Y" "$N" "$(_fmt_tid)" "$head" "$detail"
+  else
+    printf '%s[WARN]%s  %s%s\n' "$Y" "$N" "$(_fmt_tid)" "$head"
+  fi
  WARN_N=$((WARN_N + 1))
 }
 info() {
--- a/scripts/taler-monitoring/metrics.sh
+++ b/scripts/taler-monitoring/metrics.sh
--- a/scripts/taler-monitoring/taler-monitoring.sh
+++ b/scripts/taler-monitoring/taler-monitoring.sh
@ -26,6 +26,7 @@ Phases:
  sanity   public + optional server
  server   server-side only (SSH)
  e2e      withdraw + pay (small amounts; remote aborts on login/KYC)
+  ladder   GOA withdraw ladder (auto-account + explorer pool + timings)
  all      urls + inside + versions + sanity + e2e  (SSH phases only on koopa)

 Options:
@ -56,6 +57,10 @@ Env (same meaning):
  TALER_DOMAIN  BANK_PUBLIC  EXCHANGE_PUBLIC  MERCHANT_PUBLIC  EXPECT_CURRENCY
  TALER_DOMAINS_CONF  SKIP_SSH=1
  NO_COLOR=1 / CLICOLOR=0   disable green/yellow/red tags (default: coloured)
+  SKIP_SSH=1    NO_COLOR=1
+  PERF_WARN_MS  PERF_FAIL_MS   (urls latency; default 8000 / 20000)
+  KOOPA_SSH  KOOPA_SSH_FALLBACKS   (default koopa → koopa-external)
+  METRICS_LOAD=0   skip host/container RAM/CPU probes (e2e/ladder/inside)
 EOF
 }

@ -96,7 +101,7 @@ while [ $# -gt 0 ]; do
      CURRENCY_OVERRIDE="$2"; shift 2
      ;;
    --no-probe) NO_PROBE=1; shift ;;
-    urls|inside|versions|sanity|server|e2e|all) PHASES+=("$1"); shift ;;
+    urls|inside|versions|sanity|server|e2e|ladder|goa-ladder|all) PHASES+=("$1"); shift ;;
    *)
      # bare domain shorthand: ./taler-monitoring.sh taler.net
      if [[ "$1" == *.* && "$1" != *://* && "$1" != -* ]]; then
@ -140,6 +145,30 @@ export WITHDRAW_AMT PAY_AMT CREDIT_AMT MERCHANT_INSTANCE
 export E2E_FAKE_INCOMING E2E_REMOTE E2E_VARIABLE E2E_ATM_MAX
 export E2E_WITHDRAW_VALUES E2E_PAY_VALUES
 export PAIVANA_PUBLIC E2E_PAIVANA E2E_PAIVANA_TEMPLATE E2E_PAIVANA_AMOUNT E2E_PAIVANA_INSTANCE
+# Ladder: withdraw then pay — 0 + random mids + max-1 + max (see check_goa_ladder.sh).
+# Defaults so set -u export is safe when vars were never set by caller.
+: "${LADDER_STEPS:=23}"
+: "${LADDER_MIN_AMOUNT:=0.000001}"
+: "${LADDER_CONFIRM_POLLS:=40}"
+: "${LADDER_MAX_RUNGS:=99}"
+: "${LADDER_TIMEOUT_S:=3600}"
+: "${LADDER_REPORT_DIR:=}"
+: "${LADDER_SETTLE_ROUNDS:=18}"
+: "${LADDER_SETTLE_SLEEP:=2}"
+: "${LADDER_MAX_AMOUNT:=4503599627370496}"
+: "${LADDER_INCLUDE_ZERO:=1}"
+: "${LADDER_INCLUDE_MAX:=1}"
+: "${LADDER_HIGH_FROM:=1000000}"
+: "${LADDER_HIGH_RUNGS:=12}"
+: "${LADDER_LOAD:=1}"
+: "${LADDER_PAY:=1}"
+: "${LADDER_WITHDRAW_SCALE:=1.5}"
+: "${LADDER_PAY_SETTLE_ROUNDS:=6}"
+export LADDER_STEPS LADDER_MIN_AMOUNT LADDER_CONFIRM_POLLS LADDER_MAX_RUNGS LADDER_TIMEOUT_S LADDER_REPORT_DIR
+export LADDER_SETTLE_ROUNDS LADDER_SETTLE_SLEEP EXP_PW_FILE EXP_USER
+export LADDER_MAX_AMOUNT LADDER_INCLUDE_ZERO LADDER_INCLUDE_MAX
+export LADDER_HIGH_FROM LADDER_HIGH_RUNGS LADDER_LOAD
+export LADDER_PAY LADDER_WITHDRAW_SCALE LADDER_PAY_SETTLE_ROUNDS
 export TALER_DOMAIN_APPLIED=1

 # Default phases
@ -200,6 +229,7 @@ for p in "${PHASES[@]}"; do
    sanity)   "$ROOT/check_sanity.sh"   || ec=1 ;;
    server)   "$ROOT/check_server.sh"   || ec=1 ;;
    e2e)      "$ROOT/check_e2e.sh"      || ec=1 ;;
+    ladder|goa-ladder) "$ROOT/check_goa_ladder.sh" || ec=1 ;;
  esac
 done
 exit "$ec"