ops: castopod/bonfire notes; amount-ladder bench

2026-07-10 11:33:09 +02:00 · 2026-07-10 11:33:09 +02:00 · 84388ae6a9
commit 84388ae6a9
parent acff09d130
9 changed files with 749 additions and 3 deletions
--- a/benchmarks/README.md
+++ b/benchmarks/README.md
@ -0,0 +1,11 @@
+# Benchmarks (koopa Taler stack)
+
+Run **from the local machine** against public endpoints.  
+**Measure load inside** the three Podman containers on koopa via `ssh hernani@koopa` + `podman exec` (no root/screen required for metrics).
+
+| Benchmark | What it does |
+|-----------|----------------|
+| [`amount-ladder/`](amount-ladder/) | Free-amount template payments across a wide GOA ladder; wall times + host/container CPU/RSS samples |
+
+Secrets: sibling repo **`koopa-admin-secrets`** (never committed here).  
+Ephemeral run output: **`koopa-admin-log/.tmp/`** (gitignored).
--- a/benchmarks/amount-ladder/README.md
+++ b/benchmarks/amount-ladder/README.md
@ -0,0 +1,90 @@
+# Amount-ladder payment benchmark
+
+## Model
+
+| Side | Machine | Role |
+|------|---------|------|
+| **Drive (external)** | Local machine (this tree) | `taler-wallet-cli`, orchestration, CSV/RESULTS |
+| **SUT (internal)** | koopa containers | merchant / exchange / bank under load |
+| **Measure (internal)** | `ssh hernani@koopa` | `loadavg`, `ps` in each container |
+
+```
+[ Local-machine wallet-cli ] --HTTPS--> taler.hacktivism.ch / exchange / bank
+       |                              ^
+       | ssh sample_load              | pasta :9010/:9011/:9012
+       v                              |
+[ hernani@koopa ] --podman exec--> process CPU/RSS snapshots
+```
+
+## Prerequisites
+
+1. Stack healthy (`check_*-health.sh` all green; public `/config` + exchange `/keys` 200).
+2. Local wallet CLI (default path in script, override `MONO=`).
+3. Secrets in **`../koopa-admin-secrets/koopa/host-root/`**:
+   - `taler-bank/bank-admin-password.txt`
+   - `taler-bank/bank-bench-fat-password.txt` (created if missing)
+   - `taler-merchant/merchant-goa-demo-cp4zqk-password.txt`
+4. SSH: `hernani@koopa` BatchMode, `podman` without su.
+
+## Amount ladder (~20 payments: small / medium / large)
+
+| Band | Amounts (GOA) |
+|------|----------------|
+| **small** | `0.000001` `0.00001` `0.0001` `0.001` `0.01` `0.05` `0.1` `0.5` |
+| **medium** | `1` `2` `5` `10` `25` `50` `100` |
+| **large** | `250` `500` `1000` `2500` `5000` |
+
+Default list (20 steps):
+
+```text
+GOA:0.000001 GOA:0.00001 GOA:0.0001 GOA:0.001 GOA:0.01 GOA:0.05 GOA:0.1 GOA:0.5
+GOA:1 GOA:2 GOA:5 GOA:10 GOA:25 GOA:50 GOA:100
+GOA:250 GOA:500 GOA:1000 GOA:2500 GOA:5000
+```
+
+Σ ladder ≈ **GOA:9433** → default withdraw **GOA:50000**, admin credit **GOA:200000**.
+
+Override: `AMOUNTS='GOA:1 GOA:10' ./run.sh`
+
+## Pay-template URI (important)
+
+Do **not** put a trailing slash after the template id (wallet then 404s).
+
+```text
+taler://pay-template/taler.hacktivism.ch/instances/goa-demo-cp4zqk/goa-free?amount=GOA:1
+```
+
+- Template `goa-free` is free-amount with fixed summary `"pay"` → pass **only** `amount=` (no `summary=` or merchant 409).
+- HTTP GET probe:  
+  `https://taler.hacktivism.ch/instances/goa-demo-cp4zqk/templates/goa-free` → 200.
+
+## Run
+
+```bash
+cd koopa-admin-log/benchmarks/amount-ladder
+./run.sh
+# short:
+AMOUNTS='GOA:0.05 GOA:1 GOA:10' WITHDRAW_AMT=GOA:100 ./run.sh
+```
+
+Outputs under `../../.tmp/amount-bench-<UTC>/`:
+
+| File | Content |
+|------|---------|
+| `payments.csv` | per-amount timings + status |
+| `RESULTS.md` | table summary |
+| `koopa-load-*.out` | **inside** process samples |
+| `run.log` | timeline |
+
+## Columns (RESULTS)
+
+- **handle**: wall time of `wallet handle-uri`
+- **run-until-done**: wallet background after handle
+- **total**: sum
+- **loadavg**: koopa 1‑min load after payment
+- **status**: `paid` / `error` / `insufficient` / …
+
+## Related
+
+- Partial earlier run (wrong URI): `2026/2026-07-09--amount-ladder-partial-results.md`
+- Instance notes: `configs/taler-merchant/demo-instance-goa-demo-cp4zqk.md`
--- a/benchmarks/amount-ladder/run.sh
+++ b/benchmarks/amount-ladder/run.sh
@ -0,0 +1,375 @@
+#!/bin/bash
+# Amount-ladder benchmark — drive from the local machine, measure on koopa (inside containers).
+# See README.md in this directory.
+set -eu
+
+BENCH_DIR=$(cd "$(dirname "$0")" && pwd)
+ROOT=$(cd "$BENCH_DIR/../.." && pwd)
+SECRETS="${SECRETS_DIR:-}"
+if [ -z "$SECRETS" ]; then
+  if [ -d "$ROOT/../koopa-admin-secrets" ]; then
+    SECRETS=$(cd "$ROOT/../koopa-admin-secrets" && pwd)
+  else
+    SECRETS="$ROOT/../koopa-admin-secrets"
+  fi
+fi
+
+SCRATCH="${ROOT}/.tmp/amount-bench-$(date -u +%Y%m%d-%H%M%S)"
+mkdir -p "$SCRATCH"
+chmod 700 "$SCRATCH"
+
+MONO="${MONO:-/Users/newkamek/src/taler/taler-typescript-core/packages/taler-wallet-cli/bin/taler-wallet-cli.mjs}"
+WDB="$SCRATCH/wallet.sqlite3"
+EX="${EXCHANGE_URL:-https://exchange.hacktivism.ch/}"
+KOOPA="${KOOPA_HOST:-koopa}"
+INST="${MERCHANT_INSTANCE:-goa-demo-cp4zqk}"
+TPL="${TEMPLATE_ID:-goa-free}"
+MER_HOST="${MER_HOST:-taler.hacktivism.ch}"
+FAT_USER="${FAT_USER:-bench-fat}"
+# ~20 payments: small → medium → large (atomic/tiny through multi-kilo)
+# Override: AMOUNTS='GOA:1 GOA:10' ./run.sh
+AMOUNTS="${AMOUNTS:-GOA:0.000001 GOA:0.00001 GOA:0.0001 GOA:0.001 GOA:0.01 GOA:0.05 GOA:0.1 GOA:0.5 GOA:1 GOA:2 GOA:5 GOA:10 GOA:25 GOA:50 GOA:100 GOA:250 GOA:500 GOA:1000 GOA:2500 GOA:5000}"
+# sum of default ladder ≈ GOA:9433 → withdraw headroom for multi-coin + fees
+WITHDRAW_AMT="${WITHDRAW_AMT:-GOA:50000}"
+CREDIT_AMT="${CREDIT_AMT:-GOA:200000}"
+
+BANK_ADMIN_PW_FILE="${BANK_ADMIN_PW_FILE:-$SECRETS/koopa/host-root/taler-bank/bank-admin-password.txt}"
+BANK_FAT_PW_FILE="${BANK_FAT_PW_FILE:-$SECRETS/koopa/host-root/taler-bank/bank-bench-fat-password.txt}"
+MER_PW_FILE="${MER_PW_FILE:-$SECRETS/koopa/host-root/taler-merchant/merchant-${INST}-password.txt}"
+
+wcli() { node "$MONO" --wallet-db="$WDB" --no-throttle --skip-defaults "$@"; }
+log() { echo "$(date -u +%H:%M:%S) $*" | tee -a "$SCRATCH/run.log"; }
+now() { python3 -c 'import time;print(time.time())'; }
+die() { log "FAIL $*"; exit 1; }
+
+[ -f "$MONO" ] || die "wallet cli missing: $MONO"
+[ -f "$BANK_ADMIN_PW_FILE" ] || die "missing $BANK_ADMIN_PW_FILE"
+[ -f "$MER_PW_FILE" ] || die "missing $MER_PW_FILE"
+
+ADMIN_PW=$(tr -d '\n' <"$BANK_ADMIN_PW_FILE")
+MER_PW=$(tr -d '\n' <"$MER_PW_FILE")
+if [ -f "$BANK_FAT_PW_FILE" ]; then
+  FAT_PW=$(tr -d '\n' <"$BANK_FAT_PW_FILE")
+else
+  FAT_PW=$(openssl rand -hex 12)
+  mkdir -p "$(dirname "$BANK_FAT_PW_FILE")"
+  printf '%s\n' "$FAT_PW" >"$BANK_FAT_PW_FILE"
+  chmod 600 "$BANK_FAT_PW_FILE"
+  log "generated fat password -> $BANK_FAT_PW_FILE (secrets repo)"
+fi
+
+# --- remote: run short script as hernani, capture stdout ---
+run_koopa() {
+  local id="$1"
+  shift
+  local script_local="$SCRATCH/remote-$id.sh"
+  local mark="BENCH_${id}_OK"
+  {
+    echo '#!/bin/bash'
+    echo 'set +e'
+    printf '%s\n' "$@"
+    echo "echo $mark"
+  } >"$script_local"
+  scp -o BatchMode=yes -q "$script_local" "${KOOPA}:/tmp/bench-run-$id.sh"
+  ssh -o BatchMode=yes "$KOOPA" "bash /tmp/bench-run-$id.sh; rm -f /tmp/bench-run-$id.sh" \
+    >"$SCRATCH/koopa-$id.out" 2>&1 || true
+  if ! grep -q "$mark" "$SCRATCH/koopa-$id.out" 2>/dev/null; then
+    log "WARN run_koopa $id incomplete (see koopa-$id.out)"
+    return 1
+  fi
+  return 0
+}
+
+# Measure **inside** containers (and host loadavg)
+sample_load() {
+  local tag="$1"
+  scp -o BatchMode=yes -q "$BENCH_DIR/sample-load-inside.sh" "${KOOPA}:/tmp/sample-load-inside.sh"
+  ssh -o BatchMode=yes "$KOOPA" "bash /tmp/sample-load-inside.sh $(printf %q "$tag"); rm -f /tmp/sample-load-inside.sh" \
+    >"$SCRATCH/koopa-load-$tag.out" 2>&1 || true
+}
+
+log "scratch=$SCRATCH"
+log "AMOUNTS=$AMOUNTS"
+
+# --- preflight (external, public HTTPS) ---
+for url in \
+  "https://taler.hacktivism.ch/config" \
+  "https://exchange.hacktivism.ch/keys" \
+  "https://taler.hacktivism.ch/instances/${INST}/templates/${TPL}"
+do
+  code=$(curl -sk -o /dev/null -w '%{http_code}' -m 10 "$url" || echo err)
+  log "preflight $url -> $code"
+  [ "$code" = "200" ] || die "preflight failed $url"
+done
+
+# --- bank setup on koopa (API localhost, orchestrated from the local machine) ---
+log "=== bank setup $FAT_USER credit=$CREDIT_AMT withdraw=$WITHDRAW_AMT ==="
+{
+  echo '#!/bin/bash'
+  echo 'set -euo pipefail'
+  echo "export ADMIN_PW=$(printf %q "$ADMIN_PW")"
+  echo "export FAT_PW=$(printf %q "$FAT_PW")"
+  echo "export FAT_USER=$(printf %q "$FAT_USER")"
+  echo "export CREDIT_AMT=$(printf %q "$CREDIT_AMT")"
+  echo "export WITHDRAW_AMT=$(printf %q "$WITHDRAW_AMT")"
+  cat <<'EOS'
+BANK=http://127.0.0.1:9012
+AT=$(curl -sS -u "admin:${ADMIN_PW}" -H "Content-Type: application/json" -d '{"scope":"readwrite"}' \
+  "$BANK/accounts/admin/token" | python3 -c "import sys,json;print(json.load(sys.stdin)['access_token'])")
+echo admin_tok_len=${#AT}
+curl -sS -o /tmp/acc.json -w "create:%{http_code}\n" -X POST -H "Authorization: Bearer $AT" \
+  -H "Content-Type: application/json" \
+  -d "{\"username\":\"${FAT_USER}\",\"password\":\"${FAT_PW}\",\"name\":\"Bench Fat\",\"is_public\":false,\"is_taler_exchange\":false,\"debit_threshold\":\"GOA:0\"}" \
+  "$BANK/accounts" || true
+echo create_body=$(head -c 160 /tmp/acc.json)
+python3 - <<'PY'
+import json, os
+A="0123456789ABCDEFGHJKMNPQRSTVWXYZ"
+def c32(b):
+  v=bits=0; o=[]
+  for x in b:
+    v=(v<<8)|x; bits+=8
+    while bits>=5:
+      bits-=5; o.append(A[(v>>bits)&31])
+  if bits: o.append(A[(v<<(5-bits))&31])
+  return "".join(o)
+uid=c32(os.urandom(32))
+user=os.environ["FAT_USER"]
+amt=os.environ["CREDIT_AMT"]
+payto=f"payto://x-taler-bank/bank.hacktivism.ch/{user}?receiver-name={user}&message=bench-credit"
+json.dump({"payto_uri":payto,"amount":amt,"request_uid":uid}, open("/tmp/tx.json","w"))
+print("credit", amt, "->", user, "uid", uid)
+PY
+curl -sS -o /tmp/cred.json -w "credit:%{http_code}\n" -X POST -H "Authorization: Bearer $AT" \
+  -H "Content-Type: application/json" -d @/tmp/tx.json "$BANK/accounts/admin/transactions"
+echo credit_body=$(head -c 200 /tmp/cred.json)
+BT=$(curl -sS -u "${FAT_USER}:${FAT_PW}" -H "Content-Type: application/json" -d '{"scope":"readwrite"}' \
+  "$BANK/accounts/${FAT_USER}/token" | python3 -c "import sys,json;print(json.load(sys.stdin)['access_token'])")
+echo fat_tok_len=${#BT}
+curl -sS -H "Authorization: Bearer $BT" "$BANK/accounts/${FAT_USER}" -o /tmp/fatbal.json
+echo fat_account=$(head -c 280 /tmp/fatbal.json)
+curl -sS -o /tmp/wd.json -w "wd:%{http_code}\n" -X POST -H "Authorization: Bearer $BT" \
+  -H "Content-Type: application/json" -d "{\"amount\":\"${WITHDRAW_AMT}\"}" \
+  "$BANK/accounts/${FAT_USER}/withdrawals"
+python3 - <<'PY'
+import json
+d=json.load(open("/tmp/wd.json"))
+print("wd_keys", list(d.keys())[:20])
+uri=d.get("taler_withdraw_uri") or d.get("withdrawal_uri") or ""
+wid=d.get("withdrawal_id") or d.get("id") or ""
+if not uri:
+  for v in d.values():
+    if isinstance(v, str) and v.startswith("taler"):
+      uri=v
+open("/tmp/bench-wuri.txt","w").write(uri)
+open("/tmp/bench-wid.txt","w").write(wid)
+print("WURI", uri)
+print("WID", wid)
+assert uri and wid, d
+PY
+# cleanup setup temps on host (not durable)
+rm -f /tmp/acc.json /tmp/tx.json /tmp/cred.json /tmp/fatbal.json /tmp/wd.json
+echo BENCH_banksetup_OK
+EOS
+} >"$SCRATCH/remote-banksetup.sh"
+scp -o BatchMode=yes -q "$SCRATCH/remote-banksetup.sh" "${KOOPA}:/tmp/bench-run-banksetup.sh"
+ssh -o BatchMode=yes "$KOOPA" 'bash /tmp/bench-run-banksetup.sh; rm -f /tmp/bench-run-banksetup.sh' \
+  | tee "$SCRATCH/koopa-banksetup.out"
+grep -q BENCH_banksetup_OK "$SCRATCH/koopa-banksetup.out" || die "bank setup failed"
+scp -o BatchMode=yes -q "${KOOPA}:/tmp/bench-wuri.txt" "${KOOPA}:/tmp/bench-wid.txt" "$SCRATCH/"
+ssh -o BatchMode=yes "$KOOPA" 'rm -f /tmp/bench-wuri.txt /tmp/bench-wid.txt' || true
+WURI=$(tr -d '\n' <"$SCRATCH/bench-wuri.txt")
+WID=$(tr -d '\n' <"$SCRATCH/bench-wid.txt")
+[ -n "$WURI" ] || die "missing withdraw URI"
+log "WURI=$WURI WID=$WID"
+
+log "=== load baseline (internal) ==="
+sample_load baseline
+
+log "=== wallet: exchange + withdraw (external) ==="
+wcli exchanges add "$EX" 2>&1 | tee "$SCRATCH/ex-add.out" || true
+wcli exchanges update "$EX" 2>&1 | tee "$SCRATCH/ex-upd.out" || true
+wcli exchanges tos "$EX" 2>&1 | tee "$SCRATCH/ex-tos.out" || true
+wcli exchanges accept-tos "$EX" 2>&1 | tee "$SCRATCH/ex-accept.out" || true
+
+TW0=$(now)
+# 1) Wallet selects exchange (status → selected). Must happen BEFORE bank confirm.
+wcli withdraw accept-uri --exchange "$EX" "$WURI" 2>&1 | tee "$SCRATCH/wd-accept.out" || true
+
+# 2) Wait until bank sees selection, then confirm once (libeufin rejects confirm if still unselected).
+log "=== bank: wait selected → confirm ==="
+run_koopa wdconfirm \
+  "export FAT_USER=$(printf %q "$FAT_USER")" \
+  "export FAT_PW=$(printf %q "$FAT_PW")" \
+  "export WID=$(printf %q "$WID")" \
+  'BANK=http://127.0.0.1:9012' \
+  'BT=$(curl -sS -u "${FAT_USER}:${FAT_PW}" -H "Content-Type: application/json" -d "{\"scope\":\"readwrite\"}" "$BANK/accounts/${FAT_USER}/token" | python3 -c "import sys,json;print(json.load(sys.stdin).get(\"access_token\",\"\"))")' \
+  'echo WID=$WID tok_len=${#BT}' \
+  'st=""' \
+  'for i in $(seq 1 60); do
+     curl -sS "$BANK/taler-integration/withdrawal-operation/${WID}" -o /tmp/wdst.json || true
+     st=$(python3 -c "import json;print(json.load(open(\"/tmp/wdst.json\")).get(\"status\",\"\"))" 2>/dev/null || true)
+     echo "wait_select #$i status=$st"
+     case "$st" in selected|confirmed) break ;; esac
+     sleep 1
+   done' \
+  'case "$st" in
+     selected)
+       echo confirming...
+       curl -sS -o /tmp/wdconf.out -w "confirm_http:%{http_code}\n" -X POST \
+         -H "Authorization: Bearer $BT" -H "Content-Type: application/json" -d "{}" \
+         "$BANK/accounts/${FAT_USER}/withdrawals/${WID}/confirm"
+       head -c 120 /tmp/wdconf.out; echo
+       ;;
+     confirmed) echo already_confirmed ;;
+     *) echo "WARN never selected (last=$st)" ;;
+   esac' \
+  'for i in $(seq 1 30); do
+     curl -sS "$BANK/taler-integration/withdrawal-operation/${WID}" -o /tmp/wdst.json || true
+     python3 -c "import json;d=json.load(open(\"/tmp/wdst.json\"));print(\"post\",d.get(\"status\"),\"transfer\",d.get(\"transfer_done\"))"
+     st=$(python3 -c "import json;print(json.load(open(\"/tmp/wdst.json\")).get(\"status\",\"\"))" 2>/dev/null || true)
+     td=$(python3 -c "import json;print(json.load(open(\"/tmp/wdst.json\")).get(\"transfer_done\"))" 2>/dev/null || true)
+     case "$st" in confirmed) break ;; esac
+     [ "$td" = "True" ] || [ "$td" = "true" ] && break
+     sleep 1
+   done
+   rm -f /tmp/wdst.json /tmp/wdconf.out' || true
+
+# 3) Wallet finishes planchets (may take minutes for large withdraws)
+log "=== wallet: run-until-done after confirm ==="
+for round in $(seq 1 30); do
+  log "withdraw run-until-done round $round"
+  # timeout each round so we do not hang forever on long-poll
+  set +e
+  # 120s max per round
+  perl -e 'alarm shift; exec @ARGV' 120 \
+    node "$MONO" --wallet-db="$WDB" --no-throttle --skip-defaults run-until-done \
+    2>&1 | tee -a "$SCRATCH/wd-rud.out"
+  set -e
+  wcli balance 2>&1 | tee "$SCRATCH/bal-wd-$round.out"
+  if python3 - "$SCRATCH/bal-wd-$round.out" <<'PY'
+import re,sys
+t=open(sys.argv[1]).read()
+vals=[float(x) for x in re.findall(r"GOA:([0-9.]+)", t)]
+sys.exit(0 if vals and max(vals) >= 100 else 1)
+PY
+  then
+    log "withdraw OK"
+    break
+  fi
+  sleep 1
+done
+TW1=$(now)
+python3 -c "print('withdraw_wall_s', round(float('$TW1')-float('$TW0'), 3))" | tee "$SCRATCH/withdraw-time.txt"
+wcli balance 2>&1 | tee "$SCRATCH/balance-before-pays.out"
+sample_load after-withdraw
+
+CSV="$SCRATCH/payments.csv"
+echo "idx,amount,handle_s,rud_s,total_s,status,order_id,loadavg,notes" >"$CSV"
+
+idx=0
+for AMT in $AMOUNTS; do
+  idx=$((idx + 1))
+  # no trailing slash; amount only (template has fixed summary)
+  URI="taler://pay-template/${MER_HOST}/instances/${INST}/${TPL}?amount=${AMT}"
+  echo "$URI" >"$SCRATCH/uri-$idx.txt"
+  log "=== pay #$idx $AMT ==="
+  log "uri=$URI"
+
+  load_before=$(ssh -o BatchMode=yes "$KOOPA" 'cut -d" " -f1 /proc/loadavg' 2>/dev/null || echo "?")
+  sample_load "pay$idx" &
+  SPID=$!
+
+  T0=$(now)
+  set +e
+  wcli handle-uri --yes --non-interactive "$URI" 2>&1 | tee "$SCRATCH/pay-$idx-handle.out"
+  T1=$(now)
+  wcli run-until-done 2>&1 | tee "$SCRATCH/pay-$idx-rud.out"
+  T2=$(now)
+  set -e
+
+  handle_s=$(python3 -c "print(round(float('$T1')-float('$T0'),3))")
+  rud_s=$(python3 -c "print(round(float('$T2')-float('$T1'),3))")
+  total_s=$(python3 -c "print(round(float('$T2')-float('$T0'),3))")
+  wait "$SPID" 2>/dev/null || true
+  load_after=$(ssh -o BatchMode=yes "$KOOPA" 'cut -d" " -f1 /proc/loadavg' 2>/dev/null || echo "?")
+
+  run_koopa "ord$idx" \
+    "export MPW=$(printf %q "$MER_PW")" \
+    "export INST=$(printf %q "$INST")" \
+    "export AMT=$(printf %q "$AMT")" \
+    'AUTH="Authorization: Bearer secret-token:${MPW}"' \
+    'curl -sk -H "$AUTH" "https://127.0.0.1:9010/instances/${INST}/private/orders?paid=YES&delta=-30" -o /tmp/ol.json' \
+    'python3 - <<PY
+import json, os
+d=json.load(open("/tmp/ol.json"))
+want=os.environ["AMT"]
+hit=None
+for o in d.get("orders") or []:
+  if o.get("amount")==want:
+    hit=o; break
+if hit:
+  print("ORDER", hit.get("order_id"), "amount", hit.get("amount"), "summary", hit.get("summary"))
+  open("/tmp/oid.txt","w").write(hit.get("order_id") or "")
+else:
+  print("NO_ORDER")
+  open("/tmp/oid.txt","w").write("")
+PY' \
+    'OID=$(tr -d "\n" </tmp/oid.txt 2>/dev/null || true)' \
+    'if [ -n "$OID" ]; then
+       curl -sk -H "$AUTH" "https://127.0.0.1:9010/instances/${INST}/private/orders/${OID}" -o /tmp/od.json
+       python3 -c "import json;d=json.load(open(\"/tmp/od.json\"));print(\"DETAIL paid\",d.get(\"paid\"),\"status\",d.get(\"order_status\"),\"amount\",(d.get(\"contract_terms\")or{}).get(\"amount\"))"
+     fi
+     rm -f /tmp/ol.json /tmp/od.json /tmp/oid.txt' || true
+
+  status=unknown
+  if grep -qE 'DETAIL paid True|DETAIL paid true' "$SCRATCH/koopa-ord$idx.out" 2>/dev/null; then
+    status=paid
+  elif grep -qiE 'insufficient balance' "$SCRATCH/pay-$idx-handle.out" 2>/dev/null; then
+    status=insufficient
+  elif grep -qiE 'error|fail|invalid|404|409|Unexpected HTTP' "$SCRATCH/pay-$idx-handle.out" 2>/dev/null; then
+    status=error
+  elif grep -qi 'paid' "$SCRATCH/pay-$idx-rud.out" 2>/dev/null; then
+    status=maybe
+  fi
+  oid=$(awk '/^ORDER /{print $2; exit}' "$SCRATCH/koopa-ord$idx.out" 2>/dev/null || true)
+  notes="load ${load_before}->${load_after}"
+  echo "$idx,$AMT,$handle_s,$rud_s,$total_s,$status,$oid,$load_after,$notes" >>"$CSV"
+  log "pay #$idx $AMT total=${total_s}s status=$status oid=$oid load=$load_after"
+  wcli balance 2>&1 | tee "$SCRATCH/bal-after-$idx.out" || true
+done
+
+sample_load final
+wcli balance 2>&1 | tee "$SCRATCH/balance-final.out" || true
+
+python3 - "$CSV" "$SCRATCH" <<'PY' | tee "$SCRATCH/RESULTS.md"
+import csv, sys
+csv_path, scratch = sys.argv[1], sys.argv[2]
+rows = list(csv.DictReader(open(csv_path)))
+print("# Amount ladder payment benchmark")
+print()
+print(f"- Scratch: `{scratch}`")
+print(f"- Drive: Local-machine wallet-cli (external) · Measure: koopa podman (internal)")
+print(f"- Template: `goa-free` on `goa-demo-cp4zqk`")
+print(f"- URI: `taler://pay-template/…/instances/…/goa-free?amount=…` (no trailing slash, no summary)")
+print()
+print("| # | Amount | handle (s) | run-until-done (s) | **total (s)** | status | order | loadavg |")
+print("|--:|--------|----------:|-------------------:|--------------:|--------|-------|--------:|")
+for r in rows:
+  print(f"| {r['idx']} | `{r['amount']}` | {r['handle_s']} | {r['rud_s']} | **{r['total_s']}** | {r['status']} | `{r.get('order_id','')}` | {r.get('loadavg','')} |")
+print()
+paid = [r for r in rows if r["status"] == "paid"]
+print(f"Paid **{len(paid)}/{len(rows)}**.")
+if paid:
+  totals = [float(r["total_s"]) for r in paid if r["total_s"]]
+  if totals:
+    print(f"**Min** {min(totals):.3f}s · **Max** {max(totals):.3f}s · **Avg** {sum(totals)/len(totals):.3f}s")
+print()
+print("Inside samples: `koopa-load-*.out` in scratch.")
+PY
+
+# durable copy of last RESULTS into benchmarks tree (no secrets)
+cp "$SCRATCH/RESULTS.md" "$BENCH_DIR/LAST_RESULTS.md"
+cp "$CSV" "$BENCH_DIR/LAST_payments.csv"
+log "DONE -> $SCRATCH/RESULTS.md (also benchmarks/amount-ladder/LAST_*)"
+cat "$SCRATCH/RESULTS.md"
--- a/benchmarks/amount-ladder/sample-load-inside.sh
+++ b/benchmarks/amount-ladder/sample-load-inside.sh
@ -0,0 +1,26 @@
+#!/bin/bash
+# Sample host + three-container CPU/RSS from koopa (run as hernani via ssh).
+# Usage (local):  ssh hernani@koopa 'bash -s' < sample-load-inside.sh
+# Or:            ./sample-load-inside.sh   # if executed ON koopa as hernani
+set -eu
+tag="${1:-manual}"
+echo "=== sample_load tag=$tag $(date -u +%FT%TZ) ==="
+echo "=== loadavg ==="
+cat /proc/loadavg
+echo "=== mem ==="
+free -h 2>/dev/null || true
+echo "=== top host cpu ==="
+ps -eo pid,pcpu,pmem,rss,etime,comm --sort=-pcpu 2>/dev/null | head -15
+echo "=== exchange (taler-exchange-hacktivism) ==="
+podman exec taler-exchange-hacktivism \
+  ps -eo pid,pcpu,pmem,rss,etime,args 2>/dev/null \
+  | grep -E 'taler-exchange|postgres -D|PID' | head -25 || true
+echo "=== merchant (taler-hacktivism) ==="
+podman exec taler-hacktivism \
+  ps -eo pid,pcpu,pmem,rss,etime,args 2>/dev/null \
+  | grep -E 'taler-merchant|nginx|postgres -D' | head -25 || true
+echo "=== bank (taler-bank-hacktivism) ==="
+podman exec taler-bank-hacktivism \
+  ps -eo pid,pcpu,pmem,rss,etime,args 2>/dev/null \
+  | grep -E 'java|libeufin|postgres -D' | head -15 || true
+echo "=== sample_load done ==="