ops: castopod/bonfire notes; amount-ladder bench

README.md

@@ -1,7 +1,52 @@
 # koopa-admin-log
 
-Ops log and config mirror for host **koopa**.
+Ops log and config mirror for host **koopa** (openSUSE Tumbleweed).
 
-**Day logs:** `2026/` (major only, same-day).
+**Day logs:** `2026/` (major only, written same day). Topic notes: `YYYY-MM-DD--topic.md`.
 
-**Secrets:** see `SECRETS.md` / `koopa-admin-secrets`.
+**Secrets:** not in this repo — see **`SECRETS.md`** and sibling **`koopa-admin-secrets`**.
+
+## Layout
+
+```
+host/                      # openSUSE host-level config + overview
+  overview/services.md     # graphical service map
+  systemd/ firewalld/ caddy/ network/
+configs/                   # mirrored app configs (caddy, taler, …)
+  taler-merchant/          # overrides + nginx (same tree style as exchange)
+  taler-exchange/          # overrides + coins
+scripts/                   # merchant, exchange, monitoring helpers
+  taler-merchant/          # merchant-model start scripts
+  taler-exchange/          # same model; archive/ = greenfield bootstrap
+2026/                      # day logs + dated topic notes
+```
+
+## Live containers (podman)
+
+| Name | Image / role | Host port |
+|------|----------------|-----------|
+| `taler-hacktivism` | merchant | **9010** |
+| `taler-exchange-hacktivism` | exchange | **9011** |
+| `taler-bank-hacktivism` | libeufin-bank (GOA, no IBAN) | **9012** |
+| `koopa-castopod` (+ mariadb/redis) | Castopod Fediverse podcast | **9020** |
+
+Start model (Taler): **root** `/root/start_base_services_*` → shell as service user → `/usr/local/bin/start_*.sh [--restart]`.  
+Start model (Castopod): `hernani` → `cd ~/koopa-castopod && podman-compose up -d`.
+
+Public hosts: `taler.` / `exchange.` / `bank.` / **`castopod.hacktivism.ch`** (Caddy → 9010/9011/9012/**9020**).
+
+Service **details**: `configs/taler-*`, `scripts/*`.  
+Host-wide picture: **`host/overview/services.md`**.  
+Router: **`../vecigate-admin-log`**.
+
+**Benchmarks** (drive from the local machine, measure on koopa): **`benchmarks/`**  
+→ amount ladder: `benchmarks/amount-ladder/`.
+
+**Exchange snapshot/restore (GOA, local images only):**  
+→ `2026/2026-07-09--exchange-snapshot-and-restore.md`
+
+**Exchange `/keys` + Wire + bank account (diagram):**  
+`configs/taler-exchange/README.md` (section "/keys, Wire and Account").
+
+**Castopod pilot content (FOSS Airwaves, user ngi, free licenses):**  
+`2026/2026-07-09--castopod-content.md` → https://castopod.hacktivism.ch/@foss

benchmarks/README.md

@@ -0,0 +1,11 @@
+# Benchmarks (koopa Taler stack)
+
+Run **from the local machine** against public endpoints.  
+**Measure load inside** the three Podman containers on koopa via `ssh hernani@koopa` + `podman exec` (no root/screen required for metrics).
+
+| Benchmark | What it does |
+|-----------|----------------|
+| [`amount-ladder/`](amount-ladder/) | Free-amount template payments across a wide GOA ladder; wall times + host/container CPU/RSS samples |
+
+Secrets: sibling repo **`koopa-admin-secrets`** (never committed here).  
+Ephemeral run output: **`koopa-admin-log/.tmp/`** (gitignored).

benchmarks/amount-ladder/README.md

@@ -0,0 +1,90 @@
+# Amount-ladder payment benchmark
+
+## Model
+
+| Side | Machine | Role |
+|------|---------|------|
+| **Drive (external)** | Local machine (this tree) | `taler-wallet-cli`, orchestration, CSV/RESULTS |
+| **SUT (internal)** | koopa containers | merchant / exchange / bank under load |
+| **Measure (internal)** | `ssh hernani@koopa` | `loadavg`, `ps` in each container |
+
+```
+[ Local-machine wallet-cli ] --HTTPS--> taler.hacktivism.ch / exchange / bank
+       |                              ^
+       | ssh sample_load              | pasta :9010/:9011/:9012
+       v                              |
+[ hernani@koopa ] --podman exec--> process CPU/RSS snapshots
+```
+
+## Prerequisites
+
+1. Stack healthy (`check_*-health.sh` all green; public `/config` + exchange `/keys` 200).
+2. Local wallet CLI (default path in script, override `MONO=`).
+3. Secrets in **`../koopa-admin-secrets/koopa/host-root/`**:
+   - `taler-bank/bank-admin-password.txt`
+   - `taler-bank/bank-bench-fat-password.txt` (created if missing)
+   - `taler-merchant/merchant-goa-demo-cp4zqk-password.txt`
+4. SSH: `hernani@koopa` BatchMode, `podman` without su.
+
+## Amount ladder (~20 payments: small / medium / large)
+
+| Band | Amounts (GOA) |
+|------|----------------|
+| **small** | `0.000001` `0.00001` `0.0001` `0.001` `0.01` `0.05` `0.1` `0.5` |
+| **medium** | `1` `2` `5` `10` `25` `50` `100` |
+| **large** | `250` `500` `1000` `2500` `5000` |
+
+Default list (20 steps):
+
+```text
+GOA:0.000001 GOA:0.00001 GOA:0.0001 GOA:0.001 GOA:0.01 GOA:0.05 GOA:0.1 GOA:0.5
+GOA:1 GOA:2 GOA:5 GOA:10 GOA:25 GOA:50 GOA:100
+GOA:250 GOA:500 GOA:1000 GOA:2500 GOA:5000
+```
+
+Σ ladder ≈ **GOA:9433** → default withdraw **GOA:50000**, admin credit **GOA:200000**.
+
+Override: `AMOUNTS='GOA:1 GOA:10' ./run.sh`
+
+## Pay-template URI (important)
+
+Do **not** put a trailing slash after the template id (wallet then 404s).
+
+```text
+taler://pay-template/taler.hacktivism.ch/instances/goa-demo-cp4zqk/goa-free?amount=GOA:1
+```
+
+- Template `goa-free` is free-amount with fixed summary `"pay"` → pass **only** `amount=` (no `summary=` or merchant 409).
+- HTTP GET probe:  
+  `https://taler.hacktivism.ch/instances/goa-demo-cp4zqk/templates/goa-free` → 200.
+
+## Run
+
+```bash
+cd koopa-admin-log/benchmarks/amount-ladder
+./run.sh
+# short:
+AMOUNTS='GOA:0.05 GOA:1 GOA:10' WITHDRAW_AMT=GOA:100 ./run.sh
+```
+
+Outputs under `../../.tmp/amount-bench-<UTC>/`:
+
+| File | Content |
+|------|---------|
+| `payments.csv` | per-amount timings + status |
+| `RESULTS.md` | table summary |
+| `koopa-load-*.out` | **inside** process samples |
+| `run.log` | timeline |
+
+## Columns (RESULTS)
+
+- **handle**: wall time of `wallet handle-uri`
+- **run-until-done**: wallet background after handle
+- **total**: sum
+- **loadavg**: koopa 1‑min load after payment
+- **status**: `paid` / `error` / `insufficient` / …
+
+## Related
+
+- Partial earlier run (wrong URI): `2026/2026-07-09--amount-ladder-partial-results.md`
+- Instance notes: `configs/taler-merchant/demo-instance-goa-demo-cp4zqk.md`

benchmarks/amount-ladder/run.sh

@@ -0,0 +1,375 @@
+#!/bin/bash
+# Amount-ladder benchmark — drive from the local machine, measure on koopa (inside containers).
+# See README.md in this directory.
+set -eu
+
+BENCH_DIR=$(cd "$(dirname "$0")" && pwd)
+ROOT=$(cd "$BENCH_DIR/../.." && pwd)
+SECRETS="${SECRETS_DIR:-}"
+if [ -z "$SECRETS" ]; then
+  if [ -d "$ROOT/../koopa-admin-secrets" ]; then
+    SECRETS=$(cd "$ROOT/../koopa-admin-secrets" && pwd)
+  else
+    SECRETS="$ROOT/../koopa-admin-secrets"
+  fi
+fi
+
+SCRATCH="${ROOT}/.tmp/amount-bench-$(date -u +%Y%m%d-%H%M%S)"
+mkdir -p "$SCRATCH"
+chmod 700 "$SCRATCH"
+
+MONO="${MONO:-/Users/newkamek/src/taler/taler-typescript-core/packages/taler-wallet-cli/bin/taler-wallet-cli.mjs}"
+WDB="$SCRATCH/wallet.sqlite3"
+EX="${EXCHANGE_URL:-https://exchange.hacktivism.ch/}"
+KOOPA="${KOOPA_HOST:-koopa}"
+INST="${MERCHANT_INSTANCE:-goa-demo-cp4zqk}"
+TPL="${TEMPLATE_ID:-goa-free}"
+MER_HOST="${MER_HOST:-taler.hacktivism.ch}"
+FAT_USER="${FAT_USER:-bench-fat}"
+# ~20 payments: small → medium → large (atomic/tiny through multi-kilo)
+# Override: AMOUNTS='GOA:1 GOA:10' ./run.sh
+AMOUNTS="${AMOUNTS:-GOA:0.000001 GOA:0.00001 GOA:0.0001 GOA:0.001 GOA:0.01 GOA:0.05 GOA:0.1 GOA:0.5 GOA:1 GOA:2 GOA:5 GOA:10 GOA:25 GOA:50 GOA:100 GOA:250 GOA:500 GOA:1000 GOA:2500 GOA:5000}"
+# sum of default ladder ≈ GOA:9433 → withdraw headroom for multi-coin + fees
+WITHDRAW_AMT="${WITHDRAW_AMT:-GOA:50000}"
+CREDIT_AMT="${CREDIT_AMT:-GOA:200000}"
+
+BANK_ADMIN_PW_FILE="${BANK_ADMIN_PW_FILE:-$SECRETS/koopa/host-root/taler-bank/bank-admin-password.txt}"
+BANK_FAT_PW_FILE="${BANK_FAT_PW_FILE:-$SECRETS/koopa/host-root/taler-bank/bank-bench-fat-password.txt}"
+MER_PW_FILE="${MER_PW_FILE:-$SECRETS/koopa/host-root/taler-merchant/merchant-${INST}-password.txt}"
+
+wcli() { node "$MONO" --wallet-db="$WDB" --no-throttle --skip-defaults "$@"; }
+log() { echo "$(date -u +%H:%M:%S) $*" | tee -a "$SCRATCH/run.log"; }
+now() { python3 -c 'import time;print(time.time())'; }
+die() { log "FAIL $*"; exit 1; }
+
+[ -f "$MONO" ] || die "wallet cli missing: $MONO"
+[ -f "$BANK_ADMIN_PW_FILE" ] || die "missing $BANK_ADMIN_PW_FILE"
+[ -f "$MER_PW_FILE" ] || die "missing $MER_PW_FILE"
+
+ADMIN_PW=$(tr -d '\n' <"$BANK_ADMIN_PW_FILE")
+MER_PW=$(tr -d '\n' <"$MER_PW_FILE")
+if [ -f "$BANK_FAT_PW_FILE" ]; then
+  FAT_PW=$(tr -d '\n' <"$BANK_FAT_PW_FILE")
+else
+  FAT_PW=$(openssl rand -hex 12)
+  mkdir -p "$(dirname "$BANK_FAT_PW_FILE")"
+  printf '%s\n' "$FAT_PW" >"$BANK_FAT_PW_FILE"
+  chmod 600 "$BANK_FAT_PW_FILE"
+  log "generated fat password -> $BANK_FAT_PW_FILE (secrets repo)"
+fi
+
+# --- remote: run short script as hernani, capture stdout ---
+run_koopa() {
+  local id="$1"
+  shift
+  local script_local="$SCRATCH/remote-$id.sh"
+  local mark="BENCH_${id}_OK"
+  {
+    echo '#!/bin/bash'
+    echo 'set +e'
+    printf '%s\n' "$@"
+    echo "echo $mark"
+  } >"$script_local"
+  scp -o BatchMode=yes -q "$script_local" "${KOOPA}:/tmp/bench-run-$id.sh"
+  ssh -o BatchMode=yes "$KOOPA" "bash /tmp/bench-run-$id.sh; rm -f /tmp/bench-run-$id.sh" \
+    >"$SCRATCH/koopa-$id.out" 2>&1 || true
+  if ! grep -q "$mark" "$SCRATCH/koopa-$id.out" 2>/dev/null; then
+    log "WARN run_koopa $id incomplete (see koopa-$id.out)"
+    return 1
+  fi
+  return 0
+}
+
+# Measure **inside** containers (and host loadavg)
+sample_load() {
+  local tag="$1"
+  scp -o BatchMode=yes -q "$BENCH_DIR/sample-load-inside.sh" "${KOOPA}:/tmp/sample-load-inside.sh"
+  ssh -o BatchMode=yes "$KOOPA" "bash /tmp/sample-load-inside.sh $(printf %q "$tag"); rm -f /tmp/sample-load-inside.sh" \
+    >"$SCRATCH/koopa-load-$tag.out" 2>&1 || true
+}
+
+log "scratch=$SCRATCH"
+log "AMOUNTS=$AMOUNTS"
+
+# --- preflight (external, public HTTPS) ---
+for url in \
+  "https://taler.hacktivism.ch/config" \
+  "https://exchange.hacktivism.ch/keys" \
+  "https://taler.hacktivism.ch/instances/${INST}/templates/${TPL}"
+do
+  code=$(curl -sk -o /dev/null -w '%{http_code}' -m 10 "$url" || echo err)
+  log "preflight $url -> $code"
+  [ "$code" = "200" ] || die "preflight failed $url"
+done
+
+# --- bank setup on koopa (API localhost, orchestrated from the local machine) ---
+log "=== bank setup $FAT_USER credit=$CREDIT_AMT withdraw=$WITHDRAW_AMT ==="
+{
+  echo '#!/bin/bash'
+  echo 'set -euo pipefail'
+  echo "export ADMIN_PW=$(printf %q "$ADMIN_PW")"
+  echo "export FAT_PW=$(printf %q "$FAT_PW")"
+  echo "export FAT_USER=$(printf %q "$FAT_USER")"
+  echo "export CREDIT_AMT=$(printf %q "$CREDIT_AMT")"
+  echo "export WITHDRAW_AMT=$(printf %q "$WITHDRAW_AMT")"
+  cat <<'EOS'
+BANK=http://127.0.0.1:9012
+AT=$(curl -sS -u "admin:${ADMIN_PW}" -H "Content-Type: application/json" -d '{"scope":"readwrite"}' \
+  "$BANK/accounts/admin/token" | python3 -c "import sys,json;print(json.load(sys.stdin)['access_token'])")
+echo admin_tok_len=${#AT}
+curl -sS -o /tmp/acc.json -w "create:%{http_code}\n" -X POST -H "Authorization: Bearer $AT" \
+  -H "Content-Type: application/json" \
+  -d "{\"username\":\"${FAT_USER}\",\"password\":\"${FAT_PW}\",\"name\":\"Bench Fat\",\"is_public\":false,\"is_taler_exchange\":false,\"debit_threshold\":\"GOA:0\"}" \
+  "$BANK/accounts" || true
+echo create_body=$(head -c 160 /tmp/acc.json)
+python3 - <<'PY'
+import json, os
+A="0123456789ABCDEFGHJKMNPQRSTVWXYZ"
+def c32(b):
+  v=bits=0; o=[]
+  for x in b:
+    v=(v<<8)|x; bits+=8
+    while bits>=5:
+      bits-=5; o.append(A[(v>>bits)&31])
+  if bits: o.append(A[(v<<(5-bits))&31])
+  return "".join(o)
+uid=c32(os.urandom(32))
+user=os.environ["FAT_USER"]
+amt=os.environ["CREDIT_AMT"]
+payto=f"payto://x-taler-bank/bank.hacktivism.ch/{user}?receiver-name={user}&message=bench-credit"
+json.dump({"payto_uri":payto,"amount":amt,"request_uid":uid}, open("/tmp/tx.json","w"))
+print("credit", amt, "->", user, "uid", uid)
+PY
+curl -sS -o /tmp/cred.json -w "credit:%{http_code}\n" -X POST -H "Authorization: Bearer $AT" \
+  -H "Content-Type: application/json" -d @/tmp/tx.json "$BANK/accounts/admin/transactions"
+echo credit_body=$(head -c 200 /tmp/cred.json)
+BT=$(curl -sS -u "${FAT_USER}:${FAT_PW}" -H "Content-Type: application/json" -d '{"scope":"readwrite"}' \
+  "$BANK/accounts/${FAT_USER}/token" | python3 -c "import sys,json;print(json.load(sys.stdin)['access_token'])")
+echo fat_tok_len=${#BT}
+curl -sS -H "Authorization: Bearer $BT" "$BANK/accounts/${FAT_USER}" -o /tmp/fatbal.json
+echo fat_account=$(head -c 280 /tmp/fatbal.json)
+curl -sS -o /tmp/wd.json -w "wd:%{http_code}\n" -X POST -H "Authorization: Bearer $BT" \
+  -H "Content-Type: application/json" -d "{\"amount\":\"${WITHDRAW_AMT}\"}" \
+  "$BANK/accounts/${FAT_USER}/withdrawals"
+python3 - <<'PY'
+import json
+d=json.load(open("/tmp/wd.json"))
+print("wd_keys", list(d.keys())[:20])
+uri=d.get("taler_withdraw_uri") or d.get("withdrawal_uri") or ""
+wid=d.get("withdrawal_id") or d.get("id") or ""
+if not uri:
+  for v in d.values():
+    if isinstance(v, str) and v.startswith("taler"):
+      uri=v
+open("/tmp/bench-wuri.txt","w").write(uri)
+open("/tmp/bench-wid.txt","w").write(wid)
+print("WURI", uri)
+print("WID", wid)
+assert uri and wid, d
+PY
+# cleanup setup temps on host (not durable)
+rm -f /tmp/acc.json /tmp/tx.json /tmp/cred.json /tmp/fatbal.json /tmp/wd.json
+echo BENCH_banksetup_OK
+EOS
+} >"$SCRATCH/remote-banksetup.sh"
+scp -o BatchMode=yes -q "$SCRATCH/remote-banksetup.sh" "${KOOPA}:/tmp/bench-run-banksetup.sh"
+ssh -o BatchMode=yes "$KOOPA" 'bash /tmp/bench-run-banksetup.sh; rm -f /tmp/bench-run-banksetup.sh' \
+  | tee "$SCRATCH/koopa-banksetup.out"
+grep -q BENCH_banksetup_OK "$SCRATCH/koopa-banksetup.out" || die "bank setup failed"
+scp -o BatchMode=yes -q "${KOOPA}:/tmp/bench-wuri.txt" "${KOOPA}:/tmp/bench-wid.txt" "$SCRATCH/"
+ssh -o BatchMode=yes "$KOOPA" 'rm -f /tmp/bench-wuri.txt /tmp/bench-wid.txt' || true
+WURI=$(tr -d '\n' <"$SCRATCH/bench-wuri.txt")
+WID=$(tr -d '\n' <"$SCRATCH/bench-wid.txt")
+[ -n "$WURI" ] || die "missing withdraw URI"
+log "WURI=$WURI WID=$WID"
+
+log "=== load baseline (internal) ==="
+sample_load baseline
+
+log "=== wallet: exchange + withdraw (external) ==="
+wcli exchanges add "$EX" 2>&1 | tee "$SCRATCH/ex-add.out" || true
+wcli exchanges update "$EX" 2>&1 | tee "$SCRATCH/ex-upd.out" || true
+wcli exchanges tos "$EX" 2>&1 | tee "$SCRATCH/ex-tos.out" || true
+wcli exchanges accept-tos "$EX" 2>&1 | tee "$SCRATCH/ex-accept.out" || true
+
+TW0=$(now)
+# 1) Wallet selects exchange (status → selected). Must happen BEFORE bank confirm.
+wcli withdraw accept-uri --exchange "$EX" "$WURI" 2>&1 | tee "$SCRATCH/wd-accept.out" || true
+
+# 2) Wait until bank sees selection, then confirm once (libeufin rejects confirm if still unselected).
+log "=== bank: wait selected → confirm ==="
+run_koopa wdconfirm \
+  "export FAT_USER=$(printf %q "$FAT_USER")" \
+  "export FAT_PW=$(printf %q "$FAT_PW")" \
+  "export WID=$(printf %q "$WID")" \
+  'BANK=http://127.0.0.1:9012' \
+  'BT=$(curl -sS -u "${FAT_USER}:${FAT_PW}" -H "Content-Type: application/json" -d "{\"scope\":\"readwrite\"}" "$BANK/accounts/${FAT_USER}/token" | python3 -c "import sys,json;print(json.load(sys.stdin).get(\"access_token\",\"\"))")' \
+  'echo WID=$WID tok_len=${#BT}' \
+  'st=""' \
+  'for i in $(seq 1 60); do
+     curl -sS "$BANK/taler-integration/withdrawal-operation/${WID}" -o /tmp/wdst.json || true
+     st=$(python3 -c "import json;print(json.load(open(\"/tmp/wdst.json\")).get(\"status\",\"\"))" 2>/dev/null || true)
+     echo "wait_select #$i status=$st"
+     case "$st" in selected|confirmed) break ;; esac
+     sleep 1
+   done' \
+  'case "$st" in
+     selected)
+       echo confirming...
+       curl -sS -o /tmp/wdconf.out -w "confirm_http:%{http_code}\n" -X POST \
+         -H "Authorization: Bearer $BT" -H "Content-Type: application/json" -d "{}" \
+         "$BANK/accounts/${FAT_USER}/withdrawals/${WID}/confirm"
+       head -c 120 /tmp/wdconf.out; echo
+       ;;
+     confirmed) echo already_confirmed ;;
+     *) echo "WARN never selected (last=$st)" ;;
+   esac' \
+  'for i in $(seq 1 30); do
+     curl -sS "$BANK/taler-integration/withdrawal-operation/${WID}" -o /tmp/wdst.json || true
+     python3 -c "import json;d=json.load(open(\"/tmp/wdst.json\"));print(\"post\",d.get(\"status\"),\"transfer\",d.get(\"transfer_done\"))"
+     st=$(python3 -c "import json;print(json.load(open(\"/tmp/wdst.json\")).get(\"status\",\"\"))" 2>/dev/null || true)
+     td=$(python3 -c "import json;print(json.load(open(\"/tmp/wdst.json\")).get(\"transfer_done\"))" 2>/dev/null || true)
+     case "$st" in confirmed) break ;; esac
+     [ "$td" = "True" ] || [ "$td" = "true" ] && break
+     sleep 1
+   done
+   rm -f /tmp/wdst.json /tmp/wdconf.out' || true
+
+# 3) Wallet finishes planchets (may take minutes for large withdraws)
+log "=== wallet: run-until-done after confirm ==="
+for round in $(seq 1 30); do
+  log "withdraw run-until-done round $round"
+  # timeout each round so we do not hang forever on long-poll
+  set +e
+  # 120s max per round
+  perl -e 'alarm shift; exec @ARGV' 120 \
+    node "$MONO" --wallet-db="$WDB" --no-throttle --skip-defaults run-until-done \
+    2>&1 | tee -a "$SCRATCH/wd-rud.out"
+  set -e
+  wcli balance 2>&1 | tee "$SCRATCH/bal-wd-$round.out"
+  if python3 - "$SCRATCH/bal-wd-$round.out" <<'PY'
+import re,sys
+t=open(sys.argv[1]).read()
+vals=[float(x) for x in re.findall(r"GOA:([0-9.]+)", t)]
+sys.exit(0 if vals and max(vals) >= 100 else 1)
+PY
+  then
+    log "withdraw OK"
+    break
+  fi
+  sleep 1
+done
+TW1=$(now)
+python3 -c "print('withdraw_wall_s', round(float('$TW1')-float('$TW0'), 3))" | tee "$SCRATCH/withdraw-time.txt"
+wcli balance 2>&1 | tee "$SCRATCH/balance-before-pays.out"
+sample_load after-withdraw
+
+CSV="$SCRATCH/payments.csv"
+echo "idx,amount,handle_s,rud_s,total_s,status,order_id,loadavg,notes" >"$CSV"
+
+idx=0
+for AMT in $AMOUNTS; do
+  idx=$((idx + 1))
+  # no trailing slash; amount only (template has fixed summary)
+  URI="taler://pay-template/${MER_HOST}/instances/${INST}/${TPL}?amount=${AMT}"
+  echo "$URI" >"$SCRATCH/uri-$idx.txt"
+  log "=== pay #$idx $AMT ==="
+  log "uri=$URI"
+
+  load_before=$(ssh -o BatchMode=yes "$KOOPA" 'cut -d" " -f1 /proc/loadavg' 2>/dev/null || echo "?")
+  sample_load "pay$idx" &
+  SPID=$!
+
+  T0=$(now)
+  set +e
+  wcli handle-uri --yes --non-interactive "$URI" 2>&1 | tee "$SCRATCH/pay-$idx-handle.out"
+  T1=$(now)
+  wcli run-until-done 2>&1 | tee "$SCRATCH/pay-$idx-rud.out"
+  T2=$(now)
+  set -e
+
+  handle_s=$(python3 -c "print(round(float('$T1')-float('$T0'),3))")
+  rud_s=$(python3 -c "print(round(float('$T2')-float('$T1'),3))")
+  total_s=$(python3 -c "print(round(float('$T2')-float('$T0'),3))")
+  wait "$SPID" 2>/dev/null || true
+  load_after=$(ssh -o BatchMode=yes "$KOOPA" 'cut -d" " -f1 /proc/loadavg' 2>/dev/null || echo "?")
+
+  run_koopa "ord$idx" \
+    "export MPW=$(printf %q "$MER_PW")" \
+    "export INST=$(printf %q "$INST")" \
+    "export AMT=$(printf %q "$AMT")" \
+    'AUTH="Authorization: Bearer secret-token:${MPW}"' \
+    'curl -sk -H "$AUTH" "https://127.0.0.1:9010/instances/${INST}/private/orders?paid=YES&delta=-30" -o /tmp/ol.json' \
+    'python3 - <<PY
+import json, os
+d=json.load(open("/tmp/ol.json"))
+want=os.environ["AMT"]
+hit=None
+for o in d.get("orders") or []:
+  if o.get("amount")==want:
+    hit=o; break
+if hit:
+  print("ORDER", hit.get("order_id"), "amount", hit.get("amount"), "summary", hit.get("summary"))
+  open("/tmp/oid.txt","w").write(hit.get("order_id") or "")
+else:
+  print("NO_ORDER")
+  open("/tmp/oid.txt","w").write("")
+PY' \
+    'OID=$(tr -d "\n" </tmp/oid.txt 2>/dev/null || true)' \
+    'if [ -n "$OID" ]; then
+       curl -sk -H "$AUTH" "https://127.0.0.1:9010/instances/${INST}/private/orders/${OID}" -o /tmp/od.json
+       python3 -c "import json;d=json.load(open(\"/tmp/od.json\"));print(\"DETAIL paid\",d.get(\"paid\"),\"status\",d.get(\"order_status\"),\"amount\",(d.get(\"contract_terms\")or{}).get(\"amount\"))"
+     fi
+     rm -f /tmp/ol.json /tmp/od.json /tmp/oid.txt' || true
+
+  status=unknown
+  if grep -qE 'DETAIL paid True|DETAIL paid true' "$SCRATCH/koopa-ord$idx.out" 2>/dev/null; then
+    status=paid
+  elif grep -qiE 'insufficient balance' "$SCRATCH/pay-$idx-handle.out" 2>/dev/null; then
+    status=insufficient
+  elif grep -qiE 'error|fail|invalid|404|409|Unexpected HTTP' "$SCRATCH/pay-$idx-handle.out" 2>/dev/null; then
+    status=error
+  elif grep -qi 'paid' "$SCRATCH/pay-$idx-rud.out" 2>/dev/null; then
+    status=maybe
+  fi
+  oid=$(awk '/^ORDER /{print $2; exit}' "$SCRATCH/koopa-ord$idx.out" 2>/dev/null || true)
+  notes="load ${load_before}->${load_after}"
+  echo "$idx,$AMT,$handle_s,$rud_s,$total_s,$status,$oid,$load_after,$notes" >>"$CSV"
+  log "pay #$idx $AMT total=${total_s}s status=$status oid=$oid load=$load_after"
+  wcli balance 2>&1 | tee "$SCRATCH/bal-after-$idx.out" || true
+done
+
+sample_load final
+wcli balance 2>&1 | tee "$SCRATCH/balance-final.out" || true
+
+python3 - "$CSV" "$SCRATCH" <<'PY' | tee "$SCRATCH/RESULTS.md"
+import csv, sys
+csv_path, scratch = sys.argv[1], sys.argv[2]
+rows = list(csv.DictReader(open(csv_path)))
+print("# Amount ladder payment benchmark")
+print()
+print(f"- Scratch: `{scratch}`")
+print(f"- Drive: Local-machine wallet-cli (external) · Measure: koopa podman (internal)")
+print(f"- Template: `goa-free` on `goa-demo-cp4zqk`")
+print(f"- URI: `taler://pay-template/…/instances/…/goa-free?amount=…` (no trailing slash, no summary)")
+print()
+print("| # | Amount | handle (s) | run-until-done (s) | **total (s)** | status | order | loadavg |")
+print("|--:|--------|----------:|-------------------:|--------------:|--------|-------|--------:|")
+for r in rows:
+  print(f"| {r['idx']} | `{r['amount']}` | {r['handle_s']} | {r['rud_s']} | **{r['total_s']}** | {r['status']} | `{r.get('order_id','')}` | {r.get('loadavg','')} |")
+print()
+paid = [r for r in rows if r["status"] == "paid"]
+print(f"Paid **{len(paid)}/{len(rows)}**.")
+if paid:
+  totals = [float(r["total_s"]) for r in paid if r["total_s"]]
+  if totals:
+    print(f"**Min** {min(totals):.3f}s · **Max** {max(totals):.3f}s · **Avg** {sum(totals)/len(totals):.3f}s")
+print()
+print("Inside samples: `koopa-load-*.out` in scratch.")
+PY
+
+# durable copy of last RESULTS into benchmarks tree (no secrets)
+cp "$SCRATCH/RESULTS.md" "$BENCH_DIR/LAST_RESULTS.md"
+cp "$CSV" "$BENCH_DIR/LAST_payments.csv"
+log "DONE -> $SCRATCH/RESULTS.md (also benchmarks/amount-ladder/LAST_*)"
+cat "$SCRATCH/RESULTS.md"

benchmarks/amount-ladder/sample-load-inside.sh

@@ -0,0 +1,26 @@
+#!/bin/bash
+# Sample host + three-container CPU/RSS from koopa (run as hernani via ssh).
+# Usage (local):  ssh hernani@koopa 'bash -s' < sample-load-inside.sh
+# Or:            ./sample-load-inside.sh   # if executed ON koopa as hernani
+set -eu
+tag="${1:-manual}"
+echo "=== sample_load tag=$tag $(date -u +%FT%TZ) ==="
+echo "=== loadavg ==="
+cat /proc/loadavg
+echo "=== mem ==="
+free -h 2>/dev/null || true
+echo "=== top host cpu ==="
+ps -eo pid,pcpu,pmem,rss,etime,comm --sort=-pcpu 2>/dev/null | head -15
+echo "=== exchange (taler-exchange-hacktivism) ==="
+podman exec taler-exchange-hacktivism \
+  ps -eo pid,pcpu,pmem,rss,etime,args 2>/dev/null \
+  | grep -E 'taler-exchange|postgres -D|PID' | head -25 || true
+echo "=== merchant (taler-hacktivism) ==="
+podman exec taler-hacktivism \
+  ps -eo pid,pcpu,pmem,rss,etime,args 2>/dev/null \
+  | grep -E 'taler-merchant|nginx|postgres -D' | head -25 || true
+echo "=== bank (taler-bank-hacktivism) ==="
+podman exec taler-bank-hacktivism \
+  ps -eo pid,pcpu,pmem,rss,etime,args 2>/dev/null \
+  | grep -E 'java|libeufin|postgres -D' | head -15 || true
+echo "=== sample_load done ==="

scripts/castopod/README.md

@@ -0,0 +1,30 @@
+# Castopod ops scripts (koopa)
+
+Live stack: `/home/hernani/koopa-castopod/` (podman-compose).  
+These scripts live in admin-log and should be **copied** to the host when changed.
+
+## Hang-avoidance rules (learned 2026-07-09)
+
+| Failure mode | What hung | Fix in scripts |
+|--------------|-----------|----------------|
+| `php spark shield:user activate` | interactive `[y,n]` without TTY → **TypeError / hang** | SQL: `UPDATE cp_users SET active=1` via `cp_activate_users` |
+| bare `curl` / image pulls | no deadline → agent waits forever | `curl --connect-timeout` + `--max-time`; `timeout(1)` around `podman-compose pull` |
+| health loops | infinite `sleep` | capped `CP_HEALTH_TRIES` + fail |
+| Wikimedia/random downloads | slow/blocked CDN | local/known files only; always `--max-time` |
+| empty optional file fields | Castopod validates empty transcript/chapters | send real `.srt` / `.json` **or** omit fields carefully |
+
+## Usage (on koopa as `hernani`)
+
+```bash
+# copy once
+mkdir -p ~/koopa-castopod/bin
+cp /path/to/koopa-admin-log/scripts/castopod/{lib.sh,status.sh,up.sh} ~/koopa-castopod/bin/
+chmod +x ~/koopa-castopod/bin/*.sh
+
+~/koopa-castopod/bin/status.sh
+~/koopa-castopod/bin/up.sh
+```
+
+Env overrides: `CP_MAX_TIME`, `CP_PULL_TIMEOUT`, `CP_HEALTH_TRIES`, `CP_BASEURL`.
+
+Passwords stay in `~/koopa-castopod/.env` and `users.env` (mode 600) — never in admin-log.

scripts/castopod/lib.sh

@@ -0,0 +1,116 @@
+# shellcheck shell=bash
+# Shared helpers for Castopod ops on koopa — fail fast, never hang forever.
+
+set -euo pipefail
+
+# --- defaults (override via env) ---
+: "${CP_BASEURL:=https://castopod.hacktivism.ch}"
+: "${CP_COMPOSE_DIR:=/home/hernani/koopa-castopod}"
+: "${CP_CONNECT_TIMEOUT:=10}"   # seconds — TCP connect
+: "${CP_MAX_TIME:=120}"         # seconds — whole HTTP transfer
+: "${CP_PULL_TIMEOUT:=600}"     # seconds — image pull
+: "${CP_HEALTH_TRIES:=30}"      # health poll attempts
+: "${CP_HEALTH_SLEEP:=5}"       # seconds between polls
+
+# curl that cannot hang forever
+cp_curl() {
+  curl -sS \
+    --connect-timeout "${CP_CONNECT_TIMEOUT}" \
+    --max-time "${CP_MAX_TIME}" \
+    --retry 2 \
+    --retry-delay 2 \
+    --retry-connrefused \
+    "$@"
+}
+
+# HTTP status only (no hang)
+cp_http_code() {
+  local url=$1
+  cp_curl -o /dev/null -w '%{http_code}' "$url" || echo "000"
+}
+
+# Run command with hard wall-clock limit (GNU coreutils timeout)
+cp_timeout() {
+  local secs=$1
+  shift
+  if command -v timeout >/dev/null 2>&1; then
+    timeout --signal=TERM --kill-after=15s "${secs}" "$@"
+  else
+    # fallback: no timeout binary — still run, but warn
+    echo "WARN: timeout(1) missing; running without wall limit: $*" >&2
+    "$@"
+  fi
+}
+
+# Non-interactive answer stream for spark / CLI that prompts [y,n] or passwords.
+# Usage: cp_yes | podman exec -i … php spark …
+# Prefer SQL for activate when possible (spark prompts hang without TTY).
+cp_yes() {
+  # enough y's for a few prompts; never block waiting for input
+  yes y 2>/dev/null | head -n 20
+}
+
+# Poll until URL returns expected code or give up
+cp_wait_http() {
+  local url=$1
+  local want=${2:-200}
+  local i code
+  for ((i = 1; i <= CP_HEALTH_TRIES; i++)); do
+    code=$(cp_http_code "$url")
+    echo "health try $i/${CP_HEALTH_TRIES}: $url → $code"
+    if [[ "$code" == "$want" || "$code" =~ ^[23] ]]; then
+      return 0
+    fi
+    sleep "${CP_HEALTH_SLEEP}"
+  done
+  echo "ERROR: $url still not healthy after ${CP_HEALTH_TRIES} tries (last=$code)" >&2
+  return 1
+}
+
+# MariaDB password from compose .env (no hang if missing)
+cp_mysql_password() {
+  local envf="${CP_COMPOSE_DIR}/.env"
+  [[ -f "$envf" ]] || { echo "ERROR: missing $envf" >&2; return 1; }
+  # shellcheck disable=SC1090
+  grep -E '^MYSQL_PASSWORD=' "$envf" | head -1 | cut -d= -f2-
+}
+
+cp_mysql() {
+  local pw
+  pw=$(cp_mysql_password)
+  podman exec koopa-castopod-mariadb \
+    mariadb -ucastopod -p"$pw" castopod "$@"
+}
+
+# Activate shield users without spark interactive prompt
+cp_activate_users() {
+  cp_mysql -e "UPDATE cp_users SET active=1 WHERE username IN ('admin','ngi');"
+}
+
+# Session login for admin UI automation (uses users.env)
+cp_admin_login() {
+  local cookie=${1:-/tmp/cp-cj}
+  local envf="${CP_COMPOSE_DIR}/users.env"
+  [[ -f "$envf" ]] || { echo "ERROR: missing $envf" >&2; return 1; }
+  # shellcheck disable=SC1090
+  source "$envf"
+  [[ -n "${ADMIN_PW:-}" ]] || { echo "ERROR: ADMIN_PW empty" >&2; return 1; }
+
+  rm -f "$cookie"
+  cp_curl -c "$cookie" -b "$cookie" "${CP_BASEURL}/cp-auth/login" -o /tmp/cp-login.html
+  local csrf
+  csrf=$(sed -n 's/.*name="csrf_test_name" value="\([^"]*\)".*/\1/p' /tmp/cp-login.html | head -1)
+  [[ -n "$csrf" ]] || { echo "ERROR: no CSRF on login page" >&2; return 1; }
+
+  local code
+  code=$(cp_curl -c "$cookie" -b "$cookie" -o /dev/null -w '%{http_code}' \
+    -X POST "${CP_BASEURL}/cp-auth/login" \
+    --data-urlencode "csrf_test_name=${csrf}" \
+    --data-urlencode "email=admin@castopod.hacktivism.ch" \
+    --data-urlencode "password=${ADMIN_PW}")
+  # 303 see other is success
+  if [[ "$code" != "303" && "$code" != "302" && "$code" != "200" ]]; then
+    echo "ERROR: login HTTP $code" >&2
+    return 1
+  fi
+}

scripts/castopod/status.sh

@@ -0,0 +1,21 @@
+#!/usr/bin/env bash
+# Quick Castopod health — bounded network, no hangs.
+set -euo pipefail
+ROOT="$(cd "$(dirname "$0")" && pwd)"
+# shellcheck source=lib.sh
+source "${ROOT}/lib.sh"
+
+echo "== podman =="
+podman ps --filter name=koopa-castopod --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}' || true
+
+echo "== local backend :9020 =="
+code=$(cp_http_code "http://127.0.0.1:9020/health" || true)
+echo "local health → ${code:-000} (307 redirect to public is OK)"
+
+echo "== public =="
+# Note: do not put bare @url in curl -w; @ means "read file" in some curl contexts.
+for path in "/" "/@foss" "/@foss/feed.xml" "/@foss/episodes/four-freedoms"; do
+  url="${CP_BASEURL}${path}"
+  code=$(cp_http_code "$url")
+  printf '  %s → %s\n' "$url" "$code"
+done

scripts/castopod/up.sh

@@ -0,0 +1,32 @@
+#!/usr/bin/env bash
+# Start Castopod stack with hard timeouts on pull/up (no infinite hang).
+set -euo pipefail
+ROOT="$(cd "$(dirname "$0")" && pwd)"
+# shellcheck source=lib.sh
+source "${ROOT}/lib.sh"
+
+cd "${CP_COMPOSE_DIR}"
+
+echo "== pull (max ${CP_PULL_TIMEOUT}s) =="
+cp_timeout "${CP_PULL_TIMEOUT}" podman-compose pull
+
+echo "== up =="
+cp_timeout 180 podman-compose up -d
+
+echo "== wait for app on :9020 =="
+# Castopod often 307 from /health to https — accept 2xx/3xx
+ok=0
+for ((i = 1; i <= CP_HEALTH_TRIES; i++)); do
+  code=$(cp_http_code "http://127.0.0.1:9020/" || true)
+  echo "try $i: local / → $code"
+  if [[ "$code" =~ ^[23] ]]; then
+    ok=1
+    break
+  fi
+  sleep "${CP_HEALTH_SLEEP}"
+done
+[[ "$ok" -eq 1 ]] || { echo "ERROR: app not answering on 9020"; podman-compose ps; exit 1; }
+
+echo "== public via Caddy =="
+cp_wait_http "${CP_BASEURL}/" || true
+echo "done."