diff --git a/scripts/taler-monitoring/README.md b/scripts/taler-monitoring/README.md new file mode 100644 index 0000000..b69f879 --- /dev/null +++ b/scripts/taler-monitoring/README.md @@ -0,0 +1,95 @@ +# taler-monitoring + +Report for the **GOA** stack with clear severity tags and **per-area test IDs**: + +| Tag | Meaning | +|-----|---------| +| `[OK]` | check passed | +| `[INFO]` | inside status (container, ports, log noise) | +| `[WARN]` | degraded but maybe not fatal | +| `[ERROR]` | component problem | +| `[BLOCKER]` | **withdraw/pay path cannot complete** because of this | + +IDs: **`www-001`**, **`inside-001`**, **`versions-001`**, **`sanity-001`**, **`server-001`**, **`e2e-001`** … +Catalog: **[TESTS.md](./TESTS.md)**. + +```text +[OK] www-001 exchange /config https://exchange…/config +[BLOCKER] e2e-015 prereq: merchant HTTP 502 +``` + +End of each phase: totals + list of **BLOCKERS** and **ERRORS**. + +## Commands + +```bash +# Local GOA stack (may use SSH for inside/e2e) +./taler-monitoring.sh # urls + inside + versions + e2e +./taler-monitoring.sh inside # containers on koopa +./taler-monitoring.sh versions # deb.taler.net + package versions vs trixie +./taler-monitoring.sh sanity +./taler-monitoring.sh e2e + +# Other domains — public HTTPS only, never SSH +./taler-monitoring.sh -d taler.net +./taler-monitoring.sh --domain taler-ops.ch +./taler-monitoring.sh -d demo.taler.net urls +./taler-monitoring.sh taler.net # bare domain = same as -d +``` + +| Domain | Bank | Exchange | Merchant | Currency | +|--------|------|----------|----------|----------| +| `hacktivism.ch` (default) | bank.hacktivism.ch | exchange.hacktivism.ch | taler.hacktivism.ch | GOA | +| `taler.net` / `demo.taler.net` | bank.demo.taler.net | exchange.demo.taler.net | backend.demo.taler.net | KUDOS | +| `taler-ops.ch` | bank.* (probe) | exchange.taler-ops.ch | backend.* (probe) | CHF | +| other | bank.DOMAIN | exchange.DOMAIN | backend/taler/merchant (probe) | any | + +**SSH only for koopa** (`hacktivism.ch` / `-d koopa`). + +Other domains: never SSH. Optional **e2e** aborts cleanly on login/KYC. + +### E2E amounts (variable) + +| | Local (koopa) | Remote | +|--|---------------|--------| +| **ATM withdraw** | 20 · 50 · 100 · 200 | 10 · 20 · 50 | +| **Pay ladder** | 0.01 … 10 | 0.01 … 1 | + +```bash +./taler-monitoring.sh e2e +./taler-monitoring.sh -d taler.net urls e2e +# customize: +E2E_WITHDRAW_VALUES="20 50 100" E2E_PAY_VALUES="0.05 1 5" ./taler-monitoring.sh e2e +E2E_VARIABLE=0 WITHDRAW_AMT=GOA:50 PAY_AMT=GOA:1 ./taler-monitoring.sh e2e # single fixed +# remote secrets: +# E2E_BANK_ADMIN_PASS=… E2E_MERCHANT_TOKEN=… +``` + +## Phases + +| Phase | What | +|-------|------| +| **inside** | SSH koopa: processes, postgres, local /config,/keys, wirewatch, recent log ERRORs | +| **versions** | `deb.taler.net` reachable (InRelease/Packages/pool `.deb`); containers can reach it + have apt source; installed Taler packages vs **trixie** | +| **sanity** | bank · exchange · merchant sections (public + server) | +| **e2e** | account → credit → withdraw → wallet → confirm → order → pay | + +```bash +# package suite (default trixie on deb.taler.net) +TALER_APT_SUITE=trixie ./taler-monitoring.sh versions +TALER_PKG_BEHIND=error ./taler-monitoring.sh versions # any behind = ERROR +``` + +E2E maps failures to blockers, e.g.: + +- `bank-confirm HTTP 409` → wallet did not select exchange +- `no GOA balance` → wirewatch / transfer +- `create order failed` → merchant auth/instance +- `Alarm clock` during pay → usually missing `handle-uri --yes` or wrong pay URI (must be `…/instances/{inst}/{oid}/?c={token}` from merchant `taler_pay_uri`) +- `insufficient balance` → withdraw incomplete + +## Needs + +- SSH `koopa` (inside/sanity server bits) +- secrets under `koopa-admin-secrets/...` for e2e +- `taler-wallet-cli` for e2e diff --git a/scripts/taler-monitoring/TESTS.md b/scripts/taler-monitoring/TESTS.md new file mode 100644 index 0000000..d670e1a --- /dev/null +++ b/scripts/taler-monitoring/TESTS.md @@ -0,0 +1,149 @@ +# taler-monitoring — test IDs by area + +Every check line is numbered **per area** as `AREA-NNN` (zero-padded): + +| Area | Phase script | Meaning | +|------|--------------|---------| +| **www** | `check_urls.sh` | public HTTPS (outside-in) | +| **inside** | `check_inside.sh` | containers / processes on koopa | +| **versions** | `check_versions.sh` | deb.taler.net available + packages vs trixie | +| **sanity** | `check_sanity.sh` | public + server per component | +| **server** | `check_server.sh` | SSH host ports / processes | +| **e2e** | `check_e2e.sh` | withdraw + pay cycle | + +Format in output: + +```text +[OK] www-001 exchange /config https://exchange…/config +[ERROR] e2e-012 bank-auth: admin token failed +[BLOCKER] e2e-015 prereq: merchant HTTP 502 +``` + +IDs are assigned **in run order** within the area (`set_area` resets the counter). Optional soft checks still consume a number when they WARN. + +--- + +## www — public URLs (`./taler-monitoring.sh urls`) + +| ID | Check | +|----|--------| +| www-… | exchange `/config`, currency, **alt_unit_names** | +| www-… | exchange `/keys` (+ alt_unit_names soft) | +| www-… | exchange `/intro/`, `/` (302→intro) | +| www-… | **exchange `/terms`** body (not empty / not API error) | +| www-… | **exchange `/privacy`** body | +| www-… | exchange `/terms/` (200 or redirect) | +| www-… | bank `/config`, currency, **alt_unit_names** | +| www-… | bank integration / webui / intro / `/` | +| www-… | **bank `/terms`** body | +| www-… | **bank `/privacy`** (or `/intro/privacy.html` fallback) | +| www-… | merchant `/config`, currency, currencies alt_unit_names | +| www-… | each merchant `exchanges[]` `/config` alt_unit_names | +| www-… | merchant `/intro/`, `/webui/`, `/` | +| www-… | **merchant `/terms`** body (dual-currency notice) | +| www-… | **merchant `/privacy`** body (must not be `not configured`) | +| www-… | merchant `/terms/` redirect | + +**Legal docs rule:** HTTP 200, non-empty body, not plain `not configured`, not merchant API JSON `code:21`. On local stack, optional content needle (terms/privacy/FADP/GOA…). + +**alt_unit_names rule:** wallet codec requires a non-empty map including scale key `"0"`. For multi-currency merchant, also follow every entry in `exchanges[]` and check that exchange’s public `/config`. + +(IDs after a failed early check may shift if later soft checks are skipped when body missing — numbering follows **executed** checks.) + +--- + +## inside — koopa SSH (`./taler-monitoring.sh inside`) + +| ID | Check (typical order) | +|----|------------------------| +| inside-001 | ssh koopa | +| inside-002+ | per-component emit: container, ports, libeufin/httpd, postgres, local `/config`/`/keys`, wirewatch, DNS pin, caddy | + +Remote lines `E|comp|LEVEL|key|detail` each become one numbered result. + +--- + +## sanity — bank · exchange · merchant (`./taler-monitoring.sh sanity`) + +| ID | Section | +|----|---------| +| sanity-001… | bank public + server | +| sanity-… | exchange public + server | +| sanity-… | merchant public + server | + +Sequential through the whole script (one `set_area sanity`). + +--- + +## versions — packages vs deb.taler.net (`./taler-monitoring.sh versions`) + +### Outside (runner / public network — no SSH) + +| ID (order) | Check | +|------------|--------| +| versions-… | DNS `deb.taler.net` | +| versions-… | HTTPS portal + apt base URL | +| versions-… | suite `InRelease` / `Release` | +| versions-… | suite `Packages` + `Packages.gz` | +| versions-… | sample pool `.deb` fetchable (Range 200/206) | +| versions-… | suite offers `taler-exchange`, `taler-merchant`, `libeufin-bank` | +| versions-… | optional `trixie-testing` Packages | +| versions-… | TLS verify (soft) | + +### Inside (SSH koopa containers) + +| ID | Check | +|----|--------| +| versions-… | ssh koopa | +| versions-… | each container → `InRelease` (pasta can reach apt repo) | +| versions-… | each container lists `deb.taler.net` in apt sources | +| versions-… | each installed `taler*` / `libeufin*` / `libtaler*` / `libdonau*` vs suite version | +| versions-… | core packages installed (`taler-exchange`, `libeufin-bank`, `taler-merchant`) | + +Outside always runs. Inside skipped with `SKIP_SSH=1` (still reports outside results). + +Compare rules: + +- **match** suite → OK +- **ahead** of suite (often testing/dev) → INFO +- **behind** suite → ERROR for core packages, WARN otherwise (`TALER_PKG_BEHIND=error` forces ERROR) + +Default suite: **trixie** (`TALER_APT_SUITE`, `TALER_APT_BASE=https://deb.taler.net/apt/debian`). + +Without SSH (`SKIP_SSH=1` or remote domain): still runs outside-in repo checks; skips container install compare. + +--- + +## server — SSH ports (`./taler-monitoring.sh server`) + +| ID | Check | +|----|--------| +| server-001 | ssh | +| server-002+ | containers, local pasta ports, processes, caddy | + +--- + +## e2e — payment path (`./taler-monitoring.sh e2e`) + +| ID | Step (approx.) | +|----|----------------| +| e2e-001 | budget info | +| e2e-002 | wallet-cli present | +| e2e-003 | mode / currency info | +| e2e-004… | secrets, reachability gates | +| e2e-… | account, credit, withdraw, confirm, coins, order, pay | +| e2e-… | balances, dig on failure | + +Blockers keep the same ID prefix: `[BLOCKER] e2e-0NN step: message`. + +--- + +## Run one area + +```bash +./taler-monitoring.sh urls # www only +./taler-monitoring.sh inside # inside only +./taler-monitoring.sh versions # deb.taler.net + package drift +./taler-monitoring.sh e2e # e2e only +./taler-monitoring.sh -d taler.net urls +``` diff --git a/scripts/taler-monitoring/check_e2e.sh b/scripts/taler-monitoring/check_e2e.sh new file mode 100755 index 0000000..7da4dc6 --- /dev/null +++ b/scripts/taler-monitoring/check_e2e.sh @@ -0,0 +1,997 @@ +#!/usr/bin/env bash +# Full withdraw + pay cycle; clear BLOCKERS when the path cannot finish. +# Skips remaining steps if overall budget exceeded (E2E_TIMEOUT, default 90s). +set -euo pipefail +ROOT=$(cd "$(dirname "$0")" && pwd) +# shellcheck source=lib.sh +source "$ROOT/lib.sh" + +: "${E2E_TIMEOUT:=180}" # ATM ladder + wirewatch lag needs room (was 55 — too tight) +: "${E2E_PAY_SECS:=22}" # hard reserve for handle-uri + settle (do not starve pay) +: "${E2E_SETTLE_ROUNDS:=18}" +: "${E2E_SETTLE_SLEEP:=3}" +E2E_START=$(date +%s) +BAL_BEFORE="(not captured)" +BAL_AFTER="(not captured)" +E2E_REPORTED=0 +e2e_left() { echo $(( E2E_TIMEOUT - ($(date +%s) - E2E_START) )); } +e2e_over() { [ "$(e2e_left)" -le 0 ]; } + +# Fixed short timeout — used even when e2e budget is exhausted / on EXIT +# stdout only for JSON; logs go to stderr file so balance parse stays clean +wcli_bal_snap() { + local out="$1" + if [ -z "${WALLET_CLI:-}" ] || [ ! -f "${WDB:-}" ]; then + echo "(no wallet)" >"$out" + return 1 + fi + local err="${out}.err" + if command -v perl >/dev/null 2>&1; then + perl -e 'alarm shift; exec @ARGV' 10 \ + node "$WALLET_CLI" --wallet-db="$WDB" --no-throttle --skip-defaults balance \ + >"$out" 2>"$err" || true + else + node "$WALLET_CLI" --wallet-db="$WDB" --no-throttle --skip-defaults balance \ + >"$out" 2>"$err" || true + fi + [ -s "$out" ] +} + +fmt_bal() { + python3 -c ' +import json,re,sys +raw=open(sys.argv[1]).read() if sys.argv[1] else "" +if not raw.strip() or raw.strip().startswith("(no"): + print(raw.strip() or "(empty)"); sys.exit(0) +# extract first JSON object with "balances" +d=None +for m in re.finditer(r"\{", raw): + try: + d=json.loads(raw[m.start():]) + if isinstance(d, dict) and "balances" in d: + break + except Exception: + d=None +if not isinstance(d, dict): + print(re.sub(r"\s+"," ", raw)[:200]); sys.exit(0) +parts=[] +for b in d.get("balances") or []: + cur=(b.get("scopeInfo") or {}).get("currency") or "?" + parts.append("%s avail=%s pendingIn=%s" % (cur, b.get("available"), b.get("pendingIncoming"))) +print("; ".join(parts) if parts else "(no balances)") +' "${1:-/dev/null}" 2>/dev/null || echo "(unreadable)" +} + +# Numeric available for currency CUR (from balance file or fresh wallet snap) +wallet_avail_num() { + local f="${1:-}" + if [ -z "$f" ] || [ ! -s "$f" ]; then + f="$SCRATCH/bal-live.out" + wcli_bal_snap "$f" || wcli balance >"$f" 2>&1 || true + fi + python3 -c ' +import json,re,sys +cur=sys.argv[2] +raw=open(sys.argv[1]).read() if sys.argv[1] else "" +d=None +for m in re.finditer(r"\{", raw): + try: + d=json.loads(raw[m.start():]) + if isinstance(d, dict) and "balances" in d: break + except Exception: + d=None +if isinstance(d, dict): + for b in d.get("balances") or []: + c=(b.get("scopeInfo") or {}).get("currency") or "" + av=b.get("available") or "" + if c==cur or av.startswith(cur+":"): + try: + print(float(av.split(":",1)[1])); sys.exit(0) + except Exception: + pass +# text fallback +m=re.search(r"%s:([0-9]+(?:\.[0-9]+)?)" % re.escape(cur), raw) +print(float(m.group(1)) if m else 0.0) +' "${f:-/dev/null}" "${CUR:-GOA}" 2>/dev/null || echo 0 +} + +# Wait for spendable balance (settlement / wirewatch lag). Returns 0 if avail > min. +# Emphasizes timing, not hard failure, while waiting. +wait_wallet_balance() { + local min_n="${1:-0}" + local rounds="${2:-15}" + local sleep_s="${3:-3}" + local r av + info "wallet settle wait" "up to ${rounds}×${sleep_s}s for avail>${min_n} ${CUR} (wirewatch lag is common)" + for r in $(seq 1 "$rounds"); do + wcli run-until-done >"$SCRATCH/settle-run.out" 2>&1 || true + wcli_bal_snap "$SCRATCH/bal-live.out" || wcli balance >"$SCRATCH/bal-live.out" 2>&1 || true + av=$(wallet_avail_num "$SCRATCH/bal-live.out") + if python3 -c "import sys; sys.exit(0 if float(sys.argv[1]) > float(sys.argv[2]) else 1)" "$av" "$min_n" 2>/dev/null; then + ok "wallet balance ready" "avail=${CUR}:${av} after ~$((r * sleep_s))s settle wait" + info "balance" "$(fmt_bal "$SCRATCH/bal-live.out")" + return 0 + fi + if [ "$r" = "1" ] || [ "$((r % 3))" = "0" ]; then + info "settlement lag" "round $r/${rounds}: avail=${CUR}:${av} (still waiting — not an error yet)" + fi + # gentle wirewatch nudge mid-wait (local only) + if [ "$r" = "4" ] || [ "$r" = "8" ]; then + if [ "${E2E_REMOTE:-0}" != "1" ] && koopa_ssh_ok; then + koopa_ssh_run 8 \ + 'podman exec taler-hacktivism-exchange-ansible systemctl restart taler-exchange-wirewatch 2>/dev/null || true' \ + >/dev/null 2>&1 || true + fi + fi + sleep "$sleep_s" + done + av=$(wallet_avail_num "$SCRATCH/bal-live.out") + warn "settlement lag" "after ${rounds}×${sleep_s}s still avail=${CUR}:${av} — coins may still be in flight" + return 1 +} + +print_balances() { + section "e2e · wallet balances" + info "BALANCE before" "$BAL_BEFORE" + # refresh after if we can + if [ -n "${SCRATCH:-}" ]; then + wcli_bal_snap "$SCRATCH/bal-end.out" || true + if [ -s "$SCRATCH/bal-end.out" ]; then + BAL_AFTER=$(fmt_bal "$SCRATCH/bal-end.out") + fi + fi + info "BALANCE after " "$BAL_AFTER" +} + +e2e_finish() { + local code="${1:-1}" + if [ "$E2E_REPORTED" = "1" ]; then + return "$code" + fi + E2E_REPORTED=1 + print_balances + summary || true + return "$code" +} + +e2e_skip_rest() { + local why="$1" + warn "e2e timeout" "budget ${E2E_TIMEOUT}s exceeded — $why" + blocker "e2e-timeout" "skipped remaining steps after ${E2E_TIMEOUT}s ($why)" + section "e2e · report" + info "WITHDRAW" "incomplete (timeout)" + info "PAY" "SKIPPED (timeout)" + # If we already started a withdrawal, dig inside for missing coins + if [ -n "${WID:-}" ]; then + dig_when_no_coins 2>/dev/null || true + fi + e2e_finish 1 + exit 1 +} + +# Area e2e-### — withdraw + pay cycle +set_area e2e +section "e2e · prerequisites" +info "e2e budget" "${E2E_TIMEOUT}s (set E2E_TIMEOUT= to change)" + +# Remote domains (not koopa): public path only, tiny amounts, abort on login/KYC +: "${E2E_REMOTE:=0}" +if [ "${LOCAL_STACK:-1}" != "1" ]; then + E2E_REMOTE=1 + SKIP_SSH=1 + E2E_FAKE_INCOMING=0 +fi + +# Currency for amounts / balance checks +CUR="${EXPECT_CURRENCY:-}" +if [ -z "$CUR" ]; then + CUR=$(curl -skS -m 8 "$BANK_PUBLIC/config" 2>/dev/null | python3 -c 'import sys,json +try: print(json.load(sys.stdin).get("currency") or "") +except Exception: print("")' 2>/dev/null || true) +fi +[ -z "$CUR" ] && CUR=$(curl -skS -m 8 "$EXCHANGE_PUBLIC/config" 2>/dev/null | python3 -c 'import sys,json +try: print(json.load(sys.stdin).get("currency") or "") +except Exception: print("")' 2>/dev/null || true) +[ -z "$CUR" ] && CUR="GOA" + +# --------------------------------------------------------------------------- +# Variable amounts +# Withdraw = typical ATM notes only (Bankomat): 20 / 50 / 100 / 200 … +# Pay = several smaller purchase amounts +# Env: +# E2E_WITHDRAW_VALUES="20 50 100" # bare numbers → prefixed with CUR +# E2E_PAY_VALUES="0.01 0.05 0.5 1 2 5" +# E2E_VARIABLE=0 # single fixed WITHDRAW_AMT / PAY_AMT +# E2E_ATM_MAX=N # try at most N ATM withdraws (budget) +# --------------------------------------------------------------------------- +: "${E2E_VARIABLE:=1}" +: "${E2E_ATM_MAX:=4}" +if [ "$E2E_REMOTE" = "1" ]; then + # remote: still ATM-shaped but smaller notes; keep cheap + : "${E2E_WITHDRAW_VALUES:=10 20 50}" + : "${E2E_PAY_VALUES:=0.01 0.05 0.1 1}" +else + # local GOA: classic ATM denominations + : "${E2E_WITHDRAW_VALUES:=20 50 100 200}" + : "${E2E_PAY_VALUES:=0.01 0.05 0.1 0.5 1 2 5 10}" +fi + +# Build CUR:amount lists +build_amt_list() { + local cur="$1"; shift + local v out="" + for v in "$@"; do + case "$v" in + *:*) out="${out}${out:+ }$v" ;; + *) out="${out}${out:+ }${cur}:${v}" ;; + esac + done + printf '%s' "$out" +} +# shellcheck disable=SC2086 +WITHDRAW_LIST=$(build_amt_list "$CUR" $E2E_WITHDRAW_VALUES) +# shellcheck disable=SC2086 +PAY_LIST=$(build_amt_list "$CUR" $E2E_PAY_VALUES) + +if [ "$E2E_VARIABLE" != "1" ]; then + if [ "$E2E_REMOTE" = "1" ]; then + WITHDRAW_LIST="${WITHDRAW_AMT:-${CUR}:20}" + PAY_LIST="${PAY_AMT:-${CUR}:0.01}" + else + WITHDRAW_LIST="${WITHDRAW_AMT:-${CUR}:20}" + PAY_LIST="${PAY_AMT:-${CUR}:0.01}" + fi +fi + +# Credit = sum of planned ATM withdraws + small buffer for fees +CREDIT_AMT=$(python3 -c ' +import sys +cur=sys.argv[1] +vals=sys.argv[2].split() +s=0.0 +for a in vals: + try: s += float(a.split(":",1)[-1]) + except Exception: pass +buf = max(10.0, s * 0.05) +print("%s:%g" % (cur, s + buf)) +' "$CUR" "$WITHDRAW_LIST" 2>/dev/null || echo "${CUR}:100") + +# First withdraw/pay for legacy single-step labels +WITHDRAW_AMT=$(printf '%s' "$WITHDRAW_LIST" | awk '{print $1}') +PAY_AMT=$(printf '%s' "$PAY_LIST" | awk '{print $1}') + +# More wall time when running ATM ladder +if [ "$E2E_VARIABLE" = "1" ]; then + n_w=$(printf '%s' "$WITHDRAW_LIST" | wc -w | tr -d ' ') + n_p=$(printf '%s' "$PAY_LIST" | wc -w | tr -d ' ') + need=$(( 40 + n_w * 35 + n_p * 20 )) + if [ "${E2E_TIMEOUT}" -lt "$need" ]; then + E2E_TIMEOUT=$need + fi + # cap ATM attempts + WITHDRAW_LIST=$(printf '%s' "$WITHDRAW_LIST" | tr ' ' '\n' | head -n "$E2E_ATM_MAX" | tr '\n' ' ' | sed 's/ *$//') +fi + +BANK_HOST=$(python3 -c 'from urllib.parse import urlparse; print(urlparse("'"$BANK_PUBLIC"'").hostname or "bank")' 2>/dev/null || echo bank) + +SCRATCH=$(mktemp -d) +WDB="$SCRATCH/wallet.sqlite3" +export PATH="/opt/homebrew/bin:/usr/local/bin:$PATH" +# Always dump balances on any exit (timeout, blocker, signal, success) +trap 'ec=$?; e2e_finish "$ec"; rm -rf "$SCRATCH"; exit "$ec"' EXIT + +# Abort cleanly on login / KYC / registration barriers (esp. remote domains) +e2e_abort_auth() { + local step="$1" msg="$2" + warn "$step" "$msg" + blocker "$step" "abort (login/KYC): $msg" + section "e2e · report" + info "ABORTED" "login or KYC blocked further e2e — not retrying" + info "hint" "remote e2e needs open bank registration + credit path; set E2E_BANK_ADMIN_PASS / E2E_MERCHANT_TOKEN if you have them" + e2e_finish 1 + exit 1 +} + +is_auth_kyc_body() { + # stdin or file arg: true if looks like login/KYC barrier + local f="${1:-}" + local t + if [ -n "$f" ] && [ -f "$f" ]; then t=$(head -c 800 "$f" 2>/dev/null || true) + else t=$(cat 2>/dev/null || true); fi + printf '%s' "$t" | grep -qiE 'kyc|legitim|login required|unauthorized|forbidden|captcha|challenge|tan_|not.?allowed|registration.?disabled|admin.?only|permission.?denied|401|403' +} + +WALLET_CLI=$(find_wallet_cli) || { + blocker "prereq" "taler-wallet-cli not found (set WALLET_CLI=)" + exit 1 +} +ok "wallet-cli ($WALLET_CLI)" +info "e2e mode" "$([ "$E2E_REMOTE" = "1" ] && echo "remote/public domain (no SSH)" || echo "local koopa stack")" +info "currency" "$CUR" +info "ATM withdraw ladder" "$WITHDRAW_LIST (credit $CREDIT_AMT)" +info "pay ladder" "$PAY_LIST" +info "e2e budget" "${E2E_TIMEOUT}s" + +# Local stack: koopa secrets. Remote: only explicit env (never leak local passwords to foreign banks). +if [ "$E2E_REMOTE" = "1" ]; then + ADMIN_PASS="${E2E_BANK_ADMIN_PASS:-}" + MPW="${E2E_MERCHANT_TOKEN:-${MERCHANT_TOKEN:-}}" +else + ADMIN_PASS="${E2E_BANK_ADMIN_PASS:-$(read_secret "taler-bank/bank-admin-password.txt" || true)}" + MPW="${E2E_MERCHANT_TOKEN:-${MERCHANT_TOKEN:-$(read_secret "taler-merchant/merchant-goa-demo-cp4zqk-password.txt" || true)}}" +fi +if [ -n "$ADMIN_PASS" ]; then + ok "bank admin secret" +elif [ "$E2E_REMOTE" = "1" ]; then + warn "bank admin secret" "missing — will try public registration only" +else + blocker "prereq" "bank admin password missing (SECRETS_ROOT or koopa /root)" + exit 1 +fi +if [ -n "$MPW" ]; then + ok "merchant secret (instance ${MERCHANT_INSTANCE})" +elif [ "$E2E_REMOTE" = "1" ]; then + warn "merchant secret" "missing — order create may fail (set E2E_MERCHANT_TOKEN)" +else + blocker "prereq" "merchant instance password missing" + exit 1 +fi + +# Public reachability gates (remote: soft-skip if bank/merchant absent) +for pair in \ + "bank|$BANK_PUBLIC/config" \ + "exchange|$EXCHANGE_PUBLIC/config" \ + "exchange-keys|$EXCHANGE_PUBLIC/keys" \ + "bank-integration|$BANK_PUBLIC/taler-integration/config" \ + "merchant|$MERCHANT_PUBLIC/config" +do + IFS='|' read -r name url <<<"$pair" + code=$(http_code "$url") + if [ "$code" = "200" ]; then + ok "reachable $name" + else + if [ "$E2E_REMOTE" = "1" ] && [[ "$name" == bank* || "$name" == merchant ]]; then + warn "reachable $name" "HTTP $code — e2e may abort" + else + blocker "prereq" "$name HTTP $code ($url) — cannot start payment cycle" + fi + fi +done +# Remote: need at least bank+exchange+merchant for a full cycle +if [ "$E2E_REMOTE" = "1" ]; then + bc=$(http_code "$BANK_PUBLIC/config") + ec=$(http_code "$EXCHANGE_PUBLIC/config") + mc=$(http_code "$MERCHANT_PUBLIC/config") + if [ "$bc" != "200" ] || [ "$ec" != "200" ]; then + e2e_abort_auth "prereq" "bank/exchange not publicly usable (bank=$bc exchange=$ec) — skip e2e" + fi + if [ "$mc" != "200" ]; then + warn "prereq" "merchant HTTP $mc — will try withdraw only if possible" + fi +fi +if [ "${#BLOCKERS[@]}" -gt 0 ]; then + exit 1 +fi + +USER="mon$(date +%s | tail -c 6)" +USER_PW=$(python3 -c 'import secrets;print(secrets.token_urlsafe(10))') +info "e2e user" "$USER withdraw=$WITHDRAW_AMT pay=$PAY_AMT" +BANK="$BANK_PUBLIC" +INST="$MERCHANT_INSTANCE" + +# $1 = max seconds for this call (optional); rest = wallet-cli args +wcli() { + local maxc=12 + if [[ "${1:-}" =~ ^[0-9]+$ ]]; then + maxc=$1 + shift + fi + e2e_over && return 124 + local cap + cap=$(e2e_left) + [ "$cap" -gt "$maxc" ] && cap=$maxc + [ "$cap" -lt 3 ] && return 124 + if command -v perl >/dev/null 2>&1; then + perl -e 'alarm shift; exec @ARGV' "$cap" \ + node "$WALLET_CLI" --wallet-db="$WDB" --no-throttle --skip-defaults "$@" + else + node "$WALLET_CLI" --wallet-db="$WDB" --no-throttle --skip-defaults "$@" + fi +} + +# Pay must not be starved: use reserved seconds even if global budget is tight +wcli_pay() { + local cap=$E2E_PAY_SECS + local left + left=$(e2e_left) + if [ "$left" -gt "$cap" ]; then + cap=$E2E_PAY_SECS + elif [ "$left" -ge 8 ]; then + cap=$left + else + # still try once with floor 12s so Alarm clock is not the blocker + cap=12 + fi + info "pay wallet cap" "${cap}s" + if command -v perl >/dev/null 2>&1; then + perl -e 'alarm shift; exec @ARGV' "$cap" \ + node "$WALLET_CLI" --wallet-db="$WDB" --no-throttle --skip-defaults "$@" + else + node "$WALLET_CLI" --wallet-db="$WDB" --no-throttle --skip-defaults "$@" + fi +} + +# When coins never become available: run inside + targeted bank/exchange dig +dig_when_no_coins() { + section "e2e · coins missing — inside dig" + info "reason" "wallet empty after withdraw path (often wirewatch timing) — probing bank + exchange" + if [ -n "${WID:-}" ]; then + curl -sS -m 6 -o "$SCRATCH/wd-dig.json" \ + "$BANK_PUBLIC/taler-integration/withdrawal-operation/${WID}" 2>/dev/null || true + info "backend withdrawal-operation" "$(python3 -c ' +import json +try: + d=json.load(open("'"$SCRATCH"'/wd-dig.json")) + print("status=%s transfer_done=%s selection_done=%s amount=%s reserve=%s" % ( + d.get("status"), d.get("transfer_done"), d.get("selection_done"), + d.get("amount"), (d.get("selected_reserve_pub") or "-")[:24])) +except Exception as e: + print("unreadable", e) +' 2>/dev/null)" + fi + # Full component inside report (capped SSH; ignore its exit) + if [ "${SKIP_SSH}" != "1" ] && [ -x "$ROOT/check_inside.sh" ]; then + info "inside" "running check_inside.sh …" + # subshell so its PASS_N/FAIL_N/summary don't poison ours; still print to stdout + ( + # shellcheck disable=SC2030 + "$ROOT/check_inside.sh" || true + ) || true + else + warn "inside" "skipped (SKIP_SSH=1 or check_inside.sh missing)" + fi + # Extra targeted: wirewatch + recent journal + bank→exchange path + if koopa_ssh_ok; then + section "e2e · coins missing — exchange/bank focus" + DIG=$(koopa_ssh_bash 15 <<'REMOTE' || true +set +e +emit() { printf 'D|%s|%s\n' "$1" "$(printf '%s' "$2" | tr '\n' ' ' | head -c 220)"; } +EX=$(podman ps --format '{{.Names}}' | grep -i exchange | head -1) +BANK=$(podman ps --format '{{.Names}}' | grep -iE 'hacktivism-bank|taler-bank' | head -1) +[ -z "$BANK" ] && BANK=$(podman ps --format '{{.Names}}' | grep -i bank | head -1) +if [ -n "$EX" ]; then + podman exec "$EX" pgrep -af taler-exchange-wirewatch 2>/dev/null | head -1 | \ + { read -r l; [ -n "$l" ] && emit OK "wirewatch: $l" || emit ERROR "wirewatch not running"; } + podman exec "$EX" pgrep -af taler-exchange-transfer 2>/dev/null | head -1 | \ + { read -r l; [ -n "$l" ] && emit OK "transfer: $l" || emit WARN "transfer not running"; } + podman exec "$EX" pgrep -af taler-exchange-aggregator 2>/dev/null | head -1 | \ + { read -r l; [ -n "$l" ] && emit OK "aggregator: $l" || emit WARN "aggregator not running"; } + # last wirewatch log lines if journal available + j=$(podman exec "$EX" bash -c 'journalctl -u "taler-exchange-wirewatch*" -n 8 --no-pager 2>/dev/null | tail -5' 2>/dev/null | tr '\n' ';' | head -c 280) + [ -n "$j" ] && emit INFO "wirewatch journal: $j" + # reserve lookup via exchange if tools exist (best-effort) + c=$(curl -sS -m 3 -o /dev/null -w '%{http_code}' http://127.0.0.1:9011/keys 2>/dev/null || echo 000) + emit INFO "exchange /keys HTTP $c" +else + emit ERROR "no exchange container" +fi +if [ -n "$BANK" ]; then + podman exec "$BANK" pgrep -af 'MainKt serve|libeufin-bank serve' 2>/dev/null | head -1 | \ + { read -r l; [ -n "$l" ] && emit OK "libeufin: $l" || emit ERROR "libeufin not running"; } + c=$(curl -sS -m 3 -o /dev/null -w '%{http_code}' http://127.0.0.1:9012/taler-integration/config 2>/dev/null || echo 000) + emit INFO "bank integration /config HTTP $c" +else + emit ERROR "no bank container" +fi +REMOTE +) + while IFS= read -r line; do + case "$line" in + D\|*) + IFS="|" read -r _ lvl msg <<<"$line" + case "$lvl" in + OK) ok "dig $msg" ;; + ERROR) err "dig" "$msg" ;; + WARN) warn "dig" "$msg" ;; + INFO) info "dig" "$msg" ;; + esac + ;; + esac + done <<<"$DIG" + fi +} + +# Baseline wallet balance (empty DB) — always shown even if we abort later +wcli_bal_snap "$SCRATCH/bal-before.out" || true +BAL_BEFORE=$(fmt_bal "$SCRATCH/bal-before.out") +info "BALANCE before" "$BAL_BEFORE" + +# --------------------------------------------------------------------------- +section "e2e · bank (account · credit · withdraw)" +# --------------------------------------------------------------------------- +AT="" +if [ -n "${ADMIN_PASS:-}" ]; then + AT=$(curl -sS -m 15 -u "admin:${ADMIN_PASS}" -H 'Content-Type: application/json' \ + -d '{"scope":"readwrite"}' \ + "$BANK/accounts/admin/token" 2>"$SCRATCH/at.err" | python3 -c 'import sys,json;print(json.load(sys.stdin).get("access_token",""))' 2>/dev/null || true) +fi +if [ -n "$AT" ]; then + ok "admin token" +elif [ "$E2E_REMOTE" = "1" ]; then + warn "admin token" "unavailable — trying open registration" +else + if is_auth_kyc_body "$SCRATCH/at.err"; then + e2e_abort_auth "bank-auth" "admin login blocked (KYC/auth) — $(head -c 80 "$SCRATCH/at.err" 2>/dev/null)" + fi + blocker "bank-auth" "admin token failed — check admin password / bank up ($(head -c 80 "$SCRATCH/at.err" 2>/dev/null))" + exit 1 +fi + +# Create account: with admin bearer if we have it, else unauthenticated registration (demo banks) +if [ -n "$AT" ]; then + code=$(curl -sS -m 15 -o "$SCRATCH/acc.json" -w '%{http_code}' -X POST \ + -H "Authorization: Bearer $AT" -H 'Content-Type: application/json' \ + -d "{\"username\":\"${USER}\",\"password\":\"${USER_PW}\",\"name\":\"Monitoring\",\"is_public\":false,\"is_taler_exchange\":false,\"debit_threshold\":\"${CUR}:1000\"}" \ + "$BANK/accounts") +else + code=$(curl -sS -m 15 -o "$SCRATCH/acc.json" -w '%{http_code}' -X POST \ + -H 'Content-Type: application/json' \ + -d "{\"username\":\"${USER}\",\"password\":\"${USER_PW}\",\"name\":\"Monitoring\",\"is_public\":false,\"is_taler_exchange\":false,\"debit_threshold\":\"${CUR}:1000\"}" \ + "$BANK/accounts") +fi +if [ "$code" = "200" ] || [ "$code" = "201" ]; then + ok "create account $USER" +elif [ "$code" = "401" ] || [ "$code" = "403" ] || is_auth_kyc_body "$SCRATCH/acc.json"; then + e2e_abort_auth "bank-account" "registration/login blocked HTTP $code — $(head -c 100 "$SCRATCH/acc.json" | tr '\n' ' ')" +else + if [ "$E2E_REMOTE" = "1" ]; then + e2e_abort_auth "bank-account" "POST /accounts HTTP $code — cannot open account on this domain" + fi + blocker "bank-account" "POST /accounts HTTP $code — $(head -c 100 "$SCRATCH/acc.json" | tr '\n' ' ')" + exit 1 +fi + +if [ -n "$AT" ]; then + export USER_NAME="$USER" CREDIT_AMT="$CREDIT_AMT" OUT="$SCRATCH/credit.json" BANK_HOST="$BANK_HOST" + python3 - <<'PY' +import json, os +ALPH = "0123456789ABCDEFGHJKMNPQRSTVWXYZ" +def crock32(b): + bits = val = 0; o = [] + for byte in b: + val = (val << 8) | byte; bits += 8 + while bits >= 5: + bits -= 5; o.append(ALPH[(val >> bits) & 31]) + if bits: o.append(ALPH[(val << (5 - bits)) & 31]) + return "".join(o) +uid = crock32(os.urandom(32)) +user = os.environ["USER_NAME"] +host = os.environ.get("BANK_HOST") or "bank" +json.dump({ + "payto_uri": f"payto://x-taler-bank/{host}/{user}?receiver-name={user}&message=mon", + "amount": os.environ["CREDIT_AMT"], + "request_uid": uid, +}, open(os.environ["OUT"], "w")) +PY + curl -sS -m 15 -o "$SCRATCH/credit.out" -H "Authorization: Bearer $AT" \ + -H 'Content-Type: application/json' -d @"$SCRATCH/credit.json" \ + "$BANK/accounts/admin/transactions" >/dev/null || true + if grep -q row_id "$SCRATCH/credit.out" 2>/dev/null; then + ok "credit $CREDIT_AMT → $USER" + else + if is_auth_kyc_body "$SCRATCH/credit.out"; then + e2e_abort_auth "bank-credit" "admin credit blocked (KYC/auth)" + fi + if [ "$E2E_REMOTE" = "1" ]; then + e2e_abort_auth "bank-credit" "admin transfer failed — no credit path on remote ($(head -c 80 "$SCRATCH/credit.out" | tr '\n' ' '))" + fi + blocker "bank-credit" "admin transfer failed — $(head -c 100 "$SCRATCH/credit.out" | tr '\n' ' ')" + exit 1 + fi +else + # No admin: remote demo may start funded or require cash-in — try withdraw later; warn + warn "bank-credit" "skipped (no admin) — withdraw may fail without balance" +fi + +UT=$(curl -sS -m 15 -u "${USER}:${USER_PW}" -H 'Content-Type: application/json' \ + -d '{"scope":"readwrite"}' \ + "$BANK/accounts/${USER}/token" 2>"$SCRATCH/ut.err" | python3 -c 'import sys,json;print(json.load(sys.stdin).get("access_token",""))' 2>/dev/null || true) +if [ -n "$UT" ]; then + ok "user token" +elif is_auth_kyc_body "$SCRATCH/ut.err" || is_auth_kyc_body /dev/null; then + e2e_abort_auth "bank-auth" "user token failed (login/KYC) for $USER" +else + e2e_abort_auth "bank-auth" "user token failed for $USER — $(head -c 80 "$SCRATCH/ut.err" 2>/dev/null)" +fi + +# --------------------------------------------------------------------------- +section "e2e · wallet setup (exchange + ToS once)" +# --------------------------------------------------------------------------- +if ! wcli exchanges add "${EXCHANGE_PUBLIC}/" >"$SCRATCH/ex-add.out" 2>&1; then + if ! grep -qiE 'already|ok' "$SCRATCH/ex-add.out"; then + blocker "wallet-exchange" "exchanges add failed — $(tail -c 160 "$SCRATCH/ex-add.out" | tr '\n' ' ')" + else + ok "wallet exchange known" + fi +else + ok "wallet add exchange" +fi +wcli exchanges update "${EXCHANGE_PUBLIC}/" >"$SCRATCH/ex-up.out" 2>&1 || true +if wcli exchanges accept-tos "${EXCHANGE_PUBLIC}/" >"$SCRATCH/ex-tos.out" 2>&1; then + ok "wallet accept ToS" +else + if grep -qiE 'pending|tos|terms' "$SCRATCH/ex-tos.out"; then + blocker "wallet-tos" "accept-tos failed — wallet may refuse withdraw ($(tail -c 120 "$SCRATCH/ex-tos.out" | tr '\n' ' '))" + else + warn "wallet accept ToS" "$(tail -c 80 "$SCRATCH/ex-tos.out" | tr '\n' ' ')" + fi +fi + +wd_status() { + curl -sS -m 8 -o "$SCRATCH/wd-st.json" -w '' \ + "$BANK/taler-integration/withdrawal-operation/${WID}" 2>/dev/null || true + python3 -c 'import json;d=json.load(open("'"$SCRATCH"'/wd-st.json"));print(d.get("status",""))' 2>/dev/null || true +} +wd_status_detail() { + python3 -c 'import json;d=json.load(open("'"$SCRATCH"'/wd-st.json")) +print("status=%s reserve=%s exchange=%s" % ( + d.get("status"), + (d.get("selected_reserve_pub") or "-")[:20], + (d.get("selected_exchange") or d.get("selected_exchange_account") or "-")[:60], +))' 2>/dev/null || echo "(no status body)" +} + +fake_incoming_speedup() { + [ "${E2E_FAKE_INCOMING}" = "1" ] || return 0 + st=$(wd_status) + RPUB=$(python3 -c 'import json;d=json.load(open("'"$SCRATCH"'/wd-st.json"));print(d.get("selected_reserve_pub") or "")' 2>/dev/null || true) + AMT=$(python3 -c 'import json;d=json.load(open("'"$SCRATCH"'/wd-st.json"));print(d.get("amount") or "'"$WITHDRAW_AMT"'")' 2>/dev/null || echo "$WITHDRAW_AMT") + DEBIT=$(python3 -c 'import json;d=json.load(open("'"$SCRATCH"'/wd-st.json"));print(d.get("sender_wire") or "")' 2>/dev/null || true) + [ -z "$DEBIT" ] && DEBIT="payto://x-taler-bank/${BANK_HOST}/${USER}?receiver-name=${USER}" + [ -z "$RPUB" ] && return 0 + [ -z "${AT:-}" ] && return 0 + WG="${BANK}/accounts/exchange/taler-wire-gateway/admin/add-incoming" + curl -sS -m 10 -o "$SCRATCH/fake-in.json" -w '%{http_code}' -X POST \ + -H "Authorization: Bearer $AT" -H 'Content-Type: application/json' \ + -d "{\"amount\":\"${AMT}\",\"reserve_pub\":\"${RPUB}\",\"debit_account\":\"${DEBIT}\"}" \ + "$WG" >/dev/null || true + if [ "$E2E_REMOTE" != "1" ] && koopa_ssh_ok; then + koopa_ssh_run 22 \ + 'podman exec taler-hacktivism-exchange-ansible bash -c "runuser -u taler-exchange-wire -- timeout 15 taler-exchange-wirewatch -c /etc/taler-exchange/taler-exchange.conf -t -L INFO 2>&1 | tail -5"' \ + >/dev/null 2>&1 || true + fi +} + +# One ATM-style withdraw: create → accept → select → confirm → settle +# returns 0 on wallet balance increase, 1 on soft fail (ladder continues) +e2e_one_withdraw() { + WITHDRAW_AMT="$1" + local tag + tag=$(printf '%s' "$WITHDRAW_AMT" | tr '.:' '__') + section "e2e · ATM withdraw $WITHDRAW_AMT" + e2e_over && { warn "ATM withdraw" "budget exhausted — skip $WITHDRAW_AMT"; return 1; } + + curl -sS -m 15 -o "$SCRATCH/wd-$tag.json" -H "Authorization: Bearer $UT" \ + -H 'Content-Type: application/json' \ + -d "{\"amount\":\"${WITHDRAW_AMT}\",\"exchange_url\":\"${EXCHANGE_PUBLIC}/\"}" \ + "$BANK/accounts/${USER}/withdrawals" + WID=$(python3 -c 'import json;d=json.load(open("'"$SCRATCH"'/wd-'"$tag"'.json"));print(d.get("withdrawal_id") or d.get("id") or "")' 2>/dev/null || true) + URI=$(python3 -c 'import json;d=json.load(open("'"$SCRATCH"'/wd-'"$tag"'.json"));print(d.get("taler_withdraw_uri") or "")' 2>/dev/null || true) + URI_CLEAN=$(python3 -c 'import json;u=json.load(open("'"$SCRATCH"'/wd-'"$tag"'.json")).get("taler_withdraw_uri") or "";print(u.replace(":443/","/"))' 2>/dev/null || true) + if [ -z "$WID" ] || [ -z "$URI_CLEAN" ]; then + warn "ATM withdraw $WITHDRAW_AMT" "create failed — $(head -c 80 "$SCRATCH/wd-$tag.json" | tr '\n' ' ')" + return 1 + fi + ok "ATM withdrawal created $WITHDRAW_AMT ($WID)" + cp "$SCRATCH/wd-$tag.json" "$SCRATCH/wd.json" + + USE_URI="$URI" + [ -z "$USE_URI" ] && USE_URI="$URI_CLEAN" + if wcli withdraw accept-uri --exchange "${EXCHANGE_PUBLIC}/" "$USE_URI" >"$SCRATCH/accept-$tag.out" 2>&1; then + ok "wallet accept $WITHDRAW_AMT" + elif [ "$USE_URI" != "$URI_CLEAN" ] && [ -n "$URI_CLEAN" ] \ + && wcli withdraw accept-uri --exchange "${EXCHANGE_PUBLIC}/" "$URI_CLEAN" >"$SCRATCH/accept-$tag.out" 2>&1; then + ok "wallet accept $WITHDRAW_AMT (no :443)" + else + warn "ATM withdraw $WITHDRAW_AMT" "accept-uri failed — $(tail -c 100 "$SCRATCH/accept-$tag.out" | tr '\n' ' ')" + return 1 + fi + cp "$SCRATCH/accept-$tag.out" "$SCRATCH/accept.out" + + st="" + for i in 1 2 3 4; do + e2e_over && break + wcli run-until-done >"$SCRATCH/sel-$tag-$i.out" 2>&1 || true + st=$(wd_status) + case "$st" in selected|confirmed|aborted) break ;; esac + done + if [ "$st" != "selected" ] && [ "$st" != "confirmed" ]; then + wcli transactions >"$SCRATCH/tx-pre.json" 2>&1 || true + RPUB=$(python3 -c ' +import re +t=open("'"$SCRATCH"'/accept.out").read()+open("'"$SCRATCH"'/tx-pre.json").read() +m=re.search(r"reserve[_ ]?pub[\"=: ]+([A-Z0-9]{40,})", t, re.I) +if not m: m=re.search(r"\"reservePub\"\s*:\s*\"([^\"]+)\"", t) +print(m.group(1) if m else "") +' 2>/dev/null || true) + EPAYTO=$(curl -sS -m 12 "$EXCHANGE_PUBLIC/keys" 2>/dev/null | python3 -c ' +import json,sys +d=json.load(sys.stdin) +acc=d.get("accounts") or [] +for a in acc: + p=a.get("payto_uri") or a.get("payto_address") or "" + if "x-taler-bank" in p or "exchange" in p: + print(p); break +else: + if acc: print(acc[0].get("payto_uri") or "") +' 2>/dev/null || true) + if [ -n "$RPUB" ] && [ -n "$EPAYTO" ]; then + curl -sS -m 12 -o "$SCRATCH/sel.json" -X POST \ + -H 'Content-Type: application/json' \ + -d "{\"reserve_pub\":\"$RPUB\",\"selected_exchange\":\"$EPAYTO\"}" \ + "$BANK/taler-integration/withdrawal-operation/${WID}" >/dev/null || true + st=$(wd_status) + fi + fi + if [ "$st" != "selected" ] && [ "$st" != "confirmed" ]; then + warn "ATM withdraw $WITHDRAW_AMT" "not selected (status=${st:-?})" + return 1 + fi + if [ "$st" != "confirmed" ]; then + code=$(curl -sS -m 15 -o "$SCRATCH/conf-$tag.json" -w '%{http_code}' -X POST \ + -H "Authorization: Bearer $UT" -H 'Content-Type: application/json' -d '{}' \ + "$BANK/accounts/${USER}/withdrawals/${WID}/confirm") + case "$code" in + 200|204) ok "bank confirm $WITHDRAW_AMT" ;; + *) warn "ATM withdraw $WITHDRAW_AMT" "confirm HTTP $code"; return 1 ;; + esac + fi + fake_incoming_speedup + + # Short per-ATM poll; full settle wait happens after the ladder (avoids false FAIL) + local ok_bal=0 r av + for r in 1 2 3 4 5 6; do + wcli run-until-done >"$SCRATCH/run-$tag-$r.out" 2>&1 || true + wcli_bal_snap "$SCRATCH/bal-$tag.out" || wcli balance >"$SCRATCH/bal-$tag.out" 2>&1 || true + av=$(wallet_avail_num "$SCRATCH/bal-$tag.out") + if python3 -c "import sys; sys.exit(0 if float(sys.argv[1]) > 0 else 1)" "$av" 2>/dev/null; then + ok_bal=1 + ok "wallet funded after ATM $WITHDRAW_AMT" "avail=${CUR}:${av}" + info "balance" "$(fmt_bal "$SCRATCH/bal-$tag.out")" + break + fi + sleep 2 + done + if [ "$ok_bal" = "1" ]; then + return 0 + fi + # Bank side often already confirmed — treat as timing lag, not hard fail + st=$(wd_status) + if [ "$st" = "confirmed" ]; then + warn "ATM withdraw $WITHDRAW_AMT" "bank confirmed; wallet not funded yet (settlement timing — will recheck later)" + return 2 + fi + warn "ATM withdraw $WITHDRAW_AMT" "no wallet balance yet (status=${st:-?})" + return 1 +} + +# One variable payment amount +e2e_one_pay() { + PAY_AMT="$1" + local tag + tag=$(printf '%s' "$PAY_AMT" | tr '.:' '__') + section "e2e · pay $PAY_AMT" + e2e_over && { warn "pay" "budget exhausted — skip $PAY_AMT"; return 1; } + if [ -z "${MPW:-}" ]; then + warn "pay $PAY_AMT" "no merchant token" + return 1 + fi + AUTH="Authorization: Bearer secret-token:${MPW}" + curl -skS -m 15 -o "$SCRATCH/ord-$tag.json" -X POST \ + -H "$AUTH" -H 'Content-Type: application/json' \ + -d "{\"order\":{\"summary\":\"monitoring pay ${PAY_AMT}\",\"amount\":\"${PAY_AMT}\",\"fulfillment_message\":\"ok\"},\"create_token\":true}" \ + "${MERCHANT_PUBLIC}/instances/${INST}/private/orders" 2>"$SCRATCH/ord-$tag.err" || true + if ! grep -q order_id "$SCRATCH/ord-$tag.json" 2>/dev/null; then + if [ "$E2E_REMOTE" != "1" ] && koopa_ssh_ok; then + koopa_ssh_run 20 \ + "curl -skS -m 12 -X POST -H 'Authorization: Bearer secret-token:${MPW}' -H 'Content-Type: application/json' -d '{\"order\":{\"summary\":\"monitoring pay\",\"amount\":\"${PAY_AMT}\",\"fulfillment_message\":\"ok\"},\"create_token\":true}' 'https://127.0.0.1:9010/instances/${INST}/private/orders'" \ + >"$SCRATCH/ord-$tag.json" 2>"$SCRATCH/ord-$tag.err" || true + fi + fi + OID=$(python3 -c 'import json,re,sys;t=open(sys.argv[1]).read() +try: print(json.loads(t).get("order_id") or "") +except Exception: + m=re.search(r"\"order_id\"\s*:\s*\"([^\"]+)\"",t); print(m.group(1) if m else "") +' "$SCRATCH/ord-$tag.json" 2>/dev/null || true) + OTOK=$(python3 -c 'import json,re,sys;t=open(sys.argv[1]).read() +try: print(json.loads(t).get("token") or "") +except Exception: + m=re.search(r"\"token\"\s*:\s*\"([^\"]+)\"",t); print(m.group(1) if m else "") +' "$SCRATCH/ord-$tag.json" 2>/dev/null || true) + if [ -z "$OID" ]; then + if is_auth_kyc_body "$SCRATCH/ord-$tag.json"; then + e2e_abort_auth "merchant-order" "merchant auth/KYC on pay $PAY_AMT" + fi + warn "pay $PAY_AMT" "order create failed — $(head -c 80 "$SCRATCH/ord-$tag.json" | tr '\n' ' ')" + return 1 + fi + ok "merchant order $OID ($PAY_AMT)" + curl -skS -m 12 -o "$SCRATCH/ord-det-$tag.json" -H "$AUTH" \ + "${MERCHANT_PUBLIC}/instances/${INST}/private/orders/${OID}" 2>/dev/null || true + PAYURI=$(python3 -c 'import json,sys +try: print(json.load(open(sys.argv[1])).get("taler_pay_uri") or "") +except Exception: print("") +' "$SCRATCH/ord-det-$tag.json" 2>/dev/null || true) + if [ -z "$PAYURI" ] && [ -n "$OTOK" ]; then + MH=$(python3 -c 'from urllib.parse import urlparse; print(urlparse("'"$MERCHANT_PUBLIC"'").hostname or "taler.hacktivism.ch")' 2>/dev/null || echo "taler.hacktivism.ch") + PAYURI="taler://pay/${MH}/instances/${INST}/${OID}/?c=${OTOK}" + fi + if [ -z "$PAYURI" ]; then + warn "pay $PAY_AMT" "no pay URI for $OID" + return 1 + fi + if ! wcli_pay handle-uri --yes "$PAYURI" >"$SCRATCH/pay-$tag.out" 2>&1; then + warn "pay $PAY_AMT" "handle-uri failed — $(tail -c 100 "$SCRATCH/pay-$tag.out" | tr '\n' ' ')" + return 1 + fi + ok "wallet handle pay $PAY_AMT" + local r + for r in 1 2 3; do + wcli_pay run-until-done >"$SCRATCH/pay-run-$tag.out" 2>&1 || true + wcli_pay transactions >"$SCRATCH/tx-$tag.out" 2>&1 || true + curl -skS -m 8 -o "$SCRATCH/ord-paid-$tag.json" -H "$AUTH" \ + "${MERCHANT_PUBLIC}/instances/${INST}/private/orders/${OID}" 2>/dev/null || true + if grep -qiE 'payment|paid|Payment' "$SCRATCH/tx-$tag.out" 2>/dev/null \ + || grep -qiE 'done|paid|success|Payment' "$SCRATCH/pay-$tag.out" 2>/dev/null \ + || python3 -c 'import json,sys +d=json.load(open(sys.argv[1])) +sys.exit(0 if d.get("paid") is True or str(d.get("order_status","")).lower()=="paid" else 1) +' "$SCRATCH/ord-paid-$tag.json" 2>/dev/null; then + ok "payment settled $PAY_AMT (order $OID)" + return 0 + fi + done + warn "pay $PAY_AMT" "not settled for order $OID" + return 1 +} + +# --------------------------------------------------------------------------- +section "e2e · ATM withdraw ladder" +# --------------------------------------------------------------------------- +WITHDRAW_OK=0 +WITHDRAW_OK_N=0 +WITHDRAW_LAG_N=0 +WITHDRAW_FAIL_N=0 +WITHDRAW_REPORT="" +for WITHDRAW_AMT in $WITHDRAW_LIST; do + e2e_over && { warn "ATM ladder" "time budget low — stopping more ATM withdraws (not a protocol error)"; break; } + set +e + e2e_one_withdraw "$WITHDRAW_AMT" + wc=$? + set -e + case "$wc" in + 0) + WITHDRAW_OK=1 + WITHDRAW_OK_N=$((WITHDRAW_OK_N + 1)) + WITHDRAW_REPORT="${WITHDRAW_REPORT}${WITHDRAW_REPORT:+ }${WITHDRAW_AMT}=OK" + ;; + 2) + WITHDRAW_LAG_N=$((WITHDRAW_LAG_N + 1)) + WITHDRAW_REPORT="${WITHDRAW_REPORT}${WITHDRAW_REPORT:+ }${WITHDRAW_AMT}=LAG" + ;; + *) + WITHDRAW_FAIL_N=$((WITHDRAW_FAIL_N + 1)) + WITHDRAW_REPORT="${WITHDRAW_REPORT}${WITHDRAW_REPORT:+ }${WITHDRAW_AMT}=FAIL" + ;; + esac +done +info "ATM withdraw summary" "$WITHDRAW_REPORT (ok=$WITHDRAW_OK_N lag=$WITHDRAW_LAG_N fail=$WITHDRAW_FAIL_N)" + +# Settlement catch-up: bank may have confirmed while wallet was still empty +section "e2e · wallet settlement (timing)" +if ! wait_wallet_balance 0 "${E2E_SETTLE_ROUNDS}" "${E2E_SETTLE_SLEEP}"; then + if [ "$WITHDRAW_LAG_N" -gt 0 ] || [ "$WITHDRAW_OK_N" -gt 0 ]; then + warn "settlement timing" "ATM path reached bank confirm (lag=$WITHDRAW_LAG_N ok=$WITHDRAW_OK_N) but wallet empty after wait — TIME lag, not protocol error" + fi +fi +av_now=$(wallet_avail_num) +if python3 -c "import sys; sys.exit(0 if float(sys.argv[1]) > 0 else 1)" "$av_now" 2>/dev/null; then + WITHDRAW_OK=1 + if [ "$WITHDRAW_OK_N" = "0" ]; then + warn "settlement timing" "coins arrived after ATM ladder (${CUR}:${av_now}) — earlier 'FAIL' was timing lag" + WITHDRAW_REPORT="${WITHDRAW_REPORT} → late-OK avail=${CUR}:${av_now}" + fi + ok "spendable balance for payments" "${CUR}:${av_now}" +else + if [ "$WITHDRAW_OK" != "1" ]; then + warn "withdraw" "still no spendable ${CUR} after settle wait" + # diagnostic dig only — do not treat as hard stop if we can still see bank state + if [ "$E2E_REMOTE" != "1" ]; then + dig_when_no_coins || true + fi + # one last balance after dig (dig takes wall time — often enough for wirewatch) + wait_wallet_balance 0 8 3 || true + av_now=$(wallet_avail_num) + if python3 -c "import sys; sys.exit(0 if float(sys.argv[1]) > 0 else 1)" "$av_now" 2>/dev/null; then + WITHDRAW_OK=1 + warn "settlement timing" "coins present after dig wait (${CUR}:${av_now}) — timing, not blocker" + else + blocker "withdraw-settle" "no spendable ${CUR} after ATM ladder + settle wait ($WITHDRAW_LIST)" + section "e2e · report" + info "WITHDRAW" "$WITHDRAW_REPORT — no coins after extended wait" + info "PAY" "SKIPPED (no spendable balance after settle wait)" + exit 1 + fi + fi +fi + +# --------------------------------------------------------------------------- +section "e2e · variable payments (if balance allows)" +# --------------------------------------------------------------------------- +PAY_OK=0 +PAY_OK_N=0 +PAY_FAIL_N=0 +PAY_SKIP_N=0 +PAY_REPORT="" +if [ -z "${MPW:-}" ]; then + if [ "$E2E_REMOTE" = "1" ]; then + e2e_abort_auth "merchant-order" "no merchant token — set E2E_MERCHANT_TOKEN" + fi + blocker "merchant-order" "no merchant token" + exit 1 +fi + +for PAY_AMT in $PAY_LIST; do + e2e_over && { warn "pay ladder" "time budget low — stopping more payments"; break; } + av_now=$(wallet_avail_num) + pay_n=$(python3 -c 'import sys; print(float(sys.argv[1].split(":",1)[-1]))' "$PAY_AMT" 2>/dev/null || echo 0) + if ! python3 -c "import sys; sys.exit(0 if float(sys.argv[1]) + 1e-12 >= float(sys.argv[2]) else 1)" "$av_now" "$pay_n" 2>/dev/null; then + warn "pay $PAY_AMT" "skip — insufficient avail ${CUR}:${av_now} (need ${pay_n})" + PAY_SKIP_N=$((PAY_SKIP_N + 1)) + PAY_REPORT="${PAY_REPORT}${PAY_REPORT:+ }${PAY_AMT}=SKIP(bal)" + continue + fi + set +e + e2e_one_pay "$PAY_AMT" + pc=$? + set -e + if [ "$pc" = "0" ]; then + PAY_OK=1 + PAY_OK_N=$((PAY_OK_N + 1)) + PAY_REPORT="${PAY_REPORT}${PAY_REPORT:+ }${PAY_AMT}=OK" + else + PAY_FAIL_N=$((PAY_FAIL_N + 1)) + PAY_REPORT="${PAY_REPORT}${PAY_REPORT:+ }${PAY_AMT}=FAIL" + fi +done +info "pay summary" "$PAY_REPORT (ok=$PAY_OK_N fail=$PAY_FAIL_N skip=$PAY_SKIP_N)" +if [ "$PAY_OK" != "1" ]; then + av_now=$(wallet_avail_num) + if python3 -c "import sys; sys.exit(0 if float(sys.argv[1]) > 0 else 1)" "$av_now" 2>/dev/null; then + warn "pay" "balance ${CUR}:${av_now} but no pay amount completed — check merchant/timing (not necessarily empty wallet)" + else + blocker "pay-settle" "no payment succeeded and no spendable balance ($PAY_LIST)" + fi +fi + +section "e2e · report" +info "user" "$USER" +info "ATM withdraws" "$WITHDRAW_REPORT" +info "payments" "$PAY_REPORT" +info "final balance" "$(fmt_bal "$SCRATCH/bal-live.out" 2>/dev/null || echo "(n/a)")" +info "scratch" "$SCRATCH" +# Success if we had coins and at least one pay, OR coins + only pay skips (nothing affordable) +if [ "${#BLOCKERS[@]}" -eq 0 ]; then + if [ "$WITHDRAW_OK" = "1" ] && [ "$PAY_OK" = "1" ]; then + exit 0 + fi + if [ "$WITHDRAW_OK" = "1" ] && [ "$PAY_OK_N" = "0" ] && [ "$PAY_FAIL_N" = "0" ]; then + warn "pay" "no payment tried/completed — withdraw OK" + exit 0 + fi + if [ "$WITHDRAW_OK" = "1" ] && [ "$PAY_OK" != "1" ]; then + # money there, pays failed → soft exit 1 without inventing blockers if already warned + exit 1 + fi +fi +exit 1 diff --git a/scripts/taler-monitoring/check_inside.sh b/scripts/taler-monitoring/check_inside.sh new file mode 100755 index 0000000..aba6115 --- /dev/null +++ b/scripts/taler-monitoring/check_inside.sh @@ -0,0 +1,141 @@ +#!/usr/bin/env bash +# Inside status for bank / exchange / merchant. Hard-capped SSH — never hang forever. +set -euo pipefail +ROOT=$(cd "$(dirname "$0")" && pwd) +# shellcheck source=lib.sh +source "$ROOT/lib.sh" + +# Area inside-### — container / process state on koopa (SSH) +set_area inside +section "inside · collect from koopa" + +if [ "${SKIP_SSH}" = "1" ]; then + warn "ssh" "SKIP_SSH=1 — skipped" + summary + exit 0 +fi + +if ! koopa_ssh_ok; then + err "ssh" "cannot reach ${KOOPA_SSH} in ${SSH_CONNECT_TIMEOUT}s — set SKIP_SSH=1 to skip inside" + summary + exit 1 +fi +ok "ssh ${KOOPA_SSH}" + +# One short remote script (≤ SSH_CMD_TIMEOUT). Every slow step is local curl -m 3 or quick pgrep. +RAW=$( + koopa_ssh_bash "${SSH_CMD_TIMEOUT}" <<'REMOTE' || true +set +e +emit() { printf 'E|%s|%s|%s|%s\n' "$1" "$2" "$3" "$(printf '%s' "${4:-}" | tr '\n\r' ' ' | head -c 200)"; } +# quick curl +hc() { curl -skS -m 3 -o /tmp/mb -w '%{http_code}' "$1" 2>/dev/null || echo 000; } +# quick process check inside container (pgrep only) +hasp() { podman exec "$1" pgrep -f "$2" >/dev/null 2>&1; } + +BANK=$(podman ps --format '{{.Names}}' 2>/dev/null | grep -iE 'hacktivism-bank|taler-bank' | head -1) +[ -z "$BANK" ] && BANK=$(podman ps --format '{{.Names}}' 2>/dev/null | grep -i bank | head -1) +EX=$(podman ps --format '{{.Names}}' 2>/dev/null | grep -i exchange | head -1) +MER=$(podman ps --format '{{.Names}}' 2>/dev/null | grep -E '^taler-hacktivism$' | head -1) +[ -z "$MER" ] && MER=$(podman ps --format '{{.Names}}' 2>/dev/null | grep -iE 'merchant|hacktivism' | grep -viE 'bank|exchange' | head -1) + +# Domain resolve inside container (wirewatch needs bank.hacktivism.ch → real IP) +# emit: comp LEVEL dns "host → ip" or fail +check_dns() { + local comp="$1" ctr="$2" host="$3" + local line ip code + # Prefer IPv4 (wirewatch/libcurl often happier; avoid dead AAAA) + line=$(podman exec "$ctr" getent ahostsv4 "$host" 2>/dev/null | head -1) + [ -z "$line" ] && line=$(podman exec "$ctr" getent hosts "$host" 2>/dev/null | head -1) + ip=$(echo "$line" | awk '{print $1}') + if [ -z "$ip" ]; then + emit "$comp" ERROR "dns $host" "no resolve — run pin-container-hosts.sh" + return 1 + fi + # 127.0.0.1 is almost always wrong for public bank/exchange from inside pasta + if [ "$ip" = "127.0.0.1" ] || [ "$ip" = "::1" ]; then + emit "$comp" ERROR "dns $host" "$ip (loopback — wirewatch will fail)" + return 1 + fi + code=$(podman exec "$ctr" curl -skS -m 3 -o /dev/null -w '%{http_code}' "https://${host}/config" 2>/dev/null || echo 000) + if [ "$code" = "200" ]; then + emit "$comp" OK "dns $host" "→ $ip /config=$code" + else + emit "$comp" WARN "dns $host" "→ $ip /config=$code" + fi +} + +if [ -z "$BANK" ]; then emit bank ERROR container "not running" +else + emit bank INFO container "$(podman ps --filter name=$BANK --format '{{.Names}} {{.Status}}' | head -1)" + emit bank INFO ports "$(podman ps --filter name=$BANK --format '{{.Ports}}' | head -1)" + hasp "$BANK" 'MainKt serve|libeufin-bank serve' && emit bank OK libeufin "running" || emit bank ERROR libeufin "not running — API/withdraw dead" + podman exec "$BANK" pg_isready -q 2>/dev/null && emit bank OK postgres "ready" || emit bank ERROR postgres "not ready" + c=$(hc http://127.0.0.1:9012/config) + [ "$c" = "200" ] && emit bank OK "local /config" "HTTP $c" || emit bank ERROR "local /config" "HTTP $c" + c=$(hc http://127.0.0.1:9012/taler-integration/config) + [ "$c" = "200" ] && emit bank OK "local integration" "HTTP $c" || emit bank ERROR "local integration" "HTTP $c" + hasp "$BANK" 'nginx' && emit bank OK nginx ":9013" || emit bank WARN nginx "not running" + check_dns bank "$BANK" bank.hacktivism.ch || true + check_dns bank "$BANK" exchange.hacktivism.ch || true +fi + +if [ -z "$EX" ]; then emit exchange ERROR container "not running" +else + emit exchange INFO container "$(podman ps --filter name=$EX --format '{{.Names}} {{.Status}}' | head -1)" + c=$(hc http://127.0.0.1:9011/config) + [ "$c" = "200" ] && emit exchange OK "local /config" "HTTP $c" || emit exchange ERROR "local /config" "HTTP $c" + c=$(curl -sS -m 5 -o /dev/null -w '%{http_code}' http://127.0.0.1:9011/keys 2>/dev/null || echo 000) + [ "$c" = "200" ] && emit exchange OK "local /keys" "HTTP $c" || emit exchange ERROR "local /keys" "HTTP $c" + hasp "$EX" 'taler-exchange-httpd' && emit exchange OK httpd "running" || emit exchange ERROR httpd "not running" + hasp "$EX" 'taler-exchange-wirewatch' && emit exchange OK wirewatch "running" || emit exchange ERROR wirewatch "not running — withdraw stuck after bank confirm" + hasp "$EX" 'taler-exchange-aggregator' && emit exchange OK aggregator "running" || emit exchange WARN aggregator "not running" + hasp "$EX" 'taler-exchange-transfer' && emit exchange OK transfer "running" || emit exchange WARN transfer "not running" + # critical for wire gateway + check_dns exchange "$EX" bank.hacktivism.ch || true + check_dns exchange "$EX" exchange.hacktivism.ch || true + check_dns exchange "$EX" taler.hacktivism.ch || true +fi + +if [ -z "$MER" ]; then emit merchant ERROR container "not running" +else + emit merchant INFO container "$(podman ps --filter name=$MER --format '{{.Names}} {{.Status}}' | head -1)" + c=$(hc https://127.0.0.1:9010/config) + [ "$c" = "200" ] && emit merchant OK "local /config" "HTTP $c" || emit merchant ERROR "local /config" "HTTP $c" + hasp "$MER" 'taler-merchant-httpd' && emit merchant OK httpd "running" || emit merchant ERROR httpd "not running" + hasp "$MER" 'taler-merchant-wirewatch' && emit merchant OK wirewatch "running" || emit merchant WARN wirewatch "not running" + hasp "$MER" 'taler-merchant-depositcheck' && emit merchant OK depositcheck "running" || emit merchant WARN depositcheck "not running" + check_dns merchant "$MER" bank.hacktivism.ch || true + check_dns merchant "$MER" exchange.hacktivism.ch || true + check_dns merchant "$MER" taler.hacktivism.ch || true +fi + +if systemctl is-active caddy >/dev/null 2>&1 || pgrep -x caddy >/dev/null 2>&1; then + emit caddy OK process "active" +else + emit caddy ERROR process "not active" +fi +echo DONE +REMOTE +) + +if [ -z "$RAW" ] || ! echo "$RAW" | grep -q '^E|'; then + err "ssh" "remote timed out or empty (cap ${SSH_CMD_TIMEOUT}s)" + summary + exit 1 +fi + +while IFS= read -r line; do + case "$line" in + E\|*) + IFS='|' read -r _ comp level key detail <<<"$line" + case "$level" in + OK) ok "[$comp] $key${detail:+ ($detail)}" ;; + ERROR) err "$comp" "$key" "$detail" ;; + WARN) warn "[$comp] $key" "$detail" ;; + INFO) info "[$comp] $key" "$detail" ;; + esac + ;; + esac +done <<<"$RAW" + +summary diff --git a/scripts/taler-monitoring/check_sanity.sh b/scripts/taler-monitoring/check_sanity.sh new file mode 100755 index 0000000..1486e9b --- /dev/null +++ b/scripts/taler-monitoring/check_sanity.sh @@ -0,0 +1,281 @@ +#!/usr/bin/env bash +# Sanity checks for bank · exchange · merchant (public + server-side). +# Sections are independent; continues after failures. +set -euo pipefail +ROOT=$(cd "$(dirname "$0")" && pwd) +# shellcheck source=lib.sh +source "$ROOT/lib.sh" + +tmp=$(mktemp -d) +trap 'rm -rf "$tmp"' EXIT + +expect_code() { + local label="$1" want="$2" url="$3" + local code + code=$(http_code "$url") + case ",$want," in + *",$code,"*) ok "$label" ;; + *) fail "$label" "HTTP $code (want $want) $url" ;; + esac +} + +json_currency() { + python3 -c 'import json,sys; d=json.load(open(sys.argv[1])); print(d.get("currency") or "")' "$1" 2>/dev/null || true +} + +# Area sanity-### — public + optional server-side per component +set_area sanity + +# --------------------------------------------------------------------------- +section "sanity · bank" +# --------------------------------------------------------------------------- +expect_code "bank public /config" 200 "$BANK_PUBLIC/config" +expect_code "bank public /taler-integration/config" 200 "$BANK_PUBLIC/taler-integration/config" +expect_code "bank public /webui/" 200 "$BANK_PUBLIC/webui/" + +code=$(http_body "$BANK_PUBLIC/config" "$tmp/bank-config.json") +if [ "$code" = "200" ]; then + cur=$(json_currency "$tmp/bank-config.json") + [ "$cur" = "GOA" ] && ok "bank currency GOA" || fail "bank currency" "got ${cur:-?}" + # wire type / name if present + python3 - "$tmp/bank-config.json" <<'PY' 2>/dev/null && ok "bank config JSON object" || fail "bank config JSON" +import json,sys +d=json.load(open(sys.argv[1])) +sys.exit(0 if isinstance(d, dict) and d.get("currency") else 1) +PY + if json_has_alt_unit_names "$tmp/bank-config.json" >/tmp/alt-bank-s.$$ 2>&1; then + ok "bank /config alt_unit_names" "$(tr '\n' '; ' /dev/null || echo 000) +echo "LOCAL_CONFIG=$code" +code2=$(curl -sS -m 5 -o /dev/null -w '%{http_code}' http://127.0.0.1:9012/taler-integration/config 2>/dev/null || echo 000) +echo "LOCAL_INT=$code2" +if podman exec "$BANK" bash -c 'pgrep -f "MainKt serve|libeufin-bank serve" >/dev/null' 2>/dev/null; then + echo "LIBEUFIN=1" +else + echo "LIBEUFIN=0" +fi +if podman exec "$BANK" bash -c 'pg_isready -q' 2>/dev/null; then + echo "PG=1" +else + echo "PG=0" +fi +# in-container health if present +if podman exec "$BANK" test -x /usr/local/bin/check_bank-health.sh 2>/dev/null; then + podman exec "$BANK" /usr/local/bin/check_bank-health.sh 2>&1 | sed 's/^/HEALTH /' | tail -20 + echo "HEALTH_EC=${PIPESTATUS[0]}" +fi +REMOTE +) + echo "$BOUT" | grep -q '^CTR=.\+' && ok "bank container $(echo "$BOUT" | sed -n 's/^CTR=//p' | head -1)" || fail "bank container" "not found" + echo "$BOUT" | grep -q 'LOCAL_CONFIG=200' && ok "bank local :9012/config" || fail "bank local :9012/config" + echo "$BOUT" | grep -q 'LOCAL_INT=200' && ok "bank local :9012/taler-integration/config" || fail "bank local integration" + echo "$BOUT" | grep -q 'LIBEUFIN=1' && ok "bank libeufin-bank process" || fail "bank libeufin-bank process" + echo "$BOUT" | grep -q 'PG=1' && ok "bank postgres ready" || warn "bank postgres" "pg_isready failed" + if echo "$BOUT" | grep -q 'HEALTH '; then + if echo "$BOUT" | grep -qE 'HEALTH_EC=0|ALL CRITICAL CHECKS PASSED'; then + ok "bank check_bank-health.sh" + else + # health script may false-fail process grep; warn not fail if local config ok + warn "bank check_bank-health.sh" "non-zero or incomplete" + fi + fi +else + warn "bank server-side" "ssh ${KOOPA_SSH} unavailable" +fi + +# --------------------------------------------------------------------------- +section "sanity · exchange" +# --------------------------------------------------------------------------- +expect_code "exchange public /config" 200 "$EXCHANGE_PUBLIC/config" +expect_code "exchange public /keys" 200 "$EXCHANGE_PUBLIC/keys" +expect_code "exchange public /terms" 200 "$EXCHANGE_PUBLIC/terms" + +code=$(http_body "$EXCHANGE_PUBLIC/config" "$tmp/ex-config.json") +if [ "$code" = "200" ]; then + cur=$(json_currency "$tmp/ex-config.json") + [ "$cur" = "GOA" ] && ok "exchange currency GOA" || fail "exchange currency" "got ${cur:-?}" + if json_has_alt_unit_names "$tmp/ex-config.json" "GOA" >/tmp/alt-ex-s.$$ 2>&1; then + ok "exchange /config alt_unit_names" "$(tr '\n' '; ' /dev/null || echo 000) +echo "LOCAL_CONFIG=$code" +codek=$(curl -sS -m 8 -o /dev/null -w '%{http_code}' http://127.0.0.1:9011/keys 2>/dev/null || echo 000) +echo "LOCAL_KEYS=$codek" +if [ -n "$EX" ]; then + if podman exec "$EX" bash -c 'pgrep -f taler-exchange-httpd >/dev/null' 2>/dev/null; then + echo "HTTPD=1" + else + echo "HTTPD=0" + fi + for p in taler-exchange-secmod-rsa taler-exchange-secmod-eddsa taler-exchange-wirewatch taler-exchange-aggregator; do + if podman exec "$EX" bash -c "pgrep -f $p >/dev/null" 2>/dev/null; then + echo "PROC_$p=1" + else + echo "PROC_$p=0" + fi + done + if podman exec "$EX" test -x /usr/local/bin/check_exchange-health.sh 2>/dev/null; then + SKIP_ENSURE=1 podman exec -e SKIP_ENSURE=1 "$EX" /usr/local/bin/check_exchange-health.sh 2>&1 | sed 's/^/HEALTH /' | tail -25 + fi +fi +REMOTE +) + echo "$EOUT" | grep -q '^CTR=.\+' && ok "exchange container $(echo "$EOUT" | sed -n 's/^CTR=//p' | head -1)" || fail "exchange container" + echo "$EOUT" | grep -q 'LOCAL_CONFIG=200' && ok "exchange local :9011/config" || fail "exchange local :9011/config" + echo "$EOUT" | grep -q 'LOCAL_KEYS=200' && ok "exchange local :9011/keys" || fail "exchange local :9011/keys" + echo "$EOUT" | grep -q 'HTTPD=1' && ok "exchange-httpd process" || warn "exchange-httpd process" "not detected" + echo "$EOUT" | grep -q 'PROC_taler-exchange-wirewatch=1' && ok "exchange wirewatch" || warn "exchange wirewatch" "not running" + echo "$EOUT" | grep -q 'PROC_taler-exchange-aggregator=1' && ok "exchange aggregator" || warn "exchange aggregator" "not running" +else + warn "exchange server-side" "ssh unavailable" +fi + +# --------------------------------------------------------------------------- +section "sanity · merchant" +# --------------------------------------------------------------------------- +expect_code "merchant public /config" 200 "$MERCHANT_PUBLIC/config" +expect_code "merchant public /webui/" 200 "$MERCHANT_PUBLIC/webui/" + +code=$(http_body "$MERCHANT_PUBLIC/config" "$tmp/mer-config.json") +if [ "$code" = "200" ]; then + if python3 - "$tmp/mer-config.json" "$EXCHANGE_PUBLIC" <<'PY' +import json,sys +d=json.load(open(sys.argv[1])) +want=sys.argv[2].rstrip("/") +curs=list((d.get("currencies") or {}).keys()) +ex=d.get("exchanges") or [] +urls=[] +for e in ex: + if isinstance(e, dict): + u=e.get("base_url") or e.get("url") or e.get("exchange_base_url") or "" + if u: urls.append(u.rstrip("/")) + elif isinstance(e, str): + urls.append(e.rstrip("/")) +ok_goa = "GOA" in curs or any((e.get("currency") if isinstance(e, dict) else None)=="GOA" for e in ex) +ok_ex = any(want in u or "exchange.hacktivism.ch" in u for u in urls) +print("currencies", curs) +print("exchanges", urls[:5]) +sys.exit(0 if ok_goa and ok_ex else 1) +PY + then + ok "merchant config GOA + exchange.hacktivism.ch" + else + fail "merchant config GOA + exchange" "see currencies/exchanges" + fi + if json_has_alt_unit_names "$tmp/mer-config.json" >/tmp/alt-mer-s.$$ 2>&1; then + ok "merchant currencies alt_unit_names" "$(tr '\n' '; ' /dev/null || echo 000) +echo "LOCAL_CONFIG=$code" +if [ -n "$MER" ]; then + if podman exec "$MER" bash -c 'pgrep -f taler-merchant-httpd >/dev/null' 2>/dev/null; then + echo "HTTPD=1" + else + echo "HTTPD=0" + fi + if podman exec "$MER" test -x /usr/local/bin/check_merchant-health.sh 2>/dev/null; then + SKIP_ENSURE=1 podman exec -e SKIP_ENSURE=1 "$MER" /usr/local/bin/check_merchant-health.sh 2>&1 | sed 's/^/HEALTH /' | tail -30 + fi +fi +REMOTE +) + echo "$MOUT" | grep -q '^CTR=.\+' && ok "merchant container $(echo "$MOUT" | sed -n 's/^CTR=//p' | head -1)" || fail "merchant container" + echo "$MOUT" | grep -q 'LOCAL_CONFIG=200' && ok "merchant local :9010/config" || fail "merchant local :9010/config" + echo "$MOUT" | grep -q 'HTTPD=1' && ok "merchant-httpd process" || warn "merchant-httpd process" "not detected" + if echo "$MOUT" | grep -q 'HEALTH '; then + if echo "$MOUT" | grep -qiE 'ALL CRITICAL|HEALTH_EC=0|\[OK\]'; then + ok "merchant check_merchant-health.sh (sample OK)" + else + warn "merchant check_merchant-health.sh" "see remote output" + fi + fi +else + warn "merchant server-side" "ssh unavailable" +fi + +summary diff --git a/scripts/taler-monitoring/check_server.sh b/scripts/taler-monitoring/check_server.sh new file mode 100755 index 0000000..961b4c8 --- /dev/null +++ b/scripts/taler-monitoring/check_server.sh @@ -0,0 +1,128 @@ +#!/usr/bin/env bash +# Server-side component checks on koopa (via SSH). +set -euo pipefail +ROOT=$(cd "$(dirname "$0")" && pwd) +# shellcheck source=lib.sh +source "$ROOT/lib.sh" + +# Area server-### — host/container ports via SSH +set_area server +section "server · ssh ${KOOPA_SSH}" + +if ! koopa_ssh_ok; then + fail "ssh ${KOOPA_SSH}" "unreachable within ${SSH_CONNECT_TIMEOUT}s (SKIP_SSH=1 to skip)" + summary + exit 1 +fi +ok "ssh ${KOOPA_SSH}" + +# Run a remote script; collect structured lines +REMOTE=$(koopa_ssh_bash "${SSH_CMD_TIMEOUT}" <<'REMOTE' +set +e +report() { printf 'R|%s|%s|%s\n' "$1" "$2" "$3"; } + +# containers +for name in taler-hacktivism-bank taler-hacktivism-exchange-ansible taler-hacktivism; do + if podman ps --format '{{.Names}}' 2>/dev/null | grep -qx "$name"; then + st=$(podman ps --filter "name=^${name}$" --format '{{.Status}}' | head -1) + report OK "container $name" "$st" + else + # soft match + hit=$(podman ps --format '{{.Names}}' 2>/dev/null | grep -E "$name|bank|exchange|hacktivism" | head -3 | tr '\n' ' ') + if podman ps --format '{{.Names}}' 2>/dev/null | grep -q bank && [ "$name" = taler-hacktivism-bank ]; then + bn=$(podman ps --format '{{.Names}}' | grep -i bank | head -1) + report OK "container bank ($bn)" "$(podman ps --filter name="$bn" --format '{{.Status}}' | head -1)" + elif podman ps --format '{{.Names}}' 2>/dev/null | grep -qi exchange && echo "$name" | grep -qi exchange; then + en=$(podman ps --format '{{.Names}}' | grep -i exchange | head -1) + report OK "container exchange ($en)" "$(podman ps --filter name="$en" --format '{{.Status}}' | head -1)" + elif podman ps --format '{{.Names}}' 2>/dev/null | grep -qx taler-hacktivism && [ "$name" = taler-hacktivism ]; then + report OK "container taler-hacktivism" "$(podman inspect -f '{{.State.Status}}' taler-hacktivism 2>/dev/null)" + else + report FAIL "container $name" "not running (seen: $hit)" + fi + fi +done + +# local HTTP on pasta ports +for spec in \ + "bank-api|http://127.0.0.1:9012/config|200" \ + "bank-integration|http://127.0.0.1:9012/taler-integration/config|200" \ + "landing|http://127.0.0.1:9013/intro/|200" \ + "exchange|http://127.0.0.1:9011/config|200" \ + "merchant|https://127.0.0.1:9010/config|200" +do + IFS='|' read -r id url want <<<"$spec" + code=$(curl -skS -m 5 -o /tmp/mon-s.out -w '%{http_code}' "$url" 2>/dev/null || echo 000) + if [ "$code" = "$want" ]; then + report OK "local $id : ${url#*//}" "HTTP $code" + else + report FAIL "local $id" "HTTP $code want $want" + fi +done + +# processes inside bank +BANK=$(podman ps --format '{{.Names}}' | grep -iE 'bank|hacktivism-bank' | head -1) +if [ -n "$BANK" ]; then + if podman exec "$BANK" bash -c 'pgrep -f "MainKt serve|libeufin-bank serve" >/dev/null' 2>/dev/null; then + report OK "libeufin-bank process" "in $BANK" + else + report FAIL "libeufin-bank process" "not running in $BANK" + fi + if podman exec "$BANK" bash -c 'pg_isready -q' 2>/dev/null; then + report OK "postgres (bank)" "ready" + else + report WARN "postgres (bank)" "pg_isready failed" + fi + if podman exec "$BANK" bash -c 'pgrep -x nginx >/dev/null' 2>/dev/null; then + report OK "nginx landing" "in $BANK" + else + report WARN "nginx landing" "not seen in $BANK" + fi +else + report FAIL "bank container" "none" +fi + +# exchange process / systemd if any +EX=$(podman ps --format '{{.Names}}' | grep -i exchange | head -1) +if [ -n "$EX" ]; then + if podman exec "$EX" bash -c 'pgrep -f taler-exchange-httpd >/dev/null || systemctl is-active taler-exchange-httpd 2>/dev/null | grep -q active' 2>/dev/null; then + report OK "exchange-httpd" "in $EX" + else + # config answering is enough + report WARN "exchange-httpd process" "not detected; port check above" + fi +fi + +MER=$(podman ps --format '{{.Names}}' | grep -E '^taler-hacktivism$' | head -1) +[ -z "$MER" ] && MER=$(podman ps --format '{{.Names}}' | grep -i merchant | head -1) +if [ -n "$MER" ]; then + if podman exec "$MER" bash -c 'pgrep -f taler-merchant-httpd >/dev/null || true; curl -sk -m 3 -o /dev/null -w %{http_code} https://127.0.0.1:9010/config' 2>/dev/null | grep -q 200; then + report OK "merchant-httpd" "responds in $MER" + else + report WARN "merchant-httpd" "check manually in $MER" + fi +fi + +# caddy on host +if systemctl is-active caddy >/dev/null 2>&1 || pgrep -x caddy >/dev/null 2>&1; then + report OK "caddy" "active" +else + report WARN "caddy" "not detected as active" +fi +REMOTE +) + +while IFS= read -r line; do + case "$line" in + R\|*) + IFS='|' read -r _ st label detail <<<"$line" + case "$st" in + OK) ok "$label${detail:+ ($detail)}" ;; + FAIL) fail "$label" "$detail" ;; + WARN) warn "$label" "$detail" ;; + esac + ;; + esac +done <<<"$REMOTE" + +summary diff --git a/scripts/taler-monitoring/check_urls.sh b/scripts/taler-monitoring/check_urls.sh new file mode 100755 index 0000000..65561a4 --- /dev/null +++ b/scripts/taler-monitoring/check_urls.sh @@ -0,0 +1,255 @@ +#!/usr/bin/env bash +# Outside-in public HTTPS checks (no SSH). +set -euo pipefail +ROOT=$(cd "$(dirname "$0")" && pwd) +# shellcheck source=lib.sh +source "$ROOT/lib.sh" + +# Area www-### — public HTTPS (outside-in) +set_area www +section "www · public URLs · ${TALER_DOMAIN:-?} (outside-in, no SSH)" + +tmp=$(mktemp -d) +trap 'rm -rf "$tmp"' EXIT + +check_url() { + local label="$1" expect="$2" url="$3" + local code + code=$(http_code "$url") + case ",$expect," in + *",$code,"*) ok "$label $url" ;; + *) fail "$label $url" "got $code want $expect" ;; + esac +} + +# Soft check: OK on expect, WARN on foreign stack if down, ERROR on local stack +check_url_soft() { + local label="$1" expect="$2" url="$3" + local code + code=$(http_code "$url") + case ",$expect," in + *",$code,"*) ok "$label $url" ;; + *) + if [ "${LOCAL_STACK:-1}" = "0" ]; then + warn "$label $url" "got $code (optional on remote domain)" + else + fail "$label $url" "got $code want $expect" + fi + ;; + esac +} + +expect_currency() { + local label="$1" file="$2" want="${EXPECT_CURRENCY:-}" + local cur + cur=$(python3 -c 'import json,sys;print(json.load(open(sys.argv[1])).get("currency",""))' "$file" 2>/dev/null || true) + if [ -z "$want" ]; then + info "$label currency" "${cur:-?}" + return + fi + if [ "$cur" = "$want" ]; then + ok "$label currency=$want" + else + fail "$label currency" "got ${cur:-?} want $want" + fi +} + +# Legal docs: /terms and /privacy must be HTTP 200 with a real document body +# (not empty, not "not configured", not JSON API error). +# $1=label $2=url $3=optional needle regex (case-insensitive) for local stack +check_legal_doc() { + local label="$1" url="$2" needle="${3:-}" + local f code soft + soft=0 + [ "${LOCAL_STACK:-1}" = "0" ] && soft=1 + f=$(mktemp) + code=$(curl -skS --max-redirs 5 -L -m "${TIMEOUT}" \ + -H "Accept: text/html,text/markdown,text/plain,*/*" \ + -o "$f" -w '%{http_code}' "$url" 2>/dev/null || echo 000) + if [ "$code" != "200" ]; then + rm -f "$f" + if [ "$soft" = "1" ]; then + warn "$label" "HTTP $code — $url" + else + fail "$label" "HTTP $code — $url" + fi + return + fi + if [ ! -s "$f" ]; then + rm -f "$f" + fail "$label" "empty body — $url" + return + fi + # merchant returns plain "not configured" when PRIVACY_ETAG missing + if grep -qiE '^(not configured)\s*$' "$f" 2>/dev/null \ + || grep -qiE '"code"\s*:\s*21' "$f" 2>/dev/null; then + rm -f "$f" + fail "$label" "not configured / API error — $url" + return + fi + if [ -n "$needle" ] && [ "${LOCAL_STACK:-1}" = "1" ]; then + if ! grep -qiE "$needle" "$f" 2>/dev/null; then + warn "$label content" "missing /$needle/ — $url" + rm -f "$f" + return + fi + fi + ok "$label" "HTTP 200 · $(wc -c <"$f" | tr -d ' ') bytes" + rm -f "$f" +} + +# --- exchange (core; always required) --- www-001 … +check_url "exchange /config" 200 "$EXCHANGE_PUBLIC/config" +code=$(http_body "$EXCHANGE_PUBLIC/config" "$tmp/ec.json") +if [ "$code" = "200" ]; then + expect_currency "exchange" "$tmp/ec.json" + # currency_specification.alt_unit_names (wallet codec) + if json_has_alt_unit_names "$tmp/ec.json" "${EXPECT_CURRENCY:-}" >/tmp/alt-ex.$$ 2>&1; then + ok "exchange /config alt_unit_names" "$(tr '\n' '; ' /tmp/alt-bank.$$ 2>&1; then + ok "bank /config alt_unit_names" "$(tr '\n' '; ' /tmp/alt-mer.$$ 2>&1; then + ok "merchant /config currencies alt_unit_names" "$(tr '\n' '; ' command + +lint-exchange-goa taler-harness deployment lint-exchange-url https://exchange.hacktivism.ch/ diff --git a/scripts/taler-monitoring/harness-tests.txt b/scripts/taler-monitoring/harness-tests.txt new file mode 100644 index 0000000..1c63e3b --- /dev/null +++ b/scripts/taler-monitoring/harness-tests.txt @@ -0,0 +1,41 @@ +# Real-life–relevant taler-harness integration tests +# (from: taler-harness list-integrationtests) +# +# Criteria: everyday user/merchant/bank paths — withdraw, pay, deposit, refund, +# libeufin bank, templates/paywall. Excludes KYC edge cases, experimental, +# timetravel, perf, backup, mailbox, most fault-injection. +# +# These spin up a local TESTKUDOS stack (same client code paths as production). +# For tests against *public* demos, see harness-live.txt / `./taler-monitoring.sh live`. + +# --- Core path: get money + spend it (the GOA story) --- +simple-payment +withdrawal-bank-integrated +payment + +# --- Withdraw variants users actually hit --- +withdrawal-manual +withdrawal-external +withdrawal-idempotent + +# --- Pay variants shops use --- +payment-template +paywall-flow +payment-idempotency +payment-zero +payment-abort +otp + +# --- After pay: coins / merchant ops --- +deposit +refund +refund-auto +wallet-refresh + +# --- Bank stack we run (libeufin / regional) --- +libeufin-bank +bank-api + +# --- Client hygiene --- +wallet-config +term-of-service-format diff --git a/scripts/taler-monitoring/lib.sh b/scripts/taler-monitoring/lib.sh new file mode 100755 index 0000000..d405a2d --- /dev/null +++ b/scripts/taler-monitoring/lib.sh @@ -0,0 +1,536 @@ +# shellcheck shell=bash +# Shared helpers for taler-monitoring (laptop or koopa). + +# Default stack = GOA / hacktivism (overridden by TALER_DOMAIN / --domain) +: "${TALER_DOMAIN:=hacktivism.ch}" +: "${BANK_PUBLIC:=https://bank.hacktivism.ch}" +: "${EXCHANGE_PUBLIC:=https://exchange.hacktivism.ch}" +: "${MERCHANT_PUBLIC:=https://taler.hacktivism.ch}" +: "${BANK_LOCAL:=http://127.0.0.1:9012}" +: "${EXCHANGE_LOCAL:=http://127.0.0.1:9011}" +: "${MERCHANT_LOCAL:=https://127.0.0.1:9010}" +: "${LANDING_LOCAL:=http://127.0.0.1:9013}" +: "${KOOPA_SSH:=koopa}" +: "${MERCHANT_INSTANCE:=goa-demo-cp4zqk}" +: "${WITHDRAW_AMT:=GOA:20}" # single-shot fallback; e2e ladder uses ATM notes +: "${PAY_AMT:=GOA:0.01}" +: "${CREDIT_AMT:=GOA:400}" # covers ATM ladder 20+50+100+200 +: "${TIMEOUT:=12}" +: "${E2E_TIMEOUT:=55}" # whole e2e budget; skip rest when exceeded +: "${E2E_PAY_SECS:=22}" # dedicated seconds for pay handle-uri (avoid Alarm clock) +# Devtest: inject reserve credit via wire-gateway admin/add-incoming (optional). +# Default off once wirewatch DNS works; set E2E_FAKE_INCOMING=1 to force. +: "${E2E_FAKE_INCOMING:=0}" +# SSH must never hang the monitoring run +: "${SSH_CONNECT_TIMEOUT:=3}" +: "${SSH_CMD_TIMEOUT:=12}" # hard cap for whole remote script (seconds) +: "${SKIP_SSH:=0}" +# Expected currency for public /config checks (empty = report only, don't fail) +: "${EXPECT_CURRENCY:=GOA}" +# 1 = this is the local koopa/hacktivism stack (inside/e2e/SSH make sense) +: "${LOCAL_STACK:=1}" +# Probe merchant host candidates when applying a generic domain (0=off) +: "${TALER_DOMAIN_PROBE:=1}" + +BANK_PUBLIC=${BANK_PUBLIC%/} +EXCHANGE_PUBLIC=${EXCHANGE_PUBLIC%/} +MERCHANT_PUBLIC=${MERCHANT_PUBLIC%/} + +# --------------------------------------------------------------------------- +# Domain presets → public bank / exchange / merchant base URLs +# +# TALER_DOMAIN=hacktivism.ch (default, GOA, local stack) +# TALER_DOMAIN=taler.net → demo.taler.net (KUDOS) +# TALER_DOMAIN=demo.taler.net +# TALER_DOMAIN=taler-ops.ch → exchange.taler-ops.ch (CHF; bank/merchant if up) +# TALER_DOMAIN=example.org → bank/exchange/backend|taler|merchant.example.org +# +# Explicit BANK_PUBLIC / EXCHANGE_PUBLIC / MERCHANT_PUBLIC still win if set +# *after* apply_taler_domain, or pass full URLs via --bank/--exchange/--merchant. +# --------------------------------------------------------------------------- +_normalize_domain() { + local d="$1" + d="${d#https://}" + d="${d#http://}" + d="${d%%/*}" + d="${d%%:*}" + # Strip service host prefix only if a real domain remains (has a dot). + # e.g. bank.demo.taler.net → demo.taler.net, taler.hacktivism.ch → hacktivism.ch + # but NOT taler.net → net + case "$d" in + bank.*|exchange.*|taler.*|backend.*|merchant.*|shop.*|libeufin.*) + rest="${d#*.}" + if [[ "$rest" == *.* ]]; then + d="$rest" + fi + ;; + esac + printf '%s' "$d" +} + +_probe_https_config() { + # 0 if https://$1/config returns 200 + local host="$1" code + code=$(curl -skS --max-redirs 0 -m 4 -o /dev/null -w '%{http_code}' "https://${host}/config" 2>/dev/null || echo 000) + [ "$code" = "200" ] +} + +apply_taler_domain() { + local raw="${1:-}" + local d + [ -n "$raw" ] || return 0 + d=$(_normalize_domain "$raw") + TALER_DOMAIN="$d" + + case "$d" in + # Local koopa stack only — SSH / inside / e2e allowed + koopa|hacktivism.ch|hacktivism) + BANK_PUBLIC="https://bank.hacktivism.ch" + EXCHANGE_PUBLIC="https://exchange.hacktivism.ch" + MERCHANT_PUBLIC="https://taler.hacktivism.ch" + EXPECT_CURRENCY="GOA" + LOCAL_STACK=1 + SKIP_SSH=0 + TALER_DOMAIN="hacktivism.ch" + : "${WITHDRAW_AMT:=GOA:20}" + : "${PAY_AMT:=GOA:0.01}" + : "${CREDIT_AMT:=GOA:400}" + ;; + taler.net|demo.taler.net) + # Official public demo (KUDOS) — public only, never SSH; tiny e2e amounts + BANK_PUBLIC="https://bank.demo.taler.net" + EXCHANGE_PUBLIC="https://exchange.demo.taler.net" + MERCHANT_PUBLIC="https://backend.demo.taler.net" + EXPECT_CURRENCY="KUDOS" + LOCAL_STACK=0 + SKIP_SSH=1 + MERCHANT_INSTANCE="${MERCHANT_INSTANCE:-sandbox}" + WITHDRAW_AMT="${WITHDRAW_AMT:-KUDOS:20}" + PAY_AMT="${PAY_AMT:-KUDOS:0.01}" + CREDIT_AMT="${CREDIT_AMT:-KUDOS:100}" + TALER_DOMAIN="demo.taler.net" + ;; + taler-ops.ch) + # Public CHF exchange; bank/merchant hosts vary — probe common names + EXCHANGE_PUBLIC="https://exchange.taler-ops.ch" + BANK_PUBLIC="https://bank.taler-ops.ch" + MERCHANT_PUBLIC="https://backend.taler-ops.ch" + EXPECT_CURRENCY="CHF" + LOCAL_STACK=0 + SKIP_SSH=1 + WITHDRAW_AMT="${WITHDRAW_AMT:-CHF:20}" + PAY_AMT="${PAY_AMT:-CHF:0.01}" + CREDIT_AMT="${CREDIT_AMT:-CHF:100}" + if [ "${TALER_DOMAIN_PROBE}" = "1" ]; then + local h + for h in bank.taler-ops.ch bank.demo.taler-ops.ch; do + _probe_https_config "$h" && { BANK_PUBLIC="https://$h"; break; } + done + for h in backend.taler-ops.ch merchant.taler-ops.ch taler.taler-ops.ch shop.taler-ops.ch; do + _probe_https_config "$h" && { MERCHANT_PUBLIC="https://$h"; break; } + done + fi + ;; + *) + # Any other domain — public HTTPS only, never SSH to koopa + BANK_PUBLIC="https://bank.${d}" + EXCHANGE_PUBLIC="https://exchange.${d}" + MERCHANT_PUBLIC="https://backend.${d}" + EXPECT_CURRENCY="${EXPECT_CURRENCY:-}" # unknown — don't hard-fail currency + LOCAL_STACK=0 + SKIP_SSH=1 + if [ "${TALER_DOMAIN_PROBE}" = "1" ]; then + local h + for h in "backend.${d}" "taler.${d}" "merchant.${d}" "shop.${d}"; do + _probe_https_config "$h" && { MERCHANT_PUBLIC="https://$h"; break; } + done + for h in "bank.${d}" "libeufin.${d}"; do + _probe_https_config "$h" && { BANK_PUBLIC="https://$h"; break; } + done + _probe_https_config "exchange.${d}" || true + fi + ;; + esac + + BANK_PUBLIC=${BANK_PUBLIC%/} + EXCHANGE_PUBLIC=${EXCHANGE_PUBLIC%/} + MERCHANT_PUBLIC=${MERCHANT_PUBLIC%/} + + # Hard rule: only the local koopa/hacktivism stack may use SSH + if [ "${LOCAL_STACK}" != "1" ]; then + SKIP_SSH=1 + fi +} + +# Apply TALER_DOMAIN from env once (CLI exports TALER_DOMAIN_APPLIED=1 after overrides). +if [ "${TALER_DOMAIN_APPLIED:-0}" != "1" ] \ + && [ -n "${TALER_DOMAIN:-}" ] && [ "${TALER_DOMAIN}" != "hacktivism.ch" ]; then + apply_taler_domain "$TALER_DOMAIN" + TALER_DOMAIN_APPLIED=1 +fi + +# Safe SSH: publickey only, short connect, overall alarm so we never block forever. +SSH_BASE_OPTS=( + -o BatchMode=yes + -o ConnectTimeout="${SSH_CONNECT_TIMEOUT}" + -o ConnectionAttempts=1 + -o ServerAliveInterval=3 + -o ServerAliveCountMax=2 + -o StrictHostKeyChecking=accept-new + -o PreferredAuthentications=publickey + -o PasswordAuthentication=no + -o KbdInteractiveAuthentication=no + -o GSSAPIAuthentication=no + -o NumberOfPasswordPrompts=0 +) + +# Hard wall-clock timeout so ssh/curl never block the monitoring run forever. +with_timeout() { + local secs="$1"; shift + if command -v gtimeout >/dev/null 2>&1; then + gtimeout --kill-after=2 "$secs" "$@" + return $? + fi + if command -v timeout >/dev/null 2>&1; then + timeout -k 2 "$secs" "$@" 2>/dev/null || timeout --kill-after=2 "$secs" "$@" + return $? + fi + # Portable: perl alarm + process group kill + perl -e ' + use strict; use warnings; + my $secs = shift @ARGV; + my $pid = fork(); + die "fork: $!" unless defined $pid; + if ($pid == 0) { + setpgrp(0, 0); + exec @ARGV; + exit 127; + } + $SIG{ALRM} = sub { + kill "TERM", -$pid; + select(undef, undef, undef, 1.0); + kill "KILL", -$pid; + exit 124; + }; + alarm $secs; + waitpid($pid, 0); + my $code = $? >> 8; + alarm 0; + exit $code; + ' "$secs" "$@" +} + +# Probe: 0 if koopa SSH works quickly +koopa_ssh_ok() { + [ "${SKIP_SSH}" = "1" ] && return 1 + with_timeout $((SSH_CONNECT_TIMEOUT + 3)) \ + ssh "${SSH_BASE_OPTS[@]}" "${KOOPA_SSH}" 'echo ok' >/dev/null 2>&1 +} + +# Run remote bash -s with optional stdin script; hard-capped +# usage: koopa_ssh_bash [timeout_secs] <<'EOF' ... EOF +# or: koopa_ssh_run timeout_secs 'remote command' +koopa_ssh_run() { + local t="${1:-$SSH_CMD_TIMEOUT}" + shift + with_timeout "$t" ssh "${SSH_BASE_OPTS[@]}" "${KOOPA_SSH}" "$@" +} + +koopa_ssh_bash() { + local t="${1:-$SSH_CMD_TIMEOUT}" + with_timeout "$t" ssh "${SSH_BASE_OPTS[@]}" "${KOOPA_SSH}" 'bash -s' +} + +if [ "${NO_COLOR:-0}" = "1" ] || [ ! -t 1 ]; then + G= R= Y= C= N= B= +else + G=$'\e[32m'; R=$'\e[31m'; Y=$'\e[33m'; C=$'\e[36m'; B=$'\e[1m'; N=$'\e[0m' +fi + +PASS_N=0 +FAIL_N=0 +WARN_N=0 +INFO_N=0 +BLOCKERS=() # human-readable payment/withdraw blockers +ERRORS=() # all ERROR lines (component scope) + +# Test IDs by area: www-001, e2e-001, inside-001, … +# Usage: set_area www then each ok/fail/warn/info/blocker/err auto-numbers. +TEST_AREA="" +TEST_N=0 +# Last issued id (www-001); set by _take_tid — not via $(…) so TEST_N persists. +LAST_TID="" +set_area() { + TEST_AREA="$1" + TEST_N=0 + LAST_TID="" +} +# Assign next id into LAST_TID (must not run in a subshell). +_take_tid() { + LAST_TID="" + if [ -z "${TEST_AREA:-}" ]; then + return + fi + TEST_N=$((TEST_N + 1)) + LAST_TID=$(printf '%s-%03d' "$TEST_AREA" "$TEST_N") +} +_fmt_tid() { + # prefix "www-001 " or empty + if [ -n "${LAST_TID:-}" ]; then + printf '%s ' "$LAST_TID" + fi +} + +ok() { + local label="$1" + _take_tid + printf '%s[OK]%s %s%s\n' "$G" "$N" "$(_fmt_tid)" "$label" + PASS_N=$((PASS_N + 1)) +} +# component-scoped error: err bank "libeufin down" "detail" +err() { + local comp="$1" msg="$2" detail="${3:-}" + _take_tid + printf '%s[ERROR]%s %s%s: %s%s\n' "$R" "$N" "$(_fmt_tid)" "$comp" "$msg" "${detail:+ — $detail}" + FAIL_N=$((FAIL_N + 1)) + ERRORS+=("${LAST_TID:+$LAST_TID }[$comp] $msg${detail:+ — $detail}") +} +# legacy fail label ... +fail() { + local label="$1" detail="${2:-}" + _take_tid + printf '%s[ERROR]%s %s%s%s\n' "$R" "$N" "$(_fmt_tid)" "$label" "${detail:+ — $detail}" + FAIL_N=$((FAIL_N + 1)) + ERRORS+=("${LAST_TID:+$LAST_TID }$label${detail:+ — $detail}") +} +warn() { + local label="$1" detail="${2:-}" + _take_tid + printf '%s[WARN]%s %s%s%s\n' "$Y" "$N" "$(_fmt_tid)" "$label" "${detail:+ — $detail}" + WARN_N=$((WARN_N + 1)) +} +info() { + local label="$1" detail="${2:-}" + _take_tid + if [ -n "$detail" ]; then + printf '%s[INFO]%s %s%s — %s\n' "$C" "$N" "$(_fmt_tid)" "$label" "$detail" + else + printf '%s[INFO]%s %s%s\n' "$C" "$N" "$(_fmt_tid)" "$label" + fi + INFO_N=$((INFO_N + 1)) +} +blocker() { + # Payment/withdraw path cannot proceed because of this + local step="$1" msg="$2" + _take_tid + printf '%s[BLOCKER]%s %s%s: %s\n' "$R$B" "$N" "$(_fmt_tid)" "$step" "$msg" + BLOCKERS+=("${LAST_TID:+$LAST_TID }[$step] $msg") + FAIL_N=$((FAIL_N + 1)) + ERRORS+=("BLOCKER ${LAST_TID:+$LAST_TID }[$step] $msg") +} +section() { printf '\n%s== %s ==%s\n' "$B" "$*" "$N"; } + +summary() { + echo "" + if [ "${#BLOCKERS[@]}" -gt 0 ]; then + printf '%s--- BLOCKERS (fix/withdraw path) ---%s\n' "$R$B" "$N" + local b + for b in "${BLOCKERS[@]}"; do + printf '%s • %s%s\n' "$R" "$b" "$N" + done + fi + if [ "${#ERRORS[@]}" -gt 0 ] && [ "${#BLOCKERS[@]}" -lt "${#ERRORS[@]}" ]; then + printf '%s--- ERRORS ---%s\n' "$R" "$N" + local e + for e in "${ERRORS[@]}"; do + case "$e" in BLOCKER*) continue ;; esac + printf '%s • %s%s\n' "$R" "$e" "$N" + done + fi + printf 'totals: %s%d OK%s' "$G" "$PASS_N" "$N" + [ "$FAIL_N" -gt 0 ] && printf ', %s%d ERROR%s' "$R" "$FAIL_N" "$N" + [ "$WARN_N" -gt 0 ] && printf ', %s%d WARN%s' "$Y" "$WARN_N" "$N" + [ "$INFO_N" -gt 0 ] && printf ', %d INFO' "$INFO_N" + [ "${#BLOCKERS[@]}" -gt 0 ] && printf ', %s%d BLOCKER%s' "$R" "${#BLOCKERS[@]}" "$N" + printf '\n' + [ "$FAIL_N" -eq 0 ] +} + +http_code() { + local url="$1"; shift + curl -skS --max-redirs 0 -m "${TIMEOUT}" -o /dev/null -w '%{http_code}' "$@" "$url" 2>/dev/null || echo 000 +} + +http_body() { + local url="$1" out="$2"; shift 2 + curl -skS --max-redirs 0 -m "${TIMEOUT}" -o "$out" -w '%{http_code}' "$@" "$url" 2>/dev/null || echo 000 +} + +# --------------------------------------------------------------------------- +# Currency unit map checks (wallet codec: alt_unit_names must include "0") +# --------------------------------------------------------------------------- +# Returns 0 if JSON body at $1 has usable alt_unit_names. +# Supports: +# - exchange/bank: currency_specification.alt_unit_names +# - merchant: currencies..alt_unit_names for each code +# Optional $2 = required currency code for currency_specification.currency +json_has_alt_unit_names() { + local file="$1" want_cur="${2:-}" + python3 - "$file" "$want_cur" <<'PY' +import json, sys +path, want = sys.argv[1], sys.argv[2] +try: + d = json.load(open(path)) +except Exception as e: + print(f"json-error: {e}") + sys.exit(2) + +def check_au(au, label): + if not isinstance(au, dict) or not au: + print(f"{label}: missing/empty alt_unit_names") + return False + if "0" not in au or not str(au.get("0") or "").strip(): + print(f"{label}: alt_unit_names missing non-empty key \"0\" (have {sorted(au.keys())})") + return False + print(f"{label}: alt_unit_names ok (0={au.get('0')!r}, n={len(au)})") + return True + +ok = True +cs = d.get("currency_specification") +if isinstance(cs, dict): + if want and cs.get("currency") and cs.get("currency") != want: + print(f"currency_specification.currency={cs.get('currency')!r} want {want!r}") + ok = False + if not check_au(cs.get("alt_unit_names"), "currency_specification"): + ok = False +elif "currency_specification" in d: + print("currency_specification: not an object") + ok = False + +curs = d.get("currencies") +if isinstance(curs, dict) and curs: + for code, spec in curs.items(): + if not isinstance(spec, dict): + print(f"currencies.{code}: not an object") + ok = False + continue + if not check_au(spec.get("alt_unit_names"), f"currencies.{code}"): + ok = False + +if not isinstance(cs, dict) and not (isinstance(curs, dict) and curs): + # neither shape — fail + print("no currency_specification or currencies map") + ok = False + +sys.exit(0 if ok else 1) +PY +} + +# Check one exchange base URL's /config for alt_unit_names. +# $1=label $2=base_url $3=expected currency (optional) $4=strict(1) or soft(0) +check_exchange_alt_units() { + local label="$1" base="$2" want_cur="${3:-}" strict="${4:-1}" + local f code + base="${base%/}" + f=$(mktemp) + code=$(http_body "${base}/config" "$f") + if [ "$code" != "200" ]; then + rm -f "$f" + if [ "$strict" = "1" ]; then + fail "$label /config" "HTTP $code ($base)" + else + warn "$label /config" "HTTP $code ($base)" + fi + return + fi + local out ec + set +e + out=$(json_has_alt_unit_names "$f" "$want_cur" 2>&1) + ec=$? + set -e + rm -f "$f" + if [ "$ec" -eq 0 ]; then + ok "$label alt_unit_names" "$(echo "$out" | tr '\n' '; ' | sed 's/; $//')" + else + if [ "$strict" = "1" ]; then + fail "$label alt_unit_names" "$(echo "$out" | tr '\n' '; ' | sed 's/; $//')" + else + warn "$label alt_unit_names" "$(echo "$out" | tr '\n' '; ' | sed 's/; $//')" + fi + fi +} + +# From merchant /config JSON file: walk exchanges[] and check each base_url/config. +# Local stack hosts (hacktivism.ch or $EXCHANGE_PUBLIC host) are strict; others soft. +check_merchant_listed_exchanges_alt_units() { + local mer_json="$1" + local list + list=$(python3 - "$mer_json" <<'PY' +import json, sys +d = json.load(open(sys.argv[1])) +for e in d.get("exchanges") or []: + if not isinstance(e, dict): + continue + u = (e.get("base_url") or e.get("url") or "").rstrip("/") + c = e.get("currency") or "" + if u: + print(f"{c}\t{u}") +PY +) + if [ -z "$list" ]; then + fail "merchant exchanges[]" "empty — no exchanges to check for alt_unit_names" + return + fi + local line cur url strict host + while IFS=$'\t' read -r cur url; do + [ -n "$url" ] || continue + host="${url#https://}"; host="${host#http://}"; host="${host%%/*}" + strict=1 + case "$host" in + *hacktivism.ch) strict=1 ;; + *) + # foreign exchange (e.g. taler-ops) — soft unless it is our configured EXCHANGE_PUBLIC + if [ "$url" = "${EXCHANGE_PUBLIC}" ] || [ "$url" = "${EXCHANGE_PUBLIC}/" ]; then + strict=1 + else + strict=0 + fi + ;; + esac + check_exchange_alt_units "exchange ${cur:-?} ${host}" "$url" "$cur" "$strict" + done <<<"$list" +} + +SECRETS_ROOT="${SECRETS_ROOT:-}" +if [ -z "$SECRETS_ROOT" ]; then + for d in \ + "$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)/koopa-admin-secrets/koopa/host-root" \ + "/Users/newkamek/src/koopa/koopa-admin-secrets/koopa/host-root" \ + "$HOME/src/koopa/koopa-admin-secrets/koopa/host-root" + do + if [ -d "$d/taler-bank" ]; then SECRETS_ROOT=$d; break; fi + done +fi + +read_secret() { + local rel="$1" + if [ -n "${SECRETS_ROOT:-}" ] && [ -f "${SECRETS_ROOT}/${rel}" ]; then + tr -d '\n' <"${SECRETS_ROOT}/${rel}" + return 0 + fi + koopa_ssh_ok || return 1 + koopa_ssh_run 10 "tr -d '\\n' /dev/null || true" 2>/dev/null +} + +find_wallet_cli() { + if [ -n "${WALLET_CLI:-}" ] && [ -f "$WALLET_CLI" ]; then + echo "$WALLET_CLI"; return 0 + fi + for c in \ + /Users/newkamek/src/taler/taler-typescript-core/packages/taler-wallet-cli/bin/taler-wallet-cli.mjs \ + "$(command -v taler-wallet-cli 2>/dev/null || true)" + do + [ -n "$c" ] && [ -f "$c" ] && { echo "$c"; return 0; } + done + return 1 +} diff --git a/scripts/taler-monitoring/taler-monitoring.sh b/scripts/taler-monitoring/taler-monitoring.sh new file mode 100755 index 0000000..321525b --- /dev/null +++ b/scripts/taler-monitoring/taler-monitoring.sh @@ -0,0 +1,198 @@ +#!/usr/bin/env bash +# taler-monitoring — public URL / stack checks for a Taler domain +# +# ./taler-monitoring.sh # local GOA (urls + inside + e2e) +# ./taler-monitoring.sh -d taler.net urls # public demo, no SSH +# ./taler-monitoring.sh --domain taler-ops.ch # public ops, no SSH +# TALER_DOMAIN=demo.taler.net ./taler-monitoring.sh urls +# +# Tags: [OK] [INFO] [WARN] [ERROR] [BLOCKER] +# Exit 0 only if every selected phase exits 0. + +set -euo pipefail +ROOT=$(cd "$(dirname "$0")" && pwd) + +usage() { + cat <<'EOF' +taler-monitoring — bank / exchange / merchant checks + +Usage: + ./taler-monitoring.sh [options] [phases...] + +Phases: + urls public HTTPS (no SSH) ← default with --domain + inside container status via SSH (local stack only) + versions taler packages vs deb.taler.net trixie + repo availability + sanity public + optional server + server server-side only (SSH) + e2e withdraw + pay (small amounts; remote aborts on login/KYC) + all urls + inside + versions + sanity + e2e (SSH phases only on koopa) + +Options: + -d, --domain DOMAIN target domain + koopa / hacktivism.ch → local stack, SSH ok + everything else → no SSH; e2e optional (tiny amounts) + presets: koopa | hacktivism.ch | taler.net | taler-ops.ch + generic: bank/exchange/backend. + --bank URL override bank base (https://…) + --exchange URL override exchange base + --merchant URL override merchant base + --currency CODE expected currency (GOA, KUDOS, CHF, …); empty = report only + --no-probe do not probe alternate merchant/bank hostnames + -h, --help + +Examples: + ./taler-monitoring.sh -d taler.net + ./taler-monitoring.sh -d taler-ops.ch urls + ./taler-monitoring.sh -d demo.taler.net --currency KUDOS + ./taler-monitoring.sh --exchange https://exchange.taler-ops.ch urls + +Env (same meaning): + TALER_DOMAIN BANK_PUBLIC EXCHANGE_PUBLIC MERCHANT_PUBLIC EXPECT_CURRENCY + SKIP_SSH=1 NO_COLOR=1 +EOF +} + +# shellcheck source=lib.sh +source "$ROOT/lib.sh" + +PHASES=() +DOMAIN_SET=0 +BANK_OVERRIDE="" +EXCHANGE_OVERRIDE="" +MERCHANT_OVERRIDE="" +CURRENCY_OVERRIDE="" +NO_PROBE=0 + +while [ $# -gt 0 ]; do + case "$1" in + -h|--help) usage; exit 0 ;; + -d|--domain) + [ $# -ge 2 ] || { echo "missing arg for $1" >&2; exit 2; } + apply_taler_domain "$2" + DOMAIN_SET=1 + shift 2 + ;; + --bank) + [ $# -ge 2 ] || { echo "missing arg for $1" >&2; exit 2; } + BANK_OVERRIDE="${2%/}"; shift 2 + ;; + --exchange) + [ $# -ge 2 ] || { echo "missing arg for $1" >&2; exit 2; } + EXCHANGE_OVERRIDE="${2%/}"; shift 2 + ;; + --merchant) + [ $# -ge 2 ] || { echo "missing arg for $1" >&2; exit 2; } + MERCHANT_OVERRIDE="${2%/}"; shift 2 + ;; + --currency) + [ $# -ge 2 ] || { echo "missing arg for $1" >&2; exit 2; } + CURRENCY_OVERRIDE="$2"; shift 2 + ;; + --no-probe) NO_PROBE=1; shift ;; + urls|inside|versions|sanity|server|e2e|all) PHASES+=("$1"); shift ;; + *) + # bare domain shorthand: ./taler-monitoring.sh taler.net + if [[ "$1" == *.* && "$1" != *://* && "$1" != -* ]]; then + apply_taler_domain "$1" + DOMAIN_SET=1 + shift + else + echo "Unknown: $1" >&2; usage >&2; exit 2 + fi + ;; + esac +done + +if [ "$NO_PROBE" = "1" ]; then + TALER_DOMAIN_PROBE=0 +fi +if [ -n "$CURRENCY_OVERRIDE" ]; then + EXPECT_CURRENCY="$CURRENCY_OVERRIDE" +fi +if [ -n "$BANK_OVERRIDE" ]; then BANK_PUBLIC="$BANK_OVERRIDE"; fi +if [ -n "$EXCHANGE_OVERRIDE" ]; then EXCHANGE_PUBLIC="$EXCHANGE_OVERRIDE"; fi +if [ -n "$MERCHANT_OVERRIDE" ]; then MERCHANT_PUBLIC="$MERCHANT_OVERRIDE"; fi + +# Only koopa may use SSH. Remote domains: public + optional e2e (no SSH). +if [ "${LOCAL_STACK:-1}" != "1" ]; then + SKIP_SSH=1 +fi + +# Remote: no SSH; ATM withdraw ladder set in check_e2e (smaller notes) +if [ "${LOCAL_STACK}" != "1" ]; then + E2E_FAKE_INCOMING=0 + E2E_REMOTE=1 + : "${E2E_WITHDRAW_VALUES:=10 20 50}" + : "${E2E_PAY_VALUES:=0.01 0.05 0.1 1}" +fi + +# Export so check_*.sh (re-source lib) see the same targets via env +export TALER_DOMAIN BANK_PUBLIC EXCHANGE_PUBLIC MERCHANT_PUBLIC +export EXPECT_CURRENCY SKIP_SSH LOCAL_STACK TALER_DOMAIN_PROBE +export WITHDRAW_AMT PAY_AMT CREDIT_AMT MERCHANT_INSTANCE +export E2E_FAKE_INCOMING E2E_REMOTE E2E_VARIABLE E2E_ATM_MAX +export E2E_WITHDRAW_VALUES E2E_PAY_VALUES +export TALER_DOMAIN_APPLIED=1 + +# Default phases +if [ "${#PHASES[@]}" -eq 0 ]; then + if [ "${LOCAL_STACK}" = "1" ]; then + PHASES=(urls inside versions e2e) # koopa: SSH ok + else + PHASES=(urls) # remote default: public only; pass "e2e" to try + fi +fi + +OUT_PHASES=() +for p in "${PHASES[@]}"; do + if [ "$p" = "all" ]; then + if [ "${LOCAL_STACK}" = "1" ]; then + OUT_PHASES+=(urls inside versions sanity e2e) + else + # remote: repo index check only (no installed pkgs without SSH) + OUT_PHASES+=(urls versions e2e) + fi + else + OUT_PHASES+=("$p") + fi +done +PHASES=() +seen=" " +for p in "${OUT_PHASES[@]}"; do + # Drop SSH-only phases for non-koopa (e2e is allowed — public path only) + # versions still runs: outside-in deb.taler.net checks; container parts soft-skip if no SSH + if [ "${LOCAL_STACK}" != "1" ] || [ "${SKIP_SSH}" = "1" ]; then + case "$p" in inside|server) + echo "[INFO] skip phase '$p' (only koopa uses SSH)" >&2 + continue + ;; + esac + fi + case "$seen" in *" $p "*) ;; *) PHASES+=("$p"); seen="$seen$p " ;; esac +done + +if [ "${#PHASES[@]}" -eq 0 ]; then + PHASES=(urls) +fi + +printf 'target domain=%s\n' "${TALER_DOMAIN}" +printf ' bank %s\n' "$BANK_PUBLIC" +printf ' exchange %s\n' "$EXCHANGE_PUBLIC" +printf ' merchant %s\n' "$MERCHANT_PUBLIC" +printf ' currency expect=%s skip_ssh=%s\n' "${EXPECT_CURRENCY:-any}" "$SKIP_SSH" + +chmod +x "$ROOT"/check_*.sh 2>/dev/null || true + +ec=0 +for p in "${PHASES[@]}"; do + case "$p" in + urls) "$ROOT/check_urls.sh" || ec=1 ;; + inside) "$ROOT/check_inside.sh" || ec=1 ;; + versions) "$ROOT/check_versions.sh" || ec=1 ;; + sanity) "$ROOT/check_sanity.sh" || ec=1 ;; + server) "$ROOT/check_server.sh" || ec=1 ;; + e2e) "$ROOT/check_e2e.sh" || ec=1 ;; + esac +done +exit "$ec"