monitoring: taler-monitoring www/inside/sanity/server/e2e scaffold
This commit is contained in:
Hernâni Marques
parent
fed083d0d5
commit
95499a7f59
11 changed files with 2828 additions and 0 deletions
95
scripts/taler-monitoring/README.md
Normal file
95
scripts/taler-monitoring/README.md
Normal file
|
|
@ -0,0 +1,95 @@
|
|||
# taler-monitoring
|
||||
|
||||
Report for the **GOA** stack with clear severity tags and **per-area test IDs**:
|
||||
|
||||
| Tag | Meaning |
|
||||
|-----|---------|
|
||||
| `[OK]` | check passed |
|
||||
| `[INFO]` | inside status (container, ports, log noise) |
|
||||
| `[WARN]` | degraded but maybe not fatal |
|
||||
| `[ERROR]` | component problem |
|
||||
| `[BLOCKER]` | **withdraw/pay path cannot complete** because of this |
|
||||
|
||||
IDs: **`www-001`**, **`inside-001`**, **`versions-001`**, **`sanity-001`**, **`server-001`**, **`e2e-001`** …
|
||||
Catalog: **[TESTS.md](./TESTS.md)**.
|
||||
|
||||
```text
|
||||
[OK] www-001 exchange /config https://exchange…/config
|
||||
[BLOCKER] e2e-015 prereq: merchant HTTP 502
|
||||
```
|
||||
|
||||
End of each phase: totals + list of **BLOCKERS** and **ERRORS**.
|
||||
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
# Local GOA stack (may use SSH for inside/e2e)
|
||||
./taler-monitoring.sh # urls + inside + versions + e2e
|
||||
./taler-monitoring.sh inside # containers on koopa
|
||||
./taler-monitoring.sh versions # deb.taler.net + package versions vs trixie
|
||||
./taler-monitoring.sh sanity
|
||||
./taler-monitoring.sh e2e
|
||||
|
||||
# Other domains — public HTTPS only, never SSH
|
||||
./taler-monitoring.sh -d taler.net
|
||||
./taler-monitoring.sh --domain taler-ops.ch
|
||||
./taler-monitoring.sh -d demo.taler.net urls
|
||||
./taler-monitoring.sh taler.net # bare domain = same as -d
|
||||
```
|
||||
|
||||
| Domain | Bank | Exchange | Merchant | Currency |
|
||||
|--------|------|----------|----------|----------|
|
||||
| `hacktivism.ch` (default) | bank.hacktivism.ch | exchange.hacktivism.ch | taler.hacktivism.ch | GOA |
|
||||
| `taler.net` / `demo.taler.net` | bank.demo.taler.net | exchange.demo.taler.net | backend.demo.taler.net | KUDOS |
|
||||
| `taler-ops.ch` | bank.* (probe) | exchange.taler-ops.ch | backend.* (probe) | CHF |
|
||||
| other | bank.DOMAIN | exchange.DOMAIN | backend/taler/merchant (probe) | any |
|
||||
|
||||
**SSH only for koopa** (`hacktivism.ch` / `-d koopa`).
|
||||
|
||||
Other domains: never SSH. Optional **e2e** aborts cleanly on login/KYC.
|
||||
|
||||
### E2E amounts (variable)
|
||||
|
||||
| | Local (koopa) | Remote |
|
||||
|--|---------------|--------|
|
||||
| **ATM withdraw** | 20 · 50 · 100 · 200 | 10 · 20 · 50 |
|
||||
| **Pay ladder** | 0.01 … 10 | 0.01 … 1 |
|
||||
|
||||
```bash
|
||||
./taler-monitoring.sh e2e
|
||||
./taler-monitoring.sh -d taler.net urls e2e
|
||||
# customize:
|
||||
E2E_WITHDRAW_VALUES="20 50 100" E2E_PAY_VALUES="0.05 1 5" ./taler-monitoring.sh e2e
|
||||
E2E_VARIABLE=0 WITHDRAW_AMT=GOA:50 PAY_AMT=GOA:1 ./taler-monitoring.sh e2e # single fixed
|
||||
# remote secrets:
|
||||
# E2E_BANK_ADMIN_PASS=… E2E_MERCHANT_TOKEN=…
|
||||
```
|
||||
|
||||
## Phases
|
||||
|
||||
| Phase | What |
|
||||
|-------|------|
|
||||
| **inside** | SSH koopa: processes, postgres, local /config,/keys, wirewatch, recent log ERRORs |
|
||||
| **versions** | `deb.taler.net` reachable (InRelease/Packages/pool `.deb`); containers can reach it + have apt source; installed Taler packages vs **trixie** |
|
||||
| **sanity** | bank · exchange · merchant sections (public + server) |
|
||||
| **e2e** | account → credit → withdraw → wallet → confirm → order → pay |
|
||||
|
||||
```bash
|
||||
# package suite (default trixie on deb.taler.net)
|
||||
TALER_APT_SUITE=trixie ./taler-monitoring.sh versions
|
||||
TALER_PKG_BEHIND=error ./taler-monitoring.sh versions # any behind = ERROR
|
||||
```
|
||||
|
||||
E2E maps failures to blockers, e.g.:
|
||||
|
||||
- `bank-confirm HTTP 409` → wallet did not select exchange
|
||||
- `no GOA balance` → wirewatch / transfer
|
||||
- `create order failed` → merchant auth/instance
|
||||
- `Alarm clock` during pay → usually missing `handle-uri --yes` or wrong pay URI (must be `…/instances/{inst}/{oid}/?c={token}` from merchant `taler_pay_uri`)
|
||||
- `insufficient balance` → withdraw incomplete
|
||||
|
||||
## Needs
|
||||
|
||||
- SSH `koopa` (inside/sanity server bits)
|
||||
- secrets under `koopa-admin-secrets/...` for e2e
|
||||
- `taler-wallet-cli` for e2e
|
||||
149
scripts/taler-monitoring/TESTS.md
Normal file
149
scripts/taler-monitoring/TESTS.md
Normal file
|
|
@ -0,0 +1,149 @@
|
|||
# taler-monitoring — test IDs by area
|
||||
|
||||
Every check line is numbered **per area** as `AREA-NNN` (zero-padded):
|
||||
|
||||
| Area | Phase script | Meaning |
|
||||
|------|--------------|---------|
|
||||
| **www** | `check_urls.sh` | public HTTPS (outside-in) |
|
||||
| **inside** | `check_inside.sh` | containers / processes on koopa |
|
||||
| **versions** | `check_versions.sh` | deb.taler.net available + packages vs trixie |
|
||||
| **sanity** | `check_sanity.sh` | public + server per component |
|
||||
| **server** | `check_server.sh` | SSH host ports / processes |
|
||||
| **e2e** | `check_e2e.sh` | withdraw + pay cycle |
|
||||
|
||||
Format in output:
|
||||
|
||||
```text
|
||||
[OK] www-001 exchange /config https://exchange…/config
|
||||
[ERROR] e2e-012 bank-auth: admin token failed
|
||||
[BLOCKER] e2e-015 prereq: merchant HTTP 502
|
||||
```
|
||||
|
||||
IDs are assigned **in run order** within the area (`set_area` resets the counter). Optional soft checks still consume a number when they WARN.
|
||||
|
||||
---
|
||||
|
||||
## www — public URLs (`./taler-monitoring.sh urls`)
|
||||
|
||||
| ID | Check |
|
||||
|----|--------|
|
||||
| www-… | exchange `/config`, currency, **alt_unit_names** |
|
||||
| www-… | exchange `/keys` (+ alt_unit_names soft) |
|
||||
| www-… | exchange `/intro/`, `/` (302→intro) |
|
||||
| www-… | **exchange `/terms`** body (not empty / not API error) |
|
||||
| www-… | **exchange `/privacy`** body |
|
||||
| www-… | exchange `/terms/` (200 or redirect) |
|
||||
| www-… | bank `/config`, currency, **alt_unit_names** |
|
||||
| www-… | bank integration / webui / intro / `/` |
|
||||
| www-… | **bank `/terms`** body |
|
||||
| www-… | **bank `/privacy`** (or `/intro/privacy.html` fallback) |
|
||||
| www-… | merchant `/config`, currency, currencies alt_unit_names |
|
||||
| www-… | each merchant `exchanges[]` `/config` alt_unit_names |
|
||||
| www-… | merchant `/intro/`, `/webui/`, `/` |
|
||||
| www-… | **merchant `/terms`** body (dual-currency notice) |
|
||||
| www-… | **merchant `/privacy`** body (must not be `not configured`) |
|
||||
| www-… | merchant `/terms/` redirect |
|
||||
|
||||
**Legal docs rule:** HTTP 200, non-empty body, not plain `not configured`, not merchant API JSON `code:21`. On local stack, optional content needle (terms/privacy/FADP/GOA…).
|
||||
|
||||
**alt_unit_names rule:** wallet codec requires a non-empty map including scale key `"0"`. For multi-currency merchant, also follow every entry in `exchanges[]` and check that exchange’s public `/config`.
|
||||
|
||||
(IDs after a failed early check may shift if later soft checks are skipped when body missing — numbering follows **executed** checks.)
|
||||
|
||||
---
|
||||
|
||||
## inside — koopa SSH (`./taler-monitoring.sh inside`)
|
||||
|
||||
| ID | Check (typical order) |
|
||||
|----|------------------------|
|
||||
| inside-001 | ssh koopa |
|
||||
| inside-002+ | per-component emit: container, ports, libeufin/httpd, postgres, local `/config`/`/keys`, wirewatch, DNS pin, caddy |
|
||||
|
||||
Remote lines `E|comp|LEVEL|key|detail` each become one numbered result.
|
||||
|
||||
---
|
||||
|
||||
## sanity — bank · exchange · merchant (`./taler-monitoring.sh sanity`)
|
||||
|
||||
| ID | Section |
|
||||
|----|---------|
|
||||
| sanity-001… | bank public + server |
|
||||
| sanity-… | exchange public + server |
|
||||
| sanity-… | merchant public + server |
|
||||
|
||||
Sequential through the whole script (one `set_area sanity`).
|
||||
|
||||
---
|
||||
|
||||
## versions — packages vs deb.taler.net (`./taler-monitoring.sh versions`)
|
||||
|
||||
### Outside (runner / public network — no SSH)
|
||||
|
||||
| ID (order) | Check |
|
||||
|------------|--------|
|
||||
| versions-… | DNS `deb.taler.net` |
|
||||
| versions-… | HTTPS portal + apt base URL |
|
||||
| versions-… | suite `InRelease` / `Release` |
|
||||
| versions-… | suite `Packages` + `Packages.gz` |
|
||||
| versions-… | sample pool `.deb` fetchable (Range 200/206) |
|
||||
| versions-… | suite offers `taler-exchange`, `taler-merchant`, `libeufin-bank` |
|
||||
| versions-… | optional `trixie-testing` Packages |
|
||||
| versions-… | TLS verify (soft) |
|
||||
|
||||
### Inside (SSH koopa containers)
|
||||
|
||||
| ID | Check |
|
||||
|----|--------|
|
||||
| versions-… | ssh koopa |
|
||||
| versions-… | each container → `InRelease` (pasta can reach apt repo) |
|
||||
| versions-… | each container lists `deb.taler.net` in apt sources |
|
||||
| versions-… | each installed `taler*` / `libeufin*` / `libtaler*` / `libdonau*` vs suite version |
|
||||
| versions-… | core packages installed (`taler-exchange`, `libeufin-bank`, `taler-merchant`) |
|
||||
|
||||
Outside always runs. Inside skipped with `SKIP_SSH=1` (still reports outside results).
|
||||
|
||||
Compare rules:
|
||||
|
||||
- **match** suite → OK
|
||||
- **ahead** of suite (often testing/dev) → INFO
|
||||
- **behind** suite → ERROR for core packages, WARN otherwise (`TALER_PKG_BEHIND=error` forces ERROR)
|
||||
|
||||
Default suite: **trixie** (`TALER_APT_SUITE`, `TALER_APT_BASE=https://deb.taler.net/apt/debian`).
|
||||
|
||||
Without SSH (`SKIP_SSH=1` or remote domain): still runs outside-in repo checks; skips container install compare.
|
||||
|
||||
---
|
||||
|
||||
## server — SSH ports (`./taler-monitoring.sh server`)
|
||||
|
||||
| ID | Check |
|
||||
|----|--------|
|
||||
| server-001 | ssh |
|
||||
| server-002+ | containers, local pasta ports, processes, caddy |
|
||||
|
||||
---
|
||||
|
||||
## e2e — payment path (`./taler-monitoring.sh e2e`)
|
||||
|
||||
| ID | Step (approx.) |
|
||||
|----|----------------|
|
||||
| e2e-001 | budget info |
|
||||
| e2e-002 | wallet-cli present |
|
||||
| e2e-003 | mode / currency info |
|
||||
| e2e-004… | secrets, reachability gates |
|
||||
| e2e-… | account, credit, withdraw, confirm, coins, order, pay |
|
||||
| e2e-… | balances, dig on failure |
|
||||
|
||||
Blockers keep the same ID prefix: `[BLOCKER] e2e-0NN step: message`.
|
||||
|
||||
---
|
||||
|
||||
## Run one area
|
||||
|
||||
```bash
|
||||
./taler-monitoring.sh urls # www only
|
||||
./taler-monitoring.sh inside # inside only
|
||||
./taler-monitoring.sh versions # deb.taler.net + package drift
|
||||
./taler-monitoring.sh e2e # e2e only
|
||||
./taler-monitoring.sh -d taler.net urls
|
||||
```
|
||||
997
scripts/taler-monitoring/check_e2e.sh
Executable file
997
scripts/taler-monitoring/check_e2e.sh
Executable file
|
|
@ -0,0 +1,997 @@
|
|||
#!/usr/bin/env bash
|
||||
# Full withdraw + pay cycle; clear BLOCKERS when the path cannot finish.
|
||||
# Skips remaining steps if overall budget exceeded (E2E_TIMEOUT, default 90s).
|
||||
set -euo pipefail
|
||||
ROOT=$(cd "$(dirname "$0")" && pwd)
|
||||
# shellcheck source=lib.sh
|
||||
source "$ROOT/lib.sh"
|
||||
|
||||
: "${E2E_TIMEOUT:=180}" # ATM ladder + wirewatch lag needs room (was 55 — too tight)
|
||||
: "${E2E_PAY_SECS:=22}" # hard reserve for handle-uri + settle (do not starve pay)
|
||||
: "${E2E_SETTLE_ROUNDS:=18}"
|
||||
: "${E2E_SETTLE_SLEEP:=3}"
|
||||
E2E_START=$(date +%s)
|
||||
BAL_BEFORE="(not captured)"
|
||||
BAL_AFTER="(not captured)"
|
||||
E2E_REPORTED=0
|
||||
e2e_left() { echo $(( E2E_TIMEOUT - ($(date +%s) - E2E_START) )); }
|
||||
e2e_over() { [ "$(e2e_left)" -le 0 ]; }
|
||||
|
||||
# Fixed short timeout — used even when e2e budget is exhausted / on EXIT
|
||||
# stdout only for JSON; logs go to stderr file so balance parse stays clean
|
||||
wcli_bal_snap() {
|
||||
local out="$1"
|
||||
if [ -z "${WALLET_CLI:-}" ] || [ ! -f "${WDB:-}" ]; then
|
||||
echo "(no wallet)" >"$out"
|
||||
return 1
|
||||
fi
|
||||
local err="${out}.err"
|
||||
if command -v perl >/dev/null 2>&1; then
|
||||
perl -e 'alarm shift; exec @ARGV' 10 \
|
||||
node "$WALLET_CLI" --wallet-db="$WDB" --no-throttle --skip-defaults balance \
|
||||
>"$out" 2>"$err" || true
|
||||
else
|
||||
node "$WALLET_CLI" --wallet-db="$WDB" --no-throttle --skip-defaults balance \
|
||||
>"$out" 2>"$err" || true
|
||||
fi
|
||||
[ -s "$out" ]
|
||||
}
|
||||
|
||||
fmt_bal() {
|
||||
python3 -c '
|
||||
import json,re,sys
|
||||
raw=open(sys.argv[1]).read() if sys.argv[1] else ""
|
||||
if not raw.strip() or raw.strip().startswith("(no"):
|
||||
print(raw.strip() or "(empty)"); sys.exit(0)
|
||||
# extract first JSON object with "balances"
|
||||
d=None
|
||||
for m in re.finditer(r"\{", raw):
|
||||
try:
|
||||
d=json.loads(raw[m.start():])
|
||||
if isinstance(d, dict) and "balances" in d:
|
||||
break
|
||||
except Exception:
|
||||
d=None
|
||||
if not isinstance(d, dict):
|
||||
print(re.sub(r"\s+"," ", raw)[:200]); sys.exit(0)
|
||||
parts=[]
|
||||
for b in d.get("balances") or []:
|
||||
cur=(b.get("scopeInfo") or {}).get("currency") or "?"
|
||||
parts.append("%s avail=%s pendingIn=%s" % (cur, b.get("available"), b.get("pendingIncoming")))
|
||||
print("; ".join(parts) if parts else "(no balances)")
|
||||
' "${1:-/dev/null}" 2>/dev/null || echo "(unreadable)"
|
||||
}
|
||||
|
||||
# Numeric available for currency CUR (from balance file or fresh wallet snap)
|
||||
wallet_avail_num() {
|
||||
local f="${1:-}"
|
||||
if [ -z "$f" ] || [ ! -s "$f" ]; then
|
||||
f="$SCRATCH/bal-live.out"
|
||||
wcli_bal_snap "$f" || wcli balance >"$f" 2>&1 || true
|
||||
fi
|
||||
python3 -c '
|
||||
import json,re,sys
|
||||
cur=sys.argv[2]
|
||||
raw=open(sys.argv[1]).read() if sys.argv[1] else ""
|
||||
d=None
|
||||
for m in re.finditer(r"\{", raw):
|
||||
try:
|
||||
d=json.loads(raw[m.start():])
|
||||
if isinstance(d, dict) and "balances" in d: break
|
||||
except Exception:
|
||||
d=None
|
||||
if isinstance(d, dict):
|
||||
for b in d.get("balances") or []:
|
||||
c=(b.get("scopeInfo") or {}).get("currency") or ""
|
||||
av=b.get("available") or ""
|
||||
if c==cur or av.startswith(cur+":"):
|
||||
try:
|
||||
print(float(av.split(":",1)[1])); sys.exit(0)
|
||||
except Exception:
|
||||
pass
|
||||
# text fallback
|
||||
m=re.search(r"%s:([0-9]+(?:\.[0-9]+)?)" % re.escape(cur), raw)
|
||||
print(float(m.group(1)) if m else 0.0)
|
||||
' "${f:-/dev/null}" "${CUR:-GOA}" 2>/dev/null || echo 0
|
||||
}
|
||||
|
||||
# Wait for spendable balance (settlement / wirewatch lag). Returns 0 if avail > min.
|
||||
# Emphasizes timing, not hard failure, while waiting.
|
||||
wait_wallet_balance() {
|
||||
local min_n="${1:-0}"
|
||||
local rounds="${2:-15}"
|
||||
local sleep_s="${3:-3}"
|
||||
local r av
|
||||
info "wallet settle wait" "up to ${rounds}×${sleep_s}s for avail>${min_n} ${CUR} (wirewatch lag is common)"
|
||||
for r in $(seq 1 "$rounds"); do
|
||||
wcli run-until-done >"$SCRATCH/settle-run.out" 2>&1 || true
|
||||
wcli_bal_snap "$SCRATCH/bal-live.out" || wcli balance >"$SCRATCH/bal-live.out" 2>&1 || true
|
||||
av=$(wallet_avail_num "$SCRATCH/bal-live.out")
|
||||
if python3 -c "import sys; sys.exit(0 if float(sys.argv[1]) > float(sys.argv[2]) else 1)" "$av" "$min_n" 2>/dev/null; then
|
||||
ok "wallet balance ready" "avail=${CUR}:${av} after ~$((r * sleep_s))s settle wait"
|
||||
info "balance" "$(fmt_bal "$SCRATCH/bal-live.out")"
|
||||
return 0
|
||||
fi
|
||||
if [ "$r" = "1" ] || [ "$((r % 3))" = "0" ]; then
|
||||
info "settlement lag" "round $r/${rounds}: avail=${CUR}:${av} (still waiting — not an error yet)"
|
||||
fi
|
||||
# gentle wirewatch nudge mid-wait (local only)
|
||||
if [ "$r" = "4" ] || [ "$r" = "8" ]; then
|
||||
if [ "${E2E_REMOTE:-0}" != "1" ] && koopa_ssh_ok; then
|
||||
koopa_ssh_run 8 \
|
||||
'podman exec taler-hacktivism-exchange-ansible systemctl restart taler-exchange-wirewatch 2>/dev/null || true' \
|
||||
>/dev/null 2>&1 || true
|
||||
fi
|
||||
fi
|
||||
sleep "$sleep_s"
|
||||
done
|
||||
av=$(wallet_avail_num "$SCRATCH/bal-live.out")
|
||||
warn "settlement lag" "after ${rounds}×${sleep_s}s still avail=${CUR}:${av} — coins may still be in flight"
|
||||
return 1
|
||||
}
|
||||
|
||||
print_balances() {
|
||||
section "e2e · wallet balances"
|
||||
info "BALANCE before" "$BAL_BEFORE"
|
||||
# refresh after if we can
|
||||
if [ -n "${SCRATCH:-}" ]; then
|
||||
wcli_bal_snap "$SCRATCH/bal-end.out" || true
|
||||
if [ -s "$SCRATCH/bal-end.out" ]; then
|
||||
BAL_AFTER=$(fmt_bal "$SCRATCH/bal-end.out")
|
||||
fi
|
||||
fi
|
||||
info "BALANCE after " "$BAL_AFTER"
|
||||
}
|
||||
|
||||
e2e_finish() {
|
||||
local code="${1:-1}"
|
||||
if [ "$E2E_REPORTED" = "1" ]; then
|
||||
return "$code"
|
||||
fi
|
||||
E2E_REPORTED=1
|
||||
print_balances
|
||||
summary || true
|
||||
return "$code"
|
||||
}
|
||||
|
||||
e2e_skip_rest() {
|
||||
local why="$1"
|
||||
warn "e2e timeout" "budget ${E2E_TIMEOUT}s exceeded — $why"
|
||||
blocker "e2e-timeout" "skipped remaining steps after ${E2E_TIMEOUT}s ($why)"
|
||||
section "e2e · report"
|
||||
info "WITHDRAW" "incomplete (timeout)"
|
||||
info "PAY" "SKIPPED (timeout)"
|
||||
# If we already started a withdrawal, dig inside for missing coins
|
||||
if [ -n "${WID:-}" ]; then
|
||||
dig_when_no_coins 2>/dev/null || true
|
||||
fi
|
||||
e2e_finish 1
|
||||
exit 1
|
||||
}
|
||||
|
||||
# Area e2e-### — withdraw + pay cycle
|
||||
set_area e2e
|
||||
section "e2e · prerequisites"
|
||||
info "e2e budget" "${E2E_TIMEOUT}s (set E2E_TIMEOUT= to change)"
|
||||
|
||||
# Remote domains (not koopa): public path only, tiny amounts, abort on login/KYC
|
||||
: "${E2E_REMOTE:=0}"
|
||||
if [ "${LOCAL_STACK:-1}" != "1" ]; then
|
||||
E2E_REMOTE=1
|
||||
SKIP_SSH=1
|
||||
E2E_FAKE_INCOMING=0
|
||||
fi
|
||||
|
||||
# Currency for amounts / balance checks
|
||||
CUR="${EXPECT_CURRENCY:-}"
|
||||
if [ -z "$CUR" ]; then
|
||||
CUR=$(curl -skS -m 8 "$BANK_PUBLIC/config" 2>/dev/null | python3 -c 'import sys,json
|
||||
try: print(json.load(sys.stdin).get("currency") or "")
|
||||
except Exception: print("")' 2>/dev/null || true)
|
||||
fi
|
||||
[ -z "$CUR" ] && CUR=$(curl -skS -m 8 "$EXCHANGE_PUBLIC/config" 2>/dev/null | python3 -c 'import sys,json
|
||||
try: print(json.load(sys.stdin).get("currency") or "")
|
||||
except Exception: print("")' 2>/dev/null || true)
|
||||
[ -z "$CUR" ] && CUR="GOA"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Variable amounts
|
||||
# Withdraw = typical ATM notes only (Bankomat): 20 / 50 / 100 / 200 …
|
||||
# Pay = several smaller purchase amounts
|
||||
# Env:
|
||||
# E2E_WITHDRAW_VALUES="20 50 100" # bare numbers → prefixed with CUR
|
||||
# E2E_PAY_VALUES="0.01 0.05 0.5 1 2 5"
|
||||
# E2E_VARIABLE=0 # single fixed WITHDRAW_AMT / PAY_AMT
|
||||
# E2E_ATM_MAX=N # try at most N ATM withdraws (budget)
|
||||
# ---------------------------------------------------------------------------
|
||||
: "${E2E_VARIABLE:=1}"
|
||||
: "${E2E_ATM_MAX:=4}"
|
||||
if [ "$E2E_REMOTE" = "1" ]; then
|
||||
# remote: still ATM-shaped but smaller notes; keep cheap
|
||||
: "${E2E_WITHDRAW_VALUES:=10 20 50}"
|
||||
: "${E2E_PAY_VALUES:=0.01 0.05 0.1 1}"
|
||||
else
|
||||
# local GOA: classic ATM denominations
|
||||
: "${E2E_WITHDRAW_VALUES:=20 50 100 200}"
|
||||
: "${E2E_PAY_VALUES:=0.01 0.05 0.1 0.5 1 2 5 10}"
|
||||
fi
|
||||
|
||||
# Build CUR:amount lists
|
||||
build_amt_list() {
|
||||
local cur="$1"; shift
|
||||
local v out=""
|
||||
for v in "$@"; do
|
||||
case "$v" in
|
||||
*:*) out="${out}${out:+ }$v" ;;
|
||||
*) out="${out}${out:+ }${cur}:${v}" ;;
|
||||
esac
|
||||
done
|
||||
printf '%s' "$out"
|
||||
}
|
||||
# shellcheck disable=SC2086
|
||||
WITHDRAW_LIST=$(build_amt_list "$CUR" $E2E_WITHDRAW_VALUES)
|
||||
# shellcheck disable=SC2086
|
||||
PAY_LIST=$(build_amt_list "$CUR" $E2E_PAY_VALUES)
|
||||
|
||||
if [ "$E2E_VARIABLE" != "1" ]; then
|
||||
if [ "$E2E_REMOTE" = "1" ]; then
|
||||
WITHDRAW_LIST="${WITHDRAW_AMT:-${CUR}:20}"
|
||||
PAY_LIST="${PAY_AMT:-${CUR}:0.01}"
|
||||
else
|
||||
WITHDRAW_LIST="${WITHDRAW_AMT:-${CUR}:20}"
|
||||
PAY_LIST="${PAY_AMT:-${CUR}:0.01}"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Credit = sum of planned ATM withdraws + small buffer for fees
|
||||
CREDIT_AMT=$(python3 -c '
|
||||
import sys
|
||||
cur=sys.argv[1]
|
||||
vals=sys.argv[2].split()
|
||||
s=0.0
|
||||
for a in vals:
|
||||
try: s += float(a.split(":",1)[-1])
|
||||
except Exception: pass
|
||||
buf = max(10.0, s * 0.05)
|
||||
print("%s:%g" % (cur, s + buf))
|
||||
' "$CUR" "$WITHDRAW_LIST" 2>/dev/null || echo "${CUR}:100")
|
||||
|
||||
# First withdraw/pay for legacy single-step labels
|
||||
WITHDRAW_AMT=$(printf '%s' "$WITHDRAW_LIST" | awk '{print $1}')
|
||||
PAY_AMT=$(printf '%s' "$PAY_LIST" | awk '{print $1}')
|
||||
|
||||
# More wall time when running ATM ladder
|
||||
if [ "$E2E_VARIABLE" = "1" ]; then
|
||||
n_w=$(printf '%s' "$WITHDRAW_LIST" | wc -w | tr -d ' ')
|
||||
n_p=$(printf '%s' "$PAY_LIST" | wc -w | tr -d ' ')
|
||||
need=$(( 40 + n_w * 35 + n_p * 20 ))
|
||||
if [ "${E2E_TIMEOUT}" -lt "$need" ]; then
|
||||
E2E_TIMEOUT=$need
|
||||
fi
|
||||
# cap ATM attempts
|
||||
WITHDRAW_LIST=$(printf '%s' "$WITHDRAW_LIST" | tr ' ' '\n' | head -n "$E2E_ATM_MAX" | tr '\n' ' ' | sed 's/ *$//')
|
||||
fi
|
||||
|
||||
BANK_HOST=$(python3 -c 'from urllib.parse import urlparse; print(urlparse("'"$BANK_PUBLIC"'").hostname or "bank")' 2>/dev/null || echo bank)
|
||||
|
||||
SCRATCH=$(mktemp -d)
|
||||
WDB="$SCRATCH/wallet.sqlite3"
|
||||
export PATH="/opt/homebrew/bin:/usr/local/bin:$PATH"
|
||||
# Always dump balances on any exit (timeout, blocker, signal, success)
|
||||
trap 'ec=$?; e2e_finish "$ec"; rm -rf "$SCRATCH"; exit "$ec"' EXIT
|
||||
|
||||
# Abort cleanly on login / KYC / registration barriers (esp. remote domains)
|
||||
e2e_abort_auth() {
|
||||
local step="$1" msg="$2"
|
||||
warn "$step" "$msg"
|
||||
blocker "$step" "abort (login/KYC): $msg"
|
||||
section "e2e · report"
|
||||
info "ABORTED" "login or KYC blocked further e2e — not retrying"
|
||||
info "hint" "remote e2e needs open bank registration + credit path; set E2E_BANK_ADMIN_PASS / E2E_MERCHANT_TOKEN if you have them"
|
||||
e2e_finish 1
|
||||
exit 1
|
||||
}
|
||||
|
||||
is_auth_kyc_body() {
|
||||
# stdin or file arg: true if looks like login/KYC barrier
|
||||
local f="${1:-}"
|
||||
local t
|
||||
if [ -n "$f" ] && [ -f "$f" ]; then t=$(head -c 800 "$f" 2>/dev/null || true)
|
||||
else t=$(cat 2>/dev/null || true); fi
|
||||
printf '%s' "$t" | grep -qiE 'kyc|legitim|login required|unauthorized|forbidden|captcha|challenge|tan_|not.?allowed|registration.?disabled|admin.?only|permission.?denied|401|403'
|
||||
}
|
||||
|
||||
WALLET_CLI=$(find_wallet_cli) || {
|
||||
blocker "prereq" "taler-wallet-cli not found (set WALLET_CLI=)"
|
||||
exit 1
|
||||
}
|
||||
ok "wallet-cli ($WALLET_CLI)"
|
||||
info "e2e mode" "$([ "$E2E_REMOTE" = "1" ] && echo "remote/public domain (no SSH)" || echo "local koopa stack")"
|
||||
info "currency" "$CUR"
|
||||
info "ATM withdraw ladder" "$WITHDRAW_LIST (credit $CREDIT_AMT)"
|
||||
info "pay ladder" "$PAY_LIST"
|
||||
info "e2e budget" "${E2E_TIMEOUT}s"
|
||||
|
||||
# Local stack: koopa secrets. Remote: only explicit env (never leak local passwords to foreign banks).
|
||||
if [ "$E2E_REMOTE" = "1" ]; then
|
||||
ADMIN_PASS="${E2E_BANK_ADMIN_PASS:-}"
|
||||
MPW="${E2E_MERCHANT_TOKEN:-${MERCHANT_TOKEN:-}}"
|
||||
else
|
||||
ADMIN_PASS="${E2E_BANK_ADMIN_PASS:-$(read_secret "taler-bank/bank-admin-password.txt" || true)}"
|
||||
MPW="${E2E_MERCHANT_TOKEN:-${MERCHANT_TOKEN:-$(read_secret "taler-merchant/merchant-goa-demo-cp4zqk-password.txt" || true)}}"
|
||||
fi
|
||||
if [ -n "$ADMIN_PASS" ]; then
|
||||
ok "bank admin secret"
|
||||
elif [ "$E2E_REMOTE" = "1" ]; then
|
||||
warn "bank admin secret" "missing — will try public registration only"
|
||||
else
|
||||
blocker "prereq" "bank admin password missing (SECRETS_ROOT or koopa /root)"
|
||||
exit 1
|
||||
fi
|
||||
if [ -n "$MPW" ]; then
|
||||
ok "merchant secret (instance ${MERCHANT_INSTANCE})"
|
||||
elif [ "$E2E_REMOTE" = "1" ]; then
|
||||
warn "merchant secret" "missing — order create may fail (set E2E_MERCHANT_TOKEN)"
|
||||
else
|
||||
blocker "prereq" "merchant instance password missing"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Public reachability gates (remote: soft-skip if bank/merchant absent)
|
||||
for pair in \
|
||||
"bank|$BANK_PUBLIC/config" \
|
||||
"exchange|$EXCHANGE_PUBLIC/config" \
|
||||
"exchange-keys|$EXCHANGE_PUBLIC/keys" \
|
||||
"bank-integration|$BANK_PUBLIC/taler-integration/config" \
|
||||
"merchant|$MERCHANT_PUBLIC/config"
|
||||
do
|
||||
IFS='|' read -r name url <<<"$pair"
|
||||
code=$(http_code "$url")
|
||||
if [ "$code" = "200" ]; then
|
||||
ok "reachable $name"
|
||||
else
|
||||
if [ "$E2E_REMOTE" = "1" ] && [[ "$name" == bank* || "$name" == merchant ]]; then
|
||||
warn "reachable $name" "HTTP $code — e2e may abort"
|
||||
else
|
||||
blocker "prereq" "$name HTTP $code ($url) — cannot start payment cycle"
|
||||
fi
|
||||
fi
|
||||
done
|
||||
# Remote: need at least bank+exchange+merchant for a full cycle
|
||||
if [ "$E2E_REMOTE" = "1" ]; then
|
||||
bc=$(http_code "$BANK_PUBLIC/config")
|
||||
ec=$(http_code "$EXCHANGE_PUBLIC/config")
|
||||
mc=$(http_code "$MERCHANT_PUBLIC/config")
|
||||
if [ "$bc" != "200" ] || [ "$ec" != "200" ]; then
|
||||
e2e_abort_auth "prereq" "bank/exchange not publicly usable (bank=$bc exchange=$ec) — skip e2e"
|
||||
fi
|
||||
if [ "$mc" != "200" ]; then
|
||||
warn "prereq" "merchant HTTP $mc — will try withdraw only if possible"
|
||||
fi
|
||||
fi
|
||||
if [ "${#BLOCKERS[@]}" -gt 0 ]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
USER="mon$(date +%s | tail -c 6)"
|
||||
USER_PW=$(python3 -c 'import secrets;print(secrets.token_urlsafe(10))')
|
||||
info "e2e user" "$USER withdraw=$WITHDRAW_AMT pay=$PAY_AMT"
|
||||
BANK="$BANK_PUBLIC"
|
||||
INST="$MERCHANT_INSTANCE"
|
||||
|
||||
# $1 = max seconds for this call (optional); rest = wallet-cli args
|
||||
wcli() {
|
||||
local maxc=12
|
||||
if [[ "${1:-}" =~ ^[0-9]+$ ]]; then
|
||||
maxc=$1
|
||||
shift
|
||||
fi
|
||||
e2e_over && return 124
|
||||
local cap
|
||||
cap=$(e2e_left)
|
||||
[ "$cap" -gt "$maxc" ] && cap=$maxc
|
||||
[ "$cap" -lt 3 ] && return 124
|
||||
if command -v perl >/dev/null 2>&1; then
|
||||
perl -e 'alarm shift; exec @ARGV' "$cap" \
|
||||
node "$WALLET_CLI" --wallet-db="$WDB" --no-throttle --skip-defaults "$@"
|
||||
else
|
||||
node "$WALLET_CLI" --wallet-db="$WDB" --no-throttle --skip-defaults "$@"
|
||||
fi
|
||||
}
|
||||
|
||||
# Pay must not be starved: use reserved seconds even if global budget is tight
|
||||
wcli_pay() {
|
||||
local cap=$E2E_PAY_SECS
|
||||
local left
|
||||
left=$(e2e_left)
|
||||
if [ "$left" -gt "$cap" ]; then
|
||||
cap=$E2E_PAY_SECS
|
||||
elif [ "$left" -ge 8 ]; then
|
||||
cap=$left
|
||||
else
|
||||
# still try once with floor 12s so Alarm clock is not the blocker
|
||||
cap=12
|
||||
fi
|
||||
info "pay wallet cap" "${cap}s"
|
||||
if command -v perl >/dev/null 2>&1; then
|
||||
perl -e 'alarm shift; exec @ARGV' "$cap" \
|
||||
node "$WALLET_CLI" --wallet-db="$WDB" --no-throttle --skip-defaults "$@"
|
||||
else
|
||||
node "$WALLET_CLI" --wallet-db="$WDB" --no-throttle --skip-defaults "$@"
|
||||
fi
|
||||
}
|
||||
|
||||
# When coins never become available: run inside + targeted bank/exchange dig
|
||||
dig_when_no_coins() {
|
||||
section "e2e · coins missing — inside dig"
|
||||
info "reason" "wallet empty after withdraw path (often wirewatch timing) — probing bank + exchange"
|
||||
if [ -n "${WID:-}" ]; then
|
||||
curl -sS -m 6 -o "$SCRATCH/wd-dig.json" \
|
||||
"$BANK_PUBLIC/taler-integration/withdrawal-operation/${WID}" 2>/dev/null || true
|
||||
info "backend withdrawal-operation" "$(python3 -c '
|
||||
import json
|
||||
try:
|
||||
d=json.load(open("'"$SCRATCH"'/wd-dig.json"))
|
||||
print("status=%s transfer_done=%s selection_done=%s amount=%s reserve=%s" % (
|
||||
d.get("status"), d.get("transfer_done"), d.get("selection_done"),
|
||||
d.get("amount"), (d.get("selected_reserve_pub") or "-")[:24]))
|
||||
except Exception as e:
|
||||
print("unreadable", e)
|
||||
' 2>/dev/null)"
|
||||
fi
|
||||
# Full component inside report (capped SSH; ignore its exit)
|
||||
if [ "${SKIP_SSH}" != "1" ] && [ -x "$ROOT/check_inside.sh" ]; then
|
||||
info "inside" "running check_inside.sh …"
|
||||
# subshell so its PASS_N/FAIL_N/summary don't poison ours; still print to stdout
|
||||
(
|
||||
# shellcheck disable=SC2030
|
||||
"$ROOT/check_inside.sh" || true
|
||||
) || true
|
||||
else
|
||||
warn "inside" "skipped (SKIP_SSH=1 or check_inside.sh missing)"
|
||||
fi
|
||||
# Extra targeted: wirewatch + recent journal + bank→exchange path
|
||||
if koopa_ssh_ok; then
|
||||
section "e2e · coins missing — exchange/bank focus"
|
||||
DIG=$(koopa_ssh_bash 15 <<'REMOTE' || true
|
||||
set +e
|
||||
emit() { printf 'D|%s|%s\n' "$1" "$(printf '%s' "$2" | tr '\n' ' ' | head -c 220)"; }
|
||||
EX=$(podman ps --format '{{.Names}}' | grep -i exchange | head -1)
|
||||
BANK=$(podman ps --format '{{.Names}}' | grep -iE 'hacktivism-bank|taler-bank' | head -1)
|
||||
[ -z "$BANK" ] && BANK=$(podman ps --format '{{.Names}}' | grep -i bank | head -1)
|
||||
if [ -n "$EX" ]; then
|
||||
podman exec "$EX" pgrep -af taler-exchange-wirewatch 2>/dev/null | head -1 | \
|
||||
{ read -r l; [ -n "$l" ] && emit OK "wirewatch: $l" || emit ERROR "wirewatch not running"; }
|
||||
podman exec "$EX" pgrep -af taler-exchange-transfer 2>/dev/null | head -1 | \
|
||||
{ read -r l; [ -n "$l" ] && emit OK "transfer: $l" || emit WARN "transfer not running"; }
|
||||
podman exec "$EX" pgrep -af taler-exchange-aggregator 2>/dev/null | head -1 | \
|
||||
{ read -r l; [ -n "$l" ] && emit OK "aggregator: $l" || emit WARN "aggregator not running"; }
|
||||
# last wirewatch log lines if journal available
|
||||
j=$(podman exec "$EX" bash -c 'journalctl -u "taler-exchange-wirewatch*" -n 8 --no-pager 2>/dev/null | tail -5' 2>/dev/null | tr '\n' ';' | head -c 280)
|
||||
[ -n "$j" ] && emit INFO "wirewatch journal: $j"
|
||||
# reserve lookup via exchange if tools exist (best-effort)
|
||||
c=$(curl -sS -m 3 -o /dev/null -w '%{http_code}' http://127.0.0.1:9011/keys 2>/dev/null || echo 000)
|
||||
emit INFO "exchange /keys HTTP $c"
|
||||
else
|
||||
emit ERROR "no exchange container"
|
||||
fi
|
||||
if [ -n "$BANK" ]; then
|
||||
podman exec "$BANK" pgrep -af 'MainKt serve|libeufin-bank serve' 2>/dev/null | head -1 | \
|
||||
{ read -r l; [ -n "$l" ] && emit OK "libeufin: $l" || emit ERROR "libeufin not running"; }
|
||||
c=$(curl -sS -m 3 -o /dev/null -w '%{http_code}' http://127.0.0.1:9012/taler-integration/config 2>/dev/null || echo 000)
|
||||
emit INFO "bank integration /config HTTP $c"
|
||||
else
|
||||
emit ERROR "no bank container"
|
||||
fi
|
||||
REMOTE
|
||||
)
|
||||
while IFS= read -r line; do
|
||||
case "$line" in
|
||||
D\|*)
|
||||
IFS="|" read -r _ lvl msg <<<"$line"
|
||||
case "$lvl" in
|
||||
OK) ok "dig $msg" ;;
|
||||
ERROR) err "dig" "$msg" ;;
|
||||
WARN) warn "dig" "$msg" ;;
|
||||
INFO) info "dig" "$msg" ;;
|
||||
esac
|
||||
;;
|
||||
esac
|
||||
done <<<"$DIG"
|
||||
fi
|
||||
}
|
||||
|
||||
# Baseline wallet balance (empty DB) — always shown even if we abort later
|
||||
wcli_bal_snap "$SCRATCH/bal-before.out" || true
|
||||
BAL_BEFORE=$(fmt_bal "$SCRATCH/bal-before.out")
|
||||
info "BALANCE before" "$BAL_BEFORE"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
section "e2e · bank (account · credit · withdraw)"
|
||||
# ---------------------------------------------------------------------------
|
||||
AT=""
|
||||
if [ -n "${ADMIN_PASS:-}" ]; then
|
||||
AT=$(curl -sS -m 15 -u "admin:${ADMIN_PASS}" -H 'Content-Type: application/json' \
|
||||
-d '{"scope":"readwrite"}' \
|
||||
"$BANK/accounts/admin/token" 2>"$SCRATCH/at.err" | python3 -c 'import sys,json;print(json.load(sys.stdin).get("access_token",""))' 2>/dev/null || true)
|
||||
fi
|
||||
if [ -n "$AT" ]; then
|
||||
ok "admin token"
|
||||
elif [ "$E2E_REMOTE" = "1" ]; then
|
||||
warn "admin token" "unavailable — trying open registration"
|
||||
else
|
||||
if is_auth_kyc_body "$SCRATCH/at.err"; then
|
||||
e2e_abort_auth "bank-auth" "admin login blocked (KYC/auth) — $(head -c 80 "$SCRATCH/at.err" 2>/dev/null)"
|
||||
fi
|
||||
blocker "bank-auth" "admin token failed — check admin password / bank up ($(head -c 80 "$SCRATCH/at.err" 2>/dev/null))"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Create account: with admin bearer if we have it, else unauthenticated registration (demo banks)
|
||||
if [ -n "$AT" ]; then
|
||||
code=$(curl -sS -m 15 -o "$SCRATCH/acc.json" -w '%{http_code}' -X POST \
|
||||
-H "Authorization: Bearer $AT" -H 'Content-Type: application/json' \
|
||||
-d "{\"username\":\"${USER}\",\"password\":\"${USER_PW}\",\"name\":\"Monitoring\",\"is_public\":false,\"is_taler_exchange\":false,\"debit_threshold\":\"${CUR}:1000\"}" \
|
||||
"$BANK/accounts")
|
||||
else
|
||||
code=$(curl -sS -m 15 -o "$SCRATCH/acc.json" -w '%{http_code}' -X POST \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "{\"username\":\"${USER}\",\"password\":\"${USER_PW}\",\"name\":\"Monitoring\",\"is_public\":false,\"is_taler_exchange\":false,\"debit_threshold\":\"${CUR}:1000\"}" \
|
||||
"$BANK/accounts")
|
||||
fi
|
||||
if [ "$code" = "200" ] || [ "$code" = "201" ]; then
|
||||
ok "create account $USER"
|
||||
elif [ "$code" = "401" ] || [ "$code" = "403" ] || is_auth_kyc_body "$SCRATCH/acc.json"; then
|
||||
e2e_abort_auth "bank-account" "registration/login blocked HTTP $code — $(head -c 100 "$SCRATCH/acc.json" | tr '\n' ' ')"
|
||||
else
|
||||
if [ "$E2E_REMOTE" = "1" ]; then
|
||||
e2e_abort_auth "bank-account" "POST /accounts HTTP $code — cannot open account on this domain"
|
||||
fi
|
||||
blocker "bank-account" "POST /accounts HTTP $code — $(head -c 100 "$SCRATCH/acc.json" | tr '\n' ' ')"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ -n "$AT" ]; then
|
||||
export USER_NAME="$USER" CREDIT_AMT="$CREDIT_AMT" OUT="$SCRATCH/credit.json" BANK_HOST="$BANK_HOST"
|
||||
python3 - <<'PY'
|
||||
import json, os
|
||||
ALPH = "0123456789ABCDEFGHJKMNPQRSTVWXYZ"
|
||||
def crock32(b):
|
||||
bits = val = 0; o = []
|
||||
for byte in b:
|
||||
val = (val << 8) | byte; bits += 8
|
||||
while bits >= 5:
|
||||
bits -= 5; o.append(ALPH[(val >> bits) & 31])
|
||||
if bits: o.append(ALPH[(val << (5 - bits)) & 31])
|
||||
return "".join(o)
|
||||
uid = crock32(os.urandom(32))
|
||||
user = os.environ["USER_NAME"]
|
||||
host = os.environ.get("BANK_HOST") or "bank"
|
||||
json.dump({
|
||||
"payto_uri": f"payto://x-taler-bank/{host}/{user}?receiver-name={user}&message=mon",
|
||||
"amount": os.environ["CREDIT_AMT"],
|
||||
"request_uid": uid,
|
||||
}, open(os.environ["OUT"], "w"))
|
||||
PY
|
||||
curl -sS -m 15 -o "$SCRATCH/credit.out" -H "Authorization: Bearer $AT" \
|
||||
-H 'Content-Type: application/json' -d @"$SCRATCH/credit.json" \
|
||||
"$BANK/accounts/admin/transactions" >/dev/null || true
|
||||
if grep -q row_id "$SCRATCH/credit.out" 2>/dev/null; then
|
||||
ok "credit $CREDIT_AMT → $USER"
|
||||
else
|
||||
if is_auth_kyc_body "$SCRATCH/credit.out"; then
|
||||
e2e_abort_auth "bank-credit" "admin credit blocked (KYC/auth)"
|
||||
fi
|
||||
if [ "$E2E_REMOTE" = "1" ]; then
|
||||
e2e_abort_auth "bank-credit" "admin transfer failed — no credit path on remote ($(head -c 80 "$SCRATCH/credit.out" | tr '\n' ' '))"
|
||||
fi
|
||||
blocker "bank-credit" "admin transfer failed — $(head -c 100 "$SCRATCH/credit.out" | tr '\n' ' ')"
|
||||
exit 1
|
||||
fi
|
||||
else
|
||||
# No admin: remote demo may start funded or require cash-in — try withdraw later; warn
|
||||
warn "bank-credit" "skipped (no admin) — withdraw may fail without balance"
|
||||
fi
|
||||
|
||||
UT=$(curl -sS -m 15 -u "${USER}:${USER_PW}" -H 'Content-Type: application/json' \
|
||||
-d '{"scope":"readwrite"}' \
|
||||
"$BANK/accounts/${USER}/token" 2>"$SCRATCH/ut.err" | python3 -c 'import sys,json;print(json.load(sys.stdin).get("access_token",""))' 2>/dev/null || true)
|
||||
if [ -n "$UT" ]; then
|
||||
ok "user token"
|
||||
elif is_auth_kyc_body "$SCRATCH/ut.err" || is_auth_kyc_body /dev/null; then
|
||||
e2e_abort_auth "bank-auth" "user token failed (login/KYC) for $USER"
|
||||
else
|
||||
e2e_abort_auth "bank-auth" "user token failed for $USER — $(head -c 80 "$SCRATCH/ut.err" 2>/dev/null)"
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
section "e2e · wallet setup (exchange + ToS once)"
|
||||
# ---------------------------------------------------------------------------
|
||||
if ! wcli exchanges add "${EXCHANGE_PUBLIC}/" >"$SCRATCH/ex-add.out" 2>&1; then
|
||||
if ! grep -qiE 'already|ok' "$SCRATCH/ex-add.out"; then
|
||||
blocker "wallet-exchange" "exchanges add failed — $(tail -c 160 "$SCRATCH/ex-add.out" | tr '\n' ' ')"
|
||||
else
|
||||
ok "wallet exchange known"
|
||||
fi
|
||||
else
|
||||
ok "wallet add exchange"
|
||||
fi
|
||||
wcli exchanges update "${EXCHANGE_PUBLIC}/" >"$SCRATCH/ex-up.out" 2>&1 || true
|
||||
if wcli exchanges accept-tos "${EXCHANGE_PUBLIC}/" >"$SCRATCH/ex-tos.out" 2>&1; then
|
||||
ok "wallet accept ToS"
|
||||
else
|
||||
if grep -qiE 'pending|tos|terms' "$SCRATCH/ex-tos.out"; then
|
||||
blocker "wallet-tos" "accept-tos failed — wallet may refuse withdraw ($(tail -c 120 "$SCRATCH/ex-tos.out" | tr '\n' ' '))"
|
||||
else
|
||||
warn "wallet accept ToS" "$(tail -c 80 "$SCRATCH/ex-tos.out" | tr '\n' ' ')"
|
||||
fi
|
||||
fi
|
||||
|
||||
wd_status() {
|
||||
curl -sS -m 8 -o "$SCRATCH/wd-st.json" -w '' \
|
||||
"$BANK/taler-integration/withdrawal-operation/${WID}" 2>/dev/null || true
|
||||
python3 -c 'import json;d=json.load(open("'"$SCRATCH"'/wd-st.json"));print(d.get("status",""))' 2>/dev/null || true
|
||||
}
|
||||
wd_status_detail() {
|
||||
python3 -c 'import json;d=json.load(open("'"$SCRATCH"'/wd-st.json"))
|
||||
print("status=%s reserve=%s exchange=%s" % (
|
||||
d.get("status"),
|
||||
(d.get("selected_reserve_pub") or "-")[:20],
|
||||
(d.get("selected_exchange") or d.get("selected_exchange_account") or "-")[:60],
|
||||
))' 2>/dev/null || echo "(no status body)"
|
||||
}
|
||||
|
||||
fake_incoming_speedup() {
|
||||
[ "${E2E_FAKE_INCOMING}" = "1" ] || return 0
|
||||
st=$(wd_status)
|
||||
RPUB=$(python3 -c 'import json;d=json.load(open("'"$SCRATCH"'/wd-st.json"));print(d.get("selected_reserve_pub") or "")' 2>/dev/null || true)
|
||||
AMT=$(python3 -c 'import json;d=json.load(open("'"$SCRATCH"'/wd-st.json"));print(d.get("amount") or "'"$WITHDRAW_AMT"'")' 2>/dev/null || echo "$WITHDRAW_AMT")
|
||||
DEBIT=$(python3 -c 'import json;d=json.load(open("'"$SCRATCH"'/wd-st.json"));print(d.get("sender_wire") or "")' 2>/dev/null || true)
|
||||
[ -z "$DEBIT" ] && DEBIT="payto://x-taler-bank/${BANK_HOST}/${USER}?receiver-name=${USER}"
|
||||
[ -z "$RPUB" ] && return 0
|
||||
[ -z "${AT:-}" ] && return 0
|
||||
WG="${BANK}/accounts/exchange/taler-wire-gateway/admin/add-incoming"
|
||||
curl -sS -m 10 -o "$SCRATCH/fake-in.json" -w '%{http_code}' -X POST \
|
||||
-H "Authorization: Bearer $AT" -H 'Content-Type: application/json' \
|
||||
-d "{\"amount\":\"${AMT}\",\"reserve_pub\":\"${RPUB}\",\"debit_account\":\"${DEBIT}\"}" \
|
||||
"$WG" >/dev/null || true
|
||||
if [ "$E2E_REMOTE" != "1" ] && koopa_ssh_ok; then
|
||||
koopa_ssh_run 22 \
|
||||
'podman exec taler-hacktivism-exchange-ansible bash -c "runuser -u taler-exchange-wire -- timeout 15 taler-exchange-wirewatch -c /etc/taler-exchange/taler-exchange.conf -t -L INFO 2>&1 | tail -5"' \
|
||||
>/dev/null 2>&1 || true
|
||||
fi
|
||||
}
|
||||
|
||||
# One ATM-style withdraw: create → accept → select → confirm → settle
|
||||
# returns 0 on wallet balance increase, 1 on soft fail (ladder continues)
|
||||
e2e_one_withdraw() {
|
||||
WITHDRAW_AMT="$1"
|
||||
local tag
|
||||
tag=$(printf '%s' "$WITHDRAW_AMT" | tr '.:' '__')
|
||||
section "e2e · ATM withdraw $WITHDRAW_AMT"
|
||||
e2e_over && { warn "ATM withdraw" "budget exhausted — skip $WITHDRAW_AMT"; return 1; }
|
||||
|
||||
curl -sS -m 15 -o "$SCRATCH/wd-$tag.json" -H "Authorization: Bearer $UT" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "{\"amount\":\"${WITHDRAW_AMT}\",\"exchange_url\":\"${EXCHANGE_PUBLIC}/\"}" \
|
||||
"$BANK/accounts/${USER}/withdrawals"
|
||||
WID=$(python3 -c 'import json;d=json.load(open("'"$SCRATCH"'/wd-'"$tag"'.json"));print(d.get("withdrawal_id") or d.get("id") or "")' 2>/dev/null || true)
|
||||
URI=$(python3 -c 'import json;d=json.load(open("'"$SCRATCH"'/wd-'"$tag"'.json"));print(d.get("taler_withdraw_uri") or "")' 2>/dev/null || true)
|
||||
URI_CLEAN=$(python3 -c 'import json;u=json.load(open("'"$SCRATCH"'/wd-'"$tag"'.json")).get("taler_withdraw_uri") or "";print(u.replace(":443/","/"))' 2>/dev/null || true)
|
||||
if [ -z "$WID" ] || [ -z "$URI_CLEAN" ]; then
|
||||
warn "ATM withdraw $WITHDRAW_AMT" "create failed — $(head -c 80 "$SCRATCH/wd-$tag.json" | tr '\n' ' ')"
|
||||
return 1
|
||||
fi
|
||||
ok "ATM withdrawal created $WITHDRAW_AMT ($WID)"
|
||||
cp "$SCRATCH/wd-$tag.json" "$SCRATCH/wd.json"
|
||||
|
||||
USE_URI="$URI"
|
||||
[ -z "$USE_URI" ] && USE_URI="$URI_CLEAN"
|
||||
if wcli withdraw accept-uri --exchange "${EXCHANGE_PUBLIC}/" "$USE_URI" >"$SCRATCH/accept-$tag.out" 2>&1; then
|
||||
ok "wallet accept $WITHDRAW_AMT"
|
||||
elif [ "$USE_URI" != "$URI_CLEAN" ] && [ -n "$URI_CLEAN" ] \
|
||||
&& wcli withdraw accept-uri --exchange "${EXCHANGE_PUBLIC}/" "$URI_CLEAN" >"$SCRATCH/accept-$tag.out" 2>&1; then
|
||||
ok "wallet accept $WITHDRAW_AMT (no :443)"
|
||||
else
|
||||
warn "ATM withdraw $WITHDRAW_AMT" "accept-uri failed — $(tail -c 100 "$SCRATCH/accept-$tag.out" | tr '\n' ' ')"
|
||||
return 1
|
||||
fi
|
||||
cp "$SCRATCH/accept-$tag.out" "$SCRATCH/accept.out"
|
||||
|
||||
st=""
|
||||
for i in 1 2 3 4; do
|
||||
e2e_over && break
|
||||
wcli run-until-done >"$SCRATCH/sel-$tag-$i.out" 2>&1 || true
|
||||
st=$(wd_status)
|
||||
case "$st" in selected|confirmed|aborted) break ;; esac
|
||||
done
|
||||
if [ "$st" != "selected" ] && [ "$st" != "confirmed" ]; then
|
||||
wcli transactions >"$SCRATCH/tx-pre.json" 2>&1 || true
|
||||
RPUB=$(python3 -c '
|
||||
import re
|
||||
t=open("'"$SCRATCH"'/accept.out").read()+open("'"$SCRATCH"'/tx-pre.json").read()
|
||||
m=re.search(r"reserve[_ ]?pub[\"=: ]+([A-Z0-9]{40,})", t, re.I)
|
||||
if not m: m=re.search(r"\"reservePub\"\s*:\s*\"([^\"]+)\"", t)
|
||||
print(m.group(1) if m else "")
|
||||
' 2>/dev/null || true)
|
||||
EPAYTO=$(curl -sS -m 12 "$EXCHANGE_PUBLIC/keys" 2>/dev/null | python3 -c '
|
||||
import json,sys
|
||||
d=json.load(sys.stdin)
|
||||
acc=d.get("accounts") or []
|
||||
for a in acc:
|
||||
p=a.get("payto_uri") or a.get("payto_address") or ""
|
||||
if "x-taler-bank" in p or "exchange" in p:
|
||||
print(p); break
|
||||
else:
|
||||
if acc: print(acc[0].get("payto_uri") or "")
|
||||
' 2>/dev/null || true)
|
||||
if [ -n "$RPUB" ] && [ -n "$EPAYTO" ]; then
|
||||
curl -sS -m 12 -o "$SCRATCH/sel.json" -X POST \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "{\"reserve_pub\":\"$RPUB\",\"selected_exchange\":\"$EPAYTO\"}" \
|
||||
"$BANK/taler-integration/withdrawal-operation/${WID}" >/dev/null || true
|
||||
st=$(wd_status)
|
||||
fi
|
||||
fi
|
||||
if [ "$st" != "selected" ] && [ "$st" != "confirmed" ]; then
|
||||
warn "ATM withdraw $WITHDRAW_AMT" "not selected (status=${st:-?})"
|
||||
return 1
|
||||
fi
|
||||
if [ "$st" != "confirmed" ]; then
|
||||
code=$(curl -sS -m 15 -o "$SCRATCH/conf-$tag.json" -w '%{http_code}' -X POST \
|
||||
-H "Authorization: Bearer $UT" -H 'Content-Type: application/json' -d '{}' \
|
||||
"$BANK/accounts/${USER}/withdrawals/${WID}/confirm")
|
||||
case "$code" in
|
||||
200|204) ok "bank confirm $WITHDRAW_AMT" ;;
|
||||
*) warn "ATM withdraw $WITHDRAW_AMT" "confirm HTTP $code"; return 1 ;;
|
||||
esac
|
||||
fi
|
||||
fake_incoming_speedup
|
||||
|
||||
# Short per-ATM poll; full settle wait happens after the ladder (avoids false FAIL)
|
||||
local ok_bal=0 r av
|
||||
for r in 1 2 3 4 5 6; do
|
||||
wcli run-until-done >"$SCRATCH/run-$tag-$r.out" 2>&1 || true
|
||||
wcli_bal_snap "$SCRATCH/bal-$tag.out" || wcli balance >"$SCRATCH/bal-$tag.out" 2>&1 || true
|
||||
av=$(wallet_avail_num "$SCRATCH/bal-$tag.out")
|
||||
if python3 -c "import sys; sys.exit(0 if float(sys.argv[1]) > 0 else 1)" "$av" 2>/dev/null; then
|
||||
ok_bal=1
|
||||
ok "wallet funded after ATM $WITHDRAW_AMT" "avail=${CUR}:${av}"
|
||||
info "balance" "$(fmt_bal "$SCRATCH/bal-$tag.out")"
|
||||
break
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
if [ "$ok_bal" = "1" ]; then
|
||||
return 0
|
||||
fi
|
||||
# Bank side often already confirmed — treat as timing lag, not hard fail
|
||||
st=$(wd_status)
|
||||
if [ "$st" = "confirmed" ]; then
|
||||
warn "ATM withdraw $WITHDRAW_AMT" "bank confirmed; wallet not funded yet (settlement timing — will recheck later)"
|
||||
return 2
|
||||
fi
|
||||
warn "ATM withdraw $WITHDRAW_AMT" "no wallet balance yet (status=${st:-?})"
|
||||
return 1
|
||||
}
|
||||
|
||||
# One variable payment amount
|
||||
e2e_one_pay() {
|
||||
PAY_AMT="$1"
|
||||
local tag
|
||||
tag=$(printf '%s' "$PAY_AMT" | tr '.:' '__')
|
||||
section "e2e · pay $PAY_AMT"
|
||||
e2e_over && { warn "pay" "budget exhausted — skip $PAY_AMT"; return 1; }
|
||||
if [ -z "${MPW:-}" ]; then
|
||||
warn "pay $PAY_AMT" "no merchant token"
|
||||
return 1
|
||||
fi
|
||||
AUTH="Authorization: Bearer secret-token:${MPW}"
|
||||
curl -skS -m 15 -o "$SCRATCH/ord-$tag.json" -X POST \
|
||||
-H "$AUTH" -H 'Content-Type: application/json' \
|
||||
-d "{\"order\":{\"summary\":\"monitoring pay ${PAY_AMT}\",\"amount\":\"${PAY_AMT}\",\"fulfillment_message\":\"ok\"},\"create_token\":true}" \
|
||||
"${MERCHANT_PUBLIC}/instances/${INST}/private/orders" 2>"$SCRATCH/ord-$tag.err" || true
|
||||
if ! grep -q order_id "$SCRATCH/ord-$tag.json" 2>/dev/null; then
|
||||
if [ "$E2E_REMOTE" != "1" ] && koopa_ssh_ok; then
|
||||
koopa_ssh_run 20 \
|
||||
"curl -skS -m 12 -X POST -H 'Authorization: Bearer secret-token:${MPW}' -H 'Content-Type: application/json' -d '{\"order\":{\"summary\":\"monitoring pay\",\"amount\":\"${PAY_AMT}\",\"fulfillment_message\":\"ok\"},\"create_token\":true}' 'https://127.0.0.1:9010/instances/${INST}/private/orders'" \
|
||||
>"$SCRATCH/ord-$tag.json" 2>"$SCRATCH/ord-$tag.err" || true
|
||||
fi
|
||||
fi
|
||||
OID=$(python3 -c 'import json,re,sys;t=open(sys.argv[1]).read()
|
||||
try: print(json.loads(t).get("order_id") or "")
|
||||
except Exception:
|
||||
m=re.search(r"\"order_id\"\s*:\s*\"([^\"]+)\"",t); print(m.group(1) if m else "")
|
||||
' "$SCRATCH/ord-$tag.json" 2>/dev/null || true)
|
||||
OTOK=$(python3 -c 'import json,re,sys;t=open(sys.argv[1]).read()
|
||||
try: print(json.loads(t).get("token") or "")
|
||||
except Exception:
|
||||
m=re.search(r"\"token\"\s*:\s*\"([^\"]+)\"",t); print(m.group(1) if m else "")
|
||||
' "$SCRATCH/ord-$tag.json" 2>/dev/null || true)
|
||||
if [ -z "$OID" ]; then
|
||||
if is_auth_kyc_body "$SCRATCH/ord-$tag.json"; then
|
||||
e2e_abort_auth "merchant-order" "merchant auth/KYC on pay $PAY_AMT"
|
||||
fi
|
||||
warn "pay $PAY_AMT" "order create failed — $(head -c 80 "$SCRATCH/ord-$tag.json" | tr '\n' ' ')"
|
||||
return 1
|
||||
fi
|
||||
ok "merchant order $OID ($PAY_AMT)"
|
||||
curl -skS -m 12 -o "$SCRATCH/ord-det-$tag.json" -H "$AUTH" \
|
||||
"${MERCHANT_PUBLIC}/instances/${INST}/private/orders/${OID}" 2>/dev/null || true
|
||||
PAYURI=$(python3 -c 'import json,sys
|
||||
try: print(json.load(open(sys.argv[1])).get("taler_pay_uri") or "")
|
||||
except Exception: print("")
|
||||
' "$SCRATCH/ord-det-$tag.json" 2>/dev/null || true)
|
||||
if [ -z "$PAYURI" ] && [ -n "$OTOK" ]; then
|
||||
MH=$(python3 -c 'from urllib.parse import urlparse; print(urlparse("'"$MERCHANT_PUBLIC"'").hostname or "taler.hacktivism.ch")' 2>/dev/null || echo "taler.hacktivism.ch")
|
||||
PAYURI="taler://pay/${MH}/instances/${INST}/${OID}/?c=${OTOK}"
|
||||
fi
|
||||
if [ -z "$PAYURI" ]; then
|
||||
warn "pay $PAY_AMT" "no pay URI for $OID"
|
||||
return 1
|
||||
fi
|
||||
if ! wcli_pay handle-uri --yes "$PAYURI" >"$SCRATCH/pay-$tag.out" 2>&1; then
|
||||
warn "pay $PAY_AMT" "handle-uri failed — $(tail -c 100 "$SCRATCH/pay-$tag.out" | tr '\n' ' ')"
|
||||
return 1
|
||||
fi
|
||||
ok "wallet handle pay $PAY_AMT"
|
||||
local r
|
||||
for r in 1 2 3; do
|
||||
wcli_pay run-until-done >"$SCRATCH/pay-run-$tag.out" 2>&1 || true
|
||||
wcli_pay transactions >"$SCRATCH/tx-$tag.out" 2>&1 || true
|
||||
curl -skS -m 8 -o "$SCRATCH/ord-paid-$tag.json" -H "$AUTH" \
|
||||
"${MERCHANT_PUBLIC}/instances/${INST}/private/orders/${OID}" 2>/dev/null || true
|
||||
if grep -qiE 'payment|paid|Payment' "$SCRATCH/tx-$tag.out" 2>/dev/null \
|
||||
|| grep -qiE 'done|paid|success|Payment' "$SCRATCH/pay-$tag.out" 2>/dev/null \
|
||||
|| python3 -c 'import json,sys
|
||||
d=json.load(open(sys.argv[1]))
|
||||
sys.exit(0 if d.get("paid") is True or str(d.get("order_status","")).lower()=="paid" else 1)
|
||||
' "$SCRATCH/ord-paid-$tag.json" 2>/dev/null; then
|
||||
ok "payment settled $PAY_AMT (order $OID)"
|
||||
return 0
|
||||
fi
|
||||
done
|
||||
warn "pay $PAY_AMT" "not settled for order $OID"
|
||||
return 1
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
section "e2e · ATM withdraw ladder"
|
||||
# ---------------------------------------------------------------------------
|
||||
WITHDRAW_OK=0
|
||||
WITHDRAW_OK_N=0
|
||||
WITHDRAW_LAG_N=0
|
||||
WITHDRAW_FAIL_N=0
|
||||
WITHDRAW_REPORT=""
|
||||
for WITHDRAW_AMT in $WITHDRAW_LIST; do
|
||||
e2e_over && { warn "ATM ladder" "time budget low — stopping more ATM withdraws (not a protocol error)"; break; }
|
||||
set +e
|
||||
e2e_one_withdraw "$WITHDRAW_AMT"
|
||||
wc=$?
|
||||
set -e
|
||||
case "$wc" in
|
||||
0)
|
||||
WITHDRAW_OK=1
|
||||
WITHDRAW_OK_N=$((WITHDRAW_OK_N + 1))
|
||||
WITHDRAW_REPORT="${WITHDRAW_REPORT}${WITHDRAW_REPORT:+ }${WITHDRAW_AMT}=OK"
|
||||
;;
|
||||
2)
|
||||
WITHDRAW_LAG_N=$((WITHDRAW_LAG_N + 1))
|
||||
WITHDRAW_REPORT="${WITHDRAW_REPORT}${WITHDRAW_REPORT:+ }${WITHDRAW_AMT}=LAG"
|
||||
;;
|
||||
*)
|
||||
WITHDRAW_FAIL_N=$((WITHDRAW_FAIL_N + 1))
|
||||
WITHDRAW_REPORT="${WITHDRAW_REPORT}${WITHDRAW_REPORT:+ }${WITHDRAW_AMT}=FAIL"
|
||||
;;
|
||||
esac
|
||||
done
|
||||
info "ATM withdraw summary" "$WITHDRAW_REPORT (ok=$WITHDRAW_OK_N lag=$WITHDRAW_LAG_N fail=$WITHDRAW_FAIL_N)"
|
||||
|
||||
# Settlement catch-up: bank may have confirmed while wallet was still empty
|
||||
section "e2e · wallet settlement (timing)"
|
||||
if ! wait_wallet_balance 0 "${E2E_SETTLE_ROUNDS}" "${E2E_SETTLE_SLEEP}"; then
|
||||
if [ "$WITHDRAW_LAG_N" -gt 0 ] || [ "$WITHDRAW_OK_N" -gt 0 ]; then
|
||||
warn "settlement timing" "ATM path reached bank confirm (lag=$WITHDRAW_LAG_N ok=$WITHDRAW_OK_N) but wallet empty after wait — TIME lag, not protocol error"
|
||||
fi
|
||||
fi
|
||||
av_now=$(wallet_avail_num)
|
||||
if python3 -c "import sys; sys.exit(0 if float(sys.argv[1]) > 0 else 1)" "$av_now" 2>/dev/null; then
|
||||
WITHDRAW_OK=1
|
||||
if [ "$WITHDRAW_OK_N" = "0" ]; then
|
||||
warn "settlement timing" "coins arrived after ATM ladder (${CUR}:${av_now}) — earlier 'FAIL' was timing lag"
|
||||
WITHDRAW_REPORT="${WITHDRAW_REPORT} → late-OK avail=${CUR}:${av_now}"
|
||||
fi
|
||||
ok "spendable balance for payments" "${CUR}:${av_now}"
|
||||
else
|
||||
if [ "$WITHDRAW_OK" != "1" ]; then
|
||||
warn "withdraw" "still no spendable ${CUR} after settle wait"
|
||||
# diagnostic dig only — do not treat as hard stop if we can still see bank state
|
||||
if [ "$E2E_REMOTE" != "1" ]; then
|
||||
dig_when_no_coins || true
|
||||
fi
|
||||
# one last balance after dig (dig takes wall time — often enough for wirewatch)
|
||||
wait_wallet_balance 0 8 3 || true
|
||||
av_now=$(wallet_avail_num)
|
||||
if python3 -c "import sys; sys.exit(0 if float(sys.argv[1]) > 0 else 1)" "$av_now" 2>/dev/null; then
|
||||
WITHDRAW_OK=1
|
||||
warn "settlement timing" "coins present after dig wait (${CUR}:${av_now}) — timing, not blocker"
|
||||
else
|
||||
blocker "withdraw-settle" "no spendable ${CUR} after ATM ladder + settle wait ($WITHDRAW_LIST)"
|
||||
section "e2e · report"
|
||||
info "WITHDRAW" "$WITHDRAW_REPORT — no coins after extended wait"
|
||||
info "PAY" "SKIPPED (no spendable balance after settle wait)"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
section "e2e · variable payments (if balance allows)"
|
||||
# ---------------------------------------------------------------------------
|
||||
PAY_OK=0
|
||||
PAY_OK_N=0
|
||||
PAY_FAIL_N=0
|
||||
PAY_SKIP_N=0
|
||||
PAY_REPORT=""
|
||||
if [ -z "${MPW:-}" ]; then
|
||||
if [ "$E2E_REMOTE" = "1" ]; then
|
||||
e2e_abort_auth "merchant-order" "no merchant token — set E2E_MERCHANT_TOKEN"
|
||||
fi
|
||||
blocker "merchant-order" "no merchant token"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
for PAY_AMT in $PAY_LIST; do
|
||||
e2e_over && { warn "pay ladder" "time budget low — stopping more payments"; break; }
|
||||
av_now=$(wallet_avail_num)
|
||||
pay_n=$(python3 -c 'import sys; print(float(sys.argv[1].split(":",1)[-1]))' "$PAY_AMT" 2>/dev/null || echo 0)
|
||||
if ! python3 -c "import sys; sys.exit(0 if float(sys.argv[1]) + 1e-12 >= float(sys.argv[2]) else 1)" "$av_now" "$pay_n" 2>/dev/null; then
|
||||
warn "pay $PAY_AMT" "skip — insufficient avail ${CUR}:${av_now} (need ${pay_n})"
|
||||
PAY_SKIP_N=$((PAY_SKIP_N + 1))
|
||||
PAY_REPORT="${PAY_REPORT}${PAY_REPORT:+ }${PAY_AMT}=SKIP(bal)"
|
||||
continue
|
||||
fi
|
||||
set +e
|
||||
e2e_one_pay "$PAY_AMT"
|
||||
pc=$?
|
||||
set -e
|
||||
if [ "$pc" = "0" ]; then
|
||||
PAY_OK=1
|
||||
PAY_OK_N=$((PAY_OK_N + 1))
|
||||
PAY_REPORT="${PAY_REPORT}${PAY_REPORT:+ }${PAY_AMT}=OK"
|
||||
else
|
||||
PAY_FAIL_N=$((PAY_FAIL_N + 1))
|
||||
PAY_REPORT="${PAY_REPORT}${PAY_REPORT:+ }${PAY_AMT}=FAIL"
|
||||
fi
|
||||
done
|
||||
info "pay summary" "$PAY_REPORT (ok=$PAY_OK_N fail=$PAY_FAIL_N skip=$PAY_SKIP_N)"
|
||||
if [ "$PAY_OK" != "1" ]; then
|
||||
av_now=$(wallet_avail_num)
|
||||
if python3 -c "import sys; sys.exit(0 if float(sys.argv[1]) > 0 else 1)" "$av_now" 2>/dev/null; then
|
||||
warn "pay" "balance ${CUR}:${av_now} but no pay amount completed — check merchant/timing (not necessarily empty wallet)"
|
||||
else
|
||||
blocker "pay-settle" "no payment succeeded and no spendable balance ($PAY_LIST)"
|
||||
fi
|
||||
fi
|
||||
|
||||
section "e2e · report"
|
||||
info "user" "$USER"
|
||||
info "ATM withdraws" "$WITHDRAW_REPORT"
|
||||
info "payments" "$PAY_REPORT"
|
||||
info "final balance" "$(fmt_bal "$SCRATCH/bal-live.out" 2>/dev/null || echo "(n/a)")"
|
||||
info "scratch" "$SCRATCH"
|
||||
# Success if we had coins and at least one pay, OR coins + only pay skips (nothing affordable)
|
||||
if [ "${#BLOCKERS[@]}" -eq 0 ]; then
|
||||
if [ "$WITHDRAW_OK" = "1" ] && [ "$PAY_OK" = "1" ]; then
|
||||
exit 0
|
||||
fi
|
||||
if [ "$WITHDRAW_OK" = "1" ] && [ "$PAY_OK_N" = "0" ] && [ "$PAY_FAIL_N" = "0" ]; then
|
||||
warn "pay" "no payment tried/completed — withdraw OK"
|
||||
exit 0
|
||||
fi
|
||||
if [ "$WITHDRAW_OK" = "1" ] && [ "$PAY_OK" != "1" ]; then
|
||||
# money there, pays failed → soft exit 1 without inventing blockers if already warned
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
exit 1
|
||||
141
scripts/taler-monitoring/check_inside.sh
Executable file
141
scripts/taler-monitoring/check_inside.sh
Executable file
|
|
@ -0,0 +1,141 @@
|
|||
#!/usr/bin/env bash
|
||||
# Inside status for bank / exchange / merchant. Hard-capped SSH — never hang forever.
|
||||
set -euo pipefail
|
||||
ROOT=$(cd "$(dirname "$0")" && pwd)
|
||||
# shellcheck source=lib.sh
|
||||
source "$ROOT/lib.sh"
|
||||
|
||||
# Area inside-### — container / process state on koopa (SSH)
|
||||
set_area inside
|
||||
section "inside · collect from koopa"
|
||||
|
||||
if [ "${SKIP_SSH}" = "1" ]; then
|
||||
warn "ssh" "SKIP_SSH=1 — skipped"
|
||||
summary
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if ! koopa_ssh_ok; then
|
||||
err "ssh" "cannot reach ${KOOPA_SSH} in ${SSH_CONNECT_TIMEOUT}s — set SKIP_SSH=1 to skip inside"
|
||||
summary
|
||||
exit 1
|
||||
fi
|
||||
ok "ssh ${KOOPA_SSH}"
|
||||
|
||||
# One short remote script (≤ SSH_CMD_TIMEOUT). Every slow step is local curl -m 3 or quick pgrep.
|
||||
RAW=$(
|
||||
koopa_ssh_bash "${SSH_CMD_TIMEOUT}" <<'REMOTE' || true
|
||||
set +e
|
||||
emit() { printf 'E|%s|%s|%s|%s\n' "$1" "$2" "$3" "$(printf '%s' "${4:-}" | tr '\n\r' ' ' | head -c 200)"; }
|
||||
# quick curl
|
||||
hc() { curl -skS -m 3 -o /tmp/mb -w '%{http_code}' "$1" 2>/dev/null || echo 000; }
|
||||
# quick process check inside container (pgrep only)
|
||||
hasp() { podman exec "$1" pgrep -f "$2" >/dev/null 2>&1; }
|
||||
|
||||
BANK=$(podman ps --format '{{.Names}}' 2>/dev/null | grep -iE 'hacktivism-bank|taler-bank' | head -1)
|
||||
[ -z "$BANK" ] && BANK=$(podman ps --format '{{.Names}}' 2>/dev/null | grep -i bank | head -1)
|
||||
EX=$(podman ps --format '{{.Names}}' 2>/dev/null | grep -i exchange | head -1)
|
||||
MER=$(podman ps --format '{{.Names}}' 2>/dev/null | grep -E '^taler-hacktivism$' | head -1)
|
||||
[ -z "$MER" ] && MER=$(podman ps --format '{{.Names}}' 2>/dev/null | grep -iE 'merchant|hacktivism' | grep -viE 'bank|exchange' | head -1)
|
||||
|
||||
# Domain resolve inside container (wirewatch needs bank.hacktivism.ch → real IP)
|
||||
# emit: comp LEVEL dns "host → ip" or fail
|
||||
check_dns() {
|
||||
local comp="$1" ctr="$2" host="$3"
|
||||
local line ip code
|
||||
# Prefer IPv4 (wirewatch/libcurl often happier; avoid dead AAAA)
|
||||
line=$(podman exec "$ctr" getent ahostsv4 "$host" 2>/dev/null | head -1)
|
||||
[ -z "$line" ] && line=$(podman exec "$ctr" getent hosts "$host" 2>/dev/null | head -1)
|
||||
ip=$(echo "$line" | awk '{print $1}')
|
||||
if [ -z "$ip" ]; then
|
||||
emit "$comp" ERROR "dns $host" "no resolve — run pin-container-hosts.sh"
|
||||
return 1
|
||||
fi
|
||||
# 127.0.0.1 is almost always wrong for public bank/exchange from inside pasta
|
||||
if [ "$ip" = "127.0.0.1" ] || [ "$ip" = "::1" ]; then
|
||||
emit "$comp" ERROR "dns $host" "$ip (loopback — wirewatch will fail)"
|
||||
return 1
|
||||
fi
|
||||
code=$(podman exec "$ctr" curl -skS -m 3 -o /dev/null -w '%{http_code}' "https://${host}/config" 2>/dev/null || echo 000)
|
||||
if [ "$code" = "200" ]; then
|
||||
emit "$comp" OK "dns $host" "→ $ip /config=$code"
|
||||
else
|
||||
emit "$comp" WARN "dns $host" "→ $ip /config=$code"
|
||||
fi
|
||||
}
|
||||
|
||||
if [ -z "$BANK" ]; then emit bank ERROR container "not running"
|
||||
else
|
||||
emit bank INFO container "$(podman ps --filter name=$BANK --format '{{.Names}} {{.Status}}' | head -1)"
|
||||
emit bank INFO ports "$(podman ps --filter name=$BANK --format '{{.Ports}}' | head -1)"
|
||||
hasp "$BANK" 'MainKt serve|libeufin-bank serve' && emit bank OK libeufin "running" || emit bank ERROR libeufin "not running — API/withdraw dead"
|
||||
podman exec "$BANK" pg_isready -q 2>/dev/null && emit bank OK postgres "ready" || emit bank ERROR postgres "not ready"
|
||||
c=$(hc http://127.0.0.1:9012/config)
|
||||
[ "$c" = "200" ] && emit bank OK "local /config" "HTTP $c" || emit bank ERROR "local /config" "HTTP $c"
|
||||
c=$(hc http://127.0.0.1:9012/taler-integration/config)
|
||||
[ "$c" = "200" ] && emit bank OK "local integration" "HTTP $c" || emit bank ERROR "local integration" "HTTP $c"
|
||||
hasp "$BANK" 'nginx' && emit bank OK nginx ":9013" || emit bank WARN nginx "not running"
|
||||
check_dns bank "$BANK" bank.hacktivism.ch || true
|
||||
check_dns bank "$BANK" exchange.hacktivism.ch || true
|
||||
fi
|
||||
|
||||
if [ -z "$EX" ]; then emit exchange ERROR container "not running"
|
||||
else
|
||||
emit exchange INFO container "$(podman ps --filter name=$EX --format '{{.Names}} {{.Status}}' | head -1)"
|
||||
c=$(hc http://127.0.0.1:9011/config)
|
||||
[ "$c" = "200" ] && emit exchange OK "local /config" "HTTP $c" || emit exchange ERROR "local /config" "HTTP $c"
|
||||
c=$(curl -sS -m 5 -o /dev/null -w '%{http_code}' http://127.0.0.1:9011/keys 2>/dev/null || echo 000)
|
||||
[ "$c" = "200" ] && emit exchange OK "local /keys" "HTTP $c" || emit exchange ERROR "local /keys" "HTTP $c"
|
||||
hasp "$EX" 'taler-exchange-httpd' && emit exchange OK httpd "running" || emit exchange ERROR httpd "not running"
|
||||
hasp "$EX" 'taler-exchange-wirewatch' && emit exchange OK wirewatch "running" || emit exchange ERROR wirewatch "not running — withdraw stuck after bank confirm"
|
||||
hasp "$EX" 'taler-exchange-aggregator' && emit exchange OK aggregator "running" || emit exchange WARN aggregator "not running"
|
||||
hasp "$EX" 'taler-exchange-transfer' && emit exchange OK transfer "running" || emit exchange WARN transfer "not running"
|
||||
# critical for wire gateway
|
||||
check_dns exchange "$EX" bank.hacktivism.ch || true
|
||||
check_dns exchange "$EX" exchange.hacktivism.ch || true
|
||||
check_dns exchange "$EX" taler.hacktivism.ch || true
|
||||
fi
|
||||
|
||||
if [ -z "$MER" ]; then emit merchant ERROR container "not running"
|
||||
else
|
||||
emit merchant INFO container "$(podman ps --filter name=$MER --format '{{.Names}} {{.Status}}' | head -1)"
|
||||
c=$(hc https://127.0.0.1:9010/config)
|
||||
[ "$c" = "200" ] && emit merchant OK "local /config" "HTTP $c" || emit merchant ERROR "local /config" "HTTP $c"
|
||||
hasp "$MER" 'taler-merchant-httpd' && emit merchant OK httpd "running" || emit merchant ERROR httpd "not running"
|
||||
hasp "$MER" 'taler-merchant-wirewatch' && emit merchant OK wirewatch "running" || emit merchant WARN wirewatch "not running"
|
||||
hasp "$MER" 'taler-merchant-depositcheck' && emit merchant OK depositcheck "running" || emit merchant WARN depositcheck "not running"
|
||||
check_dns merchant "$MER" bank.hacktivism.ch || true
|
||||
check_dns merchant "$MER" exchange.hacktivism.ch || true
|
||||
check_dns merchant "$MER" taler.hacktivism.ch || true
|
||||
fi
|
||||
|
||||
if systemctl is-active caddy >/dev/null 2>&1 || pgrep -x caddy >/dev/null 2>&1; then
|
||||
emit caddy OK process "active"
|
||||
else
|
||||
emit caddy ERROR process "not active"
|
||||
fi
|
||||
echo DONE
|
||||
REMOTE
|
||||
)
|
||||
|
||||
if [ -z "$RAW" ] || ! echo "$RAW" | grep -q '^E|'; then
|
||||
err "ssh" "remote timed out or empty (cap ${SSH_CMD_TIMEOUT}s)"
|
||||
summary
|
||||
exit 1
|
||||
fi
|
||||
|
||||
while IFS= read -r line; do
|
||||
case "$line" in
|
||||
E\|*)
|
||||
IFS='|' read -r _ comp level key detail <<<"$line"
|
||||
case "$level" in
|
||||
OK) ok "[$comp] $key${detail:+ ($detail)}" ;;
|
||||
ERROR) err "$comp" "$key" "$detail" ;;
|
||||
WARN) warn "[$comp] $key" "$detail" ;;
|
||||
INFO) info "[$comp] $key" "$detail" ;;
|
||||
esac
|
||||
;;
|
||||
esac
|
||||
done <<<"$RAW"
|
||||
|
||||
summary
|
||||
281
scripts/taler-monitoring/check_sanity.sh
Executable file
281
scripts/taler-monitoring/check_sanity.sh
Executable file
|
|
@ -0,0 +1,281 @@
|
|||
#!/usr/bin/env bash
|
||||
# Sanity checks for bank · exchange · merchant (public + server-side).
|
||||
# Sections are independent; continues after failures.
|
||||
set -euo pipefail
|
||||
ROOT=$(cd "$(dirname "$0")" && pwd)
|
||||
# shellcheck source=lib.sh
|
||||
source "$ROOT/lib.sh"
|
||||
|
||||
tmp=$(mktemp -d)
|
||||
trap 'rm -rf "$tmp"' EXIT
|
||||
|
||||
expect_code() {
|
||||
local label="$1" want="$2" url="$3"
|
||||
local code
|
||||
code=$(http_code "$url")
|
||||
case ",$want," in
|
||||
*",$code,"*) ok "$label" ;;
|
||||
*) fail "$label" "HTTP $code (want $want) $url" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
json_currency() {
|
||||
python3 -c 'import json,sys; d=json.load(open(sys.argv[1])); print(d.get("currency") or "")' "$1" 2>/dev/null || true
|
||||
}
|
||||
|
||||
# Area sanity-### — public + optional server-side per component
|
||||
set_area sanity
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
section "sanity · bank"
|
||||
# ---------------------------------------------------------------------------
|
||||
expect_code "bank public /config" 200 "$BANK_PUBLIC/config"
|
||||
expect_code "bank public /taler-integration/config" 200 "$BANK_PUBLIC/taler-integration/config"
|
||||
expect_code "bank public /webui/" 200 "$BANK_PUBLIC/webui/"
|
||||
|
||||
code=$(http_body "$BANK_PUBLIC/config" "$tmp/bank-config.json")
|
||||
if [ "$code" = "200" ]; then
|
||||
cur=$(json_currency "$tmp/bank-config.json")
|
||||
[ "$cur" = "GOA" ] && ok "bank currency GOA" || fail "bank currency" "got ${cur:-?}"
|
||||
# wire type / name if present
|
||||
python3 - "$tmp/bank-config.json" <<'PY' 2>/dev/null && ok "bank config JSON object" || fail "bank config JSON"
|
||||
import json,sys
|
||||
d=json.load(open(sys.argv[1]))
|
||||
sys.exit(0 if isinstance(d, dict) and d.get("currency") else 1)
|
||||
PY
|
||||
if json_has_alt_unit_names "$tmp/bank-config.json" >/tmp/alt-bank-s.$$ 2>&1; then
|
||||
ok "bank /config alt_unit_names" "$(tr '\n' '; ' </tmp/alt-bank-s.$$ | sed 's/; $//')"
|
||||
else
|
||||
fail "bank /config alt_unit_names" "$(tr '\n' '; ' </tmp/alt-bank-s.$$ | sed 's/; $//')"
|
||||
fi
|
||||
rm -f /tmp/alt-bank-s.$$
|
||||
fi
|
||||
|
||||
# server-side bank
|
||||
if koopa_ssh_ok; then
|
||||
BOUT=$(koopa_ssh_bash 40 <<'REMOTE' || true
|
||||
set +e
|
||||
BANK=$(podman ps --format '{{.Names}}' | grep -iE 'hacktivism-bank|taler-bank' | head -1)
|
||||
[ -z "$BANK" ] && BANK=$(podman ps --format '{{.Names}}' | grep -i bank | head -1)
|
||||
echo "CTR=$BANK"
|
||||
if [ -z "$BANK" ]; then echo "NOCTR"; exit 0; fi
|
||||
code=$(curl -sS -m 5 -o /dev/null -w '%{http_code}' http://127.0.0.1:9012/config 2>/dev/null || echo 000)
|
||||
echo "LOCAL_CONFIG=$code"
|
||||
code2=$(curl -sS -m 5 -o /dev/null -w '%{http_code}' http://127.0.0.1:9012/taler-integration/config 2>/dev/null || echo 000)
|
||||
echo "LOCAL_INT=$code2"
|
||||
if podman exec "$BANK" bash -c 'pgrep -f "MainKt serve|libeufin-bank serve" >/dev/null' 2>/dev/null; then
|
||||
echo "LIBEUFIN=1"
|
||||
else
|
||||
echo "LIBEUFIN=0"
|
||||
fi
|
||||
if podman exec "$BANK" bash -c 'pg_isready -q' 2>/dev/null; then
|
||||
echo "PG=1"
|
||||
else
|
||||
echo "PG=0"
|
||||
fi
|
||||
# in-container health if present
|
||||
if podman exec "$BANK" test -x /usr/local/bin/check_bank-health.sh 2>/dev/null; then
|
||||
podman exec "$BANK" /usr/local/bin/check_bank-health.sh 2>&1 | sed 's/^/HEALTH /' | tail -20
|
||||
echo "HEALTH_EC=${PIPESTATUS[0]}"
|
||||
fi
|
||||
REMOTE
|
||||
)
|
||||
echo "$BOUT" | grep -q '^CTR=.\+' && ok "bank container $(echo "$BOUT" | sed -n 's/^CTR=//p' | head -1)" || fail "bank container" "not found"
|
||||
echo "$BOUT" | grep -q 'LOCAL_CONFIG=200' && ok "bank local :9012/config" || fail "bank local :9012/config"
|
||||
echo "$BOUT" | grep -q 'LOCAL_INT=200' && ok "bank local :9012/taler-integration/config" || fail "bank local integration"
|
||||
echo "$BOUT" | grep -q 'LIBEUFIN=1' && ok "bank libeufin-bank process" || fail "bank libeufin-bank process"
|
||||
echo "$BOUT" | grep -q 'PG=1' && ok "bank postgres ready" || warn "bank postgres" "pg_isready failed"
|
||||
if echo "$BOUT" | grep -q 'HEALTH '; then
|
||||
if echo "$BOUT" | grep -qE 'HEALTH_EC=0|ALL CRITICAL CHECKS PASSED'; then
|
||||
ok "bank check_bank-health.sh"
|
||||
else
|
||||
# health script may false-fail process grep; warn not fail if local config ok
|
||||
warn "bank check_bank-health.sh" "non-zero or incomplete"
|
||||
fi
|
||||
fi
|
||||
else
|
||||
warn "bank server-side" "ssh ${KOOPA_SSH} unavailable"
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
section "sanity · exchange"
|
||||
# ---------------------------------------------------------------------------
|
||||
expect_code "exchange public /config" 200 "$EXCHANGE_PUBLIC/config"
|
||||
expect_code "exchange public /keys" 200 "$EXCHANGE_PUBLIC/keys"
|
||||
expect_code "exchange public /terms" 200 "$EXCHANGE_PUBLIC/terms"
|
||||
|
||||
code=$(http_body "$EXCHANGE_PUBLIC/config" "$tmp/ex-config.json")
|
||||
if [ "$code" = "200" ]; then
|
||||
cur=$(json_currency "$tmp/ex-config.json")
|
||||
[ "$cur" = "GOA" ] && ok "exchange currency GOA" || fail "exchange currency" "got ${cur:-?}"
|
||||
if json_has_alt_unit_names "$tmp/ex-config.json" "GOA" >/tmp/alt-ex-s.$$ 2>&1; then
|
||||
ok "exchange /config alt_unit_names" "$(tr '\n' '; ' </tmp/alt-ex-s.$$ | sed 's/; $//')"
|
||||
else
|
||||
fail "exchange /config alt_unit_names" "$(tr '\n' '; ' </tmp/alt-ex-s.$$ | sed 's/; $//')"
|
||||
fi
|
||||
rm -f /tmp/alt-ex-s.$$
|
||||
fi
|
||||
|
||||
code=$(http_body "$EXCHANGE_PUBLIC/keys" "$tmp/ex-keys.json")
|
||||
if [ "$code" = "200" ]; then
|
||||
python3 - "$tmp/ex-keys.json" <<'PY'
|
||||
import json,sys,time
|
||||
d=json.load(open(sys.argv[1]))
|
||||
sk=d.get("signkeys") or []
|
||||
acc=d.get("accounts") or []
|
||||
den=d.get("denominations") or []
|
||||
# count denoms roughly
|
||||
n=0
|
||||
for g in den:
|
||||
if isinstance(g, dict):
|
||||
n += len(g.get("denoms") or [])
|
||||
print(f"signkeys={len(sk)} accounts={len(acc)} denom_groups={len(den)} denoms~={n}")
|
||||
sys.exit(0 if sk and (acc or n) else 1)
|
||||
PY
|
||||
ec=$?
|
||||
detail=$(python3 - "$tmp/ex-keys.json" <<'PY'
|
||||
import json,sys
|
||||
d=json.load(open(sys.argv[1]))
|
||||
sk=d.get("signkeys") or []
|
||||
acc=d.get("accounts") or []
|
||||
den=d.get("denominations") or []
|
||||
n=sum(len(g.get("denoms") or []) for g in den if isinstance(g, dict))
|
||||
print(f"signkeys={len(sk)} accounts={len(acc)} denoms~={n}")
|
||||
PY
|
||||
)
|
||||
[ "$ec" -eq 0 ] && ok "exchange /keys usable ($detail)" || fail "exchange /keys usable" "$detail"
|
||||
fi
|
||||
|
||||
# /wire if exposed
|
||||
wcode=$(http_code "$EXCHANGE_PUBLIC/wire")
|
||||
if [ "$wcode" = "200" ]; then
|
||||
ok "exchange public /wire"
|
||||
else
|
||||
warn "exchange public /wire" "HTTP $wcode (accounts may only be in /keys)"
|
||||
fi
|
||||
|
||||
if koopa_ssh_ok; then
|
||||
EOUT=$(koopa_ssh_bash 40 <<'REMOTE' || true
|
||||
set +e
|
||||
EX=$(podman ps --format '{{.Names}}' | grep -i exchange | head -1)
|
||||
echo "CTR=$EX"
|
||||
code=$(curl -sS -m 5 -o /dev/null -w '%{http_code}' http://127.0.0.1:9011/config 2>/dev/null || echo 000)
|
||||
echo "LOCAL_CONFIG=$code"
|
||||
codek=$(curl -sS -m 8 -o /dev/null -w '%{http_code}' http://127.0.0.1:9011/keys 2>/dev/null || echo 000)
|
||||
echo "LOCAL_KEYS=$codek"
|
||||
if [ -n "$EX" ]; then
|
||||
if podman exec "$EX" bash -c 'pgrep -f taler-exchange-httpd >/dev/null' 2>/dev/null; then
|
||||
echo "HTTPD=1"
|
||||
else
|
||||
echo "HTTPD=0"
|
||||
fi
|
||||
for p in taler-exchange-secmod-rsa taler-exchange-secmod-eddsa taler-exchange-wirewatch taler-exchange-aggregator; do
|
||||
if podman exec "$EX" bash -c "pgrep -f $p >/dev/null" 2>/dev/null; then
|
||||
echo "PROC_$p=1"
|
||||
else
|
||||
echo "PROC_$p=0"
|
||||
fi
|
||||
done
|
||||
if podman exec "$EX" test -x /usr/local/bin/check_exchange-health.sh 2>/dev/null; then
|
||||
SKIP_ENSURE=1 podman exec -e SKIP_ENSURE=1 "$EX" /usr/local/bin/check_exchange-health.sh 2>&1 | sed 's/^/HEALTH /' | tail -25
|
||||
fi
|
||||
fi
|
||||
REMOTE
|
||||
)
|
||||
echo "$EOUT" | grep -q '^CTR=.\+' && ok "exchange container $(echo "$EOUT" | sed -n 's/^CTR=//p' | head -1)" || fail "exchange container"
|
||||
echo "$EOUT" | grep -q 'LOCAL_CONFIG=200' && ok "exchange local :9011/config" || fail "exchange local :9011/config"
|
||||
echo "$EOUT" | grep -q 'LOCAL_KEYS=200' && ok "exchange local :9011/keys" || fail "exchange local :9011/keys"
|
||||
echo "$EOUT" | grep -q 'HTTPD=1' && ok "exchange-httpd process" || warn "exchange-httpd process" "not detected"
|
||||
echo "$EOUT" | grep -q 'PROC_taler-exchange-wirewatch=1' && ok "exchange wirewatch" || warn "exchange wirewatch" "not running"
|
||||
echo "$EOUT" | grep -q 'PROC_taler-exchange-aggregator=1' && ok "exchange aggregator" || warn "exchange aggregator" "not running"
|
||||
else
|
||||
warn "exchange server-side" "ssh unavailable"
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
section "sanity · merchant"
|
||||
# ---------------------------------------------------------------------------
|
||||
expect_code "merchant public /config" 200 "$MERCHANT_PUBLIC/config"
|
||||
expect_code "merchant public /webui/" 200 "$MERCHANT_PUBLIC/webui/"
|
||||
|
||||
code=$(http_body "$MERCHANT_PUBLIC/config" "$tmp/mer-config.json")
|
||||
if [ "$code" = "200" ]; then
|
||||
if python3 - "$tmp/mer-config.json" "$EXCHANGE_PUBLIC" <<'PY'
|
||||
import json,sys
|
||||
d=json.load(open(sys.argv[1]))
|
||||
want=sys.argv[2].rstrip("/")
|
||||
curs=list((d.get("currencies") or {}).keys())
|
||||
ex=d.get("exchanges") or []
|
||||
urls=[]
|
||||
for e in ex:
|
||||
if isinstance(e, dict):
|
||||
u=e.get("base_url") or e.get("url") or e.get("exchange_base_url") or ""
|
||||
if u: urls.append(u.rstrip("/"))
|
||||
elif isinstance(e, str):
|
||||
urls.append(e.rstrip("/"))
|
||||
ok_goa = "GOA" in curs or any((e.get("currency") if isinstance(e, dict) else None)=="GOA" for e in ex)
|
||||
ok_ex = any(want in u or "exchange.hacktivism.ch" in u for u in urls)
|
||||
print("currencies", curs)
|
||||
print("exchanges", urls[:5])
|
||||
sys.exit(0 if ok_goa and ok_ex else 1)
|
||||
PY
|
||||
then
|
||||
ok "merchant config GOA + exchange.hacktivism.ch"
|
||||
else
|
||||
fail "merchant config GOA + exchange" "see currencies/exchanges"
|
||||
fi
|
||||
if json_has_alt_unit_names "$tmp/mer-config.json" >/tmp/alt-mer-s.$$ 2>&1; then
|
||||
ok "merchant currencies alt_unit_names" "$(tr '\n' '; ' </tmp/alt-mer-s.$$ | sed 's/; $//')"
|
||||
else
|
||||
fail "merchant currencies alt_unit_names" "$(tr '\n' '; ' </tmp/alt-mer-s.$$ | sed 's/; $//')"
|
||||
fi
|
||||
rm -f /tmp/alt-mer-s.$$
|
||||
# Each exchange listed on merchant /config must publish alt_unit_names on its own /config
|
||||
check_merchant_listed_exchanges_alt_units "$tmp/mer-config.json"
|
||||
fi
|
||||
|
||||
# demo instance reachable?
|
||||
INST="${MERCHANT_INSTANCE}"
|
||||
icode=$(http_code "$MERCHANT_PUBLIC/instances/${INST}/config")
|
||||
if [ "$icode" = "200" ]; then
|
||||
ok "merchant instance ${INST} /config"
|
||||
else
|
||||
# some deployments use private only
|
||||
warn "merchant instance ${INST} /config" "HTTP $icode"
|
||||
fi
|
||||
|
||||
if koopa_ssh_ok; then
|
||||
MOUT=$(koopa_ssh_bash 40 <<'REMOTE' || true
|
||||
set +e
|
||||
MER=$(podman ps --format '{{.Names}}' | grep -E '^taler-hacktivism$' | head -1)
|
||||
[ -z "$MER" ] && MER=$(podman ps --format '{{.Names}}' | grep -iE 'merchant|hacktivism' | grep -viE 'bank|exchange' | head -1)
|
||||
echo "CTR=$MER"
|
||||
code=$(curl -skS -m 5 -o /dev/null -w '%{http_code}' https://127.0.0.1:9010/config 2>/dev/null || echo 000)
|
||||
echo "LOCAL_CONFIG=$code"
|
||||
if [ -n "$MER" ]; then
|
||||
if podman exec "$MER" bash -c 'pgrep -f taler-merchant-httpd >/dev/null' 2>/dev/null; then
|
||||
echo "HTTPD=1"
|
||||
else
|
||||
echo "HTTPD=0"
|
||||
fi
|
||||
if podman exec "$MER" test -x /usr/local/bin/check_merchant-health.sh 2>/dev/null; then
|
||||
SKIP_ENSURE=1 podman exec -e SKIP_ENSURE=1 "$MER" /usr/local/bin/check_merchant-health.sh 2>&1 | sed 's/^/HEALTH /' | tail -30
|
||||
fi
|
||||
fi
|
||||
REMOTE
|
||||
)
|
||||
echo "$MOUT" | grep -q '^CTR=.\+' && ok "merchant container $(echo "$MOUT" | sed -n 's/^CTR=//p' | head -1)" || fail "merchant container"
|
||||
echo "$MOUT" | grep -q 'LOCAL_CONFIG=200' && ok "merchant local :9010/config" || fail "merchant local :9010/config"
|
||||
echo "$MOUT" | grep -q 'HTTPD=1' && ok "merchant-httpd process" || warn "merchant-httpd process" "not detected"
|
||||
if echo "$MOUT" | grep -q 'HEALTH '; then
|
||||
if echo "$MOUT" | grep -qiE 'ALL CRITICAL|HEALTH_EC=0|\[OK\]'; then
|
||||
ok "merchant check_merchant-health.sh (sample OK)"
|
||||
else
|
||||
warn "merchant check_merchant-health.sh" "see remote output"
|
||||
fi
|
||||
fi
|
||||
else
|
||||
warn "merchant server-side" "ssh unavailable"
|
||||
fi
|
||||
|
||||
summary
|
||||
128
scripts/taler-monitoring/check_server.sh
Executable file
128
scripts/taler-monitoring/check_server.sh
Executable file
|
|
@ -0,0 +1,128 @@
|
|||
#!/usr/bin/env bash
|
||||
# Server-side component checks on koopa (via SSH).
|
||||
set -euo pipefail
|
||||
ROOT=$(cd "$(dirname "$0")" && pwd)
|
||||
# shellcheck source=lib.sh
|
||||
source "$ROOT/lib.sh"
|
||||
|
||||
# Area server-### — host/container ports via SSH
|
||||
set_area server
|
||||
section "server · ssh ${KOOPA_SSH}"
|
||||
|
||||
if ! koopa_ssh_ok; then
|
||||
fail "ssh ${KOOPA_SSH}" "unreachable within ${SSH_CONNECT_TIMEOUT}s (SKIP_SSH=1 to skip)"
|
||||
summary
|
||||
exit 1
|
||||
fi
|
||||
ok "ssh ${KOOPA_SSH}"
|
||||
|
||||
# Run a remote script; collect structured lines
|
||||
REMOTE=$(koopa_ssh_bash "${SSH_CMD_TIMEOUT}" <<'REMOTE'
|
||||
set +e
|
||||
report() { printf 'R|%s|%s|%s\n' "$1" "$2" "$3"; }
|
||||
|
||||
# containers
|
||||
for name in taler-hacktivism-bank taler-hacktivism-exchange-ansible taler-hacktivism; do
|
||||
if podman ps --format '{{.Names}}' 2>/dev/null | grep -qx "$name"; then
|
||||
st=$(podman ps --filter "name=^${name}$" --format '{{.Status}}' | head -1)
|
||||
report OK "container $name" "$st"
|
||||
else
|
||||
# soft match
|
||||
hit=$(podman ps --format '{{.Names}}' 2>/dev/null | grep -E "$name|bank|exchange|hacktivism" | head -3 | tr '\n' ' ')
|
||||
if podman ps --format '{{.Names}}' 2>/dev/null | grep -q bank && [ "$name" = taler-hacktivism-bank ]; then
|
||||
bn=$(podman ps --format '{{.Names}}' | grep -i bank | head -1)
|
||||
report OK "container bank ($bn)" "$(podman ps --filter name="$bn" --format '{{.Status}}' | head -1)"
|
||||
elif podman ps --format '{{.Names}}' 2>/dev/null | grep -qi exchange && echo "$name" | grep -qi exchange; then
|
||||
en=$(podman ps --format '{{.Names}}' | grep -i exchange | head -1)
|
||||
report OK "container exchange ($en)" "$(podman ps --filter name="$en" --format '{{.Status}}' | head -1)"
|
||||
elif podman ps --format '{{.Names}}' 2>/dev/null | grep -qx taler-hacktivism && [ "$name" = taler-hacktivism ]; then
|
||||
report OK "container taler-hacktivism" "$(podman inspect -f '{{.State.Status}}' taler-hacktivism 2>/dev/null)"
|
||||
else
|
||||
report FAIL "container $name" "not running (seen: $hit)"
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
# local HTTP on pasta ports
|
||||
for spec in \
|
||||
"bank-api|http://127.0.0.1:9012/config|200" \
|
||||
"bank-integration|http://127.0.0.1:9012/taler-integration/config|200" \
|
||||
"landing|http://127.0.0.1:9013/intro/|200" \
|
||||
"exchange|http://127.0.0.1:9011/config|200" \
|
||||
"merchant|https://127.0.0.1:9010/config|200"
|
||||
do
|
||||
IFS='|' read -r id url want <<<"$spec"
|
||||
code=$(curl -skS -m 5 -o /tmp/mon-s.out -w '%{http_code}' "$url" 2>/dev/null || echo 000)
|
||||
if [ "$code" = "$want" ]; then
|
||||
report OK "local $id : ${url#*//}" "HTTP $code"
|
||||
else
|
||||
report FAIL "local $id" "HTTP $code want $want"
|
||||
fi
|
||||
done
|
||||
|
||||
# processes inside bank
|
||||
BANK=$(podman ps --format '{{.Names}}' | grep -iE 'bank|hacktivism-bank' | head -1)
|
||||
if [ -n "$BANK" ]; then
|
||||
if podman exec "$BANK" bash -c 'pgrep -f "MainKt serve|libeufin-bank serve" >/dev/null' 2>/dev/null; then
|
||||
report OK "libeufin-bank process" "in $BANK"
|
||||
else
|
||||
report FAIL "libeufin-bank process" "not running in $BANK"
|
||||
fi
|
||||
if podman exec "$BANK" bash -c 'pg_isready -q' 2>/dev/null; then
|
||||
report OK "postgres (bank)" "ready"
|
||||
else
|
||||
report WARN "postgres (bank)" "pg_isready failed"
|
||||
fi
|
||||
if podman exec "$BANK" bash -c 'pgrep -x nginx >/dev/null' 2>/dev/null; then
|
||||
report OK "nginx landing" "in $BANK"
|
||||
else
|
||||
report WARN "nginx landing" "not seen in $BANK"
|
||||
fi
|
||||
else
|
||||
report FAIL "bank container" "none"
|
||||
fi
|
||||
|
||||
# exchange process / systemd if any
|
||||
EX=$(podman ps --format '{{.Names}}' | grep -i exchange | head -1)
|
||||
if [ -n "$EX" ]; then
|
||||
if podman exec "$EX" bash -c 'pgrep -f taler-exchange-httpd >/dev/null || systemctl is-active taler-exchange-httpd 2>/dev/null | grep -q active' 2>/dev/null; then
|
||||
report OK "exchange-httpd" "in $EX"
|
||||
else
|
||||
# config answering is enough
|
||||
report WARN "exchange-httpd process" "not detected; port check above"
|
||||
fi
|
||||
fi
|
||||
|
||||
MER=$(podman ps --format '{{.Names}}' | grep -E '^taler-hacktivism$' | head -1)
|
||||
[ -z "$MER" ] && MER=$(podman ps --format '{{.Names}}' | grep -i merchant | head -1)
|
||||
if [ -n "$MER" ]; then
|
||||
if podman exec "$MER" bash -c 'pgrep -f taler-merchant-httpd >/dev/null || true; curl -sk -m 3 -o /dev/null -w %{http_code} https://127.0.0.1:9010/config' 2>/dev/null | grep -q 200; then
|
||||
report OK "merchant-httpd" "responds in $MER"
|
||||
else
|
||||
report WARN "merchant-httpd" "check manually in $MER"
|
||||
fi
|
||||
fi
|
||||
|
||||
# caddy on host
|
||||
if systemctl is-active caddy >/dev/null 2>&1 || pgrep -x caddy >/dev/null 2>&1; then
|
||||
report OK "caddy" "active"
|
||||
else
|
||||
report WARN "caddy" "not detected as active"
|
||||
fi
|
||||
REMOTE
|
||||
)
|
||||
|
||||
while IFS= read -r line; do
|
||||
case "$line" in
|
||||
R\|*)
|
||||
IFS='|' read -r _ st label detail <<<"$line"
|
||||
case "$st" in
|
||||
OK) ok "$label${detail:+ ($detail)}" ;;
|
||||
FAIL) fail "$label" "$detail" ;;
|
||||
WARN) warn "$label" "$detail" ;;
|
||||
esac
|
||||
;;
|
||||
esac
|
||||
done <<<"$REMOTE"
|
||||
|
||||
summary
|
||||
255
scripts/taler-monitoring/check_urls.sh
Executable file
255
scripts/taler-monitoring/check_urls.sh
Executable file
|
|
@ -0,0 +1,255 @@
|
|||
#!/usr/bin/env bash
|
||||
# Outside-in public HTTPS checks (no SSH).
|
||||
set -euo pipefail
|
||||
ROOT=$(cd "$(dirname "$0")" && pwd)
|
||||
# shellcheck source=lib.sh
|
||||
source "$ROOT/lib.sh"
|
||||
|
||||
# Area www-### — public HTTPS (outside-in)
|
||||
set_area www
|
||||
section "www · public URLs · ${TALER_DOMAIN:-?} (outside-in, no SSH)"
|
||||
|
||||
tmp=$(mktemp -d)
|
||||
trap 'rm -rf "$tmp"' EXIT
|
||||
|
||||
check_url() {
|
||||
local label="$1" expect="$2" url="$3"
|
||||
local code
|
||||
code=$(http_code "$url")
|
||||
case ",$expect," in
|
||||
*",$code,"*) ok "$label $url" ;;
|
||||
*) fail "$label $url" "got $code want $expect" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# Soft check: OK on expect, WARN on foreign stack if down, ERROR on local stack
|
||||
check_url_soft() {
|
||||
local label="$1" expect="$2" url="$3"
|
||||
local code
|
||||
code=$(http_code "$url")
|
||||
case ",$expect," in
|
||||
*",$code,"*) ok "$label $url" ;;
|
||||
*)
|
||||
if [ "${LOCAL_STACK:-1}" = "0" ]; then
|
||||
warn "$label $url" "got $code (optional on remote domain)"
|
||||
else
|
||||
fail "$label $url" "got $code want $expect"
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
expect_currency() {
|
||||
local label="$1" file="$2" want="${EXPECT_CURRENCY:-}"
|
||||
local cur
|
||||
cur=$(python3 -c 'import json,sys;print(json.load(open(sys.argv[1])).get("currency",""))' "$file" 2>/dev/null || true)
|
||||
if [ -z "$want" ]; then
|
||||
info "$label currency" "${cur:-?}"
|
||||
return
|
||||
fi
|
||||
if [ "$cur" = "$want" ]; then
|
||||
ok "$label currency=$want"
|
||||
else
|
||||
fail "$label currency" "got ${cur:-?} want $want"
|
||||
fi
|
||||
}
|
||||
|
||||
# Legal docs: /terms and /privacy must be HTTP 200 with a real document body
|
||||
# (not empty, not "not configured", not JSON API error).
|
||||
# $1=label $2=url $3=optional needle regex (case-insensitive) for local stack
|
||||
check_legal_doc() {
|
||||
local label="$1" url="$2" needle="${3:-}"
|
||||
local f code soft
|
||||
soft=0
|
||||
[ "${LOCAL_STACK:-1}" = "0" ] && soft=1
|
||||
f=$(mktemp)
|
||||
code=$(curl -skS --max-redirs 5 -L -m "${TIMEOUT}" \
|
||||
-H "Accept: text/html,text/markdown,text/plain,*/*" \
|
||||
-o "$f" -w '%{http_code}' "$url" 2>/dev/null || echo 000)
|
||||
if [ "$code" != "200" ]; then
|
||||
rm -f "$f"
|
||||
if [ "$soft" = "1" ]; then
|
||||
warn "$label" "HTTP $code — $url"
|
||||
else
|
||||
fail "$label" "HTTP $code — $url"
|
||||
fi
|
||||
return
|
||||
fi
|
||||
if [ ! -s "$f" ]; then
|
||||
rm -f "$f"
|
||||
fail "$label" "empty body — $url"
|
||||
return
|
||||
fi
|
||||
# merchant returns plain "not configured" when PRIVACY_ETAG missing
|
||||
if grep -qiE '^(not configured)\s*$' "$f" 2>/dev/null \
|
||||
|| grep -qiE '"code"\s*:\s*21' "$f" 2>/dev/null; then
|
||||
rm -f "$f"
|
||||
fail "$label" "not configured / API error — $url"
|
||||
return
|
||||
fi
|
||||
if [ -n "$needle" ] && [ "${LOCAL_STACK:-1}" = "1" ]; then
|
||||
if ! grep -qiE "$needle" "$f" 2>/dev/null; then
|
||||
warn "$label content" "missing /$needle/ — $url"
|
||||
rm -f "$f"
|
||||
return
|
||||
fi
|
||||
fi
|
||||
ok "$label" "HTTP 200 · $(wc -c <"$f" | tr -d ' ') bytes"
|
||||
rm -f "$f"
|
||||
}
|
||||
|
||||
# --- exchange (core; always required) --- www-001 …
|
||||
check_url "exchange /config" 200 "$EXCHANGE_PUBLIC/config"
|
||||
code=$(http_body "$EXCHANGE_PUBLIC/config" "$tmp/ec.json")
|
||||
if [ "$code" = "200" ]; then
|
||||
expect_currency "exchange" "$tmp/ec.json"
|
||||
# currency_specification.alt_unit_names (wallet codec)
|
||||
if json_has_alt_unit_names "$tmp/ec.json" "${EXPECT_CURRENCY:-}" >/tmp/alt-ex.$$ 2>&1; then
|
||||
ok "exchange /config alt_unit_names" "$(tr '\n' '; ' </tmp/alt-ex.$$ | sed 's/; $//')"
|
||||
else
|
||||
fail "exchange /config alt_unit_names" "$(tr '\n' '; ' </tmp/alt-ex.$$ | sed 's/; $//')"
|
||||
fi
|
||||
rm -f /tmp/alt-ex.$$
|
||||
fi
|
||||
check_url "exchange /keys" 200 "$EXCHANGE_PUBLIC/keys"
|
||||
# /keys should also expose currency_specification.alt_unit_names when present
|
||||
code=$(http_body "$EXCHANGE_PUBLIC/keys" "$tmp/ek.json")
|
||||
if [ "$code" = "200" ]; then
|
||||
if python3 - "$tmp/ek.json" <<'PY'
|
||||
import json,sys
|
||||
d=json.load(open(sys.argv[1]))
|
||||
cs=d.get("currency_specification") or {}
|
||||
au=cs.get("alt_unit_names") if isinstance(cs,dict) else None
|
||||
if not isinstance(au, dict) or "0" not in au:
|
||||
# older keys without embedded spec: not a hard fail if /config is good
|
||||
sys.exit(2)
|
||||
sys.exit(0)
|
||||
PY
|
||||
then
|
||||
ok "exchange /keys alt_unit_names"
|
||||
else
|
||||
ec=$?
|
||||
if [ "$ec" = "2" ]; then
|
||||
warn "exchange /keys alt_unit_names" "no currency_specification in /keys (ok if /config has it)"
|
||||
else
|
||||
fail "exchange /keys alt_unit_names"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
check_url_soft "exchange /intro/" 200 "$EXCHANGE_PUBLIC/intro/"
|
||||
# Root should land on intro (302/301 then 200 on follow is checked separately)
|
||||
check_url_soft "exchange /" 302,301,200 "$EXCHANGE_PUBLIC/"
|
||||
|
||||
# Terms + privacy (legal docs)
|
||||
check_legal_doc "exchange /terms" "$EXCHANGE_PUBLIC/terms" "terms|GOA|exploration|FADP|revDSG|privacy"
|
||||
check_legal_doc "exchange /privacy" "$EXCHANGE_PUBLIC/privacy" "privacy|FADP|revDSG|data|GOA|exploration"
|
||||
# trailing slash: 200 or redirect to bare path
|
||||
code=$(http_code "$EXCHANGE_PUBLIC/terms/")
|
||||
case "$code" in
|
||||
200|301|302) ok "exchange /terms/" "HTTP $code" ;;
|
||||
*)
|
||||
if [ "${LOCAL_STACK:-1}" = "1" ]; then warn "exchange /terms/" "HTTP $code"
|
||||
else warn "exchange /terms/" "HTTP $code"
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
|
||||
# --- bank ---
|
||||
if [ "${LOCAL_STACK:-1}" = "0" ]; then
|
||||
check_url_soft "bank /config" 200 "$BANK_PUBLIC/config"
|
||||
else
|
||||
check_url "bank /config" 200 "$BANK_PUBLIC/config"
|
||||
fi
|
||||
code=$(http_body "$BANK_PUBLIC/config" "$tmp/bc.json")
|
||||
if [ "$code" = "200" ]; then
|
||||
expect_currency "bank" "$tmp/bc.json"
|
||||
if json_has_alt_unit_names "$tmp/bc.json" >/tmp/alt-bank.$$ 2>&1; then
|
||||
ok "bank /config alt_unit_names" "$(tr '\n' '; ' </tmp/alt-bank.$$ | sed 's/; $//')"
|
||||
else
|
||||
fail "bank /config alt_unit_names" "$(tr '\n' '; ' </tmp/alt-bank.$$ | sed 's/; $//')"
|
||||
fi
|
||||
rm -f /tmp/alt-bank.$$
|
||||
check_url_soft "bank /taler-integration/config" 200 "$BANK_PUBLIC/taler-integration/config"
|
||||
check_url_soft "bank /webui/" 200 "$BANK_PUBLIC/webui/"
|
||||
check_url_soft "bank /intro/" 200 "$BANK_PUBLIC/intro/"
|
||||
check_url_soft "bank /" 302,301,200 "$BANK_PUBLIC/"
|
||||
fi
|
||||
|
||||
# Bank legal docs (landing nginx via Caddy /terms* /privacy* or /intro/*)
|
||||
check_legal_doc "bank /terms" "$BANK_PUBLIC/terms" "terms|GOA|exploration|bank|FADP|revDSG"
|
||||
# Prefer /privacy; fall back to /intro/privacy.html for older deploys
|
||||
code=$(http_code "$BANK_PUBLIC/privacy")
|
||||
if [ "$code" = "200" ]; then
|
||||
check_legal_doc "bank /privacy" "$BANK_PUBLIC/privacy" "privacy|FADP|revDSG|data|GOA|bank"
|
||||
else
|
||||
if [ "${LOCAL_STACK:-1}" = "1" ]; then
|
||||
check_legal_doc "bank /privacy (or /intro/privacy.html)" \
|
||||
"$BANK_PUBLIC/intro/privacy.html" "privacy|FADP|revDSG|data|GOA|bank"
|
||||
# still report bare /privacy failure for local
|
||||
warn "bank /privacy" "HTTP $code — prefer Caddy handle /privacy* → landing"
|
||||
else
|
||||
check_url_soft "bank /privacy" 200 "$BANK_PUBLIC/privacy"
|
||||
fi
|
||||
fi
|
||||
|
||||
# --- merchant ---
|
||||
if [ "${LOCAL_STACK:-1}" = "0" ]; then
|
||||
check_url_soft "merchant /config" 200 "$MERCHANT_PUBLIC/config"
|
||||
else
|
||||
check_url "merchant /config" 200 "$MERCHANT_PUBLIC/config"
|
||||
fi
|
||||
code=$(http_body "$MERCHANT_PUBLIC/config" "$tmp/mc.json")
|
||||
if [ "$code" = "200" ]; then
|
||||
want="${EXPECT_CURRENCY:-}"
|
||||
if python3 - "$tmp/mc.json" "$want" <<'PY'
|
||||
import json,sys
|
||||
try:
|
||||
d=json.load(open(sys.argv[1]))
|
||||
except Exception:
|
||||
sys.exit(2)
|
||||
want=sys.argv[2]
|
||||
if not want:
|
||||
sys.exit(0)
|
||||
curs=list((d.get("currencies") or {}).keys())
|
||||
ex=d.get("exchanges") or []
|
||||
ok = want in curs or any((e.get("currency") if isinstance(e,dict) else None)==want for e in ex)
|
||||
if not ok and isinstance(d.get("currency"), str):
|
||||
ok = d["currency"]==want
|
||||
sys.exit(0 if ok else 1)
|
||||
PY
|
||||
then
|
||||
ok "merchant /config (${want:-currency} ok)"
|
||||
else
|
||||
ec=$?
|
||||
if [ "$ec" = "2" ]; then
|
||||
warn "merchant /config" "non-JSON body"
|
||||
elif [ -n "$want" ]; then
|
||||
fail "merchant /config currency" "want $want"
|
||||
else
|
||||
info "merchant /config" "ok"
|
||||
fi
|
||||
fi
|
||||
# merchant-local currency maps (GOA + CHF, …)
|
||||
if json_has_alt_unit_names "$tmp/mc.json" >/tmp/alt-mer.$$ 2>&1; then
|
||||
ok "merchant /config currencies alt_unit_names" "$(tr '\n' '; ' </tmp/alt-mer.$$ | sed 's/; $//')"
|
||||
else
|
||||
fail "merchant /config currencies alt_unit_names" "$(tr '\n' '; ' </tmp/alt-mer.$$ | sed 's/; $//')"
|
||||
fi
|
||||
rm -f /tmp/alt-mer.$$
|
||||
# Follow each exchange listed in merchant /config and require its /config alt_unit_names
|
||||
check_merchant_listed_exchanges_alt_units "$tmp/mc.json"
|
||||
check_url_soft "merchant /intro/" 200 "$MERCHANT_PUBLIC/intro/"
|
||||
check_url_soft "merchant /webui/" 200 "$MERCHANT_PUBLIC/webui/"
|
||||
check_url_soft "merchant /" 302,301,200 "$MERCHANT_PUBLIC/"
|
||||
fi
|
||||
|
||||
# Merchant legal docs
|
||||
check_legal_doc "merchant /terms" "$MERCHANT_PUBLIC/terms" "terms|dual|GOA|CHF|explorational|merchant"
|
||||
check_legal_doc "merchant /privacy" "$MERCHANT_PUBLIC/privacy" "privacy|FADP|revDSG|data|GOA|CHF|merchant"
|
||||
code=$(http_code "$MERCHANT_PUBLIC/terms/")
|
||||
case "$code" in
|
||||
200|301|302) ok "merchant /terms/" "HTTP $code" ;;
|
||||
*) warn "merchant /terms/" "HTTP $code (expect 302 → /terms)" ;;
|
||||
esac
|
||||
|
||||
summary
|
||||
7
scripts/taler-monitoring/harness-live.txt
Normal file
7
scripts/taler-monitoring/harness-live.txt
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
# Optional: taler-harness against *our* exchange (needs a harness version
|
||||
# that accepts current /config JSON). Older installs often FAIL codec checks
|
||||
# even when the exchange is fine — trust `./taler-monitoring.sh urls` first.
|
||||
#
|
||||
# id<TAB>command
|
||||
|
||||
lint-exchange-goa taler-harness deployment lint-exchange-url https://exchange.hacktivism.ch/
|
||||
41
scripts/taler-monitoring/harness-tests.txt
Normal file
41
scripts/taler-monitoring/harness-tests.txt
Normal file
|
|
@ -0,0 +1,41 @@
|
|||
# Real-life–relevant taler-harness integration tests
|
||||
# (from: taler-harness list-integrationtests)
|
||||
#
|
||||
# Criteria: everyday user/merchant/bank paths — withdraw, pay, deposit, refund,
|
||||
# libeufin bank, templates/paywall. Excludes KYC edge cases, experimental,
|
||||
# timetravel, perf, backup, mailbox, most fault-injection.
|
||||
#
|
||||
# These spin up a local TESTKUDOS stack (same client code paths as production).
|
||||
# For tests against *public* demos, see harness-live.txt / `./taler-monitoring.sh live`.
|
||||
|
||||
# --- Core path: get money + spend it (the GOA story) ---
|
||||
simple-payment
|
||||
withdrawal-bank-integrated
|
||||
payment
|
||||
|
||||
# --- Withdraw variants users actually hit ---
|
||||
withdrawal-manual
|
||||
withdrawal-external
|
||||
withdrawal-idempotent
|
||||
|
||||
# --- Pay variants shops use ---
|
||||
payment-template
|
||||
paywall-flow
|
||||
payment-idempotency
|
||||
payment-zero
|
||||
payment-abort
|
||||
otp
|
||||
|
||||
# --- After pay: coins / merchant ops ---
|
||||
deposit
|
||||
refund
|
||||
refund-auto
|
||||
wallet-refresh
|
||||
|
||||
# --- Bank stack we run (libeufin / regional) ---
|
||||
libeufin-bank
|
||||
bank-api
|
||||
|
||||
# --- Client hygiene ---
|
||||
wallet-config
|
||||
term-of-service-format
|
||||
536
scripts/taler-monitoring/lib.sh
Executable file
536
scripts/taler-monitoring/lib.sh
Executable file
|
|
@ -0,0 +1,536 @@
|
|||
# shellcheck shell=bash
|
||||
# Shared helpers for taler-monitoring (laptop or koopa).
|
||||
|
||||
# Default stack = GOA / hacktivism (overridden by TALER_DOMAIN / --domain)
|
||||
: "${TALER_DOMAIN:=hacktivism.ch}"
|
||||
: "${BANK_PUBLIC:=https://bank.hacktivism.ch}"
|
||||
: "${EXCHANGE_PUBLIC:=https://exchange.hacktivism.ch}"
|
||||
: "${MERCHANT_PUBLIC:=https://taler.hacktivism.ch}"
|
||||
: "${BANK_LOCAL:=http://127.0.0.1:9012}"
|
||||
: "${EXCHANGE_LOCAL:=http://127.0.0.1:9011}"
|
||||
: "${MERCHANT_LOCAL:=https://127.0.0.1:9010}"
|
||||
: "${LANDING_LOCAL:=http://127.0.0.1:9013}"
|
||||
: "${KOOPA_SSH:=koopa}"
|
||||
: "${MERCHANT_INSTANCE:=goa-demo-cp4zqk}"
|
||||
: "${WITHDRAW_AMT:=GOA:20}" # single-shot fallback; e2e ladder uses ATM notes
|
||||
: "${PAY_AMT:=GOA:0.01}"
|
||||
: "${CREDIT_AMT:=GOA:400}" # covers ATM ladder 20+50+100+200
|
||||
: "${TIMEOUT:=12}"
|
||||
: "${E2E_TIMEOUT:=55}" # whole e2e budget; skip rest when exceeded
|
||||
: "${E2E_PAY_SECS:=22}" # dedicated seconds for pay handle-uri (avoid Alarm clock)
|
||||
# Devtest: inject reserve credit via wire-gateway admin/add-incoming (optional).
|
||||
# Default off once wirewatch DNS works; set E2E_FAKE_INCOMING=1 to force.
|
||||
: "${E2E_FAKE_INCOMING:=0}"
|
||||
# SSH must never hang the monitoring run
|
||||
: "${SSH_CONNECT_TIMEOUT:=3}"
|
||||
: "${SSH_CMD_TIMEOUT:=12}" # hard cap for whole remote script (seconds)
|
||||
: "${SKIP_SSH:=0}"
|
||||
# Expected currency for public /config checks (empty = report only, don't fail)
|
||||
: "${EXPECT_CURRENCY:=GOA}"
|
||||
# 1 = this is the local koopa/hacktivism stack (inside/e2e/SSH make sense)
|
||||
: "${LOCAL_STACK:=1}"
|
||||
# Probe merchant host candidates when applying a generic domain (0=off)
|
||||
: "${TALER_DOMAIN_PROBE:=1}"
|
||||
|
||||
BANK_PUBLIC=${BANK_PUBLIC%/}
|
||||
EXCHANGE_PUBLIC=${EXCHANGE_PUBLIC%/}
|
||||
MERCHANT_PUBLIC=${MERCHANT_PUBLIC%/}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Domain presets → public bank / exchange / merchant base URLs
|
||||
#
|
||||
# TALER_DOMAIN=hacktivism.ch (default, GOA, local stack)
|
||||
# TALER_DOMAIN=taler.net → demo.taler.net (KUDOS)
|
||||
# TALER_DOMAIN=demo.taler.net
|
||||
# TALER_DOMAIN=taler-ops.ch → exchange.taler-ops.ch (CHF; bank/merchant if up)
|
||||
# TALER_DOMAIN=example.org → bank/exchange/backend|taler|merchant.example.org
|
||||
#
|
||||
# Explicit BANK_PUBLIC / EXCHANGE_PUBLIC / MERCHANT_PUBLIC still win if set
|
||||
# *after* apply_taler_domain, or pass full URLs via --bank/--exchange/--merchant.
|
||||
# ---------------------------------------------------------------------------
|
||||
_normalize_domain() {
|
||||
local d="$1"
|
||||
d="${d#https://}"
|
||||
d="${d#http://}"
|
||||
d="${d%%/*}"
|
||||
d="${d%%:*}"
|
||||
# Strip service host prefix only if a real domain remains (has a dot).
|
||||
# e.g. bank.demo.taler.net → demo.taler.net, taler.hacktivism.ch → hacktivism.ch
|
||||
# but NOT taler.net → net
|
||||
case "$d" in
|
||||
bank.*|exchange.*|taler.*|backend.*|merchant.*|shop.*|libeufin.*)
|
||||
rest="${d#*.}"
|
||||
if [[ "$rest" == *.* ]]; then
|
||||
d="$rest"
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
printf '%s' "$d"
|
||||
}
|
||||
|
||||
_probe_https_config() {
|
||||
# 0 if https://$1/config returns 200
|
||||
local host="$1" code
|
||||
code=$(curl -skS --max-redirs 0 -m 4 -o /dev/null -w '%{http_code}' "https://${host}/config" 2>/dev/null || echo 000)
|
||||
[ "$code" = "200" ]
|
||||
}
|
||||
|
||||
apply_taler_domain() {
|
||||
local raw="${1:-}"
|
||||
local d
|
||||
[ -n "$raw" ] || return 0
|
||||
d=$(_normalize_domain "$raw")
|
||||
TALER_DOMAIN="$d"
|
||||
|
||||
case "$d" in
|
||||
# Local koopa stack only — SSH / inside / e2e allowed
|
||||
koopa|hacktivism.ch|hacktivism)
|
||||
BANK_PUBLIC="https://bank.hacktivism.ch"
|
||||
EXCHANGE_PUBLIC="https://exchange.hacktivism.ch"
|
||||
MERCHANT_PUBLIC="https://taler.hacktivism.ch"
|
||||
EXPECT_CURRENCY="GOA"
|
||||
LOCAL_STACK=1
|
||||
SKIP_SSH=0
|
||||
TALER_DOMAIN="hacktivism.ch"
|
||||
: "${WITHDRAW_AMT:=GOA:20}"
|
||||
: "${PAY_AMT:=GOA:0.01}"
|
||||
: "${CREDIT_AMT:=GOA:400}"
|
||||
;;
|
||||
taler.net|demo.taler.net)
|
||||
# Official public demo (KUDOS) — public only, never SSH; tiny e2e amounts
|
||||
BANK_PUBLIC="https://bank.demo.taler.net"
|
||||
EXCHANGE_PUBLIC="https://exchange.demo.taler.net"
|
||||
MERCHANT_PUBLIC="https://backend.demo.taler.net"
|
||||
EXPECT_CURRENCY="KUDOS"
|
||||
LOCAL_STACK=0
|
||||
SKIP_SSH=1
|
||||
MERCHANT_INSTANCE="${MERCHANT_INSTANCE:-sandbox}"
|
||||
WITHDRAW_AMT="${WITHDRAW_AMT:-KUDOS:20}"
|
||||
PAY_AMT="${PAY_AMT:-KUDOS:0.01}"
|
||||
CREDIT_AMT="${CREDIT_AMT:-KUDOS:100}"
|
||||
TALER_DOMAIN="demo.taler.net"
|
||||
;;
|
||||
taler-ops.ch)
|
||||
# Public CHF exchange; bank/merchant hosts vary — probe common names
|
||||
EXCHANGE_PUBLIC="https://exchange.taler-ops.ch"
|
||||
BANK_PUBLIC="https://bank.taler-ops.ch"
|
||||
MERCHANT_PUBLIC="https://backend.taler-ops.ch"
|
||||
EXPECT_CURRENCY="CHF"
|
||||
LOCAL_STACK=0
|
||||
SKIP_SSH=1
|
||||
WITHDRAW_AMT="${WITHDRAW_AMT:-CHF:20}"
|
||||
PAY_AMT="${PAY_AMT:-CHF:0.01}"
|
||||
CREDIT_AMT="${CREDIT_AMT:-CHF:100}"
|
||||
if [ "${TALER_DOMAIN_PROBE}" = "1" ]; then
|
||||
local h
|
||||
for h in bank.taler-ops.ch bank.demo.taler-ops.ch; do
|
||||
_probe_https_config "$h" && { BANK_PUBLIC="https://$h"; break; }
|
||||
done
|
||||
for h in backend.taler-ops.ch merchant.taler-ops.ch taler.taler-ops.ch shop.taler-ops.ch; do
|
||||
_probe_https_config "$h" && { MERCHANT_PUBLIC="https://$h"; break; }
|
||||
done
|
||||
fi
|
||||
;;
|
||||
*)
|
||||
# Any other domain — public HTTPS only, never SSH to koopa
|
||||
BANK_PUBLIC="https://bank.${d}"
|
||||
EXCHANGE_PUBLIC="https://exchange.${d}"
|
||||
MERCHANT_PUBLIC="https://backend.${d}"
|
||||
EXPECT_CURRENCY="${EXPECT_CURRENCY:-}" # unknown — don't hard-fail currency
|
||||
LOCAL_STACK=0
|
||||
SKIP_SSH=1
|
||||
if [ "${TALER_DOMAIN_PROBE}" = "1" ]; then
|
||||
local h
|
||||
for h in "backend.${d}" "taler.${d}" "merchant.${d}" "shop.${d}"; do
|
||||
_probe_https_config "$h" && { MERCHANT_PUBLIC="https://$h"; break; }
|
||||
done
|
||||
for h in "bank.${d}" "libeufin.${d}"; do
|
||||
_probe_https_config "$h" && { BANK_PUBLIC="https://$h"; break; }
|
||||
done
|
||||
_probe_https_config "exchange.${d}" || true
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
|
||||
BANK_PUBLIC=${BANK_PUBLIC%/}
|
||||
EXCHANGE_PUBLIC=${EXCHANGE_PUBLIC%/}
|
||||
MERCHANT_PUBLIC=${MERCHANT_PUBLIC%/}
|
||||
|
||||
# Hard rule: only the local koopa/hacktivism stack may use SSH
|
||||
if [ "${LOCAL_STACK}" != "1" ]; then
|
||||
SKIP_SSH=1
|
||||
fi
|
||||
}
|
||||
|
||||
# Apply TALER_DOMAIN from env once (CLI exports TALER_DOMAIN_APPLIED=1 after overrides).
|
||||
if [ "${TALER_DOMAIN_APPLIED:-0}" != "1" ] \
|
||||
&& [ -n "${TALER_DOMAIN:-}" ] && [ "${TALER_DOMAIN}" != "hacktivism.ch" ]; then
|
||||
apply_taler_domain "$TALER_DOMAIN"
|
||||
TALER_DOMAIN_APPLIED=1
|
||||
fi
|
||||
|
||||
# Safe SSH: publickey only, short connect, overall alarm so we never block forever.
|
||||
SSH_BASE_OPTS=(
|
||||
-o BatchMode=yes
|
||||
-o ConnectTimeout="${SSH_CONNECT_TIMEOUT}"
|
||||
-o ConnectionAttempts=1
|
||||
-o ServerAliveInterval=3
|
||||
-o ServerAliveCountMax=2
|
||||
-o StrictHostKeyChecking=accept-new
|
||||
-o PreferredAuthentications=publickey
|
||||
-o PasswordAuthentication=no
|
||||
-o KbdInteractiveAuthentication=no
|
||||
-o GSSAPIAuthentication=no
|
||||
-o NumberOfPasswordPrompts=0
|
||||
)
|
||||
|
||||
# Hard wall-clock timeout so ssh/curl never block the monitoring run forever.
|
||||
with_timeout() {
|
||||
local secs="$1"; shift
|
||||
if command -v gtimeout >/dev/null 2>&1; then
|
||||
gtimeout --kill-after=2 "$secs" "$@"
|
||||
return $?
|
||||
fi
|
||||
if command -v timeout >/dev/null 2>&1; then
|
||||
timeout -k 2 "$secs" "$@" 2>/dev/null || timeout --kill-after=2 "$secs" "$@"
|
||||
return $?
|
||||
fi
|
||||
# Portable: perl alarm + process group kill
|
||||
perl -e '
|
||||
use strict; use warnings;
|
||||
my $secs = shift @ARGV;
|
||||
my $pid = fork();
|
||||
die "fork: $!" unless defined $pid;
|
||||
if ($pid == 0) {
|
||||
setpgrp(0, 0);
|
||||
exec @ARGV;
|
||||
exit 127;
|
||||
}
|
||||
$SIG{ALRM} = sub {
|
||||
kill "TERM", -$pid;
|
||||
select(undef, undef, undef, 1.0);
|
||||
kill "KILL", -$pid;
|
||||
exit 124;
|
||||
};
|
||||
alarm $secs;
|
||||
waitpid($pid, 0);
|
||||
my $code = $? >> 8;
|
||||
alarm 0;
|
||||
exit $code;
|
||||
' "$secs" "$@"
|
||||
}
|
||||
|
||||
# Probe: 0 if koopa SSH works quickly
|
||||
koopa_ssh_ok() {
|
||||
[ "${SKIP_SSH}" = "1" ] && return 1
|
||||
with_timeout $((SSH_CONNECT_TIMEOUT + 3)) \
|
||||
ssh "${SSH_BASE_OPTS[@]}" "${KOOPA_SSH}" 'echo ok' >/dev/null 2>&1
|
||||
}
|
||||
|
||||
# Run remote bash -s with optional stdin script; hard-capped
|
||||
# usage: koopa_ssh_bash [timeout_secs] <<'EOF' ... EOF
|
||||
# or: koopa_ssh_run timeout_secs 'remote command'
|
||||
koopa_ssh_run() {
|
||||
local t="${1:-$SSH_CMD_TIMEOUT}"
|
||||
shift
|
||||
with_timeout "$t" ssh "${SSH_BASE_OPTS[@]}" "${KOOPA_SSH}" "$@"
|
||||
}
|
||||
|
||||
koopa_ssh_bash() {
|
||||
local t="${1:-$SSH_CMD_TIMEOUT}"
|
||||
with_timeout "$t" ssh "${SSH_BASE_OPTS[@]}" "${KOOPA_SSH}" 'bash -s'
|
||||
}
|
||||
|
||||
if [ "${NO_COLOR:-0}" = "1" ] || [ ! -t 1 ]; then
|
||||
G= R= Y= C= N= B=
|
||||
else
|
||||
G=$'\e[32m'; R=$'\e[31m'; Y=$'\e[33m'; C=$'\e[36m'; B=$'\e[1m'; N=$'\e[0m'
|
||||
fi
|
||||
|
||||
PASS_N=0
|
||||
FAIL_N=0
|
||||
WARN_N=0
|
||||
INFO_N=0
|
||||
BLOCKERS=() # human-readable payment/withdraw blockers
|
||||
ERRORS=() # all ERROR lines (component scope)
|
||||
|
||||
# Test IDs by area: www-001, e2e-001, inside-001, …
|
||||
# Usage: set_area www then each ok/fail/warn/info/blocker/err auto-numbers.
|
||||
TEST_AREA=""
|
||||
TEST_N=0
|
||||
# Last issued id (www-001); set by _take_tid — not via $(…) so TEST_N persists.
|
||||
LAST_TID=""
|
||||
set_area() {
|
||||
TEST_AREA="$1"
|
||||
TEST_N=0
|
||||
LAST_TID=""
|
||||
}
|
||||
# Assign next id into LAST_TID (must not run in a subshell).
|
||||
_take_tid() {
|
||||
LAST_TID=""
|
||||
if [ -z "${TEST_AREA:-}" ]; then
|
||||
return
|
||||
fi
|
||||
TEST_N=$((TEST_N + 1))
|
||||
LAST_TID=$(printf '%s-%03d' "$TEST_AREA" "$TEST_N")
|
||||
}
|
||||
_fmt_tid() {
|
||||
# prefix "www-001 " or empty
|
||||
if [ -n "${LAST_TID:-}" ]; then
|
||||
printf '%s ' "$LAST_TID"
|
||||
fi
|
||||
}
|
||||
|
||||
ok() {
|
||||
local label="$1"
|
||||
_take_tid
|
||||
printf '%s[OK]%s %s%s\n' "$G" "$N" "$(_fmt_tid)" "$label"
|
||||
PASS_N=$((PASS_N + 1))
|
||||
}
|
||||
# component-scoped error: err bank "libeufin down" "detail"
|
||||
err() {
|
||||
local comp="$1" msg="$2" detail="${3:-}"
|
||||
_take_tid
|
||||
printf '%s[ERROR]%s %s%s: %s%s\n' "$R" "$N" "$(_fmt_tid)" "$comp" "$msg" "${detail:+ — $detail}"
|
||||
FAIL_N=$((FAIL_N + 1))
|
||||
ERRORS+=("${LAST_TID:+$LAST_TID }[$comp] $msg${detail:+ — $detail}")
|
||||
}
|
||||
# legacy fail label ...
|
||||
fail() {
|
||||
local label="$1" detail="${2:-}"
|
||||
_take_tid
|
||||
printf '%s[ERROR]%s %s%s%s\n' "$R" "$N" "$(_fmt_tid)" "$label" "${detail:+ — $detail}"
|
||||
FAIL_N=$((FAIL_N + 1))
|
||||
ERRORS+=("${LAST_TID:+$LAST_TID }$label${detail:+ — $detail}")
|
||||
}
|
||||
warn() {
|
||||
local label="$1" detail="${2:-}"
|
||||
_take_tid
|
||||
printf '%s[WARN]%s %s%s%s\n' "$Y" "$N" "$(_fmt_tid)" "$label" "${detail:+ — $detail}"
|
||||
WARN_N=$((WARN_N + 1))
|
||||
}
|
||||
info() {
|
||||
local label="$1" detail="${2:-}"
|
||||
_take_tid
|
||||
if [ -n "$detail" ]; then
|
||||
printf '%s[INFO]%s %s%s — %s\n' "$C" "$N" "$(_fmt_tid)" "$label" "$detail"
|
||||
else
|
||||
printf '%s[INFO]%s %s%s\n' "$C" "$N" "$(_fmt_tid)" "$label"
|
||||
fi
|
||||
INFO_N=$((INFO_N + 1))
|
||||
}
|
||||
blocker() {
|
||||
# Payment/withdraw path cannot proceed because of this
|
||||
local step="$1" msg="$2"
|
||||
_take_tid
|
||||
printf '%s[BLOCKER]%s %s%s: %s\n' "$R$B" "$N" "$(_fmt_tid)" "$step" "$msg"
|
||||
BLOCKERS+=("${LAST_TID:+$LAST_TID }[$step] $msg")
|
||||
FAIL_N=$((FAIL_N + 1))
|
||||
ERRORS+=("BLOCKER ${LAST_TID:+$LAST_TID }[$step] $msg")
|
||||
}
|
||||
section() { printf '\n%s== %s ==%s\n' "$B" "$*" "$N"; }
|
||||
|
||||
summary() {
|
||||
echo ""
|
||||
if [ "${#BLOCKERS[@]}" -gt 0 ]; then
|
||||
printf '%s--- BLOCKERS (fix/withdraw path) ---%s\n' "$R$B" "$N"
|
||||
local b
|
||||
for b in "${BLOCKERS[@]}"; do
|
||||
printf '%s • %s%s\n' "$R" "$b" "$N"
|
||||
done
|
||||
fi
|
||||
if [ "${#ERRORS[@]}" -gt 0 ] && [ "${#BLOCKERS[@]}" -lt "${#ERRORS[@]}" ]; then
|
||||
printf '%s--- ERRORS ---%s\n' "$R" "$N"
|
||||
local e
|
||||
for e in "${ERRORS[@]}"; do
|
||||
case "$e" in BLOCKER*) continue ;; esac
|
||||
printf '%s • %s%s\n' "$R" "$e" "$N"
|
||||
done
|
||||
fi
|
||||
printf 'totals: %s%d OK%s' "$G" "$PASS_N" "$N"
|
||||
[ "$FAIL_N" -gt 0 ] && printf ', %s%d ERROR%s' "$R" "$FAIL_N" "$N"
|
||||
[ "$WARN_N" -gt 0 ] && printf ', %s%d WARN%s' "$Y" "$WARN_N" "$N"
|
||||
[ "$INFO_N" -gt 0 ] && printf ', %d INFO' "$INFO_N"
|
||||
[ "${#BLOCKERS[@]}" -gt 0 ] && printf ', %s%d BLOCKER%s' "$R" "${#BLOCKERS[@]}" "$N"
|
||||
printf '\n'
|
||||
[ "$FAIL_N" -eq 0 ]
|
||||
}
|
||||
|
||||
http_code() {
|
||||
local url="$1"; shift
|
||||
curl -skS --max-redirs 0 -m "${TIMEOUT}" -o /dev/null -w '%{http_code}' "$@" "$url" 2>/dev/null || echo 000
|
||||
}
|
||||
|
||||
http_body() {
|
||||
local url="$1" out="$2"; shift 2
|
||||
curl -skS --max-redirs 0 -m "${TIMEOUT}" -o "$out" -w '%{http_code}' "$@" "$url" 2>/dev/null || echo 000
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Currency unit map checks (wallet codec: alt_unit_names must include "0")
|
||||
# ---------------------------------------------------------------------------
|
||||
# Returns 0 if JSON body at $1 has usable alt_unit_names.
|
||||
# Supports:
|
||||
# - exchange/bank: currency_specification.alt_unit_names
|
||||
# - merchant: currencies.<CODE>.alt_unit_names for each code
|
||||
# Optional $2 = required currency code for currency_specification.currency
|
||||
json_has_alt_unit_names() {
|
||||
local file="$1" want_cur="${2:-}"
|
||||
python3 - "$file" "$want_cur" <<'PY'
|
||||
import json, sys
|
||||
path, want = sys.argv[1], sys.argv[2]
|
||||
try:
|
||||
d = json.load(open(path))
|
||||
except Exception as e:
|
||||
print(f"json-error: {e}")
|
||||
sys.exit(2)
|
||||
|
||||
def check_au(au, label):
|
||||
if not isinstance(au, dict) or not au:
|
||||
print(f"{label}: missing/empty alt_unit_names")
|
||||
return False
|
||||
if "0" not in au or not str(au.get("0") or "").strip():
|
||||
print(f"{label}: alt_unit_names missing non-empty key \"0\" (have {sorted(au.keys())})")
|
||||
return False
|
||||
print(f"{label}: alt_unit_names ok (0={au.get('0')!r}, n={len(au)})")
|
||||
return True
|
||||
|
||||
ok = True
|
||||
cs = d.get("currency_specification")
|
||||
if isinstance(cs, dict):
|
||||
if want and cs.get("currency") and cs.get("currency") != want:
|
||||
print(f"currency_specification.currency={cs.get('currency')!r} want {want!r}")
|
||||
ok = False
|
||||
if not check_au(cs.get("alt_unit_names"), "currency_specification"):
|
||||
ok = False
|
||||
elif "currency_specification" in d:
|
||||
print("currency_specification: not an object")
|
||||
ok = False
|
||||
|
||||
curs = d.get("currencies")
|
||||
if isinstance(curs, dict) and curs:
|
||||
for code, spec in curs.items():
|
||||
if not isinstance(spec, dict):
|
||||
print(f"currencies.{code}: not an object")
|
||||
ok = False
|
||||
continue
|
||||
if not check_au(spec.get("alt_unit_names"), f"currencies.{code}"):
|
||||
ok = False
|
||||
|
||||
if not isinstance(cs, dict) and not (isinstance(curs, dict) and curs):
|
||||
# neither shape — fail
|
||||
print("no currency_specification or currencies map")
|
||||
ok = False
|
||||
|
||||
sys.exit(0 if ok else 1)
|
||||
PY
|
||||
}
|
||||
|
||||
# Check one exchange base URL's /config for alt_unit_names.
|
||||
# $1=label $2=base_url $3=expected currency (optional) $4=strict(1) or soft(0)
|
||||
check_exchange_alt_units() {
|
||||
local label="$1" base="$2" want_cur="${3:-}" strict="${4:-1}"
|
||||
local f code
|
||||
base="${base%/}"
|
||||
f=$(mktemp)
|
||||
code=$(http_body "${base}/config" "$f")
|
||||
if [ "$code" != "200" ]; then
|
||||
rm -f "$f"
|
||||
if [ "$strict" = "1" ]; then
|
||||
fail "$label /config" "HTTP $code ($base)"
|
||||
else
|
||||
warn "$label /config" "HTTP $code ($base)"
|
||||
fi
|
||||
return
|
||||
fi
|
||||
local out ec
|
||||
set +e
|
||||
out=$(json_has_alt_unit_names "$f" "$want_cur" 2>&1)
|
||||
ec=$?
|
||||
set -e
|
||||
rm -f "$f"
|
||||
if [ "$ec" -eq 0 ]; then
|
||||
ok "$label alt_unit_names" "$(echo "$out" | tr '\n' '; ' | sed 's/; $//')"
|
||||
else
|
||||
if [ "$strict" = "1" ]; then
|
||||
fail "$label alt_unit_names" "$(echo "$out" | tr '\n' '; ' | sed 's/; $//')"
|
||||
else
|
||||
warn "$label alt_unit_names" "$(echo "$out" | tr '\n' '; ' | sed 's/; $//')"
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
# From merchant /config JSON file: walk exchanges[] and check each base_url/config.
|
||||
# Local stack hosts (hacktivism.ch or $EXCHANGE_PUBLIC host) are strict; others soft.
|
||||
check_merchant_listed_exchanges_alt_units() {
|
||||
local mer_json="$1"
|
||||
local list
|
||||
list=$(python3 - "$mer_json" <<'PY'
|
||||
import json, sys
|
||||
d = json.load(open(sys.argv[1]))
|
||||
for e in d.get("exchanges") or []:
|
||||
if not isinstance(e, dict):
|
||||
continue
|
||||
u = (e.get("base_url") or e.get("url") or "").rstrip("/")
|
||||
c = e.get("currency") or ""
|
||||
if u:
|
||||
print(f"{c}\t{u}")
|
||||
PY
|
||||
)
|
||||
if [ -z "$list" ]; then
|
||||
fail "merchant exchanges[]" "empty — no exchanges to check for alt_unit_names"
|
||||
return
|
||||
fi
|
||||
local line cur url strict host
|
||||
while IFS=$'\t' read -r cur url; do
|
||||
[ -n "$url" ] || continue
|
||||
host="${url#https://}"; host="${host#http://}"; host="${host%%/*}"
|
||||
strict=1
|
||||
case "$host" in
|
||||
*hacktivism.ch) strict=1 ;;
|
||||
*)
|
||||
# foreign exchange (e.g. taler-ops) — soft unless it is our configured EXCHANGE_PUBLIC
|
||||
if [ "$url" = "${EXCHANGE_PUBLIC}" ] || [ "$url" = "${EXCHANGE_PUBLIC}/" ]; then
|
||||
strict=1
|
||||
else
|
||||
strict=0
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
check_exchange_alt_units "exchange ${cur:-?} ${host}" "$url" "$cur" "$strict"
|
||||
done <<<"$list"
|
||||
}
|
||||
|
||||
SECRETS_ROOT="${SECRETS_ROOT:-}"
|
||||
if [ -z "$SECRETS_ROOT" ]; then
|
||||
for d in \
|
||||
"$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)/koopa-admin-secrets/koopa/host-root" \
|
||||
"/Users/newkamek/src/koopa/koopa-admin-secrets/koopa/host-root" \
|
||||
"$HOME/src/koopa/koopa-admin-secrets/koopa/host-root"
|
||||
do
|
||||
if [ -d "$d/taler-bank" ]; then SECRETS_ROOT=$d; break; fi
|
||||
done
|
||||
fi
|
||||
|
||||
read_secret() {
|
||||
local rel="$1"
|
||||
if [ -n "${SECRETS_ROOT:-}" ] && [ -f "${SECRETS_ROOT}/${rel}" ]; then
|
||||
tr -d '\n' <"${SECRETS_ROOT}/${rel}"
|
||||
return 0
|
||||
fi
|
||||
koopa_ssh_ok || return 1
|
||||
koopa_ssh_run 10 "tr -d '\\n' </root/$(basename "$rel") 2>/dev/null || true" 2>/dev/null
|
||||
}
|
||||
|
||||
find_wallet_cli() {
|
||||
if [ -n "${WALLET_CLI:-}" ] && [ -f "$WALLET_CLI" ]; then
|
||||
echo "$WALLET_CLI"; return 0
|
||||
fi
|
||||
for c in \
|
||||
/Users/newkamek/src/taler/taler-typescript-core/packages/taler-wallet-cli/bin/taler-wallet-cli.mjs \
|
||||
"$(command -v taler-wallet-cli 2>/dev/null || true)"
|
||||
do
|
||||
[ -n "$c" ] && [ -f "$c" ] && { echo "$c"; return 0; }
|
||||
done
|
||||
return 1
|
||||
}
|
||||
198
scripts/taler-monitoring/taler-monitoring.sh
Executable file
198
scripts/taler-monitoring/taler-monitoring.sh
Executable file
|
|
@ -0,0 +1,198 @@
|
|||
#!/usr/bin/env bash
|
||||
# taler-monitoring — public URL / stack checks for a Taler domain
|
||||
#
|
||||
# ./taler-monitoring.sh # local GOA (urls + inside + e2e)
|
||||
# ./taler-monitoring.sh -d taler.net urls # public demo, no SSH
|
||||
# ./taler-monitoring.sh --domain taler-ops.ch # public ops, no SSH
|
||||
# TALER_DOMAIN=demo.taler.net ./taler-monitoring.sh urls
|
||||
#
|
||||
# Tags: [OK] [INFO] [WARN] [ERROR] [BLOCKER]
|
||||
# Exit 0 only if every selected phase exits 0.
|
||||
|
||||
set -euo pipefail
|
||||
ROOT=$(cd "$(dirname "$0")" && pwd)
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
taler-monitoring — bank / exchange / merchant checks
|
||||
|
||||
Usage:
|
||||
./taler-monitoring.sh [options] [phases...]
|
||||
|
||||
Phases:
|
||||
urls public HTTPS (no SSH) ← default with --domain
|
||||
inside container status via SSH (local stack only)
|
||||
versions taler packages vs deb.taler.net trixie + repo availability
|
||||
sanity public + optional server
|
||||
server server-side only (SSH)
|
||||
e2e withdraw + pay (small amounts; remote aborts on login/KYC)
|
||||
all urls + inside + versions + sanity + e2e (SSH phases only on koopa)
|
||||
|
||||
Options:
|
||||
-d, --domain DOMAIN target domain
|
||||
koopa / hacktivism.ch → local stack, SSH ok
|
||||
everything else → no SSH; e2e optional (tiny amounts)
|
||||
presets: koopa | hacktivism.ch | taler.net | taler-ops.ch
|
||||
generic: bank/exchange/backend.<domain>
|
||||
--bank URL override bank base (https://…)
|
||||
--exchange URL override exchange base
|
||||
--merchant URL override merchant base
|
||||
--currency CODE expected currency (GOA, KUDOS, CHF, …); empty = report only
|
||||
--no-probe do not probe alternate merchant/bank hostnames
|
||||
-h, --help
|
||||
|
||||
Examples:
|
||||
./taler-monitoring.sh -d taler.net
|
||||
./taler-monitoring.sh -d taler-ops.ch urls
|
||||
./taler-monitoring.sh -d demo.taler.net --currency KUDOS
|
||||
./taler-monitoring.sh --exchange https://exchange.taler-ops.ch urls
|
||||
|
||||
Env (same meaning):
|
||||
TALER_DOMAIN BANK_PUBLIC EXCHANGE_PUBLIC MERCHANT_PUBLIC EXPECT_CURRENCY
|
||||
SKIP_SSH=1 NO_COLOR=1
|
||||
EOF
|
||||
}
|
||||
|
||||
# shellcheck source=lib.sh
|
||||
source "$ROOT/lib.sh"
|
||||
|
||||
PHASES=()
|
||||
DOMAIN_SET=0
|
||||
BANK_OVERRIDE=""
|
||||
EXCHANGE_OVERRIDE=""
|
||||
MERCHANT_OVERRIDE=""
|
||||
CURRENCY_OVERRIDE=""
|
||||
NO_PROBE=0
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
-h|--help) usage; exit 0 ;;
|
||||
-d|--domain)
|
||||
[ $# -ge 2 ] || { echo "missing arg for $1" >&2; exit 2; }
|
||||
apply_taler_domain "$2"
|
||||
DOMAIN_SET=1
|
||||
shift 2
|
||||
;;
|
||||
--bank)
|
||||
[ $# -ge 2 ] || { echo "missing arg for $1" >&2; exit 2; }
|
||||
BANK_OVERRIDE="${2%/}"; shift 2
|
||||
;;
|
||||
--exchange)
|
||||
[ $# -ge 2 ] || { echo "missing arg for $1" >&2; exit 2; }
|
||||
EXCHANGE_OVERRIDE="${2%/}"; shift 2
|
||||
;;
|
||||
--merchant)
|
||||
[ $# -ge 2 ] || { echo "missing arg for $1" >&2; exit 2; }
|
||||
MERCHANT_OVERRIDE="${2%/}"; shift 2
|
||||
;;
|
||||
--currency)
|
||||
[ $# -ge 2 ] || { echo "missing arg for $1" >&2; exit 2; }
|
||||
CURRENCY_OVERRIDE="$2"; shift 2
|
||||
;;
|
||||
--no-probe) NO_PROBE=1; shift ;;
|
||||
urls|inside|versions|sanity|server|e2e|all) PHASES+=("$1"); shift ;;
|
||||
*)
|
||||
# bare domain shorthand: ./taler-monitoring.sh taler.net
|
||||
if [[ "$1" == *.* && "$1" != *://* && "$1" != -* ]]; then
|
||||
apply_taler_domain "$1"
|
||||
DOMAIN_SET=1
|
||||
shift
|
||||
else
|
||||
echo "Unknown: $1" >&2; usage >&2; exit 2
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [ "$NO_PROBE" = "1" ]; then
|
||||
TALER_DOMAIN_PROBE=0
|
||||
fi
|
||||
if [ -n "$CURRENCY_OVERRIDE" ]; then
|
||||
EXPECT_CURRENCY="$CURRENCY_OVERRIDE"
|
||||
fi
|
||||
if [ -n "$BANK_OVERRIDE" ]; then BANK_PUBLIC="$BANK_OVERRIDE"; fi
|
||||
if [ -n "$EXCHANGE_OVERRIDE" ]; then EXCHANGE_PUBLIC="$EXCHANGE_OVERRIDE"; fi
|
||||
if [ -n "$MERCHANT_OVERRIDE" ]; then MERCHANT_PUBLIC="$MERCHANT_OVERRIDE"; fi
|
||||
|
||||
# Only koopa may use SSH. Remote domains: public + optional e2e (no SSH).
|
||||
if [ "${LOCAL_STACK:-1}" != "1" ]; then
|
||||
SKIP_SSH=1
|
||||
fi
|
||||
|
||||
# Remote: no SSH; ATM withdraw ladder set in check_e2e (smaller notes)
|
||||
if [ "${LOCAL_STACK}" != "1" ]; then
|
||||
E2E_FAKE_INCOMING=0
|
||||
E2E_REMOTE=1
|
||||
: "${E2E_WITHDRAW_VALUES:=10 20 50}"
|
||||
: "${E2E_PAY_VALUES:=0.01 0.05 0.1 1}"
|
||||
fi
|
||||
|
||||
# Export so check_*.sh (re-source lib) see the same targets via env
|
||||
export TALER_DOMAIN BANK_PUBLIC EXCHANGE_PUBLIC MERCHANT_PUBLIC
|
||||
export EXPECT_CURRENCY SKIP_SSH LOCAL_STACK TALER_DOMAIN_PROBE
|
||||
export WITHDRAW_AMT PAY_AMT CREDIT_AMT MERCHANT_INSTANCE
|
||||
export E2E_FAKE_INCOMING E2E_REMOTE E2E_VARIABLE E2E_ATM_MAX
|
||||
export E2E_WITHDRAW_VALUES E2E_PAY_VALUES
|
||||
export TALER_DOMAIN_APPLIED=1
|
||||
|
||||
# Default phases
|
||||
if [ "${#PHASES[@]}" -eq 0 ]; then
|
||||
if [ "${LOCAL_STACK}" = "1" ]; then
|
||||
PHASES=(urls inside versions e2e) # koopa: SSH ok
|
||||
else
|
||||
PHASES=(urls) # remote default: public only; pass "e2e" to try
|
||||
fi
|
||||
fi
|
||||
|
||||
OUT_PHASES=()
|
||||
for p in "${PHASES[@]}"; do
|
||||
if [ "$p" = "all" ]; then
|
||||
if [ "${LOCAL_STACK}" = "1" ]; then
|
||||
OUT_PHASES+=(urls inside versions sanity e2e)
|
||||
else
|
||||
# remote: repo index check only (no installed pkgs without SSH)
|
||||
OUT_PHASES+=(urls versions e2e)
|
||||
fi
|
||||
else
|
||||
OUT_PHASES+=("$p")
|
||||
fi
|
||||
done
|
||||
PHASES=()
|
||||
seen=" "
|
||||
for p in "${OUT_PHASES[@]}"; do
|
||||
# Drop SSH-only phases for non-koopa (e2e is allowed — public path only)
|
||||
# versions still runs: outside-in deb.taler.net checks; container parts soft-skip if no SSH
|
||||
if [ "${LOCAL_STACK}" != "1" ] || [ "${SKIP_SSH}" = "1" ]; then
|
||||
case "$p" in inside|server)
|
||||
echo "[INFO] skip phase '$p' (only koopa uses SSH)" >&2
|
||||
continue
|
||||
;;
|
||||
esac
|
||||
fi
|
||||
case "$seen" in *" $p "*) ;; *) PHASES+=("$p"); seen="$seen$p " ;; esac
|
||||
done
|
||||
|
||||
if [ "${#PHASES[@]}" -eq 0 ]; then
|
||||
PHASES=(urls)
|
||||
fi
|
||||
|
||||
printf 'target domain=%s\n' "${TALER_DOMAIN}"
|
||||
printf ' bank %s\n' "$BANK_PUBLIC"
|
||||
printf ' exchange %s\n' "$EXCHANGE_PUBLIC"
|
||||
printf ' merchant %s\n' "$MERCHANT_PUBLIC"
|
||||
printf ' currency expect=%s skip_ssh=%s\n' "${EXPECT_CURRENCY:-any}" "$SKIP_SSH"
|
||||
|
||||
chmod +x "$ROOT"/check_*.sh 2>/dev/null || true
|
||||
|
||||
ec=0
|
||||
for p in "${PHASES[@]}"; do
|
||||
case "$p" in
|
||||
urls) "$ROOT/check_urls.sh" || ec=1 ;;
|
||||
inside) "$ROOT/check_inside.sh" || ec=1 ;;
|
||||
versions) "$ROOT/check_versions.sh" || ec=1 ;;
|
||||
sanity) "$ROOT/check_sanity.sh" || ec=1 ;;
|
||||
server) "$ROOT/check_server.sh" || ec=1 ;;
|
||||
e2e) "$ROOT/check_e2e.sh" || ec=1 ;;
|
||||
esac
|
||||
done
|
||||
exit "$ec"
|
||||
Loading…
Add table
Add a link
Reference in a new issue