diff --git a/scripts/taler-monitoring/README.md b/scripts/taler-monitoring/README.md index c867ada..d8408c8 100644 --- a/scripts/taler-monitoring/README.md +++ b/scripts/taler-monitoring/README.md @@ -94,6 +94,22 @@ E2E maps failures to blockers, e.g.: - `Alarm clock` during pay → usually missing `handle-uri --yes` or wrong pay URI (must be `…/instances/{inst}/{oid}/?c={token}` from merchant `taler_pay_uri`) - `insufficient balance` → withdraw incomplete +## Performance (urls phase) + +Outside-in HTTPS RTT for bank / exchange / merchant critical paths. Each probe prints +`HTTP … · N ms` on the `[OK]`/`[WARN]`/`[ERROR]` line; end of the block prints +`perf summary` (n / min / p50 / avg / max). + +| Env | Default | Meaning | +|-----|---------|---------| +| `PERF_WARN_MS` | `8000` | WARN if latency ≥ this | +| `PERF_FAIL_MS` | `20000` | ERROR if latency ≥ this | +| `PERF_CURL_TIMEOUT` | `25` | curl max seconds per probe | + +```bash +PERF_WARN_MS=3000 PERF_FAIL_MS=10000 ./taler-monitoring.sh urls +``` + ## Needs - SSH `koopa` (inside/sanity server bits) diff --git a/scripts/taler-monitoring/TESTS.md b/scripts/taler-monitoring/TESTS.md index 28af8cc..c9dbe09 100644 --- a/scripts/taler-monitoring/TESTS.md +++ b/scripts/taler-monitoring/TESTS.md @@ -46,15 +46,15 @@ IDs are assigned **in run order** within the area (`set_area` resets the counter | www-… | **landing exposed links** (bank / merchant / exchange): parse each `/intro/` HTML, probe every own-stack `https://` + root-relative `href`/`src`/`content`, soft-check external stores/docs | | www-… | landing static: `qrcode.min.js`, `og-goa-shop.png`, `qr-logo.png`, shop-pay.js/css | | www-… | cross-links between bank ↔ merchant ↔ exchange intros (local stack) | -| www-… | **bank `/intro/demo-withdraw.json`** → `taler://withdraw/HOST:PORT/taler-integration/…` + integration op HTTP 200 | +| www-… | **bank `/intro/demo-withdraw.json`** → `taler://withdraw/HOST/taler-integration/…` (no default `:443`/`:80`; non-default port OK) + integration op HTTP 200 | | www-… | bank `/intro/auto-account.json` (earlier) → same withdraw shape, **no payto_uri**, login at `/webui/` | | www-… | **performance** (outside-in): public HTTPS RTT for bank `/config`, `/taler-integration/config`, `/webui/`, `/intro/`, `stats.json`; exchange `/config`, `/keys`, `/intro/`; merchant `/config`, `/webui/`, `/intro/` — report ms; WARN ≥ `PERF_WARN_MS` (default 8000); **ERROR ≥ `PERF_FAIL_MS` (default 20000)** | **Legal docs rule:** HTTP 200, non-empty body, not plain `not configured`, not merchant API JSON `code:21`. On local stack, optional content needle (terms/privacy/FADP/GOA…). -**Performance rule:** Measured from the **monitoring runner** (public URLs via Caddy), not container loopback. HTTP must match expect (usually 200); latency is reported on the OK line. Slow ≥ `PERF_WARN_MS` → WARN only (no ERROR on slowness alone). +**Performance rule:** Measured from the **monitoring runner** (public URLs via Caddy), not container loopback. HTTP must match expect (usually 200); **latency (ms) is on every perf OK/WARN/ERROR line**, plus a **perf summary** (n / min / p50 / avg / max). Slow ≥ `PERF_WARN_MS` (default 8000) → WARN; ≥ `PERF_FAIL_MS` (default 20000) → ERROR. -**Landing links rule:** Own-stack (bank/exchange/taler.\* + page host) must be HTTP 200 (or redirect→200). External (App Store, Play, F-Droid, wallet.taler.net, docs/git.taler.net, …) soft WARN if down. Auto-account wallet link must be `taler://withdraw/…:port/taler-integration/…`, never payto. +**Landing links rule:** Own-stack (bank/exchange/taler.\* + page host) must be HTTP 200 (or redirect→200). External (App Store, Play, F-Droid, wallet.taler.net, docs/git.taler.net, …) soft WARN if down. Auto-account wallet link must be `taler://withdraw/HOST/taler-integration/…` (default ports stripped for mobile wallets), never payto. **alt_unit_names rule:** wallet codec requires a non-empty map including scale key `"0"`. For multi-currency merchant, also follow every entry in `exchanges[]` and check that exchange’s public `/config`. diff --git a/scripts/taler-monitoring/check_urls.sh b/scripts/taler-monitoring/check_urls.sh index 2f7d1de..b974603 100755 --- a/scripts/taler-monitoring/check_urls.sh +++ b/scripts/taler-monitoring/check_urls.sh @@ -98,6 +98,77 @@ check_legal_doc() { rm -f "$f" } +# Shared shape for bank mint JSON (demo-withdraw + auto-account). +# Args: $1=json file $2=auto|demo +# stdout (ok): one detail line; for demo, optional "\twithdrawal_id" suffix. +# stderr (fail): short reason. Exit 0/1. +validate_bank_withdraw_json() { + python3 - "$1" "$2" <<'PY' +import json, re, sys +from urllib.parse import urlparse + +def check_withdraw_uri(wuri: str): + """HOST or HOST:non-default-port; default :443/:80 must be stripped (wallet fix).""" + m = re.match( + r"^taler://withdraw/([^/]+)/taler-integration/([0-9a-fA-F-]+)$", + wuri or "", + ) + if not m: + print( + "need taler://withdraw/HOST/taler-integration/ID:", + (wuri or "")[:120], + file=sys.stderr, + ) + return None + host = m.group(1) + if not host or host.startswith(":"): + print("bad withdraw host:", host, file=sys.stderr) + return None + if ":" in host: + h, _, p = host.rpartition(":") + if not h or not p.isdigit(): + print("bad withdraw host:port:", host, file=sys.stderr) + return None + if p in ("443", "80"): + print("withdraw must strip default port :%s:" % p, host, file=sys.stderr) + return None + return host, m.group(2), wuri + +path, mode = sys.argv[1], sys.argv[2] +d = json.load(open(path)) +wuri = d.get("taler_withdraw_uri") or ( + d.get("qr_payload") if mode == "auto" else "" +) or "" + +if mode == "auto": + if not d.get("ok"): + print("ok!=true", file=sys.stderr) + sys.exit(1) + if d.get("payto_uri"): + print("payto_uri must not be present", file=sys.stderr) + sys.exit(1) + if not check_withdraw_uri(wuri): + sys.exit(1) + webui = d.get("login_url") or d.get("webui") or d.get("account_url") or "" + u = urlparse(webui) + if u.scheme not in ("http", "https") or "webui" not in (u.path or ""): + print("login webui missing:", webui[:80], file=sys.stderr) + sys.exit(1) + print("%s · %s" % (d.get("username", ""), wuri[:72])) +elif mode == "demo": + if not d.get("ok", True) and "taler_withdraw_uri" not in d: + print("not ok", file=sys.stderr) + sys.exit(1) + if not check_withdraw_uri(wuri): + sys.exit(1) + print("%s\t%s" % (wuri[:80], d.get("withdrawal_id") or "")) +else: + print("bad mode:", mode, file=sys.stderr) + sys.exit(2) +sys.exit(0) +PY +} + # --- exchange (core; always required) --- www-001 … check_url "exchange /config" 200 "$EXCHANGE_PUBLIC/config" code=$(http_body "$EXCHANGE_PUBLIC/config" "$tmp/ec.json") @@ -150,6 +221,8 @@ section "www · performance · public HTTPS latency (outside-in)" # ≥ PERF_WARN_MS → WARN. ≥ PERF_FAIL_MS → ERROR (suite fails). PERF_WARN_MS="${PERF_WARN_MS:-8000}" PERF_FAIL_MS="${PERF_FAIL_MS:-20000}" +PERF_TSV="$tmp/perf.tsv" +: >"$PERF_TSV" # Measure one URL: require HTTP expect (default 200), report time_total in ms. # $1=label $2=url $3=optional expected codes (default 200) @@ -165,6 +238,8 @@ check_perf() { if (ms>0 && ms<1) ms=1 printf "%d", int(ms+0.5) }') + # Always record sample for rollup (label, ms, http, url) + printf '%s\t%s\t%s\t%s\n' "$label" "$ms" "$code" "$url" >>"$PERF_TSV" case ",$expect," in *",$code,"*) if [ "$ms" -ge "${PERF_FAIL_MS}" ] 2>/dev/null; then @@ -202,7 +277,76 @@ check_perf "perf merchant /config" "$MERCHANT_PUBLIC/config" check_perf "perf merchant /webui/" "$MERCHANT_PUBLIC/webui/" 200,301,302 check_perf "perf merchant /intro/" "$MERCHANT_PUBLIC/intro/" -info "perf note" "measured from this host (outside-in); not container loopback" +# Rollup + optional JSON for metrics_print_overall +if [ -s "$PERF_TSV" ]; then + PERF_JSON="$tmp/perf-summary.json" + perf_line=$(python3 - "$PERF_TSV" "$PERF_JSON" "$PERF_WARN_MS" "$PERF_FAIL_MS" <<'PY' +import json, sys +rows = [] +for line in open(sys.argv[1]): + parts = line.rstrip("\n").split("\t") + if len(parts) < 3: + continue + label, ms_s, code = parts[0], parts[1], parts[2] + try: + ms = int(float(ms_s)) + except Exception: + continue + rows.append({"label": label, "ms": ms, "http": code}) +vals = sorted(r["ms"] for r in rows) +n = len(vals) +if n == 0: + print("n=0") + json.dump({"n": 0}, open(sys.argv[2], "w")) + raise SystemExit(0) +def pct(p): + if n == 1: + return vals[0] + i = min(n - 1, max(0, int(round((p / 100.0) * (n - 1))))) + return vals[i] +avg = int(round(sum(vals) / n)) +warn_ms = int(sys.argv[3]) +fail_ms = int(sys.argv[4]) +slow = [r for r in rows if r["ms"] >= warn_ms] +rep = { + "n": n, + "min_ms": vals[0], + "p50_ms": pct(50), + "avg_ms": avg, + "max_ms": vals[-1], + "warn_ms": warn_ms, + "fail_ms": fail_ms, + "slow": [{"label": r["label"], "ms": r["ms"]} for r in slow], + "samples": rows, +} +# metrics_print_overall expects named buckets with n/min/p50/avg/max +out = { + "www_public_https": { + "n": n, + "min_ms": vals[0], + "p50_ms": pct(50), + "avg_ms": avg, + "max_ms": vals[-1], + }, + **{r["label"].replace(" ", "_"): {"n": 1, "min_ms": r["ms"], "p50_ms": r["ms"], "avg_ms": r["ms"], "max_ms": r["ms"]} for r in rows}, +} +json.dump(out, open(sys.argv[2], "w"), indent=2) +extra = "" +if slow: + extra = " slow: " + ", ".join("%s=%dms" % (r["label"].replace("perf ", ""), r["ms"]) for r in slow) +print( + "n=%d min=%dms p50=%dms avg=%dms max=%dms (warn≥%dms fail≥%dms)%s" + % (n, vals[0], pct(50), avg, vals[-1], warn_ms, fail_ms, extra) +) +PY + ) + info "perf summary" "$perf_line" + # Keep a copy if METRICS_DIR is set (e2e/ladder overall stats) + if [ -n "${METRICS_DIR:-}" ] && [ -d "${METRICS_DIR}" ]; then + cp -f "$PERF_JSON" "${METRICS_DIR}/perf-summary.json" 2>/dev/null || true + fi +fi +info "perf note" "measured from this host (outside-in); thresholds PERF_WARN_MS=${PERF_WARN_MS} PERF_FAIL_MS=${PERF_FAIL_MS}" # Terms + privacy (legal docs) @@ -247,31 +391,10 @@ if [ "$code" = "200" ]; then aa_code=$(http_body "$BANK_PUBLIC/intro/auto-account.json" "$tmp/aa.json") case "$aa_code" in 200) - if python3 - "$tmp/aa.json" <<'PY' -import json, re, sys -from urllib.parse import urlparse -d = json.load(open(sys.argv[1])) -if not d.get("ok"): - print("ok!=true"); sys.exit(1) -if "payto_uri" in d and d.get("payto_uri"): - print("payto_uri must not be present"); sys.exit(1) -wuri = d.get("taler_withdraw_uri") or d.get("qr_payload") or "" -wm = re.match(r"^taler://withdraw/([^/]+)/taler-integration/([0-9a-fA-F-]+)$", wuri) -if not wm: - print("need taler://withdraw/HOST:PORT/taler-integration/ID:", wuri[:120]); sys.exit(1) -if ":" not in wm.group(1): - print("withdraw missing port:", wm.group(1)); sys.exit(1) -webui = d.get("login_url") or d.get("webui") or d.get("account_url") or "" -u = urlparse(webui) -if u.scheme not in ("http", "https") or "webui" not in (u.path or ""): - print("login webui missing:", webui[:80]); sys.exit(1) -print("user=%s withdraw=%s login=%s" % (d.get("username"), wm.group(1), webui)) -sys.exit(0) -PY - then - ok "bank /intro/auto-account.json" "$(python3 -c 'import json;d=json.load(open("'"$tmp/aa.json"'"));print(d.get("username",""),"·",(d.get("taler_withdraw_uri") or "")[:72])' 2>/dev/null || true)" + if aa_detail=$(validate_bank_withdraw_json "$tmp/aa.json" auto 2>"$tmp/aa-val"); then + ok "bank /intro/auto-account.json" "$aa_detail" else - fail "bank /intro/auto-account.json" "invalid withdraw/login (HTTP body bad)" + fail "bank /intro/auto-account.json" "invalid withdraw/login ($(tr '\n' ' ' <"$tmp/aa-val" | sed 's/[[:space:]]*$//'))" fi ;; 405|501|404|502|503|000) @@ -591,29 +714,17 @@ if [ "${LOCAL_STACK:-1}" = "1" ] || [ -n "${BANK_PUBLIC:-}" ]; then dw_code=$(http_body "$BANK_PUBLIC/intro/demo-withdraw.json" "$tmp/dw.json") case "$dw_code" in 200) - if python3 - "$tmp/dw.json" <<'PY' -import json, re, sys -d = json.load(open(sys.argv[1])) -if not d.get("ok", True) and "taler_withdraw_uri" not in d: - print("not ok"); sys.exit(1) -u = d.get("taler_withdraw_uri") or "" -m = re.match(r"^taler://withdraw/([^/]+)/taler-integration/([0-9a-fA-F-]+)$", u) -if not m: - print("bad uri:", u[:120]); sys.exit(1) -if ":" not in m.group(1): - print("missing port:", m.group(1)); sys.exit(1) -print(u[:88]) -sys.exit(0) -PY - then - ok "bank /intro/demo-withdraw.json" "$(python3 -c 'import json;print(json.load(open("'"$tmp/dw.json"'")).get("taler_withdraw_uri","")[:80])' 2>/dev/null || true)" - wid=$(python3 -c 'import json;print(json.load(open("'"$tmp/dw.json"'")).get("withdrawal_id",""))' 2>/dev/null || true) + if dw_out=$(validate_bank_withdraw_json "$tmp/dw.json" demo 2>"$tmp/dw-val"); then + dw_detail=${dw_out%%$'\t'*} + wid=${dw_out#*$'\t'} + [ "$wid" = "$dw_out" ] && wid= + ok "bank /intro/demo-withdraw.json" "$dw_detail" if [ -n "$wid" ]; then check_landing_asset "bank taler-integration withdraw op" \ "$BANK_PUBLIC/taler-integration/withdrawal-operation/${wid}" fi else - fail "bank /intro/demo-withdraw.json" "invalid taler://withdraw shape" + fail "bank /intro/demo-withdraw.json" "invalid taler://withdraw ($(tr '\n' ' ' <"$tmp/dw-val" | sed 's/[[:space:]]*$//'))" fi ;; 405|501|404|502|503|000) diff --git a/scripts/taler-monitoring/lib.sh b/scripts/taler-monitoring/lib.sh index 50a54ac..9e03595 100755 --- a/scripts/taler-monitoring/lib.sh +++ b/scripts/taler-monitoring/lib.sh @@ -282,9 +282,16 @@ _fmt_tid() { } ok() { - local label="$1" + # Forms (same idea as info/warn): + # ok "what passed" + # ok "what passed" "detail / ms / bytes / …" + local label="$1" detail="${2:-}" _take_tid - printf '%s[OK]%s %s%s\n' "$G" "$N" "$(_fmt_tid)" "$label" + if [ -n "$detail" ]; then + printf '%s[OK]%s %s%s — %s\n' "$G" "$N" "$(_fmt_tid)" "$label" "$detail" + else + printf '%s[OK]%s %s%s\n' "$G" "$N" "$(_fmt_tid)" "$label" + fi PASS_N=$((PASS_N + 1)) } # component-scoped error: err bank "libeufin down" "detail" diff --git a/scripts/taler-monitoring/taler-monitoring.sh b/scripts/taler-monitoring/taler-monitoring.sh index 2eb9529..9be5740 100755 --- a/scripts/taler-monitoring/taler-monitoring.sh +++ b/scripts/taler-monitoring/taler-monitoring.sh @@ -51,6 +51,7 @@ Examples: Env (same meaning): TALER_DOMAIN BANK_PUBLIC EXCHANGE_PUBLIC MERCHANT_PUBLIC EXPECT_CURRENCY SKIP_SSH=1 NO_COLOR=1 + PERF_WARN_MS PERF_FAIL_MS (urls latency; default 8000 / 20000) EOF }