fix(monitoring): withdraw URI without :443 and show perf ms

scripts/taler-monitoring/README.md

@@ -94,6 +94,22 @@ E2E maps failures to blockers, e.g.:
 - `Alarm clock` during pay → usually missing `handle-uri --yes` or wrong pay URI (must be `…/instances/{inst}/{oid}/?c={token}` from merchant `taler_pay_uri`)
 - `insufficient balance` → withdraw incomplete  
 
+## Performance (urls phase)
+
+Outside-in HTTPS RTT for bank / exchange / merchant critical paths. Each probe prints
+`HTTP … · N ms` on the `[OK]`/`[WARN]`/`[ERROR]` line; end of the block prints
+`perf summary` (n / min / p50 / avg / max).
+
+| Env | Default | Meaning |
+|-----|---------|---------|
+| `PERF_WARN_MS` | `8000` | WARN if latency ≥ this |
+| `PERF_FAIL_MS` | `20000` | ERROR if latency ≥ this |
+| `PERF_CURL_TIMEOUT` | `25` | curl max seconds per probe |
+
+```bash
+PERF_WARN_MS=3000 PERF_FAIL_MS=10000 ./taler-monitoring.sh urls
+```
+
 ## Needs
 
 - SSH `koopa` (inside/sanity server bits)

scripts/taler-monitoring/TESTS.md

@@ -46,15 +46,15 @@ IDs are assigned **in run order** within the area (`set_area` resets the counter
 | www-… | **landing exposed links** (bank / merchant / exchange): parse each `/intro/` HTML, probe every own-stack `https://` + root-relative `href`/`src`/`content`, soft-check external stores/docs |
 | www-… | landing static: `qrcode.min.js`, `og-goa-shop.png`, `qr-logo.png`, shop-pay.js/css |
 | www-… | cross-links between bank ↔ merchant ↔ exchange intros (local stack) |
-| www-… | **bank `/intro/demo-withdraw.json`** → `taler://withdraw/HOST:PORT/taler-integration/…` + integration op HTTP 200 |
+| www-… | **bank `/intro/demo-withdraw.json`** → `taler://withdraw/HOST/taler-integration/…` (no default `:443`/`:80`; non-default port OK) + integration op HTTP 200 |
 | www-… | bank `/intro/auto-account.json` (earlier) → same withdraw shape, **no payto_uri**, login at `/webui/` |
 | www-… | **performance** (outside-in): public HTTPS RTT for bank `/config`, `/taler-integration/config`, `/webui/`, `/intro/`, `stats.json`; exchange `/config`, `/keys`, `/intro/`; merchant `/config`, `/webui/`, `/intro/` — report ms; WARN ≥ `PERF_WARN_MS` (default 8000); **ERROR ≥ `PERF_FAIL_MS` (default 20000)** |
 
 **Legal docs rule:** HTTP 200, non-empty body, not plain `not configured`, not merchant API JSON `code:21`. On local stack, optional content needle (terms/privacy/FADP/GOA…).
 
-**Performance rule:** Measured from the **monitoring runner** (public URLs via Caddy), not container loopback. HTTP must match expect (usually 200); latency is reported on the OK line. Slow ≥ `PERF_WARN_MS` → WARN only (no ERROR on slowness alone).
+**Performance rule:** Measured from the **monitoring runner** (public URLs via Caddy), not container loopback. HTTP must match expect (usually 200); **latency (ms) is on every perf OK/WARN/ERROR line**, plus a **perf summary** (n / min / p50 / avg / max). Slow ≥ `PERF_WARN_MS` (default 8000) → WARN; ≥ `PERF_FAIL_MS` (default 20000) → ERROR.
 
-**Landing links rule:** Own-stack (bank/exchange/taler.\* + page host) must be HTTP 200 (or redirect→200). External (App Store, Play, F-Droid, wallet.taler.net, docs/git.taler.net, …) soft WARN if down. Auto-account wallet link must be `taler://withdraw/…:port/taler-integration/…`, never payto.
+**Landing links rule:** Own-stack (bank/exchange/taler.\* + page host) must be HTTP 200 (or redirect→200). External (App Store, Play, F-Droid, wallet.taler.net, docs/git.taler.net, …) soft WARN if down. Auto-account wallet link must be `taler://withdraw/HOST/taler-integration/…` (default ports stripped for mobile wallets), never payto.
 
 **alt_unit_names rule:** wallet codec requires a non-empty map including scale key `"0"`. For multi-currency merchant, also follow every entry in `exchanges[]` and check that exchange’s public `/config`.
 

scripts/taler-monitoring/check_urls.sh

@@ -98,6 +98,77 @@ check_legal_doc() {
   rm -f "$f"
 }
 
+# Shared shape for bank mint JSON (demo-withdraw + auto-account).
+# Args: $1=json file  $2=auto|demo
+# stdout (ok): one detail line; for demo, optional "\twithdrawal_id" suffix.
+# stderr (fail): short reason. Exit 0/1.
+validate_bank_withdraw_json() {
+  python3 - "$1" "$2" <<'PY'
+import json, re, sys
+from urllib.parse import urlparse
+
+def check_withdraw_uri(wuri: str):
+    """HOST or HOST:non-default-port; default :443/:80 must be stripped (wallet fix)."""
+    m = re.match(
+        r"^taler://withdraw/([^/]+)/taler-integration/([0-9a-fA-F-]+)$",
+        wuri or "",
+    )
+    if not m:
+        print(
+            "need taler://withdraw/HOST/taler-integration/ID:",
+            (wuri or "")[:120],
+            file=sys.stderr,
+        )
+        return None
+    host = m.group(1)
+    if not host or host.startswith(":"):
+        print("bad withdraw host:", host, file=sys.stderr)
+        return None
+    if ":" in host:
+        h, _, p = host.rpartition(":")
+        if not h or not p.isdigit():
+            print("bad withdraw host:port:", host, file=sys.stderr)
+            return None
+        if p in ("443", "80"):
+            print("withdraw must strip default port :%s:" % p, host, file=sys.stderr)
+            return None
+    return host, m.group(2), wuri
+
+path, mode = sys.argv[1], sys.argv[2]
+d = json.load(open(path))
+wuri = d.get("taler_withdraw_uri") or (
+    d.get("qr_payload") if mode == "auto" else ""
+) or ""
+
+if mode == "auto":
+    if not d.get("ok"):
+        print("ok!=true", file=sys.stderr)
+        sys.exit(1)
+    if d.get("payto_uri"):
+        print("payto_uri must not be present", file=sys.stderr)
+        sys.exit(1)
+    if not check_withdraw_uri(wuri):
+        sys.exit(1)
+    webui = d.get("login_url") or d.get("webui") or d.get("account_url") or ""
+    u = urlparse(webui)
+    if u.scheme not in ("http", "https") or "webui" not in (u.path or ""):
+        print("login webui missing:", webui[:80], file=sys.stderr)
+        sys.exit(1)
+    print("%s · %s" % (d.get("username", ""), wuri[:72]))
+elif mode == "demo":
+    if not d.get("ok", True) and "taler_withdraw_uri" not in d:
+        print("not ok", file=sys.stderr)
+        sys.exit(1)
+    if not check_withdraw_uri(wuri):
+        sys.exit(1)
+    print("%s\t%s" % (wuri[:80], d.get("withdrawal_id") or ""))
+else:
+    print("bad mode:", mode, file=sys.stderr)
+    sys.exit(2)
+sys.exit(0)
+PY
+}
+
 # --- exchange (core; always required) ---  www-001 …
 check_url "exchange /config" 200 "$EXCHANGE_PUBLIC/config"
 code=$(http_body "$EXCHANGE_PUBLIC/config" "$tmp/ec.json")
@@ -150,6 +221,8 @@ section "www · performance · public HTTPS latency (outside-in)"
 # ≥ PERF_WARN_MS → WARN. ≥ PERF_FAIL_MS → ERROR (suite fails).
 PERF_WARN_MS="${PERF_WARN_MS:-8000}"
 PERF_FAIL_MS="${PERF_FAIL_MS:-20000}"
+PERF_TSV="$tmp/perf.tsv"
+: >"$PERF_TSV"
 
 # Measure one URL: require HTTP expect (default 200), report time_total in ms.
 # $1=label $2=url $3=optional expected codes (default 200)
@@ -165,6 +238,8 @@ check_perf() {
     if (ms>0 && ms<1) ms=1
     printf "%d", int(ms+0.5)
   }')
+  # Always record sample for rollup (label, ms, http, url)
+  printf '%s\t%s\t%s\t%s\n' "$label" "$ms" "$code" "$url" >>"$PERF_TSV"
   case ",$expect," in
     *",$code,"*)
       if [ "$ms" -ge "${PERF_FAIL_MS}" ] 2>/dev/null; then
@@ -202,7 +277,76 @@ check_perf "perf merchant /config" "$MERCHANT_PUBLIC/config"
 check_perf "perf merchant /webui/" "$MERCHANT_PUBLIC/webui/" 200,301,302
 check_perf "perf merchant /intro/" "$MERCHANT_PUBLIC/intro/"
 
-info "perf note" "measured from this host (outside-in); not container loopback"
+# Rollup + optional JSON for metrics_print_overall
+if [ -s "$PERF_TSV" ]; then
+  PERF_JSON="$tmp/perf-summary.json"
+  perf_line=$(python3 - "$PERF_TSV" "$PERF_JSON" "$PERF_WARN_MS" "$PERF_FAIL_MS" <<'PY'
+import json, sys
+rows = []
+for line in open(sys.argv[1]):
+    parts = line.rstrip("\n").split("\t")
+    if len(parts) < 3:
+        continue
+    label, ms_s, code = parts[0], parts[1], parts[2]
+    try:
+        ms = int(float(ms_s))
+    except Exception:
+        continue
+    rows.append({"label": label, "ms": ms, "http": code})
+vals = sorted(r["ms"] for r in rows)
+n = len(vals)
+if n == 0:
+    print("n=0")
+    json.dump({"n": 0}, open(sys.argv[2], "w"))
+    raise SystemExit(0)
+def pct(p):
+    if n == 1:
+        return vals[0]
+    i = min(n - 1, max(0, int(round((p / 100.0) * (n - 1)))))
+    return vals[i]
+avg = int(round(sum(vals) / n))
+warn_ms = int(sys.argv[3])
+fail_ms = int(sys.argv[4])
+slow = [r for r in rows if r["ms"] >= warn_ms]
+rep = {
+    "n": n,
+    "min_ms": vals[0],
+    "p50_ms": pct(50),
+    "avg_ms": avg,
+    "max_ms": vals[-1],
+    "warn_ms": warn_ms,
+    "fail_ms": fail_ms,
+    "slow": [{"label": r["label"], "ms": r["ms"]} for r in slow],
+    "samples": rows,
+}
+# metrics_print_overall expects named buckets with n/min/p50/avg/max
+out = {
+    "www_public_https": {
+        "n": n,
+        "min_ms": vals[0],
+        "p50_ms": pct(50),
+        "avg_ms": avg,
+        "max_ms": vals[-1],
+    },
+    **{r["label"].replace(" ", "_"): {"n": 1, "min_ms": r["ms"], "p50_ms": r["ms"], "avg_ms": r["ms"], "max_ms": r["ms"]} for r in rows},
+}
+json.dump(out, open(sys.argv[2], "w"), indent=2)
+extra = ""
+if slow:
+    extra = "  slow: " + ", ".join("%s=%dms" % (r["label"].replace("perf ", ""), r["ms"]) for r in slow)
+print(
+    "n=%d  min=%dms  p50=%dms  avg=%dms  max=%dms  (warn≥%dms fail≥%dms)%s"
+    % (n, vals[0], pct(50), avg, vals[-1], warn_ms, fail_ms, extra)
+)
+PY
+  )
+  info "perf summary" "$perf_line"
+  # Keep a copy if METRICS_DIR is set (e2e/ladder overall stats)
+  if [ -n "${METRICS_DIR:-}" ] && [ -d "${METRICS_DIR}" ]; then
+    cp -f "$PERF_JSON" "${METRICS_DIR}/perf-summary.json" 2>/dev/null || true
+  fi
+fi
+info "perf note" "measured from this host (outside-in); thresholds PERF_WARN_MS=${PERF_WARN_MS} PERF_FAIL_MS=${PERF_FAIL_MS}"
 
 
 # Terms + privacy (legal docs)
@@ -247,31 +391,10 @@ if [ "$code" = "200" ]; then
   aa_code=$(http_body "$BANK_PUBLIC/intro/auto-account.json" "$tmp/aa.json")
   case "$aa_code" in
     200)
-      if python3 - "$tmp/aa.json" <<'PY'
+      if aa_detail=$(validate_bank_withdraw_json "$tmp/aa.json" auto 2>"$tmp/aa-val"); then
-import json, re, sys
+        ok "bank /intro/auto-account.json" "$aa_detail"
-from urllib.parse import urlparse
-d = json.load(open(sys.argv[1]))
-if not d.get("ok"):
-    print("ok!=true"); sys.exit(1)
-if "payto_uri" in d and d.get("payto_uri"):
-    print("payto_uri must not be present"); sys.exit(1)
-wuri = d.get("taler_withdraw_uri") or d.get("qr_payload") or ""
-wm = re.match(r"^taler://withdraw/([^/]+)/taler-integration/([0-9a-fA-F-]+)$", wuri)
-if not wm:
-    print("need taler://withdraw/HOST:PORT/taler-integration/ID:", wuri[:120]); sys.exit(1)
-if ":" not in wm.group(1):
-    print("withdraw missing port:", wm.group(1)); sys.exit(1)
-webui = d.get("login_url") or d.get("webui") or d.get("account_url") or ""
-u = urlparse(webui)
-if u.scheme not in ("http", "https") or "webui" not in (u.path or ""):
-    print("login webui missing:", webui[:80]); sys.exit(1)
-print("user=%s withdraw=%s login=%s" % (d.get("username"), wm.group(1), webui))
-sys.exit(0)
-PY
-      then
-        ok "bank /intro/auto-account.json" "$(python3 -c 'import json;d=json.load(open("'"$tmp/aa.json"'"));print(d.get("username",""),"·",(d.get("taler_withdraw_uri") or "")[:72])' 2>/dev/null || true)"
       else
-        fail "bank /intro/auto-account.json" "invalid withdraw/login (HTTP body bad)"
+        fail "bank /intro/auto-account.json" "invalid withdraw/login ($(tr '\n' ' ' <"$tmp/aa-val" | sed 's/[[:space:]]*$//'))"
       fi
       ;;
     405|501|404|502|503|000)
@@ -591,29 +714,17 @@ if [ "${LOCAL_STACK:-1}" = "1" ] || [ -n "${BANK_PUBLIC:-}" ]; then
   dw_code=$(http_body "$BANK_PUBLIC/intro/demo-withdraw.json" "$tmp/dw.json")
   case "$dw_code" in
     200)
-      if python3 - "$tmp/dw.json" <<'PY'
+      if dw_out=$(validate_bank_withdraw_json "$tmp/dw.json" demo 2>"$tmp/dw-val"); then
-import json, re, sys
+        dw_detail=${dw_out%%$'\t'*}
-d = json.load(open(sys.argv[1]))
+        wid=${dw_out#*$'\t'}
-if not d.get("ok", True) and "taler_withdraw_uri" not in d:
+        [ "$wid" = "$dw_out" ] && wid=
-    print("not ok"); sys.exit(1)
+        ok "bank /intro/demo-withdraw.json" "$dw_detail"
-u = d.get("taler_withdraw_uri") or ""
-m = re.match(r"^taler://withdraw/([^/]+)/taler-integration/([0-9a-fA-F-]+)$", u)
-if not m:
-    print("bad uri:", u[:120]); sys.exit(1)
-if ":" not in m.group(1):
-    print("missing port:", m.group(1)); sys.exit(1)
-print(u[:88])
-sys.exit(0)
-PY
-      then
-        ok "bank /intro/demo-withdraw.json" "$(python3 -c 'import json;print(json.load(open("'"$tmp/dw.json"'")).get("taler_withdraw_uri","")[:80])' 2>/dev/null || true)"
-        wid=$(python3 -c 'import json;print(json.load(open("'"$tmp/dw.json"'")).get("withdrawal_id",""))' 2>/dev/null || true)
         if [ -n "$wid" ]; then
           check_landing_asset "bank taler-integration withdraw op" \
             "$BANK_PUBLIC/taler-integration/withdrawal-operation/${wid}"
         fi
       else
-        fail "bank /intro/demo-withdraw.json" "invalid taler://withdraw shape"
+        fail "bank /intro/demo-withdraw.json" "invalid taler://withdraw ($(tr '\n' ' ' <"$tmp/dw-val" | sed 's/[[:space:]]*$//'))"
       fi
       ;;
     405|501|404|502|503|000)

scripts/taler-monitoring/lib.sh

@@ -282,9 +282,16 @@ _fmt_tid() {
 }
 
 ok() {
-  local label="$1"
+  # Forms (same idea as info/warn):
+  #   ok "what passed"
+  #   ok "what passed" "detail / ms / bytes / …"
+  local label="$1" detail="${2:-}"
   _take_tid
+  if [ -n "$detail" ]; then
+    printf '%s[OK]%s    %s%s — %s\n' "$G" "$N" "$(_fmt_tid)" "$label" "$detail"
+  else
     printf '%s[OK]%s    %s%s\n' "$G" "$N" "$(_fmt_tid)" "$label"
+  fi
   PASS_N=$((PASS_N + 1))
 }
 # component-scoped error: err bank "libeufin down" "detail"

scripts/taler-monitoring/taler-monitoring.sh

@@ -51,6 +51,7 @@ Examples:
 Env (same meaning):
   TALER_DOMAIN  BANK_PUBLIC  EXCHANGE_PUBLIC  MERCHANT_PUBLIC  EXPECT_CURRENCY
   SKIP_SSH=1    NO_COLOR=1
+  PERF_WARN_MS  PERF_FAIL_MS   (urls latency; default 8000 / 20000)
 EOF
 }