fix 1.23.4: normalize taler:// :443/:80 + mon scan all landings

Shared lib.sh normalize_taler_uri for ladder/e2e/pay. urls phase checks public bank/exchange/merchant intros and bank mint JSON for residual default ports (ERROR on GOA/local). Unit test included.
2026-07-19 15:38:25 +02:00 · 2026-07-19 15:38:25 +02:00 · d420043863
commit d420043863
parent a103eeab9d
10 changed files with 225 additions and 12 deletions
--- a/CLI-AUTOMATION-NOTES.md
+++ b/CLI-AUTOMATION-NOTES.md
@ -145,8 +145,8 @@ Sources: `https://bank.hacktivism.ch/intro/stats.json`, exchange/merchant intro
 | **Seen** | ladder/e2e strip ports; QR checks; mint URIs with `:443` |
 | **Area** | **cli** / **bank** / **doc** |
 | **Problem** | Some bank/wallet outputs include `taler://…host:443/…`. QR validators and some clients reject or double-normalize inconsistently. |
-| **Workaround** | `sed 's/:443\//\//g'` everywhere in automation. |
-| **Wanted** | Canonical form without default ports in **all** wallet-cli and bank integration outputs; document as invariant. |
+| **Workaround** | mon ≥1.23.4: `normalize_taler_uri` (lib.sh) on ladder/e2e/pay; bank landing `demo-withdraw-api.py` normalizes mint JSON; **urls** phase fails GOA/local if public landings still emit `:443`/`:80` in `taler://` (`www.taler-uri-ports-*`, unit `tests/test_normalize_taler_uri.sh`). |
+| **Wanted** | libeufin never emits default ports (source fix). |

 ---

--- a/TESTS.md
+++ b/TESTS.md
@ -59,10 +59,13 @@ Numbering follows **executed** checks (early skip may shift later NN inside the
 | **www.webui-** | SPA fingerprints: **`/webui/version.txt`**, **`/webui/version-overlay.txt`**, `index.html` / `index.js`; optional pin via `EXPECT_WEBUI_VERSION` / `EXPECT_WEBUI_OVERLAY` / `WEBUI_OVERLAY_DENY` |
 | **www.paivana-** | local GOA paywall front (redirect to template) |
 | **www.landing-** | own-stack intro links; static assets (`qrcode.min.js` hard; `og-goa-shop.png` hard only GOA/local); **demo-withdraw.json** GOA-only; shop-pay soft; stage merchant shop: `shop-ui.js` + `shops.css` + `qrcode.min.js`; cross-links local |
+| **www.taler-uri-ports-** | **no default `:443`/`:80`** in public `taler://` URIs — bank/exchange/merchant `/intro/` HTML, `demo-withdraw.json`, `auto-account.json`, bank `stats.json` (ERROR on GOA/local). Unit: `tests/test_normalize_taler_uri.sh` |
 | **www.qr-** | QR payloads: harvest `taler://` / `payto://` / app `data-qr-url` from landings + mint JSON; **form** check; **qrencode → zbarimg** exact roundtrip; optional static QR images. Needs `qrencode` + `zbar-tools`. Skip: `QR_CHECK=0` |

 **QR rule:** `taler://withdraw/HOST/taler-integration/UUID` (no default `:443`/`:80`); `taler://pay/` / `pay-template/`; `payto://…` shape OK; decoded PNG must equal payload. Form errors on withdraw/pay/payto are **ERROR** on local/GOA.

+**Default port rule:** public and automation URIs must never keep libeufin-style `host:443` / `host:80`. Consumers use `normalize_taler_uri` (lib.sh); bank landing API normalizes in `demo-withdraw-api.py`.
+
 **Legal docs rule:** HTTP 200, non-empty body, not plain `not configured`, not merchant API JSON `code:21`. Local stack may require content needle (terms/privacy/FADP/GOA…).

 **Performance rule:** Measured from the **monitoring runner** (public via Caddy), not container loopback. Latency (ms) on every perf line + **perf summary** (n / min / p50 / avg / max).
--- a/2
+++ b/2
@ -1 +1 @@
-1.23.3
+1.23.4
--- a/VERSIONS.md
+++ b/VERSIONS.md
@ -17,6 +17,7 @@ Git tags: `vMAJOR.FEATURE.FIX` (e.g. `v1.8.0`). File `VERSION` omits the `v` pre

 | Tag | Date (UTC) | Notes |
 |-----|------------|--------|
+| **v1.23.4** | 2026-07-19 | **Bugfix / mon:** shared **normalize_taler_uri** (strip taler:// host :443/:80); ladder/e2e/pay use it. **urls** scans bank+exchange+merchant intros + demo-withdraw/auto-account/stats for default ports (**ERROR** on GOA). Unit test tests/test_normalize_taler_uri.sh. |
 | **v1.23.3** | 2026-07-19 | **Bugfix:** payments stale because max-ladder never got coins — root cause on koopa was **taler-wallet-cli missing** (all mint-OK → SKIP_ACCEPT). Hard-fail ladder prereq if CLI/helper missing; settle keeps polling after transfer_done + short run-pending; pay wait default 180s; max-ladder wrapper sets PATH/WALLET_CLI. |
 | **v1.23.2** | 2026-07-19 | **Bugfix:** max-ladder host-agent forces **RUN_TIMEOUT=10800** (koopa env had 600 → run aborted mid-ladder); install starts oneshots with **--no-block**. |
 | **v1.23.1** | 2026-07-19 | **Fix / scope:** only **`/monitoring-max-ladder/`** for GOA (drop classic `/monitoring-ladder`). Daily timer + install first-run. monpages catalog/Caddy bare redirs for max-ladder only. |
--- a/check_amount_ladder.sh
+++ b/check_amount_ladder.sh
@ -671,7 +671,12 @@ ladder_withdraw_rung() {
    "${BANK}/accounts/${EXP_USER}/withdrawals")
  ms_mint=$(elapsed_ms "$t0")
  WID=$(python3 -c 'import json;d=json.load(open("'"$SCRATCH"'/wd-'"$tag"'.json"));print(d.get("withdrawal_id") or "")' 2>/dev/null || true)
-  URI=$(python3 -c 'import json;u=json.load(open("'"$SCRATCH"'/wd-'"$tag"'.json")).get("taler_withdraw_uri") or "";print(u.replace(":443/","/"))' 2>/dev/null || true)
+  # libeufin often emits taler://withdraw/host:443/… — strip default ports (lib.sh)
+  URI=$(python3 -c 'import json;print(json.load(open("'"$SCRATCH"'/wd-'"$tag"'.json")).get("taler_withdraw_uri") or "")' 2>/dev/null || true)
+  URI=$(normalize_taler_uri "$URI")
+  if taler_uri_has_default_port "$URI"; then
+    warn bank "withdraw URI still has default port" "problem: $URI"
+  fi
  # numeric amount (for zero / settle / ceiling special-cases)
  AMT_NUM=$(python3 -c 'import sys; print(sys.argv[1].split(":",1)[-1])' "$AMT")
  IS_ZERO=0
--- a/check_e2e.sh
+++ b/check_e2e.sh
@ -944,7 +944,7 @@ e2e_one_withdraw() {
    "$BANK/accounts/${USER}/withdrawals"
  WID=$(python3 -c 'import json;d=json.load(open("'"$SCRATCH"'/wd-'"$tag"'.json"));print(d.get("withdrawal_id") or d.get("id") or "")' 2>/dev/null || true)
  URI=$(python3 -c 'import json;d=json.load(open("'"$SCRATCH"'/wd-'"$tag"'.json"));print(d.get("taler_withdraw_uri") or "")' 2>/dev/null || true)
-  URI_CLEAN=$(python3 -c 'import json;u=json.load(open("'"$SCRATCH"'/wd-'"$tag"'.json")).get("taler_withdraw_uri") or "";print(u.replace(":443/","/"))' 2>/dev/null || true)
+  URI_CLEAN=$(normalize_taler_uri "$URI")
  if [ -z "$WID" ] || [ -z "$URI_CLEAN" ]; then
    warn "Bank withdraw $WITHDRAW_AMT" "create failed — $(head -c 80 "$SCRATCH/wd-$tag.json" | tr '\n' ' ')"
    return 1
@ -952,13 +952,13 @@ e2e_one_withdraw() {
  ok "Bank withdrawal created $WITHDRAW_AMT ($WID)"
  cp "$SCRATCH/wd-$tag.json" "$SCRATCH/wd.json"

-  USE_URI="$URI"
-  [ -z "$USE_URI" ] && USE_URI="$URI_CLEAN"
+  # Always prefer stripped URI (libeufin often emits host:443)
+  USE_URI="${URI_CLEAN:-$URI}"
  if wcli withdraw accept-uri --exchange "${EXCHANGE_PUBLIC}/" "$USE_URI" >"$SCRATCH/accept-$tag.out" 2>&1; then
    ok "wallet accept $WITHDRAW_AMT"
-  elif [ "$USE_URI" != "$URI_CLEAN" ] && [ -n "$URI_CLEAN" ] \
-    && wcli withdraw accept-uri --exchange "${EXCHANGE_PUBLIC}/" "$URI_CLEAN" >"$SCRATCH/accept-$tag.out" 2>&1; then
-    ok "wallet accept $WITHDRAW_AMT (no :443)"
+  elif [ -n "$URI" ] && [ "$USE_URI" != "$URI" ] \
+    && wcli withdraw accept-uri --exchange "${EXCHANGE_PUBLIC}/" "$URI" >"$SCRATCH/accept-$tag.out" 2>&1; then
+    ok "wallet accept $WITHDRAW_AMT (raw URI fallback)"
  else
    # Prefer the real Error/ENOENT line over noise (perl -e source, stack frames)
    _acc_err=$(
--- a/check_urls.sh
+++ b/check_urls.sh
@ -1339,6 +1339,90 @@ if [ "${LOCAL_STACK:-1}" = "1" ] || [ -n "${BANK_PUBLIC:-}" ]; then
  fi
 fi

+# ---------------------------------------------------------------------------
+# taler:// URIs must not carry default ports :443 / :80 (wallet / QR rule)
+# Scans public landings on this stack (bank + exchange + merchant intros,
+# bank mint JSON). Fail hard on GOA/local; soft warn elsewhere.
+# ---------------------------------------------------------------------------
+set_group landings
+section "www · taler:// default ports (:443/:80 must be absent)"
+_port_scan_fail=0
+_port_scan_ok=0
+_port_scan_one() {
+  local label="$1" url="$2"
+  local f code n detail
+  f="$tmp/portscan-$$.body"
+  code=$(http_body "$url" "$f")
+  if [ "$code" != "200" ]; then
+    if [ "${LOCAL_STACK:-1}" = "1" ] || [ "${EXPECT_CURRENCY:-}" = "GOA" ]; then
+      # mint JSON is hard; HTML intros soft if missing
+      case "$url" in
+        *demo-withdraw.json|*auto-account.json)
+          fail "taler-uri ports · $label" "HTTP $code · $url"
+          _port_scan_fail=$((_port_scan_fail + 1))
+          ;;
+        *)
+          warn "taler-uri ports · $label" "HTTP $code · skip · $url"
+          ;;
+      esac
+    else
+      info "taler-uri ports · $label" "HTTP $code · skip"
+    fi
+    rm -f "$f"
+    return 0
+  fi
+  detail=$(python3 - "$f" <<'PY' 2>/dev/null || true
+import re, sys, json
+from pathlib import Path
+raw = Path(sys.argv[1]).read_text(errors="replace")
+# collect taler:// from JSON fields and raw HTML/JS
+uris = set(re.findall(r"taler://[A-Za-z0-9._~:/?#\[\]@!$&'()*+,;=%-]+", raw))
+# JSON-escaped sequences
+uris.update(re.findall(r"taler:\\/\\/[A-Za-z0-9._~:/?#\[\]@!$&'()*+,;=%\\-]+", raw))
+uris = {u.replace("\\/", "/") for u in uris}
+bad = []
+pat = re.compile(r"taler://\S+:(443|80)(/|$|\?|#)", re.I)
+for u in sorted(uris):
+    if pat.search(u):
+        bad.append(u[:120])
+if bad:
+    print("BAD\t%d\t%s" % (len(bad), bad[0]))
+else:
+    print("OK\t%d\t" % len(uris))
+PY
+)
+  n_kind=${detail%%$'\t'*}
+  rest=${detail#*$'\t'}
+  n_count=${rest%%$'\t'*}
+  sample=${rest#*$'\t'}
+  if [ "$n_kind" = "BAD" ]; then
+    if [ "${LOCAL_STACK:-1}" = "1" ] || [ "${EXPECT_CURRENCY:-}" = "GOA" ]; then
+      fail "taler-uri ports · $label" "default :443/:80 still present · n=${n_count} e.g. ${sample}"
+      _port_scan_fail=$((_port_scan_fail + 1))
+    else
+      warn "taler-uri ports · $label" "default port in URI · ${sample}"
+    fi
+  else
+    ok "taler-uri ports · $label" "no default ports · taler_uris=${n_count:-0} · $url"
+    _port_scan_ok=$((_port_scan_ok + 1))
+  fi
+  rm -f "$f"
+}
+if [ "${CHECK_LANDING:-1}" = "1" ]; then
+  _port_scan_one "bank intro HTML" "${BANK_PUBLIC}/intro/"
+  _port_scan_one "bank demo-withdraw.json" "${BANK_PUBLIC}/intro/demo-withdraw.json"
+  _port_scan_one "bank auto-account.json" "${BANK_PUBLIC}/intro/auto-account.json"
+  _port_scan_one "exchange intro HTML" "${EXCHANGE_PUBLIC}/intro/"
+  _port_scan_one "merchant intro HTML" "${MERCHANT_PUBLIC}/intro/"
+  # stats may embed demo withdraw URI
+  _port_scan_one "bank stats.json" "${BANK_PUBLIC}/intro/stats.json"
+  if [ "$_port_scan_fail" -eq 0 ]; then
+    ok "taler-uri ports summary" "clean · ${_port_scan_ok} resource(s) checked (bank+exchange+merchant)"
+  fi
+else
+  info "taler-uri ports" "skipped (CHECK_LANDING=0)"
+fi
+
 # Merchant shop assets — GOA: shop-pay.*; stage TESTPAYSAN: /assets/shop-ui.js +
 # shops.css + QR encoder (shop-ui modal needs /intro/qrcode.min.js or /qrcode.min.js).
 _ma=0
--- a/ladder/lib_pay.sh
+++ b/ladder/lib_pay.sh
@ -67,7 +67,7 @@ except Exception:
 ' "$SCRATCH/ord-$ptag.json" 2>/dev/null || true)
    if [ -n "$OID" ] && [ -n "$OTOK" ]; then
      PAYURI="taler://pay/${mh}/instances/${inst}/${OID}/?c=${OTOK}"
-      PAYURI=$(printf '%s' "$PAYURI" | sed 's/:443\//\//g; s/:443?/?/g')
+      PAYURI=$(normalize_taler_uri "$PAYURI")
      return 0
    fi
  fi
@ -98,7 +98,7 @@ except Exception: print("")
  if [ -z "$PAYURI" ] && [ -n "$OTOK" ]; then
    PAYURI="taler://pay/${mh}/instances/${INST}/${OID}/?c=${OTOK}"
  fi
-  PAYURI=$(printf '%s' "$PAYURI" | sed 's/:443\//\//g; s/:443?/?/g')
+  PAYURI=$(normalize_taler_uri "$PAYURI")
  [ -n "$PAYURI" ]
 }

--- a/lib.sh
+++ b/lib.sh
@ -1504,6 +1504,53 @@ load_monitoring_secrets_env() {
  export E2E_BANK_ADMIN_PASS E2E_MERCHANT_TOKEN SECRETS_ROOT KOOPA_ADMIN_SECRETS
 }

+# Strip default HTTPS/HTTP ports from taler:// URIs (libeufin often emits host:443).
+# Wallets and mon QR rules reject default :443/:80 in the authority.
+# Usage: normalize_taler_uri "taler://withdraw/host:443/…"  → prints cleaned URI
+normalize_taler_uri() {
+  local u="${1-}"
+  if [ -z "$u" ]; then
+    return 0
+  fi
+  printf '%s' "$u" | python3 -c '
+import re, sys
+u = sys.stdin.read().strip()
+# any taler://SCHEME/HOST:443|80/...
+u = re.sub(
+    r"(taler://[A-Za-z0-9._-]+/)([^/?#:]+):443(?=/|$|\?|#)",
+    r"\1\2",
+    u,
+    flags=re.I,
+)
+u = re.sub(
+    r"(taler://[A-Za-z0-9._-]+/)([^/?#:]+):80(?=/|$|\?|#)",
+    r"\1\2",
+    u,
+    flags=re.I,
+)
+# leftover path/query forms
+for a, b in (
+    (":443/", "/"),
+    (":443?", "?"),
+    (":443#", "#"),
+    (":80/", "/"),
+    (":80?", "?"),
+    (":80#", "#"),
+):
+    u = u.replace(a, b)
+# bare trailing :443 / :80 on host (no slash)
+u = re.sub(r"(taler://[A-Za-z0-9._-]+/[^/?#:]+):443$", r"\1", u, flags=re.I)
+u = re.sub(r"(taler://[A-Za-z0-9._-]+/[^/?#:]+):80$", r"\1", u, flags=re.I)
+sys.stdout.write(u)
+' 2>/dev/null || printf '%s' "$u"
+}
+
+# True (exit 0) if URI still has default port in taler:// authority.
+taler_uri_has_default_port() {
+  # host:443 or host:80 before next path/query/end
+  python3 -c "import re,sys; u=sys.argv[1] if len(sys.argv)>1 else ''; sys.exit(0 if re.search(r'taler://\\S+:(443|80)(/|$|\\?|#)', u, re.I) else 1)" "${1-}" 2>/dev/null
+}
+
 find_wallet_cli() {
  # Must return a path suitable for: node "$WALLET_CLI" …
  # Prefer *.mjs / bundled entry; plain /usr/bin/taler-wallet-cli is often a
--- a/tests/test_normalize_taler_uri.sh
+++ b/tests/test_normalize_taler_uri.sh
@ -0,0 +1,73 @@
+#!/usr/bin/env bash
+# Unit tests for normalize_taler_uri / taler_uri_has_default_port (lib.sh).
+# Run: ./tests/test_normalize_taler_uri.sh
+set -euo pipefail
+ROOT=$(cd "$(dirname "$0")/.." && pwd)
+# shellcheck source=../lib.sh
+source "$ROOT/lib.sh"
+
+pass=0
+fail=0
+assert_eq() {
+  local name="$1" got="$2" want="$3"
+  if [ "$got" = "$want" ]; then
+    pass=$((pass + 1))
+  else
+    echo "FAIL $name" >&2
+    echo "  got:  $got" >&2
+    echo "  want: $want" >&2
+    fail=$((fail + 1))
+  fi
+}
+assert_true() {
+  local name="$1"
+  if "$@"; then
+    # shift name
+    :
+  fi
+}
+
+# --- strip :443 / :80 ---
+assert_eq "withdraw :443/" \
+  "$(normalize_taler_uri 'taler://withdraw/bank.hacktivism.ch:443/taler-integration/abc-123')" \
+  "taler://withdraw/bank.hacktivism.ch/taler-integration/abc-123"
+
+assert_eq "withdraw :443 end" \
+  "$(normalize_taler_uri 'taler://withdraw/bank.example:443')" \
+  "taler://withdraw/bank.example"
+
+assert_eq "pay :443" \
+  "$(normalize_taler_uri 'taler://pay/taler.hacktivism.ch:443/instances/x/orders/y?c=z')" \
+  "taler://pay/taler.hacktivism.ch/instances/x/orders/y?c=z"
+
+assert_eq "pay-template :80" \
+  "$(normalize_taler_uri 'taler://pay-template/shop.example:80/instances/i/t')" \
+  "taler://pay-template/shop.example/instances/i/t"
+
+assert_eq "already clean" \
+  "$(normalize_taler_uri 'taler://withdraw/bank.hacktivism.ch/taler-integration/uuid')" \
+  "taler://withdraw/bank.hacktivism.ch/taler-integration/uuid"
+
+assert_eq "empty" "$(normalize_taler_uri '')" ""
+
+# non-default port kept
+assert_eq "keep :8443" \
+  "$(normalize_taler_uri 'taler://withdraw/bank.example:8443/taler-integration/x')" \
+  "taler://withdraw/bank.example:8443/taler-integration/x"
+
+# has_default_port detector
+if taler_uri_has_default_port 'taler://withdraw/h:443/p'; then
+  pass=$((pass + 1))
+else
+  echo "FAIL detector should match :443" >&2
+  fail=$((fail + 1))
+fi
+if taler_uri_has_default_port 'taler://withdraw/h/p'; then
+  echo "FAIL detector should not match clean URI" >&2
+  fail=$((fail + 1))
+else
+  pass=$((pass + 1))
+fi
+
+echo "normalize_taler_uri tests: pass=$pass fail=$fail"
+[ "$fail" -eq 0 ]
 @ -1 +1 @@
 .23.3
 .23.4