From d42004386370ed7f02665fa61e1f1637d4e2daf8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Hern=C3=A2ni=20Marques?= Date: Sun, 19 Jul 2026 15:38:25 +0200 Subject: [PATCH] fix 1.23.4: normalize taler:// :443/:80 + mon scan all landings Shared lib.sh normalize_taler_uri for ladder/e2e/pay. urls phase checks public bank/exchange/merchant intros and bank mint JSON for residual default ports (ERROR on GOA/local). Unit test included. --- CLI-AUTOMATION-NOTES.md | 4 +- TESTS.md | 3 ++ VERSION | 2 +- VERSIONS.md | 1 + check_amount_ladder.sh | 7 ++- check_e2e.sh | 12 ++--- check_urls.sh | 84 +++++++++++++++++++++++++++++++ ladder/lib_pay.sh | 4 +- lib.sh | 47 +++++++++++++++++ tests/test_normalize_taler_uri.sh | 73 +++++++++++++++++++++++++++ 10 files changed, 225 insertions(+), 12 deletions(-) create mode 100755 tests/test_normalize_taler_uri.sh diff --git a/CLI-AUTOMATION-NOTES.md b/CLI-AUTOMATION-NOTES.md index b4f7431..22f75e5 100644 --- a/CLI-AUTOMATION-NOTES.md +++ b/CLI-AUTOMATION-NOTES.md @@ -145,8 +145,8 @@ Sources: `https://bank.hacktivism.ch/intro/stats.json`, exchange/merchant intro | **Seen** | ladder/e2e strip ports; QR checks; mint URIs with `:443` | | **Area** | **cli** / **bank** / **doc** | | **Problem** | Some bank/wallet outputs include `taler://…host:443/…`. QR validators and some clients reject or double-normalize inconsistently. | -| **Workaround** | `sed 's/:443\//\//g'` everywhere in automation. | -| **Wanted** | Canonical form without default ports in **all** wallet-cli and bank integration outputs; document as invariant. | +| **Workaround** | mon ≥1.23.4: `normalize_taler_uri` (lib.sh) on ladder/e2e/pay; bank landing `demo-withdraw-api.py` normalizes mint JSON; **urls** phase fails GOA/local if public landings still emit `:443`/`:80` in `taler://` (`www.taler-uri-ports-*`, unit `tests/test_normalize_taler_uri.sh`). | +| **Wanted** | libeufin never emits default ports (source fix). | --- diff --git a/TESTS.md b/TESTS.md index 9320315..7364968 100644 --- a/TESTS.md +++ b/TESTS.md @@ -59,10 +59,13 @@ Numbering follows **executed** checks (early skip may shift later NN inside the | **www.webui-** | SPA fingerprints: **`/webui/version.txt`**, **`/webui/version-overlay.txt`**, `index.html` / `index.js`; optional pin via `EXPECT_WEBUI_VERSION` / `EXPECT_WEBUI_OVERLAY` / `WEBUI_OVERLAY_DENY` | | **www.paivana-** | local GOA paywall front (redirect to template) | | **www.landing-** | own-stack intro links; static assets (`qrcode.min.js` hard; `og-goa-shop.png` hard only GOA/local); **demo-withdraw.json** GOA-only; shop-pay soft; stage merchant shop: `shop-ui.js` + `shops.css` + `qrcode.min.js`; cross-links local | +| **www.taler-uri-ports-** | **no default `:443`/`:80`** in public `taler://` URIs — bank/exchange/merchant `/intro/` HTML, `demo-withdraw.json`, `auto-account.json`, bank `stats.json` (ERROR on GOA/local). Unit: `tests/test_normalize_taler_uri.sh` | | **www.qr-** | QR payloads: harvest `taler://` / `payto://` / app `data-qr-url` from landings + mint JSON; **form** check; **qrencode → zbarimg** exact roundtrip; optional static QR images. Needs `qrencode` + `zbar-tools`. Skip: `QR_CHECK=0` | **QR rule:** `taler://withdraw/HOST/taler-integration/UUID` (no default `:443`/`:80`); `taler://pay/` / `pay-template/`; `payto://…` shape OK; decoded PNG must equal payload. Form errors on withdraw/pay/payto are **ERROR** on local/GOA. +**Default port rule:** public and automation URIs must never keep libeufin-style `host:443` / `host:80`. Consumers use `normalize_taler_uri` (lib.sh); bank landing API normalizes in `demo-withdraw-api.py`. + **Legal docs rule:** HTTP 200, non-empty body, not plain `not configured`, not merchant API JSON `code:21`. Local stack may require content needle (terms/privacy/FADP/GOA…). **Performance rule:** Measured from the **monitoring runner** (public via Caddy), not container loopback. Latency (ms) on every perf line + **perf summary** (n / min / p50 / avg / max). diff --git a/VERSION b/VERSION index ac1df3f..27ddcc1 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.23.3 +1.23.4 diff --git a/VERSIONS.md b/VERSIONS.md index 32c243b..1da8044 100644 --- a/VERSIONS.md +++ b/VERSIONS.md @@ -17,6 +17,7 @@ Git tags: `vMAJOR.FEATURE.FIX` (e.g. `v1.8.0`). File `VERSION` omits the `v` pre | Tag | Date (UTC) | Notes | |-----|------------|--------| +| **v1.23.4** | 2026-07-19 | **Bugfix / mon:** shared **normalize_taler_uri** (strip taler:// host :443/:80); ladder/e2e/pay use it. **urls** scans bank+exchange+merchant intros + demo-withdraw/auto-account/stats for default ports (**ERROR** on GOA). Unit test tests/test_normalize_taler_uri.sh. | | **v1.23.3** | 2026-07-19 | **Bugfix:** payments stale because max-ladder never got coins — root cause on koopa was **taler-wallet-cli missing** (all mint-OK → SKIP_ACCEPT). Hard-fail ladder prereq if CLI/helper missing; settle keeps polling after transfer_done + short run-pending; pay wait default 180s; max-ladder wrapper sets PATH/WALLET_CLI. | | **v1.23.2** | 2026-07-19 | **Bugfix:** max-ladder host-agent forces **RUN_TIMEOUT=10800** (koopa env had 600 → run aborted mid-ladder); install starts oneshots with **--no-block**. | | **v1.23.1** | 2026-07-19 | **Fix / scope:** only **`/monitoring-max-ladder/`** for GOA (drop classic `/monitoring-ladder`). Daily timer + install first-run. monpages catalog/Caddy bare redirs for max-ladder only. | diff --git a/check_amount_ladder.sh b/check_amount_ladder.sh index 67b6001..2f5da98 100755 --- a/check_amount_ladder.sh +++ b/check_amount_ladder.sh @@ -671,7 +671,12 @@ ladder_withdraw_rung() { "${BANK}/accounts/${EXP_USER}/withdrawals") ms_mint=$(elapsed_ms "$t0") WID=$(python3 -c 'import json;d=json.load(open("'"$SCRATCH"'/wd-'"$tag"'.json"));print(d.get("withdrawal_id") or "")' 2>/dev/null || true) - URI=$(python3 -c 'import json;u=json.load(open("'"$SCRATCH"'/wd-'"$tag"'.json")).get("taler_withdraw_uri") or "";print(u.replace(":443/","/"))' 2>/dev/null || true) + # libeufin often emits taler://withdraw/host:443/… — strip default ports (lib.sh) + URI=$(python3 -c 'import json;print(json.load(open("'"$SCRATCH"'/wd-'"$tag"'.json")).get("taler_withdraw_uri") or "")' 2>/dev/null || true) + URI=$(normalize_taler_uri "$URI") + if taler_uri_has_default_port "$URI"; then + warn bank "withdraw URI still has default port" "problem: $URI" + fi # numeric amount (for zero / settle / ceiling special-cases) AMT_NUM=$(python3 -c 'import sys; print(sys.argv[1].split(":",1)[-1])' "$AMT") IS_ZERO=0 diff --git a/check_e2e.sh b/check_e2e.sh index d219c73..9e20c62 100755 --- a/check_e2e.sh +++ b/check_e2e.sh @@ -944,7 +944,7 @@ e2e_one_withdraw() { "$BANK/accounts/${USER}/withdrawals" WID=$(python3 -c 'import json;d=json.load(open("'"$SCRATCH"'/wd-'"$tag"'.json"));print(d.get("withdrawal_id") or d.get("id") or "")' 2>/dev/null || true) URI=$(python3 -c 'import json;d=json.load(open("'"$SCRATCH"'/wd-'"$tag"'.json"));print(d.get("taler_withdraw_uri") or "")' 2>/dev/null || true) - URI_CLEAN=$(python3 -c 'import json;u=json.load(open("'"$SCRATCH"'/wd-'"$tag"'.json")).get("taler_withdraw_uri") or "";print(u.replace(":443/","/"))' 2>/dev/null || true) + URI_CLEAN=$(normalize_taler_uri "$URI") if [ -z "$WID" ] || [ -z "$URI_CLEAN" ]; then warn "Bank withdraw $WITHDRAW_AMT" "create failed — $(head -c 80 "$SCRATCH/wd-$tag.json" | tr '\n' ' ')" return 1 @@ -952,13 +952,13 @@ e2e_one_withdraw() { ok "Bank withdrawal created $WITHDRAW_AMT ($WID)" cp "$SCRATCH/wd-$tag.json" "$SCRATCH/wd.json" - USE_URI="$URI" - [ -z "$USE_URI" ] && USE_URI="$URI_CLEAN" + # Always prefer stripped URI (libeufin often emits host:443) + USE_URI="${URI_CLEAN:-$URI}" if wcli withdraw accept-uri --exchange "${EXCHANGE_PUBLIC}/" "$USE_URI" >"$SCRATCH/accept-$tag.out" 2>&1; then ok "wallet accept $WITHDRAW_AMT" - elif [ "$USE_URI" != "$URI_CLEAN" ] && [ -n "$URI_CLEAN" ] \ - && wcli withdraw accept-uri --exchange "${EXCHANGE_PUBLIC}/" "$URI_CLEAN" >"$SCRATCH/accept-$tag.out" 2>&1; then - ok "wallet accept $WITHDRAW_AMT (no :443)" + elif [ -n "$URI" ] && [ "$USE_URI" != "$URI" ] \ + && wcli withdraw accept-uri --exchange "${EXCHANGE_PUBLIC}/" "$URI" >"$SCRATCH/accept-$tag.out" 2>&1; then + ok "wallet accept $WITHDRAW_AMT (raw URI fallback)" else # Prefer the real Error/ENOENT line over noise (perl -e source, stack frames) _acc_err=$( diff --git a/check_urls.sh b/check_urls.sh index 472aba5..f2557ff 100755 --- a/check_urls.sh +++ b/check_urls.sh @@ -1339,6 +1339,90 @@ if [ "${LOCAL_STACK:-1}" = "1" ] || [ -n "${BANK_PUBLIC:-}" ]; then fi fi +# --------------------------------------------------------------------------- +# taler:// URIs must not carry default ports :443 / :80 (wallet / QR rule) +# Scans public landings on this stack (bank + exchange + merchant intros, +# bank mint JSON). Fail hard on GOA/local; soft warn elsewhere. +# --------------------------------------------------------------------------- +set_group landings +section "www · taler:// default ports (:443/:80 must be absent)" +_port_scan_fail=0 +_port_scan_ok=0 +_port_scan_one() { + local label="$1" url="$2" + local f code n detail + f="$tmp/portscan-$$.body" + code=$(http_body "$url" "$f") + if [ "$code" != "200" ]; then + if [ "${LOCAL_STACK:-1}" = "1" ] || [ "${EXPECT_CURRENCY:-}" = "GOA" ]; then + # mint JSON is hard; HTML intros soft if missing + case "$url" in + *demo-withdraw.json|*auto-account.json) + fail "taler-uri ports · $label" "HTTP $code · $url" + _port_scan_fail=$((_port_scan_fail + 1)) + ;; + *) + warn "taler-uri ports · $label" "HTTP $code · skip · $url" + ;; + esac + else + info "taler-uri ports · $label" "HTTP $code · skip" + fi + rm -f "$f" + return 0 + fi + detail=$(python3 - "$f" <<'PY' 2>/dev/null || true +import re, sys, json +from pathlib import Path +raw = Path(sys.argv[1]).read_text(errors="replace") +# collect taler:// from JSON fields and raw HTML/JS +uris = set(re.findall(r"taler://[A-Za-z0-9._~:/?#\[\]@!$&'()*+,;=%-]+", raw)) +# JSON-escaped sequences +uris.update(re.findall(r"taler:\\/\\/[A-Za-z0-9._~:/?#\[\]@!$&'()*+,;=%\\-]+", raw)) +uris = {u.replace("\\/", "/") for u in uris} +bad = [] +pat = re.compile(r"taler://\S+:(443|80)(/|$|\?|#)", re.I) +for u in sorted(uris): + if pat.search(u): + bad.append(u[:120]) +if bad: + print("BAD\t%d\t%s" % (len(bad), bad[0])) +else: + print("OK\t%d\t" % len(uris)) +PY +) + n_kind=${detail%%$'\t'*} + rest=${detail#*$'\t'} + n_count=${rest%%$'\t'*} + sample=${rest#*$'\t'} + if [ "$n_kind" = "BAD" ]; then + if [ "${LOCAL_STACK:-1}" = "1" ] || [ "${EXPECT_CURRENCY:-}" = "GOA" ]; then + fail "taler-uri ports · $label" "default :443/:80 still present · n=${n_count} e.g. ${sample}" + _port_scan_fail=$((_port_scan_fail + 1)) + else + warn "taler-uri ports · $label" "default port in URI · ${sample}" + fi + else + ok "taler-uri ports · $label" "no default ports · taler_uris=${n_count:-0} · $url" + _port_scan_ok=$((_port_scan_ok + 1)) + fi + rm -f "$f" +} +if [ "${CHECK_LANDING:-1}" = "1" ]; then + _port_scan_one "bank intro HTML" "${BANK_PUBLIC}/intro/" + _port_scan_one "bank demo-withdraw.json" "${BANK_PUBLIC}/intro/demo-withdraw.json" + _port_scan_one "bank auto-account.json" "${BANK_PUBLIC}/intro/auto-account.json" + _port_scan_one "exchange intro HTML" "${EXCHANGE_PUBLIC}/intro/" + _port_scan_one "merchant intro HTML" "${MERCHANT_PUBLIC}/intro/" + # stats may embed demo withdraw URI + _port_scan_one "bank stats.json" "${BANK_PUBLIC}/intro/stats.json" + if [ "$_port_scan_fail" -eq 0 ]; then + ok "taler-uri ports summary" "clean · ${_port_scan_ok} resource(s) checked (bank+exchange+merchant)" + fi +else + info "taler-uri ports" "skipped (CHECK_LANDING=0)" +fi + # Merchant shop assets — GOA: shop-pay.*; stage TESTPAYSAN: /assets/shop-ui.js + # shops.css + QR encoder (shop-ui modal needs /intro/qrcode.min.js or /qrcode.min.js). _ma=0 diff --git a/ladder/lib_pay.sh b/ladder/lib_pay.sh index 6565033..01ba819 100644 --- a/ladder/lib_pay.sh +++ b/ladder/lib_pay.sh @@ -67,7 +67,7 @@ except Exception: ' "$SCRATCH/ord-$ptag.json" 2>/dev/null || true) if [ -n "$OID" ] && [ -n "$OTOK" ]; then PAYURI="taler://pay/${mh}/instances/${inst}/${OID}/?c=${OTOK}" - PAYURI=$(printf '%s' "$PAYURI" | sed 's/:443\//\//g; s/:443?/?/g') + PAYURI=$(normalize_taler_uri "$PAYURI") return 0 fi fi @@ -98,7 +98,7 @@ except Exception: print("") if [ -z "$PAYURI" ] && [ -n "$OTOK" ]; then PAYURI="taler://pay/${mh}/instances/${INST}/${OID}/?c=${OTOK}" fi - PAYURI=$(printf '%s' "$PAYURI" | sed 's/:443\//\//g; s/:443?/?/g') + PAYURI=$(normalize_taler_uri "$PAYURI") [ -n "$PAYURI" ] } diff --git a/lib.sh b/lib.sh index 9ced76b..9187afd 100755 --- a/lib.sh +++ b/lib.sh @@ -1504,6 +1504,53 @@ load_monitoring_secrets_env() { export E2E_BANK_ADMIN_PASS E2E_MERCHANT_TOKEN SECRETS_ROOT KOOPA_ADMIN_SECRETS } +# Strip default HTTPS/HTTP ports from taler:// URIs (libeufin often emits host:443). +# Wallets and mon QR rules reject default :443/:80 in the authority. +# Usage: normalize_taler_uri "taler://withdraw/host:443/…" → prints cleaned URI +normalize_taler_uri() { + local u="${1-}" + if [ -z "$u" ]; then + return 0 + fi + printf '%s' "$u" | python3 -c ' +import re, sys +u = sys.stdin.read().strip() +# any taler://SCHEME/HOST:443|80/... +u = re.sub( + r"(taler://[A-Za-z0-9._-]+/)([^/?#:]+):443(?=/|$|\?|#)", + r"\1\2", + u, + flags=re.I, +) +u = re.sub( + r"(taler://[A-Za-z0-9._-]+/)([^/?#:]+):80(?=/|$|\?|#)", + r"\1\2", + u, + flags=re.I, +) +# leftover path/query forms +for a, b in ( + (":443/", "/"), + (":443?", "?"), + (":443#", "#"), + (":80/", "/"), + (":80?", "?"), + (":80#", "#"), +): + u = u.replace(a, b) +# bare trailing :443 / :80 on host (no slash) +u = re.sub(r"(taler://[A-Za-z0-9._-]+/[^/?#:]+):443$", r"\1", u, flags=re.I) +u = re.sub(r"(taler://[A-Za-z0-9._-]+/[^/?#:]+):80$", r"\1", u, flags=re.I) +sys.stdout.write(u) +' 2>/dev/null || printf '%s' "$u" +} + +# True (exit 0) if URI still has default port in taler:// authority. +taler_uri_has_default_port() { + # host:443 or host:80 before next path/query/end + python3 -c "import re,sys; u=sys.argv[1] if len(sys.argv)>1 else ''; sys.exit(0 if re.search(r'taler://\\S+:(443|80)(/|$|\\?|#)', u, re.I) else 1)" "${1-}" 2>/dev/null +} + find_wallet_cli() { # Must return a path suitable for: node "$WALLET_CLI" … # Prefer *.mjs / bundled entry; plain /usr/bin/taler-wallet-cli is often a diff --git a/tests/test_normalize_taler_uri.sh b/tests/test_normalize_taler_uri.sh new file mode 100755 index 0000000..1db180b --- /dev/null +++ b/tests/test_normalize_taler_uri.sh @@ -0,0 +1,73 @@ +#!/usr/bin/env bash +# Unit tests for normalize_taler_uri / taler_uri_has_default_port (lib.sh). +# Run: ./tests/test_normalize_taler_uri.sh +set -euo pipefail +ROOT=$(cd "$(dirname "$0")/.." && pwd) +# shellcheck source=../lib.sh +source "$ROOT/lib.sh" + +pass=0 +fail=0 +assert_eq() { + local name="$1" got="$2" want="$3" + if [ "$got" = "$want" ]; then + pass=$((pass + 1)) + else + echo "FAIL $name" >&2 + echo " got: $got" >&2 + echo " want: $want" >&2 + fail=$((fail + 1)) + fi +} +assert_true() { + local name="$1" + if "$@"; then + # shift name + : + fi +} + +# --- strip :443 / :80 --- +assert_eq "withdraw :443/" \ + "$(normalize_taler_uri 'taler://withdraw/bank.hacktivism.ch:443/taler-integration/abc-123')" \ + "taler://withdraw/bank.hacktivism.ch/taler-integration/abc-123" + +assert_eq "withdraw :443 end" \ + "$(normalize_taler_uri 'taler://withdraw/bank.example:443')" \ + "taler://withdraw/bank.example" + +assert_eq "pay :443" \ + "$(normalize_taler_uri 'taler://pay/taler.hacktivism.ch:443/instances/x/orders/y?c=z')" \ + "taler://pay/taler.hacktivism.ch/instances/x/orders/y?c=z" + +assert_eq "pay-template :80" \ + "$(normalize_taler_uri 'taler://pay-template/shop.example:80/instances/i/t')" \ + "taler://pay-template/shop.example/instances/i/t" + +assert_eq "already clean" \ + "$(normalize_taler_uri 'taler://withdraw/bank.hacktivism.ch/taler-integration/uuid')" \ + "taler://withdraw/bank.hacktivism.ch/taler-integration/uuid" + +assert_eq "empty" "$(normalize_taler_uri '')" "" + +# non-default port kept +assert_eq "keep :8443" \ + "$(normalize_taler_uri 'taler://withdraw/bank.example:8443/taler-integration/x')" \ + "taler://withdraw/bank.example:8443/taler-integration/x" + +# has_default_port detector +if taler_uri_has_default_port 'taler://withdraw/h:443/p'; then + pass=$((pass + 1)) +else + echo "FAIL detector should match :443" >&2 + fail=$((fail + 1)) +fi +if taler_uri_has_default_port 'taler://withdraw/h/p'; then + echo "FAIL detector should not match clean URI" >&2 + fail=$((fail + 1)) +else + pass=$((pass + 1)) +fi + +echo "normalize_taler_uri tests: pass=$pass fail=$fail" +[ "$fail" -eq 0 ]