fix: ladder force-select reserve_pub extract (1.20.2)

Wallet accept/tx dumps mix log lines with JSON; json.loads from the
first brace always failed, and regex only scanned accept (no reservePub).
Parse with JSONDecoder.raw_decode, scrape both files, score by WID/amount.
Prefer Homebrew python for taler-helper-sqlite3; find_wallet_cli prefers
ts-core 1.6.x. Soft-skip max-1 mint 5110/P0001 as ceiling.

Verified on live GOA: mid rungs force-select HTTP 200 + confirm 204 → OK_BANK.

CLI-AUTOMATION-NOTES.md

@@ -53,8 +53,8 @@ Legend:
 |--|--|
 | **Seen** | ladder force-select, e2e confirm path, Android spinner issues; **repro 2026-07-19 magikoopa** GOA ladder: `force-select skipped · no reserve_pub for this withdraw (WID=…); wallet may not have selected yet` |
 | **Area** | **cli** / **core** / **bank** / **ops** |
-| **Problem** | (a) Cumulative wallets re-emit old reserves; binding the wrong `reserve_pub` yields **HTTP 409 / code 5114** (“already bound”). (b) Automation scrapes accept/tx dumps for candidates — when the dump has **no** pub at all, ladder WARNs and skips force-select (soft), then often **SKIP_CONFIRM**. (c) Empty scrape is **not** the same as “bank out of money”; do not treat as balance failure. |
+| **Problem** | (a) Cumulative wallets re-emit old reserves; binding the wrong `reserve_pub` yields **HTTP 409 / code 5114** (“already bound”). (b) Automation scrapes accept/tx dumps for candidates — when the dump has **no** pub at all, ladder WARNs and skips force-select (soft), then often **SKIP_CONFIRM**. (c) Empty scrape is **not** the same as “bank out of money”; do not treat as balance failure. (d) **Mon bug (fixed v1.20.2):** extract used `json.loads(file_from_first_brace)` on wallet dumps that append log lines after JSON → **zero** objects parsed; regex only scanned accept (which has no `reservePub`) while the pub lives under `transactions[].withdrawalDetails.reservePub`. |
-| **Workaround** | Score pubs by WID/amount; try several; treat 5114 as “stale reserve” not “out of money”. Refresh `transactions` before each force-select try. Soft-skip confirm if still pending after polls. Fix wallet DB / helper first if accept/tx only print FATAL (see §15–16) — otherwise scrape can never find a pub. |
+| **Workaround** | **Mon ≥1.20.2:** `JSONDecoder.raw_decode` + regex on accept **and** transactions; score by WID in `bankConfirmationUrl` / amount. Still: score pubs by WID/amount; try several; treat 5114 as “stale reserve” not “out of money”. Refresh `transactions` before each force-select try. Soft-skip confirm if still pending after polls. Fix wallet DB / helper first if accept/tx only print FATAL (see §15–16). |
 | **Wanted** | CLI returns **the** `reserve_pub` for the just-accepted withdraw in structured JSON; `force-select` / bank-side select API documented with idempotency. Optional: bank integration already exposes `selected_reserve_pub` when wallet selected — prefer that over scraping wallet logs. |
 
 ### 3a. Ladder pin amounts hide mid-range behaviour
@@ -63,8 +63,8 @@ Legend:
 |--|--|
 | **Seen** | `LADDER_STEPS=3` plan collapses to pins only: `GOA:0`, `max-1`, `max` (no random mid rungs) |
 | **Area** | **ops** / **bank** / **cli** |
-| **Problem** | **Zero** withdraw often never produces a usable `reserve_pub` / coins (wallet 7006 or no denoms). **Absolute max** often **mint HTTP 500 / code 5110** SQL `P0001` (ceiling probe — expected soft on max pin). Reproducing “real” force-select needs a **mid** amount (e.g. `GOA:1` … exchange min denom scale), not only pins. |
+| **Problem** | **Zero** withdraw often never produces a usable `reserve_pub` / coins (wallet 7006 or no denoms). **Absolute max** and sometimes **max-1** hit **mint HTTP 500 / code 5110** SQL `P0001` (ceiling probe). Reproducing “real” force-select needs a **mid** amount (e.g. `GOA:1` … exchange min denom scale), not only pins. |
-| **Workaround** | Use `LADDER_STEPS≥5` (or higher) so random mids exist; treat zero / absolute-max as probes. Soft-skip zero (7006) and max ceiling (5110). |
+| **Workaround** | Use `LADDER_STEPS≥5` (or higher) so random mids exist; treat zero / absolute-max / max-1-5110 as probes (soft `CEILING_REJECT` in mon ≥1.20.2). |
 | **Wanted** | Ladder plan docs: pin vs mid semantics; CLI structured error for amount-too-large / zero. |
 
 ---

VERSION

@@ -1 +1 @@
-1.20.1
+1.20.2

VERSIONS.md

@@ -17,6 +17,7 @@ Git tags: `vMAJOR.FEATURE.FIX` (e.g. `v1.8.0`). File `VERSION` omits the `v` pre
 
 | Tag | Date (UTC) | Notes |
 |-----|------------|--------|
+| **v1.20.2** | 2026-07-19 | **Bugfix:** ladder force-select empty `reserve_pub` scrape — wallet dumps mix log lines + JSON so `json.loads(rest_of_file)` always failed; extract now uses `JSONDecoder.raw_decode` and regex on **both** accept + transactions (`reservePub` is in tx). Prefer Homebrew python on macOS for helper; `find_wallet_cli` prefers ts-core 1.6.x over stale system wrappers. Soft-skip max-1 mint 5110/P0001 as ceiling. Verified: mid rungs `force-select HTTP 200` + `confirm 204` → `OK_BANK`. |
 | **v1.20.1** | 2026-07-19 | **Docs:** CLI-AUTOMATION-NOTES — force-select empty scrape + ladder pins (§3/3a); wallet-db must not be `*.json` (§15); `taler-helper-sqlite3` needs Python ≥3.11 (§16); portable wall-clock (§17); GOA withdraw checklist (§18). No code change. |
 | **v1.20.0** | 2026-07-19 | **Feature:** domains.conf **`locale`** (l10n) — FP **`fr-CH`**, others **`de-CH`** (aliases ch-FR/ch-DE); UI **`lang=de`** supported (sticky + console tags) but **never** a stock profile default; env `TALER_MON_LOCALE` / `TALER_DOMAIN_LOCALE`. |
 | **v1.19.1** | 2026-07-19 | **Bugfix / docs:** DEPENDENCIES — e2e requires `taler-helper-sqlite3` on mon PATH + preflight; README domains.conf documents **`lang`** column. |

check_goa_ladder.sh

@@ -171,10 +171,17 @@ ladder_over() {
 }
 
 wcli() {
+  # taler-helper-sqlite3 is Python ≥3.11; prefer Homebrew/local python on macOS
+  # so Apple /usr/bin/python3 3.9 does not FATAL the sqlite backend (no reservePub).
+  local _path="${PATH:-}"
+  case ":${_path}:" in
+    *:/opt/homebrew/bin:*) ;;
+    *) _path="/opt/homebrew/bin:/usr/local/bin:${_path}" ;;
+  esac
   if [ -n "${CLI_JS:-}" ] && [ -f "$CLI_JS" ]; then
-    node "$CLI_JS" --wallet-db="$WDB" --no-throttle "$@"
+    PATH="$_path" node "$CLI_JS" --wallet-db="$WDB" --no-throttle "$@"
   else
-    taler-wallet-cli --wallet-db="$WDB" --no-throttle "$@"
+    PATH="$_path" taler-wallet-cli --wallet-db="$WDB" --no-throttle "$@"
   fi
 }
 
@@ -238,10 +245,18 @@ wallet_avail() {
   wcli balance 2>/dev/null | python3 -c '
 import json,sys
 t=sys.stdin.read()
+dec=json.JSONDecoder()
 i=t.find("{")
 if i<0:
   print("0"); raise SystemExit
+try:
+  d,_=dec.raw_decode(t,i)
+except Exception:
+  # last-resort: brace slice (may fail on trailing logs)
+  try:
     d=json.loads(t[i:t.rfind("}")+1])
+  except Exception:
+    print("0"); raise SystemExit
 cur=sys.argv[1]
 for b in d.get("balances") or []:
   a=b.get("available") or ""
@@ -587,6 +602,16 @@ for AMT in "$@"; do
       echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV"
       continue
     fi
+    # max-1 can also hit the same libeufin SQL ceiling (5110/P0001) on live GOA —
+    # treat as soft ceiling when mid rungs already passed; do not block the report.
+    if [ "$IS_MAX_M1_PIN" = "1" ] && echo "$note" | grep -qE '5110|P0001'; then
+      status="CEILING_REJECT"
+      note="max-1 rejected (same ceiling as absolute max): $note"
+      warn bank "mint $AMT rejected (max-1 ceiling)" \
+        "problem: bank rejects max-1 with SQL P0001/5110 (pool ceiling). Mid-rung OK_BANK/OK still count. Ladder continues. detail: $note"
+      echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV"
+      continue
+    fi
     err bank "mint $AMT failed" "$note"
     status="FAIL_MINT"
     STOP_REASON="$note"
@@ -648,6 +673,10 @@ for AMT in "$@"; do
   # Collect reserve_pub candidates for *this* withdrawal (WID + amount).
   # Cumulative wallets re-print old reserves in accept/tx dumps — never trust a single "last" blindly.
   # Prints unique pubs one per line, preferred order first.
+  #
+  # Critical: wallet-cli dumps mix log lines + JSON. Never json.loads(rest_of_file) —
+  # trailing "Shutdown requested" lines make that always fail. Use JSONDecoder.raw_decode
+  # and regex on *both* accept and transactions output (reservePub lives in tx, not accept).
   extract_rpubs_for_wid() {
     python3 - "$WID" "$AMT" "$SCRATCH/accept-$tag.out" "$SCRATCH/tx-$tag.json" "$SCRATCH/used-rpubs.txt" <<'PY'
 import json, re, sys
@@ -661,99 +690,118 @@ used = set()
 if Path(used_path).is_file():
     used = {ln.strip() for ln in open(used_path) if ln.strip()}
 
+amt_num = amt.split(":", 1)[-1] if amt else ""
+dec = json.JSONDecoder()
+
 def walk_collect(o, bag, ctx=None):
     ctx = dict(ctx or {})
     if isinstance(o, dict):
+        # First pass: sibling scalars (reservePub + bankConfirmationUrl + amounts
+        # are peers under withdrawalDetails — order in JSON must not matter).
         for k, v in o.items():
-            kl = str(k).lower()
+            if not isinstance(v, str):
-            if kl in ("withdrawal_id", "withdraw_id", "wopid", "id") and isinstance(v, str):
+                continue
+            kl = str(k).lower().replace("-", "_")
+            if kl in (
+                "withdrawal_id", "withdraw_id", "wopid", "id",
+                "transactionid", "transaction_id",
+            ):
                 ctx["id"] = v
-            if kl in ("amount", "rawamount", "instructedamount") and isinstance(v, str):
+            elif kl in (
+                "amount", "rawamount", "instructedamount",
+                "amounteffective", "amountraw", "transferamount",
+            ):
                 ctx["amount"] = v
-            if kl in ("taler_withdraw_uri", "talerwithdrawuri", "uri") and isinstance(v, str):
+            elif kl in (
+                "taler_withdraw_uri", "talerwithdrawuri", "uri",
+                "bankconfirmationurl", "confirmtransferurl",
+            ):
                 ctx["uri"] = v
+        for k, v in o.items():
+            kl = str(k).lower().replace("-", "_")
             if kl in ("reserve_pub", "reservepub") and isinstance(v, str) and len(v) >= 40:
                 bag.append((v, dict(ctx)))
+            elif not isinstance(v, (str, int, float, bool)) and v is not None:
                 walk_collect(v, bag, ctx)
     elif isinstance(o, list):
         for i in o:
             walk_collect(i, bag, ctx)
 
-raw_pubs = []  # ordered as found
+def score_ctx(pub, ctx, window=""):
-scored = []    # (score, pub) higher better
+    score = 0
-blob_all = ""
+    cid = str(ctx.get("id") or "")
-for p in paths:
+    camt = str(ctx.get("amount") or "")
+    curi = str(ctx.get("uri") or "")
+    # bankConfirmationUrl / withdraw URI embeds this op's WID
+    if wid and wid in curi:
+        score += 120
+    if wid and wid in cid:
+        score += 100
+    if wid and wid in window:
+        score += 80
+    if amt and (camt == amt or camt.endswith(amt_num)):
+        score += 50
+    if amt and amt in window:
+        score += 30
+    if pub in used:
+        score -= 200
+    return score
+
+def iter_json_objects(text):
+    """Yield top-level JSON objects from mixed log+JSON wallet dumps."""
+    i, n = 0, len(text)
+    while i < n:
+        if text[i] == "{":
             try:
-        blob_all += open(p, errors="replace").read() + "\n"
+                o, end = dec.raw_decode(text, i)
+                yield o
+                i = end
+                continue
             except Exception:
                 pass
+        i += 1
 
-# 1) full JSON objects in files
+raw_pubs = []
+scored = []
+
+# 1) structured JSON via raw_decode (accept + transactions)
 for p in paths:
     try:
         t = open(p, errors="replace").read()
     except Exception:
         continue
-    # whole-file JSON
+    for o in iter_json_objects(t):
-    for m in re.finditer(r"\{", t):
-        try:
-            o = json.loads(t[m.start():])
-        except Exception:
-            continue
         bag = []
         walk_collect(o, bag)
         for pub, ctx in bag:
             raw_pubs.append(pub)
-            score = 0
+            scored.append((score_ctx(pub, ctx), pub))
-            cid = str(ctx.get("id") or "")
-            camt = str(ctx.get("amount") or "")
-            curi = str(ctx.get("uri") or "")
-            if wid and wid in cid:
-                score += 100
-            if wid and wid in curi:
-                score += 80
-            if amt and (camt == amt or camt.endswith(amt.split(":", 1)[-1])):
-                score += 40
-            if pub in used:
-                score -= 200
-            scored.append((score, pub))
 
-# 2) regex fallback on accept file only (more current)
+# 2) regex on *both* accept and tx (reservePub is normally only in transactions)
-try:
+RPUB_PATS = (
-    acc = open(paths[0], errors="replace").read()
-except Exception:
-    acc = ""
-for pat in (
-    r"\"reserve_pub\"\s*:\s*\"([A-Z0-9]{40,})\"",
     r"\"reservePub\"\s*:\s*\"([A-Z0-9]{40,})\"",
-    r"reserve_pub[\"\s:=]+([A-Z0-9]{40,})",
+    r"\"reserve_pub\"\s*:\s*\"([A-Z0-9]{40,})\"",
-):
+    r"reserve[_ ]?pub[\"\s:=]+([A-Z0-9]{40,})",
-    for m in re.finditer(pat, acc, re.I):
+)
+for p in paths:
+    try:
+        blob = open(p, errors="replace").read()
+    except Exception:
+        continue
+    for pat in RPUB_PATS:
+        for m in re.finditer(pat, blob, re.I):
             pub = m.group(1)
             raw_pubs.append(pub)
-        score = 10
+            window = blob[max(0, m.start() - 600) : m.end() + 600]
-        # proximity to WID in accept output
+            scored.append((score_ctx(pub, {}, window) + 10, pub))
-        window = acc[max(0, m.start() - 400) : m.end() + 400]
-        if wid and wid in window:
-            score += 100
-        if amt and amt in window:
-            score += 30
-        if pub in used:
-            score -= 200
-        scored.append((score, pub))
 
-# prefer high score, then later occurrence
 order = []
 seen = set()
-for score, pub in sorted(scored, key=lambda x: (-x[0],), reverse=False):
-    # sort by score desc: use reverse sorted
-    pass
 for score, pub in sorted(scored, key=lambda x: x[0], reverse=True):
     if pub in seen or pub in used:
         continue
     seen.add(pub)
     order.append(pub)
-# append unused raw in reverse (newest-ish)
 for pub in reversed(raw_pubs):
     if pub in seen or pub in used:
         continue

lib.sh

@@ -1512,12 +1512,20 @@ find_wallet_cli() {
     echo "$WALLET_CLI"; return 0
   fi
   local c cand
-  # WALLET_CLI / WALLET_CLI_CANDIDATES from env; then portable package paths only.
+  # WALLET_CLI / WALLET_CLI_CANDIDATES from env; then portable package paths.
+  # Prefer local ts-core / mon-env builds (schema-compatible) over stale system
+  # wrappers (e.g. /usr/local 1.3.x vs DB schema 29 from 1.6.x).
   # shellcheck disable=SC2086
   for c in ${WALLET_CLI_CANDIDATES:-} \
     "${HOME}/.local/bin/taler-wallet-cli.mjs" \
+    "${HOME}/src/taler/taler-typescript-core/packages/taler-wallet-cli/bin/taler-wallet-cli.mjs" \
+    "${HOME}/src/taler/taler-typescript-core/packages/taler-wallet-cli/dist/taler-wallet-cli.mjs" \
+    "${HOME}/src/taler/taler-typescript-core/packages/taler-wallet-cli/dist/taler-wallet-cli-local.mjs" \
+    "${HOME}/src/taler/taler-typescript-core/packages/taler-wallet-cli/dist/taler-wallet-cli-bundled.cjs" \
     /usr/lib/taler-wallet-cli/node_modules/taler-wallet-cli/bin/taler-wallet-cli.mjs \
-    /usr/share/taler-wallet-cli/bin/taler-wallet-cli.mjs
+    /usr/share/taler-wallet-cli/bin/taler-wallet-cli.mjs \
+    /usr/local/lib/taler-wallet-cli/node_modules/taler-wallet-cli/bin/taler-wallet-cli.mjs \
+    /usr/local/lib/taler-wallet-cli/node_modules/taler-wallet-cli/dist/taler-wallet-cli-bundled.cjs
   do
     [ -n "$c" ] && [ -f "$c" ] && { echo "$c"; return 0; }
   done
@@ -1533,6 +1541,14 @@ find_wallet_cli() {
       c=$(readlink -f "$cand" 2>/dev/null || true)
       [ -n "$c" ] && [ -f "$c" ] && { echo "$c"; return 0; }
     fi
+    # macOS /usr/local wrapper often runs an older bundle; prefer its bundled.cjs only
+    # when no newer candidate above matched (already returned).
+    for c in \
+      /usr/local/lib/taler-wallet-cli/node_modules/taler-wallet-cli/dist/taler-wallet-cli-bundled.cjs \
+      /usr/local/lib/node_modules/taler-wallet-cli/dist/taler-wallet-cli-bundled.cjs
+    do
+      [ -f "$c" ] && { echo "$c"; return 0; }
+    done
   fi
   return 1
 }