diff --git a/CLI-AUTOMATION-NOTES.md b/CLI-AUTOMATION-NOTES.md index ae15ba5..057cef2 100644 --- a/CLI-AUTOMATION-NOTES.md +++ b/CLI-AUTOMATION-NOTES.md @@ -53,8 +53,8 @@ Legend: |--|--| | **Seen** | ladder force-select, e2e confirm path, Android spinner issues; **repro 2026-07-19 magikoopa** GOA ladder: `force-select skipped · no reserve_pub for this withdraw (WID=…); wallet may not have selected yet` | | **Area** | **cli** / **core** / **bank** / **ops** | -| **Problem** | (a) Cumulative wallets re-emit old reserves; binding the wrong `reserve_pub` yields **HTTP 409 / code 5114** (“already bound”). (b) Automation scrapes accept/tx dumps for candidates — when the dump has **no** pub at all, ladder WARNs and skips force-select (soft), then often **SKIP_CONFIRM**. (c) Empty scrape is **not** the same as “bank out of money”; do not treat as balance failure. | -| **Workaround** | Score pubs by WID/amount; try several; treat 5114 as “stale reserve” not “out of money”. Refresh `transactions` before each force-select try. Soft-skip confirm if still pending after polls. Fix wallet DB / helper first if accept/tx only print FATAL (see §15–16) — otherwise scrape can never find a pub. | +| **Problem** | (a) Cumulative wallets re-emit old reserves; binding the wrong `reserve_pub` yields **HTTP 409 / code 5114** (“already bound”). (b) Automation scrapes accept/tx dumps for candidates — when the dump has **no** pub at all, ladder WARNs and skips force-select (soft), then often **SKIP_CONFIRM**. (c) Empty scrape is **not** the same as “bank out of money”; do not treat as balance failure. (d) **Mon bug (fixed v1.20.2):** extract used `json.loads(file_from_first_brace)` on wallet dumps that append log lines after JSON → **zero** objects parsed; regex only scanned accept (which has no `reservePub`) while the pub lives under `transactions[].withdrawalDetails.reservePub`. | +| **Workaround** | **Mon ≥1.20.2:** `JSONDecoder.raw_decode` + regex on accept **and** transactions; score by WID in `bankConfirmationUrl` / amount. Still: score pubs by WID/amount; try several; treat 5114 as “stale reserve” not “out of money”. Refresh `transactions` before each force-select try. Soft-skip confirm if still pending after polls. Fix wallet DB / helper first if accept/tx only print FATAL (see §15–16). | | **Wanted** | CLI returns **the** `reserve_pub` for the just-accepted withdraw in structured JSON; `force-select` / bank-side select API documented with idempotency. Optional: bank integration already exposes `selected_reserve_pub` when wallet selected — prefer that over scraping wallet logs. | ### 3a. Ladder pin amounts hide mid-range behaviour @@ -63,8 +63,8 @@ Legend: |--|--| | **Seen** | `LADDER_STEPS=3` plan collapses to pins only: `GOA:0`, `max-1`, `max` (no random mid rungs) | | **Area** | **ops** / **bank** / **cli** | -| **Problem** | **Zero** withdraw often never produces a usable `reserve_pub` / coins (wallet 7006 or no denoms). **Absolute max** often **mint HTTP 500 / code 5110** SQL `P0001` (ceiling probe — expected soft on max pin). Reproducing “real” force-select needs a **mid** amount (e.g. `GOA:1` … exchange min denom scale), not only pins. | -| **Workaround** | Use `LADDER_STEPS≥5` (or higher) so random mids exist; treat zero / absolute-max as probes. Soft-skip zero (7006) and max ceiling (5110). | +| **Problem** | **Zero** withdraw often never produces a usable `reserve_pub` / coins (wallet 7006 or no denoms). **Absolute max** and sometimes **max-1** hit **mint HTTP 500 / code 5110** SQL `P0001` (ceiling probe). Reproducing “real” force-select needs a **mid** amount (e.g. `GOA:1` … exchange min denom scale), not only pins. | +| **Workaround** | Use `LADDER_STEPS≥5` (or higher) so random mids exist; treat zero / absolute-max / max-1-5110 as probes (soft `CEILING_REJECT` in mon ≥1.20.2). | | **Wanted** | Ladder plan docs: pin vs mid semantics; CLI structured error for amount-too-large / zero. | --- diff --git a/VERSION b/VERSION index 0044d6c..769e37e 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.20.1 +1.20.2 diff --git a/VERSIONS.md b/VERSIONS.md index 263ffcc..ec8e050 100644 --- a/VERSIONS.md +++ b/VERSIONS.md @@ -17,6 +17,7 @@ Git tags: `vMAJOR.FEATURE.FIX` (e.g. `v1.8.0`). File `VERSION` omits the `v` pre | Tag | Date (UTC) | Notes | |-----|------------|--------| +| **v1.20.2** | 2026-07-19 | **Bugfix:** ladder force-select empty `reserve_pub` scrape — wallet dumps mix log lines + JSON so `json.loads(rest_of_file)` always failed; extract now uses `JSONDecoder.raw_decode` and regex on **both** accept + transactions (`reservePub` is in tx). Prefer Homebrew python on macOS for helper; `find_wallet_cli` prefers ts-core 1.6.x over stale system wrappers. Soft-skip max-1 mint 5110/P0001 as ceiling. Verified: mid rungs `force-select HTTP 200` + `confirm 204` → `OK_BANK`. | | **v1.20.1** | 2026-07-19 | **Docs:** CLI-AUTOMATION-NOTES — force-select empty scrape + ladder pins (§3/3a); wallet-db must not be `*.json` (§15); `taler-helper-sqlite3` needs Python ≥3.11 (§16); portable wall-clock (§17); GOA withdraw checklist (§18). No code change. | | **v1.20.0** | 2026-07-19 | **Feature:** domains.conf **`locale`** (l10n) — FP **`fr-CH`**, others **`de-CH`** (aliases ch-FR/ch-DE); UI **`lang=de`** supported (sticky + console tags) but **never** a stock profile default; env `TALER_MON_LOCALE` / `TALER_DOMAIN_LOCALE`. | | **v1.19.1** | 2026-07-19 | **Bugfix / docs:** DEPENDENCIES — e2e requires `taler-helper-sqlite3` on mon PATH + preflight; README domains.conf documents **`lang`** column. | diff --git a/check_goa_ladder.sh b/check_goa_ladder.sh index 410205f..560537d 100755 --- a/check_goa_ladder.sh +++ b/check_goa_ladder.sh @@ -171,10 +171,17 @@ ladder_over() { } wcli() { + # taler-helper-sqlite3 is Python ≥3.11; prefer Homebrew/local python on macOS + # so Apple /usr/bin/python3 3.9 does not FATAL the sqlite backend (no reservePub). + local _path="${PATH:-}" + case ":${_path}:" in + *:/opt/homebrew/bin:*) ;; + *) _path="/opt/homebrew/bin:/usr/local/bin:${_path}" ;; + esac if [ -n "${CLI_JS:-}" ] && [ -f "$CLI_JS" ]; then - node "$CLI_JS" --wallet-db="$WDB" --no-throttle "$@" + PATH="$_path" node "$CLI_JS" --wallet-db="$WDB" --no-throttle "$@" else - taler-wallet-cli --wallet-db="$WDB" --no-throttle "$@" + PATH="$_path" taler-wallet-cli --wallet-db="$WDB" --no-throttle "$@" fi } @@ -238,10 +245,18 @@ wallet_avail() { wcli balance 2>/dev/null | python3 -c ' import json,sys t=sys.stdin.read() +dec=json.JSONDecoder() i=t.find("{") if i<0: print("0"); raise SystemExit -d=json.loads(t[i:t.rfind("}")+1]) +try: + d,_=dec.raw_decode(t,i) +except Exception: + # last-resort: brace slice (may fail on trailing logs) + try: + d=json.loads(t[i:t.rfind("}")+1]) + except Exception: + print("0"); raise SystemExit cur=sys.argv[1] for b in d.get("balances") or []: a=b.get("available") or "" @@ -587,6 +602,16 @@ for AMT in "$@"; do echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" continue fi + # max-1 can also hit the same libeufin SQL ceiling (5110/P0001) on live GOA — + # treat as soft ceiling when mid rungs already passed; do not block the report. + if [ "$IS_MAX_M1_PIN" = "1" ] && echo "$note" | grep -qE '5110|P0001'; then + status="CEILING_REJECT" + note="max-1 rejected (same ceiling as absolute max): $note" + warn bank "mint $AMT rejected (max-1 ceiling)" \ + "problem: bank rejects max-1 with SQL P0001/5110 (pool ceiling). Mid-rung OK_BANK/OK still count. Ladder continues. detail: $note" + echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" + continue + fi err bank "mint $AMT failed" "$note" status="FAIL_MINT" STOP_REASON="$note" @@ -648,6 +673,10 @@ for AMT in "$@"; do # Collect reserve_pub candidates for *this* withdrawal (WID + amount). # Cumulative wallets re-print old reserves in accept/tx dumps — never trust a single "last" blindly. # Prints unique pubs one per line, preferred order first. + # + # Critical: wallet-cli dumps mix log lines + JSON. Never json.loads(rest_of_file) — + # trailing "Shutdown requested" lines make that always fail. Use JSONDecoder.raw_decode + # and regex on *both* accept and transactions output (reservePub lives in tx, not accept). extract_rpubs_for_wid() { python3 - "$WID" "$AMT" "$SCRATCH/accept-$tag.out" "$SCRATCH/tx-$tag.json" "$SCRATCH/used-rpubs.txt" <<'PY' import json, re, sys @@ -661,99 +690,118 @@ used = set() if Path(used_path).is_file(): used = {ln.strip() for ln in open(used_path) if ln.strip()} +amt_num = amt.split(":", 1)[-1] if amt else "" +dec = json.JSONDecoder() + def walk_collect(o, bag, ctx=None): ctx = dict(ctx or {}) if isinstance(o, dict): + # First pass: sibling scalars (reservePub + bankConfirmationUrl + amounts + # are peers under withdrawalDetails — order in JSON must not matter). for k, v in o.items(): - kl = str(k).lower() - if kl in ("withdrawal_id", "withdraw_id", "wopid", "id") and isinstance(v, str): + if not isinstance(v, str): + continue + kl = str(k).lower().replace("-", "_") + if kl in ( + "withdrawal_id", "withdraw_id", "wopid", "id", + "transactionid", "transaction_id", + ): ctx["id"] = v - if kl in ("amount", "rawamount", "instructedamount") and isinstance(v, str): + elif kl in ( + "amount", "rawamount", "instructedamount", + "amounteffective", "amountraw", "transferamount", + ): ctx["amount"] = v - if kl in ("taler_withdraw_uri", "talerwithdrawuri", "uri") and isinstance(v, str): + elif kl in ( + "taler_withdraw_uri", "talerwithdrawuri", "uri", + "bankconfirmationurl", "confirmtransferurl", + ): ctx["uri"] = v + for k, v in o.items(): + kl = str(k).lower().replace("-", "_") if kl in ("reserve_pub", "reservepub") and isinstance(v, str) and len(v) >= 40: bag.append((v, dict(ctx))) - walk_collect(v, bag, ctx) + elif not isinstance(v, (str, int, float, bool)) and v is not None: + walk_collect(v, bag, ctx) elif isinstance(o, list): for i in o: walk_collect(i, bag, ctx) -raw_pubs = [] # ordered as found -scored = [] # (score, pub) higher better -blob_all = "" -for p in paths: - try: - blob_all += open(p, errors="replace").read() + "\n" - except Exception: - pass +def score_ctx(pub, ctx, window=""): + score = 0 + cid = str(ctx.get("id") or "") + camt = str(ctx.get("amount") or "") + curi = str(ctx.get("uri") or "") + # bankConfirmationUrl / withdraw URI embeds this op's WID + if wid and wid in curi: + score += 120 + if wid and wid in cid: + score += 100 + if wid and wid in window: + score += 80 + if amt and (camt == amt or camt.endswith(amt_num)): + score += 50 + if amt and amt in window: + score += 30 + if pub in used: + score -= 200 + return score -# 1) full JSON objects in files +def iter_json_objects(text): + """Yield top-level JSON objects from mixed log+JSON wallet dumps.""" + i, n = 0, len(text) + while i < n: + if text[i] == "{": + try: + o, end = dec.raw_decode(text, i) + yield o + i = end + continue + except Exception: + pass + i += 1 + +raw_pubs = [] +scored = [] + +# 1) structured JSON via raw_decode (accept + transactions) for p in paths: try: t = open(p, errors="replace").read() except Exception: continue - # whole-file JSON - for m in re.finditer(r"\{", t): - try: - o = json.loads(t[m.start():]) - except Exception: - continue + for o in iter_json_objects(t): bag = [] walk_collect(o, bag) for pub, ctx in bag: raw_pubs.append(pub) - score = 0 - cid = str(ctx.get("id") or "") - camt = str(ctx.get("amount") or "") - curi = str(ctx.get("uri") or "") - if wid and wid in cid: - score += 100 - if wid and wid in curi: - score += 80 - if amt and (camt == amt or camt.endswith(amt.split(":", 1)[-1])): - score += 40 - if pub in used: - score -= 200 - scored.append((score, pub)) + scored.append((score_ctx(pub, ctx), pub)) -# 2) regex fallback on accept file only (more current) -try: - acc = open(paths[0], errors="replace").read() -except Exception: - acc = "" -for pat in ( - r"\"reserve_pub\"\s*:\s*\"([A-Z0-9]{40,})\"", +# 2) regex on *both* accept and tx (reservePub is normally only in transactions) +RPUB_PATS = ( r"\"reservePub\"\s*:\s*\"([A-Z0-9]{40,})\"", - r"reserve_pub[\"\s:=]+([A-Z0-9]{40,})", -): - for m in re.finditer(pat, acc, re.I): - pub = m.group(1) - raw_pubs.append(pub) - score = 10 - # proximity to WID in accept output - window = acc[max(0, m.start() - 400) : m.end() + 400] - if wid and wid in window: - score += 100 - if amt and amt in window: - score += 30 - if pub in used: - score -= 200 - scored.append((score, pub)) + r"\"reserve_pub\"\s*:\s*\"([A-Z0-9]{40,})\"", + r"reserve[_ ]?pub[\"\s:=]+([A-Z0-9]{40,})", +) +for p in paths: + try: + blob = open(p, errors="replace").read() + except Exception: + continue + for pat in RPUB_PATS: + for m in re.finditer(pat, blob, re.I): + pub = m.group(1) + raw_pubs.append(pub) + window = blob[max(0, m.start() - 600) : m.end() + 600] + scored.append((score_ctx(pub, {}, window) + 10, pub)) -# prefer high score, then later occurrence order = [] seen = set() -for score, pub in sorted(scored, key=lambda x: (-x[0],), reverse=False): - # sort by score desc: use reverse sorted - pass for score, pub in sorted(scored, key=lambda x: x[0], reverse=True): if pub in seen or pub in used: continue seen.add(pub) order.append(pub) -# append unused raw in reverse (newest-ish) for pub in reversed(raw_pubs): if pub in seen or pub in used: continue diff --git a/lib.sh b/lib.sh index 0f92126..9ced76b 100755 --- a/lib.sh +++ b/lib.sh @@ -1512,12 +1512,20 @@ find_wallet_cli() { echo "$WALLET_CLI"; return 0 fi local c cand - # WALLET_CLI / WALLET_CLI_CANDIDATES from env; then portable package paths only. + # WALLET_CLI / WALLET_CLI_CANDIDATES from env; then portable package paths. + # Prefer local ts-core / mon-env builds (schema-compatible) over stale system + # wrappers (e.g. /usr/local 1.3.x vs DB schema 29 from 1.6.x). # shellcheck disable=SC2086 for c in ${WALLET_CLI_CANDIDATES:-} \ "${HOME}/.local/bin/taler-wallet-cli.mjs" \ + "${HOME}/src/taler/taler-typescript-core/packages/taler-wallet-cli/bin/taler-wallet-cli.mjs" \ + "${HOME}/src/taler/taler-typescript-core/packages/taler-wallet-cli/dist/taler-wallet-cli.mjs" \ + "${HOME}/src/taler/taler-typescript-core/packages/taler-wallet-cli/dist/taler-wallet-cli-local.mjs" \ + "${HOME}/src/taler/taler-typescript-core/packages/taler-wallet-cli/dist/taler-wallet-cli-bundled.cjs" \ /usr/lib/taler-wallet-cli/node_modules/taler-wallet-cli/bin/taler-wallet-cli.mjs \ - /usr/share/taler-wallet-cli/bin/taler-wallet-cli.mjs + /usr/share/taler-wallet-cli/bin/taler-wallet-cli.mjs \ + /usr/local/lib/taler-wallet-cli/node_modules/taler-wallet-cli/bin/taler-wallet-cli.mjs \ + /usr/local/lib/taler-wallet-cli/node_modules/taler-wallet-cli/dist/taler-wallet-cli-bundled.cjs do [ -n "$c" ] && [ -f "$c" ] && { echo "$c"; return 0; } done @@ -1533,6 +1541,14 @@ find_wallet_cli() { c=$(readlink -f "$cand" 2>/dev/null || true) [ -n "$c" ] && [ -f "$c" ] && { echo "$c"; return 0; } fi + # macOS /usr/local wrapper often runs an older bundle; prefer its bundled.cjs only + # when no newer candidate above matched (already returned). + for c in \ + /usr/local/lib/taler-wallet-cli/node_modules/taler-wallet-cli/dist/taler-wallet-cli-bundled.cjs \ + /usr/local/lib/node_modules/taler-wallet-cli/dist/taler-wallet-cli-bundled.cjs + do + [ -f "$c" ] && { echo "$c"; return 0; } + done fi return 1 }