fix: ladder force-select reserve_pub extract (1.20.2)
Wallet accept/tx dumps mix log lines with JSON; json.loads from the first brace always failed, and regex only scanned accept (no reservePub). Parse with JSONDecoder.raw_decode, scrape both files, score by WID/amount. Prefer Homebrew python for taler-helper-sqlite3; find_wallet_cli prefers ts-core 1.6.x. Soft-skip max-1 mint 5110/P0001 as ceiling. Verified on live GOA: mid rungs force-select HTTP 200 + confirm 204 → OK_BANK.
This commit is contained in:
Hernâni Marques
parent
ac01e4ecfe
commit
80643e6b3f
5 changed files with 135 additions and 70 deletions
|
|
@ -53,8 +53,8 @@ Legend:
|
|||
|--|--|
|
||||
| **Seen** | ladder force-select, e2e confirm path, Android spinner issues; **repro 2026-07-19 magikoopa** GOA ladder: `force-select skipped · no reserve_pub for this withdraw (WID=…); wallet may not have selected yet` |
|
||||
| **Area** | **cli** / **core** / **bank** / **ops** |
|
||||
| **Problem** | (a) Cumulative wallets re-emit old reserves; binding the wrong `reserve_pub` yields **HTTP 409 / code 5114** (“already bound”). (b) Automation scrapes accept/tx dumps for candidates — when the dump has **no** pub at all, ladder WARNs and skips force-select (soft), then often **SKIP_CONFIRM**. (c) Empty scrape is **not** the same as “bank out of money”; do not treat as balance failure. |
|
||||
| **Workaround** | Score pubs by WID/amount; try several; treat 5114 as “stale reserve” not “out of money”. Refresh `transactions` before each force-select try. Soft-skip confirm if still pending after polls. Fix wallet DB / helper first if accept/tx only print FATAL (see §15–16) — otherwise scrape can never find a pub. |
|
||||
| **Problem** | (a) Cumulative wallets re-emit old reserves; binding the wrong `reserve_pub` yields **HTTP 409 / code 5114** (“already bound”). (b) Automation scrapes accept/tx dumps for candidates — when the dump has **no** pub at all, ladder WARNs and skips force-select (soft), then often **SKIP_CONFIRM**. (c) Empty scrape is **not** the same as “bank out of money”; do not treat as balance failure. (d) **Mon bug (fixed v1.20.2):** extract used `json.loads(file_from_first_brace)` on wallet dumps that append log lines after JSON → **zero** objects parsed; regex only scanned accept (which has no `reservePub`) while the pub lives under `transactions[].withdrawalDetails.reservePub`. |
|
||||
| **Workaround** | **Mon ≥1.20.2:** `JSONDecoder.raw_decode` + regex on accept **and** transactions; score by WID in `bankConfirmationUrl` / amount. Still: score pubs by WID/amount; try several; treat 5114 as “stale reserve” not “out of money”. Refresh `transactions` before each force-select try. Soft-skip confirm if still pending after polls. Fix wallet DB / helper first if accept/tx only print FATAL (see §15–16). |
|
||||
| **Wanted** | CLI returns **the** `reserve_pub` for the just-accepted withdraw in structured JSON; `force-select` / bank-side select API documented with idempotency. Optional: bank integration already exposes `selected_reserve_pub` when wallet selected — prefer that over scraping wallet logs. |
|
||||
|
||||
### 3a. Ladder pin amounts hide mid-range behaviour
|
||||
|
|
@ -63,8 +63,8 @@ Legend:
|
|||
|--|--|
|
||||
| **Seen** | `LADDER_STEPS=3` plan collapses to pins only: `GOA:0`, `max-1`, `max` (no random mid rungs) |
|
||||
| **Area** | **ops** / **bank** / **cli** |
|
||||
| **Problem** | **Zero** withdraw often never produces a usable `reserve_pub` / coins (wallet 7006 or no denoms). **Absolute max** often **mint HTTP 500 / code 5110** SQL `P0001` (ceiling probe — expected soft on max pin). Reproducing “real” force-select needs a **mid** amount (e.g. `GOA:1` … exchange min denom scale), not only pins. |
|
||||
| **Workaround** | Use `LADDER_STEPS≥5` (or higher) so random mids exist; treat zero / absolute-max as probes. Soft-skip zero (7006) and max ceiling (5110). |
|
||||
| **Problem** | **Zero** withdraw often never produces a usable `reserve_pub` / coins (wallet 7006 or no denoms). **Absolute max** and sometimes **max-1** hit **mint HTTP 500 / code 5110** SQL `P0001` (ceiling probe). Reproducing “real” force-select needs a **mid** amount (e.g. `GOA:1` … exchange min denom scale), not only pins. |
|
||||
| **Workaround** | Use `LADDER_STEPS≥5` (or higher) so random mids exist; treat zero / absolute-max / max-1-5110 as probes (soft `CEILING_REJECT` in mon ≥1.20.2). |
|
||||
| **Wanted** | Ladder plan docs: pin vs mid semantics; CLI structured error for amount-too-large / zero. |
|
||||
|
||||
---
|
||||
|
|
|
|||
2
VERSION
2
VERSION
|
|
@ -1 +1 @@
|
|||
1.20.1
|
||||
1.20.2
|
||||
|
|
|
|||
|
|
@ -17,6 +17,7 @@ Git tags: `vMAJOR.FEATURE.FIX` (e.g. `v1.8.0`). File `VERSION` omits the `v` pre
|
|||
|
||||
| Tag | Date (UTC) | Notes |
|
||||
|-----|------------|--------|
|
||||
| **v1.20.2** | 2026-07-19 | **Bugfix:** ladder force-select empty `reserve_pub` scrape — wallet dumps mix log lines + JSON so `json.loads(rest_of_file)` always failed; extract now uses `JSONDecoder.raw_decode` and regex on **both** accept + transactions (`reservePub` is in tx). Prefer Homebrew python on macOS for helper; `find_wallet_cli` prefers ts-core 1.6.x over stale system wrappers. Soft-skip max-1 mint 5110/P0001 as ceiling. Verified: mid rungs `force-select HTTP 200` + `confirm 204` → `OK_BANK`. |
|
||||
| **v1.20.1** | 2026-07-19 | **Docs:** CLI-AUTOMATION-NOTES — force-select empty scrape + ladder pins (§3/3a); wallet-db must not be `*.json` (§15); `taler-helper-sqlite3` needs Python ≥3.11 (§16); portable wall-clock (§17); GOA withdraw checklist (§18). No code change. |
|
||||
| **v1.20.0** | 2026-07-19 | **Feature:** domains.conf **`locale`** (l10n) — FP **`fr-CH`**, others **`de-CH`** (aliases ch-FR/ch-DE); UI **`lang=de`** supported (sticky + console tags) but **never** a stock profile default; env `TALER_MON_LOCALE` / `TALER_DOMAIN_LOCALE`. |
|
||||
| **v1.19.1** | 2026-07-19 | **Bugfix / docs:** DEPENDENCIES — e2e requires `taler-helper-sqlite3` on mon PATH + preflight; README domains.conf documents **`lang`** column. |
|
||||
|
|
|
|||
|
|
@ -171,10 +171,17 @@ ladder_over() {
|
|||
}
|
||||
|
||||
wcli() {
|
||||
# taler-helper-sqlite3 is Python ≥3.11; prefer Homebrew/local python on macOS
|
||||
# so Apple /usr/bin/python3 3.9 does not FATAL the sqlite backend (no reservePub).
|
||||
local _path="${PATH:-}"
|
||||
case ":${_path}:" in
|
||||
*:/opt/homebrew/bin:*) ;;
|
||||
*) _path="/opt/homebrew/bin:/usr/local/bin:${_path}" ;;
|
||||
esac
|
||||
if [ -n "${CLI_JS:-}" ] && [ -f "$CLI_JS" ]; then
|
||||
node "$CLI_JS" --wallet-db="$WDB" --no-throttle "$@"
|
||||
PATH="$_path" node "$CLI_JS" --wallet-db="$WDB" --no-throttle "$@"
|
||||
else
|
||||
taler-wallet-cli --wallet-db="$WDB" --no-throttle "$@"
|
||||
PATH="$_path" taler-wallet-cli --wallet-db="$WDB" --no-throttle "$@"
|
||||
fi
|
||||
}
|
||||
|
||||
|
|
@ -238,10 +245,18 @@ wallet_avail() {
|
|||
wcli balance 2>/dev/null | python3 -c '
|
||||
import json,sys
|
||||
t=sys.stdin.read()
|
||||
dec=json.JSONDecoder()
|
||||
i=t.find("{")
|
||||
if i<0:
|
||||
print("0"); raise SystemExit
|
||||
try:
|
||||
d,_=dec.raw_decode(t,i)
|
||||
except Exception:
|
||||
# last-resort: brace slice (may fail on trailing logs)
|
||||
try:
|
||||
d=json.loads(t[i:t.rfind("}")+1])
|
||||
except Exception:
|
||||
print("0"); raise SystemExit
|
||||
cur=sys.argv[1]
|
||||
for b in d.get("balances") or []:
|
||||
a=b.get("available") or ""
|
||||
|
|
@ -587,6 +602,16 @@ for AMT in "$@"; do
|
|||
echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV"
|
||||
continue
|
||||
fi
|
||||
# max-1 can also hit the same libeufin SQL ceiling (5110/P0001) on live GOA —
|
||||
# treat as soft ceiling when mid rungs already passed; do not block the report.
|
||||
if [ "$IS_MAX_M1_PIN" = "1" ] && echo "$note" | grep -qE '5110|P0001'; then
|
||||
status="CEILING_REJECT"
|
||||
note="max-1 rejected (same ceiling as absolute max): $note"
|
||||
warn bank "mint $AMT rejected (max-1 ceiling)" \
|
||||
"problem: bank rejects max-1 with SQL P0001/5110 (pool ceiling). Mid-rung OK_BANK/OK still count. Ladder continues. detail: $note"
|
||||
echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV"
|
||||
continue
|
||||
fi
|
||||
err bank "mint $AMT failed" "$note"
|
||||
status="FAIL_MINT"
|
||||
STOP_REASON="$note"
|
||||
|
|
@ -648,6 +673,10 @@ for AMT in "$@"; do
|
|||
# Collect reserve_pub candidates for *this* withdrawal (WID + amount).
|
||||
# Cumulative wallets re-print old reserves in accept/tx dumps — never trust a single "last" blindly.
|
||||
# Prints unique pubs one per line, preferred order first.
|
||||
#
|
||||
# Critical: wallet-cli dumps mix log lines + JSON. Never json.loads(rest_of_file) —
|
||||
# trailing "Shutdown requested" lines make that always fail. Use JSONDecoder.raw_decode
|
||||
# and regex on *both* accept and transactions output (reservePub lives in tx, not accept).
|
||||
extract_rpubs_for_wid() {
|
||||
python3 - "$WID" "$AMT" "$SCRATCH/accept-$tag.out" "$SCRATCH/tx-$tag.json" "$SCRATCH/used-rpubs.txt" <<'PY'
|
||||
import json, re, sys
|
||||
|
|
@ -661,99 +690,118 @@ used = set()
|
|||
if Path(used_path).is_file():
|
||||
used = {ln.strip() for ln in open(used_path) if ln.strip()}
|
||||
|
||||
amt_num = amt.split(":", 1)[-1] if amt else ""
|
||||
dec = json.JSONDecoder()
|
||||
|
||||
def walk_collect(o, bag, ctx=None):
|
||||
ctx = dict(ctx or {})
|
||||
if isinstance(o, dict):
|
||||
# First pass: sibling scalars (reservePub + bankConfirmationUrl + amounts
|
||||
# are peers under withdrawalDetails — order in JSON must not matter).
|
||||
for k, v in o.items():
|
||||
kl = str(k).lower()
|
||||
if kl in ("withdrawal_id", "withdraw_id", "wopid", "id") and isinstance(v, str):
|
||||
if not isinstance(v, str):
|
||||
continue
|
||||
kl = str(k).lower().replace("-", "_")
|
||||
if kl in (
|
||||
"withdrawal_id", "withdraw_id", "wopid", "id",
|
||||
"transactionid", "transaction_id",
|
||||
):
|
||||
ctx["id"] = v
|
||||
if kl in ("amount", "rawamount", "instructedamount") and isinstance(v, str):
|
||||
elif kl in (
|
||||
"amount", "rawamount", "instructedamount",
|
||||
"amounteffective", "amountraw", "transferamount",
|
||||
):
|
||||
ctx["amount"] = v
|
||||
if kl in ("taler_withdraw_uri", "talerwithdrawuri", "uri") and isinstance(v, str):
|
||||
elif kl in (
|
||||
"taler_withdraw_uri", "talerwithdrawuri", "uri",
|
||||
"bankconfirmationurl", "confirmtransferurl",
|
||||
):
|
||||
ctx["uri"] = v
|
||||
for k, v in o.items():
|
||||
kl = str(k).lower().replace("-", "_")
|
||||
if kl in ("reserve_pub", "reservepub") and isinstance(v, str) and len(v) >= 40:
|
||||
bag.append((v, dict(ctx)))
|
||||
elif not isinstance(v, (str, int, float, bool)) and v is not None:
|
||||
walk_collect(v, bag, ctx)
|
||||
elif isinstance(o, list):
|
||||
for i in o:
|
||||
walk_collect(i, bag, ctx)
|
||||
|
||||
raw_pubs = [] # ordered as found
|
||||
scored = [] # (score, pub) higher better
|
||||
blob_all = ""
|
||||
for p in paths:
|
||||
def score_ctx(pub, ctx, window=""):
|
||||
score = 0
|
||||
cid = str(ctx.get("id") or "")
|
||||
camt = str(ctx.get("amount") or "")
|
||||
curi = str(ctx.get("uri") or "")
|
||||
# bankConfirmationUrl / withdraw URI embeds this op's WID
|
||||
if wid and wid in curi:
|
||||
score += 120
|
||||
if wid and wid in cid:
|
||||
score += 100
|
||||
if wid and wid in window:
|
||||
score += 80
|
||||
if amt and (camt == amt or camt.endswith(amt_num)):
|
||||
score += 50
|
||||
if amt and amt in window:
|
||||
score += 30
|
||||
if pub in used:
|
||||
score -= 200
|
||||
return score
|
||||
|
||||
def iter_json_objects(text):
|
||||
"""Yield top-level JSON objects from mixed log+JSON wallet dumps."""
|
||||
i, n = 0, len(text)
|
||||
while i < n:
|
||||
if text[i] == "{":
|
||||
try:
|
||||
blob_all += open(p, errors="replace").read() + "\n"
|
||||
o, end = dec.raw_decode(text, i)
|
||||
yield o
|
||||
i = end
|
||||
continue
|
||||
except Exception:
|
||||
pass
|
||||
i += 1
|
||||
|
||||
# 1) full JSON objects in files
|
||||
raw_pubs = []
|
||||
scored = []
|
||||
|
||||
# 1) structured JSON via raw_decode (accept + transactions)
|
||||
for p in paths:
|
||||
try:
|
||||
t = open(p, errors="replace").read()
|
||||
except Exception:
|
||||
continue
|
||||
# whole-file JSON
|
||||
for m in re.finditer(r"\{", t):
|
||||
try:
|
||||
o = json.loads(t[m.start():])
|
||||
except Exception:
|
||||
continue
|
||||
for o in iter_json_objects(t):
|
||||
bag = []
|
||||
walk_collect(o, bag)
|
||||
for pub, ctx in bag:
|
||||
raw_pubs.append(pub)
|
||||
score = 0
|
||||
cid = str(ctx.get("id") or "")
|
||||
camt = str(ctx.get("amount") or "")
|
||||
curi = str(ctx.get("uri") or "")
|
||||
if wid and wid in cid:
|
||||
score += 100
|
||||
if wid and wid in curi:
|
||||
score += 80
|
||||
if amt and (camt == amt or camt.endswith(amt.split(":", 1)[-1])):
|
||||
score += 40
|
||||
if pub in used:
|
||||
score -= 200
|
||||
scored.append((score, pub))
|
||||
scored.append((score_ctx(pub, ctx), pub))
|
||||
|
||||
# 2) regex fallback on accept file only (more current)
|
||||
try:
|
||||
acc = open(paths[0], errors="replace").read()
|
||||
except Exception:
|
||||
acc = ""
|
||||
for pat in (
|
||||
r"\"reserve_pub\"\s*:\s*\"([A-Z0-9]{40,})\"",
|
||||
# 2) regex on *both* accept and tx (reservePub is normally only in transactions)
|
||||
RPUB_PATS = (
|
||||
r"\"reservePub\"\s*:\s*\"([A-Z0-9]{40,})\"",
|
||||
r"reserve_pub[\"\s:=]+([A-Z0-9]{40,})",
|
||||
):
|
||||
for m in re.finditer(pat, acc, re.I):
|
||||
r"\"reserve_pub\"\s*:\s*\"([A-Z0-9]{40,})\"",
|
||||
r"reserve[_ ]?pub[\"\s:=]+([A-Z0-9]{40,})",
|
||||
)
|
||||
for p in paths:
|
||||
try:
|
||||
blob = open(p, errors="replace").read()
|
||||
except Exception:
|
||||
continue
|
||||
for pat in RPUB_PATS:
|
||||
for m in re.finditer(pat, blob, re.I):
|
||||
pub = m.group(1)
|
||||
raw_pubs.append(pub)
|
||||
score = 10
|
||||
# proximity to WID in accept output
|
||||
window = acc[max(0, m.start() - 400) : m.end() + 400]
|
||||
if wid and wid in window:
|
||||
score += 100
|
||||
if amt and amt in window:
|
||||
score += 30
|
||||
if pub in used:
|
||||
score -= 200
|
||||
scored.append((score, pub))
|
||||
window = blob[max(0, m.start() - 600) : m.end() + 600]
|
||||
scored.append((score_ctx(pub, {}, window) + 10, pub))
|
||||
|
||||
# prefer high score, then later occurrence
|
||||
order = []
|
||||
seen = set()
|
||||
for score, pub in sorted(scored, key=lambda x: (-x[0],), reverse=False):
|
||||
# sort by score desc: use reverse sorted
|
||||
pass
|
||||
for score, pub in sorted(scored, key=lambda x: x[0], reverse=True):
|
||||
if pub in seen or pub in used:
|
||||
continue
|
||||
seen.add(pub)
|
||||
order.append(pub)
|
||||
# append unused raw in reverse (newest-ish)
|
||||
for pub in reversed(raw_pubs):
|
||||
if pub in seen or pub in used:
|
||||
continue
|
||||
|
|
|
|||
20
lib.sh
20
lib.sh
|
|
@ -1512,12 +1512,20 @@ find_wallet_cli() {
|
|||
echo "$WALLET_CLI"; return 0
|
||||
fi
|
||||
local c cand
|
||||
# WALLET_CLI / WALLET_CLI_CANDIDATES from env; then portable package paths only.
|
||||
# WALLET_CLI / WALLET_CLI_CANDIDATES from env; then portable package paths.
|
||||
# Prefer local ts-core / mon-env builds (schema-compatible) over stale system
|
||||
# wrappers (e.g. /usr/local 1.3.x vs DB schema 29 from 1.6.x).
|
||||
# shellcheck disable=SC2086
|
||||
for c in ${WALLET_CLI_CANDIDATES:-} \
|
||||
"${HOME}/.local/bin/taler-wallet-cli.mjs" \
|
||||
"${HOME}/src/taler/taler-typescript-core/packages/taler-wallet-cli/bin/taler-wallet-cli.mjs" \
|
||||
"${HOME}/src/taler/taler-typescript-core/packages/taler-wallet-cli/dist/taler-wallet-cli.mjs" \
|
||||
"${HOME}/src/taler/taler-typescript-core/packages/taler-wallet-cli/dist/taler-wallet-cli-local.mjs" \
|
||||
"${HOME}/src/taler/taler-typescript-core/packages/taler-wallet-cli/dist/taler-wallet-cli-bundled.cjs" \
|
||||
/usr/lib/taler-wallet-cli/node_modules/taler-wallet-cli/bin/taler-wallet-cli.mjs \
|
||||
/usr/share/taler-wallet-cli/bin/taler-wallet-cli.mjs
|
||||
/usr/share/taler-wallet-cli/bin/taler-wallet-cli.mjs \
|
||||
/usr/local/lib/taler-wallet-cli/node_modules/taler-wallet-cli/bin/taler-wallet-cli.mjs \
|
||||
/usr/local/lib/taler-wallet-cli/node_modules/taler-wallet-cli/dist/taler-wallet-cli-bundled.cjs
|
||||
do
|
||||
[ -n "$c" ] && [ -f "$c" ] && { echo "$c"; return 0; }
|
||||
done
|
||||
|
|
@ -1533,6 +1541,14 @@ find_wallet_cli() {
|
|||
c=$(readlink -f "$cand" 2>/dev/null || true)
|
||||
[ -n "$c" ] && [ -f "$c" ] && { echo "$c"; return 0; }
|
||||
fi
|
||||
# macOS /usr/local wrapper often runs an older bundle; prefer its bundled.cjs only
|
||||
# when no newer candidate above matched (already returned).
|
||||
for c in \
|
||||
/usr/local/lib/taler-wallet-cli/node_modules/taler-wallet-cli/dist/taler-wallet-cli-bundled.cjs \
|
||||
/usr/local/lib/node_modules/taler-wallet-cli/dist/taler-wallet-cli-bundled.cjs
|
||||
do
|
||||
[ -f "$c" ] && { echo "$c"; return 0; }
|
||||
done
|
||||
fi
|
||||
return 1
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue