taler-monitoring/CLI-AUTOMATION-NOTES.md

CLI automation notes (taler-wallet-cli + monitoring)

Living list of recurring issues seen while driving GOA (hacktivism) and stage TESTPAYSAN via CLI (taler-wallet-cli, bank/merchant HTTP, monitoring e2e/ladder). Ordered roughly by how often they hurt automation and how useful a fix in wallet-cli / wallet-core (or clearer APIs) would be.

Companions:

• Android GUI → android-test/GUI-AUTOMATION-NOTES.md
• Git / suite tree → VERSIONS.md

Scripts: check_e2e.sh, check_amount_ladder.sh, taler-monitoring.sh.

Legend:

Tag	Meaning
cli	wallet-cli UX / flags / hang behaviour
core	wallet-core protocol / state machine
bank	libeufin-bank / integration API
ops	secrets, paths, multi-host
doc	missing or wrong documentation

---

1. run-until-done is unusable for automation

Seen	e2e, ladder, Android notes, macOS comments: hangs / banned in monitoring
Area	cli / core
Problem	taler-wallet-cli run-until-done (or long shepherd runs) often never returns or blocks CI-length timeouts. Monitoring forbids it and polls balance + bank transfer_done instead.
Workaround	advanced serve + socket (WSOCK) or pure poll loops (LADDER_SETTLE_*, e2e settle wait).
Wanted for wallet-cli	Bounded wait: e.g. `run-until-done --timeout=Ns --exit-on=withdrawn

---

2. Withdraw completion is not observable from CLI alone

Seen	e2e “coins missing after withdraw”; ladder OK_BANK vs wallet avail
Area	cli / core / bank
Problem	After accept-uri + bank confirm, coins may lag (wirewatch). CLI has no single “withdraw settled” command with clear success/failure; automation reimplements status from bank integration JSON + balance polls.
Workaround	Poll balance + …/withdrawal-operation/{id} (status, transfer_done, selected_reserve_pub).
Wanted	wallet-cli withdrawals wait --id=… --timeout= or JSON event stream: selected → confirmed → coins-available.

---

3. reserve_pub / force-select is fragile (5114 + empty scrape)

Seen	ladder force-select, e2e confirm path, Android spinner issues; repro 2026-07-19 magikoopa GOA ladder: force-select skipped · no reserve_pub for this withdraw (WID=…); wallet may not have selected yet
Area	cli / core / bank / ops
Problem	(a) Cumulative wallets re-emit old reserves; binding the wrong reserve_pub yields HTTP 409 / code 5114 (“already bound”). (b) Automation scrapes accept/tx dumps for candidates — when the dump has no pub at all, ladder WARNs and skips force-select (soft), then often SKIP_CONFIRM. (c) Empty scrape is not the same as “bank out of money”; do not treat as balance failure. (d) Mon bug (fixed v1.20.2): extract used json.loads(file_from_first_brace) on wallet dumps that append log lines after JSON → zero objects parsed; regex only scanned accept (which has no reservePub) while the pub lives under transactions[].withdrawalDetails.reservePub.
Workaround	Mon ≥1.20.2: JSONDecoder.raw_decode + regex on accept and transactions; score by WID in bankConfirmationUrl / amount. Still: score pubs by WID/amount; try several; treat 5114 as “stale reserve” not “out of money”. Refresh transactions before each force-select try. Soft-skip confirm if still pending after polls. Fix wallet DB / helper first if accept/tx only print FATAL (see §15–16).
Wanted	CLI returns the reserve_pub for the just-accepted withdraw in structured JSON; force-select / bank-side select API documented with idempotency. Optional: bank integration already exposes selected_reserve_pub when wallet selected — prefer that over scraping wallet logs.

3a. Ladder pin amounts hide mid-range behaviour

Seen	LADDER_STEPS=3 plan collapses to pins only: GOA:0, max-1, max (no random mid rungs)
Area	ops / bank / cli
Problem	Zero withdraw often never produces a usable reserve_pub / coins (wallet 7006 or no denoms). Absolute max and sometimes max-1 hit mint HTTP 500 / code 5110 SQL P0001 (ceiling probe). Reproducing “real” force-select needs a mid amount (e.g. GOA:1 … exchange min denom scale), not only pins.
Workaround	Use LADDER_STEPS≥5 (or higher) so random mids exist; treat zero / absolute-max / max-1-5110 as probes (soft CEILING_REJECT in mon ≥1.20.2).
Wanted	Ladder plan docs: pin vs mid semantics; CLI structured error for amount-too-large / zero.

3b. Mint fails with bank SQL 5110 / P0001 (server-side; recurring)

Seen	Several times on GOA (bank.hacktivism.ch), classic ladder high/random mids + ceiling pins and max-search bisect — not a one-off. Same class of error across 2026-07 runs (magikoopa / mon ladder).
Area	bank (libeufin / taler-corebank + Postgres) / ops — not mon SQL, not wallet-cli at mint time
Call site	mon POST ${BANK}/accounts/explorer/withdrawals with {"amount":"GOA:…"} only. Wallet/exchange not involved until mint returns 200 + URI.
Response	HTTP 500, body e.g. { code: 5110, hint: "Unexpected sql error with state P0001" }
Codes	5110 = BANK_UNMANAGED_EXCEPTION (“no known exception types captured the exception”). P0001 = PostgreSQL SQLSTATE for application RAISE EXCEPTION (function/trigger), not a mon bug and not max_wire config alone.
Config check	Public /config still allows huge wires (max_wire_transfer_amount ≈ GOA:4503599627370496.99999999, default_debit_threshold same order). Failing mints are often far below that (e.g. ~Kilo-GOA).
API mismatch	Corebank docs: insufficient funds should be HTTP 409. Getting 500/5110 means the bank is leaking an unmapped SQL error instead of a clean client error.
Problem	Live mint “ceiling” is opaque and unstable. Do not treat 5110 as wallet/reserve_pub failure when mint never succeeded. Do not assume “only absolute pin:max fails” — mid and max-search probes hit the same error.
Workaround (mon ≥1.21)	Soft CEILING_REJECT on 5110/P0001 (classic mids + max pins); max-ladder treats mint reject as too-high upper bound and bisects. Keep mid-range rungs for real withdraw/confirm tests.
Wanted (bank)	Map the SQL condition to a structured error (insufficient funds / amount too large / debit limit) — not BANK_UNMANAGED_EXCEPTION. Log full RAISE text server-side.

Full max-search cascade (primary repro log, 2026-07-19, magikoopa / GOA)

Mon phase max-ladder withdraw hunt. Every reject below is the same bank body:

mint HTTP 500 { code :5110, hint : Unexpected sql error with state P0001 }

Mon treats each as soft too-high (CEILING_REJECT / hi:=amount). lo stays 0 until the first successful mint.

Phase A — pure descent (no success yet): hi falls Tera → Giga → Mega while lo=0

Probe	Amount (alt)	Raw GOA:…	Result
(earlier)	hi already 467.409 Tera-GOA	467409181353380	5110 (prior probe)
4	154.93 Tera-GOA	154929915073879	5110 → hi:=
5	16.20 Tera-GOA	16198385073981	5110 → hi:=
6	1.417 Tera-GOA	1416945306636	5110 → hi:=
7	282.49 Giga-GOA	282485970397	5110 → hi:=
8	132.18 Giga-GOA	132183255296	5110 → hi:=
9	7.417 Giga-GOA	7416937654	5110 → hi:=
10	711.38 Mega-GOA	711383774	5110 → hi:=
11	395.88 Mega-GOA	395879334	5110 → hi:=
12	119.26 Mega-GOA	119263564	5110 → hi:=
13	5.992 Mega-GOA	5992111	5110 → hi:=

Phase B — first success raises lo; still 5110 just above

Probe	Amount (alt)	Raw	Result
14	424.114 Kilo-GOA	424114	OK mint → accept → force-select PNK5YRKT… → confirm 204 → settle transfer_done (wallet avail still 0) · lo:=424114
15	1.594 Mega-GOA	1594158	5110 → hi:=
16	822.256 Kilo-GOA	822256	OK mint → force-select SVKV4E9G… → confirm → settle · lo:=822256 (best so far)

Phase C — fine bisect above best: every step 5110 until convergence

Probe	Amount (alt)	Raw	Result
17	1.145 Mega-GOA	1144904	5110
18	970.26 Kilo-GOA	970260	5110
19	893.198 Kilo-GOA	893198	5110
20	856.993 Kilo-GOA	856993	5110
21	839.445 Kilo-GOA	839445	5110
22	830.806 Kilo-GOA	830806	5110
23	826.52 Kilo-GOA	826520	5110
24	824.385 Kilo-GOA	824385	5110
25	823.32 Kilo-GOA	823320	5110
26	822.788 Kilo-GOA	822788	5110 → hi:=

Converged (mon):

max-search best withdrawable 822.256 Kilo-GOA (GOA:822256)
  · lo=822.256 Kilo-GOA · hi=822.788 Kilo-GOA · probes=26

Reading this log

1. Same error from ~467 Tera-GOA down to ~823 Kilo-GOA — not a single “absurd pin:max” case; the bank returns 5110/P0001 for a wide continuous range of amounts.
2. Successes are real but sparse: only GOA:424114 and GOA:822256 minted in this hunt (both full bank confirm + transfer_done). Everything strictly larger than 822256 in the fine band failed at mint.
3. lo=0 for probes 4–13 means max-search had no successful withdraw yet while already rejecting Tera/Giga/Mega — so 5110 is not “only after pool drained by this run.”
4. Settle notes wallet avail=GOA:0 after OK bank path — coins lag / pendingIncoming (pay-wait board, § pay wait UX); separate from mint 5110.
5. Landing recent_withdraws at 14:42 CEST lists the same two OK amounts (424114, 822256) with reserves matching force-select prefixes above.

Landing page: much larger withdraws same day, earlier (still on explorer)

Public bank intro: https://bank.hacktivism.ch/intro/stats.json (HTML landing uses this). Snapshot 2026-07-19 ~14:47 CEST.

Pool still looks full in the UI:

Field	Snapshot
balance_explorer	4.5036 Peta-GOA (GOA:4503599627250215.32499924)
withdraws.total_amount	4.5036 Peta-GOA (292 ops)
withdraws.last_24h	3.2366 Peta-GOA (45 ops)

Dated explorer withdraws that succeeded before this max-search tight band (recent_withdraws, account=explorer):

When (CEST)	Amount (alt)	Raw	Notes
2026-07-19 14:19	1.9761 Giga-GOA	GOA:1976113563	~2h before max-search best
2026-07-19 14:19	1.4495 Giga-GOA	GOA:1449489732	same minute
2026-07-19 14:26	201.636 Mega-GOA	GOA:201635991
2026-07-19 14:26	144.53 Mega-GOA	GOA:144530029
2026-07-19 14:29	3.4099 Mega-GOA	GOA:3409891
2026-07-19 14:29	1.8009 Mega-GOA	GOA:1800875
2026-07-19 14:42	822.256 Kilo-GOA	GOA:822256	= max-search best (probe 16)
2026-07-19 14:42	424.114 Kilo-GOA	GOA:424114	= max-search first OK (probe 14)

Contrast: earlier the same day, explorer minted ~1.5–2 Giga-GOA singles; later the same API returns 5110 for ~823 Kilo-GOA and for all of Tera→Mega probes in the cascade. Landing balance_explorer still shows ~Peta-GOA. So:

• not “config max_wire forbids Kilo-GOA”;
• not “landing says explorer is empty”;
• live mintability collapsed / became amount-sensitive relative to history — bank SQL path is wrong or mis-mapped.

Ops checklist when filing bank bug: attach (1) this max-search probe table, (2) landing recent_withdraws + balance_explorer at the same timestamps, (3) bank journal / Postgres P0001 RAISE text for one failing mint.

Also seen (same error class): classic ladder e.g. GOA:747827072844319 → 500/5110/P0001; absolute max / max-1 ceiling pins → soft CEILING_REJECT.

Exchange intro (not mint path): https://exchange.hacktivism.ch/intro/stats.json — wire-in can still be Peta-scale; use bank landing for explorer mint history.

---

4. Exchange not auto-known for bank deep links / withdraw URIs

Seen	Android endless spinner; GOA bank.hacktivism.ch withdraw; fixed in app branch fix/bank-withdraw-auto-exchange
Area	core / cli
Problem	Withdraw URI names an exchange the wallet has never added → stuck loading / empty exchange list. CLI requires explicit exchanges add + update + accept-tos every fresh DB.
Workaround	e2e always adds exchange before bank withdraw ladder; Android needs ensureExchange-style fix.
Wanted	CLI/core: on accept-uri withdraw, auto-add exchange from URI/bank details (allowCompletion), then surface ToS if needed in one command.

---

5. Default ports in URIs (:443 / :80) break round-trips

Seen	ladder/e2e strip ports; QR checks; mint URIs with :443
Area	cli / bank / doc
Problem	Some bank/wallet outputs include taler://…host:443/…. QR validators and some clients reject or double-normalize inconsistently.
Workaround	sed 's/:443\//\//g' everywhere in automation.
Wanted	Canonical form without default ports in all wallet-cli and bank integration outputs; document as invariant.

---

6. No first-class “demo withdraw” / explorer mint in wallet-cli

Seen	e2e uses bank admin + bank withdraw create; ladder/gui-chain reimplement explorer mint in Python
Area	cli / ops
Problem	Automation always hand-rolls Basic auth → token → POST …/withdrawals → taler_withdraw_uri. Easy to get wrong (auth header, amount currency).
Workaround	Shared Python in goa-chain / ladder / e2e; secrets files.
Wanted	Optional helper: wallet-cli testing mint-withdraw --bank=URL --user=explorer --password-file=… --amount=GOA:10 (or bank-side only tool in libeufin) emitting a clean URI.

---

7. Long-lived wallet process required for pay/withdraw progress

Seen	e2e advanced serve / shepherd socket
Area	cli
Problem	One-shot CLI invocations do not keep background work alive; without serve/shepherd, withdraw/pay may stall mid-flight.
Workaround	Start serve once per e2e run; route wcli through the socket.
Wanted	Documented, supported “session mode”: start/stop serve; or each mutating command optionally --background-until=… with timeout.

---

8. ToS accept is a separate step that fails silently or blocks

Seen	e2e exchanges accept-tos; ladder after add; Android GUI ToS screens
Area	cli / core
Problem	Fresh exchange → operations need ToS; CLI must call accept-tos explicitly. Failures are easy to miss in multi-step scripts.
Workaround	Always accept-tos after update in e2e/ladder.
Wanted	accept-uri --accept-tos or auto-prompt with non-interactive --yes that covers exchange ToS for that URI’s exchange.

---

9. Pay template / public order path is merchant-shaped, not CLI-shaped

Seen	e2e shop templates, stage farmer shops, ladder private orders
Area	cli / doc
Problem	Wallet-cli pays via handle-uri on taler://pay/…; creating the order is always custom curl (template POST or private order + token). No unified “pay this amount to instance” for public templates.
Workaround	Monitoring builds URI externally then handle-uri --yes.
Wanted	Documented recipe only, or wallet-cli testing pay-template --base=… --instance=… --id=… for demos.

---

10. Amount / currency parsing edge cases

Seen	ladder min denom 0.01 vs GOA 1e-6; zero withdraw rejected (HTTP 409 / amount too low); max wire ceilings
Area	cli / bank
Problem	CURRENCY:0 and sub-min amounts fail differently per stack; CLI error strings are not machine-stable.
Workaround	Soft-skip zero rung; clamp ladder min to exchange denoms / bank max_wire.
Wanted	Structured errors (AMOUNT_too_small, currency_unknown) in JSON mode; wallet-cli amount validate --exchange=.

---

11. Finding the right taler-wallet-cli binary

Seen	find_wallet_cli in lib.sh; hardcoded laptop paths; wrapper vs .mjs
Area	ops / cli
Problem	Debian package is a shell wrapper; some tools need node …/taler-wallet-cli.mjs. Hardcoded /Users/… paths break other hosts.
Workaround	find_wallet_cli search list; WALLET_CLI= override.
Wanted	Single install story: wallet-cli --version JSON with path + libversion; no need to pass .mjs to node by hand.

---

12. Secrets and multi-stack confusion

Seen	explorer vs admin password; GOA secrets used on stage; ladder EXP_PW
Area	ops
Problem	CLI does not know “which stack”; wrong password → 401 mid-ladder. Not a wallet-cli bug, but every CLI automation hits it.
Workaround	SECRETS_ROOT, stage SSH, STACK= profiles.
Wanted	Optional ~/.config/taler/stacks.d/goa.env convention documented next to wallet-cli; still no secrets in repo.

---

13. No stable machine-readable “step result” for scripts

Seen	e2e greps accept output; ladder scrapes JSON from mixed stdout
Area	cli
Problem	Human logs + occasional JSON blobs; hard to parse reliably.
Workaround	Python scrapers, temp files, tee.
Wanted	Global --json / --ndjson for all commands; one object per completed operation with ok, op, ids, amounts.

---

14. Pay settlement wait is symmetric to withdraw pain

Seen	e2e pay settle loops; ladder pay settle rounds
Area	cli / core
Problem	After handle-uri pay, success is “order paid” / balance drop / merchant order status — not one CLI wait.
Workaround	Short poll loops; never run-until-done.
Wanted	Same as §1–2: bounded wait on transaction id / order id.

---

15. --wallet-db=*.json is rejected (“memory backend not supported”)

Seen	2026-07-19 magikoopa wallet-cli 1.6.8 / 1.6.11 bundle: any command with --wallet-db=/tmp/foo.json fails immediately with Error: memory backend not supported
Area	cli / ops
Problem	Current node host treats storage paths ending in .json as unsupported memory backend and throws before opening sqlite. Old notes / local tests that used WALLET_DB=….json break. Accept/transactions then never emit reserve_pub → ladder §3 empty-scrape WARNs.
Workaround	Use a non-.json path, e.g. --wallet-db=$SCRATCH/wallet.sqlite3 (ladder already prefers wallet.sqlite3). Never pass ….json as wallet-db.
Wanted	Clearer error: wallet-db path must not end in .json; use a directory or .sqlite3 path. Document legal path forms in --help.

---

16. taler-helper-sqlite3 requires Python ≥ 3.11 on PATH

Seen	magikoopa: helper is a Python script (#!/usr/bin/env python3); with Apple /usr/bin/python3 3.9.6 → FATAL: python version >=3.11 required but running on 3.9.6 in accept/tx logs; wallet may still exit 0 on some paths while stdout is FATAL-only
Area	ops / cli
Problem	Wallet spawns taler-helper-sqlite3 from PATH. Wrong python3 ⇒ sqlite backend fails ⇒ no real wallet state ⇒ no reserve_pub for force-select.
Workaround	Put Python ≥3.11 (or 3.12+) first on PATH for mon jobs; ensure command -v taler-helper-sqlite3 works; smoke: taler-helper-sqlite3 starts without FATAL. Prefer Homebrew / user python3.11+ over macOS system 3.9 for agent PATH.
Wanted	Helper shebang or wrapper that finds a suitable python; wallet preflight error if helper HELLO fails (not silent empty logs).

---

17. Portable wall-clock for scripts (no GNU timeout assumed)

Seen	macOS agent shells: timeout: command not found; e2e already tries timeout then gtimeout then perl -e alarm
Area	ops / doc
Problem	GNU coreutils timeout is not on stock macOS. Hard-wiring it breaks portable repro and CI on Darwin.
Workaround	Prefer python3 subprocess.Popen(…).wait(timeout=…) for outer budgets; for wallet-cli inner calls keep e2e order: timeout → gtimeout → perl alarm. Document both.
Wanted	One suite helper run_with_budget SECS cmd… in lib.sh used by ladder/e2e/smoke.

---

18. How to withdraw (operator checklist, GOA laptop)

Portable path that works when §§15–16 are fixed:

1. PATH: taler-helper-sqlite3 + python3 ≥ 3.11 ahead of system 3.9.
2. Wallet DB: WDB=$TMP/wallet.sqlite3 — not *.json.
3. Exchange: exchanges add → update -f → accept-tos for https://exchange.hacktivism.ch/.
4. Bank mint (explorer): token with explorer password from SECRETS_ROOT / EXP_PW; POST …/accounts/explorer/withdrawals with e.g. {"amount":"GOA:1"} (avoid pure 0 and absolute max pins for happy-path tests).
5. Strip :443 from taler_withdraw_uri.
6. withdraw accept-uri --exchange URL URI.
7. Bank: poll …/taler-integration/withdrawal-operation/{WID} → when selected, confirm; if stuck pending, force-select with scraped reserve_pub (§3).
8. Settle: poll wallet balance / bank transfer_done — no run-until-done (§1).

If step 6’s stdout is only FATAL: python version… or memory backend not supported, stop and fix §15–16 before debugging force-select.

---

Priority shortlist for taler-wallet-cli / core

If only a few changes land, these unlock the most automation:

1. Bounded run-until-done / wait-for-state (§1, §2, §14)
2. Structured JSON on every command (§13)
3. Auto-add exchange + ToS on accept-uri (§4, §8)
4. Canonical URIs without :443 (§5)
5. Emit reserve_pub for last withdraw (§3)
6. Clear wallet-db path rules + helper preflight (§15, §16)

---

How we work around today (monitoring)

Issue	Monitoring behaviour
run-until-done	Disabled; balance + bank poll
serve	Optional long-lived socket in e2e
exchange	Explicit add/update/accept-tos
ladder explorer	read_secret / EXP_PW_FILE
stage maxima	bank max_wire + keys min denom
force-select	multi-rpub try; soft-skip confirm; empty scrape → WARN not balance fail
higher amounts	mint 5110/P0001 recurring server-side (§3b); landing shows much larger dated explorer withdraws (Giga/Mega same day); soft CEILING_REJECT + max-ladder hi; alt units in report
pay vs balance	never order more than wallet available (soft SKIP_BALANCE); pendingIncoming does not count as spendable
pay wait UX	after withdraw, board lists OK/OK_BANK in start order with live wallet pendingIncoming/done + timer (LADDER_PAY_WAIT_AVAILABLE_S, default 90s); same wallet DB as withdraw
wallet-db path	Prefer *.sqlite3 (never *.json as db path)
helper / python	mon PATH must include helper + python≥3.11

---

How to add an issue

### N. short title
| | |
|--|--|
| **Seen** | where / which stack |
| **Area** | cli / core / bank / ops / doc |
| **Problem** | … |
| **Workaround** | … |
| **Wanted** | concrete CLI/core behaviour |