taler-monitoring/CLI-AUTOMATION-NOTES.md

CLI automation notes (taler-wallet-cli + monitoring)

Living list of recurring issues seen while driving GOA (hacktivism) and stage TESTPAYSAN via CLI (taler-wallet-cli, bank/merchant HTTP, monitoring e2e/ladder). Ordered roughly by how often they hurt automation and how useful a fix in wallet-cli / wallet-core (or clearer APIs) would be.

Companions:

• Android GUI → android-test/GUI-AUTOMATION-NOTES.md
• Git / suite tree → VERSIONS.md

Scripts: check_e2e.sh, check_amount_ladder.sh, taler-monitoring.sh.

Legend:

Tag	Meaning
cli	wallet-cli UX / flags / hang behaviour
core	wallet-core protocol / state machine
bank	libeufin-bank / integration API
ops	secrets, paths, multi-host
doc	missing or wrong documentation

---

1. run-until-done is unusable for automation

Seen	e2e, ladder, Android notes, macOS comments: hangs / banned in monitoring
Area	cli / core
Problem	taler-wallet-cli run-until-done (or long shepherd runs) often never returns or blocks CI-length timeouts. Monitoring forbids it and polls balance + bank transfer_done instead.
Workaround	advanced serve + socket (WSOCK) or pure poll loops (LADDER_SETTLE_*, e2e settle wait).
Wanted for wallet-cli	Bounded wait: e.g. `run-until-done --timeout=Ns --exit-on=withdrawn

---

2. Withdraw completion is not observable from CLI alone

Seen	e2e “coins missing after withdraw”; ladder OK_BANK vs wallet avail
Area	cli / core / bank
Problem	After accept-uri + bank confirm, coins may lag (wirewatch). CLI has no single “withdraw settled” command with clear success/failure; automation reimplements status from bank integration JSON + balance polls.
Workaround	Poll balance + …/withdrawal-operation/{id} (status, transfer_done, selected_reserve_pub).
Wanted	wallet-cli withdrawals wait --id=… --timeout= or JSON event stream: selected → confirmed → coins-available.

---

3. reserve_pub / force-select is fragile (5114 + empty scrape)

Seen	ladder force-select, e2e confirm path, Android spinner issues; repro 2026-07-19 magikoopa GOA ladder: force-select skipped · no reserve_pub for this withdraw (WID=…); wallet may not have selected yet
Area	cli / core / bank / ops
Problem	(a) Cumulative wallets re-emit old reserves; binding the wrong reserve_pub yields HTTP 409 / code 5114 (“already bound”). (b) Automation scrapes accept/tx dumps for candidates — when the dump has no pub at all, ladder WARNs and skips force-select (soft), then often SKIP_CONFIRM. (c) Empty scrape is not the same as “bank out of money”; do not treat as balance failure. (d) Mon bug (fixed v1.20.2): extract used json.loads(file_from_first_brace) on wallet dumps that append log lines after JSON → zero objects parsed; regex only scanned accept (which has no reservePub) while the pub lives under transactions[].withdrawalDetails.reservePub.
Workaround	Mon ≥1.20.2: JSONDecoder.raw_decode + regex on accept and transactions; score by WID in bankConfirmationUrl / amount. Still: score pubs by WID/amount; try several; treat 5114 as “stale reserve” not “out of money”. Refresh transactions before each force-select try. Soft-skip confirm if still pending after polls. Fix wallet DB / helper first if accept/tx only print FATAL (see §15–16).
Wanted	CLI returns the reserve_pub for the just-accepted withdraw in structured JSON; force-select / bank-side select API documented with idempotency. Optional: bank integration already exposes selected_reserve_pub when wallet selected — prefer that over scraping wallet logs.

3a. Ladder pin amounts hide mid-range behaviour

Seen	LADDER_STEPS=3 plan collapses to pins only: GOA:0, max-1, max (no random mid rungs)
Area	ops / bank / cli
Problem	Zero withdraw often never produces a usable reserve_pub / coins (wallet 7006 or no denoms). Absolute max and sometimes max-1 hit mint HTTP 500 / code 5110 SQL P0001 (ceiling probe). Reproducing “real” force-select needs a mid amount (e.g. GOA:1 … exchange min denom scale), not only pins.
Workaround	Use LADDER_STEPS≥5 (or higher) so random mids exist; treat zero / absolute-max / max-1-5110 as probes (soft CEILING_REJECT in mon ≥1.20.2).
Wanted	Ladder plan docs: pin vs mid semantics; CLI structured error for amount-too-large / zero.

3b. Mint fails with bank SQL 5110 / P0001 (server-side; recurring)

Seen	Several times on GOA (bank.hacktivism.ch), classic ladder high/random mids + ceiling pins and max-search bisect — not a one-off. Same class of error across 2026-07 runs (magikoopa / mon ladder).
Area	bank (libeufin / taler-corebank + Postgres) / ops — not mon SQL, not wallet-cli at mint time
Call site	mon POST ${BANK}/accounts/explorer/withdrawals with {"amount":"GOA:…"} only. Wallet/exchange not involved until mint returns 200 + URI.
Response	HTTP 500, body e.g. { code: 5110, hint: "Unexpected sql error with state P0001" }
Codes	5110 = BANK_UNMANAGED_EXCEPTION (“no known exception types captured the exception”). P0001 = PostgreSQL SQLSTATE for application RAISE EXCEPTION (function/trigger), not a mon bug and not max_wire config alone.
Config check	Public /config still allows huge wires (max_wire_transfer_amount ≈ GOA:4503599627370496.99999999, default_debit_threshold same order). Failing mints are often far below that (e.g. ~Kilo-GOA).
API mismatch	Corebank docs: insufficient funds should be HTTP 409. Getting 500/5110 means the bank is leaking an unmapped SQL error instead of a clean client error.
Problem	Live mint “ceiling” is opaque and unstable. Do not treat 5110 as wallet/reserve_pub failure when mint never succeeded. Do not assume “only absolute pin:max fails” — mid and max-search probes hit the same error.
Workaround (mon ≥1.21)	Soft CEILING_REJECT on 5110/P0001 (classic mids + max pins); max-ladder treats mint reject as too-high upper bound and bisects. Keep mid-range rungs for real withdraw/confirm tests.
Wanted (bank)	Map the SQL condition to a structured error (insufficient funds / amount too large / debit limit) — not BANK_UNMANAGED_EXCEPTION. Log full RAISE text server-side.

What max-search saw (example run, 2026-07-19)

Ladder max-search log (magikoopa):

Time (local mon log)	Probe	Result
(bisect)	lo ≈ GOA:822256 (822.256 Kilo-GOA)	still treatable as success path / lower bound
probe ~24	GOA:824385 (824.385 Kilo-GOA)	mint HTTP 500 / 5110 / P0001 → soft too-high
after	hi := GOA:824385	continues bisect

So the live single-mint fail band was ~822–825 Kilo-GOA in that run — not “only Tera/Peta pins”.

Landing page shows much higher amounts actually gotten (same day)

Public bank intro stats: https://bank.hacktivism.ch/intro/stats.json (and the HTML landing that consumes them). Snapshot 2026-07-19 ~14:47 CEST (generated_at_human).

Explorer balance still enormous (so this is not “pool empty” in the UI sense):

Field	Value (snapshot)
balance_explorer	4.5036 Peta-GOA (GOA:4503599627250215.32499924)
cumulative withdraws.total_amount	4.5036 Peta-GOA (292 ops)
withdraws.last_24h	3.2366 Peta-GOA (45 ops)

Dated single withdraws that did mint (from recent_withdraws, account=explorer) — all before or around the ~824k fail band above:

When (CEST)	Amount (alt)	Raw
2026-07-19 14:19	1.9761 Giga-GOA	GOA:1976113563
2026-07-19 14:19	1.4495 Giga-GOA	GOA:1449489732
2026-07-19 14:26	201.636 Mega-GOA	GOA:201635991
2026-07-19 14:26	144.53 Mega-GOA	GOA:144530029
2026-07-19 14:29	3.4099 Mega-GOA	GOA:3409891
2026-07-19 14:29	1.8009 Mega-GOA	GOA:1800875
2026-07-19 14:42	822.256 Kilo-GOA	GOA:822256 (matches max-search lo)
2026-07-19 14:42	424.114 Kilo-GOA	GOA:424114

Same calendar day, ~1.5–2 hours earlier, explorer successfully recorded Giga-GOA and Mega-GOA withdraws — orders of magnitude above the ~824k amount that later returned 5110/P0001.

Takeaway: landing/stats prove higher single withdraws were possible and recorded; the 5110 failure is not explained by “config max_wire forbids Kilo-GOA” or by the public explorer balance display. Ops should compare landing recent_withdraws + balance_explorer with mon mint 5110 times when filing bank bugs.

Earlier mon observations (same error class, larger pins): classic ladder e.g. GOA:747827072844319 → mint 500/5110/P0001; absolute max / max-1 ceiling probes — see also classic soft CEILING_REJECT and max-ladder hi bound.

Exchange intro (context, not the mint path): https://exchange.hacktivism.ch/intro/stats.json — e.g. snapshot same day: wire_in_amount ~Peta-GOA scale, withdraw_amount ~GOA:1.79e6 coin-side; bank landing is the right place for explorer mint history.

---

4. Exchange not auto-known for bank deep links / withdraw URIs

Seen	Android endless spinner; GOA bank.hacktivism.ch withdraw; fixed in app branch fix/bank-withdraw-auto-exchange
Area	core / cli
Problem	Withdraw URI names an exchange the wallet has never added → stuck loading / empty exchange list. CLI requires explicit exchanges add + update + accept-tos every fresh DB.
Workaround	e2e always adds exchange before bank withdraw ladder; Android needs ensureExchange-style fix.
Wanted	CLI/core: on accept-uri withdraw, auto-add exchange from URI/bank details (allowCompletion), then surface ToS if needed in one command.

---

5. Default ports in URIs (:443 / :80) break round-trips

Seen	ladder/e2e strip ports; QR checks; mint URIs with :443
Area	cli / bank / doc
Problem	Some bank/wallet outputs include taler://…host:443/…. QR validators and some clients reject or double-normalize inconsistently.
Workaround	sed 's/:443\//\//g' everywhere in automation.
Wanted	Canonical form without default ports in all wallet-cli and bank integration outputs; document as invariant.

---

6. No first-class “demo withdraw” / explorer mint in wallet-cli

Seen	e2e uses bank admin + bank withdraw create; ladder/gui-chain reimplement explorer mint in Python
Area	cli / ops
Problem	Automation always hand-rolls Basic auth → token → POST …/withdrawals → taler_withdraw_uri. Easy to get wrong (auth header, amount currency).
Workaround	Shared Python in goa-chain / ladder / e2e; secrets files.
Wanted	Optional helper: wallet-cli testing mint-withdraw --bank=URL --user=explorer --password-file=… --amount=GOA:10 (or bank-side only tool in libeufin) emitting a clean URI.

---

7. Long-lived wallet process required for pay/withdraw progress

Seen	e2e advanced serve / shepherd socket
Area	cli
Problem	One-shot CLI invocations do not keep background work alive; without serve/shepherd, withdraw/pay may stall mid-flight.
Workaround	Start serve once per e2e run; route wcli through the socket.
Wanted	Documented, supported “session mode”: start/stop serve; or each mutating command optionally --background-until=… with timeout.

---

8. ToS accept is a separate step that fails silently or blocks

Seen	e2e exchanges accept-tos; ladder after add; Android GUI ToS screens
Area	cli / core
Problem	Fresh exchange → operations need ToS; CLI must call accept-tos explicitly. Failures are easy to miss in multi-step scripts.
Workaround	Always accept-tos after update in e2e/ladder.
Wanted	accept-uri --accept-tos or auto-prompt with non-interactive --yes that covers exchange ToS for that URI’s exchange.

---

9. Pay template / public order path is merchant-shaped, not CLI-shaped

Seen	e2e shop templates, stage farmer shops, ladder private orders
Area	cli / doc
Problem	Wallet-cli pays via handle-uri on taler://pay/…; creating the order is always custom curl (template POST or private order + token). No unified “pay this amount to instance” for public templates.
Workaround	Monitoring builds URI externally then handle-uri --yes.
Wanted	Documented recipe only, or wallet-cli testing pay-template --base=… --instance=… --id=… for demos.

---

10. Amount / currency parsing edge cases

Seen	ladder min denom 0.01 vs GOA 1e-6; zero withdraw rejected (HTTP 409 / amount too low); max wire ceilings
Area	cli / bank
Problem	CURRENCY:0 and sub-min amounts fail differently per stack; CLI error strings are not machine-stable.
Workaround	Soft-skip zero rung; clamp ladder min to exchange denoms / bank max_wire.
Wanted	Structured errors (AMOUNT_too_small, currency_unknown) in JSON mode; wallet-cli amount validate --exchange=.

---

11. Finding the right taler-wallet-cli binary

Seen	find_wallet_cli in lib.sh; hardcoded laptop paths; wrapper vs .mjs
Area	ops / cli
Problem	Debian package is a shell wrapper; some tools need node …/taler-wallet-cli.mjs. Hardcoded /Users/… paths break other hosts.
Workaround	find_wallet_cli search list; WALLET_CLI= override.
Wanted	Single install story: wallet-cli --version JSON with path + libversion; no need to pass .mjs to node by hand.

---

12. Secrets and multi-stack confusion

Seen	explorer vs admin password; GOA secrets used on stage; ladder EXP_PW
Area	ops
Problem	CLI does not know “which stack”; wrong password → 401 mid-ladder. Not a wallet-cli bug, but every CLI automation hits it.
Workaround	SECRETS_ROOT, stage SSH, STACK= profiles.
Wanted	Optional ~/.config/taler/stacks.d/goa.env convention documented next to wallet-cli; still no secrets in repo.

---

13. No stable machine-readable “step result” for scripts

Seen	e2e greps accept output; ladder scrapes JSON from mixed stdout
Area	cli
Problem	Human logs + occasional JSON blobs; hard to parse reliably.
Workaround	Python scrapers, temp files, tee.
Wanted	Global --json / --ndjson for all commands; one object per completed operation with ok, op, ids, amounts.

---

14. Pay settlement wait is symmetric to withdraw pain

Seen	e2e pay settle loops; ladder pay settle rounds
Area	cli / core
Problem	After handle-uri pay, success is “order paid” / balance drop / merchant order status — not one CLI wait.
Workaround	Short poll loops; never run-until-done.
Wanted	Same as §1–2: bounded wait on transaction id / order id.

---

15. --wallet-db=*.json is rejected (“memory backend not supported”)

Seen	2026-07-19 magikoopa wallet-cli 1.6.8 / 1.6.11 bundle: any command with --wallet-db=/tmp/foo.json fails immediately with Error: memory backend not supported
Area	cli / ops
Problem	Current node host treats storage paths ending in .json as unsupported memory backend and throws before opening sqlite. Old notes / local tests that used WALLET_DB=….json break. Accept/transactions then never emit reserve_pub → ladder §3 empty-scrape WARNs.
Workaround	Use a non-.json path, e.g. --wallet-db=$SCRATCH/wallet.sqlite3 (ladder already prefers wallet.sqlite3). Never pass ….json as wallet-db.
Wanted	Clearer error: wallet-db path must not end in .json; use a directory or .sqlite3 path. Document legal path forms in --help.

---

16. taler-helper-sqlite3 requires Python ≥ 3.11 on PATH

Seen	magikoopa: helper is a Python script (#!/usr/bin/env python3); with Apple /usr/bin/python3 3.9.6 → FATAL: python version >=3.11 required but running on 3.9.6 in accept/tx logs; wallet may still exit 0 on some paths while stdout is FATAL-only
Area	ops / cli
Problem	Wallet spawns taler-helper-sqlite3 from PATH. Wrong python3 ⇒ sqlite backend fails ⇒ no real wallet state ⇒ no reserve_pub for force-select.
Workaround	Put Python ≥3.11 (or 3.12+) first on PATH for mon jobs; ensure command -v taler-helper-sqlite3 works; smoke: taler-helper-sqlite3 starts without FATAL. Prefer Homebrew / user python3.11+ over macOS system 3.9 for agent PATH.
Wanted	Helper shebang or wrapper that finds a suitable python; wallet preflight error if helper HELLO fails (not silent empty logs).

---

17. Portable wall-clock for scripts (no GNU timeout assumed)

Seen	macOS agent shells: timeout: command not found; e2e already tries timeout then gtimeout then perl -e alarm
Area	ops / doc
Problem	GNU coreutils timeout is not on stock macOS. Hard-wiring it breaks portable repro and CI on Darwin.
Workaround	Prefer python3 subprocess.Popen(…).wait(timeout=…) for outer budgets; for wallet-cli inner calls keep e2e order: timeout → gtimeout → perl alarm. Document both.
Wanted	One suite helper run_with_budget SECS cmd… in lib.sh used by ladder/e2e/smoke.

---

18. How to withdraw (operator checklist, GOA laptop)

Portable path that works when §§15–16 are fixed:

1. PATH: taler-helper-sqlite3 + python3 ≥ 3.11 ahead of system 3.9.
2. Wallet DB: WDB=$TMP/wallet.sqlite3 — not *.json.
3. Exchange: exchanges add → update -f → accept-tos for https://exchange.hacktivism.ch/.
4. Bank mint (explorer): token with explorer password from SECRETS_ROOT / EXP_PW; POST …/accounts/explorer/withdrawals with e.g. {"amount":"GOA:1"} (avoid pure 0 and absolute max pins for happy-path tests).
5. Strip :443 from taler_withdraw_uri.
6. withdraw accept-uri --exchange URL URI.
7. Bank: poll …/taler-integration/withdrawal-operation/{WID} → when selected, confirm; if stuck pending, force-select with scraped reserve_pub (§3).
8. Settle: poll wallet balance / bank transfer_done — no run-until-done (§1).

If step 6’s stdout is only FATAL: python version… or memory backend not supported, stop and fix §15–16 before debugging force-select.

---

Priority shortlist for taler-wallet-cli / core

If only a few changes land, these unlock the most automation:

1. Bounded run-until-done / wait-for-state (§1, §2, §14)
2. Structured JSON on every command (§13)
3. Auto-add exchange + ToS on accept-uri (§4, §8)
4. Canonical URIs without :443 (§5)
5. Emit reserve_pub for last withdraw (§3)
6. Clear wallet-db path rules + helper preflight (§15, §16)

---

How we work around today (monitoring)

Issue	Monitoring behaviour
run-until-done	Disabled; balance + bank poll
serve	Optional long-lived socket in e2e
exchange	Explicit add/update/accept-tos
ladder explorer	read_secret / EXP_PW_FILE
stage maxima	bank max_wire + keys min denom
force-select	multi-rpub try; soft-skip confirm; empty scrape → WARN not balance fail
higher amounts	mint 5110/P0001 recurring server-side (§3b); landing shows much larger dated explorer withdraws (Giga/Mega same day); soft CEILING_REJECT + max-ladder hi; alt units in report
pay vs balance	never order more than wallet available (soft SKIP_BALANCE); pendingIncoming does not count as spendable
pay wait UX	after withdraw, board lists OK/OK_BANK in start order with live wallet pendingIncoming/done + timer (LADDER_PAY_WAIT_AVAILABLE_S, default 90s); same wallet DB as withdraw
wallet-db path	Prefer *.sqlite3 (never *.json as db path)
helper / python	mon PATH must include helper + python≥3.11

---

How to add an issue

### N. short title
| | |
|--|--|
| **Seen** | where / which stack |
| **Area** | cli / core / bank / ops / doc |
| **Problem** | … |
| **Workaround** | … |
| **Wanted** | concrete CLI/core behaviour |