taler-monitoring/CLI-AUTOMATION-NOTES.md

CLI automation notes (taler-wallet-cli + monitoring)

Living list of recurring issues seen while driving GOA (hacktivism) and stage TESTPAYSAN via CLI (taler-wallet-cli, bank/merchant HTTP, monitoring e2e/ladder). Ordered roughly by how often they hurt automation and how useful a fix in wallet-cli / wallet-core (or clearer APIs) would be.

Companions:

• Android GUI → android-test/GUI-AUTOMATION-NOTES.md
• Git / suite tree → VERSIONS.md

Scripts: check_e2e.sh, check_amount_ladder.sh, taler-monitoring.sh.

Legend:

Tag	Meaning
cli	wallet-cli UX / flags / hang behaviour
core	wallet-core protocol / state machine
bank	libeufin-bank / integration API
ops	secrets, paths, multi-host
doc	missing or wrong documentation

---

1. run-until-done is unusable for automation

Seen	e2e, ladder, Android notes, macOS comments: hangs / banned in monitoring
Area	cli / core
Problem	taler-wallet-cli run-until-done (or long shepherd runs) often never returns or blocks CI-length timeouts. Monitoring forbids it and polls balance + bank transfer_done instead.
Workaround	advanced serve + socket (WSOCK) or pure poll loops (LADDER_SETTLE_*, e2e settle wait).
Wanted for wallet-cli	Bounded wait: e.g. `run-until-done --timeout=Ns --exit-on=withdrawn

---

2. Withdraw completion is not observable from CLI alone

Seen	e2e “coins missing after withdraw”; ladder OK_BANK vs wallet avail
Area	cli / core / bank
Problem	After accept-uri + bank confirm, coins may lag (wirewatch). CLI has no single “withdraw settled” command with clear success/failure; automation reimplements status from bank integration JSON + balance polls.
Workaround	Poll balance + …/withdrawal-operation/{id} (status, transfer_done, selected_reserve_pub).
Wanted	wallet-cli withdrawals wait --id=… --timeout= or JSON event stream: selected → confirmed → coins-available.

---

3. reserve_pub / force-select is fragile (5114 + empty scrape)

Seen	ladder force-select, e2e confirm path, Android spinner issues; repro 2026-07-19 magikoopa GOA ladder: force-select skipped · no reserve_pub for this withdraw (WID=…); wallet may not have selected yet
Area	cli / core / bank / ops
Problem	(a) Cumulative wallets re-emit old reserves; binding the wrong reserve_pub yields HTTP 409 / code 5114 (“already bound”). (b) Automation scrapes accept/tx dumps for candidates — when the dump has no pub at all, ladder WARNs and skips force-select (soft), then often SKIP_CONFIRM. (c) Empty scrape is not the same as “bank out of money”; do not treat as balance failure. (d) Mon bug (fixed v1.20.2): extract used json.loads(file_from_first_brace) on wallet dumps that append log lines after JSON → zero objects parsed; regex only scanned accept (which has no reservePub) while the pub lives under transactions[].withdrawalDetails.reservePub.
Workaround	Mon ≥1.20.2: JSONDecoder.raw_decode + regex on accept and transactions; score by WID in bankConfirmationUrl / amount. Still: score pubs by WID/amount; try several; treat 5114 as “stale reserve” not “out of money”. Refresh transactions before each force-select try. Soft-skip confirm if still pending after polls. Fix wallet DB / helper first if accept/tx only print FATAL (see §15–16).
Wanted	CLI returns the reserve_pub for the just-accepted withdraw in structured JSON; force-select / bank-side select API documented with idempotency. Optional: bank integration already exposes selected_reserve_pub when wallet selected — prefer that over scraping wallet logs.

3a. Ladder pin amounts hide mid-range behaviour

Seen	LADDER_STEPS=3 plan collapses to pins only: GOA:0, max-1, max (no random mid rungs)
Area	ops / bank / cli
Problem	Zero withdraw often never produces a usable reserve_pub / coins (wallet 7006 or no denoms). Absolute max and sometimes max-1 hit mint HTTP 500 / code 5110 SQL P0001 (ceiling probe). Reproducing “real” force-select needs a mid amount (e.g. GOA:1 … exchange min denom scale), not only pins.
Workaround	Use LADDER_STEPS≥5 (or higher) so random mids exist; treat zero / absolute-max / max-1-5110 as probes (soft CEILING_REJECT in mon ≥1.20.2).
Wanted	Ladder plan docs: pin vs mid semantics; CLI structured error for amount-too-large / zero.

3b. We do not manage to withdraw in time (GOA) — bank path unreliable, not load

Bottom line	On live hacktivism GOA, automation does not get spendable coins in time for the pay phase (default wait 90 s). Max-search may report a “best” amount and bank may even show transfer_done; pay still sees available=GOA:0 with large pendingIncoming. Coins might arrive later (wirewatch / wallet) — at least not within mon’s budget. Recurring (several runs, 2026-07-19).
Area	bank + wire/wallet settle — not mon overload, not “laptop too slow”
Not the cause: ecosystem load	Outside-in check 2026-07-19 ~14:56 CEST: bank/exchange/merchant all HTTP 200, config ~50–150 ms, exchange /keys ~0.5 s. Container loadavg ~1.5–1.7, RSS bank ~737 MiB / exchange ~670 MiB, bank probes 47 ms, wirewatch_running: true. Stack is healthy under moderate load — failures are application/SQL/protocol, not “host on fire.”
Call site (mint)	POST ${BANK}/accounts/explorer/withdrawals → often HTTP 500 { code: 5110, hint: "Unexpected sql error with state P0001" } (BANK_UNMANAGED_EXCEPTION; Postgres P0001 = RAISE EXCEPTION). Should often be clean 409 (insufficient funds); instead opaque 500.
Config	/config still allows ~2⁵² wire max; live mint fails far below that. Explorer balance_explorer on landing still ~4.5 Peta-GOA.

Analysis — where withdraw dies

Stage	What we see	Manage to withdraw?
Mint (high amounts)	5110/P0001 while hi falls Tera→Giga→Mega→Kilo with lo=0	No — withdrawal never created
Mint (some mid amounts)	occasional HTTP 200 + URI	Mint only — not done
Confirm	force-select 200 then stuck selected, confirm HTTP 500 → mon SKIP_CONFIRM (bounds unchanged)	No — bank never wires; not out-of-money (§3 / 5114 class)
Bank settle	rare full path: confirm 204 + transfer_done	Bank side “OK_BANK”
Wallet coins	almost always available=0 within wait, large pendingIncoming, board done=0	Not in time for pay → SKIP_BALANCE / skip max-pay

Pay-wait board (same cumulative wallet), example:

summary: tsv_ok=2 · wallet_wd=14 (pendingIncoming=14 done=0)
timer 0s left — available still GOA:0
pendingIncoming=GOA:335776

Reading: only 2 TSV rows are full OK/OK_BANK, but the wallet holds 14 withdrawal txs still pendingIncoming and none done after the wait. Max-search also leaves many mint-OK / confirm-failed withdraws that inflate pendingIncoming and may never complete. After 90 s + short run-pending: still no spendable balance → we do not manage to withdraw in time for pay (whether coins would appear later is unknown to the job).

What did work (sparse) — for contrast

When / run	What worked	Amounts
Landing 2026-07-19 14:19–14:29 CEST	Explorer singles recorded on bank intro stats	~1.5–2 Giga-GOA, ~145–202 Mega-GOA, ~1.8–3.4 Mega-GOA
Max-search run A (same day)	Full bank path (mint→confirm→transfer_done) for two amounts only; mon best	GOA:424114, GOA:822256 (~822 Kilo best)
Max-search run B (later)	Even fewer full successes; many mint OK + SKIP_CONFIRM; mon best collapsed	e.g. full success around GOA:12276 / GOA:17257 (~17 Kilo best)
Pay after either	Spendable available within 90 s	GOA:0 (not in time)

So: landing/history prove large withdraws were possible earlier the same day; automation does not manage a usable withdraw in time (best amount drops run-to-run; coins not available inside mon’s wait).

Mon behaviour (workaround only)

• 5110 → soft CEILING_REJECT / max-search hi:=
• Full OK only raises lo; SKIP_CONFIRM does not
• Pay waits up to LADDER_PAY_WAIT_AVAILABLE_S (default 90) with withdraw board; then SKIP_BALANCE if still 0
• Does not fix bank SQL mapping, stuck confirm, or wirewatch→coins lag

Wanted (bank / exchange ops)

1. Map mint P0001 to structured errors (not 5110).
2. Confirm after force-select must not stick selected + HTTP 500.
3. After transfer_done, coins must become wallet available (or document delay + API to wait).
4. File bugs with: max-search “what worked” table + landing recent_withdraws/balance_explorer + load note (healthy) + pay board tsv_ok vs wallet_wd/pendingIncoming/done.

Sources: https://bank.hacktivism.ch/intro/stats.json, exchange/merchant intro stats, mon max-ladder logs 2026-07-19.

---

4. Exchange not auto-known for bank deep links / withdraw URIs

Seen	Android endless spinner; GOA bank.hacktivism.ch withdraw; fixed in app branch fix/bank-withdraw-auto-exchange
Area	core / cli
Problem	Withdraw URI names an exchange the wallet has never added → stuck loading / empty exchange list. CLI requires explicit exchanges add + update + accept-tos every fresh DB.
Workaround	e2e always adds exchange before bank withdraw ladder; Android needs ensureExchange-style fix.
Wanted	CLI/core: on accept-uri withdraw, auto-add exchange from URI/bank details (allowCompletion), then surface ToS if needed in one command.

---

5. Default ports in URIs (:443 / :80) break round-trips

Seen	ladder/e2e strip ports; QR checks; mint URIs with :443
Area	cli / bank / doc
Problem	Some bank/wallet outputs include taler://…host:443/…. QR validators and some clients reject or double-normalize inconsistently.
Workaround	mon ≥1.23.4: normalize_taler_uri (lib.sh) on ladder/e2e/pay; bank landing demo-withdraw-api.py normalizes mint JSON; urls phase fails GOA/local if public landings still emit :443/:80 in taler:// (www.taler-uri-ports-*, unit tests/test_normalize_taler_uri.sh).
Wanted	libeufin never emits default ports (source fix).

---

6. No first-class “demo withdraw” / explorer mint in wallet-cli

Seen	e2e uses bank admin + bank withdraw create; ladder/gui-chain reimplement explorer mint in Python
Area	cli / ops
Problem	Automation always hand-rolls Basic auth → token → POST …/withdrawals → taler_withdraw_uri. Easy to get wrong (auth header, amount currency).
Workaround	Shared Python in goa-chain / ladder / e2e; secrets files.
Wanted	Optional helper: wallet-cli testing mint-withdraw --bank=URL --user=explorer --password-file=… --amount=GOA:10 (or bank-side only tool in libeufin) emitting a clean URI.

---

7. Long-lived wallet process required for pay/withdraw progress

Seen	e2e advanced serve / shepherd socket
Area	cli
Problem	One-shot CLI invocations do not keep background work alive; without serve/shepherd, withdraw/pay may stall mid-flight.
Workaround	Start serve once per e2e run; route wcli through the socket.
Wanted	Documented, supported “session mode”: start/stop serve; or each mutating command optionally --background-until=… with timeout.

---

8. ToS accept is a separate step that fails silently or blocks

Seen	e2e exchanges accept-tos; ladder after add; Android GUI ToS screens
Area	cli / core
Problem	Fresh exchange → operations need ToS; CLI must call accept-tos explicitly. Failures are easy to miss in multi-step scripts.
Workaround	Always accept-tos after update in e2e/ladder.
Wanted	accept-uri --accept-tos or auto-prompt with non-interactive --yes that covers exchange ToS for that URI’s exchange.

---

9. Pay template / public order path is merchant-shaped, not CLI-shaped

Seen	e2e shop templates, stage farmer shops, ladder private orders
Area	cli / doc
Problem	Wallet-cli pays via handle-uri on taler://pay/…; creating the order is always custom curl (template POST or private order + token). No unified “pay this amount to instance” for public templates.
Workaround	Monitoring builds URI externally then handle-uri --yes.
Wanted	Documented recipe only, or wallet-cli testing pay-template --base=… --instance=… --id=… for demos.

---

10. Amount / currency parsing edge cases

Seen	ladder min denom 0.01 vs GOA 1e-6; zero withdraw rejected (HTTP 409 / amount too low); max wire ceilings
Area	cli / bank
Problem	CURRENCY:0 and sub-min amounts fail differently per stack; CLI error strings are not machine-stable.
Workaround	Soft-skip zero rung; clamp ladder min to exchange denoms / bank max_wire.
Wanted	Structured errors (AMOUNT_too_small, currency_unknown) in JSON mode; wallet-cli amount validate --exchange=.

---

11. Finding the right taler-wallet-cli binary

Seen	find_wallet_cli in lib.sh; hardcoded laptop paths; wrapper vs .mjs
Area	ops / cli
Problem	Debian package is a shell wrapper; some tools need node …/taler-wallet-cli.mjs. Hardcoded /Users/… paths break other hosts.
Workaround	find_wallet_cli search list; WALLET_CLI= override.
Wanted	Single install story: wallet-cli --version JSON with path + libversion; no need to pass .mjs to node by hand.

---

12. Secrets and multi-stack confusion

Seen	explorer vs admin password; GOA secrets used on stage; ladder EXP_PW
Area	ops
Problem	CLI does not know “which stack”; wrong password → 401 mid-ladder. Not a wallet-cli bug, but every CLI automation hits it.
Workaround	SECRETS_ROOT, stage SSH, STACK= profiles.
Wanted	Optional ~/.config/taler/stacks.d/goa.env convention documented next to wallet-cli; still no secrets in repo.

---

13. No stable machine-readable “step result” for scripts

Seen	e2e greps accept output; ladder scrapes JSON from mixed stdout
Area	cli
Problem	Human logs + occasional JSON blobs; hard to parse reliably.
Workaround	Python scrapers, temp files, tee.
Wanted	Global --json / --ndjson for all commands; one object per completed operation with ok, op, ids, amounts.

---

14. Pay settlement wait is symmetric to withdraw pain

Seen	e2e pay settle loops; ladder pay settle rounds
Area	cli / core
Problem	After handle-uri pay, success is “order paid” / balance drop / merchant order status — not one CLI wait.
Workaround	Short poll loops; never run-until-done.
Wanted	Same as §1–2: bounded wait on transaction id / order id.

---

15. --wallet-db=*.json is rejected (“memory backend not supported”)

Seen	2026-07-19 magikoopa wallet-cli 1.6.8 / 1.6.11 bundle: any command with --wallet-db=/tmp/foo.json fails immediately with Error: memory backend not supported
Area	cli / ops
Problem	Current node host treats storage paths ending in .json as unsupported memory backend and throws before opening sqlite. Old notes / local tests that used WALLET_DB=….json break. Accept/transactions then never emit reserve_pub → ladder §3 empty-scrape WARNs.
Workaround	Use a non-.json path, e.g. --wallet-db=$SCRATCH/wallet.sqlite3 (ladder already prefers wallet.sqlite3). Never pass ….json as wallet-db.
Wanted	Clearer error: wallet-db path must not end in .json; use a directory or .sqlite3 path. Document legal path forms in --help.

---

16. taler-helper-sqlite3 requires Python ≥ 3.11 on PATH

Seen	magikoopa: helper is a Python script (#!/usr/bin/env python3); with Apple /usr/bin/python3 3.9.6 → FATAL: python version >=3.11 required but running on 3.9.6 in accept/tx logs; wallet may still exit 0 on some paths while stdout is FATAL-only
Area	ops / cli
Problem	Wallet spawns taler-helper-sqlite3 from PATH. Wrong python3 ⇒ sqlite backend fails ⇒ no real wallet state ⇒ no reserve_pub for force-select.
Workaround	Put Python ≥3.11 (or 3.12+) first on PATH for mon jobs; ensure command -v taler-helper-sqlite3 works; smoke: taler-helper-sqlite3 starts without FATAL. Prefer Homebrew / user python3.11+ over macOS system 3.9 for agent PATH.
Wanted	Helper shebang or wrapper that finds a suitable python; wallet preflight error if helper HELLO fails (not silent empty logs).

---

17. Portable wall-clock for scripts (no GNU timeout assumed)

Seen	macOS agent shells: timeout: command not found; e2e already tries timeout then gtimeout then perl -e alarm
Area	ops / doc
Problem	GNU coreutils timeout is not on stock macOS. Hard-wiring it breaks portable repro and CI on Darwin.
Workaround	Prefer python3 subprocess.Popen(…).wait(timeout=…) for outer budgets; for wallet-cli inner calls keep e2e order: timeout → gtimeout → perl alarm. Document both.
Wanted	One suite helper run_with_budget SECS cmd… in lib.sh used by ladder/e2e/smoke.

---

18. How to withdraw (operator checklist, GOA laptop)

Portable path that works when §§15–16 are fixed:

1. PATH: taler-helper-sqlite3 + python3 ≥ 3.11 ahead of system 3.9.
2. Wallet DB: WDB=$TMP/wallet.sqlite3 — not *.json.
3. Exchange: exchanges add → update -f → accept-tos for https://exchange.hacktivism.ch/.
4. Bank mint (explorer): token with explorer password from SECRETS_ROOT / EXP_PW; POST …/accounts/explorer/withdrawals with e.g. {"amount":"GOA:1"} (avoid pure 0 and absolute max pins for happy-path tests).
5. Strip :443 from taler_withdraw_uri.
6. withdraw accept-uri --exchange URL URI.
7. Bank: poll …/taler-integration/withdrawal-operation/{WID} → when selected, confirm; if stuck pending, force-select with scraped reserve_pub (§3).
8. Settle: poll wallet balance / bank transfer_done — no run-until-done (§1).

If step 6’s stdout is only FATAL: python version… or memory backend not supported, stop and fix §15–16 before debugging force-select.

---

Priority shortlist for taler-wallet-cli / core

If only a few changes land, these unlock the most automation:

1. Bounded run-until-done / wait-for-state (§1, §2, §14)
2. Structured JSON on every command (§13)
3. Auto-add exchange + ToS on accept-uri (§4, §8)
4. Canonical URIs without :443 (§5)
5. Emit reserve_pub for last withdraw (§3)
6. Clear wallet-db path rules + helper preflight (§15, §16)

---

How we work around today (monitoring)

Issue	Monitoring behaviour
run-until-done	Disabled; balance + bank poll
serve	Optional long-lived socket in e2e
exchange	Explicit add/update/accept-tos
ladder explorer	read_secret / EXP_PW_FILE
stage maxima	bank max_wire + keys min denom
force-select	multi-rpub try; soft-skip confirm; empty scrape → WARN not balance fail
higher amounts / withdraw fail	§3b: we do not manage to withdraw in time on GOA (healthy load); mint 5110/P0001 + SKIP_CONFIRM + pendingIncoming; pay still available=0 after 90s; landing had Giga/Mega earlier same day
pay vs balance	never order more than wallet available (soft SKIP_BALANCE); pendingIncoming does not count as spendable
pay wait UX	board: OK/OK_BANK start order + live pendingIncoming/done; often tsv_ok≪wallet_wd and done=0 after 90s → skip pay
wallet-db path	Prefer *.sqlite3 (never *.json as db path)
helper / python	mon PATH must include helper + python≥3.11

---

How to add an issue

### N. short title
| | |
|--|--|
| **Seen** | where / which stack |
| **Area** | cli / core / bank / ops / doc |
| **Problem** | … |
| **Workaround** | … |
| **Wanted** | concrete CLI/core behaviour |