# CLI automation notes (`taler-wallet-cli` + monitoring)

Living list of **recurring issues** seen while driving GOA (hacktivism) and
stage TESTPAYSAN via **CLI** (`taler-wallet-cli`, bank/merchant HTTP, monitoring
e2e/ladder). Ordered roughly by **how often they hurt automation** and how
useful a fix in **wallet-cli / wallet-core** (or clearer APIs) would be.

Companions:
- Android GUI → [`android-test/GUI-AUTOMATION-NOTES.md`](./android-test/GUI-AUTOMATION-NOTES.md)
- Git / suite tree → [`VERSIONS.md`](./VERSIONS.md)

Scripts: `check_e2e.sh`, `check_amount_ladder.sh`, `taler-monitoring.sh`.

Legend:

| Tag | Meaning |
|-----|---------|
| **cli** | wallet-cli UX / flags / hang behaviour |
| **core** | wallet-core protocol / state machine |
| **bank** | libeufin-bank / integration API |
| **ops** | secrets, paths, multi-host |
| **doc** | missing or wrong documentation |

---

## 1. `run-until-done` is unusable for automation

| | |
|--|--|
| **Seen** | e2e, ladder, Android notes, macOS comments: *hangs / banned in monitoring* |
| **Area** | **cli** / **core** |
| **Problem** | `taler-wallet-cli run-until-done` (or long shepherd runs) often **never returns** or blocks CI-length timeouts. Monitoring **forbids** it and polls balance + bank `transfer_done` instead. |
| **Workaround** | `advanced serve` + socket (`WSOCK`) or pure poll loops (`LADDER_SETTLE_*`, e2e settle wait). |
| **Wanted for wallet-cli** | Bounded wait: e.g. `run-until-done --timeout=Ns --exit-on=withdrawn|paid|idle`; stable non-zero exit on timeout; progress on stdout/JSON. |

---

## 2. Withdraw completion is not observable from CLI alone

| | |
|--|--|
| **Seen** | e2e “coins missing after withdraw”; ladder `OK_BANK` vs wallet avail |
| **Area** | **cli** / **core** / **bank** |
| **Problem** | After `accept-uri` + bank confirm, coins may lag (wirewatch). CLI has no single “withdraw settled” command with clear success/failure; automation reimplements status from bank integration JSON + balance polls. |
| **Workaround** | Poll `balance` + `…/withdrawal-operation/{id}` (`status`, `transfer_done`, `selected_reserve_pub`). |
| **Wanted** | `wallet-cli withdrawals wait --id=… --timeout=` or JSON event stream: `selected → confirmed → coins-available`. |

---

## 3. `reserve_pub` / force-select is fragile (5114 + empty scrape)

| | |
|--|--|
| **Seen** | ladder force-select, e2e confirm path, Android spinner issues; **repro 2026-07-19 magikoopa** GOA ladder: `force-select skipped · no reserve_pub for this withdraw (WID=…); wallet may not have selected yet` |
| **Area** | **cli** / **core** / **bank** / **ops** |
| **Problem** | (a) Cumulative wallets re-emit old reserves; binding the wrong `reserve_pub` yields **HTTP 409 / code 5114** (“already bound”). (b) Automation scrapes accept/tx dumps for candidates — when the dump has **no** pub at all, ladder WARNs and skips force-select (soft), then often **SKIP_CONFIRM**. (c) Empty scrape is **not** the same as “bank out of money”; do not treat as balance failure. (d) **Mon bug (fixed v1.20.2):** extract used `json.loads(file_from_first_brace)` on wallet dumps that append log lines after JSON → **zero** objects parsed; regex only scanned accept (which has no `reservePub`) while the pub lives under `transactions[].withdrawalDetails.reservePub`. |
| **Workaround** | **Mon ≥1.20.2:** `JSONDecoder.raw_decode` + regex on accept **and** transactions; score by WID in `bankConfirmationUrl` / amount. Still: score pubs by WID/amount; try several; treat 5114 as “stale reserve” not “out of money”. Refresh `transactions` before each force-select try. Soft-skip confirm if still pending after polls. Fix wallet DB / helper first if accept/tx only print FATAL (see §15–16). |
| **Wanted** | CLI returns **the** `reserve_pub` for the just-accepted withdraw in structured JSON; `force-select` / bank-side select API documented with idempotency. Optional: bank integration already exposes `selected_reserve_pub` when wallet selected — prefer that over scraping wallet logs. |

### 3a. Ladder pin amounts hide mid-range behaviour

| | |
|--|--|
| **Seen** | `LADDER_STEPS=3` plan collapses to pins only: `GOA:0`, `max-1`, `max` (no random mid rungs) |
| **Area** | **ops** / **bank** / **cli** |
| **Problem** | **Zero** withdraw often never produces a usable `reserve_pub` / coins (wallet 7006 or no denoms). **Absolute max** and sometimes **max-1** hit **mint HTTP 500 / code 5110** SQL `P0001` (ceiling probe). Reproducing “real” force-select needs a **mid** amount (e.g. `GOA:1` … exchange min denom scale), not only pins. |
| **Workaround** | Use `LADDER_STEPS≥5` (or higher) so random mids exist; treat zero / absolute-max / max-1-5110 as probes (soft `CEILING_REJECT` in mon ≥1.20.2). |
| **Wanted** | Ladder plan docs: pin vs mid semantics; CLI structured error for amount-too-large / zero. |

### 3b. Mint fails with bank SQL `5110` / `P0001` (server-side; recurring)

| | |
|--|--|
| **Seen** | **Several times** on GOA (`bank.hacktivism.ch`), classic ladder high/random mids + ceiling pins **and** max-search bisect — not a one-off. Same class of error across 2026-07 runs (magikoopa / mon ladder). |
| **Area** | **bank** (libeufin / taler-corebank + **Postgres**) / **ops** — **not** mon SQL, **not** wallet-cli at mint time |
| **Call site** | mon `POST ${BANK}/accounts/explorer/withdrawals` with `{"amount":"GOA:…"}` only. Wallet/exchange not involved until mint returns 200 + URI. |
| **Response** | HTTP **500**, body e.g. `{ code: 5110, hint: "Unexpected sql error with state P0001" }` |
| **Codes** | **`5110` = `BANK_UNMANAGED_EXCEPTION`** (“no known exception types captured the exception”). **`P0001`** = PostgreSQL SQLSTATE for application `RAISE EXCEPTION` (function/trigger), not a mon bug and not max_wire config alone. |
| **Config check** | Public `/config` still allows huge wires (`max_wire_transfer_amount` ≈ `GOA:4503599627370496.99999999`, `default_debit_threshold` same order). Failing mints are often **far below** that (e.g. ~Kilo-GOA). |
| **API mismatch** | Corebank docs: insufficient funds should be HTTP **409**. Getting **500/5110** means the bank is **leaking** an unmapped SQL error instead of a clean client error. |
| **Problem** | Live mint “ceiling” is **opaque and unstable**. Do **not** treat 5110 as wallet/`reserve_pub` failure when mint never succeeded. Do **not** assume “only absolute pin:max fails” — mid and max-search probes hit the same error. |
| **Workaround (mon ≥1.21)** | Soft `CEILING_REJECT` on 5110/P0001 (classic mids + max pins); **max-ladder** treats mint reject as too-high upper bound and bisects. Keep mid-range rungs for real withdraw/confirm tests. |
| **Wanted (bank)** | Map the SQL condition to a structured error (insufficient funds / amount too large / debit limit) — **not** `BANK_UNMANAGED_EXCEPTION`. Log full `RAISE` text server-side. |

#### What max-search saw (example run, 2026-07-19)

Ladder max-search log (magikoopa):

| Time (local mon log) | Probe | Result |
|----------------------|-------|--------|
| (bisect) | lo ≈ **GOA:822256** (822.256 Kilo-GOA) | still treatable as success path / lower bound |
| probe ~24 | **GOA:824385** (824.385 Kilo-GOA) | mint **HTTP 500 / 5110 / P0001** → soft too-high |
| after | hi := **GOA:824385** | continues bisect |

So the **live single-mint fail band** was ~**822–825 Kilo-GOA** in that run — **not** “only Tera/Peta pins”.

#### Landing page shows **much higher** amounts actually gotten (same day)

Public bank intro stats: `https://bank.hacktivism.ch/intro/stats.json` (and the HTML landing that consumes them). Snapshot **2026-07-19 ~14:47 CEST** (`generated_at_human`).

**Explorer balance still enormous** (so this is not “pool empty” in the UI sense):

| Field | Value (snapshot) |
|-------|------------------|
| `balance_explorer` | **4.5036 Peta-GOA** (`GOA:4503599627250215.32499924`) |
| cumulative `withdraws.total_amount` | **4.5036 Peta-GOA** (292 ops) |
| `withdraws.last_24h` | **3.2366 Peta-GOA** (45 ops) |

**Dated single withdraws that *did* mint** (from `recent_withdraws`, account=`explorer`) — all **before** or around the ~824k fail band above:

| When (CEST) | Amount (alt) | Raw |
|-------------|--------------|-----|
| **2026-07-19 14:19** | **1.9761 Giga-GOA** | `GOA:1976113563` |
| **2026-07-19 14:19** | **1.4495 Giga-GOA** | `GOA:1449489732` |
| **2026-07-19 14:26** | **201.636 Mega-GOA** | `GOA:201635991` |
| **2026-07-19 14:26** | **144.53 Mega-GOA** | `GOA:144530029` |
| **2026-07-19 14:29** | **3.4099 Mega-GOA** | `GOA:3409891` |
| **2026-07-19 14:29** | **1.8009 Mega-GOA** | `GOA:1800875` |
| **2026-07-19 14:42** | 822.256 Kilo-GOA | `GOA:822256` (matches max-search **lo**) |
| **2026-07-19 14:42** | 424.114 Kilo-GOA | `GOA:424114` |

Same calendar day, **~1.5–2 hours earlier**, explorer successfully recorded **Giga-GOA** and **Mega-GOA** withdraws — orders of magnitude **above** the ~824k amount that later returned **5110/P0001**.

**Takeaway:** landing/stats prove higher single withdraws were **possible and recorded**; the 5110 failure is **not** explained by “config max_wire forbids Kilo-GOA” or by the public explorer balance display. Ops should compare **landing `recent_withdraws` + `balance_explorer`** with mon mint 5110 times when filing bank bugs.

**Earlier mon observations (same error class, larger pins):** classic ladder e.g. `GOA:747827072844319` → mint 500/5110/P0001; absolute max / max-1 ceiling probes — see also classic soft `CEILING_REJECT` and max-ladder hi bound.

**Exchange intro (context, not the mint path):** `https://exchange.hacktivism.ch/intro/stats.json` — e.g. snapshot same day: `wire_in_amount` ~Peta-GOA scale, `withdraw_amount` ~`GOA:1.79e6` coin-side; bank landing is the right place for **explorer mint** history.

---

## 4. Exchange not auto-known for bank deep links / withdraw URIs

| | |
|--|--|
| **Seen** | Android endless spinner; GOA `bank.hacktivism.ch` withdraw; fixed in app branch `fix/bank-withdraw-auto-exchange` |
| **Area** | **core** / **cli** |
| **Problem** | Withdraw URI names an exchange the wallet has never added → stuck loading / empty exchange list. CLI requires explicit `exchanges add` + `update` + `accept-tos` every fresh DB. |
| **Workaround** | e2e always adds exchange before bank withdraw ladder; Android needs ensureExchange-style fix. |
| **Wanted** | CLI/core: on `accept-uri` withdraw, **auto-add** exchange from URI/bank details (`allowCompletion`), then surface ToS if needed in one command. |

---

## 5. Default ports in URIs (`:443` / `:80`) break round-trips

| | |
|--|--|
| **Seen** | ladder/e2e strip ports; QR checks; mint URIs with `:443` |
| **Area** | **cli** / **bank** / **doc** |
| **Problem** | Some bank/wallet outputs include `taler://…host:443/…`. QR validators and some clients reject or double-normalize inconsistently. |
| **Workaround** | `sed 's/:443\//\//g'` everywhere in automation. |
| **Wanted** | Canonical form without default ports in **all** wallet-cli and bank integration outputs; document as invariant. |

---

## 6. No first-class “demo withdraw” / explorer mint in wallet-cli

| | |
|--|--|
| **Seen** | e2e uses bank admin + bank withdraw create; ladder/gui-chain reimplement explorer mint in Python |
| **Area** | **cli** / **ops** |
| **Problem** | Automation always hand-rolls Basic auth → token → `POST …/withdrawals` → `taler_withdraw_uri`. Easy to get wrong (auth header, amount currency). |
| **Workaround** | Shared Python in `goa-chain` / ladder / e2e; secrets files. |
| **Wanted** | Optional helper: `wallet-cli testing mint-withdraw --bank=URL --user=explorer --password-file=… --amount=GOA:10` (or bank-side only tool in libeufin) emitting a clean URI. |

---

## 7. Long-lived wallet process required for pay/withdraw progress

| | |
|--|--|
| **Seen** | e2e `advanced serve` / shepherd socket |
| **Area** | **cli** |
| **Problem** | One-shot CLI invocations do not keep background work alive; without serve/shepherd, withdraw/pay may stall mid-flight. |
| **Workaround** | Start serve once per e2e run; route `wcli` through the socket. |
| **Wanted** | Documented, supported “session mode”: start/stop serve; or each mutating command optionally `--background-until=…` with timeout. |

---

## 8. ToS accept is a separate step that fails silently or blocks

| | |
|--|--|
| **Seen** | e2e `exchanges accept-tos`; ladder after add; Android GUI ToS screens |
| **Area** | **cli** / **core** |
| **Problem** | Fresh exchange → operations need ToS; CLI must call `accept-tos` explicitly. Failures are easy to miss in multi-step scripts. |
| **Workaround** | Always `accept-tos` after `update` in e2e/ladder. |
| **Wanted** | `accept-uri --accept-tos` or auto-prompt with non-interactive `--yes` that covers exchange ToS for that URI’s exchange. |

---

## 9. Pay template / public order path is merchant-shaped, not CLI-shaped

| | |
|--|--|
| **Seen** | e2e shop templates, stage farmer shops, ladder private orders |
| **Area** | **cli** / **doc** |
| **Problem** | Wallet-cli pays via `handle-uri` on `taler://pay/…`; creating the order is always custom curl (template POST or private order + token). No unified “pay this amount to instance” for public templates. |
| **Workaround** | Monitoring builds URI externally then `handle-uri --yes`. |
| **Wanted** | Documented recipe only, or `wallet-cli testing pay-template --base=… --instance=… --id=…` for demos. |

---

## 10. Amount / currency parsing edge cases

| | |
|--|--|
| **Seen** | ladder min denom 0.01 vs GOA 1e-6; zero withdraw rejected (HTTP 409 / amount too low); max wire ceilings |
| **Area** | **cli** / **bank** |
| **Problem** | `CURRENCY:0` and sub-min amounts fail differently per stack; CLI error strings are not machine-stable. |
| **Workaround** | Soft-skip zero rung; clamp ladder min to exchange denoms / bank max_wire. |
| **Wanted** | Structured errors (`AMOUNT_too_small`, `currency_unknown`) in JSON mode; `wallet-cli amount validate --exchange=`. |

---

## 11. Finding the right `taler-wallet-cli` binary

| | |
|--|--|
| **Seen** | `find_wallet_cli` in `lib.sh`; hardcoded laptop paths; wrapper vs `.mjs` |
| **Area** | **ops** / **cli** |
| **Problem** | Debian package is a shell wrapper; some tools need `node …/taler-wallet-cli.mjs`. Hardcoded `/Users/…` paths break other hosts. |
| **Workaround** | `find_wallet_cli` search list; `WALLET_CLI=` override. |
| **Wanted** | Single install story: `wallet-cli --version` JSON with path + libversion; no need to pass `.mjs` to node by hand. |

---

## 12. Secrets and multi-stack confusion

| | |
|--|--|
| **Seen** | explorer vs admin password; GOA secrets used on stage; ladder EXP_PW |
| **Area** | **ops** |
| **Problem** | CLI does not know “which stack”; wrong password → 401 mid-ladder. Not a wallet-cli bug, but every CLI automation hits it. |
| **Workaround** | `SECRETS_ROOT`, stage SSH, `STACK=` profiles. |
| **Wanted** | Optional `~/.config/taler/stacks.d/goa.env` convention documented next to wallet-cli; still no secrets in repo. |

---

## 13. No stable machine-readable “step result” for scripts

| | |
|--|--|
| **Seen** | e2e greps accept output; ladder scrapes JSON from mixed stdout |
| **Area** | **cli** |
| **Problem** | Human logs + occasional JSON blobs; hard to parse reliably. |
| **Workaround** | Python scrapers, temp files, `tee`. |
| **Wanted** | Global `--json` / `--ndjson` for all commands; one object per completed operation with `ok`, `op`, `ids`, `amounts`. |

---

## 14. Pay settlement wait is symmetric to withdraw pain

| | |
|--|--|
| **Seen** | e2e pay settle loops; ladder pay settle rounds |
| **Area** | **cli** / **core** |
| **Problem** | After `handle-uri` pay, success is “order paid” / balance drop / merchant order status — not one CLI wait. |
| **Workaround** | Short poll loops; never run-until-done. |
| **Wanted** | Same as §1–2: bounded wait on transaction id / order id. |

---

## 15. `--wallet-db=*.json` is rejected (“memory backend not supported”)

| | |
|--|--|
| **Seen** | **2026-07-19 magikoopa** wallet-cli **1.6.8** / **1.6.11** bundle: any command with `--wallet-db=/tmp/foo.json` fails immediately with `Error: memory backend not supported` |
| **Area** | **cli** / **ops** |
| **Problem** | Current node host treats storage paths ending in **`.json` as unsupported memory backend** and throws before opening sqlite. Old notes / local tests that used `WALLET_DB=….json` break. Accept/transactions then never emit `reserve_pub` → ladder §3 empty-scrape WARNs. |
| **Workaround** | Use a **non-`.json` path**, e.g. `--wallet-db=$SCRATCH/wallet.sqlite3` (ladder already prefers `wallet.sqlite3`). Never pass `….json` as wallet-db. |
| **Wanted** | Clearer error: `wallet-db path must not end in .json; use a directory or .sqlite3 path`. Document legal path forms in `--help`. |

---

## 16. `taler-helper-sqlite3` requires Python ≥ 3.11 on PATH

| | |
|--|--|
| **Seen** | magikoopa: helper is a **Python** script (`#!/usr/bin/env python3`); with Apple `/usr/bin/python3` **3.9.6** → `FATAL: python version >=3.11 required but running on 3.9.6` in accept/tx logs; wallet may still exit 0 on some paths while stdout is FATAL-only |
| **Area** | **ops** / **cli** |
| **Problem** | Wallet spawns `taler-helper-sqlite3` from **PATH**. Wrong `python3` ⇒ sqlite backend fails ⇒ no real wallet state ⇒ no `reserve_pub` for force-select. |
| **Workaround** | Put **Python ≥3.11** (or 3.12+) first on PATH for mon jobs; ensure `command -v taler-helper-sqlite3` works; smoke: `taler-helper-sqlite3` starts without FATAL. Prefer Homebrew / user `python3.11+` over macOS system 3.9 for agent PATH. |
| **Wanted** | Helper shebang or wrapper that finds a suitable python; wallet preflight error if helper HELLO fails (not silent empty logs). |

---

## 17. Portable wall-clock for scripts (no GNU `timeout` assumed)

| | |
|--|--|
| **Seen** | macOS agent shells: `timeout: command not found`; e2e already tries `timeout` then `gtimeout` then `perl -e alarm` |
| **Area** | **ops** / **doc** |
| **Problem** | GNU coreutils `timeout` is not on stock macOS. Hard-wiring it breaks portable repro and CI on Darwin. |
| **Workaround** | Prefer **python3** `subprocess.Popen(…).wait(timeout=…)` for outer budgets; for wallet-cli inner calls keep e2e order: `timeout` → `gtimeout` → perl alarm. Document both. |
| **Wanted** | One suite helper `run_with_budget SECS cmd…` in `lib.sh` used by ladder/e2e/smoke. |

---

## 18. How to withdraw (operator checklist, GOA laptop)

Portable path that works when §§15–16 are fixed:

1. **PATH:** `taler-helper-sqlite3` + **python3 ≥ 3.11** ahead of system 3.9.  
2. **Wallet DB:** `WDB=$TMP/wallet.sqlite3` — **not** `*.json`.  
3. **Exchange:** `exchanges add` → `update -f` → `accept-tos` for `https://exchange.hacktivism.ch/`.  
4. **Bank mint (explorer):** token with explorer password from `SECRETS_ROOT` / `EXP_PW`; `POST …/accounts/explorer/withdrawals` with e.g. `{"amount":"GOA:1"}` (avoid pure **0** and absolute **max** pins for happy-path tests).  
5. **Strip** `:443` from `taler_withdraw_uri`.  
6. **`withdraw accept-uri --exchange URL URI`**.  
7. **Bank:** poll `…/taler-integration/withdrawal-operation/{WID}` → when `selected`, confirm; if stuck `pending`, force-select with scraped `reserve_pub` (§3).  
8. **Settle:** poll wallet balance / bank `transfer_done` — **no** `run-until-done` (§1).

If step 6’s stdout is only `FATAL: python version…` or `memory backend not supported`, **stop** and fix §15–16 before debugging force-select.

---

## Priority shortlist for `taler-wallet-cli` / core

If only a few changes land, these unlock the most automation:

1. **Bounded `run-until-done` / wait-for-state** (§1, §2, §14)  
2. **Structured JSON on every command** (§13)  
3. **Auto-add exchange + ToS on accept-uri** (§4, §8)  
4. **Canonical URIs without :443** (§5)  
5. **Emit reserve_pub for last withdraw** (§3)  
6. **Clear wallet-db path rules + helper preflight** (§15, §16)  

---

## How we work around today (monitoring)

| Issue | Monitoring behaviour |
|-------|----------------------|
| run-until-done | Disabled; balance + bank poll |
| serve | Optional long-lived socket in e2e |
| exchange | Explicit add/update/accept-tos |
| ladder explorer | `read_secret` / EXP_PW_FILE |
| stage maxima | bank `max_wire` + keys min denom |
| force-select | multi-rpub try; soft-skip confirm; empty scrape → WARN not balance fail |
| higher amounts | mint **5110/P0001** recurring server-side (§3b); landing shows much larger **dated** explorer withdraws (Giga/Mega same day); soft `CEILING_REJECT` + max-ladder hi; alt units in report |
| pay vs balance | never order more than wallet **available** (soft `SKIP_BALANCE`); pendingIncoming does not count as spendable |
| pay wait UX | after withdraw, board lists OK/OK_BANK in **start order** with live wallet `pendingIncoming`/`done` + timer (`LADDER_PAY_WAIT_AVAILABLE_S`, default 90s); same wallet DB as withdraw |
| wallet-db path | Prefer `*.sqlite3` (never `*.json` as db path) |
| helper / python | mon PATH must include helper + python≥3.11 |

---

## How to add an issue

```markdown
### N. short title
| | |
|--|--|
| **Seen** | where / which stack |
| **Area** | cli / core / bank / ops / doc |
| **Problem** | … |
| **Workaround** | … |
| **Wanted** | concrete CLI/core behaviour |
```