hernani/taler-monitoring
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
taler-monitoring/CLI-AUTOMATION-NOTES.md
Hernâni Marques 5c312833e1
docs: §3b mint 5110/P0001 with dated landing withdraw examples 
Record recurring bank SQL 5110/P0001 as server-side (not mon/wallet),
max-search fail band ~824k GOA, and same-day landing recent_withdraws
that still show Giga/Mega explorer mints succeeded earlier.
2026-07-19 14:49:08 +02:00

359 lines
21 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# CLI automation notes (`taler-wallet-cli` + monitoring)
Living list of **recurring issues** seen while driving GOA (hacktivism) and
stage TESTPAYSAN via **CLI** (`taler-wallet-cli`, bank/merchant HTTP, monitoring
e2e/ladder). Ordered roughly by **how often they hurt automation** and how
useful a fix in **wallet-cli / wallet-core** (or clearer APIs) would be.
Companions:
- Android GUI → [`android-test/GUI-AUTOMATION-NOTES.md`](./android-test/GUI-AUTOMATION-NOTES.md)
- Git / suite tree → [`VERSIONS.md`](./VERSIONS.md)
Scripts: `check_e2e.sh`, `check_amount_ladder.sh`, `taler-monitoring.sh`.
Legend:
| Tag | Meaning |
|-----|---------|
| **cli** | wallet-cli UX / flags / hang behaviour |
| **core** | wallet-core protocol / state machine |
| **bank** | libeufin-bank / integration API |
| **ops** | secrets, paths, multi-host |
| **doc** | missing or wrong documentation |
---
## 1. `run-until-done` is unusable for automation
| | |
|--|--|
| **Seen** | e2e, ladder, Android notes, macOS comments: *hangs / banned in monitoring* |
| **Area** | **cli** / **core** |
| **Problem** | `taler-wallet-cli run-until-done` (or long shepherd runs) often **never returns** or blocks CI-length timeouts. Monitoring **forbids** it and polls balance + bank `transfer_done` instead. |
| **Workaround** | `advanced serve` + socket (`WSOCK`) or pure poll loops (`LADDER_SETTLE_*`, e2e settle wait). |
| **Wanted for wallet-cli** | Bounded wait: e.g. `run-until-done --timeout=Ns --exit-on=withdrawn|paid|idle`; stable non-zero exit on timeout; progress on stdout/JSON. |
---
## 2. Withdraw completion is not observable from CLI alone
| | |
|--|--|
| **Seen** | e2e “coins missing after withdraw”; ladder `OK_BANK` vs wallet avail |
| **Area** | **cli** / **core** / **bank** |
| **Problem** | After `accept-uri` + bank confirm, coins may lag (wirewatch). CLI has no single “withdraw settled” command with clear success/failure; automation reimplements status from bank integration JSON + balance polls. |
| **Workaround** | Poll `balance` + `…/withdrawal-operation/{id}` (`status`, `transfer_done`, `selected_reserve_pub`). |
| **Wanted** | `wallet-cli withdrawals wait --id=… --timeout=` or JSON event stream: `selected → confirmed → coins-available`. |
---
## 3. `reserve_pub` / force-select is fragile (5114 + empty scrape)
| | |
|--|--|
| **Seen** | ladder force-select, e2e confirm path, Android spinner issues; **repro 2026-07-19 magikoopa** GOA ladder: `force-select skipped · no reserve_pub for this withdraw (WID=…); wallet may not have selected yet` |
| **Area** | **cli** / **core** / **bank** / **ops** |
| **Problem** | (a) Cumulative wallets re-emit old reserves; binding the wrong `reserve_pub` yields **HTTP 409 / code 5114** (“already bound”). (b) Automation scrapes accept/tx dumps for candidates — when the dump has **no** pub at all, ladder WARNs and skips force-select (soft), then often **SKIP_CONFIRM**. (c) Empty scrape is **not** the same as “bank out of money”; do not treat as balance failure. (d) **Mon bug (fixed v1.20.2):** extract used `json.loads(file_from_first_brace)` on wallet dumps that append log lines after JSON → **zero** objects parsed; regex only scanned accept (which has no `reservePub`) while the pub lives under `transactions[].withdrawalDetails.reservePub`. |
| **Workaround** | **Mon ≥1.20.2:** `JSONDecoder.raw_decode` + regex on accept **and** transactions; score by WID in `bankConfirmationUrl` / amount. Still: score pubs by WID/amount; try several; treat 5114 as “stale reserve” not “out of money”. Refresh `transactions` before each force-select try. Soft-skip confirm if still pending after polls. Fix wallet DB / helper first if accept/tx only print FATAL (see §15–16). |
| **Wanted** | CLI returns **the** `reserve_pub` for the just-accepted withdraw in structured JSON; `force-select` / bank-side select API documented with idempotency. Optional: bank integration already exposes `selected_reserve_pub` when wallet selected — prefer that over scraping wallet logs. |
### 3a. Ladder pin amounts hide mid-range behaviour
| | |
|--|--|
| **Seen** | `LADDER_STEPS=3` plan collapses to pins only: `GOA:0`, `max-1`, `max` (no random mid rungs) |
| **Area** | **ops** / **bank** / **cli** |
| **Problem** | **Zero** withdraw often never produces a usable `reserve_pub` / coins (wallet 7006 or no denoms). **Absolute max** and sometimes **max-1** hit **mint HTTP 500 / code 5110** SQL `P0001` (ceiling probe). Reproducing “real” force-select needs a **mid** amount (e.g. `GOA:1` … exchange min denom scale), not only pins. |
| **Workaround** | Use `LADDER_STEPS≥5` (or higher) so random mids exist; treat zero / absolute-max / max-1-5110 as probes (soft `CEILING_REJECT` in mon ≥1.20.2). |
| **Wanted** | Ladder plan docs: pin vs mid semantics; CLI structured error for amount-too-large / zero. |
### 3b. Mint fails with bank SQL `5110` / `P0001` (server-side; recurring)
| | |
|--|--|
| **Seen** | **Several times** on GOA (`bank.hacktivism.ch`), classic ladder high/random mids + ceiling pins **and** max-search bisect — not a one-off. Same class of error across 2026-07 runs (magikoopa / mon ladder). |
| **Area** | **bank** (libeufin / taler-corebank + **Postgres**) / **ops** — **not** mon SQL, **not** wallet-cli at mint time |
| **Call site** | mon `POST ${BANK}/accounts/explorer/withdrawals` with `{"amount":"GOA:…"}` only. Wallet/exchange not involved until mint returns 200 + URI. |
| **Response** | HTTP **500**, body e.g. `{ code: 5110, hint: "Unexpected sql error with state P0001" }` |
| **Codes** | **`5110` = `BANK_UNMANAGED_EXCEPTION`** (“no known exception types captured the exception”). **`P0001`** = PostgreSQL SQLSTATE for application `RAISE EXCEPTION` (function/trigger), not a mon bug and not max_wire config alone. |
| **Config check** | Public `/config` still allows huge wires (`max_wire_transfer_amount` ≈ `GOA:4503599627370496.99999999`, `default_debit_threshold` same order). Failing mints are often **far below** that (e.g. ~Kilo-GOA). |
| **API mismatch** | Corebank docs: insufficient funds should be HTTP **409**. Getting **500/5110** means the bank is **leaking** an unmapped SQL error instead of a clean client error. |
| **Problem** | Live mint “ceiling” is **opaque and unstable**. Do **not** treat 5110 as wallet/`reserve_pub` failure when mint never succeeded. Do **not** assume “only absolute pin:max fails” — mid and max-search probes hit the same error. |
| **Workaround (mon ≥1.21)** | Soft `CEILING_REJECT` on 5110/P0001 (classic mids + max pins); **max-ladder** treats mint reject as too-high upper bound and bisects. Keep mid-range rungs for real withdraw/confirm tests. |
| **Wanted (bank)** | Map the SQL condition to a structured error (insufficient funds / amount too large / debit limit) — **not** `BANK_UNMANAGED_EXCEPTION`. Log full `RAISE` text server-side. |
#### What max-search saw (example run, 2026-07-19)
Ladder max-search log (magikoopa):
| Time (local mon log) | Probe | Result |
|----------------------|-------|--------|
| (bisect) | lo ≈ **GOA:822256** (822.256 Kilo-GOA) | still treatable as success path / lower bound |
| probe ~24 | **GOA:824385** (824.385 Kilo-GOA) | mint **HTTP 500 / 5110 / P0001** → soft too-high |
| after | hi := **GOA:824385** | continues bisect |
So the **live single-mint fail band** was ~**822–825 Kilo-GOA** in that run — **not** “only Tera/Peta pins”.
#### Landing page shows **much higher** amounts actually gotten (same day)
Public bank intro stats: `https://bank.hacktivism.ch/intro/stats.json` (and the HTML landing that consumes them). Snapshot **2026-07-19 ~14:47 CEST** (`generated_at_human`).
**Explorer balance still enormous** (so this is not “pool empty” in the UI sense):
| Field | Value (snapshot) |
|-------|------------------|
| `balance_explorer` | **4.5036 Peta-GOA** (`GOA:4503599627250215.32499924`) |
| cumulative `withdraws.total_amount` | **4.5036 Peta-GOA** (292 ops) |
| `withdraws.last_24h` | **3.2366 Peta-GOA** (45 ops) |
**Dated single withdraws that *did* mint** (from `recent_withdraws`, account=`explorer`) — all **before** or around the ~824k fail band above:
| When (CEST) | Amount (alt) | Raw |
|-------------|--------------|-----|
| **2026-07-19 14:19** | **1.9761 Giga-GOA** | `GOA:1976113563` |
| **2026-07-19 14:19** | **1.4495 Giga-GOA** | `GOA:1449489732` |
| **2026-07-19 14:26** | **201.636 Mega-GOA** | `GOA:201635991` |
| **2026-07-19 14:26** | **144.53 Mega-GOA** | `GOA:144530029` |
| **2026-07-19 14:29** | **3.4099 Mega-GOA** | `GOA:3409891` |
| **2026-07-19 14:29** | **1.8009 Mega-GOA** | `GOA:1800875` |
| **2026-07-19 14:42** | 822.256 Kilo-GOA | `GOA:822256` (matches max-search **lo**) |
| **2026-07-19 14:42** | 424.114 Kilo-GOA | `GOA:424114` |
Same calendar day, **~1.5–2 hours earlier**, explorer successfully recorded **Giga-GOA** and **Mega-GOA** withdraws — orders of magnitude **above** the ~824k amount that later returned **5110/P0001**.
**Takeaway:** landing/stats prove higher single withdraws were **possible and recorded**; the 5110 failure is **not** explained by “config max_wire forbids Kilo-GOA” or by the public explorer balance display. Ops should compare **landing `recent_withdraws` + `balance_explorer`** with mon mint 5110 times when filing bank bugs.
**Earlier mon observations (same error class, larger pins):** classic ladder e.g. `GOA:747827072844319` → mint 500/5110/P0001; absolute max / max-1 ceiling probes — see also classic soft `CEILING_REJECT` and max-ladder hi bound.
**Exchange intro (context, not the mint path):** `https://exchange.hacktivism.ch/intro/stats.json` — e.g. snapshot same day: `wire_in_amount` ~Peta-GOA scale, `withdraw_amount` ~`GOA:1.79e6` coin-side; bank landing is the right place for **explorer mint** history.
---
## 4. Exchange not auto-known for bank deep links / withdraw URIs
| | |
|--|--|
| **Seen** | Android endless spinner; GOA `bank.hacktivism.ch` withdraw; fixed in app branch `fix/bank-withdraw-auto-exchange` |
| **Area** | **core** / **cli** |
| **Problem** | Withdraw URI names an exchange the wallet has never added → stuck loading / empty exchange list. CLI requires explicit `exchanges add` + `update` + `accept-tos` every fresh DB. |
| **Workaround** | e2e always adds exchange before bank withdraw ladder; Android needs ensureExchange-style fix. |
| **Wanted** | CLI/core: on `accept-uri` withdraw, **auto-add** exchange from URI/bank details (`allowCompletion`), then surface ToS if needed in one command. |
---
## 5. Default ports in URIs (`:443` / `:80`) break round-trips
| | |
|--|--|
| **Seen** | ladder/e2e strip ports; QR checks; mint URIs with `:443` |
| **Area** | **cli** / **bank** / **doc** |
| **Problem** | Some bank/wallet outputs include `taler://…host:443/…`. QR validators and some clients reject or double-normalize inconsistently. |
| **Workaround** | `sed 's/:443\//\//g'` everywhere in automation. |
| **Wanted** | Canonical form without default ports in **all** wallet-cli and bank integration outputs; document as invariant. |
---
## 6. No first-class “demo withdraw” / explorer mint in wallet-cli
| | |
|--|--|
| **Seen** | e2e uses bank admin + bank withdraw create; ladder/gui-chain reimplement explorer mint in Python |
| **Area** | **cli** / **ops** |
| **Problem** | Automation always hand-rolls Basic auth → token → `POST …/withdrawals` → `taler_withdraw_uri`. Easy to get wrong (auth header, amount currency). |
| **Workaround** | Shared Python in `goa-chain` / ladder / e2e; secrets files. |
| **Wanted** | Optional helper: `wallet-cli testing mint-withdraw --bank=URL --user=explorer --password-file=… --amount=GOA:10` (or bank-side only tool in libeufin) emitting a clean URI. |
---
## 7. Long-lived wallet process required for pay/withdraw progress
| | |
|--|--|
| **Seen** | e2e `advanced serve` / shepherd socket |
| **Area** | **cli** |
| **Problem** | One-shot CLI invocations do not keep background work alive; without serve/shepherd, withdraw/pay may stall mid-flight. |
| **Workaround** | Start serve once per e2e run; route `wcli` through the socket. |
| **Wanted** | Documented, supported “session mode”: start/stop serve; or each mutating command optionally `--background-until=…` with timeout. |
---
## 8. ToS accept is a separate step that fails silently or blocks
| | |
|--|--|
| **Seen** | e2e `exchanges accept-tos`; ladder after add; Android GUI ToS screens |
| **Area** | **cli** / **core** |
| **Problem** | Fresh exchange → operations need ToS; CLI must call `accept-tos` explicitly. Failures are easy to miss in multi-step scripts. |
| **Workaround** | Always `accept-tos` after `update` in e2e/ladder. |
| **Wanted** | `accept-uri --accept-tos` or auto-prompt with non-interactive `--yes` that covers exchange ToS for that URI’s exchange. |
---
## 9. Pay template / public order path is merchant-shaped, not CLI-shaped
| | |
|--|--|
| **Seen** | e2e shop templates, stage farmer shops, ladder private orders |
| **Area** | **cli** / **doc** |
| **Problem** | Wallet-cli pays via `handle-uri` on `taler://pay/…`; creating the order is always custom curl (template POST or private order + token). No unified “pay this amount to instance” for public templates. |
| **Workaround** | Monitoring builds URI externally then `handle-uri --yes`. |
| **Wanted** | Documented recipe only, or `wallet-cli testing pay-template --base=… --instance=… --id=…` for demos. |
---
## 10. Amount / currency parsing edge cases
| | |
|--|--|
| **Seen** | ladder min denom 0.01 vs GOA 1e-6; zero withdraw rejected (HTTP 409 / amount too low); max wire ceilings |
| **Area** | **cli** / **bank** |
| **Problem** | `CURRENCY:0` and sub-min amounts fail differently per stack; CLI error strings are not machine-stable. |
| **Workaround** | Soft-skip zero rung; clamp ladder min to exchange denoms / bank max_wire. |
| **Wanted** | Structured errors (`AMOUNT_too_small`, `currency_unknown`) in JSON mode; `wallet-cli amount validate --exchange=`. |
---
## 11. Finding the right `taler-wallet-cli` binary
| | |
|--|--|
| **Seen** | `find_wallet_cli` in `lib.sh`; hardcoded laptop paths; wrapper vs `.mjs` |
| **Area** | **ops** / **cli** |
| **Problem** | Debian package is a shell wrapper; some tools need `node …/taler-wallet-cli.mjs`. Hardcoded `/Users/…` paths break other hosts. |
| **Workaround** | `find_wallet_cli` search list; `WALLET_CLI=` override. |
| **Wanted** | Single install story: `wallet-cli --version` JSON with path + libversion; no need to pass `.mjs` to node by hand. |
---
## 12. Secrets and multi-stack confusion
| | |
|--|--|
| **Seen** | explorer vs admin password; GOA secrets used on stage; ladder EXP_PW |
| **Area** | **ops** |
| **Problem** | CLI does not know “which stack”; wrong password → 401 mid-ladder. Not a wallet-cli bug, but every CLI automation hits it. |
| **Workaround** | `SECRETS_ROOT`, stage SSH, `STACK=` profiles. |
| **Wanted** | Optional `~/.config/taler/stacks.d/goa.env` convention documented next to wallet-cli; still no secrets in repo. |
---
## 13. No stable machine-readable “step result” for scripts
| | |
|--|--|
| **Seen** | e2e greps accept output; ladder scrapes JSON from mixed stdout |
| **Area** | **cli** |
| **Problem** | Human logs + occasional JSON blobs; hard to parse reliably. |
| **Workaround** | Python scrapers, temp files, `tee`. |
| **Wanted** | Global `--json` / `--ndjson` for all commands; one object per completed operation with `ok`, `op`, `ids`, `amounts`. |
---
## 14. Pay settlement wait is symmetric to withdraw pain
| | |
|--|--|
| **Seen** | e2e pay settle loops; ladder pay settle rounds |
| **Area** | **cli** / **core** |
| **Problem** | After `handle-uri` pay, success is “order paid” / balance drop / merchant order status — not one CLI wait. |
| **Workaround** | Short poll loops; never run-until-done. |
| **Wanted** | Same as §1–2: bounded wait on transaction id / order id. |
---
## 15. `--wallet-db=*.json` is rejected (“memory backend not supported”)
| | |
|--|--|
| **Seen** | **2026-07-19 magikoopa** wallet-cli **1.6.8** / **1.6.11** bundle: any command with `--wallet-db=/tmp/foo.json` fails immediately with `Error: memory backend not supported` |
| **Area** | **cli** / **ops** |
| **Problem** | Current node host treats storage paths ending in **`.json` as unsupported memory backend** and throws before opening sqlite. Old notes / local tests that used `WALLET_DB=….json` break. Accept/transactions then never emit `reserve_pub` → ladder §3 empty-scrape WARNs. |
| **Workaround** | Use a **non-`.json` path**, e.g. `--wallet-db=$SCRATCH/wallet.sqlite3` (ladder already prefers `wallet.sqlite3`). Never pass `….json` as wallet-db. |
| **Wanted** | Clearer error: `wallet-db path must not end in .json; use a directory or .sqlite3 path`. Document legal path forms in `--help`. |
---
## 16. `taler-helper-sqlite3` requires Python ≥ 3.11 on PATH
| | |
|--|--|
| **Seen** | magikoopa: helper is a **Python** script (`#!/usr/bin/env python3`); with Apple `/usr/bin/python3` **3.9.6** → `FATAL: python version >=3.11 required but running on 3.9.6` in accept/tx logs; wallet may still exit 0 on some paths while stdout is FATAL-only |
| **Area** | **ops** / **cli** |
| **Problem** | Wallet spawns `taler-helper-sqlite3` from **PATH**. Wrong `python3` ⇒ sqlite backend fails ⇒ no real wallet state ⇒ no `reserve_pub` for force-select. |
| **Workaround** | Put **Python ≥3.11** (or 3.12+) first on PATH for mon jobs; ensure `command -v taler-helper-sqlite3` works; smoke: `taler-helper-sqlite3` starts without FATAL. Prefer Homebrew / user `python3.11+` over macOS system 3.9 for agent PATH. |
| **Wanted** | Helper shebang or wrapper that finds a suitable python; wallet preflight error if helper HELLO fails (not silent empty logs). |
---
## 17. Portable wall-clock for scripts (no GNU `timeout` assumed)
| | |
|--|--|
| **Seen** | macOS agent shells: `timeout: command not found`; e2e already tries `timeout` then `gtimeout` then `perl -e alarm` |
| **Area** | **ops** / **doc** |
| **Problem** | GNU coreutils `timeout` is not on stock macOS. Hard-wiring it breaks portable repro and CI on Darwin. |
| **Workaround** | Prefer **python3** `subprocess.Popen(…).wait(timeout=…)` for outer budgets; for wallet-cli inner calls keep e2e order: `timeout` → `gtimeout` → perl alarm. Document both. |
| **Wanted** | One suite helper `run_with_budget SECS cmd…` in `lib.sh` used by ladder/e2e/smoke. |
---
## 18. How to withdraw (operator checklist, GOA laptop)
Portable path that works when §§15–16 are fixed:
1. **PATH:** `taler-helper-sqlite3` + **python3 ≥ 3.11** ahead of system 3.9.
2. **Wallet DB:** `WDB=$TMP/wallet.sqlite3` — **not** `*.json`.
3. **Exchange:** `exchanges add` → `update -f` → `accept-tos` for `https://exchange.hacktivism.ch/`.
4. **Bank mint (explorer):** token with explorer password from `SECRETS_ROOT` / `EXP_PW`; `POST …/accounts/explorer/withdrawals` with e.g. `{"amount":"GOA:1"}` (avoid pure **0** and absolute **max** pins for happy-path tests).
5. **Strip** `:443` from `taler_withdraw_uri`.
6. **`withdraw accept-uri --exchange URL URI`**.
7. **Bank:** poll `…/taler-integration/withdrawal-operation/{WID}` → when `selected`, confirm; if stuck `pending`, force-select with scraped `reserve_pub` (§3).
8. **Settle:** poll wallet balance / bank `transfer_done` — **no** `run-until-done` (§1).
If step 6’s stdout is only `FATAL: python version…` or `memory backend not supported`, **stop** and fix §15–16 before debugging force-select.
---
## Priority shortlist for `taler-wallet-cli` / core
If only a few changes land, these unlock the most automation:
1. **Bounded `run-until-done` / wait-for-state** (§1, §2, §14)
2. **Structured JSON on every command** (§13)
3. **Auto-add exchange + ToS on accept-uri** (§4, §8)
4. **Canonical URIs without :443** (§5)
5. **Emit reserve_pub for last withdraw** (§3)
6. **Clear wallet-db path rules + helper preflight** (§15, §16)
---
## How we work around today (monitoring)
| Issue | Monitoring behaviour |
|-------|----------------------|
| run-until-done | Disabled; balance + bank poll |
| serve | Optional long-lived socket in e2e |
| exchange | Explicit add/update/accept-tos |
| ladder explorer | `read_secret` / EXP_PW_FILE |
| stage maxima | bank `max_wire` + keys min denom |
| force-select | multi-rpub try; soft-skip confirm; empty scrape → WARN not balance fail |
| higher amounts | mint **5110/P0001** recurring server-side (§3b); landing shows much larger **dated** explorer withdraws (Giga/Mega same day); soft `CEILING_REJECT` + max-ladder hi; alt units in report |
| pay vs balance | never order more than wallet **available** (soft `SKIP_BALANCE`); pendingIncoming does not count as spendable |
| pay wait UX | after withdraw, board lists OK/OK_BANK in **start order** with live wallet `pendingIncoming`/`done` + timer (`LADDER_PAY_WAIT_AVAILABLE_S`, default 90s); same wallet DB as withdraw |
| wallet-db path | Prefer `*.sqlite3` (never `*.json` as db path) |
| helper / python | mon PATH must include helper + python≥3.11 |
---
## How to add an issue
```markdown
### N. short title
| | |
|--|--|
| **Seen** | where / which stack |
| **Area** | cli / core / bank / ops / doc |
| **Problem** | … |
| **Workaround** | … |
| **Wanted** | concrete CLI/core behaviour |
```
Reference in a new issue View git blame Copy permalink