hernani/taler-monitoring
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
taler-monitoring/CLI-AUTOMATION-NOTES.md
Hernâni Marques d420043863
fix 1.23.4: normalize taler:// :443/:80 + mon scan all landings 
Shared lib.sh normalize_taler_uri for ladder/e2e/pay. urls phase checks
public bank/exchange/merchant intros and bank mint JSON for residual
default ports (ERROR on GOA/local). Unit test included.
2026-07-19 15:38:25 +02:00

356 lines
21 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# CLI automation notes (`taler-wallet-cli` + monitoring)
Living list of **recurring issues** seen while driving GOA (hacktivism) and
stage TESTPAYSAN via **CLI** (`taler-wallet-cli`, bank/merchant HTTP, monitoring
e2e/ladder). Ordered roughly by **how often they hurt automation** and how
useful a fix in **wallet-cli / wallet-core** (or clearer APIs) would be.
Companions:
- Android GUI → [`android-test/GUI-AUTOMATION-NOTES.md`](./android-test/GUI-AUTOMATION-NOTES.md)
- Git / suite tree → [`VERSIONS.md`](./VERSIONS.md)
Scripts: `check_e2e.sh`, `check_amount_ladder.sh`, `taler-monitoring.sh`.
Legend:
| Tag | Meaning |
|-----|---------|
| **cli** | wallet-cli UX / flags / hang behaviour |
| **core** | wallet-core protocol / state machine |
| **bank** | libeufin-bank / integration API |
| **ops** | secrets, paths, multi-host |
| **doc** | missing or wrong documentation |
---
## 1. `run-until-done` is unusable for automation
| | |
|--|--|
| **Seen** | e2e, ladder, Android notes, macOS comments: *hangs / banned in monitoring* |
| **Area** | **cli** / **core** |
| **Problem** | `taler-wallet-cli run-until-done` (or long shepherd runs) often **never returns** or blocks CI-length timeouts. Monitoring **forbids** it and polls balance + bank `transfer_done` instead. |
| **Workaround** | `advanced serve` + socket (`WSOCK`) or pure poll loops (`LADDER_SETTLE_*`, e2e settle wait). |
| **Wanted for wallet-cli** | Bounded wait: e.g. `run-until-done --timeout=Ns --exit-on=withdrawn|paid|idle`; stable non-zero exit on timeout; progress on stdout/JSON. |
---
## 2. Withdraw completion is not observable from CLI alone
| | |
|--|--|
| **Seen** | e2e “coins missing after withdraw”; ladder `OK_BANK` vs wallet avail |
| **Area** | **cli** / **core** / **bank** |
| **Problem** | After `accept-uri` + bank confirm, coins may lag (wirewatch). CLI has no single “withdraw settled” command with clear success/failure; automation reimplements status from bank integration JSON + balance polls. |
| **Workaround** | Poll `balance` + `…/withdrawal-operation/{id}` (`status`, `transfer_done`, `selected_reserve_pub`). |
| **Wanted** | `wallet-cli withdrawals wait --id=… --timeout=` or JSON event stream: `selected → confirmed → coins-available`. |
---
## 3. `reserve_pub` / force-select is fragile (5114 + empty scrape)
| | |
|--|--|
| **Seen** | ladder force-select, e2e confirm path, Android spinner issues; **repro 2026-07-19 magikoopa** GOA ladder: `force-select skipped · no reserve_pub for this withdraw (WID=…); wallet may not have selected yet` |
| **Area** | **cli** / **core** / **bank** / **ops** |
| **Problem** | (a) Cumulative wallets re-emit old reserves; binding the wrong `reserve_pub` yields **HTTP 409 / code 5114** (“already bound”). (b) Automation scrapes accept/tx dumps for candidates — when the dump has **no** pub at all, ladder WARNs and skips force-select (soft), then often **SKIP_CONFIRM**. (c) Empty scrape is **not** the same as “bank out of money”; do not treat as balance failure. (d) **Mon bug (fixed v1.20.2):** extract used `json.loads(file_from_first_brace)` on wallet dumps that append log lines after JSON → **zero** objects parsed; regex only scanned accept (which has no `reservePub`) while the pub lives under `transactions[].withdrawalDetails.reservePub`. |
| **Workaround** | **Mon ≥1.20.2:** `JSONDecoder.raw_decode` + regex on accept **and** transactions; score by WID in `bankConfirmationUrl` / amount. Still: score pubs by WID/amount; try several; treat 5114 as “stale reserve” not “out of money”. Refresh `transactions` before each force-select try. Soft-skip confirm if still pending after polls. Fix wallet DB / helper first if accept/tx only print FATAL (see §15–16). |
| **Wanted** | CLI returns **the** `reserve_pub` for the just-accepted withdraw in structured JSON; `force-select` / bank-side select API documented with idempotency. Optional: bank integration already exposes `selected_reserve_pub` when wallet selected — prefer that over scraping wallet logs. |
### 3a. Ladder pin amounts hide mid-range behaviour
| | |
|--|--|
| **Seen** | `LADDER_STEPS=3` plan collapses to pins only: `GOA:0`, `max-1`, `max` (no random mid rungs) |
| **Area** | **ops** / **bank** / **cli** |
| **Problem** | **Zero** withdraw often never produces a usable `reserve_pub` / coins (wallet 7006 or no denoms). **Absolute max** and sometimes **max-1** hit **mint HTTP 500 / code 5110** SQL `P0001` (ceiling probe). Reproducing “real” force-select needs a **mid** amount (e.g. `GOA:1` … exchange min denom scale), not only pins. |
| **Workaround** | Use `LADDER_STEPS≥5` (or higher) so random mids exist; treat zero / absolute-max / max-1-5110 as probes (soft `CEILING_REJECT` in mon ≥1.20.2). |
| **Wanted** | Ladder plan docs: pin vs mid semantics; CLI structured error for amount-too-large / zero. |
### 3b. We do **not** manage to withdraw **in time** (GOA) — bank path unreliable, not load
| | |
|--|--|
| **Bottom line** | On live **hacktivism GOA**, automation **does not get spendable coins in time** for the pay phase (default wait **90 s**). Max-search may report a “best” amount and bank may even show `transfer_done`; pay still sees **`available=GOA:0`** with large **`pendingIncoming`**. Coins might arrive **later** (wirewatch / wallet) — **at least not within mon’s budget**. Recurring (several runs, 2026-07-19). |
| **Area** | **bank** + wire/wallet settle — **not** mon overload, **not** “laptop too slow” |
| **Not the cause: ecosystem load** | Outside-in check **2026-07-19 ~14:56 CEST**: bank/exchange/merchant all **HTTP 200**, config ~**50–150 ms**, exchange `/keys` ~**0.5 s**. Container **loadavg ~1.5–1.7**, RSS bank **~737 MiB** / exchange **~670 MiB**, bank probes **47 ms**, **`wirewatch_running: true`**. Stack is **healthy under moderate load** — failures are **application/SQL/protocol**, not “host on fire.” |
| **Call site (mint)** | `POST ${BANK}/accounts/explorer/withdrawals` → often HTTP **500** `{ code: 5110, hint: "Unexpected sql error with state P0001" }` (**`BANK_UNMANAGED_EXCEPTION`**; Postgres **`P0001`** = `RAISE EXCEPTION`). Should often be clean **409** (insufficient funds); instead opaque **500**. |
| **Config** | `/config` still allows ~**2⁵²** wire max; live mint fails **far below** that. Explorer **balance_explorer** on landing still **~4.5 Peta-GOA**. |
#### Analysis — where withdraw dies
| Stage | What we see | Manage to withdraw? |
|-------|-------------|---------------------|
| **Mint (high amounts)** | 5110/P0001 while hi falls Tera→Giga→Mega→Kilo with **lo=0** | **No** — withdrawal never created |
| **Mint (some mid amounts)** | occasional **HTTP 200** + URI | Mint only — not done |
| **Confirm** | force-select 200 then stuck **`selected`**, confirm **HTTP 500** → mon **`SKIP_CONFIRM`** (bounds unchanged) | **No** — bank never wires; **not** out-of-money (§3 / 5114 class) |
| **Bank settle** | rare full path: confirm 204 + `transfer_done` | Bank side “OK_BANK” |
| **Wallet coins** | almost always **`available=0`** within wait, large **`pendingIncoming`**, board **`done=0`** | **Not in time** for pay → **SKIP_BALANCE** / skip max-pay |
**Pay-wait board (same cumulative wallet), example:**
```text
summary: tsv_ok=2 · wallet_wd=14 (pendingIncoming=14 done=0)
timer 0s left — available still GOA:0
pendingIncoming=GOA:335776
```
Reading: only **2** TSV rows are full OK/OK_BANK, but the wallet holds **14** withdrawal txs still **pendingIncoming** and **none done** after the wait. Max-search also leaves many **mint-OK / confirm-failed** withdraws that **inflate pendingIncoming** and may never complete. After **90 s** + short `run-pending`: **still no spendable balance** → **we do not manage to withdraw in time** for pay (whether coins would appear later is unknown to the job).
#### What *did* work (sparse) — for contrast
| When / run | What worked | Amounts |
|------------|-------------|---------|
| Landing **2026-07-19 14:19–14:29 CEST** | Explorer singles **recorded** on bank intro stats | **~1.5–2 Giga-GOA**, **~145–202 Mega-GOA**, **~1.8–3.4 Mega-GOA** |
| Max-search run A (same day) | Full bank path (mint→confirm→`transfer_done`) for **two** amounts only; mon best | **`GOA:424114`**, **`GOA:822256`** (~822 Kilo best) |
| Max-search run B (later) | Even fewer full successes; many mint OK + **SKIP_CONFIRM**; mon best collapsed | e.g. full success around **`GOA:12276`** / **`GOA:17257`** (~**17 Kilo** best) |
| Pay after either | Spendable **available** within **90 s** | **`GOA:0`** (not in time) |
So: **landing/history prove large withdraws were possible earlier the same day**; **automation does not manage a usable withdraw in time** (best amount drops run-to-run; coins not available inside mon’s wait).
#### Mon behaviour (workaround only)
- 5110 → soft **`CEILING_REJECT`** / max-search **`hi:=`**
- Full OK only raises **`lo`**; **SKIP_CONFIRM** does not
- Pay waits up to **`LADDER_PAY_WAIT_AVAILABLE_S`** (default 90) with withdraw board; then **SKIP_BALANCE** if still 0
- **Does not fix** bank SQL mapping, stuck confirm, or wirewatch→coins lag
#### Wanted (bank / exchange ops)
1. Map mint **P0001** to structured errors (not **5110**).
2. Confirm after force-select must not stick **selected** + HTTP 500.
3. After `transfer_done`, coins must become wallet **available** (or document delay + API to wait).
4. File bugs with: max-search “what worked” table + landing `recent_withdraws`/`balance_explorer` + load note (healthy) + pay board `tsv_ok` vs `wallet_wd`/`pendingIncoming`/`done`.
Sources: `https://bank.hacktivism.ch/intro/stats.json`, exchange/merchant intro stats, mon max-ladder logs 2026-07-19.
---
## 4. Exchange not auto-known for bank deep links / withdraw URIs
| | |
|--|--|
| **Seen** | Android endless spinner; GOA `bank.hacktivism.ch` withdraw; fixed in app branch `fix/bank-withdraw-auto-exchange` |
| **Area** | **core** / **cli** |
| **Problem** | Withdraw URI names an exchange the wallet has never added → stuck loading / empty exchange list. CLI requires explicit `exchanges add` + `update` + `accept-tos` every fresh DB. |
| **Workaround** | e2e always adds exchange before bank withdraw ladder; Android needs ensureExchange-style fix. |
| **Wanted** | CLI/core: on `accept-uri` withdraw, **auto-add** exchange from URI/bank details (`allowCompletion`), then surface ToS if needed in one command. |
---
## 5. Default ports in URIs (`:443` / `:80`) break round-trips
| | |
|--|--|
| **Seen** | ladder/e2e strip ports; QR checks; mint URIs with `:443` |
| **Area** | **cli** / **bank** / **doc** |
| **Problem** | Some bank/wallet outputs include `taler://…host:443/…`. QR validators and some clients reject or double-normalize inconsistently. |
| **Workaround** | mon ≥1.23.4: `normalize_taler_uri` (lib.sh) on ladder/e2e/pay; bank landing `demo-withdraw-api.py` normalizes mint JSON; **urls** phase fails GOA/local if public landings still emit `:443`/`:80` in `taler://` (`www.taler-uri-ports-*`, unit `tests/test_normalize_taler_uri.sh`). |
| **Wanted** | libeufin never emits default ports (source fix). |
---
## 6. No first-class “demo withdraw” / explorer mint in wallet-cli
| | |
|--|--|
| **Seen** | e2e uses bank admin + bank withdraw create; ladder/gui-chain reimplement explorer mint in Python |
| **Area** | **cli** / **ops** |
| **Problem** | Automation always hand-rolls Basic auth → token → `POST …/withdrawals` → `taler_withdraw_uri`. Easy to get wrong (auth header, amount currency). |
| **Workaround** | Shared Python in `goa-chain` / ladder / e2e; secrets files. |
| **Wanted** | Optional helper: `wallet-cli testing mint-withdraw --bank=URL --user=explorer --password-file=… --amount=GOA:10` (or bank-side only tool in libeufin) emitting a clean URI. |
---
## 7. Long-lived wallet process required for pay/withdraw progress
| | |
|--|--|
| **Seen** | e2e `advanced serve` / shepherd socket |
| **Area** | **cli** |
| **Problem** | One-shot CLI invocations do not keep background work alive; without serve/shepherd, withdraw/pay may stall mid-flight. |
| **Workaround** | Start serve once per e2e run; route `wcli` through the socket. |
| **Wanted** | Documented, supported “session mode”: start/stop serve; or each mutating command optionally `--background-until=…` with timeout. |
---
## 8. ToS accept is a separate step that fails silently or blocks
| | |
|--|--|
| **Seen** | e2e `exchanges accept-tos`; ladder after add; Android GUI ToS screens |
| **Area** | **cli** / **core** |
| **Problem** | Fresh exchange → operations need ToS; CLI must call `accept-tos` explicitly. Failures are easy to miss in multi-step scripts. |
| **Workaround** | Always `accept-tos` after `update` in e2e/ladder. |
| **Wanted** | `accept-uri --accept-tos` or auto-prompt with non-interactive `--yes` that covers exchange ToS for that URI’s exchange. |
---
## 9. Pay template / public order path is merchant-shaped, not CLI-shaped
| | |
|--|--|
| **Seen** | e2e shop templates, stage farmer shops, ladder private orders |
| **Area** | **cli** / **doc** |
| **Problem** | Wallet-cli pays via `handle-uri` on `taler://pay/…`; creating the order is always custom curl (template POST or private order + token). No unified “pay this amount to instance” for public templates. |
| **Workaround** | Monitoring builds URI externally then `handle-uri --yes`. |
| **Wanted** | Documented recipe only, or `wallet-cli testing pay-template --base=… --instance=… --id=…` for demos. |
---
## 10. Amount / currency parsing edge cases
| | |
|--|--|
| **Seen** | ladder min denom 0.01 vs GOA 1e-6; zero withdraw rejected (HTTP 409 / amount too low); max wire ceilings |
| **Area** | **cli** / **bank** |
| **Problem** | `CURRENCY:0` and sub-min amounts fail differently per stack; CLI error strings are not machine-stable. |
| **Workaround** | Soft-skip zero rung; clamp ladder min to exchange denoms / bank max_wire. |
| **Wanted** | Structured errors (`AMOUNT_too_small`, `currency_unknown`) in JSON mode; `wallet-cli amount validate --exchange=`. |
---
## 11. Finding the right `taler-wallet-cli` binary
| | |
|--|--|
| **Seen** | `find_wallet_cli` in `lib.sh`; hardcoded laptop paths; wrapper vs `.mjs` |
| **Area** | **ops** / **cli** |
| **Problem** | Debian package is a shell wrapper; some tools need `node …/taler-wallet-cli.mjs`. Hardcoded `/Users/…` paths break other hosts. |
| **Workaround** | `find_wallet_cli` search list; `WALLET_CLI=` override. |
| **Wanted** | Single install story: `wallet-cli --version` JSON with path + libversion; no need to pass `.mjs` to node by hand. |
---
## 12. Secrets and multi-stack confusion
| | |
|--|--|
| **Seen** | explorer vs admin password; GOA secrets used on stage; ladder EXP_PW |
| **Area** | **ops** |
| **Problem** | CLI does not know “which stack”; wrong password → 401 mid-ladder. Not a wallet-cli bug, but every CLI automation hits it. |
| **Workaround** | `SECRETS_ROOT`, stage SSH, `STACK=` profiles. |
| **Wanted** | Optional `~/.config/taler/stacks.d/goa.env` convention documented next to wallet-cli; still no secrets in repo. |
---
## 13. No stable machine-readable “step result” for scripts
| | |
|--|--|
| **Seen** | e2e greps accept output; ladder scrapes JSON from mixed stdout |
| **Area** | **cli** |
| **Problem** | Human logs + occasional JSON blobs; hard to parse reliably. |
| **Workaround** | Python scrapers, temp files, `tee`. |
| **Wanted** | Global `--json` / `--ndjson` for all commands; one object per completed operation with `ok`, `op`, `ids`, `amounts`. |
---
## 14. Pay settlement wait is symmetric to withdraw pain
| | |
|--|--|
| **Seen** | e2e pay settle loops; ladder pay settle rounds |
| **Area** | **cli** / **core** |
| **Problem** | After `handle-uri` pay, success is “order paid” / balance drop / merchant order status — not one CLI wait. |
| **Workaround** | Short poll loops; never run-until-done. |
| **Wanted** | Same as §1–2: bounded wait on transaction id / order id. |
---
## 15. `--wallet-db=*.json` is rejected (“memory backend not supported”)
| | |
|--|--|
| **Seen** | **2026-07-19 magikoopa** wallet-cli **1.6.8** / **1.6.11** bundle: any command with `--wallet-db=/tmp/foo.json` fails immediately with `Error: memory backend not supported` |
| **Area** | **cli** / **ops** |
| **Problem** | Current node host treats storage paths ending in **`.json` as unsupported memory backend** and throws before opening sqlite. Old notes / local tests that used `WALLET_DB=….json` break. Accept/transactions then never emit `reserve_pub` → ladder §3 empty-scrape WARNs. |
| **Workaround** | Use a **non-`.json` path**, e.g. `--wallet-db=$SCRATCH/wallet.sqlite3` (ladder already prefers `wallet.sqlite3`). Never pass `….json` as wallet-db. |
| **Wanted** | Clearer error: `wallet-db path must not end in .json; use a directory or .sqlite3 path`. Document legal path forms in `--help`. |
---
## 16. `taler-helper-sqlite3` requires Python ≥ 3.11 on PATH
| | |
|--|--|
| **Seen** | magikoopa: helper is a **Python** script (`#!/usr/bin/env python3`); with Apple `/usr/bin/python3` **3.9.6** → `FATAL: python version >=3.11 required but running on 3.9.6` in accept/tx logs; wallet may still exit 0 on some paths while stdout is FATAL-only |
| **Area** | **ops** / **cli** |
| **Problem** | Wallet spawns `taler-helper-sqlite3` from **PATH**. Wrong `python3` ⇒ sqlite backend fails ⇒ no real wallet state ⇒ no `reserve_pub` for force-select. |
| **Workaround** | Put **Python ≥3.11** (or 3.12+) first on PATH for mon jobs; ensure `command -v taler-helper-sqlite3` works; smoke: `taler-helper-sqlite3` starts without FATAL. Prefer Homebrew / user `python3.11+` over macOS system 3.9 for agent PATH. |
| **Wanted** | Helper shebang or wrapper that finds a suitable python; wallet preflight error if helper HELLO fails (not silent empty logs). |
---
## 17. Portable wall-clock for scripts (no GNU `timeout` assumed)
| | |
|--|--|
| **Seen** | macOS agent shells: `timeout: command not found`; e2e already tries `timeout` then `gtimeout` then `perl -e alarm` |
| **Area** | **ops** / **doc** |
| **Problem** | GNU coreutils `timeout` is not on stock macOS. Hard-wiring it breaks portable repro and CI on Darwin. |
| **Workaround** | Prefer **python3** `subprocess.Popen(…).wait(timeout=…)` for outer budgets; for wallet-cli inner calls keep e2e order: `timeout` → `gtimeout` → perl alarm. Document both. |
| **Wanted** | One suite helper `run_with_budget SECS cmd…` in `lib.sh` used by ladder/e2e/smoke. |
---
## 18. How to withdraw (operator checklist, GOA laptop)
Portable path that works when §§15–16 are fixed:
1. **PATH:** `taler-helper-sqlite3` + **python3 ≥ 3.11** ahead of system 3.9.
2. **Wallet DB:** `WDB=$TMP/wallet.sqlite3` — **not** `*.json`.
3. **Exchange:** `exchanges add` → `update -f` → `accept-tos` for `https://exchange.hacktivism.ch/`.
4. **Bank mint (explorer):** token with explorer password from `SECRETS_ROOT` / `EXP_PW`; `POST …/accounts/explorer/withdrawals` with e.g. `{"amount":"GOA:1"}` (avoid pure **0** and absolute **max** pins for happy-path tests).
5. **Strip** `:443` from `taler_withdraw_uri`.
6. **`withdraw accept-uri --exchange URL URI`**.
7. **Bank:** poll `…/taler-integration/withdrawal-operation/{WID}` → when `selected`, confirm; if stuck `pending`, force-select with scraped `reserve_pub` (§3).
8. **Settle:** poll wallet balance / bank `transfer_done` — **no** `run-until-done` (§1).
If step 6’s stdout is only `FATAL: python version…` or `memory backend not supported`, **stop** and fix §15–16 before debugging force-select.
---
## Priority shortlist for `taler-wallet-cli` / core
If only a few changes land, these unlock the most automation:
1. **Bounded `run-until-done` / wait-for-state** (§1, §2, §14)
2. **Structured JSON on every command** (§13)
3. **Auto-add exchange + ToS on accept-uri** (§4, §8)
4. **Canonical URIs without :443** (§5)
5. **Emit reserve_pub for last withdraw** (§3)
6. **Clear wallet-db path rules + helper preflight** (§15, §16)
---
## How we work around today (monitoring)
| Issue | Monitoring behaviour |
|-------|----------------------|
| run-until-done | Disabled; balance + bank poll |
| serve | Optional long-lived socket in e2e |
| exchange | Explicit add/update/accept-tos |
| ladder explorer | `read_secret` / EXP_PW_FILE |
| stage maxima | bank `max_wire` + keys min denom |
| force-select | multi-rpub try; soft-skip confirm; empty scrape → WARN not balance fail |
| higher amounts / withdraw fail | **§3b: we do not manage to withdraw in time** on GOA (healthy load); mint 5110/P0001 + SKIP_CONFIRM + pendingIncoming; pay still available=0 after 90s; landing had Giga/Mega earlier same day |
| pay vs balance | never order more than wallet **available** (soft `SKIP_BALANCE`); pendingIncoming does not count as spendable |
| pay wait UX | board: OK/OK_BANK start order + live pendingIncoming/`done`; often `tsv_ok≪wallet_wd` and **done=0** after 90s → skip pay |
| wallet-db path | Prefer `*.sqlite3` (never `*.json` as db path) |
| helper / python | mon PATH must include helper + python≥3.11 |
---
## How to add an issue
```markdown
### N. short title
| | |
|--|--|
| **Seen** | where / which stack |
| **Area** | cli / core / bank / ops / doc |
| **Problem** | … |
| **Workaround** | … |
| **Wanted** | concrete CLI/core behaviour |
```
Reference in a new issue View git blame Copy permalink