diff --git a/CLI-AUTOMATION-NOTES.md b/CLI-AUTOMATION-NOTES.md index 69bcbc3..7d1cc25 100644 --- a/CLI-AUTOMATION-NOTES.md +++ b/CLI-AUTOMATION-NOTES.md @@ -67,121 +67,62 @@ Legend: | **Workaround** | Use `LADDER_STEPS≥5` (or higher) so random mids exist; treat zero / absolute-max / max-1-5110 as probes (soft `CEILING_REJECT` in mon ≥1.20.2). | | **Wanted** | Ladder plan docs: pin vs mid semantics; CLI structured error for amount-too-large / zero. | -### 3b. Mint fails with bank SQL `5110` / `P0001` (server-side; recurring) +### 3b. We do **not** manage to withdraw (GOA) — bank path broken, not load | | | |--|--| -| **Seen** | **Several times** on GOA (`bank.hacktivism.ch`), classic ladder high/random mids + ceiling pins **and** max-search bisect — not a one-off. Same class of error across 2026-07 runs (magikoopa / mon ladder). | -| **Area** | **bank** (libeufin / taler-corebank + **Postgres**) / **ops** — **not** mon SQL, **not** wallet-cli at mint time | -| **Call site** | mon `POST ${BANK}/accounts/explorer/withdrawals` with `{"amount":"GOA:…"}` only. Wallet/exchange not involved until mint returns 200 + URI. | -| **Response** | HTTP **500**, body e.g. `{ code: 5110, hint: "Unexpected sql error with state P0001" }` | -| **Codes** | **`5110` = `BANK_UNMANAGED_EXCEPTION`** (“no known exception types captured the exception”). **`P0001`** = PostgreSQL SQLSTATE for application `RAISE EXCEPTION` (function/trigger), not a mon bug and not max_wire config alone. | -| **Config check** | Public `/config` still allows huge wires (`max_wire_transfer_amount` ≈ `GOA:4503599627370496.99999999`, `default_debit_threshold` same order). Failing mints are often **far below** that (e.g. ~Kilo-GOA). | -| **API mismatch** | Corebank docs: insufficient funds should be HTTP **409**. Getting **500/5110** means the bank is **leaking** an unmapped SQL error instead of a clean client error. | -| **Problem** | Live mint “ceiling” is **opaque and unstable**. Do **not** treat 5110 as wallet/`reserve_pub` failure when mint never succeeded. Do **not** assume “only absolute pin:max fails” — mid and max-search probes hit the same error. | -| **Workaround (mon ≥1.21)** | Soft `CEILING_REJECT` on 5110/P0001 (classic mids + max pins); **max-ladder** treats mint reject as too-high upper bound and bisects. Keep mid-range rungs for real withdraw/confirm tests. | -| **Wanted (bank)** | Map the SQL condition to a structured error (insufficient funds / amount too large / debit limit) — **not** `BANK_UNMANAGED_EXCEPTION`. Log full `RAISE` text server-side. | +| **Bottom line** | On live **hacktivism GOA**, automation **does not reliably complete a withdraw to spendable coins**. Max-search may report a “best” amount; pay then sees **`available=GOA:0`**. This is **recurring (several runs, 2026-07-19)**. | +| **Area** | **bank** + wire/wallet settle — **not** mon overload, **not** “laptop too slow” | +| **Not the cause: ecosystem load** | Outside-in check **2026-07-19 ~14:56 CEST**: bank/exchange/merchant all **HTTP 200**, config ~**50–150 ms**, exchange `/keys` ~**0.5 s**. Container **loadavg ~1.5–1.7**, RSS bank **~737 MiB** / exchange **~670 MiB**, bank probes **47 ms**, **`wirewatch_running: true`**. Stack is **healthy under moderate load** — failures are **application/SQL/protocol**, not “host on fire.” | +| **Call site (mint)** | `POST ${BANK}/accounts/explorer/withdrawals` → often HTTP **500** `{ code: 5110, hint: "Unexpected sql error with state P0001" }` (**`BANK_UNMANAGED_EXCEPTION`**; Postgres **`P0001`** = `RAISE EXCEPTION`). Should often be clean **409** (insufficient funds); instead opaque **500**. | +| **Config** | `/config` still allows ~**2⁵²** wire max; live mint fails **far below** that. Explorer **balance_explorer** on landing still **~4.5 Peta-GOA**. | -#### Full max-search cascade (primary repro log, 2026-07-19, magikoopa / GOA) +#### Analysis — where withdraw dies -Mon phase **max-ladder** withdraw hunt. Every reject below is the **same** bank body: +| Stage | What we see | Manage to withdraw? | +|-------|-------------|---------------------| +| **Mint (high amounts)** | 5110/P0001 while hi falls Tera→Giga→Mega→Kilo with **lo=0** | **No** — withdrawal never created | +| **Mint (some mid amounts)** | occasional **HTTP 200** + URI | Mint only — not done | +| **Confirm** | force-select 200 then stuck **`selected`**, confirm **HTTP 500** → mon **`SKIP_CONFIRM`** (bounds unchanged) | **No** — bank never wires; **not** out-of-money (§3 / 5114 class) | +| **Bank settle** | rare full path: confirm 204 + `transfer_done` | Bank side “OK_BANK” | +| **Wallet coins** | almost always **`available=0`**, large **`pendingIncoming`**, board **`done=0`** | **No spendable coins** → pay **SKIP_BALANCE** / skip max-pay | + +**Pay-wait board (same cumulative wallet), example:** ```text -mint HTTP 500 { code :5110, hint : Unexpected sql error with state P0001 } +summary: tsv_ok=2 · wallet_wd=14 (pendingIncoming=14 done=0) +timer 0s left — available still GOA:0 +pendingIncoming=GOA:335776 ``` -Mon treats each as soft too-high (`CEILING_REJECT` / `hi:=amount`). **`lo` stays 0** until the first successful mint. +Reading: only **2** TSV rows are full OK/OK_BANK, but the wallet holds **14** withdrawal txs still **pendingIncoming** and **none done**. Max-search leaves many **mint-OK / confirm-failed** (and lagging) withdraws in the DB; they **inflate pendingIncoming** without ever becoming **available**. After **90 s** wait + short `run-pending` nudge: **still no spendable balance** → **we do not manage to withdraw** in the sense that matters (pay). -**Phase A — pure descent (no success yet): hi falls Tera → Giga → Mega while lo=0** +#### What *did* work (sparse) — for contrast -| Probe | Amount (alt) | Raw `GOA:…` | Result | -|------:|--------------|-------------|--------| -| (earlier) | hi already **467.409 Tera-GOA** | `467409181353380` | 5110 (prior probe) | -| 4 | **154.93 Tera-GOA** | `154929915073879` | 5110 → hi:= | -| 5 | **16.20 Tera-GOA** | `16198385073981` | 5110 → hi:= | -| 6 | **1.417 Tera-GOA** | `1416945306636` | 5110 → hi:= | -| 7 | **282.49 Giga-GOA** | `282485970397` | 5110 → hi:= | -| 8 | **132.18 Giga-GOA** | `132183255296` | 5110 → hi:= | -| 9 | **7.417 Giga-GOA** | `7416937654` | 5110 → hi:= | -| 10 | **711.38 Mega-GOA** | `711383774` | 5110 → hi:= | -| 11 | **395.88 Mega-GOA** | `395879334` | 5110 → hi:= | -| 12 | **119.26 Mega-GOA** | `119263564` | 5110 → hi:= | -| 13 | **5.992 Mega-GOA** | `5992111` | 5110 → hi:= | +| When / run | What worked | Amounts | +|------------|-------------|---------| +| Landing **2026-07-19 14:19–14:29 CEST** | Explorer singles **recorded** on bank intro stats | **~1.5–2 Giga-GOA**, **~145–202 Mega-GOA**, **~1.8–3.4 Mega-GOA** | +| Max-search run A (same day) | Full bank path (mint→confirm→`transfer_done`) for **two** amounts only; mon best | **`GOA:424114`**, **`GOA:822256`** (~822 Kilo best) | +| Max-search run B (later) | Even fewer full successes; many mint OK + **SKIP_CONFIRM**; mon best collapsed | e.g. full success around **`GOA:12276`** / **`GOA:17257`** (~**17 Kilo** best) | +| Pay after either | Spendable **available** | **`GOA:0`** (fail) | -**Phase B — first success raises lo; still 5110 just above** +So: **landing/history prove large withdraws were possible earlier the same day**; **automation now does not manage a usable withdraw** (best amount drops run-to-run; coins never spendable). -| Probe | Amount (alt) | Raw | Result | -|------:|--------------|-----|--------| -| **14** | **424.114 Kilo-GOA** | **`424114`** | **OK mint** → accept → force-select `PNK5YRKT…` → confirm 204 → settle `transfer_done` (wallet **avail still 0**) · **lo:=424114** | -| 15 | **1.594 Mega-GOA** | `1594158` | 5110 → hi:= | -| **16** | **822.256 Kilo-GOA** | **`822256`** | **OK mint** → force-select `SVKV4E9G…` → confirm → settle · **lo:=822256** (best so far) | +#### Mon behaviour (workaround only) -**Phase C — fine bisect above best: every step 5110 until convergence** +- 5110 → soft **`CEILING_REJECT`** / max-search **`hi:=`** +- Full OK only raises **`lo`**; **SKIP_CONFIRM** does not +- Pay waits up to **`LADDER_PAY_WAIT_AVAILABLE_S`** (default 90) with withdraw board; then **SKIP_BALANCE** if still 0 +- **Does not fix** bank SQL mapping, stuck confirm, or wirewatch→coins lag -| Probe | Amount (alt) | Raw | Result | -|------:|--------------|-----|--------| -| 17 | 1.145 Mega-GOA | `1144904` | 5110 | -| 18 | 970.26 Kilo-GOA | `970260` | 5110 | -| 19 | 893.198 Kilo-GOA | `893198` | 5110 | -| 20 | 856.993 Kilo-GOA | `856993` | 5110 | -| 21 | 839.445 Kilo-GOA | `839445` | 5110 | -| 22 | 830.806 Kilo-GOA | `830806` | 5110 | -| 23 | 826.52 Kilo-GOA | `826520` | 5110 | -| 24 | 824.385 Kilo-GOA | `824385` | 5110 | -| 25 | 823.32 Kilo-GOA | `823320` | 5110 | -| 26 | 822.788 Kilo-GOA | `822788` | 5110 → hi:= | +#### Wanted (bank / exchange ops) -**Converged (mon):** +1. Map mint **P0001** to structured errors (not **5110**). +2. Confirm after force-select must not stick **selected** + HTTP 500. +3. After `transfer_done`, coins must become wallet **available** (or document delay + API to wait). +4. File bugs with: max-search “what worked” table + landing `recent_withdraws`/`balance_explorer` + load note (healthy) + pay board `tsv_ok` vs `wallet_wd`/`pendingIncoming`/`done`. -```text -max-search best withdrawable 822.256 Kilo-GOA (GOA:822256) - · lo=822.256 Kilo-GOA · hi=822.788 Kilo-GOA · probes=26 -``` - -**Reading this log** - -1. **Same error from ~467 Tera-GOA down to ~823 Kilo-GOA** — not a single “absurd pin:max” case; the bank returns **5110/P0001** for a **wide continuous range** of amounts. -2. **Successes are real but sparse:** only **`GOA:424114`** and **`GOA:822256`** minted in this hunt (both full bank confirm + `transfer_done`). Everything strictly larger than **822256** in the fine band failed at **mint**. -3. **lo=0 for probes 4–13** means max-search had **no** successful withdraw yet while already rejecting Tera/Giga/Mega — so 5110 is **not** “only after pool drained by this run.” -4. Settle notes **wallet avail=GOA:0** after OK bank path — coins lag / `pendingIncoming` (pay-wait board, § pay wait UX); separate from mint 5110. -5. Landing `recent_withdraws` at **14:42 CEST** lists the same two OK amounts (`424114`, `822256`) with reserves matching force-select prefixes above. - -#### Landing page: much larger withdraws **same day, earlier** (still on explorer) - -Public bank intro: `https://bank.hacktivism.ch/intro/stats.json` (HTML landing uses this). Snapshot **2026-07-19 ~14:47 CEST**. - -**Pool still looks full in the UI:** - -| Field | Snapshot | -|-------|----------| -| `balance_explorer` | **4.5036 Peta-GOA** (`GOA:4503599627250215.32499924`) | -| `withdraws.total_amount` | **4.5036 Peta-GOA** (292 ops) | -| `withdraws.last_24h` | **3.2366 Peta-GOA** (45 ops) | - -**Dated explorer withdraws that succeeded *before* this max-search tight band** (`recent_withdraws`, account=`explorer`): - -| When (CEST) | Amount (alt) | Raw | Notes | -|-------------|--------------|-----|--------| -| **2026-07-19 14:19** | **1.9761 Giga-GOA** | `GOA:1976113563` | ~2h before max-search best | -| **2026-07-19 14:19** | **1.4495 Giga-GOA** | `GOA:1449489732` | same minute | -| **2026-07-19 14:26** | **201.636 Mega-GOA** | `GOA:201635991` | | -| **2026-07-19 14:26** | **144.53 Mega-GOA** | `GOA:144530029` | | -| **2026-07-19 14:29** | **3.4099 Mega-GOA** | `GOA:3409891` | | -| **2026-07-19 14:29** | **1.8009 Mega-GOA** | `GOA:1800875` | | -| **2026-07-19 14:42** | 822.256 Kilo-GOA | `GOA:822256` | = max-search **best** (probe 16) | -| **2026-07-19 14:42** | 424.114 Kilo-GOA | `GOA:424114` | = max-search first OK (probe 14) | - -**Contrast:** earlier the **same day**, explorer minted **~1.5–2 Giga-GOA** singles; later the same API returns **5110** for **~823 Kilo-GOA** and for all of Tera→Mega probes in the cascade. Landing **balance_explorer** still shows **~Peta-GOA**. So: - -- not “config max_wire forbids Kilo-GOA”; -- not “landing says explorer is empty”; -- **live mintability collapsed / became amount-sensitive** relative to history — bank SQL path is wrong or mis-mapped. - -**Ops checklist when filing bank bug:** attach (1) this max-search probe table, (2) landing `recent_withdraws` + `balance_explorer` at the same timestamps, (3) bank journal / Postgres `P0001` RAISE text for one failing mint. - -**Also seen (same error class):** classic ladder e.g. `GOA:747827072844319` → 500/5110/P0001; absolute max / max-1 ceiling pins → soft `CEILING_REJECT`. - -**Exchange intro (not mint path):** `https://exchange.hacktivism.ch/intro/stats.json` — wire-in can still be Peta-scale; use **bank** landing for explorer mint history. +Sources: `https://bank.hacktivism.ch/intro/stats.json`, exchange/merchant intro stats, mon max-ladder logs 2026-07-19. --- @@ -393,9 +334,9 @@ If only a few changes land, these unlock the most automation: | ladder explorer | `read_secret` / EXP_PW_FILE | | stage maxima | bank `max_wire` + keys min denom | | force-select | multi-rpub try; soft-skip confirm; empty scrape → WARN not balance fail | -| higher amounts | mint **5110/P0001** recurring server-side (§3b); landing shows much larger **dated** explorer withdraws (Giga/Mega same day); soft `CEILING_REJECT` + max-ladder hi; alt units in report | +| higher amounts / withdraw fail | **§3b: we do not manage to withdraw** (spendable coins) on GOA despite healthy load; mint 5110/P0001 + SKIP_CONFIRM + pendingIncoming pile-up; landing had Giga/Mega earlier same day | | pay vs balance | never order more than wallet **available** (soft `SKIP_BALANCE`); pendingIncoming does not count as spendable | -| pay wait UX | after withdraw, board lists OK/OK_BANK in **start order** with live wallet `pendingIncoming`/`done` + timer (`LADDER_PAY_WAIT_AVAILABLE_S`, default 90s); same wallet DB as withdraw | +| pay wait UX | board: OK/OK_BANK start order + live pendingIncoming/`done`; often `tsv_ok≪wallet_wd` and **done=0** after 90s → skip pay | | wallet-db path | Prefer `*.sqlite3` (never `*.json` as db path) | | helper / python | mon PATH must include helper + python≥3.11 |