diff --git a/CLI-AUTOMATION-NOTES.md b/CLI-AUTOMATION-NOTES.md index b4f7431..b6ac18f 100644 --- a/CLI-AUTOMATION-NOTES.md +++ b/CLI-AUTOMATION-NOTES.md @@ -9,7 +9,7 @@ Companions: - Android GUI → [`android-test/GUI-AUTOMATION-NOTES.md`](./android-test/GUI-AUTOMATION-NOTES.md) - Git / suite tree → [`VERSIONS.md`](./VERSIONS.md) -Scripts: `check_e2e.sh`, `check_amount_ladder.sh`, `taler-monitoring.sh`. +Scripts: `check_e2e.sh`, `check_goa_ladder.sh`, `taler-monitoring.sh`. Legend: @@ -47,82 +47,15 @@ Legend: --- -## 3. `reserve_pub` / force-select is fragile (5114 + empty scrape) +## 3. `reserve_pub` / force-select is fragile (5114) | | | |--|--| -| **Seen** | ladder force-select, e2e confirm path, Android spinner issues; **repro 2026-07-19 magikoopa** GOA ladder: `force-select skipped · no reserve_pub for this withdraw (WID=…); wallet may not have selected yet` | -| **Area** | **cli** / **core** / **bank** / **ops** | -| **Problem** | (a) Cumulative wallets re-emit old reserves; binding the wrong `reserve_pub` yields **HTTP 409 / code 5114** (“already bound”). (b) Automation scrapes accept/tx dumps for candidates — when the dump has **no** pub at all, ladder WARNs and skips force-select (soft), then often **SKIP_CONFIRM**. (c) Empty scrape is **not** the same as “bank out of money”; do not treat as balance failure. (d) **Mon bug (fixed v1.20.2):** extract used `json.loads(file_from_first_brace)` on wallet dumps that append log lines after JSON → **zero** objects parsed; regex only scanned accept (which has no `reservePub`) while the pub lives under `transactions[].withdrawalDetails.reservePub`. | -| **Workaround** | **Mon ≥1.20.2:** `JSONDecoder.raw_decode` + regex on accept **and** transactions; score by WID in `bankConfirmationUrl` / amount. Still: score pubs by WID/amount; try several; treat 5114 as “stale reserve” not “out of money”. Refresh `transactions` before each force-select try. Soft-skip confirm if still pending after polls. Fix wallet DB / helper first if accept/tx only print FATAL (see §15–16). | -| **Wanted** | CLI returns **the** `reserve_pub` for the just-accepted withdraw in structured JSON; `force-select` / bank-side select API documented with idempotency. Optional: bank integration already exposes `selected_reserve_pub` when wallet selected — prefer that over scraping wallet logs. | - -### 3a. Ladder pin amounts hide mid-range behaviour - -| | | -|--|--| -| **Seen** | `LADDER_STEPS=3` plan collapses to pins only: `GOA:0`, `max-1`, `max` (no random mid rungs) | -| **Area** | **ops** / **bank** / **cli** | -| **Problem** | **Zero** withdraw often never produces a usable `reserve_pub` / coins (wallet 7006 or no denoms). **Absolute max** and sometimes **max-1** hit **mint HTTP 500 / code 5110** SQL `P0001` (ceiling probe). Reproducing “real” force-select needs a **mid** amount (e.g. `GOA:1` … exchange min denom scale), not only pins. | -| **Workaround** | Use `LADDER_STEPS≥5` (or higher) so random mids exist; treat zero / absolute-max / max-1-5110 as probes (soft `CEILING_REJECT` in mon ≥1.20.2). | -| **Wanted** | Ladder plan docs: pin vs mid semantics; CLI structured error for amount-too-large / zero. | - -### 3b. We do **not** manage to withdraw **in time** (GOA) — bank path unreliable, not load - -| | | -|--|--| -| **Bottom line** | On live **hacktivism GOA**, automation **does not get spendable coins in time** for the pay phase (default wait **90 s**). Max-search may report a “best” amount and bank may even show `transfer_done`; pay still sees **`available=GOA:0`** with large **`pendingIncoming`**. Coins might arrive **later** (wirewatch / wallet) — **at least not within mon’s budget**. Recurring (several runs, 2026-07-19). | -| **Area** | **bank** + wire/wallet settle — **not** mon overload, **not** “laptop too slow” | -| **Not the cause: ecosystem load** | Outside-in check **2026-07-19 ~14:56 CEST**: bank/exchange/merchant all **HTTP 200**, config ~**50–150 ms**, exchange `/keys` ~**0.5 s**. Container **loadavg ~1.5–1.7**, RSS bank **~737 MiB** / exchange **~670 MiB**, bank probes **47 ms**, **`wirewatch_running: true`**. Stack is **healthy under moderate load** — failures are **application/SQL/protocol**, not “host on fire.” | -| **Call site (mint)** | `POST ${BANK}/accounts/explorer/withdrawals` → often HTTP **500** `{ code: 5110, hint: "Unexpected sql error with state P0001" }` (**`BANK_UNMANAGED_EXCEPTION`**; Postgres **`P0001`** = `RAISE EXCEPTION`). Should often be clean **409** (insufficient funds); instead opaque **500**. | -| **Config** | `/config` still allows ~**2⁵²** wire max; live mint fails **far below** that. Explorer **balance_explorer** on landing still **~4.5 Peta-GOA**. | - -#### Analysis — where withdraw dies - -| Stage | What we see | Manage to withdraw? | -|-------|-------------|---------------------| -| **Mint (high amounts)** | 5110/P0001 while hi falls Tera→Giga→Mega→Kilo with **lo=0** | **No** — withdrawal never created | -| **Mint (some mid amounts)** | occasional **HTTP 200** + URI | Mint only — not done | -| **Confirm** | force-select 200 then stuck **`selected`**, confirm **HTTP 500** → mon **`SKIP_CONFIRM`** (bounds unchanged) | **No** — bank never wires; **not** out-of-money (§3 / 5114 class) | -| **Bank settle** | rare full path: confirm 204 + `transfer_done` | Bank side “OK_BANK” | -| **Wallet coins** | almost always **`available=0`** within wait, large **`pendingIncoming`**, board **`done=0`** | **Not in time** for pay → **SKIP_BALANCE** / skip max-pay | - -**Pay-wait board (same cumulative wallet), example:** - -```text -summary: tsv_ok=2 · wallet_wd=14 (pendingIncoming=14 done=0) -timer 0s left — available still GOA:0 -pendingIncoming=GOA:335776 -``` - -Reading: only **2** TSV rows are full OK/OK_BANK, but the wallet holds **14** withdrawal txs still **pendingIncoming** and **none done** after the wait. Max-search also leaves many **mint-OK / confirm-failed** withdraws that **inflate pendingIncoming** and may never complete. After **90 s** + short `run-pending`: **still no spendable balance** → **we do not manage to withdraw in time** for pay (whether coins would appear later is unknown to the job). - -#### What *did* work (sparse) — for contrast - -| When / run | What worked | Amounts | -|------------|-------------|---------| -| Landing **2026-07-19 14:19–14:29 CEST** | Explorer singles **recorded** on bank intro stats | **~1.5–2 Giga-GOA**, **~145–202 Mega-GOA**, **~1.8–3.4 Mega-GOA** | -| Max-search run A (same day) | Full bank path (mint→confirm→`transfer_done`) for **two** amounts only; mon best | **`GOA:424114`**, **`GOA:822256`** (~822 Kilo best) | -| Max-search run B (later) | Even fewer full successes; many mint OK + **SKIP_CONFIRM**; mon best collapsed | e.g. full success around **`GOA:12276`** / **`GOA:17257`** (~**17 Kilo** best) | -| Pay after either | Spendable **available** within **90 s** | **`GOA:0`** (not in time) | - -So: **landing/history prove large withdraws were possible earlier the same day**; **automation does not manage a usable withdraw in time** (best amount drops run-to-run; coins not available inside mon’s wait). - -#### Mon behaviour (workaround only) - -- 5110 → soft **`CEILING_REJECT`** / max-search **`hi:=`** -- Full OK only raises **`lo`**; **SKIP_CONFIRM** does not -- Pay waits up to **`LADDER_PAY_WAIT_AVAILABLE_S`** (default 90) with withdraw board; then **SKIP_BALANCE** if still 0 -- **Does not fix** bank SQL mapping, stuck confirm, or wirewatch→coins lag - -#### Wanted (bank / exchange ops) - -1. Map mint **P0001** to structured errors (not **5110**). -2. Confirm after force-select must not stick **selected** + HTTP 500. -3. After `transfer_done`, coins must become wallet **available** (or document delay + API to wait). -4. File bugs with: max-search “what worked” table + landing `recent_withdraws`/`balance_explorer` + load note (healthy) + pay board `tsv_ok` vs `wallet_wd`/`pendingIncoming`/`done`. - -Sources: `https://bank.hacktivism.ch/intro/stats.json`, exchange/merchant intro stats, mon max-ladder logs 2026-07-19. +| **Seen** | ladder force-select, e2e confirm path, Android spinner issues | +| **Area** | **cli** / **core** / **bank** | +| **Problem** | Cumulative wallets re-emit old reserves; binding the wrong `reserve_pub` yields **HTTP 409 / code 5114** (“already bound”). Automation scrapes accept/tx dumps for candidates. | +| **Workaround** | Score pubs by WID/amount; try several; treat 5114 as “stale reserve” not “out of money”. | +| **Wanted** | CLI returns **the** `reserve_pub` for the just-accepted withdraw in structured JSON; `force-select` / bank-side select API documented with idempotency. | --- @@ -258,59 +191,6 @@ Sources: `https://bank.hacktivism.ch/intro/stats.json`, exchange/merchant intro --- -## 15. `--wallet-db=*.json` is rejected (“memory backend not supported”) - -| | | -|--|--| -| **Seen** | **2026-07-19 magikoopa** wallet-cli **1.6.8** / **1.6.11** bundle: any command with `--wallet-db=/tmp/foo.json` fails immediately with `Error: memory backend not supported` | -| **Area** | **cli** / **ops** | -| **Problem** | Current node host treats storage paths ending in **`.json` as unsupported memory backend** and throws before opening sqlite. Old notes / local tests that used `WALLET_DB=….json` break. Accept/transactions then never emit `reserve_pub` → ladder §3 empty-scrape WARNs. | -| **Workaround** | Use a **non-`.json` path**, e.g. `--wallet-db=$SCRATCH/wallet.sqlite3` (ladder already prefers `wallet.sqlite3`). Never pass `….json` as wallet-db. | -| **Wanted** | Clearer error: `wallet-db path must not end in .json; use a directory or .sqlite3 path`. Document legal path forms in `--help`. | - ---- - -## 16. `taler-helper-sqlite3` requires Python ≥ 3.11 on PATH - -| | | -|--|--| -| **Seen** | magikoopa: helper is a **Python** script (`#!/usr/bin/env python3`); with Apple `/usr/bin/python3` **3.9.6** → `FATAL: python version >=3.11 required but running on 3.9.6` in accept/tx logs; wallet may still exit 0 on some paths while stdout is FATAL-only | -| **Area** | **ops** / **cli** | -| **Problem** | Wallet spawns `taler-helper-sqlite3` from **PATH**. Wrong `python3` ⇒ sqlite backend fails ⇒ no real wallet state ⇒ no `reserve_pub` for force-select. | -| **Workaround** | Put **Python ≥3.11** (or 3.12+) first on PATH for mon jobs; ensure `command -v taler-helper-sqlite3` works; smoke: `taler-helper-sqlite3` starts without FATAL. Prefer Homebrew / user `python3.11+` over macOS system 3.9 for agent PATH. | -| **Wanted** | Helper shebang or wrapper that finds a suitable python; wallet preflight error if helper HELLO fails (not silent empty logs). | - ---- - -## 17. Portable wall-clock for scripts (no GNU `timeout` assumed) - -| | | -|--|--| -| **Seen** | macOS agent shells: `timeout: command not found`; e2e already tries `timeout` then `gtimeout` then `perl -e alarm` | -| **Area** | **ops** / **doc** | -| **Problem** | GNU coreutils `timeout` is not on stock macOS. Hard-wiring it breaks portable repro and CI on Darwin. | -| **Workaround** | Prefer **python3** `subprocess.Popen(…).wait(timeout=…)` for outer budgets; for wallet-cli inner calls keep e2e order: `timeout` → `gtimeout` → perl alarm. Document both. | -| **Wanted** | One suite helper `run_with_budget SECS cmd…` in `lib.sh` used by ladder/e2e/smoke. | - ---- - -## 18. How to withdraw (operator checklist, GOA laptop) - -Portable path that works when §§15–16 are fixed: - -1. **PATH:** `taler-helper-sqlite3` + **python3 ≥ 3.11** ahead of system 3.9. -2. **Wallet DB:** `WDB=$TMP/wallet.sqlite3` — **not** `*.json`. -3. **Exchange:** `exchanges add` → `update -f` → `accept-tos` for `https://exchange.hacktivism.ch/`. -4. **Bank mint (explorer):** token with explorer password from `SECRETS_ROOT` / `EXP_PW`; `POST …/accounts/explorer/withdrawals` with e.g. `{"amount":"GOA:1"}` (avoid pure **0** and absolute **max** pins for happy-path tests). -5. **Strip** `:443` from `taler_withdraw_uri`. -6. **`withdraw accept-uri --exchange URL URI`**. -7. **Bank:** poll `…/taler-integration/withdrawal-operation/{WID}` → when `selected`, confirm; if stuck `pending`, force-select with scraped `reserve_pub` (§3). -8. **Settle:** poll wallet balance / bank `transfer_done` — **no** `run-until-done` (§1). - -If step 6’s stdout is only `FATAL: python version…` or `memory backend not supported`, **stop** and fix §15–16 before debugging force-select. - ---- - ## Priority shortlist for `taler-wallet-cli` / core If only a few changes land, these unlock the most automation: @@ -320,7 +200,6 @@ If only a few changes land, these unlock the most automation: 3. **Auto-add exchange + ToS on accept-uri** (§4, §8) 4. **Canonical URIs without :443** (§5) 5. **Emit reserve_pub for last withdraw** (§3) -6. **Clear wallet-db path rules + helper preflight** (§15, §16) --- @@ -333,12 +212,7 @@ If only a few changes land, these unlock the most automation: | exchange | Explicit add/update/accept-tos | | ladder explorer | `read_secret` / EXP_PW_FILE | | stage maxima | bank `max_wire` + keys min denom | -| force-select | multi-rpub try; soft-skip confirm; empty scrape → WARN not balance fail | -| higher amounts / withdraw fail | **§3b: we do not manage to withdraw in time** on GOA (healthy load); mint 5110/P0001 + SKIP_CONFIRM + pendingIncoming; pay still available=0 after 90s; landing had Giga/Mega earlier same day | -| pay vs balance | never order more than wallet **available** (soft `SKIP_BALANCE`); pendingIncoming does not count as spendable | -| pay wait UX | board: OK/OK_BANK start order + live pendingIncoming/`done`; often `tsv_ok≪wallet_wd` and **done=0** after 90s → skip pay | -| wallet-db path | Prefer `*.sqlite3` (never `*.json` as db path) | -| helper / python | mon PATH must include helper + python≥3.11 | +| force-select | multi-rpub try; soft-skip confirm | --- diff --git a/README.md b/README.md index 2c08f4e..998c65b 100644 --- a/README.md +++ b/README.md @@ -274,14 +274,12 @@ Other domains: never SSH. Optional **e2e** aborts cleanly on login/KYC. ```bash ./taler-monitoring.sh e2e ./taler-monitoring.sh -d taler.net urls e2e -# Amount ladder (any currency): classic 0 → … → max, then optional pay -./taler-monitoring.sh ladder # classic (current domain) +# Amount ladder (withdraw 0 → … → max, then pay): same script for GOA + stage +./taler-monitoring.sh ladder # GOA / current domain ./taler-monitoring.sh -d stage.lefrancpaysan.ch ladder # TESTPAYSAN maxima -# Max-search: highest withdrawable + highest payable (≤ available; GOA free template goa-free) -./taler-monitoring.sh max-ladder # Stage auto-reads bank max_wire_transfer_amount (e.g. 2000) and min denom (0.01). # Explorer password: francpaysan-stage-user …/bank-explorer-password.txt (or EXP_PW=). -# Override: LADDER_MODE=max LADDER_MAX_PROBES=16 LADDER_MAX_AMOUNT=100 LADDER_STEPS=7 LADDER_PAY=0 … +# Override: LADDER_MAX_AMOUNT=100 LADDER_STEPS=7 LADDER_PAY=0 … # customize e2e: E2E_WITHDRAW_VALUES="20 50 100" E2E_PAY_VALUES="0.05 1 5" ./taler-monitoring.sh e2e E2E_VARIABLE=0 WITHDRAW_AMT=GOA:50 PAY_AMT=GOA:1 ./taler-monitoring.sh e2e # single fixed @@ -306,7 +304,7 @@ E2E_VARIABLE=0 WITHDRAW_AMT=GOA:50 PAY_AMT=GOA:1 ./taler-monitoring.sh e2e # s | **versions** | `deb.taler.net` reachable (InRelease/Packages/pool `.deb`); containers can reach it + have apt source; installed Taler packages vs **trixie** | | **sanity** | bank · exchange · merchant sections (public + server) | | **e2e** | account → credit → withdraw → wallet → confirm → order → pay | -| **ladder** | amount ladder modules: classic 23-rung wd+pay, or max-search wd+pay (`max-ladder`) | +| **ladder** | GOA withdraw: fixed **0** + low bands + **random high ranges** + fixed **max** | ```bash # package suite (default trixie on deb.taler.net) @@ -396,14 +394,8 @@ LADDER_LOAD=0 ./taler-monitoring.sh ladder | `METRICS_LOAD` | `1` | `0` = skip all host/container load probes | | `LADDER_LOAD` | `1` | `0` = skip load in ladder only | | `LADDER_PAY` | `1` | `0` = skip payment ladder after withdraws | -| `LADDER_MODE` | `classic` | default **classic** 23-rung wd+pay; `max` = max-search wd+pay (`max-ladder`) | -| `LADDER_MAX_PROBES` | `32` | max-search probe budget; `0` = until converge / timeout | -| `LADDER_HIGH_RUNGS` | `6` | max-search: first N probes random-high, then bisect | -| `LADDER_MAX_TOL_REL` | `0.001` | max-search relative lo/hi gap to stop | -| `LADDER_FREE_TEMPLATE` | `goa-free` | public free-amount pay template (hacktivism) | -| `LADDER_USE_FREE_TEMPLATE` | `1` | create pays via template POST `{"amount":…}` | -| `LADDER_WITHDRAW_SCALE` | `1.5` | classic: withdraw mids / pay mids ratio (funding headroom) | -| `LADDER_STEPS` | `23` | classic: rungs for withdraw and pay (0 + mids + max−1 + max) | +| `LADDER_WITHDRAW_SCALE` | `1.5` | withdraw mids / pay mids ratio (funding headroom) | +| `LADDER_STEPS` | `23` | rungs for both withdraw and pay (0 + mids + max−1 + max) | | `METRICS_LOAD_SSH_TIMEOUT` | `90` | seconds for remote load python | ## Needs diff --git a/TESTS.md b/TESTS.md index 9320315..5c0f76a 100644 --- a/TESTS.md +++ b/TESTS.md @@ -25,7 +25,7 @@ Every check line has a **global** run number and a **grouped** id: | **sanity** | `check_sanity.sh` | `bank` `exchange` `merchant` | | **server** | `check_server.sh` | (flat `server-NN` or host groups) | | **e2e** | `check_e2e.sh` | `prereq` `load` `bank` `wallet` `bankwd` `settle` `pay` `shop` `paivana` `dig` `report` | -| **ladder** | `check_amount_ladder.sh` + `ladder/lib_pay.sh` | `plan` `load` `withdraw` `pay` `report` — classic / max-search; phases `ladder` · `max-ladder` · `goa-ladder` (alias) | +| **ladder** | `check_goa_ladder.sh` | `plan` `load` `withdraw` `pay` `report` | **Why groups:** flat `www-001`…`www-080` was hard to talk about. `www.bank-04` / `e2e.bankwd-02` pin the failure to a logical block. @@ -141,7 +141,7 @@ Standalone: `host-agent/run-aptdeploy.sh` (on koopa local; else ssh `$DEPLOY_SSH - OSV CVE query when package+version known → **ERROR** on hits - Never SSH / never podman on the *targets* | **sanity.bank-** / **.exchange-** / **.merchant-** | public + optional server-side per component | -| **ladder.plan-** / **.load-** / **.withdraw-** / **.pay-** / **.report-** | amount ladder classic or max-search (any currency; stage uses `max_wire`) | +| **ladder.plan-** / **.load-** / **.withdraw-** / **.pay-** / **.report-** | amount ladder (GOA ceiling or stage TESTPAYSAN `max_wire`) | | **server-** | SSH host ports / processes (flat unless grouped later) | | **auth401.*** | merchant 401 / case matrix (WebUI APIs): **signup** `POST /instances` MIX id, **login** Basic user low vs MIX/UPPER, full **case** matrix, **password** bad/empty, **pwchange** `POST private/auth` + re-login, **admin-create** `POST /management/instances` (if admin token), **bearer**, **durable mon401**, **webui** SPA `toLowerCase` probe (`check_auth401.sh`) | diff --git a/VERSION b/VERSION index a6c2798..3989355 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.23.0 +1.20.0 diff --git a/VERSIONS.md b/VERSIONS.md index 3e7f783..76bb76b 100644 --- a/VERSIONS.md +++ b/VERSIONS.md @@ -17,11 +17,6 @@ Git tags: `vMAJOR.FEATURE.FIX` (e.g. `v1.8.0`). File `VERSION` omits the `v` pre | Tag | Date (UTC) | Notes | |-----|------------|--------| -| **v1.23.0** | 2026-07-19 | **Feature:** GOA daily mon pages **`/monitoring-ladder/`** (classic) + **`/monitoring-max-ladder/`** (max-search) on bank/exchange/taler.hacktivism.ch. Host-agent timers once/day + install starts first round immediately. monpages catalog/path allow; Caddy bare redirs. | -| **v1.22.1** | 2026-07-19 | **Bugfix (UX):** pay wait after withdraw no longer looks hung — shows **withdraw board in start order** (OK/OK_BANK from TSV) with live wallet status (`pendingIncoming` → `done`) + timer + aggregate available/pendingIncoming on the **same** wallet DB. Short `run-pending` nudge only (no run-until-done). Default wait 90s (`LADDER_PAY_WAIT_AVAILABLE_S`). | -| **v1.22.0** | 2026-07-19 | **Feature:** modular amount ladder — `ladder/lib_pay.sh` (pay) + withdraw in `check_amount_ladder.sh`; `ladder/extract_rpubs.py`. **max-ladder** hunts highest **withdrawable** and highest **payable** (≤ wallet available). GOA pays use public free-amount template **`goa-free`** (`LADDER_FREE_TEMPLATE`). Classic still default 23-rung wd+pay. Phase alias `goa-ladder` → amount ladder (shim removed). | -| **v1.21.0** | 2026-07-19 | **Feature:** generic **amount ladder** (`check_amount_ladder.sh`; compat shim `check_goa_ladder.sh`). Default **classic** 23-rung withdraw (+ pay when enabled); **max-search** (`max-ladder` / `LADDER_MODE=max`) hunts highest mintable amount (probes default 32, `0`=until timeout). **Pay:** never overspend — soft `SKIP_BALANCE` if amount > wallet available. **Alt units:** report/logs show human names (e.g. `8.15 Tera-GOA (GOA:…)`) + raw. **Fixes bundled:** external extract (no stdin SyntaxError); force-select multi-rpub; soft `CEILING_REJECT` on mint 5110/P0001; crash-proof report. CLI-AUTOMATION-NOTES §3/3a/3b/§15–18. | -| **v1.20.1** | 2026-07-19 | **Docs:** CLI-AUTOMATION-NOTES — force-select empty scrape + ladder pins (§3/3a); wallet-db must not be `*.json` (§15); `taler-helper-sqlite3` needs Python ≥3.11 (§16); portable wall-clock (§17); GOA withdraw checklist (§18). No code change. *(Superseded notes folded into **v1.21.0** feature release.)* | | **v1.20.0** | 2026-07-19 | **Feature:** domains.conf **`locale`** (l10n) — FP **`fr-CH`**, others **`de-CH`** (aliases ch-FR/ch-DE); UI **`lang=de`** supported (sticky + console tags) but **never** a stock profile default; env `TALER_MON_LOCALE` / `TALER_DOMAIN_LOCALE`. | | **v1.19.1** | 2026-07-19 | **Bugfix / docs:** DEPENDENCIES — e2e requires `taler-helper-sqlite3` on mon PATH + preflight; README domains.conf documents **`lang`** column. | | **v1.19.0** | 2026-07-19 | **Feature:** domains.conf **`lang`** column (en\|fr) is the global UI language default per stack; i18n + HTML read `TALER_DOMAIN_LANG` / conf (FP profiles `lang=fr`). | diff --git a/check_amount_ladder.sh b/check_goa_ladder.sh similarity index 52% rename from check_amount_ladder.sh rename to check_goa_ladder.sh index 3bc13ed..410205f 100755 --- a/check_amount_ladder.sh +++ b/check_goa_ladder.sh @@ -1,36 +1,27 @@ #!/usr/bin/env bash -# check_amount_ladder.sh — generic withdraw/pay amount ladder (any Taler currency) -# -# Orchestrator for the modular amount ladder: -# Phase A withdraw — this file (classic plan or max-search) -# Phase B pay — ladder/lib_pay.sh (classic list or max payable) -# Force-select — ladder/extract_rpubs.py -# See ladder/README.md. +# check_goa_ladder.sh — withdraw/pay amount ladder (GOA + stage TESTPAYSAN) # # Flow (bank landings): # 1) GET /intro/auto-account.json → personal *account-* (balance 0) # 2) Mint pool withdrawals as explorer + confirm when selected # 3) wallet-cli accept-uri only (no run-until-done) # 4) bank confirm when selected; settle = balance + transfer_done -# 5) pay via free-amount template (GOA: goa-free) or private orders # -# Modes (LADDER_MODE): -# classic — default 23-rung withdraw + matching pay plan (pins 0, max-1, max) -# max — max-search: highest withdrawable AND highest payable -# (pay never exceeds wallet available; GOA uses public free-amount -# template goa-free by default) +# Amounts: random strictly increasing; pins 0, max-1, max. +# Phase A: withdraw ladder (cumulative wallet). Phase B: pay ladder. +# +# Stacks: +# GOA — LADDER_MAX_AMOUNT = libeufin ceiling; explorer from koopa-admin-secrets +# TESTPAYSAN stage — auto max_wire from bank /config, min denom from /keys; +# explorer from stagepaysan secrets / SSH # # Usage: -# ./taler-monitoring.sh ladder # classic -# ./taler-monitoring.sh max-ladder # LADDER_MODE=max -# ./taler-monitoring.sh goa-ladder # alias of ladder (phase name only) +# ./taler-monitoring.sh ladder # ./taler-monitoring.sh -d stage.lefrancpaysan.ch ladder -# LADDER_MODE=max LADDER_MAX_PROBES=32 ./taler-monitoring.sh ladder +# LADDER_MAX_AMOUNT=100 LADDER_STEPS=7 ./taler-monitoring.sh -d stage.lefrancpaysan.ch ladder # -# Env: LADDER_MODE, LADDER_STEPS, LADDER_MAX_AMOUNT, LADDER_MIN_AMOUNT, -# LADDER_MAX_PROBES, LADDER_MAX_TOL_REL, LADDER_HIGH_RUNGS, LADDER_STACK_AUTO, -# LADDER_PAY, LADDER_FREE_TEMPLATE, LADDER_FREE_TEMPLATE_INSTANCE, -# EXP_PW / EXP_PW_FILE, CLI_JS, MERCHANT_INSTANCE, … +# Env (see body): LADDER_STEPS, LADDER_MAX_AMOUNT, LADDER_MIN_AMOUNT, LADDER_STACK_AUTO, +# LADDER_PAY, EXP_PW / EXP_PW_FILE, CLI_JS, MERCHANT_INSTANCE, … set -euo pipefail ROOT=$(cd "$(dirname "$0")" && pwd) # shellcheck source=lib.sh @@ -38,7 +29,7 @@ source "$ROOT/lib.sh" # When invoked standalone (not via taler-monitoring.sh), still load secrets.env load_monitoring_secrets_env 2>/dev/null || true -# Area ladder.* — withdraw/pay amount ladder (any EXPECT_CURRENCY / stack) +# Area ladder.* — withdraw/pay amount ladder (GOA + TESTPAYSAN) # Groups: ladder.plan / ladder.load / ladder.withdraw / ladder.pay / ladder.report set_area ladder set_group plan @@ -73,26 +64,8 @@ elapsed_ms() { # Public variable template for stage pays (optional; fixed templates used when amount maps) : "${LADDER_PAY_TEMPLATE:=}" : "${LADDER_PAY_TEMPLATE_INSTANCE:=fermes-des-collines}" -# Free-amount public template (hacktivism: goa-free on goa-demo-cp4zqk — any GOA amount) -: "${LADDER_FREE_TEMPLATE:=goa-free}" -: "${LADDER_FREE_TEMPLATE_INSTANCE:=}" -# 1 = create pays via public free template POST (amount in body); 0 = private orders only -: "${LADDER_USE_FREE_TEMPLATE:=1}" -# classic | max (max = highest withdrawable + highest payable; phase alias max-ladder) -: "${LADDER_MODE:=classic}" -# max-search: probe budget (0 = run until convergence / LADDER_TIMEOUT_S only) -: "${LADDER_MAX_PROBES:=32}" -: "${LADDER_MAX_TOL_REL:=0.001}" -# first N probes = random high (log-uniform); then log-bisect between lo/hi -: "${LADDER_HIGH_RUNGS:=6}" -# optional seed for reproducible max-search (empty = time-based) -: "${LADDER_MAX_SEED:=}" -# After withdraw, wait for spendable available>0 before pay (secs total; shows countdown) -: "${LADDER_PAY_WAIT_AVAILABLE_S:=90}" -# Historic libeufin-ish absolute ceiling used as default LADDER_MAX_AMOUNT on GOA -LADDER_ABS_CEILING="4503599627370496" -GOA_LADDER_CEILING="${LADDER_ABS_CEILING}" # back-compat alias +GOA_LADDER_CEILING="4503599627370496" # Resolve wallet-cli .mjs (no hardcoded laptop path) if [ -z "${CLI_JS}" ] || [ ! -f "${CLI_JS}" ]; then @@ -157,7 +130,7 @@ print(min(vals) if vals else "") [ -n "$min_denom" ] || min_denom="0.01" # Only rewrite when caller left GOA defaults (explicit LADDER_MAX_AMOUNT=… kept) - if [ -n "$max_wire" ] && { [ "${LADDER_MAX_AMOUNT}" = "${GOA_LADDER_CEILING}" ] || [ "${LADDER_MAX_AMOUNT}" = "${LADDER_ABS_CEILING}" ] || [ "${LADDER_MAX_AMOUNT}" = "4503599627370496" ]; }; then + if [ -n "$max_wire" ] && { [ "${LADDER_MAX_AMOUNT}" = "${GOA_LADDER_CEILING}" ] || [ "${LADDER_MAX_AMOUNT}" = "4503599627370496" ]; }; then LADDER_MAX_AMOUNT="$max_wire" fi if [ -n "$min_denom" ] && { [ "${LADDER_MIN_AMOUNT}" = "0.000001" ] || [ "${LADDER_MIN_AMOUNT}" = "0.00000001" ]; }; then @@ -179,21 +152,6 @@ print(min(vals) if vals else "") info "ladder stack" "TESTPAYSAN · max=${CUR}:${LADDER_MAX_AMOUNT} min=${CUR}:${LADDER_MIN_AMOUNT} steps=${LADDER_STEPS} (from bank max_wire / keys denoms)" } apply_ladder_stack_defaults - -# Normalize mode (max-ladder phase / aliases) -case "${LADDER_MODE}" in - max|MAX|max-search|maxsearch|find-max|findmax) LADDER_MODE=max ;; - classic|CLASSIC|default|plan|"" ) LADDER_MODE=classic ;; - *) - warn ladder "unknown LADDER_MODE=${LADDER_MODE} — using classic" - LADDER_MODE=classic - ;; -esac -# max mode: withdraw + pay hunts (LADDER_PAY=0 still disables pay entirely) -: "${LADDER_FREE_TEMPLATE_INSTANCE:=${MERCHANT_INSTANCE}}" -export LADDER_MODE LADDER_MAX_PROBES LADDER_MAX_TOL_REL LADDER_HIGH_RUNGS LADDER_MAX_SEED LADDER_PAY -export LADDER_FREE_TEMPLATE LADDER_FREE_TEMPLATE_INSTANCE LADDER_USE_FREE_TEMPLATE LADDER_PAY_WAIT_AVAILABLE_S - SCRATCH=$(mktemp -d) WDB="$SCRATCH/wallet.sqlite3" REPORT_DIR="${LADDER_REPORT_DIR:-$SCRATCH}" @@ -213,17 +171,10 @@ ladder_over() { } wcli() { - # taler-helper-sqlite3 is Python ≥3.11; prefer Homebrew/local python on macOS - # so Apple /usr/bin/python3 3.9 does not FATAL the sqlite backend (no reservePub). - local _path="${PATH:-}" - case ":${_path}:" in - *:/opt/homebrew/bin:*) ;; - *) _path="/opt/homebrew/bin:/usr/local/bin:${_path}" ;; - esac if [ -n "${CLI_JS:-}" ] && [ -f "$CLI_JS" ]; then - PATH="$_path" node "$CLI_JS" --wallet-db="$WDB" --no-throttle "$@" + node "$CLI_JS" --wallet-db="$WDB" --no-throttle "$@" else - PATH="$_path" taler-wallet-cli --wallet-db="$WDB" --no-throttle "$@" + taler-wallet-cli --wallet-db="$WDB" --no-throttle "$@" fi } @@ -287,18 +238,10 @@ wallet_avail() { wcli balance 2>/dev/null | python3 -c ' import json,sys t=sys.stdin.read() -dec=json.JSONDecoder() i=t.find("{") if i<0: print("0"); raise SystemExit -try: - d,_=dec.raw_decode(t,i) -except Exception: - # last-resort: brace slice (may fail on trailing logs) - try: - d=json.loads(t[i:t.rfind("}")+1]) - except Exception: - print("0"); raise SystemExit +d=json.loads(t[i:t.rfind("}")+1]) cur=sys.argv[1] for b in d.get("balances") or []: a=b.get("available") or "" @@ -308,31 +251,6 @@ print("0") ' "$CUR" 2>/dev/null || echo "0" } -# pendingIncoming for same currency (same wallet DB as withdraw) — "0" if missing -wallet_pending_in() { - wcli balance 2>/dev/null | python3 -c ' -import json,sys -t=sys.stdin.read() -dec=json.JSONDecoder() -i=t.find("{") -if i<0: - print("0"); raise SystemExit -try: - d,_=dec.raw_decode(t,i) -except Exception: - try: - d=json.loads(t[i:t.rfind("}")+1]) - except Exception: - print("0"); raise SystemExit -cur=sys.argv[1] -for b in d.get("balances") or []: - a=b.get("pendingIncoming") or b.get("pending_incoming") or "" - if a.startswith(cur+":"): - print(a.split(":",1)[1]); raise SystemExit -print("0") -' "$CUR" 2>/dev/null || echo "0" -} - # Build paired pay + withdraw ladders (same step count, shared random shape). # Pay: [0] + log-uniform mids + [max-1] + [max] # Wd: [0] + max(mid, mid×scale) mids + [max-1] + [max] (mids higher → enough to spend) @@ -466,9 +384,6 @@ PY # shellcheck source=metrics.sh source "$ROOT/metrics.sh" -# Payment module (classic + max-search payable) -# shellcheck source=ladder/lib_pay.sh -source "$ROOT/ladder/lib_pay.sh" METRICS_DIR="$REPORT_DIR" ALT_UNITS_FILE="${REPORT_DIR}/alt_unit_names.json" export METRICS_DIR CUR WDB CLI_JS ALT_UNITS_FILE @@ -478,16 +393,11 @@ if [ "${LADDER_LOAD:-1}" = "0" ]; then export METRICS_LOAD fi -if [ "${LADDER_MODE}" = "max" ]; then - section "ladder · ${CUR} max-search (find highest withdrawable)" -else - section "ladder · ${CUR} classic (0 → random → max · ${LADDER_STEPS} steps)" -fi +section "ladder · ${CUR} withdraw (0 → random → max · ${LADDER_STEPS} steps)" info "bank" "$BANK" info "exchange" "$EX" info "currency" "$CUR" -info "ladder mode" "${LADDER_MODE}" -# alt_unit_names for human amounts (Kilo-GOA / Mega-GOA / … + base in parentheses) +# alt_unit_names for human amounts (Kilo-GOA / Mega-GOA / … + GOA in parentheses) if metrics_load_alt_units "${EX%/}/config"; then info "alt_unit_names" "from ${EX%/}/config → $ALT_UNITS_FILE" else @@ -495,12 +405,8 @@ else fi info "budget" "${LADDER_TIMEOUT_S}s" _max_m1=$(python3 -c 'import sys; from decimal import Decimal; m=Decimal(sys.argv[1]); print(m-1 if m>1 else m)' "${LADDER_MAX_AMOUNT}" 2>/dev/null || echo "${LADDER_MAX_AMOUNT}-1") -if [ "${LADDER_MODE}" = "max" ]; then - info "max-search" "probes≤${LADDER_MAX_PROBES} high_rand=${LADDER_HIGH_RUNGS} tol=${LADDER_MAX_TOL_REL} ceiling=${CUR}:${LADDER_MAX_AMOUNT}" -else - info "steps" "${LADDER_STEPS} (0 + random≥${LADDER_MIN_AMOUNT} + max-1=${CUR}:${_max_m1} + max=${CUR}:${LADDER_MAX_AMOUNT})" - info "withdraw_scale" "${LADDER_WITHDRAW_SCALE}× pay mids (fund pay ladder)" -fi +info "steps" "${LADDER_STEPS} (0 + random≥${LADDER_MIN_AMOUNT} + max-1=${CUR}:${_max_m1} + max=${CUR}:${LADDER_MAX_AMOUNT})" +info "withdraw_scale" "${LADDER_WITHDRAW_SCALE}× pay mids (fund pay ladder)" info "pay_phase" "$([ "${LADDER_PAY}" = "1" ] && echo enabled || echo disabled)" info "currency/domain" "${CUR} · ${TALER_DOMAIN:-?} · bank=${BANK}" @@ -572,39 +478,24 @@ wallet_prepare "main" ms_tos=$(elapsed_ms "$t0") ok "wallet exchange + ToS (${ms_tos}ms) — cumulative DB for withdraw+pay (no run-until-done)" +build_ladder_pair "$SCRATCH/ladder-wd-plan.txt" "$SCRATCH/ladder-pay-plan.txt" +LADDER_LIST=$(tr -d '\n' <"$SCRATCH/ladder-wd-plan.txt") +PAY_LIST=$(tr -d '\n' <"$SCRATCH/ladder-pay-plan.txt") : "${LADDER_MAX_RUNGS:=99}" -if [ "${LADDER_MODE}" = "max" ]; then - info "ladder mode" "max-search (highest withdrawable + highest payable ${CUR})" - info "max-search params" "probes≤${LADDER_MAX_PROBES} high_rand=${LADDER_HIGH_RUNGS} tol_rel=${LADDER_MAX_TOL_REL} ceiling=${CUR}:${LADDER_MAX_AMOUNT}" - info "max-search pay" "LADDER_PAY=${LADDER_PAY} free_template=${LADDER_FREE_TEMPLATE}@${LADDER_FREE_TEMPLATE_INSTANCE:-$INST}" - LADDER_LIST="" - PAY_LIST="" - LADDER_N=0 - set -- - printf '%s\n' "# max-search (amounts chosen adaptively)" >"$SCRATCH/ladder-plan.txt" - : >"$SCRATCH/ladder-wd-plan.txt" - : >"$SCRATCH/ladder-pay-plan.txt" -else - info "ladder mode" "classic (0 → random mids → max-1 → max)" - build_ladder_pair "$SCRATCH/ladder-wd-plan.txt" "$SCRATCH/ladder-pay-plan.txt" - LADDER_LIST=$(tr -d '\n' <"$SCRATCH/ladder-wd-plan.txt") - PAY_LIST=$(tr -d '\n' <"$SCRATCH/ladder-pay-plan.txt") - # shellcheck disable=SC2086 - set -- $LADDER_LIST - if [ "$#" -gt "$LADDER_MAX_RUNGS" ]; then - # shellcheck disable=SC2046 - set -- $(printf '%s\n' "$@" | head -n "$LADDER_MAX_RUNGS") - fi - LADDER_N=$# - info "withdraw plan" "$*" - info "withdraw plan (alt)" "$(format_amount_list_alt "$@")" - info "pay plan" "$PAY_LIST" - # shellcheck disable=SC2086 - info "pay plan (alt)" "$(format_amount_list_alt $PAY_LIST)" - printf '%s\n' "$@" >"$SCRATCH/ladder-plan.txt" - # shellcheck disable=SC2086 - printf '%s\n' $PAY_LIST >"$SCRATCH/ladder-pay-plan-lines.txt" 2>/dev/null || true +# shellcheck disable=SC2086 +set -- $LADDER_LIST +if [ "$#" -gt "$LADDER_MAX_RUNGS" ]; then + # shellcheck disable=SC2046 + set -- $(printf '%s\n' "$@" | head -n "$LADDER_MAX_RUNGS") fi +LADDER_N=$# +info "withdraw plan" "$*" +info "withdraw plan (alt)" "$(format_amount_list_alt "$@")" +info "pay plan" "$PAY_LIST" +# shellcheck disable=SC2086 +info "pay plan (alt)" "$(format_amount_list_alt $PAY_LIST)" +printf '%s\n' "$@" >"$SCRATCH/ladder-plan.txt" +printf '%s\n' $PAY_LIST >"$SCRATCH/ladder-pay-plan-lines.txt" 2>/dev/null || true OK_N=0 FAIL_N_L=0 @@ -612,27 +503,32 @@ PAY_OK_N=0 PAY_FAIL_N=0 STOP_REASON="" STOP_AMOUNT="" -MAX_BEST_AMT="" -MAX_LO="0" -MAX_HI="${LADDER_MAX_AMOUNT}" declare -a RUNG_JSON=() -# --------------------------------------------------------------------------- -# One withdraw rung. Caller sets: AMT, range_note, rung. -# Returns 0 = soft continue (ok/skip/ceiling); 1 = hard stop. -# Sets: status, AMT_NUM, WID, note, ms_*. -# --------------------------------------------------------------------------- -ladder_withdraw_rung() { +set_group withdraw +section "ladder · phase A · withdraw (${LADDER_N} rungs)" +rung=0 +for AMT in "$@"; do + rung=$((rung + 1)) if ladder_over; then STOP_REASON="timeout budget ${LADDER_TIMEOUT_S}s" warn ladder "time budget exhausted" \ "problem: LADDER_TIMEOUT_S=${LADDER_TIMEOUT_S}s reached after ${OK_N} ok rungs; remaining amounts not tried" - return 1 + break fi - # section header is printed by classic/max controller (avoids double banners) + section "ladder · rung $rung $AMT · $(format_amount_alt "$AMT")" tag=$(printf '%s' "$AMT" | tr '.:' '__') - # range_note set by caller (classic pins / max-search labels) + # TSV "range" column: fixed pins at ends, random mids (refined after AMT_NUM) + if [ "$rung" -eq 1 ]; then + range_note="pin:0" + elif [ "$rung" -eq "$LADDER_N" ]; then + range_note="pin:max" + elif [ "$rung" -eq $((LADDER_N - 1)) ] && [ "$LADDER_N" -ge 3 ]; then + range_note="pin:max-1" + else + range_note="random" + fi t_rung=$(now_ms) ms_mint=0 ms_accept=0 ms_confirm=0 ms_settle=0 note="" @@ -680,7 +576,7 @@ ladder_withdraw_rung() { "problem: bank will not create a GOA:0 withdrawal (zero amount probe). Ladder continues. detail: $note" echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" OK_N=$((OK_N + 1)) - return 0 + continue fi # Absolute max is a ceiling probe — bank often returns SQL P0001/5110; do not abort. if [ "$IS_MAX_PIN" = "1" ]; then @@ -689,35 +585,7 @@ ladder_withdraw_rung() { warn bank "mint $AMT rejected (ceiling)" \ "problem: bank rejects absolute LADDER_MAX_AMOUNT (often SQL P0001/5110). max-1 rung is the last expected success. Ladder continues to report. detail: $note" echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" - return 0 - fi - # max-1 can also hit the same libeufin SQL ceiling (5110/P0001) on live GOA — - # treat as soft ceiling when mid rungs already passed; do not block the report. - if [ "$IS_MAX_M1_PIN" = "1" ] && echo "$note" | grep -qE '5110|P0001'; then - status="CEILING_REJECT" - note="max-1 rejected (same ceiling as absolute max): $note" - warn bank "mint $AMT rejected (max-1 ceiling)" \ - "problem: bank rejects max-1 with SQL P0001/5110 (pool ceiling). Mid-rung OK_BANK/OK still count. Ladder continues. detail: $note" - echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" - return 0 - fi - # max-search: any mint reject is "too high" / ceiling — never hard-stop the hunt - if [ "${LADDER_MODE}" = "max" ]; then - status="CEILING_REJECT" - note="max-search too-high mint: $note" - warn bank "mint $AMT rejected (max-search)" \ - "problem: mint failed while hunting max withdrawable — treat as upper bound. detail: $note" - echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" - return 0 - fi - # Classic mid/high random rungs: same 5110/P0001 pool ceiling as pins — soft, not FAIL_MINT. - if echo "$note" | grep -qE '5110|P0001'; then - status="CEILING_REJECT" - note="amount above live mint ceiling (5110/P0001): $note" - warn bank "mint $AMT rejected (ceiling)" \ - "problem: bank mint HTTP 500/5110 SQL P0001 on high amount (not only pin:max). Ladder continues; use max-ladder to bound hi. detail: $note" - echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" - return 0 + continue fi err bank "mint $AMT failed" "$note" status="FAIL_MINT" @@ -725,7 +593,7 @@ ladder_withdraw_rung() { STOP_AMOUNT="$AMT" echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" FAIL_N_L=$((FAIL_N_L + 1)) - return 1 + break fi ok "mint $AMT ($WID) ${ms_mint}ms" @@ -755,13 +623,7 @@ ladder_withdraw_rung() { "problem: wallet code 7006 — no denominations match this amount (below smallest coin or not combinable). Ladder continues with next rung. detail: $note" fi echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" - return 0 - fi - if [ "${LADDER_MODE}" = "max" ]; then - status="SKIP_ACCEPT" - warn wallet "accept $AMT skipped (max-search)" "$note" - echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" - return 0 + continue fi err wallet "accept $AMT" "$note" status="FAIL_ACCEPT" @@ -769,7 +631,7 @@ ladder_withdraw_rung() { STOP_AMOUNT="$AMT" echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" FAIL_N_L=$((FAIL_N_L + 1)) - return 1 + break fi # Confirm ASAP when bank status is selected. No run-until-done (hangs on macOS/wallet). @@ -786,20 +648,120 @@ ladder_withdraw_rung() { # Collect reserve_pub candidates for *this* withdrawal (WID + amount). # Cumulative wallets re-print old reserves in accept/tx dumps — never trust a single "last" blindly. # Prints unique pubs one per line, preferred order first. - # - # Critical: wallet-cli dumps mix log lines + JSON. Never json.loads(rest_of_file) — - # trailing "Shutdown requested" lines make that always fail. Use JSONDecoder.raw_decode - # and regex on *both* accept and transactions output (reservePub lives in tx, not accept). extract_rpubs_for_wid() { - # External module — never feed extract via bash heredoc (refactor tools - # lives in ladder/extract_rpubs.py). - local _ext="$ROOT/ladder/extract_rpubs.py" - if [ ! -f "$_ext" ]; then - warn bank "force-select extract missing" "problem: $_ext not found" - return 0 - fi - python3 "$_ext" "$WID" "$AMT" \ - "$SCRATCH/accept-$tag.out" "$SCRATCH/tx-$tag.json" "$SCRATCH/used-rpubs.txt" + python3 - "$WID" "$AMT" "$SCRATCH/accept-$tag.out" "$SCRATCH/tx-$tag.json" "$SCRATCH/used-rpubs.txt" <<'PY' +import json, re, sys +from pathlib import Path + +wid = sys.argv[1] +amt = sys.argv[2] +paths = sys.argv[3:5] +used_path = sys.argv[5] +used = set() +if Path(used_path).is_file(): + used = {ln.strip() for ln in open(used_path) if ln.strip()} + +def walk_collect(o, bag, ctx=None): + ctx = dict(ctx or {}) + if isinstance(o, dict): + for k, v in o.items(): + kl = str(k).lower() + if kl in ("withdrawal_id", "withdraw_id", "wopid", "id") and isinstance(v, str): + ctx["id"] = v + if kl in ("amount", "rawamount", "instructedamount") and isinstance(v, str): + ctx["amount"] = v + if kl in ("taler_withdraw_uri", "talerwithdrawuri", "uri") and isinstance(v, str): + ctx["uri"] = v + if kl in ("reserve_pub", "reservepub") and isinstance(v, str) and len(v) >= 40: + bag.append((v, dict(ctx))) + walk_collect(v, bag, ctx) + elif isinstance(o, list): + for i in o: + walk_collect(i, bag, ctx) + +raw_pubs = [] # ordered as found +scored = [] # (score, pub) higher better +blob_all = "" +for p in paths: + try: + blob_all += open(p, errors="replace").read() + "\n" + except Exception: + pass + +# 1) full JSON objects in files +for p in paths: + try: + t = open(p, errors="replace").read() + except Exception: + continue + # whole-file JSON + for m in re.finditer(r"\{", t): + try: + o = json.loads(t[m.start():]) + except Exception: + continue + bag = [] + walk_collect(o, bag) + for pub, ctx in bag: + raw_pubs.append(pub) + score = 0 + cid = str(ctx.get("id") or "") + camt = str(ctx.get("amount") or "") + curi = str(ctx.get("uri") or "") + if wid and wid in cid: + score += 100 + if wid and wid in curi: + score += 80 + if amt and (camt == amt or camt.endswith(amt.split(":", 1)[-1])): + score += 40 + if pub in used: + score -= 200 + scored.append((score, pub)) + +# 2) regex fallback on accept file only (more current) +try: + acc = open(paths[0], errors="replace").read() +except Exception: + acc = "" +for pat in ( + r"\"reserve_pub\"\s*:\s*\"([A-Z0-9]{40,})\"", + r"\"reservePub\"\s*:\s*\"([A-Z0-9]{40,})\"", + r"reserve_pub[\"\s:=]+([A-Z0-9]{40,})", +): + for m in re.finditer(pat, acc, re.I): + pub = m.group(1) + raw_pubs.append(pub) + score = 10 + # proximity to WID in accept output + window = acc[max(0, m.start() - 400) : m.end() + 400] + if wid and wid in window: + score += 100 + if amt and amt in window: + score += 30 + if pub in used: + score -= 200 + scored.append((score, pub)) + +# prefer high score, then later occurrence +order = [] +seen = set() +for score, pub in sorted(scored, key=lambda x: (-x[0],), reverse=False): + # sort by score desc: use reverse sorted + pass +for score, pub in sorted(scored, key=lambda x: x[0], reverse=True): + if pub in seen or pub in used: + continue + seen.add(pub) + order.append(pub) +# append unused raw in reverse (newest-ish) +for pub in reversed(raw_pubs): + if pub in seen or pub in used: + continue + seen.add(pub) + order.append(pub) +for pub in order: + print(pub) +PY } mark_rpub_used() { @@ -906,7 +868,7 @@ else: "problem: bank status='${st:-empty}' after ${LADDER_CONFIRM_POLLS} polls (want selected/confirmed). Usually reserve_pub mismatch (5114), not out-of-money — mint/accept already OK. detail: $note" ms_total=$(elapsed_ms "$t_rung") echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" - return 0 + continue fi ok "confirm $AMT ${ms_confirm}ms (client, on selected)" @@ -965,13 +927,6 @@ sys.exit(0 if Decimal(sys.argv[1]) > Decimal(sys.argv[2]) else 1) metrics_record_flow withdrawn "$AMT" || true else note="no coins / no transfer_done avail=${after} $xfer" - if [ "${LADDER_MODE}" = "max" ]; then - status="SKIP_SETTLE" - warn wallet "settle $AMT skipped (max-search)" \ - "problem: bank confirm ok but settle inconclusive — do not move max bounds. detail: $note" - echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" - return 0 - fi err wallet "settle $AMT" "$note" status="FAIL_SETTLE" STOP_REASON="$note" @@ -979,235 +934,272 @@ sys.exit(0 if Decimal(sys.argv[1]) > Decimal(sys.argv[2]) else 1) FAIL_N_L=$((FAIL_N_L + 1)) echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" metrics_report_coins "ladder-r${rung}-fail-${tag}" || true - return 1 + break fi - return 0 -} - -set_group withdraw - -# --- next amount for max-search (prints: AMT_NUM|range_note|action) --- -# action: probe | done -max_search_next() { - python3 - "$CUR" "$MAX_LO" "$MAX_HI" "$LADDER_MIN_AMOUNT" \ - "$LADDER_MAX_TOL_REL" "$rung" "$LADDER_HIGH_RUNGS" "$LADDER_MAX_SEED" \ - "${MAX_BEST_OK:-}" <<'PY' -import math, random, sys -from decimal import Decimal, ROUND_HALF_UP, ROUND_UP - -cur = sys.argv[1] -lo = Decimal(sys.argv[2] or "0") -hi = Decimal(sys.argv[3]) -min_amt = Decimal(sys.argv[4]) -tol = Decimal(sys.argv[5] or "0.001") -probe_i = int(sys.argv[6]) -high_n = max(1, int(sys.argv[7] or "4")) -seed = sys.argv[8].strip() -best = sys.argv[9].strip() - -if seed: - random.seed(int(seed) + probe_i) -else: - random.seed() - -if min_amt <= 0: - min_amt = Decimal("0.000001") -if hi <= min_amt: - print("0|max:done|done") - raise SystemExit(0) - -if lo > 0 and hi > lo: - gap = (hi - lo) / hi - abs_step = Decimal(1) if hi >= 1 else min_amt - if gap <= tol or (hi - lo) <= abs_step: - print("%s|max:done|done" % (best or str(lo))) - raise SystemExit(0) - -def fmt(v: Decimal) -> str: - if v >= 1: - q = v.quantize(Decimal(1), rounding=ROUND_HALF_UP) - return format(int(q), "d") - q = v.quantize(Decimal("0.00000001"), rounding=ROUND_UP) - s = format(q, "f").rstrip("0").rstrip(".") - return s or "0" - -def clamp(v: Decimal): - if v <= lo and lo > 0: - v = lo + (Decimal(1) if lo >= 1 else min_amt) - if v >= hi: - v = hi - (Decimal(1) if hi > 1 else min_amt) - if v < min_amt: - v = min_amt - if v <= 0: - v = min_amt - if v >= hi: - return None - return v - -if probe_i < high_n: - band_lo = max(lo if lo > 0 else min_amt, hi * Decimal("0.15")) - band_hi = hi * Decimal("0.999") - if band_hi <= band_lo: - band_lo = max(min_amt, hi * Decimal("0.5")) - band_hi = hi * Decimal("0.99") - flo, fhi = float(band_lo), float(band_hi) - if flo <= 0: - flo = float(min_amt) - if fhi <= flo: - fhi = flo * 2 - v = Decimal(str(math.exp(random.uniform(math.log(flo), math.log(fhi))))) - v = clamp(v) - if v is None: - print("%s|max:done|done" % (best or str(lo))) - raise SystemExit(0) - print("%s|max:high-rand|probe" % fmt(v)) - raise SystemExit(0) - -if lo <= 0: - flo = float(max(min_amt, hi * Decimal("0.05"))) - fhi = float(hi * Decimal("0.85")) - if fhi <= flo: - fhi = flo * 1.5 - v = Decimal(str(math.exp(random.uniform(math.log(flo), math.log(fhi))))) - label = "max:descent" -else: - v = (lo * hi).sqrt() - label = "max:bisect" -v = clamp(v) -if v is None: - print("%s|max:done|done" % (best or str(lo))) - raise SystemExit(0) -print("%s|%s|probe" % (fmt(v), label)) -PY -} - -if [ "${LADDER_MODE}" = "max" ]; then - section "ladder · phase A · max-search (find highest withdrawable ${CUR})" - info "max-search" "ceiling=${CUR}:${LADDER_MAX_AMOUNT} min=${CUR}:${LADDER_MIN_AMOUNT} probes≤${LADDER_MAX_PROBES} high_rand=${LADDER_HIGH_RUNGS} tol_rel=${LADDER_MAX_TOL_REL}" - info "max-search" "pay phase $([ "${LADDER_PAY}" = "1" ] && echo enabled || echo disabled)" - MAX_LO="0" - MAX_HI="${LADDER_MAX_AMOUNT}" - MAX_BEST_OK="" - MAX_BEST_AMT="" - rung=0 - # LADDER_MAX_PROBES=0 → no probe cap (stop on converge / timeout only) - while [ "${LADDER_MAX_PROBES}" = "0" ] || [ "$rung" -lt "${LADDER_MAX_PROBES}" ]; do - if ladder_over; then - STOP_REASON="timeout budget ${LADDER_TIMEOUT_S}s" - warn ladder "time budget exhausted" \ - "problem: LADDER_TIMEOUT_S=${LADDER_TIMEOUT_S}s during max-search after ${OK_N} ok; best=${MAX_BEST_AMT:-none} · $(format_amount_alt "${MAX_BEST_AMT:-${CUR}:0}")" - break - fi - _next=$(max_search_next) - _anum=$(printf '%s' "$_next" | cut -d'|' -f1) - range_note=$(printf '%s' "$_next" | cut -d'|' -f2) - _act=$(printf '%s' "$_next" | cut -d'|' -f3) - if [ "$_act" = "done" ]; then - info "max-search" "converged best=$(format_amount_alt "${MAX_BEST_AMT:-${CUR}:0}") · lo=$(format_amount_alt "${CUR}:${MAX_LO}") · hi=$(format_amount_alt "${CUR}:${MAX_HI}") · probes=${rung}" - break - fi - AMT="${CUR}:${_anum}" - if [ -n "${_LAST_MAX_AMT:-}" ] && [ "$AMT" = "$_LAST_MAX_AMT" ]; then - info "max-search" "repeat amount $(format_amount_alt "$AMT") — tightening hi" - MAX_HI=$(python3 -c 'from decimal import Decimal; a=Decimal("'"${_anum}"'"); print(a-(1 if a>=1 else Decimal("'"${LADDER_MIN_AMOUNT}"'")))') - continue - fi - _LAST_MAX_AMT="$AMT" - rung=$((rung + 1)) - section "ladder · max probe $rung $AMT · $(format_amount_alt "$AMT") [lo=$(format_amount_alt "${CUR}:${MAX_LO}") · hi=$(format_amount_alt "${CUR}:${MAX_HI}")]" - status="FAIL" - if ! ladder_withdraw_rung; then - break - fi - AMT_NUM=$(python3 -c 'import sys; print(sys.argv[1].split(":",1)[-1])' "$AMT") - case "$status" in - OK|OK_BANK) - MAX_LO="$AMT_NUM" - MAX_BEST_OK="$AMT_NUM" - MAX_BEST_AMT="$AMT" - info "max-search" "success bound lo=$(format_amount_alt "${CUR}:${MAX_LO}") (best so far)" - ;; - CEILING_REJECT|FAIL_MINT) - MAX_HI=$(python3 -c ' -from decimal import Decimal -a=Decimal("'"$AMT_NUM"'"); h=Decimal("'"$MAX_HI"'") -print(min(a,h)) -') - info "max-search" "too-high hi:=$(format_amount_alt "${CUR}:${MAX_HI}")" - ;; - SKIP_DENOM|ZERO_SKIP|ZERO_REJECT) - info "max-search" "denom-skip $(format_amount_alt "$AMT") (bounds unchanged)" - ;; - SKIP_CONFIRM|SKIP_SETTLE|SKIP_ACCEPT) - info "max-search" "inconclusive $status $(format_amount_alt "$AMT") (bounds unchanged)" - ;; - *) - info "max-search" "status=$status $(format_amount_alt "$AMT") (bounds unchanged)" - ;; - esac - if python3 -c 'from decimal import Decimal; import sys; sys.exit(0 if Decimal(sys.argv[1])>0 and Decimal(sys.argv[2])<=Decimal(sys.argv[1]) else 1)' "$MAX_LO" "$MAX_HI" 2>/dev/null; then - info "max-search" "bounds crossed lo=$(format_amount_alt "${CUR}:${MAX_LO}") · hi=$(format_amount_alt "${CUR}:${MAX_HI}") — stop" - break - fi - done - if [ -n "${MAX_BEST_AMT:-}" ]; then - ok "max-search best withdrawable $(format_amount_alt "$MAX_BEST_AMT") · lo=$(format_amount_alt "${CUR}:${MAX_LO}") · hi=$(format_amount_alt "${CUR}:${MAX_HI}") · probes=${rung}" - else - warn ladder "max-search no successful withdraw" \ - "problem: no OK/OK_BANK within ${rung} probes; hi=$(format_amount_alt "${CUR}:${MAX_HI}")" - fi - { - echo "# max-search mode (withdraw)" - echo "best_wd=${MAX_BEST_AMT:-}" - echo "best_wd_alt=$(format_amount_alt "${MAX_BEST_AMT:-${CUR}:0}")" - echo "lo=${MAX_LO}" - echo "lo_alt=$(format_amount_alt "${CUR}:${MAX_LO}")" - echo "hi=${MAX_HI}" - echo "hi_alt=$(format_amount_alt "${CUR}:${MAX_HI}")" - echo "probes=${rung}" - } >"$SCRATCH/ladder-plan.txt" - printf '%s\n' "${MAX_BEST_AMT:-${CUR}:0}" >"$SCRATCH/ladder-wd-plan.txt" - : >"$SCRATCH/ladder-pay-plan.txt" - # pay phase uses max-search (not classic PAY_LIST) - PAY_LIST="" -else - section "ladder · phase A · withdraw (${LADDER_N} rungs)" - rung=0 - for AMT in "$@"; do - rung=$((rung + 1)) - if ladder_over; then - STOP_REASON="timeout budget ${LADDER_TIMEOUT_S}s" - warn ladder "time budget exhausted" \ - "problem: LADDER_TIMEOUT_S=${LADDER_TIMEOUT_S}s reached after ${OK_N} ok rungs; remaining amounts not tried" - break - fi - if [ "$rung" -eq 1 ]; then - range_note="pin:0" - elif [ "$rung" -eq "$LADDER_N" ]; then - range_note="pin:max" - elif [ "$rung" -eq $((LADDER_N - 1)) ] && [ "$LADDER_N" -ge 3 ]; then - range_note="pin:max-1" - else - range_note="random" - fi - section "ladder · rung $rung $AMT · $(format_amount_alt "$AMT")" - if ! ladder_withdraw_rung; then - break - fi - done -fi - +done # --------------------------------------------------------------------------- -# Phase B — payment (ladder/lib_pay.sh) -# classic: planned pay list · max: highest payable ≤ available (goa-free) +# Phase B — payment ladder (same shape: 0 → low → … → max-1 → max) # --------------------------------------------------------------------------- PAY_OK_N=0 PAY_FAIL_N=0 -MAX_PAY_BEST_AMT="" -MAX_PAY_LO="0" -MAX_PAY_HI="0" -ladder_run_pay_phase +if [ "${LADDER_PAY}" = "1" ] && [ -n "${PAY_LIST:-}" ] && [ "$FAIL_N_L" -eq 0 ]; then + set_group pay + section "ladder · phase B · pay" + metrics_report_coins "before-pay-ladder" || true + # Merchant secret (same as e2e) — stage often has only public shop templates + MPW="${E2E_MERCHANT_TOKEN:-${MERCHANT_TOKEN:-}}" + if [ -z "$MPW" ]; then + MPW=$(read_secret "taler-merchant/merchant-${INST}-password.txt" 2>/dev/null || true) + fi + if [ -z "$MPW" ]; then + MPW=$(read_secret "taler-merchant/merchant-goa-demo-cp4zqk-password.txt" 2>/dev/null || true) + fi + if [ -z "$MPW" ] && [ "${CUR}" = "TESTPAYSAN" ]; then + if [ -n "${FRANCPAYSAN_SECRETS:-}" ] && [ -f "${FRANCPAYSAN_SECRETS}/stage/default-instance-token.txt" ]; then + MPW=$(tr -d '\n\r' <"${FRANCPAYSAN_SECRETS}/stage/default-instance-token.txt") + fi + if [ -z "$MPW" ]; then + _remote_mer="${FRANCPAYSAN_REMOTE_MERCHANT_TOKEN:-}" + [ -z "$_remote_mer" ] && [ -n "${FRANCPAYSAN_REMOTE_SECRETS_ROOT:-}" ] && \ + _remote_mer="${FRANCPAYSAN_REMOTE_SECRETS_ROOT}/merchant/secrets/default-instance-token.txt" + for host in ${INSIDE_SSH:+"$INSIDE_SSH"} ${FRANCPAYSAN_SSH:+"$FRANCPAYSAN_SSH"}; do + [ -n "$host" ] && [ -n "$_remote_mer" ] || continue + MPW=$(ssh -o BatchMode=yes -o ConnectTimeout=12 "$host" \ + "tr -d '\\n\\r' <${_remote_mer} 2>/dev/null || true" \ + 2>/dev/null || true) + [ -n "$MPW" ] && break + done + unset _remote_mer + fi + fi + if [ -z "$MPW" ]; then + if [ "${CUR}" = "TESTPAYSAN" ]; then + # Stage: withdraw ladder is the main probe; pays need instance token or + # public templates for *exact* face values (not continuous ladder amounts). + info pay "no merchant token — skip pay ladder on TESTPAYSAN (withdraw maxima still covered)" + info pay "hint: set E2E_MERCHANT_TOKEN or use ./taler-monitoring.sh -d stage.lefrancpaysan.ch e2e for shop templates" + else + warn pay "no merchant token — skip pay ladder (set E2E_MERCHANT_TOKEN or secrets)" + fi + else + ok "merchant token" "instance ${INST}" + case "$MPW" in + secret-token:*) AUTH="Authorization: Bearer ${MPW}" ;; + *) AUTH="Authorization: Bearer secret-token:${MPW}" ;; + esac + wallet_prepare "main" + # shellcheck disable=SC2086 + set -- $PAY_LIST + PAY_N=$# + info "pay rungs" "$PAY_N · $*" + prung=0 + for PAMT in "$@"; do + prung=$((prung + 1)) + if ladder_over; then + warn pay "time budget exhausted" "after ${PAY_OK_N} ok pays" + break + fi + section "ladder · pay rung $prung $PAMT · $(format_amount_alt "$PAMT")" + ptag=$(printf '%s' "$PAMT" | tr '.:' '__') + if [ "$prung" -eq 1 ]; then + prange="pin:0" + elif [ "$prung" -eq "$PAY_N" ]; then + prange="pin:max" + elif [ "$prung" -eq $((PAY_N - 1)) ] && [ "$PAY_N" -ge 3 ]; then + prange="pin:max-1" + else + prange="random" + fi + PNUM=$(python3 -c 'import sys; print(sys.argv[1].split(":",1)[-1])' "$PAMT") + IS_PZERO=0 + python3 -c 'import sys; from decimal import Decimal; sys.exit(0 if Decimal(sys.argv[1])==0 else 1)' "$PNUM" 2>/dev/null && IS_PZERO=1 + IS_PMAX=0 + [ "$PNUM" = "${LADDER_MAX_AMOUNT}" ] && IS_PMAX=1 + t_pay=$(now_ms) + ms_order=0 ms_handle=0 ms_psettle=0 + pnote="" + pstatus="FAIL" + OID="-" + + metrics_report_coins "before-pay-${ptag}" || true + bal=$(wallet_avail) + + if [ "$IS_PZERO" = "1" ]; then + pstatus="ZERO_SKIP" + pnote="zero pay probe skipped" + warn pay "pay $PAMT skipped" "problem: zero amount order not useful. Ladder continues." + ms_total=$(elapsed_ms "$t_pay") + echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t${ms_handle}\t${ms_psettle}\t${ms_total}\t${OID}\t${pnote}" >>"$PAY_TSV" + PAY_OK_N=$((PAY_OK_N + 1)) + continue + fi + + # Soft: absolute max pay is a ceiling probe (unlikely affordable / merchant may reject) + if [ "$IS_PMAX" = "1" ]; then + # try once; on fail CEILING_REJECT + : + fi + + # Insufficient balance → soft skip for max pin only; hard fail for mid/max-1 + if ! python3 -c 'import sys; from decimal import Decimal; sys.exit(0 if Decimal(sys.argv[1])>=Decimal(sys.argv[2]) else 1)' "$bal" "$PNUM" 2>/dev/null; then + pnote="insufficient balance avail=${CUR}:${bal} need=${PAMT}" + ms_total=$(elapsed_ms "$t_pay") + if [ "$IS_PMAX" = "1" ]; then + pstatus="CEILING_SKIP" + warn pay "pay $PAMT skipped (ceiling)" "$pnote" + echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t0\t0\t0\t${ms_total}\t-\t${pnote}" >>"$PAY_TSV" + continue + fi + err pay "pay $PAMT" "$pnote" + pstatus="FAIL_BALANCE" + PAY_FAIL_N=$((PAY_FAIL_N + 1)) + FAIL_N_L=$((FAIL_N_L + 1)) + STOP_REASON="$pnote" + STOP_AMOUNT="$PAMT" + echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t0\t0\t0\t${ms_total}\t-\t${pnote}" >>"$PAY_TSV" + break + fi + + t0=$(now_ms) + SUM_JSON=$(python3 -c 'import json,sys; print(json.dumps(sys.argv[1]))' "ladder pay ${PAMT}" 2>/dev/null || echo '"ladder pay"') + curl -skS -m 20 -o "$SCRATCH/ord-$ptag.json" -X POST \ + -H "$AUTH" -H 'Content-Type: application/json' \ + -d "{\"order\":{\"summary\":${SUM_JSON},\"amount\":\"${PAMT}\",\"fulfillment_message\":\"ok\"},\"create_token\":true}" \ + "${MER}/instances/${INST}/private/orders" 2>"$SCRATCH/ord-$ptag.err" || true + ms_order=$(elapsed_ms "$t0") + OID=$(python3 -c 'import json,re,sys;t=open(sys.argv[1]).read() +try: print(json.loads(t).get("order_id") or "") +except Exception: + m=re.search(r"\"order_id\"\s*:\s*\"([^\"]+)\"",t); print(m.group(1) if m else "") +' "$SCRATCH/ord-$ptag.json" 2>/dev/null || true) + OTOK=$(python3 -c 'import json,re,sys;t=open(sys.argv[1]).read() +try: print(json.loads(t).get("token") or "") +except Exception: + m=re.search(r"\"token\"\s*:\s*\"([^\"]+)\"",t); print(m.group(1) if m else "") +' "$SCRATCH/ord-$ptag.json" 2>/dev/null || true) + if [ -z "$OID" ]; then + pnote="order create failed $(head -c 80 "$SCRATCH/ord-$ptag.json" 2>/dev/null | tr '\n\"' ' ')" + ms_total=$(elapsed_ms "$t_pay") + if [ "$IS_PMAX" = "1" ]; then + pstatus="CEILING_REJECT" + warn pay "pay $PAMT rejected (ceiling)" "$pnote" + echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t0\t0\t${ms_total}\t-\t${pnote}" >>"$PAY_TSV" + continue + fi + err pay "order $PAMT" "$pnote" + pstatus="FAIL_ORDER" + PAY_FAIL_N=$((PAY_FAIL_N + 1)) + FAIL_N_L=$((FAIL_N_L + 1)) + STOP_REASON="$pnote" + STOP_AMOUNT="$PAMT" + echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t0\t0\t${ms_total}\t-\t${pnote}" >>"$PAY_TSV" + break + fi + ok "order $OID ($PAMT) ${ms_order}ms" + + curl -skS -m 12 -o "$SCRATCH/ord-det-$ptag.json" -H "$AUTH" \ + "${MER}/instances/${INST}/private/orders/${OID}" 2>/dev/null || true + PAYURI=$(python3 -c 'import json,sys +try: print(json.load(open(sys.argv[1])).get("taler_pay_uri") or "") +except Exception: print("") +' "$SCRATCH/ord-det-$ptag.json" 2>/dev/null || true) + if [ -z "$PAYURI" ] && [ -n "$OTOK" ]; then + MH=$(python3 -c 'from urllib.parse import urlparse; print(urlparse("'"$MER"'").hostname or "taler.hacktivism.ch")' 2>/dev/null || echo "taler.hacktivism.ch") + PAYURI="taler://pay/${MH}/instances/${INST}/${OID}/?c=${OTOK}" + fi + PAYURI=$(printf '%s' "$PAYURI" | sed 's/:443\//\//g; s/:443?/?/g') + if [ -z "$PAYURI" ]; then + pnote="no pay URI for $OID" + ms_total=$(elapsed_ms "$t_pay") + err pay "uri $PAMT" "$pnote" + pstatus="FAIL_URI" + PAY_FAIL_N=$((PAY_FAIL_N + 1)) + FAIL_N_L=$((FAIL_N_L + 1)) + STOP_REASON="$pnote" + STOP_AMOUNT="$PAMT" + echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t0\t0\t${ms_total}\t${OID}\t${pnote}" >>"$PAY_TSV" + break + fi + + t0=$(now_ms) + if ! wcli handle-uri --yes "$PAYURI" >"$SCRATCH/pay-$ptag.out" 2>&1; then + ms_handle=$(elapsed_ms "$t0") + pnote="handle-uri failed $(tail -c 100 "$SCRATCH/pay-$ptag.out" | tr '\n\"' ' ')" + ms_total=$(elapsed_ms "$t_pay") + if [ "$IS_PMAX" = "1" ]; then + pstatus="CEILING_REJECT" + warn pay "pay $PAMT handle failed (ceiling)" "$pnote" + echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t${ms_handle}\t0\t${ms_total}\t${OID}\t${pnote}" >>"$PAY_TSV" + continue + fi + err pay "handle $PAMT" "$pnote" + pstatus="FAIL_HANDLE" + PAY_FAIL_N=$((PAY_FAIL_N + 1)) + FAIL_N_L=$((FAIL_N_L + 1)) + STOP_REASON="$pnote" + STOP_AMOUNT="$PAMT" + echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t${ms_handle}\t0\t${ms_total}\t${OID}\t${pnote}" >>"$PAY_TSV" + break + fi + ms_handle=$(elapsed_ms "$t0") + ok "handle-uri $PAMT ${ms_handle}ms" + + # Short settle polls: merchant order + wallet tx only — never run-until-done + t0=$(now_ms) + settled=0 + r=0 + while [ "$r" -lt "${LADDER_PAY_SETTLE_ROUNDS}" ]; do + r=$((r + 1)) + wcli transactions >"$SCRATCH/tx-$ptag.out" 2>&1 || true + curl -skS -m 8 -o "$SCRATCH/ord-paid-$ptag.json" -H "$AUTH" \ + "${MER}/instances/${INST}/private/orders/${OID}" 2>/dev/null || true + if grep -qiE 'payment|paid|Payment' "$SCRATCH/tx-$ptag.out" 2>/dev/null \ + || python3 -c 'import json,sys +d=json.load(open(sys.argv[1])) +sys.exit(0 if d.get("paid") is True or str(d.get("order_status","")).lower()=="paid" else 1) +' "$SCRATCH/ord-paid-$ptag.json" 2>/dev/null; then + settled=1 + break + fi + sleep 0.5 + done + ms_psettle=$(elapsed_ms "$t0") + ms_total=$(elapsed_ms "$t_pay") + after=$(wallet_avail) + if [ "$settled" = "1" ]; then + pstatus="OK" + pnote="avail=${CUR}:${after}" + ok "pay settled $PAMT → bal ${CUR}:${after} (total ${ms_total}ms)" + PAY_OK_N=$((PAY_OK_N + 1)) + echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t${ms_handle}\t${ms_psettle}\t${ms_total}\t${OID}\t${pnote}" >>"$PAY_TSV" + metrics_report_coins "after-pay-${ptag}" || true + metrics_record_flow spent "$PAMT" || true + else + pnote="not settled order=$OID avail=${CUR}:${after}" + if [ "$IS_PMAX" = "1" ]; then + pstatus="CEILING_REJECT" + warn pay "pay $PAMT not settled (ceiling)" "$pnote" + echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t${ms_handle}\t${ms_psettle}\t${ms_total}\t${OID}\t${pnote}" >>"$PAY_TSV" + continue + fi + err pay "settle $PAMT" "$pnote" + pstatus="FAIL_SETTLE" + PAY_FAIL_N=$((PAY_FAIL_N + 1)) + FAIL_N_L=$((FAIL_N_L + 1)) + STOP_REASON="$pnote" + STOP_AMOUNT="$PAMT" + echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t${ms_handle}\t${ms_psettle}\t${ms_total}\t${OID}\t${pnote}" >>"$PAY_TSV" + metrics_report_coins "after-pay-fail-${ptag}" || true + break + fi + done + metrics_report_coins "after-pay-ladder" || true + info "pay summary" "ok=$PAY_OK_N fail=$PAY_FAIL_N tsv=$PAY_TSV" + fi +elif [ "${LADDER_PAY}" = "1" ] && [ "$FAIL_N_L" -gt 0 ]; then + warn pay "skipped pay ladder" "withdraw phase already failed" +fi ms_phase=$(python3 -c 'import sys,time; print(int((time.time()-float(sys.argv[1]))*1000))' "$SECTION_T0") @@ -1227,264 +1219,92 @@ info "pay_tsv" "$PAY_TSV" # used to break shell argv quoting → "syntax error near unexpected token '('") export LADDER_REPORT_STOP_AMOUNT="${STOP_AMOUNT:-}" export LADDER_REPORT_STOP_REASON="${STOP_REASON:-}" -export LADDER_REPORT_MODE="${LADDER_MODE:-classic}" -export LADDER_REPORT_MAX_BEST="${MAX_BEST_AMT:-}" -export LADDER_REPORT_MAX_PAY_BEST="${MAX_PAY_BEST_AMT:-}" -export LADDER_REPORT_MAX_PAY_LO="${MAX_PAY_LO:-}" -export LADDER_REPORT_MAX_PAY_HI="${MAX_PAY_HI:-}" -export LADDER_REPORT_MAX_PAY_BEST_ALT="$(format_amount_alt "${MAX_PAY_BEST_AMT:-${CUR}:0}" 2>/dev/null || true)" -export LADDER_REPORT_MAX_LO="${MAX_LO:-}" -export LADDER_REPORT_MAX_HI="${MAX_HI:-}" -# Human alt_unit_names (e.g. 8.15 Tera-GOA (GOA:…)) for report + final lines -export LADDER_REPORT_MAX_BEST_ALT="$(format_amount_alt "${MAX_BEST_AMT:-${CUR}:0}" 2>/dev/null || true)" -export LADDER_REPORT_MAX_LO_ALT="$(format_amount_alt "${CUR}:${MAX_LO:-0}" 2>/dev/null || true)" -export LADDER_REPORT_MAX_HI_ALT="$(format_amount_alt "${CUR}:${MAX_HI:-0}" 2>/dev/null || true)" -export LADDER_REPORT_ALT_UNITS="${ALT_UNITS_FILE:-}" -# Report must never abort the ladder (set -e): soft-fail print/JSON only. -python3 - "$TSV" "$PAY_TSV" "$JSON" "$OK_N" "$FAIL_N_L" "$PAY_OK_N" "$PAY_FAIL_N" "$ms_phase" "$ACCT_USER" "$CUR" <<'PY' || true -import csv, json, os, sys, statistics, traceback -from decimal import Decimal, InvalidOperation, ROUND_HALF_UP +python3 - "$TSV" "$PAY_TSV" "$JSON" "$OK_N" "$FAIL_N_L" "$PAY_OK_N" "$PAY_FAIL_N" "$ms_phase" "$ACCT_USER" "$CUR" <<'PY' +import csv, json, os, sys, statistics +tsv, pay_tsv, jpath = sys.argv[1:4] +ok_n, fail_n, pay_ok, pay_fail, phase_ms, acct, cur = sys.argv[4:11] +stop_amt = os.environ.get("LADDER_REPORT_STOP_AMOUNT") or None +stop_reason = os.environ.get("LADDER_REPORT_STOP_REASON") or None -try: - tsv, pay_tsv, jpath = sys.argv[1:4] - ok_n, fail_n, pay_ok, pay_fail, phase_ms, acct, cur = sys.argv[4:11] - stop_amt = os.environ.get("LADDER_REPORT_STOP_AMOUNT") or None - stop_reason = os.environ.get("LADDER_REPORT_STOP_REASON") or None - mode = os.environ.get("LADDER_REPORT_MODE") or "classic" - max_best = os.environ.get("LADDER_REPORT_MAX_BEST") or None - max_lo = os.environ.get("LADDER_REPORT_MAX_LO") or None - max_hi = os.environ.get("LADDER_REPORT_MAX_HI") or None - max_best_alt = os.environ.get("LADDER_REPORT_MAX_BEST_ALT") or None - max_lo_alt = os.environ.get("LADDER_REPORT_MAX_LO_ALT") or None - max_hi_alt = os.environ.get("LADDER_REPORT_MAX_HI_ALT") or None - max_pay_best = os.environ.get("LADDER_REPORT_MAX_PAY_BEST") or None - max_pay_lo = os.environ.get("LADDER_REPORT_MAX_PAY_LO") or None - max_pay_hi = os.environ.get("LADDER_REPORT_MAX_PAY_HI") or None - max_pay_best_alt = os.environ.get("LADDER_REPORT_MAX_PAY_BEST_ALT") or None - alt_path = os.environ.get("LADDER_REPORT_ALT_UNITS") or "" - alt = {} - if alt_path: - try: - _loaded = json.load(open(alt_path)) - if isinstance(_loaded, dict): - alt = {str(k): str(v) for k, v in _loaded.items()} - except Exception: - alt = {} - if not isinstance(alt, dict) or not alt: - alt = {"0": cur or "GOA"} - elif "0" not in alt: - alt = dict(alt) - alt["0"] = cur or "GOA" +def load_rows(path): + rows = [] + try: + with open(path, newline="") as f: + for row in csv.DictReader(f, delimiter="\t"): + rows.append(row) + except Exception: + pass + return rows - def fmt_num(v): - try: - if v == v.to_integral(): - return format(int(v), "d") - s = format(v.quantize(Decimal("0.00000001"), rounding=ROUND_HALF_UP), "f") - return s.rstrip("0").rstrip(".") - except Exception: - return str(v) +rows = load_rows(tsv) +prows = load_rows(pay_tsv) - def format_alt(amt): - """Human alt_unit_names; never raises.""" +def nums(rs, key): + out = [] + for row in rs: try: - if not amt: - return "-" - s = str(amt).strip() - try: - if ":" in s: - c, v = s.split(":", 1) - val = Decimal(v) - else: - c, val = cur, Decimal(s) - except (InvalidOperation, ValueError, TypeError): - return s - au = alt if isinstance(alt, dict) else {"0": cur or "GOA"} - base_name = au.get("0") or c or cur or "GOA" - base_s = "%s:%s" % (c, fmt_num(val)) - if val == 0: - return "0 %s (%s)" % (base_name, base_s) - scales = [] - for k, name in au.items(): - try: - scales.append((int(k), str(name))) - except Exception: - continue - scales.sort(key=lambda x: -x[0]) - absval = abs(val) - chosen = None - for sc, name in scales: - try: - unit = Decimal(10) ** sc - if unit <= 0: - continue - coeff = absval / unit - if coeff >= 1: - chosen = (sc, name, coeff if val >= 0 else -coeff) - break - except Exception: - continue - if chosen is None: - return "%s %s (%s)" % (fmt_num(val), base_name, base_s) - sc, name, coeff = chosen - if sc == 0: - return "%s %s" % (fmt_num(val), name) - return "%s %s (%s)" % (fmt_num(coeff), name, base_s) - except Exception: - return str(amt) if amt else "-" - - def as_amt(s): - try: - if s is None or s == "": - return None - s = str(s).strip() - if ":" in s: - return s - return "%s:%s" % (cur, s) - except Exception: - return None - - def load_rows(path): - rows = [] - try: - with open(path, newline="") as f: - for row in csv.DictReader(f, delimiter="\t"): - rows.append(row) + out.append(int(row[key])) except Exception: pass - return rows + return out - rows = load_rows(tsv) - prows = load_rows(pay_tsv) - - def nums(rs, key): - out = [] - for row in rs: - try: - out.append(int(row[key])) - except Exception: - pass - return out - - def stats(xs): - if not xs: - return {"n": 0} - return { - "n": len(xs), - "min_ms": min(xs), - "max_ms": max(xs), - "avg_ms": int(sum(xs) / len(xs)), - "p50_ms": int(statistics.median(xs)), - } - - _max_block = None - if mode == "max": - _best = as_amt(max_best) if max_best else None - _lo = as_amt(max_lo) if max_lo is not None and str(max_lo) != "" else None - _hi = as_amt(max_hi) if max_hi is not None and str(max_hi) != "" else None - _pb = as_amt(max_pay_best) if max_pay_best else None - _max_block = { - "best": _best, - "best_alt": max_best_alt or (format_alt(_best) if _best else None), - "lo": max_lo, - "lo_alt": max_lo_alt or (format_alt(_lo) if _lo else None), - "hi": max_hi, - "hi_alt": max_hi_alt or (format_alt(_hi) if _hi else None), - "best_pay": _pb, - "best_pay_alt": max_pay_best_alt or (format_alt(_pb) if _pb else None), - "pay_lo": max_pay_lo, - "pay_hi": max_pay_hi, - } - - report = { - "currency": cur, - "mode": mode, - "auto_account": acct, - "ok_rungs": int(ok_n), - "fail_rungs": int(fail_n), - "pay_ok": int(pay_ok), - "pay_fail": int(pay_fail), - "max_search": _max_block, - "stopped_at_amount": stop_amt or None, - "stop_reason": stop_reason or None, - "phase_ms": int(phase_ms), - "timing": { - "mint": stats(nums(rows, "ms_mint")), - "accept": stats(nums(rows, "ms_accept")), - "confirm": stats(nums(rows, "ms_confirm")), - "settle": stats(nums(rows, "ms_settle")), - "rung_total": stats(nums(rows, "ms_total")), - "pay_order": stats(nums(prows, "ms_order")), - "pay_handle": stats(nums(prows, "ms_handle")), - "pay_settle": stats(nums(prows, "ms_settle")), - "pay_total": stats(nums(prows, "ms_total")), - }, - "rungs": rows, - "pays": prows, +def stats(xs): + if not xs: + return {"n": 0} + return { + "n": len(xs), + "min_ms": min(xs), + "max_ms": max(xs), + "avg_ms": int(sum(xs) / len(xs)), + "p50_ms": int(statistics.median(xs)), } - try: - json.dump(report, open(jpath, "w"), indent=2) - print("JSON", jpath) - except Exception as e: - print("JSON write error:", e) - print("--- withdraw speed (ms) ---") - for k in ("mint", "accept", "confirm", "settle", "rung_total"): - v = report["timing"][k] - if v.get("n"): - print(" %s n=%s min=%s p50=%s avg=%s max=%s" % ( - k.ljust(12), v["n"], v["min_ms"], v["p50_ms"], v["avg_ms"], v["max_ms"])) - print("--- pay speed (ms) ---") - for k in ("pay_order", "pay_handle", "pay_settle", "pay_total"): - v = report["timing"][k] - if v.get("n"): - print(" %s n=%s min=%s p50=%s avg=%s max=%s" % ( - k.ljust(12), v["n"], v["min_ms"], v["p50_ms"], v["avg_ms"], v["max_ms"])) - print("--- withdraw rungs ---") - for row in rows: - try: - amt = row.get("amount") or "" - aalt = format_alt(amt) if amt else "" - print(" %2s %-18s %-14s total=%sms %s" % ( - row.get("rung"), amt, row.get("status"), row.get("ms_total"), aalt)) - except Exception as e: - print(" ? row-print-error %s" % (e,)) - print("--- pay rungs ---") - for row in prows: - try: - amt = row.get("amount") or "" - aalt = format_alt(amt) if amt else "" - print(" %2s %-18s %-14s total=%sms oid=%s %s" % ( - row.get("rung"), amt, row.get("status"), row.get("ms_total"), - row.get("oid"), aalt)) - except Exception as e: - print(" ? pay-row-print-error %s" % (e,)) - if mode == "max": - print("--- max-search ---") - try: - ba = (max_best_alt or format_alt(as_amt(max_best) or "")) if max_best else "-" - la = (max_lo_alt or format_alt(as_amt(max_lo) or "")) if max_lo is not None and str(max_lo) != "" else "-" - ha = (max_hi_alt or format_alt(as_amt(max_hi) or "")) if max_hi is not None and str(max_hi) != "" else "-" - print(" best=%s" % (max_best or "-")) - print(" best_alt=%s" % ba) - print(" lo=%s" % (max_lo if max_lo is not None else "-")) - print(" lo_alt=%s" % la) - print(" hi=%s" % (max_hi if max_hi is not None else "-")) - print(" hi_alt=%s" % ha) - print(" best_pay=%s" % (max_pay_best or "-")) - print(" best_pay_alt=%s" % (max_pay_best_alt or (format_alt(as_amt(max_pay_best) or "") if max_pay_best else "-"))) - except Exception as e: - print(" max-search print error: %s" % (e,)) - if stop_amt: - try: - print("STOPPED at %s · %s: %s" % (stop_amt, format_alt(stop_amt), stop_reason)) - except Exception: - print("STOPPED at %s: %s" % (stop_amt, stop_reason)) - else: - print("Completed without hard failure (or budget stop without fail).") -except Exception: - traceback.print_exc() -sys.exit(0) +report = { + "currency": cur, + "auto_account": acct, + "ok_rungs": int(ok_n), + "fail_rungs": int(fail_n), + "pay_ok": int(pay_ok), + "pay_fail": int(pay_fail), + "stopped_at_amount": stop_amt or None, + "stop_reason": stop_reason or None, + "phase_ms": int(phase_ms), + "timing": { + "mint": stats(nums(rows, "ms_mint")), + "accept": stats(nums(rows, "ms_accept")), + "confirm": stats(nums(rows, "ms_confirm")), + "settle": stats(nums(rows, "ms_settle")), + "rung_total": stats(nums(rows, "ms_total")), + "pay_order": stats(nums(prows, "ms_order")), + "pay_handle": stats(nums(prows, "ms_handle")), + "pay_settle": stats(nums(prows, "ms_settle")), + "pay_total": stats(nums(prows, "ms_total")), + }, + "rungs": rows, + "pays": prows, +} +json.dump(report, open(jpath, "w"), indent=2) +print("JSON", jpath) +print("--- withdraw speed (ms) ---") +for k in ("mint", "accept", "confirm", "settle", "rung_total"): + v = report["timing"][k] + if v.get("n"): + print(" %s n=%s min=%s p50=%s avg=%s max=%s" % (k.ljust(12), v["n"], v["min_ms"], v["p50_ms"], v["avg_ms"], v["max_ms"])) +print("--- pay speed (ms) ---") +for k in ("pay_order", "pay_handle", "pay_settle", "pay_total"): + v = report["timing"][k] + if v.get("n"): + print(" %s n=%s min=%s p50=%s avg=%s max=%s" % (k.ljust(12), v["n"], v["min_ms"], v["p50_ms"], v["avg_ms"], v["max_ms"])) +print("--- withdraw rungs ---") +for row in rows: + print(" %2s %-16s %-12s total=%sms" % (row.get("rung"), row.get("amount"), row.get("status"), row.get("ms_total"))) +print("--- pay rungs ---") +for row in prows: + print(" %2s %-16s %-12s total=%sms oid=%s" % (row.get("rung"), row.get("amount"), row.get("status"), row.get("ms_total"), row.get("oid"))) +if stop_amt: + print("STOPPED at %s: %s" % (stop_amt, stop_reason)) +else: + print("Completed without hard failure (or budget stop without fail).") PY wcli balance 2>&1 | tee "$REPORT_DIR/balance-final.out" | tail -20 || true @@ -1515,7 +1335,7 @@ metrics_print_overall "ladder final" || true # Keep scratch if LADDER_REPORT_DIR set; else copy key files to /tmp if [ -z "${LADDER_REPORT_DIR:-}" ]; then - KEEP="/tmp/amount-ladder-report-$(date +%Y%m%d-%H%M%S)" + KEEP="/tmp/goa-ladder-report-$(date +%Y%m%d-%H%M%S)" mkdir -p "$KEEP" cp -a "$TSV" "$PAY_TSV" "$JSON" "$SCRATCH/auto-account.json" "$SCRATCH/ladder-plan.txt" \ "$SCRATCH/ladder-wd-plan.txt" "$SCRATCH/ladder-pay-plan.txt" \ @@ -1532,9 +1352,5 @@ if [ "$OK_N" -eq 0 ]; then blocker "ladder" "no successful withdraw rungs" exit 1 fi -if [ "${LADDER_MODE}" = "max" ] && [ -n "${MAX_BEST_AMT:-}" ]; then - ok "ladder finished mode=max wd_best=$(format_amount_alt "$MAX_BEST_AMT") · pay_best=$(format_amount_alt "${MAX_PAY_BEST_AMT:-${CUR}:0}") · withdraw_ok=$OK_N pay_ok=$PAY_OK_N phase=${ms_phase}ms" -else - ok "ladder finished mode=${LADDER_MODE} withdraw_ok=$OK_N pay_ok=$PAY_OK_N phase=${ms_phase}ms" -fi +ok "ladder finished withdraw_ok=$OK_N pay_ok=$PAY_OK_N phase=${ms_phase}ms" exit 0 diff --git a/check_monitoring_pages.sh b/check_monitoring_pages.sh index f4804b7..fea39cd 100755 --- a/check_monitoring_pages.sh +++ b/check_monitoring_pages.sh @@ -100,18 +100,14 @@ _host_allowed() { } # Public mon URL families (suite name "taler-monitoring" ≠ a public path): -# 1) /monitoring(+_err) — landing hosts (bare + slash) -# 2) /monitoring-ladder(+_err) — GOA classic amount ladder (daily) -# 3) /monitoring-max-ladder(+_err) — GOA max-search ladder (daily) -# 4) /taler-monitoring-surface(+_err) — taler.hacktivism.ch only -# 5) /taler-monitoring-aptdeploy(+_err) — taler.hacktivism.ch only +# 1) /monitoring(+_err) — landing hosts (bare + slash) +# 2) /taler-monitoring-surface(+_err) — taler.hacktivism.ch only +# 3) /taler-monitoring-aptdeploy(+_err) — taler.hacktivism.ch only # No /taler-monitoring page. No /taler-monitoring-mail|mattermost pages. _path_allowed() { case "$1" in monitoring|monitoring_err|\ - monitoring-ladder|monitoring-ladder_err|\ - monitoring-max-ladder|monitoring-max-ladder_err|\ taler-monitoring-surface|taler-monitoring-surface_err|\ taler-monitoring-aptdeploy|taler-monitoring-aptdeploy_err) return 0 ;; # explicit deny (never invent a mon page here) @@ -152,8 +148,6 @@ _catalog_urls() { goa) for h in bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch; do printf 'https://%s/monitoring/\n' "$h" - printf 'https://%s/monitoring-ladder/\n' "$h" - printf 'https://%s/monitoring-max-ladder/\n' "$h" done # ecosystem surface (versions / software) + apt-src deploy mon page printf 'https://taler.hacktivism.ch/taler-monitoring-surface/\n' @@ -171,12 +165,6 @@ _catalog_urls() { while IFS= read -r h; do [ -n "$h" ] || continue printf 'https://%s/monitoring/\n' "$h" - case "$h" in - *.hacktivism.ch) - printf 'https://%s/monitoring-ladder/\n' "$h" - printf 'https://%s/monitoring-max-ladder/\n' "$h" - ;; - esac done < <(_landing_allowed) printf 'https://taler.hacktivism.ch/taler-monitoring-surface/\n' printf 'https://taler.hacktivism.ch/taler-monitoring-aptdeploy/\n' diff --git a/host-agent/README.md b/host-agent/README.md index 6c8ea53..ffbbcef 100644 --- a/host-agent/README.md +++ b/host-agent/README.md @@ -23,28 +23,9 @@ Thin wrappers only set stack defaults: | `run-aptdeploy.sh` | CLI: ensure + **aptdeploy** (no HTML) | local if hostname=koopa, else **ssh $DEPLOY_SSH** | | `run-aptdeploy-monitoring.sh` | apt-src tests → `/taler-monitoring-aptdeploy*` | `hernani` @ koopa (4h) | | `run-surface-monitoring.sh` | **surface** → only `/taler-monitoring-surface*` (ecosystem + mail + mattermost + versions) | `hernani` @ koopa (hourly) | -| `run-ladder-monitoring.sh` | GOA **classic** amount ladder → `/monitoring-ladder/` on bank/exchange/taler | `hernani` @ koopa (**daily**) | -| `run-max-ladder-monitoring.sh` | GOA **max-search** ladder → `/monitoring-max-ladder/` on bank/exchange/taler | `hernani` @ koopa (**daily**) | Deprecated mail/mattermost wrappers: **`../meta/deprecated/`** (not installed). -### GOA amount ladder pages (daily) - -| URL | Phase | Timer | -|-----|-------|-------| -| `https://{bank,exchange,taler}.hacktivism.ch/monitoring-ladder/` | `ladder` (classic) | `taler-monitoring-ladder.timer` (~05:15 + jitter) | -| `https://{bank,exchange,taler}.hacktivism.ch/monitoring-max-ladder/` | `max-ladder` | `taler-monitoring-max-ladder.timer` (~06:30 + jitter) | - -```bash -systemctl --user enable --now taler-monitoring-ladder.timer -systemctl --user enable --now taler-monitoring-max-ladder.timer -# first round now: -systemctl --user start taler-monitoring-ladder.service -systemctl --user start taler-monitoring-max-ladder.service -``` - -Needs explorer secrets (`SECRETS_ROOT` / bank-explorer password). Long wall: classic ~2h, max ~3h (`RUN_TIMEOUT`). - ### apt-src deploy tests (hacktivism `/monitoring`) | Container | Mode | diff --git a/host-agent/ROOT-APPLY-MONITORING.md b/host-agent/ROOT-APPLY-MONITORING.md index 9687af2..0a594d1 100644 --- a/host-agent/ROOT-APPLY-MONITORING.md +++ b/host-agent/ROOT-APPLY-MONITORING.md @@ -48,13 +48,9 @@ There is **no** public page `/taler-monitoring`. Each family needs **bare and sl | Family | Paths | Host | Source timer | |--------|-------|------|----------------| | **monitoring** | `/monitoring` · `/monitoring/` · `/monitoring_err` · `/monitoring_err/` | bank + exchange + taler (and other landings) | `taler-monitoring-hacktivism.timer` (4h) | -| **monitoring-ladder** | `/monitoring-ladder` · `/` · `_err` | bank + exchange + taler (**GOA**) | `taler-monitoring-ladder.timer` (**daily**) | -| **monitoring-max-ladder** | `/monitoring-max-ladder` · `/` · `_err` | bank + exchange + taler (**GOA**) | `taler-monitoring-max-ladder.timer` (**daily**) | | **taler-monitoring-surface** | bare + slash + `_err` | **taler.hacktivism.ch only** | `taler-monitoring-surface.timer` (1h) | | **taler-monitoring-aptdeploy** | bare + slash + `_err` | **taler.hacktivism.ch only** | `taler-monitoring-aptdeploy.timer` (4h) | -Served via existing Caddy `handle /monitoring*` (file_server under `/var/www/monitoring-sites/{host}/`). - HTML is generated as **hernani** into `$DEPLOY_STAGING (or $HOME/monitoring-sites-staging)/`. diff --git a/host-agent/apply-monitoring-live.sh b/host-agent/apply-monitoring-live.sh index f14b7e7..4916ba8 100755 --- a/host-agent/apply-monitoring-live.sh +++ b/host-agent/apply-monitoring-live.sh @@ -57,12 +57,6 @@ urls=( https://taler.hacktivism.ch/monitoring/ https://bank.hacktivism.ch/monitoring/ https://exchange.hacktivism.ch/monitoring/ - https://taler.hacktivism.ch/monitoring-ladder/ - https://bank.hacktivism.ch/monitoring-ladder/ - https://exchange.hacktivism.ch/monitoring-ladder/ - https://taler.hacktivism.ch/monitoring-max-ladder/ - https://bank.hacktivism.ch/monitoring-max-ladder/ - https://exchange.hacktivism.ch/monitoring-max-ladder/ https://taler.hacktivism.ch/taler-monitoring-surface https://taler.hacktivism.ch/taler-monitoring-surface/ https://taler.hacktivism.ch/taler-monitoring-surface_err/ diff --git a/host-agent/install-host-agent.sh b/host-agent/install-host-agent.sh index d4624c2..3468455 100755 --- a/host-agent/install-host-agent.sh +++ b/host-agent/install-host-agent.sh @@ -63,19 +63,11 @@ install_local() { # optional aptdeploy (not a public mon page layout — v1.8.0 simplified overview) install -m 644 "$ROOT/taler-monitoring-aptdeploy.service" "$home_unit/" install -m 644 "$ROOT/taler-monitoring-aptdeploy.timer" "$home_unit/" - # GOA amount ladder pages (daily) - install -m 644 "$ROOT/taler-monitoring-ladder.service" "$home_unit/" - install -m 644 "$ROOT/taler-monitoring-ladder.timer" "$home_unit/" - install -m 644 "$ROOT/taler-monitoring-max-ladder.service" "$home_unit/" - install -m 644 "$ROOT/taler-monitoring-max-ladder.timer" "$home_unit/" - chmod +x "$ROOT/run-ladder-monitoring.sh" "$ROOT/run-max-ladder-monitoring.sh" 2>/dev/null || true systemctl --user daemon-reload systemctl --user enable --now taler-monitoring-hacktivism.path systemctl --user enable --now taler-monitoring-hacktivism.timer systemctl --user enable --now taler-monitoring-surface.timer systemctl --user enable --now taler-monitoring-aptdeploy.timer - systemctl --user enable --now taler-monitoring-ladder.timer - systemctl --user enable --now taler-monitoring-max-ladder.timer # Remove leftover mail/mattermost units if present (moved to meta/deprecated/) systemctl --user disable --now taler-monitoring-mattermost.timer 2>/dev/null || true systemctl --user disable --now taler-monitoring-mail.timer 2>/dev/null || true @@ -89,28 +81,20 @@ install_local() { systemctl --user start taler-monitoring-hacktivism.service || true systemctl --user start taler-monitoring-surface.service || true systemctl --user start taler-monitoring-aptdeploy.service || true - # first ladder rounds immediately (classic then max-search; both daily thereafter) - systemctl --user start taler-monitoring-ladder.service || true - systemctl --user start taler-monitoring-max-ladder.service || true echo "--- status ---" systemctl --user status taler-monitoring-hacktivism.path --no-pager -l | head -15 systemctl --user status taler-monitoring-hacktivism.timer --no-pager -l | head -12 systemctl --user status taler-monitoring-surface.timer --no-pager -l | head -12 systemctl --user status taler-monitoring-aptdeploy.timer --no-pager -l | head -12 - systemctl --user status taler-monitoring-ladder.timer --no-pager -l | head -12 - systemctl --user status taler-monitoring-max-ladder.timer --no-pager -l | head -12 systemctl --user list-timers --all | grep taler-monitoring || true - echo "OK host-agent:" + echo "OK host-agent (v1.8.0 simplified):" echo " · hacktivism 4h → /monitoring on bank/exchange/taler (landing stacks)" - echo " · ladder daily → /monitoring-ladder/ (classic amount ladder, GOA)" - echo " · max-ladder daily → /monitoring-max-ladder/ (max-search, GOA)" echo " · surface 1h → /taler-monitoring-surface* (ecosystem versions/software)" - echo " · aptdeploy 4h (optional tests)" + echo " · aptdeploy 4h (optional tests; not a public mon-page type)" echo " · mail/mattermost: not installed (meta/deprecated/; content in surface)" echo "suite: $mon" - echo "manual ladder: systemctl --user start taler-monitoring-ladder.service" - echo "manual max-ladder: systemctl --user start taler-monitoring-max-ladder.service" + echo "manual surface: systemctl --user start taler-monitoring-surface.service" echo "after upgrades: $ROOT/touch-software-stamp.sh" } diff --git a/host-agent/run-hacktivism-monitoring.sh b/host-agent/run-hacktivism-monitoring.sh index 9599cb1..02132b8 100755 --- a/host-agent/run-hacktivism-monitoring.sh +++ b/host-agent/run-hacktivism-monitoring.sh @@ -16,12 +16,10 @@ export INSIDE_MODE="${INSIDE_MODE:-local-podman}" export LOCAL_STACK="${LOCAL_STACK:-1}" export CONTINUE_ON_ERROR="${CONTINUE_ON_ERROR:-1}" export RUN_TIMEOUT="${RUN_TIMEOUT:-600}" -# stack checks + full monpages inventory (landings + ladder + surface + aptdeploy). +# stack checks + full monpages inventory (landings + surface + aptdeploy bare/slash). # Specialized pages also have their own timers; full outside-in inventory lives here. -# run-ladder-monitoring.sh → /monitoring-ladder* (daily classic) -# run-max-ladder-monitoring.sh → /monitoring-max-ladder* (daily max-search) -# run-aptdeploy-monitoring.sh → monpages job-only for /taler-monitoring-aptdeploy* -# run-surface-monitoring.sh → monpages job-only for /taler-monitoring-surface* +# run-aptdeploy-monitoring.sh → monpages job-only for /taler-monitoring-aptdeploy* +# run-surface-monitoring.sh → monpages job-only for /taler-monitoring-surface* export PHASES="${PHASES:-urls inside versions monpages}" export MON_HOSTS="${MON_HOSTS:-bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch}" export HTML_OUT="${HTML_OUT:-$HOME/monitoring-sites-staging}" diff --git a/host-agent/run-ladder-monitoring.sh b/host-agent/run-ladder-monitoring.sh deleted file mode 100755 index c51f4f3..0000000 --- a/host-agent/run-ladder-monitoring.sh +++ /dev/null @@ -1,43 +0,0 @@ -#!/usr/bin/env bash -# run-ladder-monitoring.sh — GOA classic amount ladder → public HTML -# -# Public (landing hosts, same Caddy /monitoring* static root): -# https://{bank,exchange,taler}.hacktivism.ch/monitoring-ladder/ -# https://{bank,exchange,taler}.hacktivism.ch/monitoring-ladder_err/ -# -# Timer: once daily (taler-monitoring-ladder.timer). Long wall clock. -# -set -uo pipefail -AGENT_DIR=$(cd "$(dirname "$0")" && pwd) - -export AGENT_LABEL="${AGENT_LABEL:-ladder-host-agent}" -export STATE_NAME="${STATE_NAME:-taler-ladder-monitoring}" -export TALER_DOMAIN="${TALER_DOMAIN:-hacktivism.ch}" -export TALER_MON_LANG="${TALER_MON_LANG:-en}" -export TALER_MON_LANG_SET=1 -export INSIDE_PODMAN="${INSIDE_PODMAN:-1}" -export INSIDE_MODE="${INSIDE_MODE:-local-podman}" -export LOCAL_STACK="${LOCAL_STACK:-1}" -export SKIP_SSH="${SKIP_SSH:-0}" -export CONTINUE_ON_ERROR="${CONTINUE_ON_ERROR:-1}" -# classic 23-rung + pay can run long -export RUN_TIMEOUT="${RUN_TIMEOUT:-7200}" - -export PHASES="ladder monpages" -export MON_HOSTS="${MON_HOSTS:-bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch}" -export HTML_OUT="${HTML_OUT:-$HOME/monitoring-sites-staging}" -export DEPLOY_WWW_ROOT="${DEPLOY_WWW_ROOT:-/var/www/monitoring-sites}" - -export HTML_OK_DIR="monitoring-ladder" -export HTML_ERR_DIR="monitoring-ladder_err" -export HTML_URL_OK="/monitoring-ladder/" -export HTML_URL_ERR="/monitoring-ladder_err/" -export PAGE_LABEL="monitoring-ladder" -export MONPAGES_INVENTORY="job" - -# Classic ladder defaults (override in ~/.config/taler-monitoring/env) -export LADDER_MODE="${LADDER_MODE:-classic}" -export LADDER_PAY="${LADDER_PAY:-1}" -export APT_DEPLOY_ENSURE=0 - -exec bash "$AGENT_DIR/run-host-report.sh" "$@" diff --git a/host-agent/run-max-ladder-monitoring.sh b/host-agent/run-max-ladder-monitoring.sh deleted file mode 100755 index c8735a8..0000000 --- a/host-agent/run-max-ladder-monitoring.sh +++ /dev/null @@ -1,42 +0,0 @@ -#!/usr/bin/env bash -# run-max-ladder-monitoring.sh — GOA max-search amount ladder → public HTML -# -# Public (landing hosts, same Caddy /monitoring* static root): -# https://{bank,exchange,taler}.hacktivism.ch/monitoring-max-ladder/ -# https://{bank,exchange,taler}.hacktivism.ch/monitoring-max-ladder_err/ -# -# Timer: once daily (taler-monitoring-max-ladder.timer). Long wall clock. -# -set -uo pipefail -AGENT_DIR=$(cd "$(dirname "$0")" && pwd) - -export AGENT_LABEL="${AGENT_LABEL:-max-ladder-host-agent}" -export STATE_NAME="${STATE_NAME:-taler-max-ladder-monitoring}" -export TALER_DOMAIN="${TALER_DOMAIN:-hacktivism.ch}" -export TALER_MON_LANG="${TALER_MON_LANG:-en}" -export TALER_MON_LANG_SET=1 -export INSIDE_PODMAN="${INSIDE_PODMAN:-1}" -export INSIDE_MODE="${INSIDE_MODE:-local-podman}" -export LOCAL_STACK="${LOCAL_STACK:-1}" -export SKIP_SSH="${SKIP_SSH:-0}" -export CONTINUE_ON_ERROR="${CONTINUE_ON_ERROR:-1}" -# max-search probes + pay hunt -export RUN_TIMEOUT="${RUN_TIMEOUT:-10800}" - -export PHASES="max-ladder monpages" -export MON_HOSTS="${MON_HOSTS:-bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch}" -export HTML_OUT="${HTML_OUT:-$HOME/monitoring-sites-staging}" -export DEPLOY_WWW_ROOT="${DEPLOY_WWW_ROOT:-/var/www/monitoring-sites}" - -export HTML_OK_DIR="monitoring-max-ladder" -export HTML_ERR_DIR="monitoring-max-ladder_err" -export HTML_URL_OK="/monitoring-max-ladder/" -export HTML_URL_ERR="/monitoring-max-ladder_err/" -export PAGE_LABEL="monitoring-max-ladder" -export MONPAGES_INVENTORY="job" - -export LADDER_MODE=max -export LADDER_PAY="${LADDER_PAY:-1}" -export APT_DEPLOY_ENSURE=0 - -exec bash "$AGENT_DIR/run-host-report.sh" "$@" diff --git a/host-agent/taler-monitoring-ladder.service b/host-agent/taler-monitoring-ladder.service deleted file mode 100644 index c93bcd8..0000000 --- a/host-agent/taler-monitoring-ladder.service +++ /dev/null @@ -1,19 +0,0 @@ -[Unit] -Description=Taler monitoring GOA classic amount ladder → /monitoring-ladder/ -Documentation=file:%h/src/taler-monitoring/host-agent/README.md -After=network-online.target -Wants=network-online.target - -[Service] -Type=oneshot -Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin -Environment=CONTINUE_ON_ERROR=1 -Environment=RUN_TIMEOUT=7200 -EnvironmentFile=-%h/.config/taler-monitoring/env -WorkingDirectory=%h/src/taler-monitoring -ExecStart=%h/src/taler-monitoring/host-agent/run-ladder-monitoring.sh -Nice=10 -TimeoutStartSec=3h - -[Install] -WantedBy=default.target diff --git a/host-agent/taler-monitoring-ladder.timer b/host-agent/taler-monitoring-ladder.timer deleted file mode 100644 index a076315..0000000 --- a/host-agent/taler-monitoring-ladder.timer +++ /dev/null @@ -1,14 +0,0 @@ -[Unit] -Description=Daily GOA classic amount ladder mon page -Documentation=file:%h/src/taler-monitoring/host-agent/README.md - -[Timer] -# Once per day (CEST morning-ish on typical hosts); randomize to avoid thundering herd -OnCalendar=*-*-* 05:15:00 -Persistent=true -AccuracySec=15min -RandomizedDelaySec=30min -Unit=taler-monitoring-ladder.service - -[Install] -WantedBy=timers.target diff --git a/host-agent/taler-monitoring-max-ladder.service b/host-agent/taler-monitoring-max-ladder.service deleted file mode 100644 index 9943edb..0000000 --- a/host-agent/taler-monitoring-max-ladder.service +++ /dev/null @@ -1,19 +0,0 @@ -[Unit] -Description=Taler monitoring GOA max-search amount ladder → /monitoring-max-ladder/ -Documentation=file:%h/src/taler-monitoring/host-agent/README.md -After=network-online.target -Wants=network-online.target - -[Service] -Type=oneshot -Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin -Environment=CONTINUE_ON_ERROR=1 -Environment=RUN_TIMEOUT=10800 -EnvironmentFile=-%h/.config/taler-monitoring/env -WorkingDirectory=%h/src/taler-monitoring -ExecStart=%h/src/taler-monitoring/host-agent/run-max-ladder-monitoring.sh -Nice=10 -TimeoutStartSec=4h - -[Install] -WantedBy=default.target diff --git a/host-agent/taler-monitoring-max-ladder.timer b/host-agent/taler-monitoring-max-ladder.timer deleted file mode 100644 index 5b16361..0000000 --- a/host-agent/taler-monitoring-max-ladder.timer +++ /dev/null @@ -1,14 +0,0 @@ -[Unit] -Description=Daily GOA max-search amount ladder mon page -Documentation=file:%h/src/taler-monitoring/host-agent/README.md - -[Timer] -# Once per day, after classic ladder window -OnCalendar=*-*-* 06:30:00 -Persistent=true -AccuracySec=15min -RandomizedDelaySec=30min -Unit=taler-monitoring-max-ladder.service - -[Install] -WantedBy=timers.target diff --git a/ladder/README.md b/ladder/README.md deleted file mode 100644 index 3c6e826..0000000 --- a/ladder/README.md +++ /dev/null @@ -1,32 +0,0 @@ -# Amount ladder modules - -**Entry:** `../check_amount_ladder.sh` via: - -```bash -./taler-monitoring.sh ladder # classic 23-rung withdraw + pay -./taler-monitoring.sh max-ladder # max-search withdraw + pay -./taler-monitoring.sh goa-ladder # legacy phase name = classic ladder -``` - -| Path | Role | -|------|------| -| `../check_amount_ladder.sh` | Orchestrator: classic / max **withdraw**, then pay phase | -| `lib_pay.sh` | **Pay** module: free-amount template, classic pay list, max **payable** ≤ available | -| `extract_rpubs.py` | Force-select: scrape `reservePub` from wallet dumps | - -There is **no** `check_goa_ladder.sh` — removed; use the phases above. - -## Modes - -- **classic** (default): 23-rung withdraw 0→mids→max-1→max, then matching pay plan (never overspend). -- **max** (`max-ladder`): adaptive max withdraw, then adaptive max pay via public free template (hacktivism: `goa-free`). - -## Env (pay) - -| Variable | Default | Meaning | -|----------|---------|---------| -| `LADDER_PAY` | `1` | `0` = skip all pays | -| `LADDER_USE_FREE_TEMPLATE` | `1` | public POST template with `{"amount":…}` | -| `LADDER_FREE_TEMPLATE` | `goa-free` | template id (hacktivism) | -| `LADDER_FREE_TEMPLATE_INSTANCE` | merchant instance | e.g. `goa-demo-cp4zqk` | -| `LADDER_PAY_WAIT_AVAILABLE_S` | `90` | after withdraw, wait for spendable **available**>0; logs **withdraw board in start order** (OK/OK_BANK) with live `pendingIncoming`→`done` + timer (same wallet DB) | diff --git a/ladder/extract_rpubs.py b/ladder/extract_rpubs.py deleted file mode 100755 index 1c939e9..0000000 --- a/ladder/extract_rpubs.py +++ /dev/null @@ -1,163 +0,0 @@ -#!/usr/bin/env python3 -"""Extract reserve_pub candidates from wallet accept + transactions dumps. - -Used by check_amount_ladder.sh force-select. Kept as a file (not a bash -heredoc) so shell refactors cannot turn Python ``continue`` into ``return``. - -Usage: - ladder_extract_rpubs.py WID AMT accept.out tx.json used-rpubs.txt -Prints one reserve pub per line, preferred first. -""" -from __future__ import annotations - -import json -import re -import sys -from pathlib import Path - - -def main(argv: list[str]) -> int: - if len(argv) < 6: - print( - "usage: ladder_extract_rpubs.py WID AMT accept.out tx.json used-rpubs.txt", - file=sys.stderr, - ) - return 2 - - wid = argv[1] - amt = argv[2] - paths = argv[3:5] - used_path = argv[5] - used: set[str] = set() - if Path(used_path).is_file(): - used = {ln.strip() for ln in open(used_path) if ln.strip()} - - amt_num = amt.split(":", 1)[-1] if amt else "" - dec = json.JSONDecoder() - - def walk_collect(o, bag, ctx=None): - ctx = dict(ctx or {}) - if isinstance(o, dict): - for k, v in o.items(): - if not isinstance(v, str): - continue - kl = str(k).lower().replace("-", "_") - if kl in ( - "withdrawal_id", - "withdraw_id", - "wopid", - "id", - "transactionid", - "transaction_id", - ): - ctx["id"] = v - elif kl in ( - "amount", - "rawamount", - "instructedamount", - "amounteffective", - "amountraw", - "transferamount", - ): - ctx["amount"] = v - elif kl in ( - "taler_withdraw_uri", - "talerwithdrawuri", - "uri", - "bankconfirmationurl", - "confirmtransferurl", - ): - ctx["uri"] = v - for k, v in o.items(): - kl = str(k).lower().replace("-", "_") - if kl in ("reserve_pub", "reservepub") and isinstance(v, str) and len(v) >= 40: - bag.append((v, dict(ctx))) - elif not isinstance(v, (str, int, float, bool)) and v is not None: - walk_collect(v, bag, ctx) - elif isinstance(o, list): - for i in o: - walk_collect(i, bag, ctx) - - def score_ctx(pub, ctx, window=""): - score = 0 - cid = str(ctx.get("id") or "") - camt = str(ctx.get("amount") or "") - curi = str(ctx.get("uri") or "") - if wid and wid in curi: - score += 120 - if wid and wid in cid: - score += 100 - if wid and wid in window: - score += 80 - if amt and (camt == amt or camt.endswith(amt_num)): - score += 50 - if amt and amt in window: - score += 30 - if pub in used: - score -= 200 - return score - - def iter_json_objects(text: str): - i, n = 0, len(text) - while i < n: - if text[i] == "{": - try: - o, end = dec.raw_decode(text, i) - yield o - i = end - continue - except Exception: - pass - i += 1 - - raw_pubs: list[str] = [] - scored: list[tuple[int, str]] = [] - - for p in paths: - try: - t = open(p, errors="replace").read() - except Exception: - continue - for o in iter_json_objects(t): - bag = [] - walk_collect(o, bag) - for pub, ctx in bag: - raw_pubs.append(pub) - scored.append((score_ctx(pub, ctx), pub)) - - rpub_pats = ( - r"\"reservePub\"\s*:\s*\"([A-Z0-9]{40,})\"", - r"\"reserve_pub\"\s*:\s*\"([A-Z0-9]{40,})\"", - r"reserve[_ ]?pub[\"\s:=]+([A-Z0-9]{40,})", - ) - for p in paths: - try: - blob = open(p, errors="replace").read() - except Exception: - continue - for pat in rpub_pats: - for m in re.finditer(pat, blob, re.I): - pub = m.group(1) - raw_pubs.append(pub) - window = blob[max(0, m.start() - 600) : m.end() + 600] - scored.append((score_ctx(pub, {}, window) + 10, pub)) - - order: list[str] = [] - seen: set[str] = set() - for score, pub in sorted(scored, key=lambda x: x[0], reverse=True): - if pub in seen or pub in used: - continue - seen.add(pub) - order.append(pub) - for pub in reversed(raw_pubs): - if pub in seen or pub in used: - continue - seen.add(pub) - order.append(pub) - for pub in order: - print(pub) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main(sys.argv)) diff --git a/ladder/lib_pay.sh b/ladder/lib_pay.sh deleted file mode 100644 index e8ae15a..0000000 --- a/ladder/lib_pay.sh +++ /dev/null @@ -1,559 +0,0 @@ -# ladder/lib_pay.sh — payment side of amount ladder (sourced by check_amount_ladder.sh) -# Expects: lib.sh + metrics.sh already sourced; CUR BANK MER INST EX SCRATCH WDB -# wcli wallet_avail format_amount_alt ladder_over now_ms elapsed_ms -# LADDER_* env, AUTH optional, PAY_TSV PAY_OK_N PAY_FAIL_N -# -# Public free-amount template (hacktivism): goa-free @ goa-demo-cp4zqk -# Never spends more than wallet available. - -: "${LADDER_FREE_TEMPLATE:=goa-free}" -: "${LADDER_FREE_TEMPLATE_INSTANCE:=}" -: "${LADDER_USE_FREE_TEMPLATE:=1}" -: "${LADDER_PAY_WAIT_AVAILABLE_S:=90}" -: "${LADDER_PAY_SETTLE_ROUNDS:=6}" - -MAX_PAY_BEST_AMT="${MAX_PAY_BEST_AMT:-}" -MAX_PAY_LO="${MAX_PAY_LO:-0}" -MAX_PAY_HI="${MAX_PAY_HI:-0}" - -ladder_pay_resolve_merchant() { - # Sets AUTH (may be empty if free template only) - MPW="${E2E_MERCHANT_TOKEN:-${MERCHANT_TOKEN:-}}" - if [ -z "$MPW" ]; then - MPW=$(read_secret "taler-merchant/merchant-${INST}-password.txt" 2>/dev/null || true) - fi - if [ -z "$MPW" ]; then - MPW=$(read_secret "taler-merchant/merchant-goa-demo-cp4zqk-password.txt" 2>/dev/null || true) - fi - if [ -z "$MPW" ] && [ "${CUR}" = "TESTPAYSAN" ]; then - if [ -n "${FRANCPAYSAN_SECRETS:-}" ] && [ -f "${FRANCPAYSAN_SECRETS}/stage/default-instance-token.txt" ]; then - MPW=$(tr -d '\n\r' <"${FRANCPAYSAN_SECRETS}/stage/default-instance-token.txt") - fi - fi - AUTH="" - if [ -n "$MPW" ]; then - case "$MPW" in - secret-token:*) AUTH="Authorization: Bearer ${MPW}" ;; - *) AUTH="Authorization: Bearer secret-token:${MPW}" ;; - esac - fi - : "${LADDER_FREE_TEMPLATE_INSTANCE:=${INST}}" -} - -# Create order for PAMT → OID OTOK PAYURI. Prefers public free template. -ladder_create_order() { - local pamt="$1" ptag="$2" - local inst mh body tpl_https sum_json - OID=""; OTOK=""; PAYURI="" - inst="${LADDER_FREE_TEMPLATE_INSTANCE:-$INST}" - mh=$(python3 -c 'from urllib.parse import urlparse; print(urlparse("'"${MER}"'").hostname or "taler.hacktivism.ch")' 2>/dev/null || echo "taler.hacktivism.ch") - - if [ "${LADDER_USE_FREE_TEMPLATE}" = "1" ] && [ -n "${LADDER_FREE_TEMPLATE:-}" ]; then - tpl_https="${MER}/instances/${inst}/templates/${LADDER_FREE_TEMPLATE}" - body=$(python3 -c 'import json,sys; print(json.dumps({"amount":sys.argv[1]}))' "$pamt" 2>/dev/null || printf '{"amount":"%s"}' "$pamt") - curl -skS -m 20 -o "$SCRATCH/ord-$ptag.json" -X POST \ - -H 'Content-Type: application/json' \ - -d "$body" \ - "$tpl_https" 2>"$SCRATCH/ord-$ptag.err" || true - OID=$(python3 -c 'import json,re,sys;t=open(sys.argv[1]).read() -try: print(json.loads(t).get("order_id") or "") -except Exception: - m=re.search(r"\"order_id\"\s*:\s*\"([^\"]+)\"",t); print(m.group(1) if m else "") -' "$SCRATCH/ord-$ptag.json" 2>/dev/null || true) - OTOK=$(python3 -c 'import json,re,sys;t=open(sys.argv[1]).read() -try: print(json.loads(t).get("token") or "") -except Exception: - m=re.search(r"\"token\"\s*:\s*\"([^\"]+)\"",t); print(m.group(1) if m else "") -' "$SCRATCH/ord-$ptag.json" 2>/dev/null || true) - if [ -n "$OID" ] && [ -n "$OTOK" ]; then - PAYURI="taler://pay/${mh}/instances/${inst}/${OID}/?c=${OTOK}" - PAYURI=$(printf '%s' "$PAYURI" | sed 's/:443\//\//g; s/:443?/?/g') - return 0 - fi - fi - - [ -n "${AUTH:-}" ] || return 1 - sum_json=$(python3 -c 'import json,sys; print(json.dumps(sys.argv[1]))' "ladder pay ${pamt}" 2>/dev/null || echo '"ladder pay"') - curl -skS -m 20 -o "$SCRATCH/ord-$ptag.json" -X POST \ - -H "$AUTH" -H 'Content-Type: application/json' \ - -d "{\"order\":{\"summary\":${sum_json},\"amount\":\"${pamt}\",\"fulfillment_message\":\"ok\"},\"create_token\":true}" \ - "${MER}/instances/${INST}/private/orders" 2>"$SCRATCH/ord-$ptag.err" || true - OID=$(python3 -c 'import json,re,sys;t=open(sys.argv[1]).read() -try: print(json.loads(t).get("order_id") or "") -except Exception: - m=re.search(r"\"order_id\"\s*:\s*\"([^\"]+)\"",t); print(m.group(1) if m else "") -' "$SCRATCH/ord-$ptag.json" 2>/dev/null || true) - OTOK=$(python3 -c 'import json,re,sys;t=open(sys.argv[1]).read() -try: print(json.loads(t).get("token") or "") -except Exception: - m=re.search(r"\"token\"\s*:\s*\"([^\"]+)\"",t); print(m.group(1) if m else "") -' "$SCRATCH/ord-$ptag.json" 2>/dev/null || true) - [ -n "$OID" ] || return 1 - curl -skS -m 12 -o "$SCRATCH/ord-det-$ptag.json" -H "$AUTH" \ - "${MER}/instances/${INST}/private/orders/${OID}" 2>/dev/null || true - PAYURI=$(python3 -c 'import json,sys -try: print(json.load(open(sys.argv[1])).get("taler_pay_uri") or "") -except Exception: print("") -' "$SCRATCH/ord-det-$ptag.json" 2>/dev/null || true) - if [ -z "$PAYURI" ] && [ -n "$OTOK" ]; then - PAYURI="taler://pay/${mh}/instances/${INST}/${OID}/?c=${OTOK}" - fi - PAYURI=$(printf '%s' "$PAYURI" | sed 's/:443\//\//g; s/:443?/?/g') - [ -n "$PAYURI" ] -} - -# One pay attempt; never overspends. Always soft (no hard ladder stop). -ladder_pay_attempt() { - local pamt="$1" prange="$2" prung="$3" - local ptag pnum bal t_pay t0 ms_order ms_handle ms_psettle after settled r pstatus pnote - ptag=$(printf '%s' "$pamt" | tr '.:' '__') - pnum=$(python3 -c 'import sys; print(sys.argv[1].split(":",1)[-1])' "$pamt") - t_pay=$(now_ms) - ms_order=0; ms_handle=0; ms_psettle=0 - pnote=""; pstatus="FAIL"; OID="-" - - bal=$(wallet_avail) - if ! python3 -c 'import sys; from decimal import Decimal; sys.exit(0 if Decimal(sys.argv[1])>0 and Decimal(sys.argv[1])>=Decimal(sys.argv[2]) else 1)' "$bal" "$pnum" 2>/dev/null; then - pstatus="SKIP_BALANCE" - pnote="skip: would spend more than available avail=${CUR}:${bal} need=${pamt}" - warn pay "pay $pamt skipped (balance)" \ - "problem: never overspend. $(format_amount_alt "${CUR}:${bal}") free vs $(format_amount_alt "$pamt")" - echo -e "${prung}\t${prange}\t${pamt}\t${pstatus}\t0\t0\t0\t$(elapsed_ms "$t_pay")\t-\t${pnote}" >>"$PAY_TSV" - return 0 - fi - - t0=$(now_ms) - if ! ladder_create_order "$pamt" "$ptag"; then - ms_order=$(elapsed_ms "$t0") - pstatus="FAIL_ORDER" - pnote="order create failed $(head -c 100 "$SCRATCH/ord-$ptag.json" 2>/dev/null | tr '\n\"' ' ')" - warn pay "order $pamt" "$pnote" - echo -e "${prung}\t${prange}\t${pamt}\t${pstatus}\t${ms_order}\t0\t0\t$(elapsed_ms "$t_pay")\t-\t${pnote}" >>"$PAY_TSV" - return 0 - fi - ms_order=$(elapsed_ms "$t0") - ok "order $OID ($pamt) ${ms_order}ms · $(format_amount_alt "$pamt")" - - t0=$(now_ms) - if ! wcli handle-uri --yes "$PAYURI" >"$SCRATCH/pay-$ptag.out" 2>&1; then - ms_handle=$(elapsed_ms "$t0") - pstatus="FAIL_HANDLE" - pnote="handle-uri failed $(tail -c 100 "$SCRATCH/pay-$ptag.out" | tr '\n\"' ' ')" - warn pay "handle $pamt" "$pnote" - echo -e "${prung}\t${prange}\t${pamt}\t${pstatus}\t${ms_order}\t${ms_handle}\t0\t$(elapsed_ms "$t_pay")\t${OID}\t${pnote}" >>"$PAY_TSV" - return 0 - fi - ms_handle=$(elapsed_ms "$t0") - ok "handle-uri $pamt ${ms_handle}ms" - - t0=$(now_ms) - settled=0 - r=0 - while [ "$r" -lt "${LADDER_PAY_SETTLE_ROUNDS}" ]; do - r=$((r + 1)) - wcli transactions >"$SCRATCH/tx-pay-$ptag.out" 2>&1 || true - if [ -n "${AUTH:-}" ]; then - curl -skS -m 8 -o "$SCRATCH/ord-paid-$ptag.json" -H "$AUTH" \ - "${MER}/instances/${INST}/private/orders/${OID}" 2>/dev/null || true - fi - if grep -qiE 'payment|paid|Payment' "$SCRATCH/tx-pay-$ptag.out" 2>/dev/null \ - || python3 -c 'import json,sys -try: - d=json.load(open(sys.argv[1])) - sys.exit(0 if d.get("paid") is True or str(d.get("order_status","")).lower()=="paid" else 1) -except Exception: - sys.exit(1) -' "$SCRATCH/ord-paid-$ptag.json" 2>/dev/null; then - settled=1 - break - fi - sleep 0.5 - done - ms_psettle=$(elapsed_ms "$t0") - after=$(wallet_avail) - if [ "$settled" = "1" ]; then - pstatus="OK" - pnote="avail=${CUR}:${after}" - ok "pay settled $pamt · $(format_amount_alt "$pamt") → bal ${CUR}:${after}" - PAY_OK_N=$((PAY_OK_N + 1)) - echo -e "${prung}\t${prange}\t${pamt}\t${pstatus}\t${ms_order}\t${ms_handle}\t${ms_psettle}\t$(elapsed_ms "$t_pay")\t${OID}\t${pnote}" >>"$PAY_TSV" - metrics_report_coins "after-pay-${ptag}" || true - metrics_record_flow spent "$pamt" || true - else - pstatus="FAIL_SETTLE" - pnote="not settled order=$OID avail=${CUR}:${after}" - warn pay "settle $pamt" "$pnote" - echo -e "${prung}\t${prange}\t${pamt}\t${pstatus}\t${ms_order}\t${ms_handle}\t${ms_psettle}\t$(elapsed_ms "$t_pay")\t${OID}\t${pnote}" >>"$PAY_TSV" - fi - return 0 -} - -# Print withdraw board: OK/OK_BANK rungs in *start order* + wallet tx state. -# Uses TSV (ladder order) + transactions dump. Call after refreshing tx dump. -# Args: left_secs available_num pendingIncoming_num -ladder_print_wd_board() { - local left="${1:-?}" bal="${2:-0}" pend="${3:-0}" - local tsv="${TSV:-}" txf="${SCRATCH:-/tmp}/tx-wait-pay.out" - local board - board=$( - python3 - "$tsv" "$txf" "${CUR:-GOA}" "$left" "$bal" "$pend" <<'PY' 2>/dev/null || true -import json, sys -from pathlib import Path - -tsv, txf, cur, left, bal, pend = sys.argv[1:7] -dec = json.JSONDecoder() - -def num(a: str) -> str: - a = (a or "").strip() - if ":" in a: - return a.split(":", 1)[1] - return a - -def iter_json(text: str): - i, n = 0, len(text) - while i < n: - if text[i] == "{": - try: - o, end = dec.raw_decode(text, i) - yield o - i = end - continue - except Exception: - pass - i += 1 - -# TSV rows in start order (OK / OK_BANK only — successful bank path) -rows = [] -if tsv and Path(tsv).is_file(): - for ln in open(tsv, errors="replace"): - if not ln.strip() or ln.startswith("rung"): - continue - p = ln.rstrip("\n").split("\t") - if len(p) < 10: - continue - status = p[3] - if status not in ("OK", "OK_BANK"): - continue - rows.append({ - "rung": p[0], - "amt": p[2], - "bank": status, - "wid": (p[9] if len(p) > 9 else "")[:12], - }) - -# Wallet withdrawal txs (chronological if timestamp present) -wds = [] -if txf and Path(txf).is_file(): - try: - text = open(txf, errors="replace").read() - except Exception: - text = "" - for o in iter_json(text): - txs = o.get("transactions") if isinstance(o, dict) else None - if not isinstance(txs, list): - continue - for t in txs: - if not isinstance(t, dict): - continue - typ = str(t.get("type") or t.get("Type") or "").lower() - if "withdraw" not in typ: - continue - amt = t.get("amountRaw") or t.get("amountEffective") or t.get("amount") or "" - st = ( - t.get("txState") - or t.get("status") - or t.get("pendingMajor") - or t.get("txMajorState") - or "" - ) - st = str(st).lower() - det = t.get("withdrawalDetails") or t.get("withdrawal_details") or {} - if not isinstance(det, dict): - det = {} - conf = det.get("confirmed") - if conf is True and not st: - st = "done" - elif conf is False and not st: - st = "pending" - # pending* / dialog / kyc → still incoming; done/abort final - if any(x in st for x in ("done", "abort", "fail", "delete")): - phase = "done" if "done" in st else st - elif st in ("", "none", "null"): - phase = "unknown" - else: - phase = "pendingIncoming" - ts = 0 - for k in ("timestamp", "timestamp_ms", "age"): - v = t.get(k) - if isinstance(v, dict): - v = v.get("t_s") or v.get("t_ms") or v.get("t_usec") - try: - ts = int(v or 0) - except Exception: - ts = 0 - if ts: - break - wds.append({"amt": str(amt), "n": num(str(amt)), "phase": phase, "st": st or "?", "ts": ts}) - -wds.sort(key=lambda x: x["ts"]) -used = [False] * len(wds) - -lines = [ - f"withdraw board (start order) · timer {left}s left · available={cur}:{bal} · pendingIncoming={cur}:{pend}" -] -if not rows: - lines.append(" (no OK/OK_BANK rows in TSV yet)") -else: - for i, r in enumerate(rows, 1): - n = num(r["amt"]) - match = None - for j, w in enumerate(wds): - if used[j]: - continue - if w["n"] == n or w["amt"] == r["amt"] or w["amt"].endswith(":" + n): - used[j] = True - match = w - break - if match is None: - wlabel = "wallet=? (no tx match)" - elif match["phase"] == "pendingIncoming": - wlabel = f"wallet=pendingIncoming ({match['st']})" - elif match["phase"] == "done": - wlabel = f"wallet=done ({match['st']})" - else: - wlabel = f"wallet={match['phase']} ({match['st']})" - wid = r["wid"] or "-" - lines.append( - f" #{i:<3} rung={r['rung']:<4} {r['amt']:<22} bank={r['bank']:<8} {wlabel} wid={wid}" - ) - pending_n = sum(1 for w in wds if w["phase"] == "pendingIncoming") - done_n = sum(1 for w in wds if w["phase"] == "done") - lines.append( - f" summary: tsv_ok={len(rows)} · wallet_wd={len(wds)} (pendingIncoming={pending_n} done={done_n})" - ) - -print("\n".join(lines)) -PY - ) - if [ -n "$board" ]; then - while IFS= read -r line || [ -n "$line" ]; do - [ -n "$line" ] && info pay "$line" - done <<<"$board" - else - info pay "timer ${left}s left · available=${CUR}:${bal} · pendingIncoming=${CUR}:${pend} · same wallet (board n/a)" - fi -} - -# Wait for spendable coins on the *same* wallet DB as withdraw (WDB). -# After OK_BANK, available is often still 0 while pendingIncoming > 0 — pay needs available. -# Shows withdraw board in start order + countdown (never looks hung). -ladder_wait_available() { - local wait_s="${1:-$LADDER_PAY_WAIT_AVAILABLE_S}" - local end now left bal pend tick=0 - end=$(( $(date +%s) + wait_s )) - bal=$(wallet_avail) - pend=$(wallet_pending_in 2>/dev/null || echo "0") - if python3 -c 'import sys; from decimal import Decimal; sys.exit(0 if Decimal(sys.argv[1])>0 else 1)' "$bal" 2>/dev/null; then - printf '%s' "$bal" - return 0 - fi - info pay "same wallet as withdraw (db=$(basename "${WDB:-wallet}")) · available=${CUR}:${bal} · pendingIncoming=${CUR}:${pend}" - info pay "waiting up to ${wait_s}s for spendable available>0 — board lists OK/OK_BANK withdraws in start order" - # initial board before loop - wcli transactions >"$SCRATCH/tx-wait-pay.out" 2>&1 || true - ladder_print_wd_board "$wait_s" "$bal" "$pend" - while python3 -c 'import sys; from decimal import Decimal; sys.exit(0 if Decimal(sys.argv[1])<=0 else 1)' "$bal" 2>/dev/null; do - now=$(date +%s) - left=$(( end - now )) - if [ "$left" -le 0 ]; then - wcli transactions >"$SCRATCH/tx-wait-pay.out" 2>&1 || true - ladder_print_wd_board "0" "$bal" "$pend" - info pay "timer 0s left — stop waiting (available still ${CUR}:0)" - break - fi - # nudge wallet (same DB); do not run-until-done (can hang) - wcli transactions >"$SCRATCH/tx-wait-pay.out" 2>&1 || true - if command -v timeout >/dev/null 2>&1 || command -v gtimeout >/dev/null 2>&1; then - local _to - _to=$(command -v gtimeout 2>/dev/null || command -v timeout) - "$_to" 8 wcli advanced run-pending >"$SCRATCH/run-pending.out" 2>&1 || true - fi - # progress board every ~5s (tick every 2s → every 3rd) - if [ $((tick % 3)) -eq 0 ]; then - ladder_print_wd_board "$left" "$bal" "$pend" - fi - sleep 2 - tick=$((tick + 1)) - bal=$(wallet_avail) - pend=$(wallet_pending_in 2>/dev/null || echo "0") - done - if python3 -c 'import sys; from decimal import Decimal; sys.exit(0 if Decimal(sys.argv[1])>0 else 1)' "$bal" 2>/dev/null; then - wcli transactions >"$SCRATCH/tx-wait-pay.out" 2>&1 || true - ladder_print_wd_board "ready" "$bal" "$pend" - ok pay "spendable coins ready available=${CUR}:${bal} · $(format_amount_alt "${CUR}:${bal}") (same wallet)" - else - warn pay "no spendable coins after ${wait_s}s" \ - "problem: available still ${CUR}:0 (pendingIncoming=${CUR}:${pend}). Board above shows which withdraws still pendingIncoming. Pay will SKIP_BALANCE / skip max-pay until wirewatch+wallet finish. Same wallet as withdraw." - fi - printf '%s' "$bal" -} - -# Classic planned pays (PAY_LIST) -ladder_run_pay_classic() { - # shellcheck disable=SC2086 - set -- $PAY_LIST - local pay_n=$# prung=0 PAMT prange PNUM - if [ "$pay_n" -eq 0 ]; then - info pay "empty pay plan — skip" - return 0 - fi - info "pay rungs" "$pay_n · $*" - for PAMT in "$@"; do - prung=$((prung + 1)) - ladder_over && break - if [ "$prung" -eq 1 ]; then - prange="pin:0" - elif [ "$prung" -eq "$pay_n" ]; then - prange="pin:max" - elif [ "$prung" -eq $((pay_n - 1)) ] && [ "$pay_n" -ge 3 ]; then - prange="pin:max-1" - else - prange="random" - fi - section "ladder · pay rung $prung $PAMT · $(format_amount_alt "$PAMT")" - PNUM=$(python3 -c 'import sys; print(sys.argv[1].split(":",1)[-1])' "$PAMT") - if python3 -c 'import sys; from decimal import Decimal; sys.exit(0 if Decimal(sys.argv[1])==0 else 1)' "$PNUM" 2>/dev/null; then - echo -e "${prung}\t${prange}\t${PAMT}\tZERO_SKIP\t0\t0\t0\t0\t-\tzero" >>"$PAY_TSV" - PAY_OK_N=$((PAY_OK_N + 1)) - continue - fi - ladder_pay_attempt "$PAMT" "$prange" "$prung" - done -} - -# Max-search: highest payable ≤ available (uses max_search_next from withdraw module) -ladder_run_pay_max_search() { - local bal prung _next _anum prange _act _last _slo _shi - section "ladder · phase B · max-search pay (highest payable ≤ available)" - bal=$(ladder_wait_available "$LADDER_PAY_WAIT_AVAILABLE_S") - MAX_PAY_LO="0" - MAX_PAY_HI="$bal" - MAX_PAY_BEST_AMT="" - if ! python3 -c 'import sys; from decimal import Decimal; sys.exit(0 if Decimal(sys.argv[1])>0 else 1)' "$MAX_PAY_HI" 2>/dev/null; then - warn pay "max-search pay skipped" \ - "problem: wallet available is ${CUR}:0 — need spendable coins (pendingIncoming is not enough)" - return 0 - fi - info "max-search pay" "hi=available $(format_amount_alt "${CUR}:${MAX_PAY_HI}") · template=${LADDER_FREE_TEMPLATE}@${LADDER_FREE_TEMPLATE_INSTANCE}" - info "pay budget" "never order more than $(format_amount_alt "${CUR}:${MAX_PAY_HI}")" - prung=0 - _last="" - while [ "${LADDER_MAX_PROBES}" = "0" ] || [ "$prung" -lt "${LADDER_MAX_PROBES}" ]; do - ladder_over && break - # reuse withdraw amount picker with pay bounds - _slo=${MAX_LO:-0}; _shi=${MAX_HI:-0} - MAX_LO=$MAX_PAY_LO - MAX_HI=$MAX_PAY_HI - if ! type max_search_next >/dev/null 2>&1; then - warn pay "max_search_next missing — cannot run pay max-search" - MAX_LO=$_slo; MAX_HI=$_shi - return 0 - fi - _next=$(max_search_next) - MAX_LO=$_slo; MAX_HI=$_shi - _anum=$(printf '%s' "$_next" | cut -d'|' -f1) - prange=$(printf '%s' "$_next" | cut -d'|' -f2 | sed 's/^max:/pay:/') - _act=$(printf '%s' "$_next" | cut -d'|' -f3) - if [ "$_act" = "done" ]; then - info "max-search pay" "converged best=$(format_amount_alt "${MAX_PAY_BEST_AMT:-${CUR}:0}") · lo=$(format_amount_alt "${CUR}:${MAX_PAY_LO}") · hi=$(format_amount_alt "${CUR}:${MAX_PAY_HI}")" - break - fi - if ! python3 -c 'import sys; from decimal import Decimal; sys.exit(0 if Decimal(sys.argv[1])<=Decimal(sys.argv[2]) else 1)' "$_anum" "$MAX_PAY_HI" 2>/dev/null; then - _anum=$MAX_PAY_HI - fi - if ! python3 -c 'import sys; from decimal import Decimal; sys.exit(0 if Decimal(sys.argv[1])>0 else 1)' "$_anum" 2>/dev/null; then - break - fi - PAMT="${CUR}:${_anum}" - if [ -n "$_last" ] && [ "$PAMT" = "$_last" ]; then - MAX_PAY_HI=$(python3 -c 'from decimal import Decimal; a=Decimal("'"$_anum"'"); print(max(Decimal(0), a-(1 if a>=1 else Decimal("0.000001"))))') - continue - fi - _last=$PAMT - prung=$((prung + 1)) - section "ladder · max pay probe $prung $PAMT · $(format_amount_alt "$PAMT") [lo=$(format_amount_alt "${CUR}:${MAX_PAY_LO}") · hi=$(format_amount_alt "${CUR}:${MAX_PAY_HI}")]" - ladder_pay_attempt "$PAMT" "$prange" "$prung" - _last_st=$(tail -1 "$PAY_TSV" 2>/dev/null | cut -f4) - case "$_last_st" in - OK) - MAX_PAY_LO=$_anum - MAX_PAY_BEST_AMT=$PAMT - info "max-search pay" "success bound lo=$(format_amount_alt "${CUR}:${MAX_PAY_LO}")" - ;; - SKIP_BALANCE|FAIL_ORDER|FAIL_HANDLE|FAIL_SETTLE) - MAX_PAY_HI=$(python3 -c 'from decimal import Decimal; a=Decimal("'"$_anum"'"); h=Decimal("'"$MAX_PAY_HI"'"); print(min(a,h))') - info "max-search pay" "too-high/fail hi:=$(format_amount_alt "${CUR}:${MAX_PAY_HI}") status=${_last_st}" - ;; - esac - if python3 -c 'from decimal import Decimal; import sys; sys.exit(0 if Decimal(sys.argv[1])>0 and Decimal(sys.argv[2])<=Decimal(sys.argv[1]) else 1)' "$MAX_PAY_LO" "$MAX_PAY_HI" 2>/dev/null; then - info "max-search pay" "bounds crossed — stop" - break - fi - done - if [ -n "${MAX_PAY_BEST_AMT:-}" ]; then - ok "max-search best payable $(format_amount_alt "$MAX_PAY_BEST_AMT") · lo=$(format_amount_alt "${CUR}:${MAX_PAY_LO}") · hi=$(format_amount_alt "${CUR}:${MAX_PAY_HI}")" - else - warn pay "max-search no successful pay" "available was ${CUR}:$(wallet_avail)" - fi - { - echo "best_pay=${MAX_PAY_BEST_AMT:-}" - echo "best_pay_alt=$(format_amount_alt "${MAX_PAY_BEST_AMT:-${CUR}:0}" 2>/dev/null || true)" - echo "pay_lo=${MAX_PAY_LO}" - echo "pay_hi=${MAX_PAY_HI}" - } >>"${SCRATCH}/ladder-plan.txt" 2>/dev/null || true - printf '%s\n' "${MAX_PAY_BEST_AMT:-}" >"${SCRATCH}/ladder-pay-plan.txt" 2>/dev/null || true -} - -# Entry: glue from check_amount_ladder.sh after withdraw phase -ladder_run_pay_phase() { - PAY_OK_N=0 - PAY_FAIL_N=0 - if [ "${LADDER_PAY}" != "1" ]; then - info pay "LADDER_PAY=0 — pay phase skipped" - return 0 - fi - if [ "${FAIL_N_L:-0}" -gt 0 ]; then - warn pay "skipped pay ladder" "withdraw phase already failed" - return 0 - fi - - set_group pay - section "ladder · phase B · pay" - metrics_report_coins "before-pay-ladder" || true - ladder_pay_resolve_merchant - - if [ -z "${AUTH:-}" ] && ! { [ "${LADDER_USE_FREE_TEMPLATE}" = "1" ] && [ -n "${LADDER_FREE_TEMPLATE:-}" ]; }; then - warn pay "no merchant token and free template off — skip pay" - return 0 - fi - if [ -n "${AUTH:-}" ]; then - ok "merchant token" "instance ${INST}" - else - info pay "public free template ${LADDER_FREE_TEMPLATE}@${LADDER_FREE_TEMPLATE_INSTANCE} (no merchant token)" - fi - - # Same cumulative wallet as withdraw (WDB=wallet-main.sqlite3) - wallet_prepare "main" - info pay "using same wallet as withdraw · WDB=${WDB:-?}" - local bal0 - bal0=$(ladder_wait_available "$LADDER_PAY_WAIT_AVAILABLE_S") - info "pay budget" "wallet available ${CUR}:${bal0} · $(format_amount_alt "${CUR}:${bal0}") — never order more than this" - - if [ "${LADDER_MODE}" = "max" ]; then - ladder_run_pay_max_search - else - ladder_run_pay_classic - fi - metrics_report_coins "after-pay-ladder" || true - info "pay summary" "ok=$PAY_OK_N fail=$PAY_FAIL_N tsv=$PAY_TSV" -} diff --git a/lib.sh b/lib.sh index 9ced76b..0f92126 100755 --- a/lib.sh +++ b/lib.sh @@ -1512,20 +1512,12 @@ find_wallet_cli() { echo "$WALLET_CLI"; return 0 fi local c cand - # WALLET_CLI / WALLET_CLI_CANDIDATES from env; then portable package paths. - # Prefer local ts-core / mon-env builds (schema-compatible) over stale system - # wrappers (e.g. /usr/local 1.3.x vs DB schema 29 from 1.6.x). + # WALLET_CLI / WALLET_CLI_CANDIDATES from env; then portable package paths only. # shellcheck disable=SC2086 for c in ${WALLET_CLI_CANDIDATES:-} \ "${HOME}/.local/bin/taler-wallet-cli.mjs" \ - "${HOME}/src/taler/taler-typescript-core/packages/taler-wallet-cli/bin/taler-wallet-cli.mjs" \ - "${HOME}/src/taler/taler-typescript-core/packages/taler-wallet-cli/dist/taler-wallet-cli.mjs" \ - "${HOME}/src/taler/taler-typescript-core/packages/taler-wallet-cli/dist/taler-wallet-cli-local.mjs" \ - "${HOME}/src/taler/taler-typescript-core/packages/taler-wallet-cli/dist/taler-wallet-cli-bundled.cjs" \ /usr/lib/taler-wallet-cli/node_modules/taler-wallet-cli/bin/taler-wallet-cli.mjs \ - /usr/share/taler-wallet-cli/bin/taler-wallet-cli.mjs \ - /usr/local/lib/taler-wallet-cli/node_modules/taler-wallet-cli/bin/taler-wallet-cli.mjs \ - /usr/local/lib/taler-wallet-cli/node_modules/taler-wallet-cli/dist/taler-wallet-cli-bundled.cjs + /usr/share/taler-wallet-cli/bin/taler-wallet-cli.mjs do [ -n "$c" ] && [ -f "$c" ] && { echo "$c"; return 0; } done @@ -1541,14 +1533,6 @@ find_wallet_cli() { c=$(readlink -f "$cand" 2>/dev/null || true) [ -n "$c" ] && [ -f "$c" ] && { echo "$c"; return 0; } fi - # macOS /usr/local wrapper often runs an older bundle; prefer its bundled.cjs only - # when no newer candidate above matched (already returned). - for c in \ - /usr/local/lib/taler-wallet-cli/node_modules/taler-wallet-cli/dist/taler-wallet-cli-bundled.cjs \ - /usr/local/lib/node_modules/taler-wallet-cli/dist/taler-wallet-cli-bundled.cjs - do - [ -f "$c" ] && { echo "$c"; return 0; } - done fi return 1 } diff --git a/metrics.sh b/metrics.sh index 5e6ac42..aac0630 100644 --- a/metrics.sh +++ b/metrics.sh @@ -70,15 +70,11 @@ path = sys.argv[2] if len(sys.argv) > 2 else "" alt = {} if path: try: - _loaded = json.load(open(path)) - if isinstance(_loaded, dict): - alt = {str(k): str(v) for k, v in _loaded.items()} + alt = json.load(open(path)) except Exception: alt = {} -if not isinstance(alt, dict) or not alt: +if not alt: alt = {"0": "GOA"} -elif "0" not in alt: - alt["0"] = "GOA" def parse(s): if ":" in s: diff --git a/site-gen/caddy-monitoring-handles.snippet b/site-gen/caddy-monitoring-handles.snippet index 1fb1e1b..5689fff 100644 --- a/site-gen/caddy-monitoring-handles.snippet +++ b/site-gen/caddy-monitoring-handles.snippet @@ -5,8 +5,6 @@ # # CORRECT (site-level named matcher — preferred): # @mon_bare path /monitoring /monitoring_err \ -# /monitoring-ladder /monitoring-ladder_err \ -# /monitoring-max-ladder /monitoring-max-ladder_err \ # /taler-monitoring-surface /taler-monitoring-surface_err \ # /taler-monitoring-aptdeploy /taler-monitoring-aptdeploy_err # redir @mon_bare {path}/ 302 @@ -37,8 +35,6 @@ # --- only inside taler.hacktivism.ch { ... } --- @mon_bare path /monitoring /monitoring_err \ - /monitoring-ladder /monitoring-ladder_err \ - /monitoring-max-ladder /monitoring-max-ladder_err \ /taler-monitoring-surface /taler-monitoring-surface_err \ /taler-monitoring-aptdeploy /taler-monitoring-aptdeploy_err \ /taler-monitoring-mail /taler-monitoring-mail_err \ @@ -63,10 +59,7 @@ } # --- bank + exchange + taler (each site block; set root host dir) --- -# /monitoring* already covers /monitoring-ladder* and /monitoring-max-ladder* - @mon_bare path /monitoring /monitoring_err \ - /monitoring-ladder /monitoring-ladder_err \ - /monitoring-max-ladder /monitoring-max-ladder_err + @mon_bare path /monitoring /monitoring_err redir @mon_bare {path}/ 302 handle /monitoring_err* { diff --git a/site-gen/console_to_html.py b/site-gen/console_to_html.py index a2c4af1..d49c839 100755 --- a/site-gen/console_to_html.py +++ b/site-gen/console_to_html.py @@ -281,11 +281,11 @@ def ui(lang: str, key: str, **kwargs) -> str: "mode_redirect": "REDIR", "pages_title": "Monitoring pages (suite HTML)", "pages_summary": "Public sticky-bar reports · phase monpages · part of taler-monitoring", - "pages_i1": "Suite name taler-monitoring ≠ URL. Public families (bare + slash each):", - "pages_i2": "1) /monitoring(+_err) · 2) /monitoring-ladder(+_err) daily classic · 3) /monitoring-max-ladder(+_err) daily max-search · 4) /taler-monitoring-surface · 5) /taler-monitoring-aptdeploy (surface/aptdeploy on taler.hacktivism.ch)", - "pages_i3": "No /taler-monitoring page; no /taler-monitoring-mail|mattermost (folded into surface). Ladder pages on GOA bank/exchange/taler landings.", + "pages_i1": "Suite name taler-monitoring ≠ URL. Only three public families (bare + slash each):", + "pages_i2": "1) /monitoring(+_err) on landing hosts · 2) /taler-monitoring-surface(+_err) · 3) /taler-monitoring-aptdeploy(+_err) on taler.hacktivism.ch", + "pages_i3": "No /taler-monitoring page; no /taler-monitoring-mail|mattermost (folded into surface)", "pages_i4": "monpages: existence + content markers; bare → slash redir (not merchant code 21)", - "pages_i5": "Host-agent: staging → DEPLOY_WWW_ROOT; Caddy /monitoring* also serves ladder and max-ladder paths", + "pages_i5": "Host-agent: staging → DEPLOY_WWW_ROOT; reverse-proxy handles only those three families", "pages_i6": "This page host: {host} · path label: {label}", "stage_title": "mytops stage monitoring · {host}", "stage_summary": "betel only · CHF stage stack · not lifeline/prod · not GOA", @@ -391,11 +391,11 @@ def ui(lang: str, key: str, **kwargs) -> str: "mode_redirect": "REDIR", "pages_title": "Pages de monitoring (HTML de la suite)", "pages_summary": "Rapports publics sticky-bar · phase monpages · partie de taler-monitoring", - "pages_i1": "Nom de suite taler-monitoring ≠ URL. Familles publiques (sans et avec /) :", - "pages_i2": "1) /monitoring(+_err) · 2) /monitoring-ladder(+_err) quotidien classic · 3) /monitoring-max-ladder(+_err) max-search · 4) surface · 5) aptdeploy (surface/aptdeploy sur taler.hacktivism.ch)", - "pages_i3": "Pas de page /taler-monitoring ; pas de /taler-monitoring-mail|mattermost (dans surface). Ladder sur landings GOA bank/exchange/taler.", + "pages_i1": "Nom de suite taler-monitoring ≠ URL. Trois familles publiques seulement (sans et avec /) :", + "pages_i2": "1) /monitoring(+_err) · 2) /taler-monitoring-surface(+_err) · 3) /taler-monitoring-aptdeploy(+_err) sur taler.hacktivism.ch", + "pages_i3": "Pas de page /taler-monitoring ; pas de /taler-monitoring-mail|mattermost (dans surface)", "pages_i4": "monpages : existence + marqueurs ; bare → slash (pas code 21 merchant)", - "pages_i5": "Host-agent : staging → DEPLOY_WWW_ROOT ; Caddy /monitoring* sert aussi ladder et max-ladder", + "pages_i5": "Host-agent : staging → DEPLOY_WWW_ROOT ; reverse-proxy seulement ces trois familles", "stage_title": "Surveillance mytops stage · {host}", "stage_summary": "betel uniquement · pile CHF stage · pas lifeline/prod · pas GOA", "stage_i1": "Hôte : betel (stage). HTML mon : stage.my.taler-ops.ch + stage.taler-ops.ch /monitoring/", diff --git a/taler-monitoring.sh b/taler-monitoring.sh index 5379f3f..35c2a99 100755 --- a/taler-monitoring.sh +++ b/taler-monitoring.sh @@ -93,9 +93,7 @@ Phases: sanity public + optional server server server-side only (SSH) e2e withdraw + pay (small amounts; remote aborts on login/KYC) - ladder amount ladder classic (default 23-rung withdraw + pay; any currency) - max-ladder max-search: highest withdrawable + highest payable (≤ available) - goa-ladder alias of ladder (legacy phase name; same check_amount_ladder.sh) + ladder withdraw/pay amount ladder (GOA ceiling or stage TESTPAYSAN max_wire) auth401 merchant Basic-auth / case matrix (HTTP 401 paths; may create throwaway instance) aptdeploy koopa podman apt-src smoke: taler-merchant in koopa-taler-deploy-test-apt-src-trixie{,-testing} @@ -267,7 +265,7 @@ while [ $# -gt 0 ]; do CURRENCY_OVERRIDE="$2"; shift 2 ;; --no-probe) NO_PROBE=1; shift ;; - urls|inside|versions|sanity|server|e2e|ladder|goa-ladder|max-ladder|maxladder|auth401|aptdeploy|apt-deploy|apt_src|surface|ecosystem|mattermost|mail|monpages|pages|devtesting|franken|fake-franken|fake_franken|all|full) PHASES+=("$1"); shift ;; + urls|inside|versions|sanity|server|e2e|ladder|goa-ladder|auth401|aptdeploy|apt-deploy|apt_src|surface|ecosystem|mattermost|mail|monpages|pages|devtesting|franken|fake-franken|fake_franken|all|full) PHASES+=("$1"); shift ;; *) # bare domain shorthand: ./taler-monitoring.sh taler.net if [[ "$1" == *.* && "$1" != *://* && "$1" != -* ]]; then @@ -318,8 +316,7 @@ export INSIDE_PROFILE INSIDE_SSH export INSIDE_BANK_CTR INSIDE_EXCHANGE_CTR INSIDE_MERCHANT_CTR export INSIDE_BANK_PORT INSIDE_EXCHANGE_PORT INSIDE_MERCHANT_PORT export INSIDE_DNS_BANK INSIDE_DNS_EXCHANGE INSIDE_DNS_MERCHANT -# Amount ladder — check_amount_ladder.sh + ladder/lib_pay.sh + ladder/extract_rpubs.py -# Phases: ladder|goa-ladder (classic), max-ladder|maxladder (max-search). +# Ladder: withdraw then pay — 0 + random mids + max-1 + max (see check_goa_ladder.sh). # Defaults so set -u export is safe when vars were never set by caller. : "${LADDER_STEPS:=23}" : "${LADDER_MIN_AMOUNT:=0.000001}" @@ -333,23 +330,16 @@ export INSIDE_DNS_BANK INSIDE_DNS_EXCHANGE INSIDE_DNS_MERCHANT : "${LADDER_INCLUDE_ZERO:=1}" : "${LADDER_INCLUDE_MAX:=1}" : "${LADDER_HIGH_FROM:=1000000}" -: "${LADDER_HIGH_RUNGS:=4}" +: "${LADDER_HIGH_RUNGS:=12}" : "${LADDER_LOAD:=1}" : "${LADDER_PAY:=1}" : "${LADDER_WITHDRAW_SCALE:=1.5}" : "${LADDER_PAY_SETTLE_ROUNDS:=6}" -: "${LADDER_MODE:=classic}" -: "${LADDER_MAX_PROBES:=32}" -: "${LADDER_MAX_TOL_REL:=0.001}" -: "${LADDER_MAX_SEED:=}" -: "${LADDER_PAY_WITH_MAX:=0}" -: "${LADDER_HIGH_RUNGS:=6}" export LADDER_STEPS LADDER_MIN_AMOUNT LADDER_CONFIRM_POLLS LADDER_MAX_RUNGS LADDER_TIMEOUT_S LADDER_REPORT_DIR export LADDER_SETTLE_ROUNDS LADDER_SETTLE_SLEEP EXP_PW_FILE EXP_USER export LADDER_MAX_AMOUNT LADDER_INCLUDE_ZERO LADDER_INCLUDE_MAX export LADDER_HIGH_FROM LADDER_HIGH_RUNGS LADDER_LOAD export LADDER_PAY LADDER_WITHDRAW_SCALE LADDER_PAY_SETTLE_ROUNDS -export LADDER_MODE LADDER_MAX_PROBES LADDER_MAX_TOL_REL LADDER_MAX_SEED LADDER_PAY_WITH_MAX export TALER_DOMAIN_APPLIED=1 i18n_init 2>/dev/null || true # Optional: auth401 and other phases may honor CONTINUE_ON_ERROR @@ -445,7 +435,7 @@ _progress_estimate_phase() { server) n=20 ;; # e2e: bank withdraw ladder emits many INFO lines (coins before/after each note) e2e) n=240 ;; - ladder|goa-ladder|max-ladder|maxladder) n=120 ;; + ladder|goa-ladder) n=120 ;; auth401) n=70 ;; aptdeploy) n=40 ;; surface|ecosystem) n=120 ;; @@ -621,15 +611,7 @@ while [ "$_phase_idx" -lt "$_n_phases" ]; do sanity) run_phase sanity "$ROOT/check_sanity.sh" || ec=1 ;; server) run_phase server "$ROOT/check_server.sh" || ec=1 ;; e2e) run_phase e2e "$ROOT/check_e2e.sh" || ec=1 ;; - ladder|goa-ladder) - run_phase ladder "$ROOT/check_amount_ladder.sh" || ec=1 - ;; - max-ladder|maxladder) - # Feature: hunt highest mint+confirm withdrawable amount (any currency) - LADDER_MODE=max - export LADDER_MODE - run_phase ladder "$ROOT/check_amount_ladder.sh" || ec=1 - ;; + ladder|goa-ladder) run_phase ladder "$ROOT/check_goa_ladder.sh" || ec=1 ;; auth401) run_phase auth401 "$ROOT/check_auth401.sh" || ec=1 ;; aptdeploy|apt-deploy|apt_src) run_phase aptdeploy "$ROOT/check_apt_deploy.sh" || ec=1 ;; surface|ecosystem) run_phase surface "$ROOT/check_surface.sh" || ec=1 ;;