diff --git a/CLI-AUTOMATION-NOTES.md b/CLI-AUTOMATION-NOTES.md index 057cef2..a541d28 100644 --- a/CLI-AUTOMATION-NOTES.md +++ b/CLI-AUTOMATION-NOTES.md @@ -9,7 +9,7 @@ Companions: - Android GUI → [`android-test/GUI-AUTOMATION-NOTES.md`](./android-test/GUI-AUTOMATION-NOTES.md) - Git / suite tree → [`VERSIONS.md`](./VERSIONS.md) -Scripts: `check_e2e.sh`, `check_goa_ladder.sh`, `taler-monitoring.sh`. +Scripts: `check_e2e.sh`, `check_amount_ladder.sh` (compat `check_goa_ladder.sh`), `taler-monitoring.sh`. Legend: @@ -67,6 +67,16 @@ Legend: | **Workaround** | Use `LADDER_STEPS≥5` (or higher) so random mids exist; treat zero / absolute-max / max-1-5110 as probes (soft `CEILING_REJECT` in mon ≥1.20.2). | | **Wanted** | Ladder plan docs: pin vs mid semantics; CLI structured error for amount-too-large / zero. | +### 3b. Higher withdraw amounts often fail at mint + +| | | +|--|--| +| **Seen** | Classic GOA ladder high/random mids and ceiling pins — e.g. `GOA:747827072844319` → `FAIL_MINT` / `STOPPED … mint HTTP 500 { code :5110, … P0001 }`; same class of error on max / max-1. | +| **Area** | **bank** / **ops** | +| **Problem** | **Larger amounts** frequently hit bank/libeufin limits (HTTP 500, Taler **5110**, SQL **P0001**). Not specific to pin:max only — mid-high ladder steps fail the same way. Do not treat as wallet/`reserve_pub` when mint never succeeded. | +| **Workaround** | Expect failures above some live ceiling; use max-search / soft ceiling handling; keep mid-range rungs for real withdraw/confirm tests. | +| **Wanted** | Clear amount-too-large from bank (not opaque SQL 500); mon reports ceiling separately from hard fails. | + --- ## 4. Exchange not auto-known for bank deep links / withdraw URIs @@ -277,6 +287,8 @@ If only a few changes land, these unlock the most automation: | ladder explorer | `read_secret` / EXP_PW_FILE | | stage maxima | bank `max_wire` + keys min denom | | force-select | multi-rpub try; soft-skip confirm; empty scrape → WARN not balance fail | +| higher amounts | mint 5110/P0001 common above live ceiling (§3b); classic soft `CEILING_REJECT`; max-ladder bounds hi; report uses alt_unit_names (Tera-GOA …) | +| pay vs balance | never order more than wallet **available** (soft `SKIP_BALANCE`); pendingIncoming does not count as spendable | | wallet-db path | Prefer `*.sqlite3` (never `*.json` as db path) | | helper / python | mon PATH must include helper + python≥3.11 | diff --git a/README.md b/README.md index 998c65b..d8cf093 100644 --- a/README.md +++ b/README.md @@ -274,12 +274,14 @@ Other domains: never SSH. Optional **e2e** aborts cleanly on login/KYC. ```bash ./taler-monitoring.sh e2e ./taler-monitoring.sh -d taler.net urls e2e -# Amount ladder (withdraw 0 → … → max, then pay): same script for GOA + stage -./taler-monitoring.sh ladder # GOA / current domain +# Amount ladder (any currency): classic 0 → … → max, then optional pay +./taler-monitoring.sh ladder # classic (current domain) ./taler-monitoring.sh -d stage.lefrancpaysan.ch ladder # TESTPAYSAN maxima +# Max-search: start high, narrow toward highest withdrawable +./taler-monitoring.sh max-ladder # Stage auto-reads bank max_wire_transfer_amount (e.g. 2000) and min denom (0.01). # Explorer password: francpaysan-stage-user …/bank-explorer-password.txt (or EXP_PW=). -# Override: LADDER_MAX_AMOUNT=100 LADDER_STEPS=7 LADDER_PAY=0 … +# Override: LADDER_MODE=max LADDER_MAX_PROBES=16 LADDER_MAX_AMOUNT=100 LADDER_STEPS=7 LADDER_PAY=0 … # customize e2e: E2E_WITHDRAW_VALUES="20 50 100" E2E_PAY_VALUES="0.05 1 5" ./taler-monitoring.sh e2e E2E_VARIABLE=0 WITHDRAW_AMT=GOA:50 PAY_AMT=GOA:1 ./taler-monitoring.sh e2e # single fixed @@ -304,7 +306,7 @@ E2E_VARIABLE=0 WITHDRAW_AMT=GOA:50 PAY_AMT=GOA:1 ./taler-monitoring.sh e2e # s | **versions** | `deb.taler.net` reachable (InRelease/Packages/pool `.deb`); containers can reach it + have apt source; installed Taler packages vs **trixie** | | **sanity** | bank · exchange · merchant sections (public + server) | | **e2e** | account → credit → withdraw → wallet → confirm → order → pay | -| **ladder** | GOA withdraw: fixed **0** + low bands + **random high ranges** + fixed **max** | +| **ladder** | amount ladder: **classic** (0 + mids + max) or **max-search** (`max-ladder` / `LADDER_MODE=max`) | ```bash # package suite (default trixie on deb.taler.net) @@ -394,8 +396,12 @@ LADDER_LOAD=0 ./taler-monitoring.sh ladder | `METRICS_LOAD` | `1` | `0` = skip all host/container load probes | | `LADDER_LOAD` | `1` | `0` = skip load in ladder only | | `LADDER_PAY` | `1` | `0` = skip payment ladder after withdraws | -| `LADDER_WITHDRAW_SCALE` | `1.5` | withdraw mids / pay mids ratio (funding headroom) | -| `LADDER_STEPS` | `23` | rungs for both withdraw and pay (0 + mids + max−1 + max) | +| `LADDER_MODE` | `classic` | default **classic** 23-rung; `max` = hunt highest withdrawable (`max-ladder`) | +| `LADDER_MAX_PROBES` | `32` | max-search probe budget; `0` = until converge / timeout | +| `LADDER_HIGH_RUNGS` | `6` | max-search: first N probes random-high, then bisect | +| `LADDER_MAX_TOL_REL` | `0.001` | max-search relative lo/hi gap to stop | +| `LADDER_WITHDRAW_SCALE` | `1.5` | classic: withdraw mids / pay mids ratio (funding headroom) | +| `LADDER_STEPS` | `23` | classic: rungs for withdraw and pay (0 + mids + max−1 + max) | | `METRICS_LOAD_SSH_TIMEOUT` | `90` | seconds for remote load python | ## Needs diff --git a/TESTS.md b/TESTS.md index 5c0f76a..0c89237 100644 --- a/TESTS.md +++ b/TESTS.md @@ -25,7 +25,7 @@ Every check line has a **global** run number and a **grouped** id: | **sanity** | `check_sanity.sh` | `bank` `exchange` `merchant` | | **server** | `check_server.sh` | (flat `server-NN` or host groups) | | **e2e** | `check_e2e.sh` | `prereq` `load` `bank` `wallet` `bankwd` `settle` `pay` `shop` `paivana` `dig` `report` | -| **ladder** | `check_goa_ladder.sh` | `plan` `load` `withdraw` `pay` `report` | +| **ladder** | `check_amount_ladder.sh` (alias `check_goa_ladder.sh`) | `plan` `load` `withdraw` `pay` `report` — modes classic / max (`max-ladder`) | **Why groups:** flat `www-001`…`www-080` was hard to talk about. `www.bank-04` / `e2e.bankwd-02` pin the failure to a logical block. diff --git a/VERSION b/VERSION index 769e37e..3500250 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.20.2 +1.21.0 diff --git a/VERSIONS.md b/VERSIONS.md index ec8e050..f8d32cd 100644 --- a/VERSIONS.md +++ b/VERSIONS.md @@ -17,8 +17,8 @@ Git tags: `vMAJOR.FEATURE.FIX` (e.g. `v1.8.0`). File `VERSION` omits the `v` pre | Tag | Date (UTC) | Notes | |-----|------------|--------| -| **v1.20.2** | 2026-07-19 | **Bugfix:** ladder force-select empty `reserve_pub` scrape — wallet dumps mix log lines + JSON so `json.loads(rest_of_file)` always failed; extract now uses `JSONDecoder.raw_decode` and regex on **both** accept + transactions (`reservePub` is in tx). Prefer Homebrew python on macOS for helper; `find_wallet_cli` prefers ts-core 1.6.x over stale system wrappers. Soft-skip max-1 mint 5110/P0001 as ceiling. Verified: mid rungs `force-select HTTP 200` + `confirm 204` → `OK_BANK`. | -| **v1.20.1** | 2026-07-19 | **Docs:** CLI-AUTOMATION-NOTES — force-select empty scrape + ladder pins (§3/3a); wallet-db must not be `*.json` (§15); `taler-helper-sqlite3` needs Python ≥3.11 (§16); portable wall-clock (§17); GOA withdraw checklist (§18). No code change. | +| **v1.21.0** | 2026-07-19 | **Feature:** generic **amount ladder** (`check_amount_ladder.sh`; compat shim `check_goa_ladder.sh`). Default **classic** 23-rung withdraw (+ pay when enabled); **max-search** (`max-ladder` / `LADDER_MODE=max`) hunts highest mintable amount (probes default 32, `0`=until timeout). **Pay:** never overspend — soft `SKIP_BALANCE` if amount > wallet available. **Alt units:** report/logs show human names (e.g. `8.15 Tera-GOA (GOA:…)`) + raw. **Fixes bundled:** external `ladder_extract_rpubs.py` (no stdin SyntaxError); force-select multi-rpub + raw_decode; 409 tries next rpub; soft `CEILING_REJECT` on mid/high mint 5110/P0001; report python never aborts ladder; helper PATH python≥3.11; wallet-db not `*.json`. CLI-AUTOMATION-NOTES §3/3a/3b/§15–18. | +| **v1.20.1** | 2026-07-19 | **Docs:** CLI-AUTOMATION-NOTES — force-select empty scrape + ladder pins (§3/3a); wallet-db must not be `*.json` (§15); `taler-helper-sqlite3` needs Python ≥3.11 (§16); portable wall-clock (§17); GOA withdraw checklist (§18). No code change. *(Superseded notes folded into **v1.21.0** feature release.)* | | **v1.20.0** | 2026-07-19 | **Feature:** domains.conf **`locale`** (l10n) — FP **`fr-CH`**, others **`de-CH`** (aliases ch-FR/ch-DE); UI **`lang=de`** supported (sticky + console tags) but **never** a stock profile default; env `TALER_MON_LOCALE` / `TALER_DOMAIN_LOCALE`. | | **v1.19.1** | 2026-07-19 | **Bugfix / docs:** DEPENDENCIES — e2e requires `taler-helper-sqlite3` on mon PATH + preflight; README domains.conf documents **`lang`** column. | | **v1.19.0** | 2026-07-19 | **Feature:** domains.conf **`lang`** column (en\|fr) is the global UI language default per stack; i18n + HTML read `TALER_DOMAIN_LANG` / conf (FP profiles `lang=fr`). | diff --git a/check_amount_ladder.sh b/check_amount_ladder.sh new file mode 100755 index 0000000..9c7e2c1 --- /dev/null +++ b/check_amount_ladder.sh @@ -0,0 +1,1729 @@ +#!/usr/bin/env bash +# check_amount_ladder.sh — generic withdraw/pay amount ladder (any Taler currency) +# +# Currency-agnostic bank landing ladder for GOA, TESTPAYSAN, KUDOS, … +# (formerly check_goa_ladder.sh — shim retained for compatibility). +# +# Flow (bank landings): +# 1) GET /intro/auto-account.json → personal *account-* (balance 0) +# 2) Mint pool withdrawals as explorer + confirm when selected +# 3) wallet-cli accept-uri only (no run-until-done) +# 4) bank confirm when selected; settle = balance + transfer_done +# +# Modes (LADDER_MODE): +# classic — random strictly increasing rungs + pins 0, max-1, max (default) +# max — free max-search: start randomly high, narrow toward highest +# amount that mint+accept+confirm (+ bank transfer_done) succeeds +# +# Phase A: withdraw. Phase B: pay (classic only by default; off in max mode). +# +# Stacks auto-tune via bank /config max_wire + exchange min denom when possible. +# +# Usage: +# ./taler-monitoring.sh ladder +# ./taler-monitoring.sh max-ladder # LADDER_MODE=max +# ./taler-monitoring.sh -d stage.lefrancpaysan.ch ladder +# LADDER_MODE=max LADDER_MAX_PROBES=16 ./taler-monitoring.sh ladder +# LADDER_MAX_AMOUNT=100 LADDER_STEPS=7 ./taler-monitoring.sh -d stage.lefrancpaysan.ch ladder +# +# Env: LADDER_MODE, LADDER_STEPS, LADDER_MAX_AMOUNT, LADDER_MIN_AMOUNT, +# LADDER_MAX_PROBES, LADDER_MAX_TOL_REL, LADDER_HIGH_RUNGS, LADDER_STACK_AUTO, +# LADDER_PAY, EXP_PW / EXP_PW_FILE, CLI_JS, MERCHANT_INSTANCE, … +set -euo pipefail +ROOT=$(cd "$(dirname "$0")" && pwd) +# shellcheck source=lib.sh +source "$ROOT/lib.sh" +# When invoked standalone (not via taler-monitoring.sh), still load secrets.env +load_monitoring_secrets_env 2>/dev/null || true + +# Area ladder.* — withdraw/pay amount ladder (any EXPECT_CURRENCY / stack) +# Groups: ladder.plan / ladder.load / ladder.withdraw / ladder.pay / ladder.report +set_area ladder +set_group plan +SECTION_T0=$(date +%s) +now_ms() { python3 -c 'import time; print(int(time.time()*1000))'; } +elapsed_ms() { + # elapsed_ms START_MS + python3 -c 'import sys; print(int(sys.argv[1]) - int(sys.argv[2]))' "$(now_ms)" "$1" +} + +: "${LADDER_TIMEOUT_S:=3600}" +: "${LADDER_SETTLE_ROUNDS:=18}" +: "${LADDER_SETTLE_SLEEP:=2}" +: "${EXP_USER:=explorer}" +# EXP_PW_FILE / EXP_PW optional overrides; otherwise read_secret + SECRETS_ROOT / stage SSH +: "${EXP_PW_FILE:=}" +: "${EXP_PW:=}" +: "${CLI_JS:=}" +# GOA libeufin amount ceiling (hacktivism). Stage auto-replaces via bank /config. +: "${LADDER_MAX_AMOUNT:=4503599627370496}" +: "${LADDER_STEPS:=23}" +: "${LADDER_LOAD:=1}" +: "${LADDER_PAY:=1}" +: "${LADDER_STACK_AUTO:=1}" +# Withdraw mids = pay mids × scale (so wallet can afford the pay ladder) +: "${LADDER_WITHDRAW_SCALE:=1.5}" +# Floor for random mids (must be ≥ smallest exchange coin; GOA min denom ≈ 0.000001) +: "${LADDER_MIN_AMOUNT:=0.000001}" +: "${LADDER_CONFIRM_POLLS:=40}" +: "${LADDER_PAY_SETTLE_ROUNDS:=6}" +: "${MERCHANT_INSTANCE:=goa-demo-cp4zqk}" +# Public variable template for stage pays (optional; fixed templates used when amount maps) +: "${LADDER_PAY_TEMPLATE:=}" +: "${LADDER_PAY_TEMPLATE_INSTANCE:=fermes-des-collines}" +# classic | max (max = find highest withdrawable; phase alias max-ladder) +: "${LADDER_MODE:=classic}" +# max-search: probe budget (0 = run until convergence / LADDER_TIMEOUT_S only) +: "${LADDER_MAX_PROBES:=32}" +: "${LADDER_MAX_TOL_REL:=0.001}" +# first N probes = random high (log-uniform); then log-bisect between lo/hi +: "${LADDER_HIGH_RUNGS:=6}" +# optional seed for reproducible max-search (empty = time-based) +: "${LADDER_MAX_SEED:=}" + +# Historic libeufin-ish absolute ceiling used as default LADDER_MAX_AMOUNT on GOA +LADDER_ABS_CEILING="4503599627370496" +GOA_LADDER_CEILING="${LADDER_ABS_CEILING}" # back-compat alias + +# Resolve wallet-cli .mjs (no hardcoded laptop path) +if [ -z "${CLI_JS}" ] || [ ! -f "${CLI_JS}" ]; then + CLI_JS=$(find_wallet_cli 2>/dev/null || true) +fi +if [ -z "${CLI_JS}" ] || [ ! -f "${CLI_JS}" ]; then + # allow PATH wrapper as last resort (wcli falls back to taler-wallet-cli) + CLI_JS="" +fi + +CUR="${EXPECT_CURRENCY:-GOA}" +BANK="${BANK_PUBLIC%/}" +EX="${EXCHANGE_PUBLIC%/}/" +MER="${MERCHANT_PUBLIC%/}" +INST="${MERCHANT_INSTANCE}" + +# --- Stack-specific ladder maxima (TESTPAYSAN stage) --- +# Uses bank max_wire_transfer_amount and exchange min denom when still on GOA defaults. +apply_ladder_stack_defaults() { + [ "${LADDER_STACK_AUTO}" = "1" ] || return 0 + if [ "${CUR}" != "TESTPAYSAN" ]; then + case "${TALER_DOMAIN:-}" in + stage.lefrancpaysan.ch|stage.bank.lefrancpaysan.ch|stage.exchange.lefrancpaysan.ch|stage.monnaie.lefrancpaysan.ch) + CUR="TESTPAYSAN" + ;; + *) return 0 ;; + esac + fi + + local bank_cfg max_wire min_denom + bank_cfg=$(curl -sS -m 12 "${BANK}/config" 2>/dev/null || true) + max_wire=$(printf '%s' "$bank_cfg" | python3 -c ' +import json,sys +try: + d=json.load(sys.stdin) +except Exception: + print("") + raise SystemExit(0) +a=d.get("max_wire_transfer_amount") or "" +print(a.split(":",1)[-1] if a else "") +' 2>/dev/null || true) + + min_denom=$(curl -sS -m 25 -H 'Accept: application/json' "${EX%/}/keys" 2>/dev/null | python3 -c ' +import json,sys +try: + d=json.load(sys.stdin) +except Exception: + print("") + raise SystemExit(0) +den=d.get("denoms") or d.get("denominations") or [] +vals=[] +for x in den if isinstance(den,list) else []: + v=x.get("value") or x.get("amount") or "" + if not v: continue + try: vals.append(float(str(v).split(":")[-1])) + except Exception: pass +print(min(vals) if vals else "") +' 2>/dev/null || true) + + # Fallback maxima if public config unreachable + [ -n "$max_wire" ] || max_wire="2000" + [ -n "$min_denom" ] || min_denom="0.01" + + # Only rewrite when caller left GOA defaults (explicit LADDER_MAX_AMOUNT=… kept) + if [ -n "$max_wire" ] && { [ "${LADDER_MAX_AMOUNT}" = "${GOA_LADDER_CEILING}" ] || [ "${LADDER_MAX_AMOUNT}" = "${LADDER_ABS_CEILING}" ] || [ "${LADDER_MAX_AMOUNT}" = "4503599627370496" ]; }; then + LADDER_MAX_AMOUNT="$max_wire" + fi + if [ -n "$min_denom" ] && { [ "${LADDER_MIN_AMOUNT}" = "0.000001" ] || [ "${LADDER_MIN_AMOUNT}" = "0.00000001" ]; }; then + LADDER_MIN_AMOUNT="$min_denom" + fi + # Slightly fewer rungs on stage (full GOA 23 still ok if user set LADDER_STEPS) + if [ "${LADDER_STEPS}" = "23" ]; then + LADDER_STEPS=15 + fi + # Stage farmer shops: private goa-demo instance does not exist + if [ "${MERCHANT_INSTANCE}" = "goa-demo-cp4zqk" ]; then + MERCHANT_INSTANCE="${LADDER_PAY_TEMPLATE_INSTANCE:-fermes-des-collines}" + INST="$MERCHANT_INSTANCE" + fi + # Prefer public templates for pays when no merchant token (set later) + : "${E2E_USE_TEMPLATES:=1}" + export E2E_USE_TEMPLATES + export LADDER_MAX_AMOUNT LADDER_MIN_AMOUNT LADDER_STEPS MERCHANT_INSTANCE + info "ladder stack" "TESTPAYSAN · max=${CUR}:${LADDER_MAX_AMOUNT} min=${CUR}:${LADDER_MIN_AMOUNT} steps=${LADDER_STEPS} (from bank max_wire / keys denoms)" +} +apply_ladder_stack_defaults + +# Normalize mode (max-ladder phase / aliases) +case "${LADDER_MODE}" in + max|MAX|max-search|maxsearch|find-max|findmax) LADDER_MODE=max ;; + classic|CLASSIC|default|plan|"" ) LADDER_MODE=classic ;; + *) + warn ladder "unknown LADDER_MODE=${LADDER_MODE} — using classic" + LADDER_MODE=classic + ;; +esac +# max mode: pay ladder off unless caller explicitly set LADDER_PAY=1 *and* LADDER_PAY_WITH_MAX=1 +if [ "$LADDER_MODE" = "max" ]; then + if [ "${LADDER_PAY_WITH_MAX:-0}" != "1" ]; then + LADDER_PAY=0 + fi +fi +export LADDER_MODE LADDER_MAX_PROBES LADDER_MAX_TOL_REL LADDER_HIGH_RUNGS LADDER_MAX_SEED LADDER_PAY + +SCRATCH=$(mktemp -d) +WDB="$SCRATCH/wallet.sqlite3" +REPORT_DIR="${LADDER_REPORT_DIR:-$SCRATCH}" +mkdir -p "$REPORT_DIR" +TSV="$REPORT_DIR/ladder-results.tsv" +PAY_TSV="$REPORT_DIR/ladder-pay-results.tsv" +JSON="$REPORT_DIR/ladder-report.json" +LOAD_BEFORE="$REPORT_DIR/load-before.json" +LOAD_AFTER="$REPORT_DIR/load-after.json" +echo -e "rung\trange\tamount\tstatus\tms_mint\tms_accept\tms_confirm\tms_settle\tms_total\twid\tnote" >"$TSV" +echo -e "rung\trange\tamount\tstatus\tms_order\tms_handle\tms_settle\tms_total\toid\tnote" >"$PAY_TSV" + +ladder_over() { + local now + now=$(date +%s) + [ $((now - SECTION_T0)) -ge "$LADDER_TIMEOUT_S" ] +} + +wcli() { + # taler-helper-sqlite3 is Python ≥3.11; prefer Homebrew/local python on macOS + # so Apple /usr/bin/python3 3.9 does not FATAL the sqlite backend (no reservePub). + local _path="${PATH:-}" + case ":${_path}:" in + *:/opt/homebrew/bin:*) ;; + *) _path="/opt/homebrew/bin:/usr/local/bin:${_path}" ;; + esac + if [ -n "${CLI_JS:-}" ] && [ -f "$CLI_JS" ]; then + PATH="$_path" node "$CLI_JS" --wallet-db="$WDB" --no-throttle "$@" + else + PATH="$_path" taler-wallet-cli --wallet-db="$WDB" --no-throttle "$@" + fi +} + +# Explorer pool password: env → file override → stack secrets → SSH +resolve_explorer_pw() { + local pw="" f="" host="" + if [ -n "${EXP_PW:-}" ]; then + printf '%s' "$EXP_PW" + return 0 + fi + if [ -n "${EXP_PW_FILE:-}" ] && [ -f "$EXP_PW_FILE" ]; then + tr -d '\n\r' <"$EXP_PW_FILE" + return 0 + fi + + # Stage TESTPAYSAN — never use GOA koopa-admin-secrets explorer pw first + if [ "${CUR}" = "TESTPAYSAN" ] \ + || [[ "${TALER_DOMAIN:-}" == stage.*lefrancpaysan* ]] \ + || [[ "${TALER_DOMAIN:-}" == *stage.lefrancpaysan* ]]; then + for f in \ + ${FRANCPAYSAN_SECRETS:+"${FRANCPAYSAN_SECRETS}/stage/bank-explorer-password.txt"} \ + "${HOME}/.config/taler-landing/stage-bank-explorer-password.txt" + do + [ -n "$f" ] && [ -f "$f" ] || continue + tr -d '\n\r' <"$f" + return 0 + done + _remote_exp="${FRANCPAYSAN_REMOTE_BANK_EXPLORER_SECRET:-}" + [ -z "$_remote_exp" ] && [ -n "${FRANCPAYSAN_REMOTE_SECRETS_ROOT:-}" ] && \ + _remote_exp="${FRANCPAYSAN_REMOTE_SECRETS_ROOT}/bank/secrets/bank-explorer-password.txt" + for host in ${INSIDE_SSH:+"$INSIDE_SSH"} ${FRANCPAYSAN_SSH:+"$FRANCPAYSAN_SSH"}; do + [ -n "$host" ] && [ -n "$_remote_exp" ] || continue + pw=$(ssh -o BatchMode=yes -o ConnectTimeout=12 "$host" \ + "tr -d '\\n\\r' <${_remote_exp} 2>/dev/null || sudo tr -d '\\n\\r' <${_remote_exp} 2>/dev/null" \ + 2>/dev/null || true) + if [ -n "$pw" ]; then + printf '%s' "$pw" + return 0 + fi + done + return 1 + fi + + # GOA / default: SECRETS_ROOT / ~/.config / koopa SSH + if pw=$(read_secret "taler-bank/bank-explorer-password.txt" 2>/dev/null) && [ -n "$pw" ]; then + printf '%s' "$pw" + return 0 + fi + for f in \ + "${SECRETS_ROOT:+${SECRETS_ROOT}/taler-bank/bank-explorer-password.txt}" \ + "${HOME}/.config/taler-landing/bank-explorer-password.txt" + do + [ -n "$f" ] && [ -f "$f" ] || continue + tr -d '\n\r' <"$f" + return 0 + done + return 1 +} + +wallet_avail() { + wcli balance 2>/dev/null | python3 -c ' +import json,sys +t=sys.stdin.read() +dec=json.JSONDecoder() +i=t.find("{") +if i<0: + print("0"); raise SystemExit +try: + d,_=dec.raw_decode(t,i) +except Exception: + # last-resort: brace slice (may fail on trailing logs) + try: + d=json.loads(t[i:t.rfind("}")+1]) + except Exception: + print("0"); raise SystemExit +cur=sys.argv[1] +for b in d.get("balances") or []: + a=b.get("available") or "" + if a.startswith(cur+":"): + print(a.split(":",1)[1]); raise SystemExit +print("0") +' "$CUR" 2>/dev/null || echo "0" +} + +# Build paired pay + withdraw ladders (same step count, shared random shape). +# Pay: [0] + log-uniform mids + [max-1] + [max] +# Wd: [0] + max(mid, mid×scale) mids + [max-1] + [max] (mids higher → enough to spend) +# Writes: $1 = withdraw list file, $2 = pay list file (space-separated CUR:amt lines as one line each) +build_ladder_pair() { + local wd_out="$1" pay_out="$2" + python3 - <<'PY' "$CUR" "${LADDER_MAX_AMOUNT}" "${LADDER_STEPS}" "${LADDER_MIN_AMOUNT}" "${LADDER_WITHDRAW_SCALE}" "$wd_out" "$pay_out" +import math, random, sys +from decimal import Decimal, ROUND_HALF_UP, ROUND_UP +from pathlib import Path + +cur = sys.argv[1] +max_amt = Decimal(sys.argv[2]) +steps = max(1, int(sys.argv[3])) +min_amt = Decimal(sys.argv[4]) +scale = Decimal(sys.argv[5]) +wd_path, pay_path = Path(sys.argv[6]), Path(sys.argv[7]) +if min_amt <= 0: + min_amt = Decimal("0.000001") +if min_amt >= max_amt: + min_amt = max_amt / Decimal(1000) +if scale < 1: + scale = Decimal(1) +max_m1 = max_amt - Decimal(1) if max_amt > 1 else max_amt + +def fmt(v: Decimal) -> str: + q = v.quantize(Decimal("0.00000001"), rounding=ROUND_HALF_UP) + if q == q.to_integral(): + return format(int(q), "d") + return format(q, "f").rstrip("0").rstrip(".") + +def quant(v: Decimal) -> Decimal: + if v >= 1: + return v.quantize(Decimal(1), rounding=ROUND_HALF_UP) + if v < min_amt: + return min_amt + return v.quantize(Decimal("0.00000001"), rounding=ROUND_UP) + +def amt(v: Decimal) -> str: + return "%s:%s" % (cur, fmt(v)) + +def make_mids(n_mid: int, hi_cap: Decimal): + if n_mid <= 0 or hi_cap <= min_amt: + return [] + lo, hi = float(min_amt), float(hi_cap) * 0.999999 + if hi <= lo: + hi = lo * 10 + cuts = sorted(math.exp(random.uniform(math.log(lo), math.log(hi))) for _ in range(n_mid)) + out, prev = [], Decimal(0) + for c in cuts: + v = quant(Decimal(str(c))) + if v <= prev: + step = min_amt if prev < 1 else max(prev * Decimal("1e-6"), Decimal(1)) + v = quant(prev + step) + if v >= hi_cap: + v = quant(hi_cap - (Decimal(1) if hi_cap > 1 else min_amt)) + if v <= prev or v >= hi_cap: + continue + out.append(v) + prev = v + return out + +# Build withdraw ladder first (full range), then pay mids = withdraw/scale +# so each pay mid is always cheaper than the matching withdraw mid. +if steps == 1: + wd_vals = [max_amt] +elif steps == 2: + wd_vals = [Decimal(0), max_amt] +else: + n_mid = max(0, steps - 3) + wd_vals = [Decimal(0)] + make_mids(n_mid, max_m1) + [max_m1, max_amt] + while len(wd_vals) > steps and len(wd_vals) > 3: + wd_vals.pop(len(wd_vals) // 2) + while len(wd_vals) < steps and len(wd_vals) >= 2: + i = max(1, len(wd_vals) - 2) + a, b = wd_vals[i - 1], wd_vals[i] + if a <= 0: + m = max(min_amt, b / 2 if b > 0 else min_amt) + else: + m = (a * b).sqrt() if a * b > 0 else (a + b) / 2 + m = quant(m) + if m <= a or m >= b: + break + wd_vals.insert(i, m) + wd_vals = wd_vals[:steps] + +pay_vals = [] +prev_p = Decimal(-1) +for i, w in enumerate(wd_vals): + is_first = i == 0 + is_last = i == len(wd_vals) - 1 + is_m1 = (not is_last) and w == max_m1 and i == len(wd_vals) - 2 + if is_first and w == 0: + pay_vals.append(Decimal(0)) + prev_p = Decimal(0) + continue + if is_last: + pay_vals.append(max_amt) + continue + if is_m1 or w == max_m1: + pay_vals.append(max_m1) + prev_p = max_m1 + continue + # pay mid = withdraw / scale (strictly less funding needed per step) + p = quant(w / scale) if scale > 0 else w + if p < min_amt and w >= min_amt: + p = min_amt + if p <= prev_p: + p = quant(prev_p + (min_amt if prev_p < 1 else Decimal(1))) + if p >= w: + # keep pay strictly below this withdraw rung when possible + p = quant(w - (Decimal(1) if w > 1 else min_amt)) if w > prev_p else prev_p + if p <= prev_p: + p = prev_p # flat ok only if stuck; still ≤ w + pay_vals.append(p) + prev_p = p + +assert len(pay_vals) == len(wd_vals) +# sanity: every non-pin pay mid ≤ matching withdraw mid +for i, (p, w) in enumerate(zip(pay_vals, wd_vals)): + if i == 0 or i >= len(wd_vals) - 2: + continue + if p > w: + pay_vals[i] = w + +wd_path.write_text(" ".join(amt(v) for v in wd_vals) + "\n") +pay_path.write_text(" ".join(amt(v) for v in pay_vals) + "\n") +print(" ".join(amt(v) for v in wd_vals)) +PY +} + +# shellcheck source=metrics.sh +source "$ROOT/metrics.sh" +METRICS_DIR="$REPORT_DIR" +ALT_UNITS_FILE="${REPORT_DIR}/alt_unit_names.json" +export METRICS_DIR CUR WDB CLI_JS ALT_UNITS_FILE +# Ladder can disable host load without killing coin metrics +if [ "${LADDER_LOAD:-1}" = "0" ]; then + METRICS_LOAD=0 + export METRICS_LOAD +fi + +if [ "${LADDER_MODE}" = "max" ]; then + section "ladder · ${CUR} max-search (find highest withdrawable)" +else + section "ladder · ${CUR} classic (0 → random → max · ${LADDER_STEPS} steps)" +fi +info "bank" "$BANK" +info "exchange" "$EX" +info "currency" "$CUR" +info "ladder mode" "${LADDER_MODE}" +# alt_unit_names for human amounts (Kilo-GOA / Mega-GOA / … + base in parentheses) +if metrics_load_alt_units "${EX%/}/config"; then + info "alt_unit_names" "from ${EX%/}/config → $ALT_UNITS_FILE" +else + warn "alt_unit_names" "using built-in SI fallback ($ALT_UNITS_FILE)" +fi +info "budget" "${LADDER_TIMEOUT_S}s" +_max_m1=$(python3 -c 'import sys; from decimal import Decimal; m=Decimal(sys.argv[1]); print(m-1 if m>1 else m)' "${LADDER_MAX_AMOUNT}" 2>/dev/null || echo "${LADDER_MAX_AMOUNT}-1") +if [ "${LADDER_MODE}" = "max" ]; then + info "max-search" "probes≤${LADDER_MAX_PROBES} high_rand=${LADDER_HIGH_RUNGS} tol=${LADDER_MAX_TOL_REL} ceiling=${CUR}:${LADDER_MAX_AMOUNT}" +else + info "steps" "${LADDER_STEPS} (0 + random≥${LADDER_MIN_AMOUNT} + max-1=${CUR}:${_max_m1} + max=${CUR}:${LADDER_MAX_AMOUNT})" + info "withdraw_scale" "${LADDER_WITHDRAW_SCALE}× pay mids (fund pay ladder)" +fi +info "pay_phase" "$([ "${LADDER_PAY}" = "1" ] && echo enabled || echo disabled)" +info "currency/domain" "${CUR} · ${TALER_DOMAIN:-?} · bank=${BANK}" + +set_group load +section "ladder · load snapshot (before withdraws)" +metrics_report_load "$LOAD_BEFORE" "ladder-start" || true +# Fresh wallet per rung — baseline empty (or last-rung DB if re-used later) +metrics_report_coins "ladder-start" || true + +if ! EXP_PW=$(resolve_explorer_pw); then + err bank "explorer password missing" \ + "set EXP_PW / EXP_PW_FILE or SECRETS_ROOT=…/koopa/host-root (taler-bank/bank-explorer-password.txt)${SECRETS_ROOT:+ · SECRETS_ROOT=${SECRETS_ROOT}}" + secrets_hint 2>/dev/null || true + exit 1 +fi +if [ -n "${SECRETS_ROOT:-}" ]; then + info "explorer secret" "resolved (SECRETS_ROOT=${SECRETS_ROOT} · stage uses stagepaysan secret first when CUR=TESTPAYSAN)" +else + info "explorer secret" "resolved via EXP_PW / EXP_PW_FILE / stage SSH / koopa" +fi +if [ -n "${CLI_JS:-}" ] && [ -f "${CLI_JS}" ]; then + info "wallet-cli" "$CLI_JS" +else + info "wallet-cli" "PATH taler-wallet-cli ($(command -v taler-wallet-cli 2>/dev/null || echo missing))" +fi + +# --- auto-account --- +t0=$(now_ms) +if ! curl -sS -m 30 -o "$SCRATCH/auto-account.json" "${BANK}/intro/auto-account.json"; then + err bank "auto-account.json unreachable" + exit 1 +fi +ms_auto=$(elapsed_ms "$t0") +if ! python3 -c 'import json; d=json.load(open("'"$SCRATCH"'/auto-account.json")); assert d.get("ok") or d.get("username")' 2>/dev/null; then + err bank "auto-account create failed" "$(head -c 120 "$SCRATCH/auto-account.json" | tr '\n' ' ')" + exit 1 +fi +ACCT_USER=$(python3 -c 'import json; print(json.load(open("'"$SCRATCH"'/auto-account.json"))["username"])') +ok "auto-account ${ACCT_USER} (${ms_auto}ms) — personal ${CUR}:0; pool=explorer" +info "auto-account password" "(see $SCRATCH/auto-account.json — not logged)" + +# --- explorer token --- +t0=$(now_ms) +TOK=$(curl -sS -m 20 -u "${EXP_USER}:${EXP_PW}" \ + -H 'Content-Type: application/json' \ + -d '{"scope":"readwrite","duration":{"d_us":3600000000}}' \ + "${BANK}/accounts/${EXP_USER}/token" \ + | python3 -c 'import json,sys; print(json.load(sys.stdin)["access_token"])') +ms_tok=$(elapsed_ms "$t0") +[ -n "$TOK" ] || { err bank "explorer token failed"; exit 1; } +ok "explorer token (${ms_tok}ms)" + +# ONE cumulative wallet for all withdraws + pays (need balance to spend). +# force-select uses *last* reserve_pub from the current accept output. +wallet_prepare() { + local label="${1:-wallet}" + WDB="$SCRATCH/wallet-${label}.sqlite3" + export WDB + if [ ! -f "$WDB" ]; then + wcli exchanges add "$EX" >"$SCRATCH/ex-add-$label.out" 2>&1 || true + wcli exchanges update "$EX" >"$SCRATCH/ex-upd-$label.out" 2>&1 || true + wcli exchanges accept-tos "$EX" >"$SCRATCH/ex-tos-$label.out" 2>&1 || true + fi +} + +t0=$(now_ms) +rm -f "$SCRATCH/wallet-main.sqlite3" +wallet_prepare "main" +ms_tos=$(elapsed_ms "$t0") +ok "wallet exchange + ToS (${ms_tos}ms) — cumulative DB for withdraw+pay (no run-until-done)" + +: "${LADDER_MAX_RUNGS:=99}" +if [ "${LADDER_MODE}" = "max" ]; then + info "ladder mode" "max-search (find highest withdrawable ${CUR})" + info "max-search params" "probes≤${LADDER_MAX_PROBES} high_rand=${LADDER_HIGH_RUNGS} tol_rel=${LADDER_MAX_TOL_REL} ceiling=${CUR}:${LADDER_MAX_AMOUNT}" + LADDER_LIST="" + PAY_LIST="" + LADDER_N=0 + set -- + printf '%s\n' "# max-search (amounts chosen adaptively)" >"$SCRATCH/ladder-plan.txt" + : >"$SCRATCH/ladder-wd-plan.txt" + : >"$SCRATCH/ladder-pay-plan.txt" +else + info "ladder mode" "classic (0 → random mids → max-1 → max)" + build_ladder_pair "$SCRATCH/ladder-wd-plan.txt" "$SCRATCH/ladder-pay-plan.txt" + LADDER_LIST=$(tr -d '\n' <"$SCRATCH/ladder-wd-plan.txt") + PAY_LIST=$(tr -d '\n' <"$SCRATCH/ladder-pay-plan.txt") + # shellcheck disable=SC2086 + set -- $LADDER_LIST + if [ "$#" -gt "$LADDER_MAX_RUNGS" ]; then + # shellcheck disable=SC2046 + set -- $(printf '%s\n' "$@" | head -n "$LADDER_MAX_RUNGS") + fi + LADDER_N=$# + info "withdraw plan" "$*" + info "withdraw plan (alt)" "$(format_amount_list_alt "$@")" + info "pay plan" "$PAY_LIST" + # shellcheck disable=SC2086 + info "pay plan (alt)" "$(format_amount_list_alt $PAY_LIST)" + printf '%s\n' "$@" >"$SCRATCH/ladder-plan.txt" + # shellcheck disable=SC2086 + printf '%s\n' $PAY_LIST >"$SCRATCH/ladder-pay-plan-lines.txt" 2>/dev/null || true +fi + +OK_N=0 +FAIL_N_L=0 +PAY_OK_N=0 +PAY_FAIL_N=0 +STOP_REASON="" +STOP_AMOUNT="" +MAX_BEST_AMT="" +MAX_LO="0" +MAX_HI="${LADDER_MAX_AMOUNT}" +declare -a RUNG_JSON=() + +# --------------------------------------------------------------------------- +# One withdraw rung. Caller sets: AMT, range_note, rung. +# Returns 0 = soft continue (ok/skip/ceiling); 1 = hard stop. +# Sets: status, AMT_NUM, WID, note, ms_*. +# --------------------------------------------------------------------------- +ladder_withdraw_rung() { + if ladder_over; then + STOP_REASON="timeout budget ${LADDER_TIMEOUT_S}s" + warn ladder "time budget exhausted" \ + "problem: LADDER_TIMEOUT_S=${LADDER_TIMEOUT_S}s reached after ${OK_N} ok rungs; remaining amounts not tried" + return 1 + fi + + # section header is printed by classic/max controller (avoids double banners) + tag=$(printf '%s' "$AMT" | tr '.:' '__') + # range_note set by caller (classic pins / max-search labels) + t_rung=$(now_ms) + ms_mint=0 ms_accept=0 ms_confirm=0 ms_settle=0 + note="" + status="FAIL" + WID="-" + FORCE_SEL_409_LOGGED=0 + + # keep cumulative main wallet + wallet_prepare "main" + + # mint from explorer pool + t0=$(now_ms) + code=$(curl -sS -m 30 -o "$SCRATCH/wd-$tag.json" -w '%{http_code}' \ + -H "Authorization: Bearer ${TOK}" \ + -H 'Content-Type: application/json' \ + -d "{\"amount\":\"${AMT}\"}" \ + "${BANK}/accounts/${EXP_USER}/withdrawals") + ms_mint=$(elapsed_ms "$t0") + WID=$(python3 -c 'import json;d=json.load(open("'"$SCRATCH"'/wd-'"$tag"'.json"));print(d.get("withdrawal_id") or "")' 2>/dev/null || true) + URI=$(python3 -c 'import json;u=json.load(open("'"$SCRATCH"'/wd-'"$tag"'.json")).get("taler_withdraw_uri") or "";print(u.replace(":443/","/"))' 2>/dev/null || true) + # numeric amount (for zero / settle / ceiling special-cases) + AMT_NUM=$(python3 -c 'import sys; print(sys.argv[1].split(":",1)[-1])' "$AMT") + IS_ZERO=0 + python3 -c 'import sys; from decimal import Decimal; sys.exit(0 if Decimal(sys.argv[1])==0 else 1)' "$AMT_NUM" 2>/dev/null && IS_ZERO=1 + IS_MAX_PIN=0 + IS_MAX_M1_PIN=0 + if [ "$AMT_NUM" = "${LADDER_MAX_AMOUNT}" ]; then + IS_MAX_PIN=1 + range_note="pin:max" + elif [ "$AMT_NUM" = "$((LADDER_MAX_AMOUNT - 1))" ] 2>/dev/null || \ + [ "$AMT_NUM" = "$(python3 -c 'print(int("'"$LADDER_MAX_AMOUNT"'")-1)')" ]; then + IS_MAX_M1_PIN=1 + range_note="pin:max-1" + fi + + if [ "$code" != "200" ] && [ "$code" != "201" ] || [ -z "$WID" ] || [ -z "$URI" ]; then + # strip quotes so STOP_REASON never breaks later shell/python argv + note="mint HTTP $code $(head -c 100 "$SCRATCH/wd-$tag.json" 2>/dev/null | tr '\n\"' ' ')" + ms_total=$(elapsed_ms "$t_rung") + if [ "$IS_ZERO" = "1" ]; then + # Probe only: bank may reject GOA:0 — record and continue ladder + status="ZERO_REJECT" + note="zero-withdraw rejected (expected possible): $note" + warn bank "mint $AMT rejected" \ + "problem: bank will not create a GOA:0 withdrawal (zero amount probe). Ladder continues. detail: $note" + echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" + OK_N=$((OK_N + 1)) + return 0 + fi + # Absolute max is a ceiling probe — bank often returns SQL P0001/5110; do not abort. + if [ "$IS_MAX_PIN" = "1" ]; then + status="CEILING_REJECT" + note="absolute max rejected (ceiling probe; max-1 is the hard pin): $note" + warn bank "mint $AMT rejected (ceiling)" \ + "problem: bank rejects absolute LADDER_MAX_AMOUNT (often SQL P0001/5110). max-1 rung is the last expected success. Ladder continues to report. detail: $note" + echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" + return 0 + fi + # max-1 can also hit the same libeufin SQL ceiling (5110/P0001) on live GOA — + # treat as soft ceiling when mid rungs already passed; do not block the report. + if [ "$IS_MAX_M1_PIN" = "1" ] && echo "$note" | grep -qE '5110|P0001'; then + status="CEILING_REJECT" + note="max-1 rejected (same ceiling as absolute max): $note" + warn bank "mint $AMT rejected (max-1 ceiling)" \ + "problem: bank rejects max-1 with SQL P0001/5110 (pool ceiling). Mid-rung OK_BANK/OK still count. Ladder continues. detail: $note" + echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" + return 0 + fi + # max-search: any mint reject is "too high" / ceiling — never hard-stop the hunt + if [ "${LADDER_MODE}" = "max" ]; then + status="CEILING_REJECT" + note="max-search too-high mint: $note" + warn bank "mint $AMT rejected (max-search)" \ + "problem: mint failed while hunting max withdrawable — treat as upper bound. detail: $note" + echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" + return 0 + fi + # Classic mid/high random rungs: same 5110/P0001 pool ceiling as pins — soft, not FAIL_MINT. + if echo "$note" | grep -qE '5110|P0001'; then + status="CEILING_REJECT" + note="amount above live mint ceiling (5110/P0001): $note" + warn bank "mint $AMT rejected (ceiling)" \ + "problem: bank mint HTTP 500/5110 SQL P0001 on high amount (not only pin:max). Ladder continues; use max-ladder to bound hi. detail: $note" + echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" + return 0 + fi + err bank "mint $AMT failed" "$note" + status="FAIL_MINT" + STOP_REASON="$note" + STOP_AMOUNT="$AMT" + echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" + FAIL_N_L=$((FAIL_N_L + 1)) + return 1 + fi + ok "mint $AMT ($WID) ${ms_mint}ms" + + before=$(wallet_avail) + + # accept + t0=$(now_ms) + if wcli withdraw accept-uri --exchange "$EX" "$URI" >"$SCRATCH/accept-$tag.out" 2>&1; then + ms_accept=$(elapsed_ms "$t0") + ok "accept-uri $AMT ${ms_accept}ms" + else + ms_accept=$(elapsed_ms "$t0") + note="accept-uri failed: $(tail -c 200 "$SCRATCH/accept-$tag.out" | tr '\n' ' ')" + ms_total=$(elapsed_ms "$t_rung") + # Wallet 7006: no denominations for this amount (0, sub-denom dust, etc.) — warn & continue + if [ "$IS_ZERO" = "1" ] || grep -qE 'code: 7006|"code"[[:space:]]*:[[:space:]]*7006|No denominations could be selected' \ + "$SCRATCH/accept-$tag.out" 2>/dev/null; then + if [ "$IS_ZERO" = "1" ]; then + status="ZERO_SKIP" + note="zero-withdraw skip (7006 / no denoms): $note" + warn wallet "accept $AMT skipped" \ + "problem: wallet code 7006 — no coin denominations for GOA:0 (zero amount cannot be withdrawn as coins). Ladder continues. detail: $note" + else + status="SKIP_DENOM" + note="skip amount (wallet 7006 no denoms): $note" + warn wallet "accept $AMT skipped" \ + "problem: wallet code 7006 — no denominations match this amount (below smallest coin or not combinable). Ladder continues with next rung. detail: $note" + fi + echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" + return 0 + fi + if [ "${LADDER_MODE}" = "max" ]; then + status="SKIP_ACCEPT" + warn wallet "accept $AMT skipped (max-search)" "$note" + echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" + return 0 + fi + err wallet "accept $AMT" "$note" + status="FAIL_ACCEPT" + STOP_REASON="$note" + STOP_AMOUNT="$AMT" + echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" + FAIL_N_L=$((FAIL_N_L + 1)) + return 1 + fi + + # Confirm ASAP when bank status is selected. No run-until-done (hangs on macOS/wallet). + # Server-side auto-confirm only watches landing withdraw-watch.ids — ladder must confirm itself. + bank_st() { + curl -sS -m 8 "${BANK}/taler-integration/withdrawal-operation/${WID}" 2>/dev/null \ + | python3 -c 'import json,sys; print((json.load(sys.stdin).get("status") or "").strip())' 2>/dev/null || true + } + do_confirm() { + curl -sS -m 15 -o "$SCRATCH/conf-$tag.json" -w '%{http_code}' -X POST \ + -H "Authorization: Bearer ${TOK}" -H 'Content-Type: application/json' -d '{}' \ + "${BANK}/accounts/${EXP_USER}/withdrawals/${WID}/confirm" + } + # Collect reserve_pub candidates for *this* withdrawal (WID + amount). + # Cumulative wallets re-print old reserves in accept/tx dumps — never trust a single "last" blindly. + # Prints unique pubs one per line, preferred order first. + # + # Critical: wallet-cli dumps mix log lines + JSON. Never json.loads(rest_of_file) — + # trailing "Shutdown requested" lines make that always fail. Use JSONDecoder.raw_decode + # and regex on *both* accept and transactions output (reservePub lives in tx, not accept). + extract_rpubs_for_wid() { + # External module — never feed extract via bash heredoc (refactor tools + # previously turned Python continue into return → SyntaxError line 91). + local _ext="$ROOT/ladder_extract_rpubs.py" + if [ ! -f "$_ext" ]; then + warn bank "force-select extract missing" "problem: $_ext not found" + return 0 + fi + python3 "$_ext" "$WID" "$AMT" \ + "$SCRATCH/accept-$tag.out" "$SCRATCH/tx-$tag.json" "$SCRATCH/used-rpubs.txt" + } + + mark_rpub_used() { + local p="$1" + [ -n "$p" ] || return 0 + mkdir -p "$SCRATCH" 2>/dev/null || true + grep -qxF "$p" "$SCRATCH/used-rpubs.txt" 2>/dev/null || echo "$p" >>"$SCRATCH/used-rpubs.txt" + } + + force_select_if_needed() { + local st_now="$1" + [ "$st_now" = "pending" ] || [ -z "$st_now" ] || return 0 + local rpub epayto code_fs any=0 + # refresh tx dump each try (wallet may attach reserve late) + wcli transactions >"$SCRATCH/tx-$tag.json" 2>&1 || true + epayto=$(curl -sS -m 10 "${EX%/}/keys" 2>/dev/null | python3 -c ' +import json,sys +d=json.load(sys.stdin) +acc=d.get("accounts") or [] +for a in acc: + p=a.get("payto_uri") or a.get("payto_address") or "" + if "x-taler-bank" in p or "exchange" in p: + print(p); break +else: + if acc: print(acc[0].get("payto_uri") or "") +' 2>/dev/null || true) + if [ -z "$epayto" ]; then + warn bank "force-select skipped" "problem: exchange payto empty from /keys" + return 0 + fi + # Try candidates until bank leaves pending (200/204) or we exhaust + while IFS= read -r rpub; do + [ -n "$rpub" ] || continue + any=1 + code_fs=$(curl -sS -m 12 -o "$SCRATCH/force-sel-$tag.json" -w '%{http_code}' -X POST \ + -H 'Content-Type: application/json' \ + -d "{\"reserve_pub\":\"${rpub}\",\"selected_exchange\":\"${epayto}\"}" \ + "${BANK}/taler-integration/withdrawal-operation/${WID}" 2>/dev/null || echo "000") + if [ "$code_fs" = "200" ] || [ "$code_fs" = "204" ]; then + info "force-select" "HTTP ${code_fs} rpub=${rpub:0:12}… (ok for WID ${WID:0:8})" + mark_rpub_used "$rpub" + return 0 + fi + if [ "$code_fs" = "409" ]; then + # 5114 = this reserve already bound to another op — not "out of money" + info "force-select" "HTTP 409 rpub=${rpub:0:12}… (stale/used reserve — not balance; trying next)" + mark_rpub_used "$rpub" + continue + fi + info "force-select" "HTTP ${code_fs} rpub=${rpub:0:12}… body=$(tr '\n' ' ' <"$SCRATCH/force-sel-$tag.json" 2>/dev/null | head -c 120)" + # other errors: still try next candidate + done < <(extract_rpubs_for_wid) + if [ "$any" != "1" ]; then + warn bank "force-select skipped" \ + "problem: no reserve_pub for this withdraw (WID=${WID:0:8}…); wallet may not have selected yet" + fi + } + + # No run-until-done (hangs / banned). Read transactions only for reserve_pub candidates. + wcli transactions >"$SCRATCH/tx-$tag.json" 2>&1 || true + + t0=$(now_ms) + conf_ok=0 + st="" + # Poll bank status; force-select while pending; confirm as soon as selected + for i in $(seq 1 "${LADDER_CONFIRM_POLLS}"); do + ladder_over && break + st=$(bank_st) + case "$st" in + selected) + ccode=$(do_confirm) + if [ "$ccode" = "204" ] || [ "$ccode" = "200" ]; then + conf_ok=1 + info "confirm" "HTTP $ccode after selected (poll $i)" + else + note="confirm HTTP $ccode" + fi + break + ;; + confirmed) + conf_ok=1 + break + ;; + aborted) + note="withdrawal aborted by bank" + break + ;; + esac + # force-select while pending — try alternate rpubs on 5114 (not out-of-money) + if [ "$st" = "pending" ] || [ -z "$st" ]; then + if [ "$i" -eq 1 ] || [ "$i" -eq 2 ] || [ $((i % 3)) -eq 0 ]; then + force_select_if_needed "$st" + fi + fi + sleep 0.35 + done + ms_confirm=$(elapsed_ms "$t0") + st=$(printf '%s' "${st:-}" | tr -d '\r\n' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//') + if [ "$conf_ok" != "1" ]; then + note="${note:-confirm timeout last=${st:-empty}}" + # Soft: not confirmed — usually stale reserve_pub (5114), NOT empty pool balance + status="SKIP_CONFIRM" + warn bank "confirm $AMT skipped" \ + "problem: bank status='${st:-empty}' after ${LADDER_CONFIRM_POLLS} polls (want selected/confirmed). Usually reserve_pub mismatch (5114), not out-of-money — mint/accept already OK. detail: $note" + ms_total=$(elapsed_ms "$t_rung") + echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" + return 0 + fi + ok "confirm $AMT ${ms_confirm}ms (client, on selected)" + + # settle: poll wallet balance + bank transfer_done only — never run-until-done + t0=$(now_ms) + settled=0 + xfer="?" + if [ "$IS_ZERO" = "1" ]; then + settled=1 + note="zero-amount: no coin delta expected" + else + for r in $(seq 1 "$LADDER_SETTLE_ROUNDS"); do + ladder_over && break + after=$(wallet_avail) + if python3 -c " +from decimal import Decimal +import sys +sys.exit(0 if Decimal(sys.argv[1]) > Decimal(sys.argv[2]) else 1) +" "$after" "$before" 2>/dev/null; then + settled=1 + break + fi + xfer=$(curl -sS -m 10 "${BANK}/taler-integration/withdrawal-operation/${WID}" \ + | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d.get("transfer_done"), d.get("status"))' 2>/dev/null || echo "?") + # bank done is enough to leave settle without hanging on wallet + if echo "$xfer" | grep -qi True; then + note="bank transfer_done (no run-until-done) avail=${after} $xfer" + break + fi + sleep "$LADDER_SETTLE_SLEEP" + done + fi + ms_settle=$(elapsed_ms "$t0") + after=$(wallet_avail) + ms_total=$(elapsed_ms "$t_rung") + if [ -z "${xfer:-}" ] || [ "$xfer" = "?" ]; then + xfer=$(curl -sS -m 10 "${BANK}/taler-integration/withdrawal-operation/${WID}" \ + | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d.get("transfer_done"), d.get("status"))' 2>/dev/null || echo "?") + fi + + if [ "$settled" = "1" ]; then + status="OK" + note="${note:-avail=${CUR}:${after}}" + ok "settle $AMT → ${CUR}:${after} (settle ${ms_settle}ms, rung ${ms_total}ms)" + OK_N=$((OK_N + 1)) + echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" + metrics_report_coins "ladder-r${rung}-after-${tag}" || true + metrics_record_flow withdrawn "$AMT" || true + elif echo "$xfer" | grep -qi True; then + status="OK_BANK" + note="bank transfer_done avail=${after} $xfer (no run-until-done)" + ok "settle $AMT bank transfer_done (wallet avail=${CUR}:${after}, ${ms_settle}ms)" + OK_N=$((OK_N + 1)) + echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" + metrics_report_coins "ladder-r${rung}-after-${tag}" || true + metrics_record_flow withdrawn "$AMT" || true + else + note="no coins / no transfer_done avail=${after} $xfer" + if [ "${LADDER_MODE}" = "max" ]; then + status="SKIP_SETTLE" + warn wallet "settle $AMT skipped (max-search)" \ + "problem: bank confirm ok but settle inconclusive — do not move max bounds. detail: $note" + echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" + return 0 + fi + err wallet "settle $AMT" "$note" + status="FAIL_SETTLE" + STOP_REASON="$note" + STOP_AMOUNT="$AMT" + FAIL_N_L=$((FAIL_N_L + 1)) + echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" + metrics_report_coins "ladder-r${rung}-fail-${tag}" || true + return 1 + fi + return 0 +} + +set_group withdraw + +# --- next amount for max-search (prints: AMT_NUM|range_note|action) --- +# action: probe | done +max_search_next() { + python3 - "$CUR" "$MAX_LO" "$MAX_HI" "$LADDER_MIN_AMOUNT" \ + "$LADDER_MAX_TOL_REL" "$rung" "$LADDER_HIGH_RUNGS" "$LADDER_MAX_SEED" \ + "${MAX_BEST_OK:-}" <<'PY' +import math, random, sys +from decimal import Decimal, ROUND_HALF_UP, ROUND_UP + +cur = sys.argv[1] +lo = Decimal(sys.argv[2] or "0") +hi = Decimal(sys.argv[3]) +min_amt = Decimal(sys.argv[4]) +tol = Decimal(sys.argv[5] or "0.001") +probe_i = int(sys.argv[6]) +high_n = max(1, int(sys.argv[7] or "4")) +seed = sys.argv[8].strip() +best = sys.argv[9].strip() + +if seed: + random.seed(int(seed) + probe_i) +else: + random.seed() + +if min_amt <= 0: + min_amt = Decimal("0.000001") +if hi <= min_amt: + print("0|max:done|done") + raise SystemExit(0) + +if lo > 0 and hi > lo: + gap = (hi - lo) / hi + abs_step = Decimal(1) if hi >= 1 else min_amt + if gap <= tol or (hi - lo) <= abs_step: + print("%s|max:done|done" % (best or str(lo))) + raise SystemExit(0) + +def fmt(v: Decimal) -> str: + if v >= 1: + q = v.quantize(Decimal(1), rounding=ROUND_HALF_UP) + return format(int(q), "d") + q = v.quantize(Decimal("0.00000001"), rounding=ROUND_UP) + s = format(q, "f").rstrip("0").rstrip(".") + return s or "0" + +def clamp(v: Decimal): + if v <= lo and lo > 0: + v = lo + (Decimal(1) if lo >= 1 else min_amt) + if v >= hi: + v = hi - (Decimal(1) if hi > 1 else min_amt) + if v < min_amt: + v = min_amt + if v <= 0: + v = min_amt + if v >= hi: + return None + return v + +if probe_i < high_n: + band_lo = max(lo if lo > 0 else min_amt, hi * Decimal("0.15")) + band_hi = hi * Decimal("0.999") + if band_hi <= band_lo: + band_lo = max(min_amt, hi * Decimal("0.5")) + band_hi = hi * Decimal("0.99") + flo, fhi = float(band_lo), float(band_hi) + if flo <= 0: + flo = float(min_amt) + if fhi <= flo: + fhi = flo * 2 + v = Decimal(str(math.exp(random.uniform(math.log(flo), math.log(fhi))))) + v = clamp(v) + if v is None: + print("%s|max:done|done" % (best or str(lo))) + raise SystemExit(0) + print("%s|max:high-rand|probe" % fmt(v)) + raise SystemExit(0) + +if lo <= 0: + flo = float(max(min_amt, hi * Decimal("0.05"))) + fhi = float(hi * Decimal("0.85")) + if fhi <= flo: + fhi = flo * 1.5 + v = Decimal(str(math.exp(random.uniform(math.log(flo), math.log(fhi))))) + label = "max:descent" +else: + v = (lo * hi).sqrt() + label = "max:bisect" +v = clamp(v) +if v is None: + print("%s|max:done|done" % (best or str(lo))) + raise SystemExit(0) +print("%s|%s|probe" % (fmt(v), label)) +PY +} + +if [ "${LADDER_MODE}" = "max" ]; then + section "ladder · phase A · max-search (find highest withdrawable ${CUR})" + info "max-search" "ceiling=${CUR}:${LADDER_MAX_AMOUNT} min=${CUR}:${LADDER_MIN_AMOUNT} probes≤${LADDER_MAX_PROBES} high_rand=${LADDER_HIGH_RUNGS} tol_rel=${LADDER_MAX_TOL_REL}" + info "max-search" "pay phase $([ "${LADDER_PAY}" = "1" ] && echo enabled || echo disabled)" + MAX_LO="0" + MAX_HI="${LADDER_MAX_AMOUNT}" + MAX_BEST_OK="" + MAX_BEST_AMT="" + rung=0 + # LADDER_MAX_PROBES=0 → no probe cap (stop on converge / timeout only) + while [ "${LADDER_MAX_PROBES}" = "0" ] || [ "$rung" -lt "${LADDER_MAX_PROBES}" ]; do + if ladder_over; then + STOP_REASON="timeout budget ${LADDER_TIMEOUT_S}s" + warn ladder "time budget exhausted" \ + "problem: LADDER_TIMEOUT_S=${LADDER_TIMEOUT_S}s during max-search after ${OK_N} ok; best=${MAX_BEST_AMT:-none} · $(format_amount_alt "${MAX_BEST_AMT:-${CUR}:0}")" + break + fi + _next=$(max_search_next) + _anum=$(printf '%s' "$_next" | cut -d'|' -f1) + range_note=$(printf '%s' "$_next" | cut -d'|' -f2) + _act=$(printf '%s' "$_next" | cut -d'|' -f3) + if [ "$_act" = "done" ]; then + info "max-search" "converged best=$(format_amount_alt "${MAX_BEST_AMT:-${CUR}:0}") · lo=$(format_amount_alt "${CUR}:${MAX_LO}") · hi=$(format_amount_alt "${CUR}:${MAX_HI}") · probes=${rung}" + break + fi + AMT="${CUR}:${_anum}" + if [ -n "${_LAST_MAX_AMT:-}" ] && [ "$AMT" = "$_LAST_MAX_AMT" ]; then + info "max-search" "repeat amount $(format_amount_alt "$AMT") — tightening hi" + MAX_HI=$(python3 -c 'from decimal import Decimal; a=Decimal("'"${_anum}"'"); print(a-(1 if a>=1 else Decimal("'"${LADDER_MIN_AMOUNT}"'")))') + continue + fi + _LAST_MAX_AMT="$AMT" + rung=$((rung + 1)) + section "ladder · max probe $rung $AMT · $(format_amount_alt "$AMT") [lo=$(format_amount_alt "${CUR}:${MAX_LO}") · hi=$(format_amount_alt "${CUR}:${MAX_HI}")]" + status="FAIL" + if ! ladder_withdraw_rung; then + break + fi + AMT_NUM=$(python3 -c 'import sys; print(sys.argv[1].split(":",1)[-1])' "$AMT") + case "$status" in + OK|OK_BANK) + MAX_LO="$AMT_NUM" + MAX_BEST_OK="$AMT_NUM" + MAX_BEST_AMT="$AMT" + info "max-search" "success bound lo=$(format_amount_alt "${CUR}:${MAX_LO}") (best so far)" + ;; + CEILING_REJECT|FAIL_MINT) + MAX_HI=$(python3 -c ' +from decimal import Decimal +a=Decimal("'"$AMT_NUM"'"); h=Decimal("'"$MAX_HI"'") +print(min(a,h)) +') + info "max-search" "too-high hi:=$(format_amount_alt "${CUR}:${MAX_HI}")" + ;; + SKIP_DENOM|ZERO_SKIP|ZERO_REJECT) + info "max-search" "denom-skip $(format_amount_alt "$AMT") (bounds unchanged)" + ;; + SKIP_CONFIRM|SKIP_SETTLE|SKIP_ACCEPT) + info "max-search" "inconclusive $status $(format_amount_alt "$AMT") (bounds unchanged)" + ;; + *) + info "max-search" "status=$status $(format_amount_alt "$AMT") (bounds unchanged)" + ;; + esac + if python3 -c 'from decimal import Decimal; import sys; sys.exit(0 if Decimal(sys.argv[1])>0 and Decimal(sys.argv[2])<=Decimal(sys.argv[1]) else 1)' "$MAX_LO" "$MAX_HI" 2>/dev/null; then + info "max-search" "bounds crossed lo=$(format_amount_alt "${CUR}:${MAX_LO}") · hi=$(format_amount_alt "${CUR}:${MAX_HI}") — stop" + break + fi + done + if [ -n "${MAX_BEST_AMT:-}" ]; then + ok "max-search best withdrawable $(format_amount_alt "$MAX_BEST_AMT") · lo=$(format_amount_alt "${CUR}:${MAX_LO}") · hi=$(format_amount_alt "${CUR}:${MAX_HI}") · probes=${rung}" + else + warn ladder "max-search no successful withdraw" \ + "problem: no OK/OK_BANK within ${rung} probes; hi=$(format_amount_alt "${CUR}:${MAX_HI}")" + fi + { + echo "# max-search mode" + echo "best=${MAX_BEST_AMT:-}" + echo "best_alt=$(format_amount_alt "${MAX_BEST_AMT:-${CUR}:0}")" + echo "lo=${MAX_LO}" + echo "lo_alt=$(format_amount_alt "${CUR}:${MAX_LO}")" + echo "hi=${MAX_HI}" + echo "hi_alt=$(format_amount_alt "${CUR}:${MAX_HI}")" + echo "probes=${rung}" + } >"$SCRATCH/ladder-plan.txt" + printf '%s\n' "${MAX_BEST_AMT:-${CUR}:0}" >"$SCRATCH/ladder-wd-plan.txt" + : >"$SCRATCH/ladder-pay-plan.txt" + PAY_LIST="" +else + section "ladder · phase A · withdraw (${LADDER_N} rungs)" + rung=0 + for AMT in "$@"; do + rung=$((rung + 1)) + if ladder_over; then + STOP_REASON="timeout budget ${LADDER_TIMEOUT_S}s" + warn ladder "time budget exhausted" \ + "problem: LADDER_TIMEOUT_S=${LADDER_TIMEOUT_S}s reached after ${OK_N} ok rungs; remaining amounts not tried" + break + fi + if [ "$rung" -eq 1 ]; then + range_note="pin:0" + elif [ "$rung" -eq "$LADDER_N" ]; then + range_note="pin:max" + elif [ "$rung" -eq $((LADDER_N - 1)) ] && [ "$LADDER_N" -ge 3 ]; then + range_note="pin:max-1" + else + range_note="random" + fi + section "ladder · rung $rung $AMT · $(format_amount_alt "$AMT")" + if ! ladder_withdraw_rung; then + break + fi + done +fi + + +# --------------------------------------------------------------------------- +# Phase B — payment ladder (same shape: 0 → low → … → max-1 → max) +# --------------------------------------------------------------------------- +PAY_OK_N=0 +PAY_FAIL_N=0 +if [ "${LADDER_PAY}" = "1" ] && [ -n "${PAY_LIST:-}" ] && [ "$FAIL_N_L" -eq 0 ]; then + set_group pay + section "ladder · phase B · pay" + metrics_report_coins "before-pay-ladder" || true + # Merchant secret (same as e2e) — stage often has only public shop templates + MPW="${E2E_MERCHANT_TOKEN:-${MERCHANT_TOKEN:-}}" + if [ -z "$MPW" ]; then + MPW=$(read_secret "taler-merchant/merchant-${INST}-password.txt" 2>/dev/null || true) + fi + if [ -z "$MPW" ]; then + MPW=$(read_secret "taler-merchant/merchant-goa-demo-cp4zqk-password.txt" 2>/dev/null || true) + fi + if [ -z "$MPW" ] && [ "${CUR}" = "TESTPAYSAN" ]; then + if [ -n "${FRANCPAYSAN_SECRETS:-}" ] && [ -f "${FRANCPAYSAN_SECRETS}/stage/default-instance-token.txt" ]; then + MPW=$(tr -d '\n\r' <"${FRANCPAYSAN_SECRETS}/stage/default-instance-token.txt") + fi + if [ -z "$MPW" ]; then + _remote_mer="${FRANCPAYSAN_REMOTE_MERCHANT_TOKEN:-}" + [ -z "$_remote_mer" ] && [ -n "${FRANCPAYSAN_REMOTE_SECRETS_ROOT:-}" ] && \ + _remote_mer="${FRANCPAYSAN_REMOTE_SECRETS_ROOT}/merchant/secrets/default-instance-token.txt" + for host in ${INSIDE_SSH:+"$INSIDE_SSH"} ${FRANCPAYSAN_SSH:+"$FRANCPAYSAN_SSH"}; do + [ -n "$host" ] && [ -n "$_remote_mer" ] || continue + MPW=$(ssh -o BatchMode=yes -o ConnectTimeout=12 "$host" \ + "tr -d '\\n\\r' <${_remote_mer} 2>/dev/null || true" \ + 2>/dev/null || true) + [ -n "$MPW" ] && break + done + unset _remote_mer + fi + fi + if [ -z "$MPW" ]; then + if [ "${CUR}" = "TESTPAYSAN" ]; then + # Stage: withdraw ladder is the main probe; pays need instance token or + # public templates for *exact* face values (not continuous ladder amounts). + info pay "no merchant token — skip pay ladder on TESTPAYSAN (withdraw maxima still covered)" + info pay "hint: set E2E_MERCHANT_TOKEN or use ./taler-monitoring.sh -d stage.lefrancpaysan.ch e2e for shop templates" + else + warn pay "no merchant token — skip pay ladder (set E2E_MERCHANT_TOKEN or secrets)" + fi + else + ok "merchant token" "instance ${INST}" + case "$MPW" in + secret-token:*) AUTH="Authorization: Bearer ${MPW}" ;; + *) AUTH="Authorization: Bearer secret-token:${MPW}" ;; + esac + wallet_prepare "main" + # shellcheck disable=SC2086 + set -- $PAY_LIST + PAY_N=$# + _bal0=$(wallet_avail) + info "pay rungs" "$PAY_N · $*" + info "pay budget" "wallet available ${CUR}:${_bal0} · $(format_amount_alt "${CUR}:${_bal0}") — never order more than this" + prung=0 + for PAMT in "$@"; do + prung=$((prung + 1)) + if ladder_over; then + warn pay "time budget exhausted" "after ${PAY_OK_N} ok pays" + break + fi + section "ladder · pay rung $prung $PAMT · $(format_amount_alt "$PAMT")" + ptag=$(printf '%s' "$PAMT" | tr '.:' '__') + if [ "$prung" -eq 1 ]; then + prange="pin:0" + elif [ "$prung" -eq "$PAY_N" ]; then + prange="pin:max" + elif [ "$prung" -eq $((PAY_N - 1)) ] && [ "$PAY_N" -ge 3 ]; then + prange="pin:max-1" + else + prange="random" + fi + PNUM=$(python3 -c 'import sys; print(sys.argv[1].split(":",1)[-1])' "$PAMT") + IS_PZERO=0 + python3 -c 'import sys; from decimal import Decimal; sys.exit(0 if Decimal(sys.argv[1])==0 else 1)' "$PNUM" 2>/dev/null && IS_PZERO=1 + IS_PMAX=0 + [ "$PNUM" = "${LADDER_MAX_AMOUNT}" ] && IS_PMAX=1 + t_pay=$(now_ms) + ms_order=0 ms_handle=0 ms_psettle=0 + pnote="" + pstatus="FAIL" + OID="-" + + metrics_report_coins "before-pay-${ptag}" || true + bal=$(wallet_avail) + + if [ "$IS_PZERO" = "1" ]; then + pstatus="ZERO_SKIP" + pnote="zero pay probe skipped" + warn pay "pay $PAMT skipped" "problem: zero amount order not useful. Ladder continues." + ms_total=$(elapsed_ms "$t_pay") + echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t${ms_handle}\t${ms_psettle}\t${ms_total}\t${OID}\t${pnote}" >>"$PAY_TSV" + PAY_OK_N=$((PAY_OK_N + 1)) + continue + fi + + # Never spend more than wallet available (spendable coins only — not pendingIncoming). + # Soft-skip any over-budget rung; do not hard-fail the ladder for insufficient funds. + if ! python3 -c 'import sys; from decimal import Decimal; sys.exit(0 if Decimal(sys.argv[1])>=Decimal(sys.argv[2]) else 1)' "$bal" "$PNUM" 2>/dev/null; then + pnote="skip: would spend more than available avail=${CUR}:${bal} need=${PAMT}" + ms_total=$(elapsed_ms "$t_pay") + pstatus="SKIP_BALANCE" + warn pay "pay $PAMT skipped (balance)" \ + "problem: pay amount exceeds wallet available — never overspend. $(format_amount_alt "${CUR}:${bal}") free vs $(format_amount_alt "$PAMT"). detail: $pnote" + echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t0\t0\t0\t${ms_total}\t-\t${pnote}" >>"$PAY_TSV" + continue + fi + + t0=$(now_ms) + SUM_JSON=$(python3 -c 'import json,sys; print(json.dumps(sys.argv[1]))' "ladder pay ${PAMT}" 2>/dev/null || echo '"ladder pay"') + curl -skS -m 20 -o "$SCRATCH/ord-$ptag.json" -X POST \ + -H "$AUTH" -H 'Content-Type: application/json' \ + -d "{\"order\":{\"summary\":${SUM_JSON},\"amount\":\"${PAMT}\",\"fulfillment_message\":\"ok\"},\"create_token\":true}" \ + "${MER}/instances/${INST}/private/orders" 2>"$SCRATCH/ord-$ptag.err" || true + ms_order=$(elapsed_ms "$t0") + OID=$(python3 -c 'import json,re,sys;t=open(sys.argv[1]).read() +try: print(json.loads(t).get("order_id") or "") +except Exception: + m=re.search(r"\"order_id\"\s*:\s*\"([^\"]+)\"",t); print(m.group(1) if m else "") +' "$SCRATCH/ord-$ptag.json" 2>/dev/null || true) + OTOK=$(python3 -c 'import json,re,sys;t=open(sys.argv[1]).read() +try: print(json.loads(t).get("token") or "") +except Exception: + m=re.search(r"\"token\"\s*:\s*\"([^\"]+)\"",t); print(m.group(1) if m else "") +' "$SCRATCH/ord-$ptag.json" 2>/dev/null || true) + if [ -z "$OID" ]; then + pnote="order create failed $(head -c 80 "$SCRATCH/ord-$ptag.json" 2>/dev/null | tr '\n\"' ' ')" + ms_total=$(elapsed_ms "$t_pay") + if [ "$IS_PMAX" = "1" ]; then + pstatus="CEILING_REJECT" + warn pay "pay $PAMT rejected (ceiling)" "$pnote" + echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t0\t0\t${ms_total}\t-\t${pnote}" >>"$PAY_TSV" + continue + fi + err pay "order $PAMT" "$pnote" + pstatus="FAIL_ORDER" + PAY_FAIL_N=$((PAY_FAIL_N + 1)) + FAIL_N_L=$((FAIL_N_L + 1)) + STOP_REASON="$pnote" + STOP_AMOUNT="$PAMT" + echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t0\t0\t${ms_total}\t-\t${pnote}" >>"$PAY_TSV" + break + fi + ok "order $OID ($PAMT) ${ms_order}ms" + + curl -skS -m 12 -o "$SCRATCH/ord-det-$ptag.json" -H "$AUTH" \ + "${MER}/instances/${INST}/private/orders/${OID}" 2>/dev/null || true + PAYURI=$(python3 -c 'import json,sys +try: print(json.load(open(sys.argv[1])).get("taler_pay_uri") or "") +except Exception: print("") +' "$SCRATCH/ord-det-$ptag.json" 2>/dev/null || true) + if [ -z "$PAYURI" ] && [ -n "$OTOK" ]; then + MH=$(python3 -c 'from urllib.parse import urlparse; print(urlparse("'"$MER"'").hostname or "taler.hacktivism.ch")' 2>/dev/null || echo "taler.hacktivism.ch") + PAYURI="taler://pay/${MH}/instances/${INST}/${OID}/?c=${OTOK}" + fi + PAYURI=$(printf '%s' "$PAYURI" | sed 's/:443\//\//g; s/:443?/?/g') + if [ -z "$PAYURI" ]; then + pnote="no pay URI for $OID" + ms_total=$(elapsed_ms "$t_pay") + err pay "uri $PAMT" "$pnote" + pstatus="FAIL_URI" + PAY_FAIL_N=$((PAY_FAIL_N + 1)) + FAIL_N_L=$((FAIL_N_L + 1)) + STOP_REASON="$pnote" + STOP_AMOUNT="$PAMT" + echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t0\t0\t${ms_total}\t${OID}\t${pnote}" >>"$PAY_TSV" + break + fi + + t0=$(now_ms) + if ! wcli handle-uri --yes "$PAYURI" >"$SCRATCH/pay-$ptag.out" 2>&1; then + ms_handle=$(elapsed_ms "$t0") + pnote="handle-uri failed $(tail -c 100 "$SCRATCH/pay-$ptag.out" | tr '\n\"' ' ')" + ms_total=$(elapsed_ms "$t_pay") + if [ "$IS_PMAX" = "1" ]; then + pstatus="CEILING_REJECT" + warn pay "pay $PAMT handle failed (ceiling)" "$pnote" + echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t${ms_handle}\t0\t${ms_total}\t${OID}\t${pnote}" >>"$PAY_TSV" + continue + fi + err pay "handle $PAMT" "$pnote" + pstatus="FAIL_HANDLE" + PAY_FAIL_N=$((PAY_FAIL_N + 1)) + FAIL_N_L=$((FAIL_N_L + 1)) + STOP_REASON="$pnote" + STOP_AMOUNT="$PAMT" + echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t${ms_handle}\t0\t${ms_total}\t${OID}\t${pnote}" >>"$PAY_TSV" + break + fi + ms_handle=$(elapsed_ms "$t0") + ok "handle-uri $PAMT ${ms_handle}ms" + + # Short settle polls: merchant order + wallet tx only — never run-until-done + t0=$(now_ms) + settled=0 + r=0 + while [ "$r" -lt "${LADDER_PAY_SETTLE_ROUNDS}" ]; do + r=$((r + 1)) + wcli transactions >"$SCRATCH/tx-$ptag.out" 2>&1 || true + curl -skS -m 8 -o "$SCRATCH/ord-paid-$ptag.json" -H "$AUTH" \ + "${MER}/instances/${INST}/private/orders/${OID}" 2>/dev/null || true + if grep -qiE 'payment|paid|Payment' "$SCRATCH/tx-$ptag.out" 2>/dev/null \ + || python3 -c 'import json,sys +d=json.load(open(sys.argv[1])) +sys.exit(0 if d.get("paid") is True or str(d.get("order_status","")).lower()=="paid" else 1) +' "$SCRATCH/ord-paid-$ptag.json" 2>/dev/null; then + settled=1 + break + fi + sleep 0.5 + done + ms_psettle=$(elapsed_ms "$t0") + ms_total=$(elapsed_ms "$t_pay") + after=$(wallet_avail) + if [ "$settled" = "1" ]; then + pstatus="OK" + pnote="avail=${CUR}:${after}" + ok "pay settled $PAMT → bal ${CUR}:${after} (total ${ms_total}ms)" + PAY_OK_N=$((PAY_OK_N + 1)) + echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t${ms_handle}\t${ms_psettle}\t${ms_total}\t${OID}\t${pnote}" >>"$PAY_TSV" + metrics_report_coins "after-pay-${ptag}" || true + metrics_record_flow spent "$PAMT" || true + else + pnote="not settled order=$OID avail=${CUR}:${after}" + if [ "$IS_PMAX" = "1" ]; then + pstatus="CEILING_REJECT" + warn pay "pay $PAMT not settled (ceiling)" "$pnote" + echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t${ms_handle}\t${ms_psettle}\t${ms_total}\t${OID}\t${pnote}" >>"$PAY_TSV" + continue + fi + err pay "settle $PAMT" "$pnote" + pstatus="FAIL_SETTLE" + PAY_FAIL_N=$((PAY_FAIL_N + 1)) + FAIL_N_L=$((FAIL_N_L + 1)) + STOP_REASON="$pnote" + STOP_AMOUNT="$PAMT" + echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t${ms_handle}\t${ms_psettle}\t${ms_total}\t${OID}\t${pnote}" >>"$PAY_TSV" + metrics_report_coins "after-pay-fail-${ptag}" || true + break + fi + done + metrics_report_coins "after-pay-ladder" || true + info "pay summary" "ok=$PAY_OK_N fail=$PAY_FAIL_N tsv=$PAY_TSV" + fi +elif [ "${LADDER_PAY}" = "1" ] && [ "$FAIL_N_L" -gt 0 ]; then + warn pay "skipped pay ladder" "withdraw phase already failed" +fi + +ms_phase=$(python3 -c 'import sys,time; print(int((time.time()-float(sys.argv[1]))*1000))' "$SECTION_T0") + +# --- report --- +set_group report +section "ladder · report" +info "auto-account" "$ACCT_USER" +info "ok_rungs" "$OK_N" +info "fail_rungs" "$FAIL_N_L" +info "pay_ok" "$PAY_OK_N" +info "pay_fail" "$PAY_FAIL_N" +info "phase_ms" "$ms_phase" +info "tsv" "$TSV" +info "pay_tsv" "$PAY_TSV" + +# speed summary via python — free-text stop reason via env (JSON " in bank errors +# used to break shell argv quoting → "syntax error near unexpected token '('") +export LADDER_REPORT_STOP_AMOUNT="${STOP_AMOUNT:-}" +export LADDER_REPORT_STOP_REASON="${STOP_REASON:-}" +export LADDER_REPORT_MODE="${LADDER_MODE:-classic}" +export LADDER_REPORT_MAX_BEST="${MAX_BEST_AMT:-}" +export LADDER_REPORT_MAX_LO="${MAX_LO:-}" +export LADDER_REPORT_MAX_HI="${MAX_HI:-}" +# Human alt_unit_names (e.g. 8.15 Tera-GOA (GOA:…)) for report + final lines +export LADDER_REPORT_MAX_BEST_ALT="$(format_amount_alt "${MAX_BEST_AMT:-${CUR}:0}" 2>/dev/null || true)" +export LADDER_REPORT_MAX_LO_ALT="$(format_amount_alt "${CUR}:${MAX_LO:-0}" 2>/dev/null || true)" +export LADDER_REPORT_MAX_HI_ALT="$(format_amount_alt "${CUR}:${MAX_HI:-0}" 2>/dev/null || true)" +export LADDER_REPORT_ALT_UNITS="${ALT_UNITS_FILE:-}" +# Report must never abort the ladder (set -e): soft-fail print/JSON only. +python3 - "$TSV" "$PAY_TSV" "$JSON" "$OK_N" "$FAIL_N_L" "$PAY_OK_N" "$PAY_FAIL_N" "$ms_phase" "$ACCT_USER" "$CUR" <<'PY' || true +import csv, json, os, sys, statistics, traceback +from decimal import Decimal, InvalidOperation, ROUND_HALF_UP + +try: + tsv, pay_tsv, jpath = sys.argv[1:4] + ok_n, fail_n, pay_ok, pay_fail, phase_ms, acct, cur = sys.argv[4:11] + stop_amt = os.environ.get("LADDER_REPORT_STOP_AMOUNT") or None + stop_reason = os.environ.get("LADDER_REPORT_STOP_REASON") or None + mode = os.environ.get("LADDER_REPORT_MODE") or "classic" + max_best = os.environ.get("LADDER_REPORT_MAX_BEST") or None + max_lo = os.environ.get("LADDER_REPORT_MAX_LO") or None + max_hi = os.environ.get("LADDER_REPORT_MAX_HI") or None + max_best_alt = os.environ.get("LADDER_REPORT_MAX_BEST_ALT") or None + max_lo_alt = os.environ.get("LADDER_REPORT_MAX_LO_ALT") or None + max_hi_alt = os.environ.get("LADDER_REPORT_MAX_HI_ALT") or None + alt_path = os.environ.get("LADDER_REPORT_ALT_UNITS") or "" + alt = {} + if alt_path: + try: + _loaded = json.load(open(alt_path)) + if isinstance(_loaded, dict): + alt = {str(k): str(v) for k, v in _loaded.items()} + except Exception: + alt = {} + if not isinstance(alt, dict) or not alt: + alt = {"0": cur or "GOA"} + elif "0" not in alt: + alt = dict(alt) + alt["0"] = cur or "GOA" + + def fmt_num(v): + try: + if v == v.to_integral(): + return format(int(v), "d") + s = format(v.quantize(Decimal("0.00000001"), rounding=ROUND_HALF_UP), "f") + return s.rstrip("0").rstrip(".") + except Exception: + return str(v) + + def format_alt(amt): + """Human alt_unit_names; never raises.""" + try: + if not amt: + return "-" + s = str(amt).strip() + try: + if ":" in s: + c, v = s.split(":", 1) + val = Decimal(v) + else: + c, val = cur, Decimal(s) + except (InvalidOperation, ValueError, TypeError): + return s + au = alt if isinstance(alt, dict) else {"0": cur or "GOA"} + base_name = au.get("0") or c or cur or "GOA" + base_s = "%s:%s" % (c, fmt_num(val)) + if val == 0: + return "0 %s (%s)" % (base_name, base_s) + scales = [] + for k, name in au.items(): + try: + scales.append((int(k), str(name))) + except Exception: + continue + scales.sort(key=lambda x: -x[0]) + absval = abs(val) + chosen = None + for sc, name in scales: + try: + unit = Decimal(10) ** sc + if unit <= 0: + continue + coeff = absval / unit + if coeff >= 1: + chosen = (sc, name, coeff if val >= 0 else -coeff) + break + except Exception: + continue + if chosen is None: + return "%s %s (%s)" % (fmt_num(val), base_name, base_s) + sc, name, coeff = chosen + if sc == 0: + return "%s %s" % (fmt_num(val), name) + return "%s %s (%s)" % (fmt_num(coeff), name, base_s) + except Exception: + return str(amt) if amt else "-" + + def as_amt(s): + try: + if s is None or s == "": + return None + s = str(s).strip() + if ":" in s: + return s + return "%s:%s" % (cur, s) + except Exception: + return None + + def load_rows(path): + rows = [] + try: + with open(path, newline="") as f: + for row in csv.DictReader(f, delimiter="\t"): + rows.append(row) + except Exception: + pass + return rows + + rows = load_rows(tsv) + prows = load_rows(pay_tsv) + + def nums(rs, key): + out = [] + for row in rs: + try: + out.append(int(row[key])) + except Exception: + pass + return out + + def stats(xs): + if not xs: + return {"n": 0} + return { + "n": len(xs), + "min_ms": min(xs), + "max_ms": max(xs), + "avg_ms": int(sum(xs) / len(xs)), + "p50_ms": int(statistics.median(xs)), + } + + _max_block = None + if mode == "max": + _best = as_amt(max_best) if max_best else None + _lo = as_amt(max_lo) if max_lo is not None and str(max_lo) != "" else None + _hi = as_amt(max_hi) if max_hi is not None and str(max_hi) != "" else None + _max_block = { + "best": _best, + "best_alt": max_best_alt or (format_alt(_best) if _best else None), + "lo": max_lo, + "lo_alt": max_lo_alt or (format_alt(_lo) if _lo else None), + "hi": max_hi, + "hi_alt": max_hi_alt or (format_alt(_hi) if _hi else None), + } + + report = { + "currency": cur, + "mode": mode, + "auto_account": acct, + "ok_rungs": int(ok_n), + "fail_rungs": int(fail_n), + "pay_ok": int(pay_ok), + "pay_fail": int(pay_fail), + "max_search": _max_block, + "stopped_at_amount": stop_amt or None, + "stop_reason": stop_reason or None, + "phase_ms": int(phase_ms), + "timing": { + "mint": stats(nums(rows, "ms_mint")), + "accept": stats(nums(rows, "ms_accept")), + "confirm": stats(nums(rows, "ms_confirm")), + "settle": stats(nums(rows, "ms_settle")), + "rung_total": stats(nums(rows, "ms_total")), + "pay_order": stats(nums(prows, "ms_order")), + "pay_handle": stats(nums(prows, "ms_handle")), + "pay_settle": stats(nums(prows, "ms_settle")), + "pay_total": stats(nums(prows, "ms_total")), + }, + "rungs": rows, + "pays": prows, + } + try: + json.dump(report, open(jpath, "w"), indent=2) + print("JSON", jpath) + except Exception as e: + print("JSON write error:", e) + + print("--- withdraw speed (ms) ---") + for k in ("mint", "accept", "confirm", "settle", "rung_total"): + v = report["timing"][k] + if v.get("n"): + print(" %s n=%s min=%s p50=%s avg=%s max=%s" % ( + k.ljust(12), v["n"], v["min_ms"], v["p50_ms"], v["avg_ms"], v["max_ms"])) + print("--- pay speed (ms) ---") + for k in ("pay_order", "pay_handle", "pay_settle", "pay_total"): + v = report["timing"][k] + if v.get("n"): + print(" %s n=%s min=%s p50=%s avg=%s max=%s" % ( + k.ljust(12), v["n"], v["min_ms"], v["p50_ms"], v["avg_ms"], v["max_ms"])) + print("--- withdraw rungs ---") + for row in rows: + try: + amt = row.get("amount") or "" + aalt = format_alt(amt) if amt else "" + print(" %2s %-18s %-14s total=%sms %s" % ( + row.get("rung"), amt, row.get("status"), row.get("ms_total"), aalt)) + except Exception as e: + print(" ? row-print-error %s" % (e,)) + print("--- pay rungs ---") + for row in prows: + try: + amt = row.get("amount") or "" + aalt = format_alt(amt) if amt else "" + print(" %2s %-18s %-14s total=%sms oid=%s %s" % ( + row.get("rung"), amt, row.get("status"), row.get("ms_total"), + row.get("oid"), aalt)) + except Exception as e: + print(" ? pay-row-print-error %s" % (e,)) + if mode == "max": + print("--- max-search ---") + try: + ba = (max_best_alt or format_alt(as_amt(max_best) or "")) if max_best else "-" + la = (max_lo_alt or format_alt(as_amt(max_lo) or "")) if max_lo is not None and str(max_lo) != "" else "-" + ha = (max_hi_alt or format_alt(as_amt(max_hi) or "")) if max_hi is not None and str(max_hi) != "" else "-" + print(" best=%s" % (max_best or "-")) + print(" best_alt=%s" % ba) + print(" lo=%s" % (max_lo if max_lo is not None else "-")) + print(" lo_alt=%s" % la) + print(" hi=%s" % (max_hi if max_hi is not None else "-")) + print(" hi_alt=%s" % ha) + except Exception as e: + print(" max-search print error: %s" % (e,)) + if stop_amt: + try: + print("STOPPED at %s · %s: %s" % (stop_amt, format_alt(stop_amt), stop_reason)) + except Exception: + print("STOPPED at %s: %s" % (stop_amt, stop_reason)) + else: + print("Completed without hard failure (or budget stop without fail).") +except Exception: + traceback.print_exc() +sys.exit(0) +PY + +wcli balance 2>&1 | tee "$REPORT_DIR/balance-final.out" | tail -20 || true + +set_group load +section "ladder · load snapshot (after withdraws)" +metrics_report_load "$LOAD_AFTER" "ladder-end" || true +if [ -f "$LOAD_BEFORE" ] && [ -f "$LOAD_AFTER" ]; then + section "metrics · ladder load delta" + metrics_print_load_delta "$LOAD_BEFORE" "$LOAD_AFTER" || true +fi +# Speed timings → metrics overall (min/p50/avg/max per phase) +if [ -f "$JSON" ]; then + python3 - "$JSON" "${METRICS_DIR}/perf-summary.json" <<'PY' 2>/dev/null || true +import json, sys +rep = json.load(open(sys.argv[1])) +out = {} +for k, v in (rep.get("timing") or {}).items(): + if isinstance(v, dict) and v.get("n"): + out[k] = v +json.dump(out, open(sys.argv[2], "w"), indent=2) +PY +fi +export METRICS_WITHDRAW_TSV="$TSV" +export METRICS_PAY_TSV="$PAY_TSV" +metrics_report_coins "ladder-end" || true +metrics_print_overall "ladder final" || true + +# Keep scratch if LADDER_REPORT_DIR set; else copy key files to /tmp +if [ -z "${LADDER_REPORT_DIR:-}" ]; then + KEEP="/tmp/amount-ladder-report-$(date +%Y%m%d-%H%M%S)" + mkdir -p "$KEEP" + cp -a "$TSV" "$PAY_TSV" "$JSON" "$SCRATCH/auto-account.json" "$SCRATCH/ladder-plan.txt" \ + "$SCRATCH/ladder-wd-plan.txt" "$SCRATCH/ladder-pay-plan.txt" \ + "$REPORT_DIR/balance-final.out" "$LOAD_BEFORE" "$LOAD_AFTER" "$KEEP/" 2>/dev/null || true + info "report_dir" "$KEEP" + echo "$KEEP" >"$SCRATCH/KEEP_PATH" +fi + +if [ "$FAIL_N_L" -gt 0 ]; then + blocker "ladder" "stopped at ${STOP_AMOUNT:-?} — ${STOP_REASON:-error}" + exit 1 +fi +if [ "$OK_N" -eq 0 ]; then + blocker "ladder" "no successful withdraw rungs" + exit 1 +fi +if [ "${LADDER_MODE}" = "max" ] && [ -n "${MAX_BEST_AMT:-}" ]; then + ok "ladder finished mode=max best=$(format_amount_alt "$MAX_BEST_AMT") · lo=$(format_amount_alt "${CUR}:${MAX_LO}") · hi=$(format_amount_alt "${CUR}:${MAX_HI}") · withdraw_ok=$OK_N phase=${ms_phase}ms" +else + ok "ladder finished mode=${LADDER_MODE} withdraw_ok=$OK_N pay_ok=$PAY_OK_N phase=${ms_phase}ms" +fi +exit 0 diff --git a/check_goa_ladder.sh b/check_goa_ladder.sh index 560537d..c32f686 100755 --- a/check_goa_ladder.sh +++ b/check_goa_ladder.sh @@ -1,1404 +1,5 @@ #!/usr/bin/env bash -# check_goa_ladder.sh — withdraw/pay amount ladder (GOA + stage TESTPAYSAN) -# -# Flow (bank landings): -# 1) GET /intro/auto-account.json → personal *account-* (balance 0) -# 2) Mint pool withdrawals as explorer + confirm when selected -# 3) wallet-cli accept-uri only (no run-until-done) -# 4) bank confirm when selected; settle = balance + transfer_done -# -# Amounts: random strictly increasing; pins 0, max-1, max. -# Phase A: withdraw ladder (cumulative wallet). Phase B: pay ladder. -# -# Stacks: -# GOA — LADDER_MAX_AMOUNT = libeufin ceiling; explorer from koopa-admin-secrets -# TESTPAYSAN stage — auto max_wire from bank /config, min denom from /keys; -# explorer from stagepaysan secrets / SSH -# -# Usage: -# ./taler-monitoring.sh ladder -# ./taler-monitoring.sh -d stage.lefrancpaysan.ch ladder -# LADDER_MAX_AMOUNT=100 LADDER_STEPS=7 ./taler-monitoring.sh -d stage.lefrancpaysan.ch ladder -# -# Env (see body): LADDER_STEPS, LADDER_MAX_AMOUNT, LADDER_MIN_AMOUNT, LADDER_STACK_AUTO, -# LADDER_PAY, EXP_PW / EXP_PW_FILE, CLI_JS, MERCHANT_INSTANCE, … -set -euo pipefail +# Compat shim — use check_amount_ladder.sh (generic currency amount ladder). +# Kept so older host-agent / docs invoking check_goa_ladder.sh still work. ROOT=$(cd "$(dirname "$0")" && pwd) -# shellcheck source=lib.sh -source "$ROOT/lib.sh" -# When invoked standalone (not via taler-monitoring.sh), still load secrets.env -load_monitoring_secrets_env 2>/dev/null || true - -# Area ladder.* — withdraw/pay amount ladder (GOA + TESTPAYSAN) -# Groups: ladder.plan / ladder.load / ladder.withdraw / ladder.pay / ladder.report -set_area ladder -set_group plan -SECTION_T0=$(date +%s) -now_ms() { python3 -c 'import time; print(int(time.time()*1000))'; } -elapsed_ms() { - # elapsed_ms START_MS - python3 -c 'import sys; print(int(sys.argv[1]) - int(sys.argv[2]))' "$(now_ms)" "$1" -} - -: "${LADDER_TIMEOUT_S:=3600}" -: "${LADDER_SETTLE_ROUNDS:=18}" -: "${LADDER_SETTLE_SLEEP:=2}" -: "${EXP_USER:=explorer}" -# EXP_PW_FILE / EXP_PW optional overrides; otherwise read_secret + SECRETS_ROOT / stage SSH -: "${EXP_PW_FILE:=}" -: "${EXP_PW:=}" -: "${CLI_JS:=}" -# GOA libeufin amount ceiling (hacktivism). Stage auto-replaces via bank /config. -: "${LADDER_MAX_AMOUNT:=4503599627370496}" -: "${LADDER_STEPS:=23}" -: "${LADDER_LOAD:=1}" -: "${LADDER_PAY:=1}" -: "${LADDER_STACK_AUTO:=1}" -# Withdraw mids = pay mids × scale (so wallet can afford the pay ladder) -: "${LADDER_WITHDRAW_SCALE:=1.5}" -# Floor for random mids (must be ≥ smallest exchange coin; GOA min denom ≈ 0.000001) -: "${LADDER_MIN_AMOUNT:=0.000001}" -: "${LADDER_CONFIRM_POLLS:=40}" -: "${LADDER_PAY_SETTLE_ROUNDS:=6}" -: "${MERCHANT_INSTANCE:=goa-demo-cp4zqk}" -# Public variable template for stage pays (optional; fixed templates used when amount maps) -: "${LADDER_PAY_TEMPLATE:=}" -: "${LADDER_PAY_TEMPLATE_INSTANCE:=fermes-des-collines}" - -GOA_LADDER_CEILING="4503599627370496" - -# Resolve wallet-cli .mjs (no hardcoded laptop path) -if [ -z "${CLI_JS}" ] || [ ! -f "${CLI_JS}" ]; then - CLI_JS=$(find_wallet_cli 2>/dev/null || true) -fi -if [ -z "${CLI_JS}" ] || [ ! -f "${CLI_JS}" ]; then - # allow PATH wrapper as last resort (wcli falls back to taler-wallet-cli) - CLI_JS="" -fi - -CUR="${EXPECT_CURRENCY:-GOA}" -BANK="${BANK_PUBLIC%/}" -EX="${EXCHANGE_PUBLIC%/}/" -MER="${MERCHANT_PUBLIC%/}" -INST="${MERCHANT_INSTANCE}" - -# --- Stack-specific ladder maxima (TESTPAYSAN stage) --- -# Uses bank max_wire_transfer_amount and exchange min denom when still on GOA defaults. -apply_ladder_stack_defaults() { - [ "${LADDER_STACK_AUTO}" = "1" ] || return 0 - if [ "${CUR}" != "TESTPAYSAN" ]; then - case "${TALER_DOMAIN:-}" in - stage.lefrancpaysan.ch|stage.bank.lefrancpaysan.ch|stage.exchange.lefrancpaysan.ch|stage.monnaie.lefrancpaysan.ch) - CUR="TESTPAYSAN" - ;; - *) return 0 ;; - esac - fi - - local bank_cfg max_wire min_denom - bank_cfg=$(curl -sS -m 12 "${BANK}/config" 2>/dev/null || true) - max_wire=$(printf '%s' "$bank_cfg" | python3 -c ' -import json,sys -try: - d=json.load(sys.stdin) -except Exception: - print("") - raise SystemExit(0) -a=d.get("max_wire_transfer_amount") or "" -print(a.split(":",1)[-1] if a else "") -' 2>/dev/null || true) - - min_denom=$(curl -sS -m 25 -H 'Accept: application/json' "${EX%/}/keys" 2>/dev/null | python3 -c ' -import json,sys -try: - d=json.load(sys.stdin) -except Exception: - print("") - raise SystemExit(0) -den=d.get("denoms") or d.get("denominations") or [] -vals=[] -for x in den if isinstance(den,list) else []: - v=x.get("value") or x.get("amount") or "" - if not v: continue - try: vals.append(float(str(v).split(":")[-1])) - except Exception: pass -print(min(vals) if vals else "") -' 2>/dev/null || true) - - # Fallback maxima if public config unreachable - [ -n "$max_wire" ] || max_wire="2000" - [ -n "$min_denom" ] || min_denom="0.01" - - # Only rewrite when caller left GOA defaults (explicit LADDER_MAX_AMOUNT=… kept) - if [ -n "$max_wire" ] && { [ "${LADDER_MAX_AMOUNT}" = "${GOA_LADDER_CEILING}" ] || [ "${LADDER_MAX_AMOUNT}" = "4503599627370496" ]; }; then - LADDER_MAX_AMOUNT="$max_wire" - fi - if [ -n "$min_denom" ] && { [ "${LADDER_MIN_AMOUNT}" = "0.000001" ] || [ "${LADDER_MIN_AMOUNT}" = "0.00000001" ]; }; then - LADDER_MIN_AMOUNT="$min_denom" - fi - # Slightly fewer rungs on stage (full GOA 23 still ok if user set LADDER_STEPS) - if [ "${LADDER_STEPS}" = "23" ]; then - LADDER_STEPS=15 - fi - # Stage farmer shops: private goa-demo instance does not exist - if [ "${MERCHANT_INSTANCE}" = "goa-demo-cp4zqk" ]; then - MERCHANT_INSTANCE="${LADDER_PAY_TEMPLATE_INSTANCE:-fermes-des-collines}" - INST="$MERCHANT_INSTANCE" - fi - # Prefer public templates for pays when no merchant token (set later) - : "${E2E_USE_TEMPLATES:=1}" - export E2E_USE_TEMPLATES - export LADDER_MAX_AMOUNT LADDER_MIN_AMOUNT LADDER_STEPS MERCHANT_INSTANCE - info "ladder stack" "TESTPAYSAN · max=${CUR}:${LADDER_MAX_AMOUNT} min=${CUR}:${LADDER_MIN_AMOUNT} steps=${LADDER_STEPS} (from bank max_wire / keys denoms)" -} -apply_ladder_stack_defaults -SCRATCH=$(mktemp -d) -WDB="$SCRATCH/wallet.sqlite3" -REPORT_DIR="${LADDER_REPORT_DIR:-$SCRATCH}" -mkdir -p "$REPORT_DIR" -TSV="$REPORT_DIR/ladder-results.tsv" -PAY_TSV="$REPORT_DIR/ladder-pay-results.tsv" -JSON="$REPORT_DIR/ladder-report.json" -LOAD_BEFORE="$REPORT_DIR/load-before.json" -LOAD_AFTER="$REPORT_DIR/load-after.json" -echo -e "rung\trange\tamount\tstatus\tms_mint\tms_accept\tms_confirm\tms_settle\tms_total\twid\tnote" >"$TSV" -echo -e "rung\trange\tamount\tstatus\tms_order\tms_handle\tms_settle\tms_total\toid\tnote" >"$PAY_TSV" - -ladder_over() { - local now - now=$(date +%s) - [ $((now - SECTION_T0)) -ge "$LADDER_TIMEOUT_S" ] -} - -wcli() { - # taler-helper-sqlite3 is Python ≥3.11; prefer Homebrew/local python on macOS - # so Apple /usr/bin/python3 3.9 does not FATAL the sqlite backend (no reservePub). - local _path="${PATH:-}" - case ":${_path}:" in - *:/opt/homebrew/bin:*) ;; - *) _path="/opt/homebrew/bin:/usr/local/bin:${_path}" ;; - esac - if [ -n "${CLI_JS:-}" ] && [ -f "$CLI_JS" ]; then - PATH="$_path" node "$CLI_JS" --wallet-db="$WDB" --no-throttle "$@" - else - PATH="$_path" taler-wallet-cli --wallet-db="$WDB" --no-throttle "$@" - fi -} - -# Explorer pool password: env → file override → stack secrets → SSH -resolve_explorer_pw() { - local pw="" f="" host="" - if [ -n "${EXP_PW:-}" ]; then - printf '%s' "$EXP_PW" - return 0 - fi - if [ -n "${EXP_PW_FILE:-}" ] && [ -f "$EXP_PW_FILE" ]; then - tr -d '\n\r' <"$EXP_PW_FILE" - return 0 - fi - - # Stage TESTPAYSAN — never use GOA koopa-admin-secrets explorer pw first - if [ "${CUR}" = "TESTPAYSAN" ] \ - || [[ "${TALER_DOMAIN:-}" == stage.*lefrancpaysan* ]] \ - || [[ "${TALER_DOMAIN:-}" == *stage.lefrancpaysan* ]]; then - for f in \ - ${FRANCPAYSAN_SECRETS:+"${FRANCPAYSAN_SECRETS}/stage/bank-explorer-password.txt"} \ - "${HOME}/.config/taler-landing/stage-bank-explorer-password.txt" - do - [ -n "$f" ] && [ -f "$f" ] || continue - tr -d '\n\r' <"$f" - return 0 - done - _remote_exp="${FRANCPAYSAN_REMOTE_BANK_EXPLORER_SECRET:-}" - [ -z "$_remote_exp" ] && [ -n "${FRANCPAYSAN_REMOTE_SECRETS_ROOT:-}" ] && \ - _remote_exp="${FRANCPAYSAN_REMOTE_SECRETS_ROOT}/bank/secrets/bank-explorer-password.txt" - for host in ${INSIDE_SSH:+"$INSIDE_SSH"} ${FRANCPAYSAN_SSH:+"$FRANCPAYSAN_SSH"}; do - [ -n "$host" ] && [ -n "$_remote_exp" ] || continue - pw=$(ssh -o BatchMode=yes -o ConnectTimeout=12 "$host" \ - "tr -d '\\n\\r' <${_remote_exp} 2>/dev/null || sudo tr -d '\\n\\r' <${_remote_exp} 2>/dev/null" \ - 2>/dev/null || true) - if [ -n "$pw" ]; then - printf '%s' "$pw" - return 0 - fi - done - return 1 - fi - - # GOA / default: SECRETS_ROOT / ~/.config / koopa SSH - if pw=$(read_secret "taler-bank/bank-explorer-password.txt" 2>/dev/null) && [ -n "$pw" ]; then - printf '%s' "$pw" - return 0 - fi - for f in \ - "${SECRETS_ROOT:+${SECRETS_ROOT}/taler-bank/bank-explorer-password.txt}" \ - "${HOME}/.config/taler-landing/bank-explorer-password.txt" - do - [ -n "$f" ] && [ -f "$f" ] || continue - tr -d '\n\r' <"$f" - return 0 - done - return 1 -} - -wallet_avail() { - wcli balance 2>/dev/null | python3 -c ' -import json,sys -t=sys.stdin.read() -dec=json.JSONDecoder() -i=t.find("{") -if i<0: - print("0"); raise SystemExit -try: - d,_=dec.raw_decode(t,i) -except Exception: - # last-resort: brace slice (may fail on trailing logs) - try: - d=json.loads(t[i:t.rfind("}")+1]) - except Exception: - print("0"); raise SystemExit -cur=sys.argv[1] -for b in d.get("balances") or []: - a=b.get("available") or "" - if a.startswith(cur+":"): - print(a.split(":",1)[1]); raise SystemExit -print("0") -' "$CUR" 2>/dev/null || echo "0" -} - -# Build paired pay + withdraw ladders (same step count, shared random shape). -# Pay: [0] + log-uniform mids + [max-1] + [max] -# Wd: [0] + max(mid, mid×scale) mids + [max-1] + [max] (mids higher → enough to spend) -# Writes: $1 = withdraw list file, $2 = pay list file (space-separated CUR:amt lines as one line each) -build_ladder_pair() { - local wd_out="$1" pay_out="$2" - python3 - <<'PY' "$CUR" "${LADDER_MAX_AMOUNT}" "${LADDER_STEPS}" "${LADDER_MIN_AMOUNT}" "${LADDER_WITHDRAW_SCALE}" "$wd_out" "$pay_out" -import math, random, sys -from decimal import Decimal, ROUND_HALF_UP, ROUND_UP -from pathlib import Path - -cur = sys.argv[1] -max_amt = Decimal(sys.argv[2]) -steps = max(1, int(sys.argv[3])) -min_amt = Decimal(sys.argv[4]) -scale = Decimal(sys.argv[5]) -wd_path, pay_path = Path(sys.argv[6]), Path(sys.argv[7]) -if min_amt <= 0: - min_amt = Decimal("0.000001") -if min_amt >= max_amt: - min_amt = max_amt / Decimal(1000) -if scale < 1: - scale = Decimal(1) -max_m1 = max_amt - Decimal(1) if max_amt > 1 else max_amt - -def fmt(v: Decimal) -> str: - q = v.quantize(Decimal("0.00000001"), rounding=ROUND_HALF_UP) - if q == q.to_integral(): - return format(int(q), "d") - return format(q, "f").rstrip("0").rstrip(".") - -def quant(v: Decimal) -> Decimal: - if v >= 1: - return v.quantize(Decimal(1), rounding=ROUND_HALF_UP) - if v < min_amt: - return min_amt - return v.quantize(Decimal("0.00000001"), rounding=ROUND_UP) - -def amt(v: Decimal) -> str: - return "%s:%s" % (cur, fmt(v)) - -def make_mids(n_mid: int, hi_cap: Decimal): - if n_mid <= 0 or hi_cap <= min_amt: - return [] - lo, hi = float(min_amt), float(hi_cap) * 0.999999 - if hi <= lo: - hi = lo * 10 - cuts = sorted(math.exp(random.uniform(math.log(lo), math.log(hi))) for _ in range(n_mid)) - out, prev = [], Decimal(0) - for c in cuts: - v = quant(Decimal(str(c))) - if v <= prev: - step = min_amt if prev < 1 else max(prev * Decimal("1e-6"), Decimal(1)) - v = quant(prev + step) - if v >= hi_cap: - v = quant(hi_cap - (Decimal(1) if hi_cap > 1 else min_amt)) - if v <= prev or v >= hi_cap: - continue - out.append(v) - prev = v - return out - -# Build withdraw ladder first (full range), then pay mids = withdraw/scale -# so each pay mid is always cheaper than the matching withdraw mid. -if steps == 1: - wd_vals = [max_amt] -elif steps == 2: - wd_vals = [Decimal(0), max_amt] -else: - n_mid = max(0, steps - 3) - wd_vals = [Decimal(0)] + make_mids(n_mid, max_m1) + [max_m1, max_amt] - while len(wd_vals) > steps and len(wd_vals) > 3: - wd_vals.pop(len(wd_vals) // 2) - while len(wd_vals) < steps and len(wd_vals) >= 2: - i = max(1, len(wd_vals) - 2) - a, b = wd_vals[i - 1], wd_vals[i] - if a <= 0: - m = max(min_amt, b / 2 if b > 0 else min_amt) - else: - m = (a * b).sqrt() if a * b > 0 else (a + b) / 2 - m = quant(m) - if m <= a or m >= b: - break - wd_vals.insert(i, m) - wd_vals = wd_vals[:steps] - -pay_vals = [] -prev_p = Decimal(-1) -for i, w in enumerate(wd_vals): - is_first = i == 0 - is_last = i == len(wd_vals) - 1 - is_m1 = (not is_last) and w == max_m1 and i == len(wd_vals) - 2 - if is_first and w == 0: - pay_vals.append(Decimal(0)) - prev_p = Decimal(0) - continue - if is_last: - pay_vals.append(max_amt) - continue - if is_m1 or w == max_m1: - pay_vals.append(max_m1) - prev_p = max_m1 - continue - # pay mid = withdraw / scale (strictly less funding needed per step) - p = quant(w / scale) if scale > 0 else w - if p < min_amt and w >= min_amt: - p = min_amt - if p <= prev_p: - p = quant(prev_p + (min_amt if prev_p < 1 else Decimal(1))) - if p >= w: - # keep pay strictly below this withdraw rung when possible - p = quant(w - (Decimal(1) if w > 1 else min_amt)) if w > prev_p else prev_p - if p <= prev_p: - p = prev_p # flat ok only if stuck; still ≤ w - pay_vals.append(p) - prev_p = p - -assert len(pay_vals) == len(wd_vals) -# sanity: every non-pin pay mid ≤ matching withdraw mid -for i, (p, w) in enumerate(zip(pay_vals, wd_vals)): - if i == 0 or i >= len(wd_vals) - 2: - continue - if p > w: - pay_vals[i] = w - -wd_path.write_text(" ".join(amt(v) for v in wd_vals) + "\n") -pay_path.write_text(" ".join(amt(v) for v in pay_vals) + "\n") -print(" ".join(amt(v) for v in wd_vals)) -PY -} - -# shellcheck source=metrics.sh -source "$ROOT/metrics.sh" -METRICS_DIR="$REPORT_DIR" -ALT_UNITS_FILE="${REPORT_DIR}/alt_unit_names.json" -export METRICS_DIR CUR WDB CLI_JS ALT_UNITS_FILE -# Ladder can disable host load without killing coin metrics -if [ "${LADDER_LOAD:-1}" = "0" ]; then - METRICS_LOAD=0 - export METRICS_LOAD -fi - -section "ladder · ${CUR} withdraw (0 → random → max · ${LADDER_STEPS} steps)" -info "bank" "$BANK" -info "exchange" "$EX" -info "currency" "$CUR" -# alt_unit_names for human amounts (Kilo-GOA / Mega-GOA / … + GOA in parentheses) -if metrics_load_alt_units "${EX%/}/config"; then - info "alt_unit_names" "from ${EX%/}/config → $ALT_UNITS_FILE" -else - warn "alt_unit_names" "using built-in SI fallback ($ALT_UNITS_FILE)" -fi -info "budget" "${LADDER_TIMEOUT_S}s" -_max_m1=$(python3 -c 'import sys; from decimal import Decimal; m=Decimal(sys.argv[1]); print(m-1 if m>1 else m)' "${LADDER_MAX_AMOUNT}" 2>/dev/null || echo "${LADDER_MAX_AMOUNT}-1") -info "steps" "${LADDER_STEPS} (0 + random≥${LADDER_MIN_AMOUNT} + max-1=${CUR}:${_max_m1} + max=${CUR}:${LADDER_MAX_AMOUNT})" -info "withdraw_scale" "${LADDER_WITHDRAW_SCALE}× pay mids (fund pay ladder)" -info "pay_phase" "$([ "${LADDER_PAY}" = "1" ] && echo enabled || echo disabled)" -info "currency/domain" "${CUR} · ${TALER_DOMAIN:-?} · bank=${BANK}" - -set_group load -section "ladder · load snapshot (before withdraws)" -metrics_report_load "$LOAD_BEFORE" "ladder-start" || true -# Fresh wallet per rung — baseline empty (or last-rung DB if re-used later) -metrics_report_coins "ladder-start" || true - -if ! EXP_PW=$(resolve_explorer_pw); then - err bank "explorer password missing" \ - "set EXP_PW / EXP_PW_FILE or SECRETS_ROOT=…/koopa/host-root (taler-bank/bank-explorer-password.txt)${SECRETS_ROOT:+ · SECRETS_ROOT=${SECRETS_ROOT}}" - secrets_hint 2>/dev/null || true - exit 1 -fi -if [ -n "${SECRETS_ROOT:-}" ]; then - info "explorer secret" "resolved (SECRETS_ROOT=${SECRETS_ROOT} · stage uses stagepaysan secret first when CUR=TESTPAYSAN)" -else - info "explorer secret" "resolved via EXP_PW / EXP_PW_FILE / stage SSH / koopa" -fi -if [ -n "${CLI_JS:-}" ] && [ -f "${CLI_JS}" ]; then - info "wallet-cli" "$CLI_JS" -else - info "wallet-cli" "PATH taler-wallet-cli ($(command -v taler-wallet-cli 2>/dev/null || echo missing))" -fi - -# --- auto-account --- -t0=$(now_ms) -if ! curl -sS -m 30 -o "$SCRATCH/auto-account.json" "${BANK}/intro/auto-account.json"; then - err bank "auto-account.json unreachable" - exit 1 -fi -ms_auto=$(elapsed_ms "$t0") -if ! python3 -c 'import json; d=json.load(open("'"$SCRATCH"'/auto-account.json")); assert d.get("ok") or d.get("username")' 2>/dev/null; then - err bank "auto-account create failed" "$(head -c 120 "$SCRATCH/auto-account.json" | tr '\n' ' ')" - exit 1 -fi -ACCT_USER=$(python3 -c 'import json; print(json.load(open("'"$SCRATCH"'/auto-account.json"))["username"])') -ok "auto-account ${ACCT_USER} (${ms_auto}ms) — personal ${CUR}:0; pool=explorer" -info "auto-account password" "(see $SCRATCH/auto-account.json — not logged)" - -# --- explorer token --- -t0=$(now_ms) -TOK=$(curl -sS -m 20 -u "${EXP_USER}:${EXP_PW}" \ - -H 'Content-Type: application/json' \ - -d '{"scope":"readwrite","duration":{"d_us":3600000000}}' \ - "${BANK}/accounts/${EXP_USER}/token" \ - | python3 -c 'import json,sys; print(json.load(sys.stdin)["access_token"])') -ms_tok=$(elapsed_ms "$t0") -[ -n "$TOK" ] || { err bank "explorer token failed"; exit 1; } -ok "explorer token (${ms_tok}ms)" - -# ONE cumulative wallet for all withdraws + pays (need balance to spend). -# force-select uses *last* reserve_pub from the current accept output. -wallet_prepare() { - local label="${1:-wallet}" - WDB="$SCRATCH/wallet-${label}.sqlite3" - export WDB - if [ ! -f "$WDB" ]; then - wcli exchanges add "$EX" >"$SCRATCH/ex-add-$label.out" 2>&1 || true - wcli exchanges update "$EX" >"$SCRATCH/ex-upd-$label.out" 2>&1 || true - wcli exchanges accept-tos "$EX" >"$SCRATCH/ex-tos-$label.out" 2>&1 || true - fi -} - -t0=$(now_ms) -rm -f "$SCRATCH/wallet-main.sqlite3" -wallet_prepare "main" -ms_tos=$(elapsed_ms "$t0") -ok "wallet exchange + ToS (${ms_tos}ms) — cumulative DB for withdraw+pay (no run-until-done)" - -build_ladder_pair "$SCRATCH/ladder-wd-plan.txt" "$SCRATCH/ladder-pay-plan.txt" -LADDER_LIST=$(tr -d '\n' <"$SCRATCH/ladder-wd-plan.txt") -PAY_LIST=$(tr -d '\n' <"$SCRATCH/ladder-pay-plan.txt") -: "${LADDER_MAX_RUNGS:=99}" -# shellcheck disable=SC2086 -set -- $LADDER_LIST -if [ "$#" -gt "$LADDER_MAX_RUNGS" ]; then - # shellcheck disable=SC2046 - set -- $(printf '%s\n' "$@" | head -n "$LADDER_MAX_RUNGS") -fi -LADDER_N=$# -info "withdraw plan" "$*" -info "withdraw plan (alt)" "$(format_amount_list_alt "$@")" -info "pay plan" "$PAY_LIST" -# shellcheck disable=SC2086 -info "pay plan (alt)" "$(format_amount_list_alt $PAY_LIST)" -printf '%s\n' "$@" >"$SCRATCH/ladder-plan.txt" -printf '%s\n' $PAY_LIST >"$SCRATCH/ladder-pay-plan-lines.txt" 2>/dev/null || true - -OK_N=0 -FAIL_N_L=0 -PAY_OK_N=0 -PAY_FAIL_N=0 -STOP_REASON="" -STOP_AMOUNT="" -declare -a RUNG_JSON=() - -set_group withdraw -section "ladder · phase A · withdraw (${LADDER_N} rungs)" -rung=0 -for AMT in "$@"; do - rung=$((rung + 1)) - if ladder_over; then - STOP_REASON="timeout budget ${LADDER_TIMEOUT_S}s" - warn ladder "time budget exhausted" \ - "problem: LADDER_TIMEOUT_S=${LADDER_TIMEOUT_S}s reached after ${OK_N} ok rungs; remaining amounts not tried" - break - fi - - section "ladder · rung $rung $AMT · $(format_amount_alt "$AMT")" - tag=$(printf '%s' "$AMT" | tr '.:' '__') - # TSV "range" column: fixed pins at ends, random mids (refined after AMT_NUM) - if [ "$rung" -eq 1 ]; then - range_note="pin:0" - elif [ "$rung" -eq "$LADDER_N" ]; then - range_note="pin:max" - elif [ "$rung" -eq $((LADDER_N - 1)) ] && [ "$LADDER_N" -ge 3 ]; then - range_note="pin:max-1" - else - range_note="random" - fi - t_rung=$(now_ms) - ms_mint=0 ms_accept=0 ms_confirm=0 ms_settle=0 - note="" - status="FAIL" - WID="-" - FORCE_SEL_409_LOGGED=0 - - # keep cumulative main wallet - wallet_prepare "main" - - # mint from explorer pool - t0=$(now_ms) - code=$(curl -sS -m 30 -o "$SCRATCH/wd-$tag.json" -w '%{http_code}' \ - -H "Authorization: Bearer ${TOK}" \ - -H 'Content-Type: application/json' \ - -d "{\"amount\":\"${AMT}\"}" \ - "${BANK}/accounts/${EXP_USER}/withdrawals") - ms_mint=$(elapsed_ms "$t0") - WID=$(python3 -c 'import json;d=json.load(open("'"$SCRATCH"'/wd-'"$tag"'.json"));print(d.get("withdrawal_id") or "")' 2>/dev/null || true) - URI=$(python3 -c 'import json;u=json.load(open("'"$SCRATCH"'/wd-'"$tag"'.json")).get("taler_withdraw_uri") or "";print(u.replace(":443/","/"))' 2>/dev/null || true) - # numeric amount (for zero / settle / ceiling special-cases) - AMT_NUM=$(python3 -c 'import sys; print(sys.argv[1].split(":",1)[-1])' "$AMT") - IS_ZERO=0 - python3 -c 'import sys; from decimal import Decimal; sys.exit(0 if Decimal(sys.argv[1])==0 else 1)' "$AMT_NUM" 2>/dev/null && IS_ZERO=1 - IS_MAX_PIN=0 - IS_MAX_M1_PIN=0 - if [ "$AMT_NUM" = "${LADDER_MAX_AMOUNT}" ]; then - IS_MAX_PIN=1 - range_note="pin:max" - elif [ "$AMT_NUM" = "$((LADDER_MAX_AMOUNT - 1))" ] 2>/dev/null || \ - [ "$AMT_NUM" = "$(python3 -c 'print(int("'"$LADDER_MAX_AMOUNT"'")-1)')" ]; then - IS_MAX_M1_PIN=1 - range_note="pin:max-1" - fi - - if [ "$code" != "200" ] && [ "$code" != "201" ] || [ -z "$WID" ] || [ -z "$URI" ]; then - # strip quotes so STOP_REASON never breaks later shell/python argv - note="mint HTTP $code $(head -c 100 "$SCRATCH/wd-$tag.json" 2>/dev/null | tr '\n\"' ' ')" - ms_total=$(elapsed_ms "$t_rung") - if [ "$IS_ZERO" = "1" ]; then - # Probe only: bank may reject GOA:0 — record and continue ladder - status="ZERO_REJECT" - note="zero-withdraw rejected (expected possible): $note" - warn bank "mint $AMT rejected" \ - "problem: bank will not create a GOA:0 withdrawal (zero amount probe). Ladder continues. detail: $note" - echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" - OK_N=$((OK_N + 1)) - continue - fi - # Absolute max is a ceiling probe — bank often returns SQL P0001/5110; do not abort. - if [ "$IS_MAX_PIN" = "1" ]; then - status="CEILING_REJECT" - note="absolute max rejected (ceiling probe; max-1 is the hard pin): $note" - warn bank "mint $AMT rejected (ceiling)" \ - "problem: bank rejects absolute LADDER_MAX_AMOUNT (often SQL P0001/5110). max-1 rung is the last expected success. Ladder continues to report. detail: $note" - echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" - continue - fi - # max-1 can also hit the same libeufin SQL ceiling (5110/P0001) on live GOA — - # treat as soft ceiling when mid rungs already passed; do not block the report. - if [ "$IS_MAX_M1_PIN" = "1" ] && echo "$note" | grep -qE '5110|P0001'; then - status="CEILING_REJECT" - note="max-1 rejected (same ceiling as absolute max): $note" - warn bank "mint $AMT rejected (max-1 ceiling)" \ - "problem: bank rejects max-1 with SQL P0001/5110 (pool ceiling). Mid-rung OK_BANK/OK still count. Ladder continues. detail: $note" - echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" - continue - fi - err bank "mint $AMT failed" "$note" - status="FAIL_MINT" - STOP_REASON="$note" - STOP_AMOUNT="$AMT" - echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" - FAIL_N_L=$((FAIL_N_L + 1)) - break - fi - ok "mint $AMT ($WID) ${ms_mint}ms" - - before=$(wallet_avail) - - # accept - t0=$(now_ms) - if wcli withdraw accept-uri --exchange "$EX" "$URI" >"$SCRATCH/accept-$tag.out" 2>&1; then - ms_accept=$(elapsed_ms "$t0") - ok "accept-uri $AMT ${ms_accept}ms" - else - ms_accept=$(elapsed_ms "$t0") - note="accept-uri failed: $(tail -c 200 "$SCRATCH/accept-$tag.out" | tr '\n' ' ')" - ms_total=$(elapsed_ms "$t_rung") - # Wallet 7006: no denominations for this amount (0, sub-denom dust, etc.) — warn & continue - if [ "$IS_ZERO" = "1" ] || grep -qE 'code: 7006|"code"[[:space:]]*:[[:space:]]*7006|No denominations could be selected' \ - "$SCRATCH/accept-$tag.out" 2>/dev/null; then - if [ "$IS_ZERO" = "1" ]; then - status="ZERO_SKIP" - note="zero-withdraw skip (7006 / no denoms): $note" - warn wallet "accept $AMT skipped" \ - "problem: wallet code 7006 — no coin denominations for GOA:0 (zero amount cannot be withdrawn as coins). Ladder continues. detail: $note" - else - status="SKIP_DENOM" - note="skip amount (wallet 7006 no denoms): $note" - warn wallet "accept $AMT skipped" \ - "problem: wallet code 7006 — no denominations match this amount (below smallest coin or not combinable). Ladder continues with next rung. detail: $note" - fi - echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" - continue - fi - err wallet "accept $AMT" "$note" - status="FAIL_ACCEPT" - STOP_REASON="$note" - STOP_AMOUNT="$AMT" - echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" - FAIL_N_L=$((FAIL_N_L + 1)) - break - fi - - # Confirm ASAP when bank status is selected. No run-until-done (hangs on macOS/wallet). - # Server-side auto-confirm only watches landing withdraw-watch.ids — ladder must confirm itself. - bank_st() { - curl -sS -m 8 "${BANK}/taler-integration/withdrawal-operation/${WID}" 2>/dev/null \ - | python3 -c 'import json,sys; print((json.load(sys.stdin).get("status") or "").strip())' 2>/dev/null || true - } - do_confirm() { - curl -sS -m 15 -o "$SCRATCH/conf-$tag.json" -w '%{http_code}' -X POST \ - -H "Authorization: Bearer ${TOK}" -H 'Content-Type: application/json' -d '{}' \ - "${BANK}/accounts/${EXP_USER}/withdrawals/${WID}/confirm" - } - # Collect reserve_pub candidates for *this* withdrawal (WID + amount). - # Cumulative wallets re-print old reserves in accept/tx dumps — never trust a single "last" blindly. - # Prints unique pubs one per line, preferred order first. - # - # Critical: wallet-cli dumps mix log lines + JSON. Never json.loads(rest_of_file) — - # trailing "Shutdown requested" lines make that always fail. Use JSONDecoder.raw_decode - # and regex on *both* accept and transactions output (reservePub lives in tx, not accept). - extract_rpubs_for_wid() { - python3 - "$WID" "$AMT" "$SCRATCH/accept-$tag.out" "$SCRATCH/tx-$tag.json" "$SCRATCH/used-rpubs.txt" <<'PY' -import json, re, sys -from pathlib import Path - -wid = sys.argv[1] -amt = sys.argv[2] -paths = sys.argv[3:5] -used_path = sys.argv[5] -used = set() -if Path(used_path).is_file(): - used = {ln.strip() for ln in open(used_path) if ln.strip()} - -amt_num = amt.split(":", 1)[-1] if amt else "" -dec = json.JSONDecoder() - -def walk_collect(o, bag, ctx=None): - ctx = dict(ctx or {}) - if isinstance(o, dict): - # First pass: sibling scalars (reservePub + bankConfirmationUrl + amounts - # are peers under withdrawalDetails — order in JSON must not matter). - for k, v in o.items(): - if not isinstance(v, str): - continue - kl = str(k).lower().replace("-", "_") - if kl in ( - "withdrawal_id", "withdraw_id", "wopid", "id", - "transactionid", "transaction_id", - ): - ctx["id"] = v - elif kl in ( - "amount", "rawamount", "instructedamount", - "amounteffective", "amountraw", "transferamount", - ): - ctx["amount"] = v - elif kl in ( - "taler_withdraw_uri", "talerwithdrawuri", "uri", - "bankconfirmationurl", "confirmtransferurl", - ): - ctx["uri"] = v - for k, v in o.items(): - kl = str(k).lower().replace("-", "_") - if kl in ("reserve_pub", "reservepub") and isinstance(v, str) and len(v) >= 40: - bag.append((v, dict(ctx))) - elif not isinstance(v, (str, int, float, bool)) and v is not None: - walk_collect(v, bag, ctx) - elif isinstance(o, list): - for i in o: - walk_collect(i, bag, ctx) - -def score_ctx(pub, ctx, window=""): - score = 0 - cid = str(ctx.get("id") or "") - camt = str(ctx.get("amount") or "") - curi = str(ctx.get("uri") or "") - # bankConfirmationUrl / withdraw URI embeds this op's WID - if wid and wid in curi: - score += 120 - if wid and wid in cid: - score += 100 - if wid and wid in window: - score += 80 - if amt and (camt == amt or camt.endswith(amt_num)): - score += 50 - if amt and amt in window: - score += 30 - if pub in used: - score -= 200 - return score - -def iter_json_objects(text): - """Yield top-level JSON objects from mixed log+JSON wallet dumps.""" - i, n = 0, len(text) - while i < n: - if text[i] == "{": - try: - o, end = dec.raw_decode(text, i) - yield o - i = end - continue - except Exception: - pass - i += 1 - -raw_pubs = [] -scored = [] - -# 1) structured JSON via raw_decode (accept + transactions) -for p in paths: - try: - t = open(p, errors="replace").read() - except Exception: - continue - for o in iter_json_objects(t): - bag = [] - walk_collect(o, bag) - for pub, ctx in bag: - raw_pubs.append(pub) - scored.append((score_ctx(pub, ctx), pub)) - -# 2) regex on *both* accept and tx (reservePub is normally only in transactions) -RPUB_PATS = ( - r"\"reservePub\"\s*:\s*\"([A-Z0-9]{40,})\"", - r"\"reserve_pub\"\s*:\s*\"([A-Z0-9]{40,})\"", - r"reserve[_ ]?pub[\"\s:=]+([A-Z0-9]{40,})", -) -for p in paths: - try: - blob = open(p, errors="replace").read() - except Exception: - continue - for pat in RPUB_PATS: - for m in re.finditer(pat, blob, re.I): - pub = m.group(1) - raw_pubs.append(pub) - window = blob[max(0, m.start() - 600) : m.end() + 600] - scored.append((score_ctx(pub, {}, window) + 10, pub)) - -order = [] -seen = set() -for score, pub in sorted(scored, key=lambda x: x[0], reverse=True): - if pub in seen or pub in used: - continue - seen.add(pub) - order.append(pub) -for pub in reversed(raw_pubs): - if pub in seen or pub in used: - continue - seen.add(pub) - order.append(pub) -for pub in order: - print(pub) -PY - } - - mark_rpub_used() { - local p="$1" - [ -n "$p" ] || return 0 - mkdir -p "$SCRATCH" 2>/dev/null || true - grep -qxF "$p" "$SCRATCH/used-rpubs.txt" 2>/dev/null || echo "$p" >>"$SCRATCH/used-rpubs.txt" - } - - force_select_if_needed() { - local st_now="$1" - [ "$st_now" = "pending" ] || [ -z "$st_now" ] || return 0 - local rpub epayto code_fs any=0 - # refresh tx dump each try (wallet may attach reserve late) - wcli transactions >"$SCRATCH/tx-$tag.json" 2>&1 || true - epayto=$(curl -sS -m 10 "${EX%/}/keys" 2>/dev/null | python3 -c ' -import json,sys -d=json.load(sys.stdin) -acc=d.get("accounts") or [] -for a in acc: - p=a.get("payto_uri") or a.get("payto_address") or "" - if "x-taler-bank" in p or "exchange" in p: - print(p); break -else: - if acc: print(acc[0].get("payto_uri") or "") -' 2>/dev/null || true) - if [ -z "$epayto" ]; then - warn bank "force-select skipped" "problem: exchange payto empty from /keys" - return 0 - fi - # Try candidates until bank leaves pending (200/204) or we exhaust - while IFS= read -r rpub; do - [ -n "$rpub" ] || continue - any=1 - code_fs=$(curl -sS -m 12 -o "$SCRATCH/force-sel-$tag.json" -w '%{http_code}' -X POST \ - -H 'Content-Type: application/json' \ - -d "{\"reserve_pub\":\"${rpub}\",\"selected_exchange\":\"${epayto}\"}" \ - "${BANK}/taler-integration/withdrawal-operation/${WID}" 2>/dev/null || echo "000") - if [ "$code_fs" = "200" ] || [ "$code_fs" = "204" ]; then - info "force-select" "HTTP ${code_fs} rpub=${rpub:0:12}… (ok for WID ${WID:0:8})" - mark_rpub_used "$rpub" - return 0 - fi - if [ "$code_fs" = "409" ]; then - # 5114 = this reserve already bound to another op — not "out of money" - info "force-select" "HTTP 409 rpub=${rpub:0:12}… (stale/used reserve — not balance; trying next)" - mark_rpub_used "$rpub" - continue - fi - info "force-select" "HTTP ${code_fs} rpub=${rpub:0:12}… body=$(tr '\n' ' ' <"$SCRATCH/force-sel-$tag.json" 2>/dev/null | head -c 120)" - # other errors: still try next candidate - done < <(extract_rpubs_for_wid) - if [ "$any" != "1" ]; then - warn bank "force-select skipped" \ - "problem: no reserve_pub for this withdraw (WID=${WID:0:8}…); wallet may not have selected yet" - fi - } - - # No run-until-done (hangs / banned). Read transactions only for reserve_pub candidates. - wcli transactions >"$SCRATCH/tx-$tag.json" 2>&1 || true - - t0=$(now_ms) - conf_ok=0 - st="" - # Poll bank status; force-select while pending; confirm as soon as selected - for i in $(seq 1 "${LADDER_CONFIRM_POLLS}"); do - ladder_over && break - st=$(bank_st) - case "$st" in - selected) - ccode=$(do_confirm) - if [ "$ccode" = "204" ] || [ "$ccode" = "200" ]; then - conf_ok=1 - info "confirm" "HTTP $ccode after selected (poll $i)" - else - note="confirm HTTP $ccode" - fi - break - ;; - confirmed) - conf_ok=1 - break - ;; - aborted) - note="withdrawal aborted by bank" - break - ;; - esac - # force-select while pending — try alternate rpubs on 5114 (not out-of-money) - if [ "$st" = "pending" ] || [ -z "$st" ]; then - if [ "$i" -eq 1 ] || [ "$i" -eq 2 ] || [ $((i % 3)) -eq 0 ]; then - force_select_if_needed "$st" - fi - fi - sleep 0.35 - done - ms_confirm=$(elapsed_ms "$t0") - st=$(printf '%s' "${st:-}" | tr -d '\r\n' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//') - if [ "$conf_ok" != "1" ]; then - note="${note:-confirm timeout last=${st:-empty}}" - # Soft: not confirmed — usually stale reserve_pub (5114), NOT empty pool balance - status="SKIP_CONFIRM" - warn bank "confirm $AMT skipped" \ - "problem: bank status='${st:-empty}' after ${LADDER_CONFIRM_POLLS} polls (want selected/confirmed). Usually reserve_pub mismatch (5114), not out-of-money — mint/accept already OK. detail: $note" - ms_total=$(elapsed_ms "$t_rung") - echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" - continue - fi - ok "confirm $AMT ${ms_confirm}ms (client, on selected)" - - # settle: poll wallet balance + bank transfer_done only — never run-until-done - t0=$(now_ms) - settled=0 - xfer="?" - if [ "$IS_ZERO" = "1" ]; then - settled=1 - note="zero-amount: no coin delta expected" - else - for r in $(seq 1 "$LADDER_SETTLE_ROUNDS"); do - ladder_over && break - after=$(wallet_avail) - if python3 -c " -from decimal import Decimal -import sys -sys.exit(0 if Decimal(sys.argv[1]) > Decimal(sys.argv[2]) else 1) -" "$after" "$before" 2>/dev/null; then - settled=1 - break - fi - xfer=$(curl -sS -m 10 "${BANK}/taler-integration/withdrawal-operation/${WID}" \ - | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d.get("transfer_done"), d.get("status"))' 2>/dev/null || echo "?") - # bank done is enough to leave settle without hanging on wallet - if echo "$xfer" | grep -qi True; then - note="bank transfer_done (no run-until-done) avail=${after} $xfer" - break - fi - sleep "$LADDER_SETTLE_SLEEP" - done - fi - ms_settle=$(elapsed_ms "$t0") - after=$(wallet_avail) - ms_total=$(elapsed_ms "$t_rung") - if [ -z "${xfer:-}" ] || [ "$xfer" = "?" ]; then - xfer=$(curl -sS -m 10 "${BANK}/taler-integration/withdrawal-operation/${WID}" \ - | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d.get("transfer_done"), d.get("status"))' 2>/dev/null || echo "?") - fi - - if [ "$settled" = "1" ]; then - status="OK" - note="${note:-avail=${CUR}:${after}}" - ok "settle $AMT → ${CUR}:${after} (settle ${ms_settle}ms, rung ${ms_total}ms)" - OK_N=$((OK_N + 1)) - echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" - metrics_report_coins "ladder-r${rung}-after-${tag}" || true - metrics_record_flow withdrawn "$AMT" || true - elif echo "$xfer" | grep -qi True; then - status="OK_BANK" - note="bank transfer_done avail=${after} $xfer (no run-until-done)" - ok "settle $AMT bank transfer_done (wallet avail=${CUR}:${after}, ${ms_settle}ms)" - OK_N=$((OK_N + 1)) - echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" - metrics_report_coins "ladder-r${rung}-after-${tag}" || true - metrics_record_flow withdrawn "$AMT" || true - else - note="no coins / no transfer_done avail=${after} $xfer" - err wallet "settle $AMT" "$note" - status="FAIL_SETTLE" - STOP_REASON="$note" - STOP_AMOUNT="$AMT" - FAIL_N_L=$((FAIL_N_L + 1)) - echo -e "${rung}\t${range_note}\t${AMT}\t${status}\t${ms_mint}\t${ms_accept}\t${ms_confirm}\t${ms_settle}\t${ms_total}\t${WID}\t${note}" >>"$TSV" - metrics_report_coins "ladder-r${rung}-fail-${tag}" || true - break - fi -done - -# --------------------------------------------------------------------------- -# Phase B — payment ladder (same shape: 0 → low → … → max-1 → max) -# --------------------------------------------------------------------------- -PAY_OK_N=0 -PAY_FAIL_N=0 -if [ "${LADDER_PAY}" = "1" ] && [ -n "${PAY_LIST:-}" ] && [ "$FAIL_N_L" -eq 0 ]; then - set_group pay - section "ladder · phase B · pay" - metrics_report_coins "before-pay-ladder" || true - # Merchant secret (same as e2e) — stage often has only public shop templates - MPW="${E2E_MERCHANT_TOKEN:-${MERCHANT_TOKEN:-}}" - if [ -z "$MPW" ]; then - MPW=$(read_secret "taler-merchant/merchant-${INST}-password.txt" 2>/dev/null || true) - fi - if [ -z "$MPW" ]; then - MPW=$(read_secret "taler-merchant/merchant-goa-demo-cp4zqk-password.txt" 2>/dev/null || true) - fi - if [ -z "$MPW" ] && [ "${CUR}" = "TESTPAYSAN" ]; then - if [ -n "${FRANCPAYSAN_SECRETS:-}" ] && [ -f "${FRANCPAYSAN_SECRETS}/stage/default-instance-token.txt" ]; then - MPW=$(tr -d '\n\r' <"${FRANCPAYSAN_SECRETS}/stage/default-instance-token.txt") - fi - if [ -z "$MPW" ]; then - _remote_mer="${FRANCPAYSAN_REMOTE_MERCHANT_TOKEN:-}" - [ -z "$_remote_mer" ] && [ -n "${FRANCPAYSAN_REMOTE_SECRETS_ROOT:-}" ] && \ - _remote_mer="${FRANCPAYSAN_REMOTE_SECRETS_ROOT}/merchant/secrets/default-instance-token.txt" - for host in ${INSIDE_SSH:+"$INSIDE_SSH"} ${FRANCPAYSAN_SSH:+"$FRANCPAYSAN_SSH"}; do - [ -n "$host" ] && [ -n "$_remote_mer" ] || continue - MPW=$(ssh -o BatchMode=yes -o ConnectTimeout=12 "$host" \ - "tr -d '\\n\\r' <${_remote_mer} 2>/dev/null || true" \ - 2>/dev/null || true) - [ -n "$MPW" ] && break - done - unset _remote_mer - fi - fi - if [ -z "$MPW" ]; then - if [ "${CUR}" = "TESTPAYSAN" ]; then - # Stage: withdraw ladder is the main probe; pays need instance token or - # public templates for *exact* face values (not continuous ladder amounts). - info pay "no merchant token — skip pay ladder on TESTPAYSAN (withdraw maxima still covered)" - info pay "hint: set E2E_MERCHANT_TOKEN or use ./taler-monitoring.sh -d stage.lefrancpaysan.ch e2e for shop templates" - else - warn pay "no merchant token — skip pay ladder (set E2E_MERCHANT_TOKEN or secrets)" - fi - else - ok "merchant token" "instance ${INST}" - case "$MPW" in - secret-token:*) AUTH="Authorization: Bearer ${MPW}" ;; - *) AUTH="Authorization: Bearer secret-token:${MPW}" ;; - esac - wallet_prepare "main" - # shellcheck disable=SC2086 - set -- $PAY_LIST - PAY_N=$# - info "pay rungs" "$PAY_N · $*" - prung=0 - for PAMT in "$@"; do - prung=$((prung + 1)) - if ladder_over; then - warn pay "time budget exhausted" "after ${PAY_OK_N} ok pays" - break - fi - section "ladder · pay rung $prung $PAMT · $(format_amount_alt "$PAMT")" - ptag=$(printf '%s' "$PAMT" | tr '.:' '__') - if [ "$prung" -eq 1 ]; then - prange="pin:0" - elif [ "$prung" -eq "$PAY_N" ]; then - prange="pin:max" - elif [ "$prung" -eq $((PAY_N - 1)) ] && [ "$PAY_N" -ge 3 ]; then - prange="pin:max-1" - else - prange="random" - fi - PNUM=$(python3 -c 'import sys; print(sys.argv[1].split(":",1)[-1])' "$PAMT") - IS_PZERO=0 - python3 -c 'import sys; from decimal import Decimal; sys.exit(0 if Decimal(sys.argv[1])==0 else 1)' "$PNUM" 2>/dev/null && IS_PZERO=1 - IS_PMAX=0 - [ "$PNUM" = "${LADDER_MAX_AMOUNT}" ] && IS_PMAX=1 - t_pay=$(now_ms) - ms_order=0 ms_handle=0 ms_psettle=0 - pnote="" - pstatus="FAIL" - OID="-" - - metrics_report_coins "before-pay-${ptag}" || true - bal=$(wallet_avail) - - if [ "$IS_PZERO" = "1" ]; then - pstatus="ZERO_SKIP" - pnote="zero pay probe skipped" - warn pay "pay $PAMT skipped" "problem: zero amount order not useful. Ladder continues." - ms_total=$(elapsed_ms "$t_pay") - echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t${ms_handle}\t${ms_psettle}\t${ms_total}\t${OID}\t${pnote}" >>"$PAY_TSV" - PAY_OK_N=$((PAY_OK_N + 1)) - continue - fi - - # Soft: absolute max pay is a ceiling probe (unlikely affordable / merchant may reject) - if [ "$IS_PMAX" = "1" ]; then - # try once; on fail CEILING_REJECT - : - fi - - # Insufficient balance → soft skip for max pin only; hard fail for mid/max-1 - if ! python3 -c 'import sys; from decimal import Decimal; sys.exit(0 if Decimal(sys.argv[1])>=Decimal(sys.argv[2]) else 1)' "$bal" "$PNUM" 2>/dev/null; then - pnote="insufficient balance avail=${CUR}:${bal} need=${PAMT}" - ms_total=$(elapsed_ms "$t_pay") - if [ "$IS_PMAX" = "1" ]; then - pstatus="CEILING_SKIP" - warn pay "pay $PAMT skipped (ceiling)" "$pnote" - echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t0\t0\t0\t${ms_total}\t-\t${pnote}" >>"$PAY_TSV" - continue - fi - err pay "pay $PAMT" "$pnote" - pstatus="FAIL_BALANCE" - PAY_FAIL_N=$((PAY_FAIL_N + 1)) - FAIL_N_L=$((FAIL_N_L + 1)) - STOP_REASON="$pnote" - STOP_AMOUNT="$PAMT" - echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t0\t0\t0\t${ms_total}\t-\t${pnote}" >>"$PAY_TSV" - break - fi - - t0=$(now_ms) - SUM_JSON=$(python3 -c 'import json,sys; print(json.dumps(sys.argv[1]))' "ladder pay ${PAMT}" 2>/dev/null || echo '"ladder pay"') - curl -skS -m 20 -o "$SCRATCH/ord-$ptag.json" -X POST \ - -H "$AUTH" -H 'Content-Type: application/json' \ - -d "{\"order\":{\"summary\":${SUM_JSON},\"amount\":\"${PAMT}\",\"fulfillment_message\":\"ok\"},\"create_token\":true}" \ - "${MER}/instances/${INST}/private/orders" 2>"$SCRATCH/ord-$ptag.err" || true - ms_order=$(elapsed_ms "$t0") - OID=$(python3 -c 'import json,re,sys;t=open(sys.argv[1]).read() -try: print(json.loads(t).get("order_id") or "") -except Exception: - m=re.search(r"\"order_id\"\s*:\s*\"([^\"]+)\"",t); print(m.group(1) if m else "") -' "$SCRATCH/ord-$ptag.json" 2>/dev/null || true) - OTOK=$(python3 -c 'import json,re,sys;t=open(sys.argv[1]).read() -try: print(json.loads(t).get("token") or "") -except Exception: - m=re.search(r"\"token\"\s*:\s*\"([^\"]+)\"",t); print(m.group(1) if m else "") -' "$SCRATCH/ord-$ptag.json" 2>/dev/null || true) - if [ -z "$OID" ]; then - pnote="order create failed $(head -c 80 "$SCRATCH/ord-$ptag.json" 2>/dev/null | tr '\n\"' ' ')" - ms_total=$(elapsed_ms "$t_pay") - if [ "$IS_PMAX" = "1" ]; then - pstatus="CEILING_REJECT" - warn pay "pay $PAMT rejected (ceiling)" "$pnote" - echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t0\t0\t${ms_total}\t-\t${pnote}" >>"$PAY_TSV" - continue - fi - err pay "order $PAMT" "$pnote" - pstatus="FAIL_ORDER" - PAY_FAIL_N=$((PAY_FAIL_N + 1)) - FAIL_N_L=$((FAIL_N_L + 1)) - STOP_REASON="$pnote" - STOP_AMOUNT="$PAMT" - echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t0\t0\t${ms_total}\t-\t${pnote}" >>"$PAY_TSV" - break - fi - ok "order $OID ($PAMT) ${ms_order}ms" - - curl -skS -m 12 -o "$SCRATCH/ord-det-$ptag.json" -H "$AUTH" \ - "${MER}/instances/${INST}/private/orders/${OID}" 2>/dev/null || true - PAYURI=$(python3 -c 'import json,sys -try: print(json.load(open(sys.argv[1])).get("taler_pay_uri") or "") -except Exception: print("") -' "$SCRATCH/ord-det-$ptag.json" 2>/dev/null || true) - if [ -z "$PAYURI" ] && [ -n "$OTOK" ]; then - MH=$(python3 -c 'from urllib.parse import urlparse; print(urlparse("'"$MER"'").hostname or "taler.hacktivism.ch")' 2>/dev/null || echo "taler.hacktivism.ch") - PAYURI="taler://pay/${MH}/instances/${INST}/${OID}/?c=${OTOK}" - fi - PAYURI=$(printf '%s' "$PAYURI" | sed 's/:443\//\//g; s/:443?/?/g') - if [ -z "$PAYURI" ]; then - pnote="no pay URI for $OID" - ms_total=$(elapsed_ms "$t_pay") - err pay "uri $PAMT" "$pnote" - pstatus="FAIL_URI" - PAY_FAIL_N=$((PAY_FAIL_N + 1)) - FAIL_N_L=$((FAIL_N_L + 1)) - STOP_REASON="$pnote" - STOP_AMOUNT="$PAMT" - echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t0\t0\t${ms_total}\t${OID}\t${pnote}" >>"$PAY_TSV" - break - fi - - t0=$(now_ms) - if ! wcli handle-uri --yes "$PAYURI" >"$SCRATCH/pay-$ptag.out" 2>&1; then - ms_handle=$(elapsed_ms "$t0") - pnote="handle-uri failed $(tail -c 100 "$SCRATCH/pay-$ptag.out" | tr '\n\"' ' ')" - ms_total=$(elapsed_ms "$t_pay") - if [ "$IS_PMAX" = "1" ]; then - pstatus="CEILING_REJECT" - warn pay "pay $PAMT handle failed (ceiling)" "$pnote" - echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t${ms_handle}\t0\t${ms_total}\t${OID}\t${pnote}" >>"$PAY_TSV" - continue - fi - err pay "handle $PAMT" "$pnote" - pstatus="FAIL_HANDLE" - PAY_FAIL_N=$((PAY_FAIL_N + 1)) - FAIL_N_L=$((FAIL_N_L + 1)) - STOP_REASON="$pnote" - STOP_AMOUNT="$PAMT" - echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t${ms_handle}\t0\t${ms_total}\t${OID}\t${pnote}" >>"$PAY_TSV" - break - fi - ms_handle=$(elapsed_ms "$t0") - ok "handle-uri $PAMT ${ms_handle}ms" - - # Short settle polls: merchant order + wallet tx only — never run-until-done - t0=$(now_ms) - settled=0 - r=0 - while [ "$r" -lt "${LADDER_PAY_SETTLE_ROUNDS}" ]; do - r=$((r + 1)) - wcli transactions >"$SCRATCH/tx-$ptag.out" 2>&1 || true - curl -skS -m 8 -o "$SCRATCH/ord-paid-$ptag.json" -H "$AUTH" \ - "${MER}/instances/${INST}/private/orders/${OID}" 2>/dev/null || true - if grep -qiE 'payment|paid|Payment' "$SCRATCH/tx-$ptag.out" 2>/dev/null \ - || python3 -c 'import json,sys -d=json.load(open(sys.argv[1])) -sys.exit(0 if d.get("paid") is True or str(d.get("order_status","")).lower()=="paid" else 1) -' "$SCRATCH/ord-paid-$ptag.json" 2>/dev/null; then - settled=1 - break - fi - sleep 0.5 - done - ms_psettle=$(elapsed_ms "$t0") - ms_total=$(elapsed_ms "$t_pay") - after=$(wallet_avail) - if [ "$settled" = "1" ]; then - pstatus="OK" - pnote="avail=${CUR}:${after}" - ok "pay settled $PAMT → bal ${CUR}:${after} (total ${ms_total}ms)" - PAY_OK_N=$((PAY_OK_N + 1)) - echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t${ms_handle}\t${ms_psettle}\t${ms_total}\t${OID}\t${pnote}" >>"$PAY_TSV" - metrics_report_coins "after-pay-${ptag}" || true - metrics_record_flow spent "$PAMT" || true - else - pnote="not settled order=$OID avail=${CUR}:${after}" - if [ "$IS_PMAX" = "1" ]; then - pstatus="CEILING_REJECT" - warn pay "pay $PAMT not settled (ceiling)" "$pnote" - echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t${ms_handle}\t${ms_psettle}\t${ms_total}\t${OID}\t${pnote}" >>"$PAY_TSV" - continue - fi - err pay "settle $PAMT" "$pnote" - pstatus="FAIL_SETTLE" - PAY_FAIL_N=$((PAY_FAIL_N + 1)) - FAIL_N_L=$((FAIL_N_L + 1)) - STOP_REASON="$pnote" - STOP_AMOUNT="$PAMT" - echo -e "${prung}\t${prange}\t${PAMT}\t${pstatus}\t${ms_order}\t${ms_handle}\t${ms_psettle}\t${ms_total}\t${OID}\t${pnote}" >>"$PAY_TSV" - metrics_report_coins "after-pay-fail-${ptag}" || true - break - fi - done - metrics_report_coins "after-pay-ladder" || true - info "pay summary" "ok=$PAY_OK_N fail=$PAY_FAIL_N tsv=$PAY_TSV" - fi -elif [ "${LADDER_PAY}" = "1" ] && [ "$FAIL_N_L" -gt 0 ]; then - warn pay "skipped pay ladder" "withdraw phase already failed" -fi - -ms_phase=$(python3 -c 'import sys,time; print(int((time.time()-float(sys.argv[1]))*1000))' "$SECTION_T0") - -# --- report --- -set_group report -section "ladder · report" -info "auto-account" "$ACCT_USER" -info "ok_rungs" "$OK_N" -info "fail_rungs" "$FAIL_N_L" -info "pay_ok" "$PAY_OK_N" -info "pay_fail" "$PAY_FAIL_N" -info "phase_ms" "$ms_phase" -info "tsv" "$TSV" -info "pay_tsv" "$PAY_TSV" - -# speed summary via python — free-text stop reason via env (JSON " in bank errors -# used to break shell argv quoting → "syntax error near unexpected token '('") -export LADDER_REPORT_STOP_AMOUNT="${STOP_AMOUNT:-}" -export LADDER_REPORT_STOP_REASON="${STOP_REASON:-}" -python3 - "$TSV" "$PAY_TSV" "$JSON" "$OK_N" "$FAIL_N_L" "$PAY_OK_N" "$PAY_FAIL_N" "$ms_phase" "$ACCT_USER" "$CUR" <<'PY' -import csv, json, os, sys, statistics -tsv, pay_tsv, jpath = sys.argv[1:4] -ok_n, fail_n, pay_ok, pay_fail, phase_ms, acct, cur = sys.argv[4:11] -stop_amt = os.environ.get("LADDER_REPORT_STOP_AMOUNT") or None -stop_reason = os.environ.get("LADDER_REPORT_STOP_REASON") or None - -def load_rows(path): - rows = [] - try: - with open(path, newline="") as f: - for row in csv.DictReader(f, delimiter="\t"): - rows.append(row) - except Exception: - pass - return rows - -rows = load_rows(tsv) -prows = load_rows(pay_tsv) - -def nums(rs, key): - out = [] - for row in rs: - try: - out.append(int(row[key])) - except Exception: - pass - return out - -def stats(xs): - if not xs: - return {"n": 0} - return { - "n": len(xs), - "min_ms": min(xs), - "max_ms": max(xs), - "avg_ms": int(sum(xs) / len(xs)), - "p50_ms": int(statistics.median(xs)), - } - -report = { - "currency": cur, - "auto_account": acct, - "ok_rungs": int(ok_n), - "fail_rungs": int(fail_n), - "pay_ok": int(pay_ok), - "pay_fail": int(pay_fail), - "stopped_at_amount": stop_amt or None, - "stop_reason": stop_reason or None, - "phase_ms": int(phase_ms), - "timing": { - "mint": stats(nums(rows, "ms_mint")), - "accept": stats(nums(rows, "ms_accept")), - "confirm": stats(nums(rows, "ms_confirm")), - "settle": stats(nums(rows, "ms_settle")), - "rung_total": stats(nums(rows, "ms_total")), - "pay_order": stats(nums(prows, "ms_order")), - "pay_handle": stats(nums(prows, "ms_handle")), - "pay_settle": stats(nums(prows, "ms_settle")), - "pay_total": stats(nums(prows, "ms_total")), - }, - "rungs": rows, - "pays": prows, -} -json.dump(report, open(jpath, "w"), indent=2) -print("JSON", jpath) -print("--- withdraw speed (ms) ---") -for k in ("mint", "accept", "confirm", "settle", "rung_total"): - v = report["timing"][k] - if v.get("n"): - print(" %s n=%s min=%s p50=%s avg=%s max=%s" % (k.ljust(12), v["n"], v["min_ms"], v["p50_ms"], v["avg_ms"], v["max_ms"])) -print("--- pay speed (ms) ---") -for k in ("pay_order", "pay_handle", "pay_settle", "pay_total"): - v = report["timing"][k] - if v.get("n"): - print(" %s n=%s min=%s p50=%s avg=%s max=%s" % (k.ljust(12), v["n"], v["min_ms"], v["p50_ms"], v["avg_ms"], v["max_ms"])) -print("--- withdraw rungs ---") -for row in rows: - print(" %2s %-16s %-12s total=%sms" % (row.get("rung"), row.get("amount"), row.get("status"), row.get("ms_total"))) -print("--- pay rungs ---") -for row in prows: - print(" %2s %-16s %-12s total=%sms oid=%s" % (row.get("rung"), row.get("amount"), row.get("status"), row.get("ms_total"), row.get("oid"))) -if stop_amt: - print("STOPPED at %s: %s" % (stop_amt, stop_reason)) -else: - print("Completed without hard failure (or budget stop without fail).") -PY - -wcli balance 2>&1 | tee "$REPORT_DIR/balance-final.out" | tail -20 || true - -set_group load -section "ladder · load snapshot (after withdraws)" -metrics_report_load "$LOAD_AFTER" "ladder-end" || true -if [ -f "$LOAD_BEFORE" ] && [ -f "$LOAD_AFTER" ]; then - section "metrics · ladder load delta" - metrics_print_load_delta "$LOAD_BEFORE" "$LOAD_AFTER" || true -fi -# Speed timings → metrics overall (min/p50/avg/max per phase) -if [ -f "$JSON" ]; then - python3 - "$JSON" "${METRICS_DIR}/perf-summary.json" <<'PY' 2>/dev/null || true -import json, sys -rep = json.load(open(sys.argv[1])) -out = {} -for k, v in (rep.get("timing") or {}).items(): - if isinstance(v, dict) and v.get("n"): - out[k] = v -json.dump(out, open(sys.argv[2], "w"), indent=2) -PY -fi -export METRICS_WITHDRAW_TSV="$TSV" -export METRICS_PAY_TSV="$PAY_TSV" -metrics_report_coins "ladder-end" || true -metrics_print_overall "ladder final" || true - -# Keep scratch if LADDER_REPORT_DIR set; else copy key files to /tmp -if [ -z "${LADDER_REPORT_DIR:-}" ]; then - KEEP="/tmp/goa-ladder-report-$(date +%Y%m%d-%H%M%S)" - mkdir -p "$KEEP" - cp -a "$TSV" "$PAY_TSV" "$JSON" "$SCRATCH/auto-account.json" "$SCRATCH/ladder-plan.txt" \ - "$SCRATCH/ladder-wd-plan.txt" "$SCRATCH/ladder-pay-plan.txt" \ - "$REPORT_DIR/balance-final.out" "$LOAD_BEFORE" "$LOAD_AFTER" "$KEEP/" 2>/dev/null || true - info "report_dir" "$KEEP" - echo "$KEEP" >"$SCRATCH/KEEP_PATH" -fi - -if [ "$FAIL_N_L" -gt 0 ]; then - blocker "ladder" "stopped at ${STOP_AMOUNT:-?} — ${STOP_REASON:-error}" - exit 1 -fi -if [ "$OK_N" -eq 0 ]; then - blocker "ladder" "no successful withdraw rungs" - exit 1 -fi -ok "ladder finished withdraw_ok=$OK_N pay_ok=$PAY_OK_N phase=${ms_phase}ms" -exit 0 +exec bash "$ROOT/check_amount_ladder.sh" "$@" diff --git a/ladder_extract_rpubs.py b/ladder_extract_rpubs.py new file mode 100755 index 0000000..1c939e9 --- /dev/null +++ b/ladder_extract_rpubs.py @@ -0,0 +1,163 @@ +#!/usr/bin/env python3 +"""Extract reserve_pub candidates from wallet accept + transactions dumps. + +Used by check_amount_ladder.sh force-select. Kept as a file (not a bash +heredoc) so shell refactors cannot turn Python ``continue`` into ``return``. + +Usage: + ladder_extract_rpubs.py WID AMT accept.out tx.json used-rpubs.txt +Prints one reserve pub per line, preferred first. +""" +from __future__ import annotations + +import json +import re +import sys +from pathlib import Path + + +def main(argv: list[str]) -> int: + if len(argv) < 6: + print( + "usage: ladder_extract_rpubs.py WID AMT accept.out tx.json used-rpubs.txt", + file=sys.stderr, + ) + return 2 + + wid = argv[1] + amt = argv[2] + paths = argv[3:5] + used_path = argv[5] + used: set[str] = set() + if Path(used_path).is_file(): + used = {ln.strip() for ln in open(used_path) if ln.strip()} + + amt_num = amt.split(":", 1)[-1] if amt else "" + dec = json.JSONDecoder() + + def walk_collect(o, bag, ctx=None): + ctx = dict(ctx or {}) + if isinstance(o, dict): + for k, v in o.items(): + if not isinstance(v, str): + continue + kl = str(k).lower().replace("-", "_") + if kl in ( + "withdrawal_id", + "withdraw_id", + "wopid", + "id", + "transactionid", + "transaction_id", + ): + ctx["id"] = v + elif kl in ( + "amount", + "rawamount", + "instructedamount", + "amounteffective", + "amountraw", + "transferamount", + ): + ctx["amount"] = v + elif kl in ( + "taler_withdraw_uri", + "talerwithdrawuri", + "uri", + "bankconfirmationurl", + "confirmtransferurl", + ): + ctx["uri"] = v + for k, v in o.items(): + kl = str(k).lower().replace("-", "_") + if kl in ("reserve_pub", "reservepub") and isinstance(v, str) and len(v) >= 40: + bag.append((v, dict(ctx))) + elif not isinstance(v, (str, int, float, bool)) and v is not None: + walk_collect(v, bag, ctx) + elif isinstance(o, list): + for i in o: + walk_collect(i, bag, ctx) + + def score_ctx(pub, ctx, window=""): + score = 0 + cid = str(ctx.get("id") or "") + camt = str(ctx.get("amount") or "") + curi = str(ctx.get("uri") or "") + if wid and wid in curi: + score += 120 + if wid and wid in cid: + score += 100 + if wid and wid in window: + score += 80 + if amt and (camt == amt or camt.endswith(amt_num)): + score += 50 + if amt and amt in window: + score += 30 + if pub in used: + score -= 200 + return score + + def iter_json_objects(text: str): + i, n = 0, len(text) + while i < n: + if text[i] == "{": + try: + o, end = dec.raw_decode(text, i) + yield o + i = end + continue + except Exception: + pass + i += 1 + + raw_pubs: list[str] = [] + scored: list[tuple[int, str]] = [] + + for p in paths: + try: + t = open(p, errors="replace").read() + except Exception: + continue + for o in iter_json_objects(t): + bag = [] + walk_collect(o, bag) + for pub, ctx in bag: + raw_pubs.append(pub) + scored.append((score_ctx(pub, ctx), pub)) + + rpub_pats = ( + r"\"reservePub\"\s*:\s*\"([A-Z0-9]{40,})\"", + r"\"reserve_pub\"\s*:\s*\"([A-Z0-9]{40,})\"", + r"reserve[_ ]?pub[\"\s:=]+([A-Z0-9]{40,})", + ) + for p in paths: + try: + blob = open(p, errors="replace").read() + except Exception: + continue + for pat in rpub_pats: + for m in re.finditer(pat, blob, re.I): + pub = m.group(1) + raw_pubs.append(pub) + window = blob[max(0, m.start() - 600) : m.end() + 600] + scored.append((score_ctx(pub, {}, window) + 10, pub)) + + order: list[str] = [] + seen: set[str] = set() + for score, pub in sorted(scored, key=lambda x: x[0], reverse=True): + if pub in seen or pub in used: + continue + seen.add(pub) + order.append(pub) + for pub in reversed(raw_pubs): + if pub in seen or pub in used: + continue + seen.add(pub) + order.append(pub) + for pub in order: + print(pub) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main(sys.argv)) diff --git a/metrics.sh b/metrics.sh index aac0630..5e6ac42 100644 --- a/metrics.sh +++ b/metrics.sh @@ -70,11 +70,15 @@ path = sys.argv[2] if len(sys.argv) > 2 else "" alt = {} if path: try: - alt = json.load(open(path)) + _loaded = json.load(open(path)) + if isinstance(_loaded, dict): + alt = {str(k): str(v) for k, v in _loaded.items()} except Exception: alt = {} -if not alt: +if not isinstance(alt, dict) or not alt: alt = {"0": "GOA"} +elif "0" not in alt: + alt["0"] = "GOA" def parse(s): if ":" in s: diff --git a/taler-monitoring.sh b/taler-monitoring.sh index 35c2a99..b89fbaa 100755 --- a/taler-monitoring.sh +++ b/taler-monitoring.sh @@ -93,7 +93,8 @@ Phases: sanity public + optional server server server-side only (SSH) e2e withdraw + pay (small amounts; remote aborts on login/KYC) - ladder withdraw/pay amount ladder (GOA ceiling or stage TESTPAYSAN max_wire) + ladder amount ladder (classic: 0→mids→max; any currency / stack ceiling) + max-ladder same script, LADDER_MODE=max — hunt highest withdrawable amount auth401 merchant Basic-auth / case matrix (HTTP 401 paths; may create throwaway instance) aptdeploy koopa podman apt-src smoke: taler-merchant in koopa-taler-deploy-test-apt-src-trixie{,-testing} @@ -265,7 +266,7 @@ while [ $# -gt 0 ]; do CURRENCY_OVERRIDE="$2"; shift 2 ;; --no-probe) NO_PROBE=1; shift ;; - urls|inside|versions|sanity|server|e2e|ladder|goa-ladder|auth401|aptdeploy|apt-deploy|apt_src|surface|ecosystem|mattermost|mail|monpages|pages|devtesting|franken|fake-franken|fake_franken|all|full) PHASES+=("$1"); shift ;; + urls|inside|versions|sanity|server|e2e|ladder|goa-ladder|max-ladder|maxladder|auth401|aptdeploy|apt-deploy|apt_src|surface|ecosystem|mattermost|mail|monpages|pages|devtesting|franken|fake-franken|fake_franken|all|full) PHASES+=("$1"); shift ;; *) # bare domain shorthand: ./taler-monitoring.sh taler.net if [[ "$1" == *.* && "$1" != *://* && "$1" != -* ]]; then @@ -316,7 +317,7 @@ export INSIDE_PROFILE INSIDE_SSH export INSIDE_BANK_CTR INSIDE_EXCHANGE_CTR INSIDE_MERCHANT_CTR export INSIDE_BANK_PORT INSIDE_EXCHANGE_PORT INSIDE_MERCHANT_PORT export INSIDE_DNS_BANK INSIDE_DNS_EXCHANGE INSIDE_DNS_MERCHANT -# Ladder: withdraw then pay — 0 + random mids + max-1 + max (see check_goa_ladder.sh). +# Amount ladder — classic or max-search (see check_amount_ladder.sh; goa-ladder alias). # Defaults so set -u export is safe when vars were never set by caller. : "${LADDER_STEPS:=23}" : "${LADDER_MIN_AMOUNT:=0.000001}" @@ -330,16 +331,23 @@ export INSIDE_DNS_BANK INSIDE_DNS_EXCHANGE INSIDE_DNS_MERCHANT : "${LADDER_INCLUDE_ZERO:=1}" : "${LADDER_INCLUDE_MAX:=1}" : "${LADDER_HIGH_FROM:=1000000}" -: "${LADDER_HIGH_RUNGS:=12}" +: "${LADDER_HIGH_RUNGS:=4}" : "${LADDER_LOAD:=1}" : "${LADDER_PAY:=1}" : "${LADDER_WITHDRAW_SCALE:=1.5}" : "${LADDER_PAY_SETTLE_ROUNDS:=6}" +: "${LADDER_MODE:=classic}" +: "${LADDER_MAX_PROBES:=32}" +: "${LADDER_MAX_TOL_REL:=0.001}" +: "${LADDER_MAX_SEED:=}" +: "${LADDER_PAY_WITH_MAX:=0}" +: "${LADDER_HIGH_RUNGS:=6}" export LADDER_STEPS LADDER_MIN_AMOUNT LADDER_CONFIRM_POLLS LADDER_MAX_RUNGS LADDER_TIMEOUT_S LADDER_REPORT_DIR export LADDER_SETTLE_ROUNDS LADDER_SETTLE_SLEEP EXP_PW_FILE EXP_USER export LADDER_MAX_AMOUNT LADDER_INCLUDE_ZERO LADDER_INCLUDE_MAX export LADDER_HIGH_FROM LADDER_HIGH_RUNGS LADDER_LOAD export LADDER_PAY LADDER_WITHDRAW_SCALE LADDER_PAY_SETTLE_ROUNDS +export LADDER_MODE LADDER_MAX_PROBES LADDER_MAX_TOL_REL LADDER_MAX_SEED LADDER_PAY_WITH_MAX export TALER_DOMAIN_APPLIED=1 i18n_init 2>/dev/null || true # Optional: auth401 and other phases may honor CONTINUE_ON_ERROR @@ -435,7 +443,7 @@ _progress_estimate_phase() { server) n=20 ;; # e2e: bank withdraw ladder emits many INFO lines (coins before/after each note) e2e) n=240 ;; - ladder|goa-ladder) n=120 ;; + ladder|goa-ladder|max-ladder|maxladder) n=120 ;; auth401) n=70 ;; aptdeploy) n=40 ;; surface|ecosystem) n=120 ;; @@ -611,7 +619,15 @@ while [ "$_phase_idx" -lt "$_n_phases" ]; do sanity) run_phase sanity "$ROOT/check_sanity.sh" || ec=1 ;; server) run_phase server "$ROOT/check_server.sh" || ec=1 ;; e2e) run_phase e2e "$ROOT/check_e2e.sh" || ec=1 ;; - ladder|goa-ladder) run_phase ladder "$ROOT/check_goa_ladder.sh" || ec=1 ;; + ladder|goa-ladder) + run_phase ladder "$ROOT/check_amount_ladder.sh" || ec=1 + ;; + max-ladder|maxladder) + # Feature: hunt highest mint+confirm withdrawable amount (any currency) + LADDER_MODE=max + export LADDER_MODE + run_phase ladder "$ROOT/check_amount_ladder.sh" || ec=1 + ;; auth401) run_phase auth401 "$ROOT/check_auth401.sh" || ec=1 ;; aptdeploy|apt-deploy|apt_src) run_phase aptdeploy "$ROOT/check_apt_deploy.sh" || ec=1 ;; surface|ecosystem) run_phase surface "$ROOT/check_surface.sh" || ec=1 ;;