scripts/taler-landing: full-scan bank collector with GOA alt units
Add a host-side Python collector that lists every bank account, pages through all ledger rows, and emits stats.json with amount_alt fields (Kilo/Mega/Peta-GOA) so large ladder withdrawals fit on the landing tiles.
This commit is contained in:
parent
51785031d6
commit
25695805b4
5 changed files with 1528 additions and 0 deletions
106
scripts/taler-landing/README.md
Normal file
106
scripts/taler-landing/README.md
Normal file
|
|
@ -0,0 +1,106 @@
|
||||||
|
# taler-landing — central stats (hernani user systemd)
|
||||||
|
|
||||||
|
Public landing numbers on
|
||||||
|
|
||||||
|
- https://bank.hacktivism.ch/intro/ → `stats.json`
|
||||||
|
- https://exchange.hacktivism.ch/intro/ → `stats.json`
|
||||||
|
- https://taler.hacktivism.ch/intro/ → `stats.json`
|
||||||
|
|
||||||
|
are produced by **one host process** as user **`hernani`**, not by three unrelated in-container crons.
|
||||||
|
|
||||||
|
## Why host / user systemd
|
||||||
|
|
||||||
|
| Concern | Approach |
|
||||||
|
|--------|----------|
|
||||||
|
| **All accounts / all money** | Python full scan + tx pagination (not capped at 80 accounts) |
|
||||||
|
| **High ladder GOA** | `amount_alt` / UI alt units (Kilo-/Mega-/Peta-GOA) — tiles stay short |
|
||||||
|
| **No root cron** | `systemctl --user` as hernani + `loginctl enable-linger` |
|
||||||
|
| **Failure safety** | failed run only updates `stats-run.json`; last good `stats.json` stays |
|
||||||
|
|
||||||
|
## Install (on koopa as hernani)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd ~/src/koopa/koopa-admin-log
|
||||||
|
./scripts/taler-landing/install-landing-stats-host.sh
|
||||||
|
|
||||||
|
# timer without login session:
|
||||||
|
sudo loginctl enable-linger hernani
|
||||||
|
|
||||||
|
# run once:
|
||||||
|
systemctl --user start taler-landing-stats.service
|
||||||
|
journalctl --user -u taler-landing-stats.service -n 50 --no-pager
|
||||||
|
systemctl --user list-timers 'taler-landing-stats*'
|
||||||
|
```
|
||||||
|
|
||||||
|
Units:
|
||||||
|
|
||||||
|
- `configs/systemd/user/taler-landing-stats.service` — oneshot collector
|
||||||
|
- `configs/systemd/user/taler-landing-stats.timer` — every **2 minutes** after boot
|
||||||
|
|
||||||
|
Installed paths:
|
||||||
|
|
||||||
|
| Path | Role |
|
||||||
|
|------|------|
|
||||||
|
| `~/.local/bin/collect-landing-stats.sh` | orchestrator |
|
||||||
|
| `~/.local/lib/taler-landing/*.py` | bank scan + alt enrich |
|
||||||
|
| `~/.local/state/taler-landing-stats/` | logs |
|
||||||
|
| `~/.config/systemd/user/taler-landing-stats.*` | user units |
|
||||||
|
|
||||||
|
## What the collector does
|
||||||
|
|
||||||
|
1. **Bank** — `collect_bank_stats.py`
|
||||||
|
- admin token → list **all** accounts
|
||||||
|
- for each account (except `SCAN_SKIP`, default **`exchange`**) page through **all** transactions
|
||||||
|
- sum credits / Taler withdraws / other debits
|
||||||
|
- emit `amount` + `amount_alt` / `amount_full`
|
||||||
|
- `podman cp` → `taler-hacktivism-bank:/var/www/bank-landing/stats.json`
|
||||||
|
|
||||||
|
2. **Exchange** — run `landing-stats-exchange.sh` inside exchange container, then **enrich** alt fields on the host and write back.
|
||||||
|
|
||||||
|
3. **Merchant** — same for merchant container.
|
||||||
|
|
||||||
|
`exchange` is skipped in the bank **flow** scan so the same GOA is not counted once as customer withdraw and again as exchange credit. **Admin**, **explorer**, and every auto-account are included.
|
||||||
|
|
||||||
|
## Secrets
|
||||||
|
|
||||||
|
Preferred order:
|
||||||
|
|
||||||
|
1. `~/src/koopa/koopa-admin-secrets/koopa/host-root/taler-bank/bank-admin-password.txt`
|
||||||
|
2. `~/.config/taler-landing/bank-*-password.txt`
|
||||||
|
3. `podman exec taler-hacktivism-bank cat /root/bank-admin-password.txt`
|
||||||
|
|
||||||
|
Explorer password optional (shared-pool balance).
|
||||||
|
|
||||||
|
## Env overrides
|
||||||
|
|
||||||
|
| Env | Default | Meaning |
|
||||||
|
|-----|---------|---------|
|
||||||
|
| `BANK_URL` | `http://127.0.0.1:9012` | libeufin loopback |
|
||||||
|
| `SCAN_SKIP` | `exchange` | usernames excluded from flow |
|
||||||
|
| `TX_PAGE` | `500` | transactions page size |
|
||||||
|
| `MAX_TX_PAGES` | `0` | `0` = unlimited pages / account |
|
||||||
|
| `ACCOUNTS_DELTA` | `-10000` | account list window |
|
||||||
|
| `ADMIN_LOG` | `%h/src/koopa/koopa-admin-log` | refresh in-container scripts |
|
||||||
|
|
||||||
|
## UI alt names
|
||||||
|
|
||||||
|
`configs/shared/goa-amount.js` formats large amounts as e.g. `1.23 Mega-GOA` (tooltip = full `GOA:…`). Landings load it as `/intro/goa-amount.js` (deploy via `deploy-landings.sh`).
|
||||||
|
|
||||||
|
## Legacy in-container bank cron
|
||||||
|
|
||||||
|
`scripts/taler-bank/landing-stats.sh` remains a **fallback** inside the bank container. Once the hernani timer is healthy, disable the old in-container minutely cron to avoid races:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
podman exec taler-hacktivism-bank crontab -l # inspect
|
||||||
|
# remove landing-stats.sh line if present
|
||||||
|
```
|
||||||
|
|
||||||
|
## Manual run
|
||||||
|
|
||||||
|
```bash
|
||||||
|
~/.local/bin/collect-landing-stats.sh
|
||||||
|
# or from checkout:
|
||||||
|
./scripts/taler-landing/collect-landing-stats.sh
|
||||||
|
|
||||||
|
curl -sS https://bank.hacktivism.ch/intro/stats.json | python3 -m json.tool | head
|
||||||
|
```
|
||||||
260
scripts/taler-landing/collect-landing-stats.sh
Executable file
260
scripts/taler-landing/collect-landing-stats.sh
Executable file
|
|
@ -0,0 +1,260 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# Central landing-stats collector for hernani@koopa (user systemd timer).
|
||||||
|
#
|
||||||
|
# - Bank: full account+ledger scan via collect_bank_stats.py → bank container
|
||||||
|
# - Exchange / merchant: existing in-container scripts, then amount_alt enrich
|
||||||
|
# - Never wipes a good stats.json on failure (writes stats-run.json only)
|
||||||
|
#
|
||||||
|
# Install: scripts/taler-landing/install-landing-stats-host.sh
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
export TZ="${TZ:-Europe/Zurich}"
|
||||||
|
export PATH="${HOME}/.local/bin:/usr/local/bin:/usr/bin:/bin${PATH:+:$PATH}"
|
||||||
|
|
||||||
|
log() { printf '%s %s\n' "$(date -Iseconds)" "$*"; }
|
||||||
|
|
||||||
|
ROOT="$(cd "$(dirname "$0")" && pwd)"
|
||||||
|
# When installed to ~/.local/bin, libs live in ~/.local/lib/taler-landing
|
||||||
|
if [ -f "$ROOT/collect_bank_stats.py" ]; then
|
||||||
|
LIB="$ROOT"
|
||||||
|
elif [ -f "${HOME}/.local/lib/taler-landing/collect_bank_stats.py" ]; then
|
||||||
|
LIB="${HOME}/.local/lib/taler-landing"
|
||||||
|
elif [ -f "${HOME}/src/koopa/koopa-admin-log/scripts/taler-landing/collect_bank_stats.py" ]; then
|
||||||
|
LIB="${HOME}/src/koopa/koopa-admin-log/scripts/taler-landing"
|
||||||
|
else
|
||||||
|
LIB="$ROOT"
|
||||||
|
fi
|
||||||
|
|
||||||
|
ADMIN_LOG="${ADMIN_LOG:-${HOME}/src/koopa/koopa-admin-log}"
|
||||||
|
SECRETS_BANK="${SECRETS_BANK:-${HOME}/src/koopa/koopa-admin-secrets/koopa/host-root/taler-bank}"
|
||||||
|
|
||||||
|
BANK_CTR="${BANK_CTR:-taler-hacktivism-bank}"
|
||||||
|
EX_CTR="${EX_CTR:-taler-hacktivism-exchange-ansible}"
|
||||||
|
MER_CTR="${MER_CTR:-taler-hacktivism}"
|
||||||
|
|
||||||
|
BANK_URL="${BANK_URL:-http://127.0.0.1:9012}"
|
||||||
|
BANK_PUBLIC_URL="${BANK_PUBLIC_URL:-https://bank.hacktivism.ch}"
|
||||||
|
EXCHANGE_CONFIG_URL="${EXCHANGE_CONFIG_URL:-https://exchange.hacktivism.ch/config}"
|
||||||
|
|
||||||
|
BANK_LANDING_IN="${BANK_LANDING_IN:-/var/www/bank-landing}"
|
||||||
|
EX_LANDING_IN="${EX_LANDING_IN:-/var/www/exchange-landing}"
|
||||||
|
MER_LANDING_IN="${MER_LANDING_IN:-/var/www/merchant-landing}"
|
||||||
|
|
||||||
|
WORKDIR="${LANDING_STATS_WORKDIR:-${XDG_RUNTIME_DIR:-/tmp}/taler-landing-stats}"
|
||||||
|
mkdir -p "$WORKDIR"
|
||||||
|
LOG_DIR="${LANDING_STATS_LOGDIR:-${HOME}/.local/state/taler-landing-stats}"
|
||||||
|
mkdir -p "$LOG_DIR"
|
||||||
|
|
||||||
|
PY="${PYTHON:-python3}"
|
||||||
|
ec_bank=0
|
||||||
|
ec_ex=0
|
||||||
|
ec_mer=0
|
||||||
|
|
||||||
|
read_pass() {
|
||||||
|
local name="$1" f
|
||||||
|
for f in \
|
||||||
|
"${SECRETS_BANK}/${name}" \
|
||||||
|
"${HOME}/.config/taler-landing/${name}" \
|
||||||
|
"/run/user/$(id -u)/taler-landing/${name}"
|
||||||
|
do
|
||||||
|
if [ -r "$f" ]; then
|
||||||
|
tr -d '\n' <"$f"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
# container (hernani can podman exec root files inside owned containers)
|
||||||
|
if podman inspect -f '{{.State.Running}}' "$BANK_CTR" 2>/dev/null | grep -qx true; then
|
||||||
|
if podman exec "$BANK_CTR" test -r "/root/${name}" 2>/dev/null; then
|
||||||
|
podman exec "$BANK_CTR" cat "/root/${name}" 2>/dev/null | tr -d '\n'
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
ctr_running() {
|
||||||
|
podman inspect -f '{{.State.Running}}' "$1" 2>/dev/null | grep -qx true
|
||||||
|
}
|
||||||
|
|
||||||
|
publish_json() {
|
||||||
|
local ctr="$1" dest_dir="$2" src_stats="$3" src_run="$4"
|
||||||
|
if ! ctr_running "$ctr"; then
|
||||||
|
log "WARN: $ctr not running — skip publish $dest_dir"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
podman exec "$ctr" mkdir -p "$dest_dir" 2>/dev/null || true
|
||||||
|
if [ -f "$src_stats" ]; then
|
||||||
|
podman cp "$src_stats" "${ctr}:${dest_dir}/stats.json"
|
||||||
|
fi
|
||||||
|
if [ -f "$src_run" ]; then
|
||||||
|
podman cp "$src_run" "${ctr}:${dest_dir}/stats-run.json"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Bank — full scan (all accounts / all money except SCAN_SKIP)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
collect_bank() {
|
||||||
|
log "bank: collect full stats via $LIB/collect_bank_stats.py"
|
||||||
|
local admin_pass explorer_pass
|
||||||
|
admin_pass="$(read_pass bank-admin-password.txt || true)"
|
||||||
|
explorer_pass="$(read_pass bank-explorer-password.txt || true)"
|
||||||
|
if [ -z "$admin_pass" ]; then
|
||||||
|
log "ERROR: bank admin password not found (secrets or container)"
|
||||||
|
printf '%s\n' '{"ok":false,"error":"no admin password","at_human":"'"$(date +"%Y-%m-%d %H:%M %Z")"'"}' \
|
||||||
|
>"$WORKDIR/bank-stats-run.json"
|
||||||
|
publish_json "$BANK_CTR" "$BANK_LANDING_IN" "" "$WORKDIR/bank-stats-run.json" || true
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# optional demo files from container
|
||||||
|
local demo_dir=""
|
||||||
|
if ctr_running "$BANK_CTR"; then
|
||||||
|
demo_dir="$WORKDIR/demo"
|
||||||
|
mkdir -p "$demo_dir"
|
||||||
|
podman cp "${BANK_CTR}:${BANK_LANDING_IN}/withdraw.uri" "$demo_dir/withdraw.uri" 2>/dev/null || true
|
||||||
|
podman cp "${BANK_CTR}:${BANK_LANDING_IN}/withdraw.amount" "$demo_dir/withdraw.amount" 2>/dev/null || true
|
||||||
|
podman cp "${BANK_CTR}:${BANK_LANDING_IN}/withdraw.created" "$demo_dir/withdraw.created" 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
|
||||||
|
set +e
|
||||||
|
BANK_URL="$BANK_URL" \
|
||||||
|
BANK_PUBLIC_URL="$BANK_PUBLIC_URL" \
|
||||||
|
EXCHANGE_CONFIG_URL="$EXCHANGE_CONFIG_URL" \
|
||||||
|
BANK_ADMIN_PASS="$admin_pass" \
|
||||||
|
BANK_EXPLORER_PASS="$explorer_pass" \
|
||||||
|
DEMO_DIR="${demo_dir}" \
|
||||||
|
SCAN_SKIP="${SCAN_SKIP:-exchange}" \
|
||||||
|
TX_PAGE="${TX_PAGE:-500}" \
|
||||||
|
MAX_TX_PAGES="${MAX_TX_PAGES:-0}" \
|
||||||
|
ACCOUNTS_DELTA="${ACCOUNTS_DELTA:--10000}" \
|
||||||
|
"$PY" "$LIB/collect_bank_stats.py" \
|
||||||
|
--out "$WORKDIR/bank-stats.json" \
|
||||||
|
--run-out "$WORKDIR/bank-stats-run.json" \
|
||||||
|
>>"$LOG_DIR/bank.log" 2>&1
|
||||||
|
ec_bank=$?
|
||||||
|
set -e
|
||||||
|
|
||||||
|
if [ "$ec_bank" -eq 0 ] && [ -f "$WORKDIR/bank-stats.json" ]; then
|
||||||
|
# merge in-container memory snapshot when helper exists
|
||||||
|
if ctr_running "$BANK_CTR" && podman exec "$BANK_CTR" test -f /usr/local/lib/landing-mem-snapshot.sh 2>/dev/null; then
|
||||||
|
set +e
|
||||||
|
mem_json=$(podman exec "$BANK_CTR" bash -c '
|
||||||
|
export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
|
||||||
|
MEM_JSON=""
|
||||||
|
# shellcheck disable=SC1091
|
||||||
|
. /usr/local/lib/landing-mem-snapshot.sh
|
||||||
|
mem_snapshot_json 2>/dev/null || true
|
||||||
|
# print MEM_JSON lines only if function set them via echo — fallback empty
|
||||||
|
if [ -n "${MEM_JSON:-}" ]; then printf "%s" "$MEM_JSON"; fi
|
||||||
|
' 2>/dev/null)
|
||||||
|
set -e
|
||||||
|
if [ -n "${mem_json:-}" ]; then
|
||||||
|
"$PY" - "$WORKDIR/bank-stats.json" <<'PY' || true
|
||||||
|
import json, sys
|
||||||
|
path = sys.argv[1]
|
||||||
|
# optional: leave memory as-is if merge fails
|
||||||
|
print("mem merge skipped (structured merge via podman helper optional)", file=sys.stderr)
|
||||||
|
PY
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
publish_json "$BANK_CTR" "$BANK_LANDING_IN" \
|
||||||
|
"$WORKDIR/bank-stats.json" "$WORKDIR/bank-stats-run.json"
|
||||||
|
log "bank: OK → ${BANK_CTR}:${BANK_LANDING_IN}/stats.json"
|
||||||
|
else
|
||||||
|
log "ERROR: bank collect failed (ec=$ec_bank) — previous stats.json kept"
|
||||||
|
[ -f "$WORKDIR/bank-stats-run.json" ] || \
|
||||||
|
printf '%s\n' '{"ok":false,"error":"collect failed","at_human":"'"$(date +"%Y-%m-%d %H:%M %Z")"'"}' \
|
||||||
|
>"$WORKDIR/bank-stats-run.json"
|
||||||
|
publish_json "$BANK_CTR" "$BANK_LANDING_IN" "" "$WORKDIR/bank-stats-run.json" || true
|
||||||
|
fi
|
||||||
|
return "$ec_bank"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Exchange / merchant — in-container generators + host enrich
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
run_incontainer_stats() {
|
||||||
|
local label="$1" ctr="$2" script_candidates="$3" landing="$4"
|
||||||
|
local script="" s
|
||||||
|
if ! ctr_running "$ctr"; then
|
||||||
|
log "WARN: $label: $ctr not running"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
# install latest script from admin-log if present
|
||||||
|
local host_src=""
|
||||||
|
case "$label" in
|
||||||
|
exchange)
|
||||||
|
host_src="$ADMIN_LOG/scripts/taler-exchange/landing-stats-exchange.sh"
|
||||||
|
;;
|
||||||
|
merchant)
|
||||||
|
host_src="$ADMIN_LOG/scripts/taler-merchant/landing-stats-merchant.sh"
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
if [ -n "$host_src" ] && [ -f "$host_src" ]; then
|
||||||
|
podman cp "$host_src" "${ctr}:/usr/local/bin/$(basename "$host_src")"
|
||||||
|
podman exec "$ctr" chmod 755 "/usr/local/bin/$(basename "$host_src")" 2>/dev/null || true
|
||||||
|
script="/usr/local/bin/$(basename "$host_src")"
|
||||||
|
else
|
||||||
|
for s in $script_candidates; do
|
||||||
|
if podman exec "$ctr" test -x "$s" 2>/dev/null || podman exec "$ctr" test -f "$s" 2>/dev/null; then
|
||||||
|
script="$s"
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
if [ -z "$script" ]; then
|
||||||
|
log "WARN: $label: no landing-stats script in $ctr"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
log "$label: run $script inside $ctr"
|
||||||
|
set +e
|
||||||
|
podman exec \
|
||||||
|
-e LANDING_DIR="$landing" \
|
||||||
|
-e TZ=Europe/Zurich \
|
||||||
|
-e PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" \
|
||||||
|
"$ctr" bash "$script" >>"$LOG_DIR/${label}.log" 2>&1
|
||||||
|
local ec=$?
|
||||||
|
set -e
|
||||||
|
if [ "$ec" -ne 0 ]; then
|
||||||
|
log "ERROR: $label stats failed (ec=$ec)"
|
||||||
|
return "$ec"
|
||||||
|
fi
|
||||||
|
# enrich with alt units on host
|
||||||
|
podman cp "${ctr}:${landing}/stats.json" "$WORKDIR/${label}-stats.json" 2>/dev/null || return 0
|
||||||
|
set +e
|
||||||
|
"$PY" "$LIB/enrich_stats_alt.py" "$WORKDIR/${label}-stats.json" \
|
||||||
|
--exchange-config "$EXCHANGE_CONFIG_URL" >>"$LOG_DIR/${label}.log" 2>&1
|
||||||
|
set -e
|
||||||
|
podman cp "$WORKDIR/${label}-stats.json" "${ctr}:${landing}/stats.json"
|
||||||
|
log "$label: OK + alt enrich → ${ctr}:${landing}/stats.json"
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
collect_exchange() {
|
||||||
|
run_incontainer_stats exchange "$EX_CTR" \
|
||||||
|
"/usr/local/bin/landing-stats-exchange.sh /usr/local/bin/landing-stats.sh" \
|
||||||
|
"$EX_LANDING_IN"
|
||||||
|
}
|
||||||
|
|
||||||
|
collect_merchant() {
|
||||||
|
run_incontainer_stats merchant "$MER_CTR" \
|
||||||
|
"/usr/local/bin/landing-stats-merchant.sh /usr/local/bin/landing-stats.sh" \
|
||||||
|
"$MER_LANDING_IN"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
main() {
|
||||||
|
log "=== taler-landing-stats start (user=$(id -un) lib=$LIB) ==="
|
||||||
|
collect_bank || ec_bank=$?
|
||||||
|
collect_exchange || ec_ex=$?
|
||||||
|
collect_merchant || ec_mer=$?
|
||||||
|
log "=== done bank=$ec_bank exchange=$ec_ex merchant=$ec_mer ==="
|
||||||
|
# non-zero if bank failed (primary public flow numbers); soft on ex/mer
|
||||||
|
if [ "$ec_bank" -ne 0 ]; then
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
main "$@"
|
||||||
816
scripts/taler-landing/collect_bank_stats.py
Executable file
816
scripts/taler-landing/collect_bank_stats.py
Executable file
|
|
@ -0,0 +1,816 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Full-scan bank landing stats (all accounts, all ledger money).
|
||||||
|
|
||||||
|
Run on koopa as hernani (or anywhere with admin API access). Writes stats.json
|
||||||
|
compatible with bank.hacktivism.ch/intro/ plus amount_alt fields for compact UI.
|
||||||
|
|
||||||
|
Env (selected):
|
||||||
|
BANK_URL default http://127.0.0.1:9012
|
||||||
|
BANK_ADMIN_USER default admin
|
||||||
|
BANK_ADMIN_PASS or BANK_ADMIN_PASS_FILE / pass via --admin-pass-file
|
||||||
|
BANK_EXPLORER_USER default explorer
|
||||||
|
BANK_EXPLORER_PASS optional (balance_explorer)
|
||||||
|
EXCHANGE_CONFIG_URL for alt_unit_names (default https://exchange.hacktivism.ch/config)
|
||||||
|
OUT output path (default stdout if -)
|
||||||
|
SCAN_SKIP comma usernames excluded from flow (default: exchange)
|
||||||
|
TX_PAGE page size for transactions delta (default 500)
|
||||||
|
MAX_TX_PAGES 0 = unlimited pages per account (default 0)
|
||||||
|
ACCOUNTS_DELTA GET /accounts?delta= (default -10000)
|
||||||
|
RECENT_WD_N recent withdraws (default 10)
|
||||||
|
TZ default Europe/Zurich
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
import urllib.error
|
||||||
|
import urllib.parse
|
||||||
|
import urllib.request
|
||||||
|
from datetime import datetime
|
||||||
|
from decimal import Decimal
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any, Dict, List, Optional, Set, Tuple
|
||||||
|
from zoneinfo import ZoneInfo
|
||||||
|
|
||||||
|
# local import (same directory)
|
||||||
|
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||||
|
from goa_amounts import ( # noqa: E402
|
||||||
|
DEFAULT_ALT,
|
||||||
|
enrich_stats_tree,
|
||||||
|
format_amount_alt,
|
||||||
|
load_alt_from_config,
|
||||||
|
parse_amount,
|
||||||
|
)
|
||||||
|
|
||||||
|
RESERVE_RE = re.compile(r"(?i)withdrawal\s+([A-Za-z0-9]+)")
|
||||||
|
|
||||||
|
|
||||||
|
def env(name: str, default: str = "") -> str:
|
||||||
|
return os.environ.get(name, default)
|
||||||
|
|
||||||
|
|
||||||
|
def read_pass_file(path: str) -> str:
|
||||||
|
p = Path(path)
|
||||||
|
if p.is_file():
|
||||||
|
return p.read_text(encoding="utf-8", errors="replace").strip()
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
def http_json(
|
||||||
|
url: str,
|
||||||
|
*,
|
||||||
|
method: str = "GET",
|
||||||
|
headers: Optional[Dict[str, str]] = None,
|
||||||
|
data: Optional[bytes] = None,
|
||||||
|
timeout: float = 30.0,
|
||||||
|
auth: Optional[Tuple[str, str]] = None,
|
||||||
|
) -> Tuple[int, Any, bytes]:
|
||||||
|
h = dict(headers or {})
|
||||||
|
if auth:
|
||||||
|
import base64
|
||||||
|
|
||||||
|
token = base64.b64encode(f"{auth[0]}:{auth[1]}".encode()).decode("ascii")
|
||||||
|
h["Authorization"] = f"Basic {token}"
|
||||||
|
req = urllib.request.Request(url, data=data, headers=h, method=method)
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||||
|
body = resp.read()
|
||||||
|
code = getattr(resp, "status", 200) or 200
|
||||||
|
except urllib.error.HTTPError as e:
|
||||||
|
body = e.read() if e.fp else b""
|
||||||
|
code = e.code
|
||||||
|
except Exception as e:
|
||||||
|
return 0, None, str(e).encode()
|
||||||
|
|
||||||
|
if not body:
|
||||||
|
return code, None, body
|
||||||
|
try:
|
||||||
|
return code, json.loads(body.decode("utf-8", errors="replace")), body
|
||||||
|
except Exception:
|
||||||
|
return code, None, body
|
||||||
|
|
||||||
|
|
||||||
|
def measure_ms(url: str, timeout: float = 8.0) -> Tuple[Optional[int], str]:
|
||||||
|
t0 = time.perf_counter()
|
||||||
|
code = "000"
|
||||||
|
try:
|
||||||
|
req = urllib.request.Request(url, method="GET")
|
||||||
|
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||||
|
resp.read()
|
||||||
|
code = str(getattr(resp, "status", 200) or 200)
|
||||||
|
except urllib.error.HTTPError as e:
|
||||||
|
code = str(e.code)
|
||||||
|
except Exception:
|
||||||
|
return None, "000"
|
||||||
|
ms = int(round((time.perf_counter() - t0) * 1000))
|
||||||
|
if ms == 0:
|
||||||
|
ms = 1
|
||||||
|
return ms, code
|
||||||
|
|
||||||
|
|
||||||
|
def get_token(bank: str, user: str, password: str) -> str:
|
||||||
|
code, data, _ = http_json(
|
||||||
|
f"{bank}/accounts/{user}/token",
|
||||||
|
method="POST",
|
||||||
|
headers={"Content-Type": "application/json"},
|
||||||
|
data=b'{"scope":"readonly"}',
|
||||||
|
auth=(user, password),
|
||||||
|
timeout=15,
|
||||||
|
)
|
||||||
|
if code not in (200, 201) or not isinstance(data, dict):
|
||||||
|
raise RuntimeError(f"token failed for {user}: HTTP {code}")
|
||||||
|
tok = data.get("access_token") or data.get("token") or ""
|
||||||
|
if not tok:
|
||||||
|
raise RuntimeError(f"token empty for {user}")
|
||||||
|
return str(tok)
|
||||||
|
|
||||||
|
|
||||||
|
def list_all_accounts(bank: str, token: str, delta: int) -> List[str]:
|
||||||
|
"""List accounts; page with start if needed."""
|
||||||
|
names: List[str] = []
|
||||||
|
seen: Set[str] = set()
|
||||||
|
start: Optional[int] = None
|
||||||
|
pages = 0
|
||||||
|
max_pages = int(env("MAX_ACCOUNT_PAGES", "50") or "50")
|
||||||
|
|
||||||
|
while pages < max_pages:
|
||||||
|
pages += 1
|
||||||
|
q = f"delta={delta}"
|
||||||
|
if start is not None:
|
||||||
|
q += f"&start={start}"
|
||||||
|
code, data, raw = http_json(
|
||||||
|
f"{bank}/accounts?{q}",
|
||||||
|
headers={"Authorization": f"Bearer {token}"},
|
||||||
|
timeout=45,
|
||||||
|
)
|
||||||
|
if code == 204 or not data:
|
||||||
|
break
|
||||||
|
if code != 200:
|
||||||
|
raise RuntimeError(f"list accounts HTTP {code}: {raw[:200]!r}")
|
||||||
|
|
||||||
|
batch: List[Dict[str, Any]] = []
|
||||||
|
if isinstance(data, dict):
|
||||||
|
if isinstance(data.get("accounts"), list):
|
||||||
|
batch = data["accounts"]
|
||||||
|
elif isinstance(data.get("users"), list):
|
||||||
|
batch = data["users"]
|
||||||
|
elif isinstance(data, list):
|
||||||
|
batch = data
|
||||||
|
|
||||||
|
if not batch:
|
||||||
|
# flat object with username? try regex fallback
|
||||||
|
text = raw.decode("utf-8", errors="replace")
|
||||||
|
for m in re.finditer(r'"username"\s*:\s*"([^"]+)"', text):
|
||||||
|
u = m.group(1)
|
||||||
|
if u not in seen:
|
||||||
|
seen.add(u)
|
||||||
|
names.append(u)
|
||||||
|
break
|
||||||
|
|
||||||
|
min_row = None
|
||||||
|
for item in batch:
|
||||||
|
if not isinstance(item, dict):
|
||||||
|
continue
|
||||||
|
u = item.get("username") or item.get("name") or ""
|
||||||
|
if u and u not in seen:
|
||||||
|
seen.add(u)
|
||||||
|
names.append(str(u))
|
||||||
|
rid = item.get("row_id") or item.get("rowId")
|
||||||
|
if rid is not None:
|
||||||
|
try:
|
||||||
|
rid_i = int(rid)
|
||||||
|
min_row = rid_i if min_row is None else min(min_row, rid_i)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
if len(batch) < abs(delta):
|
||||||
|
break
|
||||||
|
if min_row is None:
|
||||||
|
break
|
||||||
|
# next older page
|
||||||
|
start = min_row
|
||||||
|
# avoid infinite loop on same start
|
||||||
|
if pages > 1 and len(names) == len(seen):
|
||||||
|
# if no growth and full page, still advance
|
||||||
|
pass
|
||||||
|
|
||||||
|
return names
|
||||||
|
|
||||||
|
|
||||||
|
def iter_transactions(
|
||||||
|
bank: str,
|
||||||
|
token: str,
|
||||||
|
username: str,
|
||||||
|
page: int,
|
||||||
|
max_pages: int,
|
||||||
|
) -> List[Dict[str, Any]]:
|
||||||
|
"""Return all transactions for account (paginated)."""
|
||||||
|
out: List[Dict[str, Any]] = []
|
||||||
|
start: Optional[int] = None
|
||||||
|
pages = 0
|
||||||
|
safety = max_pages if max_pages > 0 else 10_000
|
||||||
|
|
||||||
|
while pages < safety:
|
||||||
|
pages += 1
|
||||||
|
q = f"delta=-{abs(page)}"
|
||||||
|
if start is not None:
|
||||||
|
q += f"&start={start}"
|
||||||
|
code, data, raw = http_json(
|
||||||
|
f"{bank}/accounts/{urllib.parse.quote(username)}/transactions?{q}",
|
||||||
|
headers={"Authorization": f"Bearer {token}"},
|
||||||
|
timeout=float(env("TX_CURL_TIMEOUT", "45") or "45"),
|
||||||
|
)
|
||||||
|
if code in (204, 404) or not data:
|
||||||
|
break
|
||||||
|
if code != 200:
|
||||||
|
# soft-fail one account
|
||||||
|
break
|
||||||
|
|
||||||
|
txs: List[Dict[str, Any]] = []
|
||||||
|
if isinstance(data, dict) and isinstance(data.get("transactions"), list):
|
||||||
|
txs = data["transactions"]
|
||||||
|
elif isinstance(data, list):
|
||||||
|
txs = data
|
||||||
|
else:
|
||||||
|
# tolerate raw array-ish
|
||||||
|
break
|
||||||
|
|
||||||
|
if not txs:
|
||||||
|
break
|
||||||
|
|
||||||
|
min_row = None
|
||||||
|
for tx in txs:
|
||||||
|
if not isinstance(tx, dict):
|
||||||
|
continue
|
||||||
|
out.append(tx)
|
||||||
|
rid = tx.get("row_id") or tx.get("rowId")
|
||||||
|
if rid is not None:
|
||||||
|
try:
|
||||||
|
rid_i = int(rid)
|
||||||
|
min_row = rid_i if min_row is None else min(min_row, rid_i)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
if len(txs) < abs(page):
|
||||||
|
break
|
||||||
|
if min_row is None:
|
||||||
|
break
|
||||||
|
start = min_row
|
||||||
|
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def tx_fields(tx: Dict[str, Any]) -> Tuple[str, str, int, str]:
|
||||||
|
"""direction, amount, unix_ts, subject"""
|
||||||
|
direction = str(tx.get("direction") or "").lower()
|
||||||
|
amount = str(tx.get("amount") or "")
|
||||||
|
subject = str(tx.get("subject") or tx.get("description") or "")
|
||||||
|
ts = 0
|
||||||
|
when = tx.get("when") or tx.get("date") or tx.get("timestamp")
|
||||||
|
if isinstance(when, dict):
|
||||||
|
# Taler AbsoluteTime: t_s seconds or t_ms
|
||||||
|
if "t_s" in when:
|
||||||
|
try:
|
||||||
|
ts = int(when["t_s"])
|
||||||
|
except Exception:
|
||||||
|
ts = 0
|
||||||
|
elif "t_ms" in when:
|
||||||
|
try:
|
||||||
|
ts = int(int(when["t_ms"]) / 1000)
|
||||||
|
except Exception:
|
||||||
|
ts = 0
|
||||||
|
elif isinstance(when, (int, float)):
|
||||||
|
ts = int(when)
|
||||||
|
if ts > 10_000_000_000: # ms
|
||||||
|
ts //= 1000
|
||||||
|
elif tx.get("t_s") is not None:
|
||||||
|
try:
|
||||||
|
ts = int(tx["t_s"])
|
||||||
|
except Exception:
|
||||||
|
ts = 0
|
||||||
|
return direction, amount, ts, subject
|
||||||
|
|
||||||
|
|
||||||
|
def reserve_from_subject(subject: str) -> str:
|
||||||
|
m = RESERVE_RE.search(subject or "")
|
||||||
|
if m:
|
||||||
|
return re.sub(r"[^A-Za-z0-9]", "", m.group(1))
|
||||||
|
# fallback: last token
|
||||||
|
parts = (subject or "").split()
|
||||||
|
if parts:
|
||||||
|
return re.sub(r"[^A-Za-z0-9]", "", parts[-1])
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
def now_parts(tz_name: str) -> Tuple[int, str, str]:
|
||||||
|
tz = ZoneInfo(tz_name)
|
||||||
|
dt = datetime.now(tz)
|
||||||
|
unix = int(dt.timestamp())
|
||||||
|
iso = dt.strftime("%Y-%m-%dT%H:%M%z")
|
||||||
|
# +0200 → +02:00
|
||||||
|
if len(iso) >= 5 and iso[-5] in "+-" and ":" not in iso[-5:]:
|
||||||
|
iso = iso[:-2] + ":" + iso[-2:]
|
||||||
|
human = dt.strftime("%Y-%m-%d %H:%M %Z")
|
||||||
|
return unix, iso, human
|
||||||
|
|
||||||
|
|
||||||
|
def human_from_unix(ts: int, tz_name: str) -> Tuple[str, str]:
|
||||||
|
if not ts:
|
||||||
|
return "", ""
|
||||||
|
tz = ZoneInfo(tz_name)
|
||||||
|
dt = datetime.fromtimestamp(ts, tz)
|
||||||
|
iso = dt.strftime("%Y-%m-%dT%H:%M%z")
|
||||||
|
if len(iso) >= 5 and iso[-5] in "+-" and ":" not in iso[-5:]:
|
||||||
|
iso = iso[:-2] + ":" + iso[-2:]
|
||||||
|
human = dt.strftime("%Y-%m-%d %H:%M %Z")
|
||||||
|
return human, iso
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
ap = argparse.ArgumentParser(description="Collect full bank landing stats")
|
||||||
|
ap.add_argument("--bank", default=env("BANK_URL", "http://127.0.0.1:9012"))
|
||||||
|
ap.add_argument("--admin-user", default=env("BANK_ADMIN_USER", "admin"))
|
||||||
|
ap.add_argument("--admin-pass", default=env("BANK_ADMIN_PASS", ""))
|
||||||
|
ap.add_argument("--admin-pass-file", default=env("BANK_ADMIN_PASS_FILE", ""))
|
||||||
|
ap.add_argument("--explorer-user", default=env("BANK_EXPLORER_USER", "explorer"))
|
||||||
|
ap.add_argument("--explorer-pass", default=env("BANK_EXPLORER_PASS", ""))
|
||||||
|
ap.add_argument("--explorer-pass-file", default=env("BANK_EXPLORER_PASS_FILE", ""))
|
||||||
|
ap.add_argument(
|
||||||
|
"--exchange-config",
|
||||||
|
default=env("EXCHANGE_CONFIG_URL", "https://exchange.hacktivism.ch/config"),
|
||||||
|
)
|
||||||
|
ap.add_argument("--out", default=env("OUT", "-"))
|
||||||
|
ap.add_argument("--run-out", default=env("RUN_OUT", ""))
|
||||||
|
ap.add_argument(
|
||||||
|
"--skip",
|
||||||
|
default=env("SCAN_SKIP", "exchange"),
|
||||||
|
help="comma usernames excluded from flow scan (default: exchange)",
|
||||||
|
)
|
||||||
|
ap.add_argument("--tx-page", type=int, default=int(env("TX_PAGE", "500") or "500"))
|
||||||
|
ap.add_argument(
|
||||||
|
"--max-tx-pages",
|
||||||
|
type=int,
|
||||||
|
default=int(env("MAX_TX_PAGES", "0") or "0"),
|
||||||
|
help="0 = unlimited pages per account",
|
||||||
|
)
|
||||||
|
ap.add_argument(
|
||||||
|
"--accounts-delta",
|
||||||
|
type=int,
|
||||||
|
default=int(env("ACCOUNTS_DELTA", "-10000") or "-10000"),
|
||||||
|
)
|
||||||
|
ap.add_argument("--recent", type=int, default=int(env("RECENT_WD_N", "10") or "10"))
|
||||||
|
ap.add_argument("--public-base", default=env("BANK_PUBLIC_URL", "https://bank.hacktivism.ch"))
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
tz_name = env("TZ", "Europe/Zurich") or "Europe/Zurich"
|
||||||
|
os.environ["TZ"] = tz_name
|
||||||
|
|
||||||
|
admin_pass = args.admin_pass or (
|
||||||
|
read_pass_file(args.admin_pass_file) if args.admin_pass_file else ""
|
||||||
|
)
|
||||||
|
explorer_pass = args.explorer_pass or (
|
||||||
|
read_pass_file(args.explorer_pass_file) if args.explorer_pass_file else ""
|
||||||
|
)
|
||||||
|
|
||||||
|
bank = args.bank.rstrip("/")
|
||||||
|
skip = {s.strip() for s in (args.skip or "").split(",") if s.strip()}
|
||||||
|
|
||||||
|
def write_run(ok: bool, err: Optional[str] = None) -> None:
|
||||||
|
if not args.run_out:
|
||||||
|
return
|
||||||
|
unix, iso, human = now_parts(tz_name)
|
||||||
|
payload = {
|
||||||
|
"ok": ok,
|
||||||
|
"at": iso,
|
||||||
|
"at_human": human,
|
||||||
|
"error": err,
|
||||||
|
}
|
||||||
|
Path(args.run_out).parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
Path(args.run_out).write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")
|
||||||
|
|
||||||
|
try:
|
||||||
|
if not admin_pass:
|
||||||
|
raise RuntimeError("no admin password (BANK_ADMIN_PASS or --admin-pass-file)")
|
||||||
|
token = get_token(bank, args.admin_user, admin_pass)
|
||||||
|
alt = load_alt_from_config(args.exchange_config)
|
||||||
|
if not alt:
|
||||||
|
alt = dict(DEFAULT_ALT)
|
||||||
|
|
||||||
|
accounts = list_all_accounts(bank, token, args.accounts_delta)
|
||||||
|
if not accounts:
|
||||||
|
raise RuntimeError("accounts list empty")
|
||||||
|
|
||||||
|
# ALL accounts except skip set (default: only exchange — avoid double-count)
|
||||||
|
scan_users = [u for u in accounts if u not in skip]
|
||||||
|
# ensure explorer is included when present
|
||||||
|
if args.explorer_user not in skip and args.explorer_user in accounts:
|
||||||
|
if args.explorer_user not in scan_users:
|
||||||
|
scan_users.append(args.explorer_user)
|
||||||
|
|
||||||
|
withdraws: List[Dict[str, Any]] = []
|
||||||
|
incomings: List[Dict[str, Any]] = []
|
||||||
|
total_in = Decimal(0)
|
||||||
|
total_wd = Decimal(0)
|
||||||
|
total_other = Decimal(0)
|
||||||
|
n_incoming = 0
|
||||||
|
scan_ok = 0
|
||||||
|
scan_empty = 0
|
||||||
|
scan_fail = 0
|
||||||
|
|
||||||
|
for uname in scan_users:
|
||||||
|
try:
|
||||||
|
txs = iter_transactions(
|
||||||
|
bank, token, uname, args.tx_page, args.max_tx_pages
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
scan_fail += 1
|
||||||
|
continue
|
||||||
|
if not txs:
|
||||||
|
scan_empty += 1
|
||||||
|
continue
|
||||||
|
scan_ok += 1
|
||||||
|
for tx in txs:
|
||||||
|
direction, amount, ts, subject = tx_fields(tx)
|
||||||
|
if not amount:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
_cur, n = parse_amount(amount)
|
||||||
|
except Exception:
|
||||||
|
continue
|
||||||
|
low = (subject or "").lower()
|
||||||
|
if direction == "credit":
|
||||||
|
total_in += n
|
||||||
|
n_incoming += 1
|
||||||
|
incomings.append(
|
||||||
|
{
|
||||||
|
"kind": "incoming",
|
||||||
|
"amount": amount if ":" in amount else f"GOA:{amount}",
|
||||||
|
"at_unix": ts,
|
||||||
|
"account": uname,
|
||||||
|
"subject": subject,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
elif direction == "debit" and "withdraw" in low:
|
||||||
|
total_wd += n
|
||||||
|
res = reserve_from_subject(subject)
|
||||||
|
withdraws.append(
|
||||||
|
{
|
||||||
|
"kind": "withdraw",
|
||||||
|
"amount": amount if ":" in amount else f"GOA:{amount}",
|
||||||
|
"at_unix": ts,
|
||||||
|
"account": uname,
|
||||||
|
"subject": subject,
|
||||||
|
"reserve": res,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
elif direction == "debit":
|
||||||
|
total_other += n
|
||||||
|
|
||||||
|
withdraws.sort(key=lambda r: r.get("at_unix") or 0, reverse=True)
|
||||||
|
incomings.sort(key=lambda r: r.get("at_unix") or 0, reverse=True)
|
||||||
|
|
||||||
|
now_u, gen_iso, gen_human = now_parts(tz_name)
|
||||||
|
day = now_u - 86400
|
||||||
|
week = now_u - 7 * 86400
|
||||||
|
|
||||||
|
w24 = Decimal(0)
|
||||||
|
w7 = Decimal(0)
|
||||||
|
n24 = 0
|
||||||
|
n7 = 0
|
||||||
|
for w in withdraws:
|
||||||
|
ts = int(w.get("at_unix") or 0)
|
||||||
|
try:
|
||||||
|
_c, n = parse_amount(w.get("amount"))
|
||||||
|
except Exception:
|
||||||
|
n = Decimal(0)
|
||||||
|
if ts >= day:
|
||||||
|
w24 += n
|
||||||
|
n24 += 1
|
||||||
|
if ts >= week:
|
||||||
|
w7 += n
|
||||||
|
n7 += 1
|
||||||
|
|
||||||
|
wallets = sorted(
|
||||||
|
{w.get("reserve") for w in withdraws if w.get("reserve")}
|
||||||
|
)
|
||||||
|
accounts_with_wd = sorted(
|
||||||
|
{w.get("account") for w in withdraws if w.get("account")}
|
||||||
|
)
|
||||||
|
users_n = sum(1 for u in accounts if u not in ("admin", "exchange"))
|
||||||
|
|
||||||
|
def pack_amt(d: Decimal) -> Dict[str, Any]:
|
||||||
|
f = format_amount_alt(f"GOA:{d}", alt)
|
||||||
|
return {
|
||||||
|
"amount": f["amount"],
|
||||||
|
"amount_alt": f["amount_alt"],
|
||||||
|
"amount_full": f["amount_full"],
|
||||||
|
"value": f["value"],
|
||||||
|
"value_str": f["value_str"],
|
||||||
|
}
|
||||||
|
|
||||||
|
fin = pack_amt(total_in)
|
||||||
|
fwd = pack_amt(total_wd)
|
||||||
|
foth = pack_amt(total_other)
|
||||||
|
fout = pack_amt(total_wd + total_other)
|
||||||
|
p24 = pack_amt(w24)
|
||||||
|
p7 = pack_amt(w7)
|
||||||
|
|
||||||
|
last = withdraws[0] if withdraws else None
|
||||||
|
last_amt = None
|
||||||
|
last_at = last_iso = last_subj = None
|
||||||
|
last_unix = None
|
||||||
|
last_alt = None
|
||||||
|
if last:
|
||||||
|
lf = format_amount_alt(last["amount"], alt)
|
||||||
|
last_amt = lf["amount"]
|
||||||
|
last_alt = lf["amount_alt"]
|
||||||
|
last_unix = last.get("at_unix")
|
||||||
|
last_at, last_iso = human_from_unix(int(last_unix or 0), tz_name)
|
||||||
|
last_subj = last.get("subject")
|
||||||
|
|
||||||
|
recent = []
|
||||||
|
for w in withdraws[: max(0, args.recent)]:
|
||||||
|
f = format_amount_alt(w["amount"], alt)
|
||||||
|
at_h, at_iso = human_from_unix(int(w.get("at_unix") or 0), tz_name)
|
||||||
|
recent.append(
|
||||||
|
{
|
||||||
|
"kind": "withdraw",
|
||||||
|
"amount": f["amount"],
|
||||||
|
"amount_alt": f["amount_alt"],
|
||||||
|
"amount_full": f["amount_full"],
|
||||||
|
"at": at_h,
|
||||||
|
"at_iso": at_iso,
|
||||||
|
"at_unix": w.get("at_unix") or None,
|
||||||
|
"account": w.get("account"),
|
||||||
|
"reserve": w.get("reserve") or "",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
recent_in = []
|
||||||
|
for row in incomings[:5]:
|
||||||
|
f = format_amount_alt(row["amount"], alt)
|
||||||
|
at_h, at_iso = human_from_unix(int(row.get("at_unix") or 0), tz_name)
|
||||||
|
recent_in.append(
|
||||||
|
{
|
||||||
|
"kind": "incoming",
|
||||||
|
"amount": f["amount"],
|
||||||
|
"amount_alt": f["amount_alt"],
|
||||||
|
"amount_full": f["amount_full"],
|
||||||
|
"at": at_h,
|
||||||
|
"at_iso": at_iso,
|
||||||
|
"at_unix": row.get("at_unix") or None,
|
||||||
|
"account": row.get("account"),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
# explorer balance
|
||||||
|
balance = "GOA:0"
|
||||||
|
bal_alt = "GOA:0"
|
||||||
|
bal_full = "GOA:0"
|
||||||
|
if explorer_pass:
|
||||||
|
try:
|
||||||
|
etok = get_token(bank, args.explorer_user, explorer_pass)
|
||||||
|
code, data, _ = http_json(
|
||||||
|
f"{bank}/accounts/{args.explorer_user}",
|
||||||
|
headers={"Authorization": f"Bearer {etok}"},
|
||||||
|
timeout=12,
|
||||||
|
)
|
||||||
|
if code == 200 and isinstance(data, dict):
|
||||||
|
balance = str(
|
||||||
|
data.get("balance", {}).get("amount")
|
||||||
|
if isinstance(data.get("balance"), dict)
|
||||||
|
else data.get("amount") or data.get("balance") or "GOA:0"
|
||||||
|
)
|
||||||
|
# sometimes balance is object with amount
|
||||||
|
if isinstance(data.get("balance"), dict) and data["balance"].get(
|
||||||
|
"amount"
|
||||||
|
):
|
||||||
|
balance = str(data["balance"]["amount"])
|
||||||
|
bf = format_amount_alt(balance, alt)
|
||||||
|
balance, bal_alt, bal_full = (
|
||||||
|
bf["amount"],
|
||||||
|
bf["amount_alt"],
|
||||||
|
bf["amount_full"],
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
else:
|
||||||
|
# try admin-read of explorer account
|
||||||
|
try:
|
||||||
|
code, data, _ = http_json(
|
||||||
|
f"{bank}/accounts/{args.explorer_user}",
|
||||||
|
headers={"Authorization": f"Bearer {token}"},
|
||||||
|
timeout=12,
|
||||||
|
)
|
||||||
|
if code == 200 and isinstance(data, dict):
|
||||||
|
if isinstance(data.get("balance"), dict):
|
||||||
|
balance = str(data["balance"].get("amount") or "GOA:0")
|
||||||
|
elif data.get("amount"):
|
||||||
|
balance = str(data["amount"])
|
||||||
|
bf = format_amount_alt(balance, alt)
|
||||||
|
balance, bal_alt, bal_full = (
|
||||||
|
bf["amount"],
|
||||||
|
bf["amount_alt"],
|
||||||
|
bf["amount_full"],
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
pub = args.public_base.rstrip("/")
|
||||||
|
config_ms, config_http = measure_ms(f"{pub}/config")
|
||||||
|
int_ms, int_http = measure_ms(f"{pub}/taler-integration/config")
|
||||||
|
webui_ms, webui_http = measure_ms(f"{pub}/webui/")
|
||||||
|
if config_http != "200":
|
||||||
|
config_ms, config_http = measure_ms(f"{bank}/config")
|
||||||
|
if int_http != "200":
|
||||||
|
int_ms, int_http = measure_ms(f"{bank}/taler-integration/config")
|
||||||
|
|
||||||
|
loadavg = ""
|
||||||
|
try:
|
||||||
|
with open("/proc/loadavg", encoding="utf-8") as f:
|
||||||
|
parts = f.read().split()
|
||||||
|
loadavg = ",".join(parts[:3])
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
total_pack = pack_amt(total_wd)
|
||||||
|
|
||||||
|
stats = {
|
||||||
|
"ok": True,
|
||||||
|
"currency": "GOA",
|
||||||
|
"timezone": tz_name,
|
||||||
|
"generated_at": gen_iso,
|
||||||
|
"generated_at_human": gen_human,
|
||||||
|
"generated_at_unix": now_u,
|
||||||
|
"source": "host collect_bank_stats.py (hernani systemd)",
|
||||||
|
"bank_url": bank,
|
||||||
|
"scan": {
|
||||||
|
"tx_page": args.tx_page,
|
||||||
|
"max_tx_pages": args.max_tx_pages,
|
||||||
|
"accounts_delta": args.accounts_delta,
|
||||||
|
"skip_users": sorted(skip),
|
||||||
|
"accounts_listed": len(accounts),
|
||||||
|
"accounts_scanned_ok": scan_ok,
|
||||||
|
"accounts_empty_tx": scan_empty,
|
||||||
|
"accounts_scan_fail": scan_fail,
|
||||||
|
"accounts_scanned": len(scan_users),
|
||||||
|
"note": "all accounts except skip_users; full tx pagination; empty includes HTTP 204",
|
||||||
|
},
|
||||||
|
"bank_accounts": {
|
||||||
|
"total": len(accounts),
|
||||||
|
"users": users_n,
|
||||||
|
"with_withdraws": len(accounts_with_wd),
|
||||||
|
},
|
||||||
|
"wallets": {
|
||||||
|
"unique_reserves": len(wallets),
|
||||||
|
"note": "unique reserve pubs from Taler withdrawals (one per wallet withdraw)",
|
||||||
|
},
|
||||||
|
"balance_explorer": balance,
|
||||||
|
"balance_explorer_alt": bal_alt,
|
||||||
|
"balance_explorer_full": bal_full,
|
||||||
|
"flow": {
|
||||||
|
"incoming": {
|
||||||
|
"label": "Incoming bank credits",
|
||||||
|
"count": n_incoming,
|
||||||
|
"amount": fin["amount"],
|
||||||
|
"amount_alt": fin["amount_alt"],
|
||||||
|
"amount_full": fin["amount_full"],
|
||||||
|
"value": fin["value"],
|
||||||
|
"value_str": fin["value_str"],
|
||||||
|
},
|
||||||
|
"withdraw": {
|
||||||
|
"label": "Taler withdrawals to wallets",
|
||||||
|
"count": len(withdraws),
|
||||||
|
"amount": fwd["amount"],
|
||||||
|
"amount_alt": fwd["amount_alt"],
|
||||||
|
"amount_full": fwd["amount_full"],
|
||||||
|
"value": fwd["value"],
|
||||||
|
"value_str": fwd["value_str"],
|
||||||
|
},
|
||||||
|
"other_out": {
|
||||||
|
"label": "Other debits (non-withdraw)",
|
||||||
|
"amount": foth["amount"],
|
||||||
|
"amount_alt": foth["amount_alt"],
|
||||||
|
"amount_full": foth["amount_full"],
|
||||||
|
"value": foth["value"],
|
||||||
|
"value_str": foth["value_str"],
|
||||||
|
},
|
||||||
|
"total_in": fin["amount"],
|
||||||
|
"total_in_alt": fin["amount_alt"],
|
||||||
|
"total_in_value": fin["value"],
|
||||||
|
"total_out": fout["amount"],
|
||||||
|
"total_out_alt": fout["amount_alt"],
|
||||||
|
"total_out_value": fout["value"],
|
||||||
|
"note": "incoming=credits; withdraw=Taler withdrawal debits; skip_users excluded (default exchange)",
|
||||||
|
},
|
||||||
|
"withdraws": {
|
||||||
|
"count": len(withdraws),
|
||||||
|
"total_amount": total_pack["amount"],
|
||||||
|
"total_amount_alt": total_pack["amount_alt"],
|
||||||
|
"total_amount_full": total_pack["amount_full"],
|
||||||
|
"total_value": total_pack["value"],
|
||||||
|
"total_value_str": total_pack["value_str"],
|
||||||
|
"last_amount": last_amt,
|
||||||
|
"last_amount_alt": last_alt,
|
||||||
|
"last_at": last_at,
|
||||||
|
"last_at_iso": last_iso,
|
||||||
|
"last_at_unix": last_unix,
|
||||||
|
"last_subject": last_subj,
|
||||||
|
"last_24h": {
|
||||||
|
"count": n24,
|
||||||
|
"amount": p24["amount"],
|
||||||
|
"amount_alt": p24["amount_alt"],
|
||||||
|
"amount_full": p24["amount_full"],
|
||||||
|
"value": p24["value"],
|
||||||
|
},
|
||||||
|
"last_7d": {
|
||||||
|
"count": n7,
|
||||||
|
"amount": p7["amount"],
|
||||||
|
"amount_alt": p7["amount_alt"],
|
||||||
|
"amount_full": p7["amount_full"],
|
||||||
|
"value": p7["value"],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"recent_withdraws": recent,
|
||||||
|
"recent_incoming": recent_in,
|
||||||
|
"demo": {
|
||||||
|
"uri": None,
|
||||||
|
"amount": None,
|
||||||
|
"created": None,
|
||||||
|
"withdrawal_id": None,
|
||||||
|
"status": None,
|
||||||
|
"ready": False,
|
||||||
|
},
|
||||||
|
"performance": {
|
||||||
|
"config_http": config_http,
|
||||||
|
"config_ms": config_ms,
|
||||||
|
"integration_http": int_http,
|
||||||
|
"integration_ms": int_ms,
|
||||||
|
"webui_http": webui_http,
|
||||||
|
"webui_ms": webui_ms,
|
||||||
|
"loadavg": loadavg,
|
||||||
|
"memory": {
|
||||||
|
"container_rss_human": "—",
|
||||||
|
"note": "memory filled by host collector merge when available",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"alt_unit_names": alt,
|
||||||
|
}
|
||||||
|
|
||||||
|
# pull demo withdraw files + memory from bank container if OUT is path and caller merged
|
||||||
|
# demo files optional via DEMO_DIR
|
||||||
|
demo_dir = env("DEMO_DIR", "")
|
||||||
|
if demo_dir:
|
||||||
|
dpath = Path(demo_dir)
|
||||||
|
uri = (dpath / "withdraw.uri").read_text().strip() if (dpath / "withdraw.uri").is_file() else ""
|
||||||
|
amt = (
|
||||||
|
(dpath / "withdraw.amount").read_text().strip()
|
||||||
|
if (dpath / "withdraw.amount").is_file()
|
||||||
|
else ""
|
||||||
|
)
|
||||||
|
created = (
|
||||||
|
(dpath / "withdraw.created").read_text().strip()
|
||||||
|
if (dpath / "withdraw.created").is_file()
|
||||||
|
else ""
|
||||||
|
)
|
||||||
|
if uri:
|
||||||
|
wid = uri.rstrip("/").split("/")[-1]
|
||||||
|
stats["demo"] = {
|
||||||
|
"uri": uri,
|
||||||
|
"amount": amt or None,
|
||||||
|
"created": created or None,
|
||||||
|
"withdrawal_id": wid,
|
||||||
|
"status": None,
|
||||||
|
"ready": True,
|
||||||
|
}
|
||||||
|
|
||||||
|
stats = enrich_stats_tree(stats, alt)
|
||||||
|
|
||||||
|
text = json.dumps(stats, indent=2, ensure_ascii=False) + "\n"
|
||||||
|
if args.out == "-" or not args.out:
|
||||||
|
sys.stdout.write(text)
|
||||||
|
else:
|
||||||
|
outp = Path(args.out)
|
||||||
|
outp.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
tmp = outp.with_suffix(outp.suffix + f".tmp.{os.getpid()}")
|
||||||
|
tmp.write_text(text, encoding="utf-8")
|
||||||
|
tmp.replace(outp)
|
||||||
|
write_run(True, None)
|
||||||
|
print(
|
||||||
|
f"ok accounts={len(accounts)} scanned={scan_ok} withdraws={len(withdraws)} "
|
||||||
|
f"total={total_pack['amount']} alt={total_pack['amount_alt']}",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
return 0
|
||||||
|
except Exception as e:
|
||||||
|
write_run(False, str(e))
|
||||||
|
print(f"error: {e}", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
40
scripts/taler-landing/enrich_stats_alt.py
Executable file
40
scripts/taler-landing/enrich_stats_alt.py
Executable file
|
|
@ -0,0 +1,40 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Add amount_alt / amount_full fields to a landing stats.json in place."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||||
|
from goa_amounts import DEFAULT_ALT, enrich_stats_tree, load_alt_from_config # noqa: E402
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("path", help="stats.json path")
|
||||||
|
ap.add_argument(
|
||||||
|
"--exchange-config",
|
||||||
|
default="https://exchange.hacktivism.ch/config",
|
||||||
|
)
|
||||||
|
ap.add_argument("-o", "--out", default="", help="default: overwrite path")
|
||||||
|
args = ap.parse_args()
|
||||||
|
p = Path(args.path)
|
||||||
|
data = json.loads(p.read_text(encoding="utf-8"))
|
||||||
|
if not isinstance(data, dict) or not data.get("ok"):
|
||||||
|
print("skip: not ok stats", file=sys.stderr)
|
||||||
|
return 0
|
||||||
|
alt = load_alt_from_config(args.exchange_config) or dict(DEFAULT_ALT)
|
||||||
|
enrich_stats_tree(data, alt)
|
||||||
|
out = Path(args.out) if args.out else p
|
||||||
|
tmp = out.with_suffix(out.suffix + f".tmp.{os.getpid()}")
|
||||||
|
tmp.write_text(json.dumps(data, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")
|
||||||
|
tmp.replace(out)
|
||||||
|
print(f"enriched {out}", file=sys.stderr)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
306
scripts/taler-landing/goa_amounts.py
Executable file
306
scripts/taler-landing/goa_amounts.py
Executable file
|
|
@ -0,0 +1,306 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""GOA (and generic CUR:amount) alt-unit formatting for landing stats.
|
||||||
|
|
||||||
|
Matches exchange currency_specification.alt_unit_names:
|
||||||
|
0→GOA, 3→Kilo-GOA, 6→Mega-GOA, … and fractional scales.
|
||||||
|
Display prefers compact alt names for large values so landing tiles fit.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from decimal import Decimal, InvalidOperation, ROUND_HALF_UP
|
||||||
|
from typing import Any, Dict, Optional, Tuple
|
||||||
|
|
||||||
|
# Default GOA ladder (same shape as exchange /config)
|
||||||
|
DEFAULT_ALT: Dict[str, str] = {
|
||||||
|
"24": "Yotta-GOA",
|
||||||
|
"21": "Zetta-GOA",
|
||||||
|
"18": "Exa-GOA",
|
||||||
|
"15": "Peta-GOA",
|
||||||
|
"12": "Tera-GOA",
|
||||||
|
"9": "Giga-GOA",
|
||||||
|
"6": "Mega-GOA",
|
||||||
|
"3": "Kilo-GOA",
|
||||||
|
"0": "GOA",
|
||||||
|
"-1": "Deci-GOA",
|
||||||
|
"-2": "Centi-GOA",
|
||||||
|
"-3": "Milli-GOA",
|
||||||
|
"-6": "Micro-GOA",
|
||||||
|
"-7": "Deci-Micro-GOA",
|
||||||
|
"-8": "Atomic-GOA",
|
||||||
|
}
|
||||||
|
|
||||||
|
# Use alt unit when |value| >= this (base units). Below: keep compact CUR:n.
|
||||||
|
ALT_THRESHOLD = Decimal("1000")
|
||||||
|
|
||||||
|
|
||||||
|
def parse_amount(s: Any) -> Tuple[str, Decimal]:
|
||||||
|
"""Parse 'GOA:12.5' or bare number → (currency, value)."""
|
||||||
|
if s is None:
|
||||||
|
return "GOA", Decimal(0)
|
||||||
|
if isinstance(s, (int, float, Decimal)):
|
||||||
|
return "GOA", Decimal(str(s))
|
||||||
|
text = str(s).strip()
|
||||||
|
if not text:
|
||||||
|
return "GOA", Decimal(0)
|
||||||
|
if ":" in text:
|
||||||
|
cur, rest = text.split(":", 1)
|
||||||
|
return (cur or "GOA").strip(), Decimal(rest.strip() or "0")
|
||||||
|
return "GOA", Decimal(text)
|
||||||
|
|
||||||
|
|
||||||
|
def fmt_coeff(v: Decimal) -> str:
|
||||||
|
if v == v.to_integral_value():
|
||||||
|
return format(int(v), "d")
|
||||||
|
# up to 4 significant fractional digits, strip trailing zeros
|
||||||
|
q = v.quantize(Decimal("0.0001"), rounding=ROUND_HALF_UP)
|
||||||
|
s = format(q, "f").rstrip("0").rstrip(".")
|
||||||
|
return s or "0"
|
||||||
|
|
||||||
|
|
||||||
|
def fmt_base(cur: str, val: Decimal) -> str:
|
||||||
|
if val == val.to_integral_value():
|
||||||
|
return f"{cur}:{int(val)}"
|
||||||
|
# preserve up to 8 fractional digits (Taler style)
|
||||||
|
s = format(val, "f").rstrip("0").rstrip(".")
|
||||||
|
return f"{cur}:{s}"
|
||||||
|
|
||||||
|
|
||||||
|
def format_amount_alt(
|
||||||
|
amount: Any,
|
||||||
|
alt: Optional[Dict[str, str]] = None,
|
||||||
|
*,
|
||||||
|
threshold: Decimal = ALT_THRESHOLD,
|
||||||
|
with_base: bool = False,
|
||||||
|
) -> Dict[str, Any]:
|
||||||
|
"""Return display fields for one amount.
|
||||||
|
|
||||||
|
Keys:
|
||||||
|
amount canonical CUR:n
|
||||||
|
amount_alt short display (e.g. "1.23 Mega-GOA")
|
||||||
|
amount_full "1.23 Mega-GOA (GOA:1230000)" when alt used
|
||||||
|
value float for JSON (may lose precision above 2^53 — also value_str)
|
||||||
|
value_str exact decimal string
|
||||||
|
"""
|
||||||
|
alt = alt or DEFAULT_ALT
|
||||||
|
try:
|
||||||
|
cur, val = parse_amount(amount)
|
||||||
|
except (InvalidOperation, ValueError, ArithmeticError):
|
||||||
|
raw = str(amount or "")
|
||||||
|
return {
|
||||||
|
"amount": raw,
|
||||||
|
"amount_alt": raw or "—",
|
||||||
|
"amount_full": raw or "—",
|
||||||
|
"value": 0.0,
|
||||||
|
"value_str": "0",
|
||||||
|
}
|
||||||
|
|
||||||
|
base = fmt_base(cur, val)
|
||||||
|
value_str = format(val, "f").rstrip("0").rstrip(".") if val != val.to_integral_value() else str(int(val))
|
||||||
|
try:
|
||||||
|
value_f = float(val)
|
||||||
|
except Exception:
|
||||||
|
value_f = 0.0
|
||||||
|
|
||||||
|
base_name = alt.get("0") or cur
|
||||||
|
if val == 0:
|
||||||
|
disp = f"0 {base_name}"
|
||||||
|
return {
|
||||||
|
"amount": base,
|
||||||
|
"amount_alt": disp,
|
||||||
|
"amount_full": disp,
|
||||||
|
"value": 0.0,
|
||||||
|
"value_str": "0",
|
||||||
|
}
|
||||||
|
|
||||||
|
scales = []
|
||||||
|
for k, name in alt.items():
|
||||||
|
try:
|
||||||
|
scales.append((int(k), str(name)))
|
||||||
|
except Exception:
|
||||||
|
continue
|
||||||
|
scales.sort(key=lambda x: -x[0])
|
||||||
|
|
||||||
|
absval = abs(val)
|
||||||
|
# Below threshold: short base form (saves noise on small demo amounts)
|
||||||
|
if absval < threshold:
|
||||||
|
# Prefer "GOA:12.5" style (matches existing landings) when small
|
||||||
|
return {
|
||||||
|
"amount": base,
|
||||||
|
"amount_alt": base,
|
||||||
|
"amount_full": base,
|
||||||
|
"value": value_f,
|
||||||
|
"value_str": value_str,
|
||||||
|
}
|
||||||
|
|
||||||
|
chosen = None
|
||||||
|
for sc, name in scales:
|
||||||
|
unit = Decimal(10) ** sc
|
||||||
|
if unit <= 0:
|
||||||
|
continue
|
||||||
|
coeff = absval / unit
|
||||||
|
if coeff >= 1:
|
||||||
|
chosen = (sc, name, coeff if val >= 0 else -coeff)
|
||||||
|
break
|
||||||
|
|
||||||
|
if chosen is None or chosen[0] == 0:
|
||||||
|
return {
|
||||||
|
"amount": base,
|
||||||
|
"amount_alt": base,
|
||||||
|
"amount_full": base,
|
||||||
|
"value": value_f,
|
||||||
|
"value_str": value_str,
|
||||||
|
}
|
||||||
|
|
||||||
|
sc, name, coeff = chosen
|
||||||
|
short = f"{fmt_coeff(coeff)} {name}"
|
||||||
|
full = f"{short} ({base})" if with_base or True else short
|
||||||
|
return {
|
||||||
|
"amount": base,
|
||||||
|
"amount_alt": short,
|
||||||
|
"amount_full": full,
|
||||||
|
"value": value_f,
|
||||||
|
"value_str": value_str,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def attach_alt(obj: Dict[str, Any], amount_key: str = "amount", alt: Optional[Dict[str, str]] = None) -> Dict[str, Any]:
|
||||||
|
"""Mutate obj: ensure amount_alt / amount_full from amount_key."""
|
||||||
|
if not isinstance(obj, dict):
|
||||||
|
return obj
|
||||||
|
src = obj.get(amount_key)
|
||||||
|
if src is None:
|
||||||
|
return obj
|
||||||
|
f = format_amount_alt(src, alt)
|
||||||
|
obj[amount_key] = f["amount"]
|
||||||
|
obj["amount_alt"] = f["amount_alt"]
|
||||||
|
obj["amount_full"] = f["amount_full"]
|
||||||
|
if "value" not in obj:
|
||||||
|
obj["value"] = f["value"]
|
||||||
|
obj["value_str"] = f["value_str"]
|
||||||
|
return obj
|
||||||
|
|
||||||
|
|
||||||
|
def load_alt_from_config(url: str, timeout: float = 8.0) -> Dict[str, str]:
|
||||||
|
"""GET exchange/bank /config → alt_unit_names (fallback DEFAULT_ALT)."""
|
||||||
|
import json
|
||||||
|
import urllib.request
|
||||||
|
|
||||||
|
try:
|
||||||
|
req = urllib.request.Request(url, headers={"Accept": "application/json"})
|
||||||
|
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||||
|
data = json.loads(resp.read().decode("utf-8", errors="replace"))
|
||||||
|
except Exception:
|
||||||
|
return dict(DEFAULT_ALT)
|
||||||
|
|
||||||
|
au = None
|
||||||
|
cs = data.get("currency_specification")
|
||||||
|
if isinstance(cs, dict):
|
||||||
|
au = cs.get("alt_unit_names")
|
||||||
|
if not au and isinstance(data.get("currencies"), dict):
|
||||||
|
for _code, spec in data["currencies"].items():
|
||||||
|
if isinstance(spec, dict) and spec.get("alt_unit_names"):
|
||||||
|
au = spec["alt_unit_names"]
|
||||||
|
break
|
||||||
|
if not isinstance(au, dict) or "0" not in au:
|
||||||
|
return dict(DEFAULT_ALT)
|
||||||
|
return {str(k): str(v) for k, v in au.items()}
|
||||||
|
|
||||||
|
|
||||||
|
def enrich_stats_tree(data: Dict[str, Any], alt: Optional[Dict[str, str]] = None) -> Dict[str, Any]:
|
||||||
|
"""Walk common landing stats.json shapes and add amount_alt fields."""
|
||||||
|
alt = alt or DEFAULT_ALT
|
||||||
|
if not isinstance(data, dict):
|
||||||
|
return data
|
||||||
|
|
||||||
|
# bank-style
|
||||||
|
for path in (
|
||||||
|
("flow", "incoming"),
|
||||||
|
("flow", "withdraw"),
|
||||||
|
("flow", "other_out"),
|
||||||
|
("withdraws",),
|
||||||
|
("withdraws", "last_24h"),
|
||||||
|
("withdraws", "last_7d"),
|
||||||
|
):
|
||||||
|
cur: Any = data
|
||||||
|
ok = True
|
||||||
|
for p in path:
|
||||||
|
if not isinstance(cur, dict) or p not in cur:
|
||||||
|
ok = False
|
||||||
|
break
|
||||||
|
cur = cur[p]
|
||||||
|
if ok and isinstance(cur, dict):
|
||||||
|
if "amount" in cur:
|
||||||
|
attach_alt(cur, "amount", alt)
|
||||||
|
if "total_amount" in cur:
|
||||||
|
f = format_amount_alt(cur["total_amount"], alt)
|
||||||
|
cur["total_amount"] = f["amount"]
|
||||||
|
cur["total_amount_alt"] = f["amount_alt"]
|
||||||
|
cur["total_amount_full"] = f["amount_full"]
|
||||||
|
if "last_amount" in cur and cur["last_amount"]:
|
||||||
|
f = format_amount_alt(cur["last_amount"], alt)
|
||||||
|
cur["last_amount"] = f["amount"]
|
||||||
|
cur["last_amount_alt"] = f["amount_alt"]
|
||||||
|
|
||||||
|
if isinstance(data.get("flow"), dict):
|
||||||
|
fl = data["flow"]
|
||||||
|
for k in ("total_in", "total_out"):
|
||||||
|
if fl.get(k):
|
||||||
|
f = format_amount_alt(fl[k], alt)
|
||||||
|
fl[k] = f["amount"]
|
||||||
|
fl[f"{k}_alt"] = f["amount_alt"]
|
||||||
|
fl[f"{k}_full"] = f["amount_full"]
|
||||||
|
|
||||||
|
if data.get("balance_explorer"):
|
||||||
|
f = format_amount_alt(data["balance_explorer"], alt)
|
||||||
|
data["balance_explorer"] = f["amount"]
|
||||||
|
data["balance_explorer_alt"] = f["amount_alt"]
|
||||||
|
data["balance_explorer_full"] = f["amount_full"]
|
||||||
|
|
||||||
|
for key in ("recent_withdraws", "recent_incoming", "recent_activity"):
|
||||||
|
rows = data.get(key)
|
||||||
|
if isinstance(rows, list):
|
||||||
|
for row in rows:
|
||||||
|
if isinstance(row, dict) and row.get("amount"):
|
||||||
|
attach_alt(row, "amount", alt)
|
||||||
|
|
||||||
|
# exchange-style top-level amounts
|
||||||
|
for k in (
|
||||||
|
"wire_in_amount",
|
||||||
|
"withdraw_amount",
|
||||||
|
"coins_remaining_amount",
|
||||||
|
):
|
||||||
|
if data.get(k):
|
||||||
|
f = format_amount_alt(data[k], alt)
|
||||||
|
data[k] = f["amount"]
|
||||||
|
data[f"{k}_alt"] = f["amount_alt"]
|
||||||
|
data[f"{k}_full"] = f["amount_full"]
|
||||||
|
|
||||||
|
for key in ("by_denom", "denom_ladder"):
|
||||||
|
rows = data.get(key)
|
||||||
|
if isinstance(rows, list):
|
||||||
|
for row in rows:
|
||||||
|
if isinstance(row, dict) and row.get("value"):
|
||||||
|
f = format_amount_alt(row["value"], alt)
|
||||||
|
row["value"] = f["amount"]
|
||||||
|
row["value_alt"] = f["amount_alt"]
|
||||||
|
|
||||||
|
# merchant dual currency
|
||||||
|
for block in data.get("by_currency") or []:
|
||||||
|
if not isinstance(block, dict):
|
||||||
|
continue
|
||||||
|
for k in ("amount_sum", "amount_paid_sum"):
|
||||||
|
if block.get(k):
|
||||||
|
f = format_amount_alt(block[k], alt)
|
||||||
|
block[k] = f["amount"]
|
||||||
|
block[f"{k}_alt"] = f["amount_alt"]
|
||||||
|
block[f"{k}_full"] = f["amount_full"]
|
||||||
|
|
||||||
|
for block in data.get("recent_activity_by_currency") or []:
|
||||||
|
if not isinstance(block, dict):
|
||||||
|
continue
|
||||||
|
for row in block.get("items") or []:
|
||||||
|
if isinstance(row, dict) and row.get("amount"):
|
||||||
|
attach_alt(row, "amount", alt)
|
||||||
|
|
||||||
|
data["alt_unit_names"] = alt
|
||||||
|
return data
|
||||||
Loading…
Add table
Add a link
Reference in a new issue