hernani/taler-monitoring
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

taler-monitoring v1.13.12

Browse source 
Multi-phase global SUMMARY, warn-filter dimmed context, SUMMARY chrome
as meta; FP stage host-agent via francpaysan-stage-user (stagepaysan);
monpages GOA + FP stage.
This commit is contained in:
Hernâni Marques 2026-07-19 04:21:16 +02:00
commit 5a5431bfa9
No known key found for this signature in database
GPG key ID: CB5738652768F7E9
88 changed files with 21996 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

9
.gitignore vendored Normal file
View file

@ -0,0 +1,9 @@
secrets.env
__pycache__/
*.pyc
android-test/apks/
android-test/out*/
android-test/artifacts/
site-gen/__pycache__/
*.log
.DS_Store

230
CLI-AUTOMATION-NOTES.md Normal file
View file

@ -0,0 +1,230 @@
# CLI automation notes (`taler-wallet-cli` + monitoring)
Living list of **recurring issues** seen while driving GOA (hacktivism) and
stage TESTPAYSAN via **CLI** (`taler-wallet-cli`, bank/merchant HTTP, monitoring
e2e/ladder). Ordered roughly by **how often they hurt automation** and how
useful a fix in **wallet-cli / wallet-core** (or clearer APIs) would be.
Companions:
- Android GUI → [`android-test/GUI-AUTOMATION-NOTES.md`](./android-test/GUI-AUTOMATION-NOTES.md)
- Git / suite tree → [`VERSIONS.md`](./VERSIONS.md)
Scripts: `check_e2e.sh`, `check_goa_ladder.sh`, `taler-monitoring.sh`.
Legend:
| Tag | Meaning |
|-----|---------|
| **cli** | wallet-cli UX / flags / hang behaviour |
| **core** | wallet-core protocol / state machine |
| **bank** | libeufin-bank / integration API |
| **ops** | secrets, paths, multi-host |
| **doc** | missing or wrong documentation |
---
## 1. `run-until-done` is unusable for automation
| | |
|--|--|
| **Seen** | e2e, ladder, Android notes, macOS comments: *hangs / banned in monitoring* |
| **Area** | **cli** / **core** |
| **Problem** | `taler-wallet-cli run-until-done` (or long shepherd runs) often **never returns** or blocks CI-length timeouts. Monitoring **forbids** it and polls balance + bank `transfer_done` instead. |
| **Workaround** | `advanced serve` + socket (`WSOCK`) or pure poll loops (`LADDER_SETTLE_*`, e2e settle wait). |
| **Wanted for wallet-cli** | Bounded wait: e.g. `run-until-done --timeout=Ns --exit-on=withdrawn|paid|idle`; stable non-zero exit on timeout; progress on stdout/JSON. |
---
## 2. Withdraw completion is not observable from CLI alone
| | |
|--|--|
| **Seen** | e2e “coins missing after withdraw”; ladder `OK_BANK` vs wallet avail |
| **Area** | **cli** / **core** / **bank** |
| **Problem** | After `accept-uri` + bank confirm, coins may lag (wirewatch). CLI has no single “withdraw settled” command with clear success/failure; automation reimplements status from bank integration JSON + balance polls. |
| **Workaround** | Poll `balance` + `…/withdrawal-operation/{id}` (`status`, `transfer_done`, `selected_reserve_pub`). |
| **Wanted** | `wallet-cli withdrawals wait --id=… --timeout=` or JSON event stream: `selected → confirmed → coins-available`. |
---
## 3. `reserve_pub` / force-select is fragile (5114)
| | |
|--|--|
| **Seen** | ladder force-select, e2e confirm path, Android spinner issues |
| **Area** | **cli** / **core** / **bank** |
| **Problem** | Cumulative wallets re-emit old reserves; binding the wrong `reserve_pub` yields **HTTP 409 / code 5114** (“already bound”). Automation scrapes accept/tx dumps for candidates. |
| **Workaround** | Score pubs by WID/amount; try several; treat 5114 as “stale reserve” not “out of money”. |
| **Wanted** | CLI returns **the** `reserve_pub` for the just-accepted withdraw in structured JSON; `force-select` / bank-side select API documented with idempotency. |
---
## 4. Exchange not auto-known for bank deep links / withdraw URIs
| | |
|--|--|
| **Seen** | Android endless spinner; GOA `bank.hacktivism.ch` withdraw; fixed in app branch `fix/bank-withdraw-auto-exchange` |
| **Area** | **core** / **cli** |
| **Problem** | Withdraw URI names an exchange the wallet has never added → stuck loading / empty exchange list. CLI requires explicit `exchanges add` + `update` + `accept-tos` every fresh DB. |
| **Workaround** | e2e always adds exchange before ATM ladder; Android needs ensureExchange-style fix. |
| **Wanted** | CLI/core: on `accept-uri` withdraw, **auto-add** exchange from URI/bank details (`allowCompletion`), then surface ToS if needed in one command. |
---
## 5. Default ports in URIs (`:443` / `:80`) break round-trips
| | |
|--|--|
| **Seen** | ladder/e2e strip ports; QR checks; mint URIs with `:443` |
| **Area** | **cli** / **bank** / **doc** |
| **Problem** | Some bank/wallet outputs include `taler://…host:443/…`. QR validators and some clients reject or double-normalize inconsistently. |
| **Workaround** | `sed 's/:443\//\//g'` everywhere in automation. |
| **Wanted** | Canonical form without default ports in **all** wallet-cli and bank integration outputs; document as invariant. |
---
## 6. No first-class “demo withdraw” / explorer mint in wallet-cli
| | |
|--|--|
| **Seen** | e2e uses bank admin + ATM create; ladder/gui-chain reimplement explorer mint in Python |
| **Area** | **cli** / **ops** |
| **Problem** | Automation always hand-rolls Basic auth → token → `POST …/withdrawals` → `taler_withdraw_uri`. Easy to get wrong (auth header, amount currency). |
| **Workaround** | Shared Python in `goa-chain` / ladder / e2e; secrets files. |
| **Wanted** | Optional helper: `wallet-cli testing mint-withdraw --bank=URL --user=explorer --password-file=… --amount=GOA:10` (or bank-side only tool in libeufin) emitting a clean URI. |
---
## 7. Long-lived wallet process required for pay/withdraw progress
| | |
|--|--|
| **Seen** | e2e `advanced serve` / shepherd socket |
| **Area** | **cli** |
| **Problem** | One-shot CLI invocations do not keep background work alive; without serve/shepherd, withdraw/pay may stall mid-flight. |
| **Workaround** | Start serve once per e2e run; route `wcli` through the socket. |
| **Wanted** | Documented, supported “session mode”: start/stop serve; or each mutating command optionally `--background-until=…` with timeout. |
---
## 8. ToS accept is a separate step that fails silently or blocks
| | |
|--|--|
| **Seen** | e2e `exchanges accept-tos`; ladder after add; Android GUI ToS screens |
| **Area** | **cli** / **core** |
| **Problem** | Fresh exchange → operations need ToS; CLI must call `accept-tos` explicitly. Failures are easy to miss in multi-step scripts. |
| **Workaround** | Always `accept-tos` after `update` in e2e/ladder. |
| **Wanted** | `accept-uri --accept-tos` or auto-prompt with non-interactive `--yes` that covers exchange ToS for that URI’s exchange. |
---
## 9. Pay template / public order path is merchant-shaped, not CLI-shaped
| | |
|--|--|
| **Seen** | e2e shop templates, stage farmer shops, ladder private orders |
| **Area** | **cli** / **doc** |
| **Problem** | Wallet-cli pays via `handle-uri` on `taler://pay/…`; creating the order is always custom curl (template POST or private order + token). No unified “pay this amount to instance” for public templates. |
| **Workaround** | Monitoring builds URI externally then `handle-uri --yes`. |
| **Wanted** | Documented recipe only, or `wallet-cli testing pay-template --base=… --instance=… --id=…` for demos. |
---
## 10. Amount / currency parsing edge cases
| | |
|--|--|
| **Seen** | ladder min denom 0.01 vs GOA 1e-6; zero withdraw rejected (HTTP 409 / amount too low); max wire ceilings |
| **Area** | **cli** / **bank** |
| **Problem** | `CURRENCY:0` and sub-min amounts fail differently per stack; CLI error strings are not machine-stable. |
| **Workaround** | Soft-skip zero rung; clamp ladder min to exchange denoms / bank max_wire. |
| **Wanted** | Structured errors (`AMOUNT_too_small`, `currency_unknown`) in JSON mode; `wallet-cli amount validate --exchange=`. |
---
## 11. Finding the right `taler-wallet-cli` binary
| | |
|--|--|
| **Seen** | `find_wallet_cli` in `lib.sh`; hardcoded laptop paths; wrapper vs `.mjs` |
| **Area** | **ops** / **cli** |
| **Problem** | Debian package is a shell wrapper; some tools need `node …/taler-wallet-cli.mjs`. Hardcoded `/Users/…` paths break other hosts. |
| **Workaround** | `find_wallet_cli` search list; `WALLET_CLI=` override. |
| **Wanted** | Single install story: `wallet-cli --version` JSON with path + libversion; no need to pass `.mjs` to node by hand. |
---
## 12. Secrets and multi-stack confusion
| | |
|--|--|
| **Seen** | explorer vs admin password; GOA secrets used on stage; ladder EXP_PW |
| **Area** | **ops** |
| **Problem** | CLI does not know “which stack”; wrong password → 401 mid-ladder. Not a wallet-cli bug, but every CLI automation hits it. |
| **Workaround** | `SECRETS_ROOT`, stage SSH, `STACK=` profiles. |
| **Wanted** | Optional `~/.config/taler/stacks.d/goa.env` convention documented next to wallet-cli; still no secrets in repo. |
---
## 13. No stable machine-readable “step result” for scripts
| | |
|--|--|
| **Seen** | e2e greps accept output; ladder scrapes JSON from mixed stdout |
| **Area** | **cli** |
| **Problem** | Human logs + occasional JSON blobs; hard to parse reliably. |
| **Workaround** | Python scrapers, temp files, `tee`. |
| **Wanted** | Global `--json` / `--ndjson` for all commands; one object per completed operation with `ok`, `op`, `ids`, `amounts`. |
---
## 14. Pay settlement wait is symmetric to withdraw pain
| | |
|--|--|
| **Seen** | e2e pay settle loops; ladder pay settle rounds |
| **Area** | **cli** / **core** |
| **Problem** | After `handle-uri` pay, success is “order paid” / balance drop / merchant order status — not one CLI wait. |
| **Workaround** | Short poll loops; never run-until-done. |
| **Wanted** | Same as §1–2: bounded wait on transaction id / order id. |
---
## Priority shortlist for `taler-wallet-cli` / core
If only a few changes land, these unlock the most automation:
1. **Bounded `run-until-done` / wait-for-state** (§1, §2, §14)
2. **Structured JSON on every command** (§13)
3. **Auto-add exchange + ToS on accept-uri** (§4, §8)
4. **Canonical URIs without :443** (§5)
5. **Emit reserve_pub for last withdraw** (§3)
---
## How we work around today (monitoring)
| Issue | Monitoring behaviour |
|-------|----------------------|
| run-until-done | Disabled; balance + bank poll |
| serve | Optional long-lived socket in e2e |
| exchange | Explicit add/update/accept-tos |
| ladder explorer | `read_secret` / EXP_PW_FILE |
| stage maxima | bank `max_wire` + keys min denom |
| force-select | multi-rpub try; soft-skip confirm |
---
## How to add an issue
```markdown
### N. short title
| | |
|--|--|
| **Seen** | where / which stack |
| **Area** | cli / core / bank / ops / doc |
| **Problem** | … |
| **Workaround** | … |
| **Wanted** | concrete CLI/core behaviour |
```

71
DEPENDENCIES.md Normal file
View file

@ -0,0 +1,71 @@
# taler-monitoring — host package dependencies
## koopa (openSUSE Tumbleweed, user `hernani`)
### Required (host-agent / GOA primary: `urls inside versions`)
| Binary | Package (zypper) | Role |
|--------|------------------|------|
| `bash` | (base) | runner |
| `curl` | `curl` | HTTPS/API probes + **monpages** FQDN checks |
| `nmap` | `nmap` | **surface** OS/service fingerprint (`SURFACE_NMAP=1`, v1.8.0+; OS detect may need root) |
| `python3` | `python313-base` (provides `/usr/bin/python3`) | JSON, HTML site-gen, metrics |
| `podman` | `podman` | **inside**: `podman exec` into taler-hacktivism* |
| `git` | `git` | pin commit SHA in monitoring HTML footer |
| `rsync` | `rsync` | optional: deploy/sync suite |
| `timeout` / `mktemp` / `getent` | coreutils / glibc | helpers |
**Status on koopa (2026-07-18):** all of the above present; rootless podman owns GOA containers.
### Recommended (full `urls` QR group)
| Binary | Package | Role |
|--------|---------|------|
| `qrencode` | **`qrencode`** | encode taler:// / payto:// → PNG |
| `zbarimg` | **`zbar`** | decode PNG; exact payload match |
Without these, QR checks **WARN** and skip encode/decode (`apt install` messages in code are Debian-oriented; on Tumbleweed use zypper).
```bash
# as root on koopa
zypper refresh
zypper in qrencode zbar
# verify as hernani:
command -v qrencode zbarimg
```
### Optional (not needed for host-agent default)
| Binary / tool | When |
|---------------|------|
| `jq` | nice-to-have; suite uses python3 for JSON |
| `dig` / `bind-utils` | debug DNS; not required for default phases |
| `ssh` | only for access mode **ssh** (laptop → `KOOPA_SSH` / stage `INSIDE_SSH`); not needed for **host-podman** |
| `node` / `pnpm` + wallet-cli | **e2e** / ladder only (+ secrets) |
| `secrets.env` / admin passwords | e2e only — never system packages |
### systemd user (no extra packages)
- `systemd` user session + **`loginctl enable-linger hernani`** (already **Linger=yes** on koopa)
- Units: `host-agent/*.service|path|timer`
### Caddy HTML publish (separate from monitoring run)
- Host Caddy already installed (root service).
- Writing `/var/www/monitoring-sites`: either root `rsync` or directory owned/writable by `hernani` (you set this up).
---
## outside-runner (macOS, outside-only timer)
| Need | Notes |
|------|--------|
| `bash`, `curl`, `python3`, `rsync`, `git` (optional) | usually Homebrew / Xcode CLT |
| **No** `podman` | phases are `urls versions` only (`SKIP_SSH=1`) |
| `qrencode` / `zbarimg` | optional for QR; `brew install qrencode zbar` if desired |
---
## Franc Paysan stage host (stagepaysan)
FP stage (`stagepaysan` via SSH alias `francpaysan-stage-user`): curl, python3, podman if inside, qrencode/zbar for full urls; suite at `~/src/taler-monitoring`.

446
README.md Normal file
View file

@ -0,0 +1,446 @@
# taler-monitoring
Checks for **GNU Taler** stacks (bank, exchange, merchant; optional surface inventory).
| Mode | What |
|------|------|
| **Outside-only** | Public HTTPS from anywhere — no SSH, no host-agent. Phases such as `urls`, `monpages`, `surface`, … |
| **Inside / local** | Where you operate the stack: podman/SSH (`inside`, `versions`, `e2e`, `ladder`, …) |
| **Host-agent + HTML** | Optional: timers run the suite and publish sticky-bar HTML (see simplified page layout below) |
Outside-only example: `./taler-monitoring.sh -d demo.taler.net urls`
Surface (ecosystem + mail + mattermost + versions): `./taler-monitoring.sh surface mattermost mail versions`
Full local GOA example: `./taler-monitoring.sh -d hacktivism.ch urls inside versions`
Repo: https://git.hacktivism.ch/hernani/taler-monitoring · tree: `~/src/taler-monitoring`
Version: `./taler-monitoring.sh --ver` (or `--version` / `-V`)
## Monitoring pages — public URL families (only these three)
**`taler-monitoring` is the suite name (git / CLI), not a public URL path.**
There is **no** mon page at `/taler-monitoring` or `/taler-monitoring/`.
Exactly **three** first-class public path families (each must work **bare and with trailing slash**; reverse-proxy should `redir` bare → slash). Error companions use `…_err` when the last run failed:
| Family | Paths (bare + slash) | Where | Purpose |
|--------|----------------------|--------|---------|
| **monitoring** | `/monitoring` · `/monitoring/` · optional `/monitoring_err` · `/monitoring_err/` | Per **landing** host (GOA bank/exchange/taler · FP · mytops stage, …) | Stack report for that host |
| **taler-monitoring-surface** | `/taler-monitoring-surface` · `/…/` · optional `…_err` | **only** `taler.hacktivism.ch` | Ecosystem surface (ports, nmap, mail, mattermost, versions) |
| **taler-monitoring-aptdeploy** | `/taler-monitoring-aptdeploy` · `/…/` · optional `…_err` | **only** `taler.hacktivism.ch` | apt-src merchant deploy tests (podman on koopa) |
**Not** public mon pages: `/taler-monitoring-mail*`, `/taler-monitoring-mattermost*`, bare `/taler-monitoring` — mail/mm are folded into **surface**.
Related:
- **Phase `monpages`:** obligatory **ERROR** if a listed page is missing, bootstrap stub, not suite HTML, or **content markers fail**. Bare paths that fall through to merchant **code 21** are ERROR (not “OK because slash works”).
- **Publish:** staging `~/monitoring-sites-staging` → `DEPLOY_WWW_ROOT` + reverse-proxy handles for **only** the three families above.
```bash
./taler-monitoring.sh monpages
./taler-monitoring.sh -d hacktivism.ch monpages
```
Host packages (koopa / runners): **[DEPENDENCIES.md](./DEPENDENCIES.md)**.
Host-agents (GOA + FP; shared report pipeline): **[host-agent/README.md](./host-agent/README.md)**.
Public HTML site-gen (same timeout/log/HTML defaults): **[site-gen/README.md](./site-gen/README.md)**.
FP stage host-agent: `host-agent/run-fp-stage-monitoring.sh` + `host-agent/env/stagepaysan.env.example` (see `host-agent/README.md`).
Report for the **GOA** stack with boxed severity badges and **grouped test IDs**
`area.group-NN` (e.g. `www.exchange-01`, `e2e.pay-03`):
| Tag | Meaning |
|-----|---------|
| `┌ OK ┐` | check passed |
| `┌ INFO ┐` | status / note |
| `┌ WARN ┐` | degraded but maybe not fatal |
| `┌ ERROR ┐` | component problem |
| `┌ BLOCKER ┐` | **withdraw/pay path cannot complete** because of this |
IDs reset per **group** inside an area — so “too many www tests” become
`www.exchange-*`, `www.bank-*`, `www.landing-*`, etc.
Catalog: **[TESTS.md](./TESTS.md)**.
Git / tags / suite tree: **[VERSIONS.md](./VERSIONS.md)**.
Optional / deprecated tooling: **[meta/](./meta/)** (firecuda, legacy site-gen, old mail/mm units).
GUI Android notes: **[android-test/GUI-AUTOMATION-NOTES.md](./android-test/GUI-AUTOMATION-NOTES.md)**.
CLI / wallet-cli recurring issues: **[CLI-AUTOMATION-NOTES.md](./CLI-AUTOMATION-NOTES.md)**.
```text
┌ OK ┐ www.exchange-01 exchange /config · HTTP 200
┌ BLOCKER┐ e2e.prereq-02 merchant HTTP 502
```
End of each phase: totals + list of **BLOCKERS** and **ERRORS** (ids included).
## Secrets (local GOA e2e)
**Do not commit passwords.** Suite tree: `~/src/taler-monitoring` · host ops (optional):
sibling `~/src/koopa/koopa-admin-secrets`.
### Local file (recommended on laptop)
```bash
cd ~/src/taler-monitoring
cp secrets.env.example secrets.env # gitignored
$EDITOR secrets.env
./taler-monitoring.sh e2e
```
See **[secrets.env.example](./secrets.env.example)** for every variable that
may appear (admin pass, merchant token, SECRETS_ROOT, progress knobs).
### Search order
1. Already-exported env (`E2E_BANK_ADMIN_PASS`, …)
2. `secrets.env` next to this suite (`MONITORING_SECRETS_ENV=` override)
3. `SECRETS_ROOT` / `KOOPA_ADMIN_SECRETS` file tree
4. Sibling `../koopa-admin-secrets/koopa/host-root`
5. `$HOME/src/koopa/koopa-admin-secrets/koopa/host-root`
6. `$HOME/.config/taler-landing/bank-admin-password.txt`
7. SSH koopa `/root/…` or bank container
### Line IDs + progress
Each check prints **global** and **grouped** ids:
```text
┌ OK ┐ #014 www.exchange-03 exchange /config · HTTP 200
┌ WARN ┐ #015 www.stats-02 stats age soft · yellow badge (not red)
┌ 42% ┐ ████████░░░░░░░░░░░░░░░░░░░░ 14/55
```
- `#014` — run-wide counter (whole `./taler-monitoring.sh` invocation)
- `www.exchange-03` — area.group-NN
- **Colours (left badge + label):**
- **OK** green · **INFO** blue/cyan · **WARN** yellow (never red)
- **ERROR** red · **BLOCKER** magenta
- Progress: badge shows **percent** (not “PROG”); bar fills **left → right**;
`done/total` on the right. Estimate auto-grows if short. Override:
`PROGRESS_TOTAL=N` / `PROGRESS_OFF=1`
- **QR (urls):** harvest `taler://` / `payto://` (+ app store `data-qr-url`) from landings and
`demo-withdraw.json` / `auto-account.json` → validate form → `qrencode` PNG → `zbarimg`
decode must match. Requires packages **`qrencode`** + **`zbar-tools`**.
## Commands
```bash
# Local GOA stack (may use SSH for inside/e2e)
./taler-monitoring.sh # urls + inside + versions + e2e
./taler-monitoring.sh inside # containers on koopa
./taler-monitoring.sh versions # deb.taler.net + package versions vs trixie
./taler-monitoring.sh sanity
./taler-monitoring.sh e2e
# FrancPaysan STAGE (TESTPAYSAN) — like hacktivism breadth:
# default phases: urls + inside + versions + e2e
./taler-monitoring.sh -d stage.lefrancpaysan.ch
./taler-monitoring.sh -d stage.lefrancpaysan.ch inside # stagepaysan only
# inside: low-priv SSH francpaysan-stage-user (User stagepaysan) → podman
# containers stage-lfp-{bank,exchange-ansible,merchant} · ports 9032/9031/9030
# stats: outside-in public …/intro/stats.json (age via generated_at_unix)
# Admin credit for e2e: E2E_BANK_ADMIN_PASS=… or stage host secrets
# (bank-admin-password.txt under /mnt/data/stagepaysan/bank/secrets/)
# Override: INSIDE_SSH=… INSIDE_PROFILE=stage-lfp
# Other domains — public HTTPS only (pass e2e/inside explicitly)
./taler-monitoring.sh -d taler.net
./taler-monitoring.sh --domain taler-ops.ch
./taler-monitoring.sh -d my.taler-ops.ch urls
./taler-monitoring.sh -d demo.taler.net urls
./taler-monitoring.sh taler.net # bare domain = same as -d
# Explicit endpoints (any mix; override profile or skip -d)
./taler-monitoring.sh --bank https://bank.demo.taler.net \
--exchange https://exchange.demo.taler.net \
--merchant https://backend.demo.taler.net --currency KUDOS urls
# Landing stats.json freshness (outside-in; age via generated_at_unix)
STATS_STALE_SECS=900 STATS_FAIL_SECS=3600 ./taler-monitoring.sh -d stage.lefrancpaysan.ch urls
# --- Full load (GOA / hacktivism) ---
# all = urls + inside + versions + sanity + e2e
# full = all + server + ladder + auth401 (long; needs secrets.env)
./taler-monitoring.sh -d hacktivism.ch full
./taler-monitoring.sh -d hacktivism.ch urls inside versions sanity server e2e ladder auth401
# SPA pin after selfbuild-webui.sh (container):
EXPECT_WEBUI_VERSION=1.6.11 EXPECT_WEBUI_OVERLAY=selfbuild-v1.6.11 \
WEBUI_OVERLAY_DENY='c778af04|master-11340' \
./taler-monitoring.sh -d hacktivism.ch urls
# Only 401 matrix (may create throwaway instance):
./taler-monitoring.sh -d hacktivism.ch auth401
# Stage FrancPaysan — durable mon401 account (password in francpaysan-secrets):
./taler-monitoring.sh -d stage.monnaie.lefrancpaysan.ch auth401
# secrets: ~/francpaysan-secrets/stage/instances/mon401.password
# ~/francpaysan-secrets/stage/default-instance-token.txt
# force throwaway create instead: AUTH401_PREFER_DURABLE=0 AUTH401_CREATE=1 …
```
### Landing `stats.json` (www.stats)
Public `GET …/intro/stats.json` must expose timestamps so monitoring can tell
whether the collector is still updating:
| Field | Role |
|-------|------|
| `generated_at_unix` | epoch seconds — **preferred** for age = now − unix |
| `generated_at` | ISO-8601 (fallback if unix missing) |
| `generated_at_human` | UI “updated” / footer string |
Also checks display payloads (bank-shape `withdraws` / accounts, or exchange-db /
merchant-db counters). Soft WARN if age > `STATS_STALE_SECS` (default 900);
ERROR if age > `STATS_FAIL_SECS` (default 3600; set `0` to disable).
Stage shared feed: same `gen_unix` on bank + exchange + monnaies.
## Domain profiles (`domains.conf`)
Each monitored stack must say **what is bank, exchange, and merchant-backend**.
Edit **[domains.conf](./domains.conf)** (or set `TALER_DOMAINS_CONF=`):
```text
# name bank exchange merchant-backend currency local [canonical]
taler-ops.ch bank.taler-ops.ch exchange.taler-ops.ch my.taler-ops.ch CHF 0
```
| Field | Meaning |
|-------|---------|
| **name** | `-d NAME` / `TALER_DOMAIN` match (add alias lines as needed) |
| **bank** | libeufin/bank public host or `https://…` |
| **exchange** | exchange public host or URL |
| **merchant** | merchant **backend** (SPA/API), e.g. `my.taler-ops.ch` |
| **currency** | `GOA` / `KUDOS` / `CHF` / … |
| **local** | `1` = koopa SSH stack; `0` = public only |
| **landing** | `1` = check `/intro` landings + assets; `0` = skip (TOPS / mytops) |
| **canonical** | optional label after alias (e.g. `my.taler-ops.ch` → `taler-ops.ch`) |
Then:
```bash
./taler-monitoring.sh -d taler-ops.ch urls
```
Overrides (always win after profile load):
```bash
./taler-monitoring.sh -d taler-ops.ch \
--merchant https://my.taler-ops.ch \
--exchange https://exchange.taler-ops.ch \
--bank https://bank.taler-ops.ch urls
```
Helpers in `lib.sh`: `set_taler_stack`, `load_domain_profile`, `apply_taler_domain`.
### Built-in profiles (see `domains.conf`)
| Domain / alias | Bank | Exchange | Merchant | Currency |
|----------------|------|----------|----------|----------|
| `hacktivism.ch` / `koopa` | bank.hacktivism.ch | exchange.hacktivism.ch | taler.hacktivism.ch | GOA |
| `taler.net` / `demo.taler.net` | bank.demo.taler.net | exchange.demo.taler.net | backend.demo.taler.net | KUDOS |
| `test.taler.net` | bank.test… | exchange.test… | backend.test… | **TESTKUDOS** · no landings |
| `lefrancpaysan.ch` | bank.… | exchange.… | **monnaie.…** (merchant) | report-only · no landings |
| **`stage.lefrancpaysan.ch`** | stage.bank.… | stage.exchange.… | **stage.monnaie.…** | **TESTPAYSAN** · landings · **inside via stagepaysan** |
| `taler-ops.ch` / `my.taler-ops.ch` | bank.taler-ops.ch | exchange.taler-ops.ch | **my.taler-ops.ch** | CHF · **no landings** |
| `stage.taler-ops.ch` | bank.stage… | exchange.stage… | **my.stage…** | CHF · **no landings** |
| unknown | bank.DOMAIN | exchange.DOMAIN | my.DOMAIN then probe | any · no landings |
**Landings:** GOA, stage LFP (`TESTPAYSAN`), and optionally demo use public `/intro` pages. **taler-ops.ch does not** — `CHECK_LANDING=0` skips intro crawl.
**SSH:**
| Stack | Host alias | User | Scope |
|-------|------------|------|--------|
| koopa / hacktivism | `koopa` | hernani | full host + containers |
| stage LFP | `francpaysan-stage-user` | **stagepaysan** | own podman only (no sudo / no prod data) |
Other domains: never SSH. Optional **e2e** aborts cleanly on login/KYC.
### E2E amounts (variable)
| | Local (koopa) | Remote |
|--|---------------|--------|
| **ATM withdraw** | 20 · 50 · 100 · 200 · **4200** (paivana) | 10 · 20 · 50 |
| **Pay ladder** | 0.01 … 10 | 0.01 … 1 |
| **Paivana** | HTTP paywall + template pay **GOA:4200** (`goa-shop` / `paivana`) | skipped |
```bash
./taler-monitoring.sh e2e
./taler-monitoring.sh -d taler.net urls e2e
# Amount ladder (withdraw 0 → … → max, then pay): same script for GOA + stage
./taler-monitoring.sh ladder # GOA / current domain
./taler-monitoring.sh -d stage.lefrancpaysan.ch ladder # TESTPAYSAN maxima
# Stage auto-reads bank max_wire_transfer_amount (e.g. 2000) and min denom (0.01).
# Explorer password: francpaysan-stage-user …/bank-explorer-password.txt (or EXP_PW=).
# Override: LADDER_MAX_AMOUNT=100 LADDER_STEPS=7 LADDER_PAY=0 …
# customize e2e:
E2E_WITHDRAW_VALUES="20 50 100" E2E_PAY_VALUES="0.05 1 5" ./taler-monitoring.sh e2e
E2E_VARIABLE=0 WITHDRAW_AMT=GOA:50 PAY_AMT=GOA:1 ./taler-monitoring.sh e2e # single fixed
# GOA shop catalog (local hacktivism): full list in E2E_SHOP_PRODUCTS; each run
# shuffles and pays E2E_SHOP_PICK_N products (default 2).
# E2E_SHOP_PRODUCTS lines: id|Product name|GOA:amount
# E2E_SHOP_PICK_N=2
# (landing QR = taler://pay-template/…/{id}; popup = live taler://pay after POST templates/{id})
# Paivana paywall (local GOA only):
# E2E_PAIVANA=1 (default) PAIVANA_PUBLIC=https://paivana.hacktivism.ch
# E2E_PAIVANA_TEMPLATE=paivana E2E_PAIVANA_AMOUNT=GOA:4200 E2E_PAIVANA_INSTANCE=goa-shop
# E2E_PAIVANA=0 # skip
# remote secrets:
# E2E_BANK_ADMIN_PASS=… E2E_MERCHANT_TOKEN=…
```
## Phases
| Phase | What |
|-------|------|
| **inside** | SSH koopa: processes, postgres, local /config,/keys, wirewatch, recent log ERRORs |
| **versions** | `deb.taler.net` reachable (InRelease/Packages/pool `.deb`); containers can reach it + have apt source; installed Taler packages vs **trixie** |
| **sanity** | bank · exchange · merchant sections (public + server) |
| **e2e** | account → credit → withdraw → wallet → confirm → order → pay |
| **ladder** | GOA withdraw: fixed **0** + low bands + **random high ranges** + fixed **max** |
```bash
# package suite (default trixie on deb.taler.net)
TALER_APT_SUITE=trixie ./taler-monitoring.sh versions
TALER_PKG_BEHIND=error ./taler-monitoring.sh versions # any behind = ERROR
```
E2E maps failures to blockers, e.g.:
- `bank-confirm HTTP 409` → wallet did not select exchange
- `no GOA balance` → wirewatch / transfer
- `create order failed` → merchant auth/instance
- `Alarm clock` during pay → usually missing `handle-uri --yes` or wrong pay URI (must be `…/instances/{inst}/{oid}/?c={token}` from merchant `taler_pay_uri`)
- `insufficient balance` → withdraw incomplete
## Performance (urls phase)
Outside-in HTTPS RTT for bank / exchange / merchant critical paths. Each probe prints
`HTTP … · N ms` on the `[OK]`/`[WARN]`/`[ERROR]` line; end of the block prints
`perf summary` (n / min / p50 / avg / max).
| Env | Default | Meaning |
|-----|---------|---------|
| `PERF_WARN_MS` | `8000` | WARN if latency ≥ this |
| `PERF_FAIL_MS` | `20000` | ERROR if latency ≥ this |
| `PERF_CURL_TIMEOUT` | `25` | curl max seconds per probe |
```bash
PERF_WARN_MS=3000 PERF_FAIL_MS=10000 ./taler-monitoring.sh urls
```
## Wallet coins (e2e · ladder)
At every relevant withdraw/pay step the wallet is snapped via `advanced dump-coins`.
Amounts use exchange **`alt_unit_names`** (Kilo-GOA, Mega-GOA, …) with the base
**GOA value in parentheses**:
```text
[INFO] coins after-ATM — amount_circ=2 Kilo-GOA (GOA:2000)
[INFO] coins after-ATM — denoms_in_circ 1 Kilo-GOA (GOA:1000)×2 (=2 Kilo-GOA (GOA:2000))
== ladder · rung 12 GOA:1000000 · 1 Mega-GOA (GOA:1000000) ==
```
Also: coin counts (in circulation / spent), status histogram, Δ vs previous snap.
History TSV: `$METRICS_DIR/coins-history.tsv`. Map loaded from
`${EXCHANGE_PUBLIC}/config` → `currency_specification.alt_unit_names`.
### Final statistics dashboard (e2e · ladder)
At the end of **e2e** and **ladder**, a boxed **FINAL STATISTICS** block prints:
| Section | Content |
|---------|---------|
| **Coins** | in circulation / spent counts + amounts + denoms (alt names) |
| **Money flow** | withdrawn events/amount, spent (paid), net (wd − spent) |
| **Tendency** | coin/amount trend over snaps (↑↓→), slope, last history rows |
| **Performance** | min / p50 / avg / max per phase; first-half vs second-half latency |
| **Load** | host loadavg + mem before/after; per-container RSS/CPU; tendencies |
Successful withdraw/pay steps append to `flow-withdrawn.txt` / `flow-spent.txt` (ladder also uses its TSVs).
## Load / memory (inside · e2e · ladder)
Host **loadavg**, **RAM used/avail**, and per-container **RSS / CPU / block I/O**
(plus postgres DB sizes and role RSS: libeufin, exchange, merchant, …) are
snapshotted via SSH whenever the stack is under load:
| Phase | When |
|-------|------|
| **inside** | end of container status |
| **e2e** | before work · after ATM withdraws · after payments · after shop · e2e-end (+ delta) |
| **ladder** | before first rung · after last rung (+ delta + timing overall) |
```bash
# LAN down? force WAN DNAT host
KOOPA_SSH=koopa-external ./taler-monitoring.sh inside
# or rely on automatic fallback (KOOPA_SSH_FALLBACKS from env)
METRICS_LOAD=0 ./taler-monitoring.sh e2e # skip host probes
LADDER_LOAD=0 ./taler-monitoring.sh ladder
```
| Env | Default | Meaning |
|-----|---------|---------|
| `KOOPA_SSH` | `koopa` | primary SSH host |
| `KOOPA_SSH_FALLBACKS` | *(from env)* | tried if primary fails |
| `METRICS_LOAD` | `1` | `0` = skip all host/container load probes |
| `LADDER_LOAD` | `1` | `0` = skip load in ladder only |
| `LADDER_PAY` | `1` | `0` = skip payment ladder after withdraws |
| `LADDER_WITHDRAW_SCALE` | `1.5` | withdraw mids / pay mids ratio (funding headroom) |
| `LADDER_STEPS` | `23` | rungs for both withdraw and pay (0 + mids + max−1 + max) |
| `METRICS_LOAD_SSH_TIMEOUT` | `90` | seconds for remote load python |
## Needs
- SSH via `KOOPA_SSH` / `KOOPA_SSH_FALLBACKS` (env) (inside / load / sanity server bits)
- secrets under `koopa-admin-secrets/...` for e2e
- `taler-wallet-cli` for e2e
## Languages (i18n) — **v1.2+**
Default **English**. Select language with `--lang en|fr` or `TALER_MON_LANG`.
FrancPaysan stacks auto-select **French** (`*lefrancpaysan*`). See `VERSIONS.md` and `i18n.sh`.
## Public monitoring pages via FQDN (v1.3.1+ / obligatory v1.7.6+ / layout v1.8.0+)
Phase **`monpages`** is **obligatory**: missing or non-suite HTML is **ERROR**.
Soft mode only with `MONPAGES_REQUIRE_PUBLIC=0` (escape hatch).
| Stack | What monpages expects (v1.9.0) |
|-------|--------------------------------|
| **GOA** | 3× `/monitoring/` + **surface** (existence **and** top/bottom/content markers) |
| **FrancPaysan** | Stack mon hosts `/monitoring/` with same content rules |
| **Nine landings (model)** | Each landing front: `/monitoring` + `/monitoring_err` (err when a failed run exists) |
```bash
./taler-monitoring.sh -d hacktivism.ch monpages
./taler-monitoring.sh -d lefrancpaysan.ch monpages
```
If pages are missing publicly: `sudo $APPLY_MONITORING_LIVE` on koopa
(FP: Infomaniak vhost for `/monitoring*`).
## Devtesting · fake-franken CHF (v1.13+)
Phase **`devtesting`** (aliases: `franken`, `fake-franken`) exercises the
Taler Operations **rusty** host (`taler-devtesting` forced command):
```bash
./taler-monitoring.sh -d stage.taler-ops.ch --currency CHF devtesting
```
Needs SSH to `devtesting@rusty.taler-ops.ch` (Host `taler-rusty-devtesting`).
Probes: CLI help, `geniban` (CH IBAN), `fake-incoming` CHF to exchange credit
IBAN. A nexus bounce *missing reserve public key* is **OK** for the plumbing
probe (no real wallet reserve required).
## Version on monitoring pages (v1.3+ / sticky bar)
The sticky bar shows the suite **tag** (e.g. `v1.11.0`) and links to that tag on Forgejo.
**v1.12.0:** Run header (**monitoring env context**: env_file, suite pin, domain/hosts/flags) is a collapsed `<details>` block above the console log — expand to inspect agent context.
**v1.11.0:** Click the sticky **error** or **warning** count to filter the console log to only those lines (other log lines hide; sticky bar, stats, and top overviews stay). Same via `#filter-error` / `#filter-warn` (or `#first-error` / `#first-warn`). Click again or **Show all** to clear.

205
TESTS.md Normal file
View file

@ -0,0 +1,205 @@
# taler-monitoring — grouped test IDs
Every check line has a **global** run number and a **grouped** id:
```text
┌ OK ┐ #001 www.exchange-01 exchange /config · HTTP 200
┌ WARN ┐ #002 www.stats-02 soft age · yellow (never red)
┌ ERROR ┐ #003 e2e.pay-03 order create failed · HTTP 502
┌ BLOCKER┐ #004 e2e.prereq-02 merchant secret missing
┌ PROG ┐ ████░░░░ 25% 4/16
```
- `#NNN` — monotonic for the whole monitoring run
- `area.group-NN` — logical block (resets per group, continues if re-entered)
| Area | Phase script | Groups (examples) |
|------|--------------|-------------------|
| **www** | `check_urls.sh` | `exchange` `perf` `stats` `bank` `merchant` `paivana` `landing` `qr` |
| **inside** | `check_inside.sh` | `ssh` `bank` `exchange` `merchant` `caddy` `load` |
| **versions** | `check_versions.sh` | `outside` `inside` `compare` |
| **aptdeploy** | `check_apt_deploy.sh` | `trixie` `trixie-testing` (koopa podman apt-src merchant smoke) |
| **surface** | `check_surface.sh` | **remote-only** ecosystem / `-d` domain inventory (not in default/all/full) |
| **monpages** | `check_monitoring_pages.sh` | public HTML via FQDN — **ERROR** if missing or markers fail (top+bottom+body); surface required on GOA (v1.9.0) |
| **surface** | `check_surface.sh` | ecosystem inventory + **nmap** OS fingerprint; mail/mattermost content reported via surface page |
| **sanity** | `check_sanity.sh` | `bank` `exchange` `merchant` |
| **server** | `check_server.sh` | (flat `server-NN` or host groups) |
| **e2e** | `check_e2e.sh` | `prereq` `load` `bank` `wallet` `atm` `settle` `pay` `shop` `paivana` `dig` `report` |
| **ladder** | `check_goa_ladder.sh` | `plan` `load` `withdraw` `pay` `report` |
**Why groups:** flat `www-001`…`www-080` was hard to talk about.
`www.bank-04` / `e2e.atm-02` pin the failure to a logical block.
Usage in scripts:
```bash
set_area www
set_group exchange # → www.exchange-01, www.exchange-02, …
set_group bank # counter resets → www.bank-01
```
`set_group` also prints a small group chip (`┌ www.exchange ┐`) so logs show section boundaries.
Re-entering the same group later (e.g. exchange legal after perf) **continues** NN — `www.exchange-03`, not a second `-01`.
Without `set_group`, IDs stay flat: `area-01`, `area-02`, …
Optional soft checks still consume a number when they WARN.
Numbering follows **executed** checks (early skip may shift later NN inside the same group).
---
## www — public URLs (`./taler-monitoring.sh urls`)
| Group | Checks (typical) |
|-------|------------------|
| **www.exchange-** | `/config`, currency, **alt_unit_names**; `/keys` (+ alt soft); `/intro/`, `/`; **`/terms`**, **`/privacy`**, `/terms/` |
| **www.perf-** | outside-in RTT: bank/exchange/merchant `/config`, keys, webui, intro — ms; WARN ≥ `PERF_WARN_MS` (8000); ERROR ≥ `PERF_FAIL_MS` (20000) |
| **www.stats-** | `/intro/stats.json` **reachable**; **timestamps** (`generated_at_unix` + human/ISO); **freshness** vs wall clock (`STATS_STALE_SECS` WARN, `STATS_FAIL_SECS` ERROR); **display fields** (bank-shape withdraws/accounts, or exchange-db / merchant-db keys); optional performance block; shared-feed equal `gen_unix` when bank collector is published to all three |
| **www.bank-** | `/config`, currency, alt_unit_names; integration/webui/intro; **auto-account.json** (required GOA/local; skip/soft off-GOA e.g. TESTPAYSAN); `/terms`, `/privacy` |
| **www.merchant-** | `/config` currency + currencies alt_unit_names; listed exchanges alt; webui/intro; **`/terms`**, **`/privacy`** |
| **www.webui-** | SPA fingerprints: **`/webui/version.txt`**, **`/webui/version-overlay.txt`**, `index.html` / `index.js`; optional pin via `EXPECT_WEBUI_VERSION` / `EXPECT_WEBUI_OVERLAY` / `WEBUI_OVERLAY_DENY` |
| **www.paivana-** | local GOA paywall front (redirect to template) |
| **www.landing-** | own-stack intro links; static assets (`qrcode.min.js` hard; `og-goa-shop.png` hard only GOA/local); **demo-withdraw.json** GOA-only; shop-pay soft; stage merchant shop: `shop-ui.js` + `shops.css` + `qrcode.min.js`; cross-links local |
| **www.qr-** | QR payloads: harvest `taler://` / `payto://` / app `data-qr-url` from landings + mint JSON; **form** check; **qrencode → zbarimg** exact roundtrip; optional static QR images. Needs `qrencode` + `zbar-tools`. Skip: `QR_CHECK=0` |
**QR rule:** `taler://withdraw/HOST/taler-integration/UUID` (no default `:443`/`:80`); `taler://pay/` / `pay-template/`; `payto://…` shape OK; decoded PNG must equal payload. Form errors on withdraw/pay/payto are **ERROR** on local/GOA.
**Legal docs rule:** HTTP 200, non-empty body, not plain `not configured`, not merchant API JSON `code:21`. Local stack may require content needle (terms/privacy/FADP/GOA…).
**Performance rule:** Measured from the **monitoring runner** (public via Caddy), not container loopback. Latency (ms) on every perf line + **perf summary** (n / min / p50 / avg / max).
**Landing links rule:** Own-stack must be HTTP 200 (or redirect→200). External stores/docs soft WARN. Auto-account wallet link must be `taler://withdraw/HOST/taler-integration/…` (default ports stripped), never payto.
**alt_unit_names rule:** non-empty map including scale key `"0"`. Multi-currency merchant: follow each `exchanges[]` public `/config`.
---
## inside — koopa SSH (`./taler-monitoring.sh inside`)
| Group | Checks |
|-------|--------|
| **inside.host-** | **host-podman**: reachability via `podman exec` on this machine (GOA host / `INSIDE_PODMAN=1`) — not SSH |
| **inside.ssh-** | **ssh**: from laptop/remote — SSH to `KOOPA_SSH` / `INSIDE_SSH`, then podman there |
| **inside.bank-** | container, libeufin, postgres, local `/config`, nginx, DNS pins |
| **inside.exchange-** | container, httpd, wirewatch, aggregator, transfer, local keys, DNS |
| **inside.merchant-** | container, httpd, wirewatch, depositcheck, DNS |
| **inside.caddy-** | host reverse-proxy process |
| **inside.load-** | host loadavg + RAM; per-container RSS/CPU |
| **inside.disk-** / **aptdeploy.disk-** / **versions.disk-** / **server.disk-** | free space via `df`: **WARN** if use ≥ `DISK_WARN_USED_PCT` (85), **ERROR** if full or use ≥ `DISK_ERR_USED_PCT` (95); host + containers |
Remote lines `E|comp|LEVEL|key|detail` each become one numbered result under that component group.
SSH: `KOOPA_SSH` (default `koopa`), then `KOOPA_SSH_FALLBACKS` when LAN is unreachable.
---
## e2e — withdraw + pay (`./taler-monitoring.sh e2e`)
| Group | Checks |
|-------|--------|
| **e2e.prereq-** | wallet-cli, currency, budgets, secrets, public reachability |
| **e2e.load-** | host/container load snapshots (before / after ATM / after pay) |
| **e2e.bank-** | account, credit, withdraw ops |
| **e2e.wallet-** | exchange + ToS, wallet setup |
| **e2e.atm-** | ATM withdraw ladder rungs |
| **e2e.settle-** | wallet settlement wait / spendable balance |
| **e2e.pay-** | variable payments |
| **e2e.shop-** | GOA shop product catalog pays |
| **e2e.paivana-** | template paywall pay |
| **e2e.dig-** | coins-missing dig (inside + withdrawal-operation) |
| **e2e.report-** | final tallies |
When filing an issue, quote the full id + label, e.g.
`e2e.atm-03 ATM withdraw 5 GOA · bank confirmed, wallet empty`.
---
## versions / sanity / ladder / server / auth401
| Area.group | Meaning |
|------------|---------|
| **versions.outside-** | deb.taler.net suite index |
| **versions.host-** | host-podman: packages via local `podman exec` (`INSIDE_PODMAN=1`) |
| **versions.ssh-** | ssh: packages via `KOOPA_SSH` / `INSIDE_SSH` from laptop/remote |
| **versions.compare-** | installed vs suite |
| **aptdeploy.trixie-fresh-** | fresh rebuild container `…-apt-src-trixie` |
| **aptdeploy.trixie-testing-fresh-** | fresh rebuild `…-trixie-testing` |
| **aptdeploy.trixie-upgrade-** | upgrade-track `…-trixie-upgrade` (install once, then mytops-style upgrade) |
| **aptdeploy.trixie-testing-upgrade-** | upgrade-track `…-trixie-testing-upgrade` |
Standalone: `host-agent/run-aptdeploy.sh` (on koopa local; else ssh `$DEPLOY_SSH` / `$APT_DEPLOY_SSH`).
### surface (remote-only public inventory)
```bash
./taler-monitoring.sh surface # taler.net / gnunet / taler-systems / mattermost / …
./taler-monitoring.sh -d hacktivism.ch surface # all things under that domain
```
- Catalog: `surface-catalog.conf` (extend as hosts are found)
- DNS, optional ICMP, TCP ports, then **protocol** checks (HTTPS/HTTP/SSH) before ERROR
- TLS cert expiry; Server-header / `/config` version hints; else **WARN: version unknown**
- OSV CVE query when package+version known → **ERROR** on hits
- Never SSH / never podman on the *targets*
| **sanity.bank-** / **.exchange-** / **.merchant-** | public + optional server-side per component |
| **ladder.plan-** / **.load-** / **.withdraw-** / **.pay-** / **.report-** | amount ladder (GOA ceiling or stage TESTPAYSAN `max_wire`) |
| **server-** | SSH host ports / processes (flat unless grouped later) |
| **auth401.*** | merchant 401 / case matrix (WebUI APIs): **signup** `POST /instances` MIX id, **login** Basic user low vs MIX/UPPER, full **case** matrix, **password** bad/empty, **pwchange** `POST private/auth` + re-login, **admin-create** `POST /management/instances` (if admin token), **bearer**, **durable mon401**, **webui** SPA `toLowerCase` probe (`check_auth401.sh`) |
### SPA pin (after selfbuild-webui)
```bash
EXPECT_WEBUI_VERSION=1.6.11 \
EXPECT_WEBUI_OVERLAY=selfbuild-v1.6.11 \
WEBUI_OVERLAY_DENY='c778af04|master-11340' \
./taler-monitoring.sh -d hacktivism.ch urls
```
Skip SPA block: `CHECK_WEBUI_SPA=0`.
### Full load (maximum phases)
| Alias | Expands to (local GOA) |
|-------|-------------------------|
| **`all`** | `urls inside versions sanity e2e` |
| **`full`** | `urls inside versions sanity server e2e ladder auth401` |
```bash
cd ~/src/taler-monitoring
# secrets.env with bank admin + merchant tokens (see secrets.env.example)
./taler-monitoring.sh -d hacktivism.ch full
# same, explicit:
./taler-monitoring.sh -d hacktivism.ch \
urls inside versions sanity server e2e ladder auth401
```
---
## Visual format (colour on)
```text
╔══════════════════════════════╗
║ www · public URLs · … ║
╚══════════════════════════════╝
┌ www.exchange ┐
┌ OK ┐ www.exchange-01 exchange /config · HTTP 200
┌ www.bank ┐
┌ OK ┐ www.bank-01 bank /config · HTTP 200
┌ ERROR ┐ www.bank-04 bank /intro/auto-account.json · invalid withdraw
┌ ERRORS · failed checks ┐
• www.bank-04 bank /intro/auto-account.json · invalid withdraw
totals: 40 OK, 1 ERROR, 2 WARN, 5 INFO
```
`NO_COLOR=1` or `CLICOLOR=0` disables boxes/colours (ASCII `[ OK ]` + `-- group --` headers).
Always use `printf --` friendly plain headers when colour is off (leading `---` is not a printf option).
## devtesting · fake-franken (v1.13+)
| ID prefix | What |
|-----------|------|
| **devtesting.cli-** | SSH to rusty · `taler-devtesting` CLI |
| **devtesting.geniban-** | synthetic CH IBAN |
| **devtesting.credit-payto-** | exchange credit IBAN for nexus |
| **devtesting.fake-incoming-** | CHF IN via nexus; bounce missing reserve pub = plumbing OK |

1
VERSION Normal file
View file

@ -0,0 +1 @@
1.13.12

112
VERSIONS.md Normal file
View file

@ -0,0 +1,112 @@
# VERSIONS — taler-monitoring
Release history for the standalone **taler-monitoring** suite
(https://git.hacktivism.ch/hernani/taler-monitoring).
## Version scheme
Tags and `VERSION` use **MAJOR.FEATURE.FIX** (three components), introduced in **v1.4.0**:
| Part | Meaning |
|------|---------|
| **MAJOR** | Breaking changes to CLI, layout, or host-agent contract |
| **FEATURE** | New capabilities (phases, HTML features, i18n packs, …) |
| **FIX** | Bugfixes and docs that do not add a feature |
Git tags: `vMAJOR.FEATURE.FIX` (e.g. `v1.8.0`). File `VERSION` omits the `v` prefix.
| Tag | Date (UTC) | Notes |
|-----|------------|--------|
| **v1.13.12** | 2026-07-19 | **Bugfix:** SUMMARY chrome (`[ WARN ] 2`, `warnings only`, OK/ERROR counts without `#NNN`) is **meta**, not warn/error — sticky counts + warn filter no longer double SUMMARY rows; filter context skips blank lines. |
| **v1.13.11** | 2026-07-19 | **Bugfix:** sticky error/warn filter keeps **±3 dimmed context lines** around each match (gray/grayscale) so phase/group context stays readable; dashed gap between distant clusters. |
| **v1.13.10** | 2026-07-19 | **Bugfix:** multi-phase runs print **one global SUMMARY** at parent end (aggregated OK/ERROR/WARN/INFO/BLOCK via `TALER_MON_STATE`); no phase SUMMARY at all (not even last phase); mid-phase still lists ERRORS/BLOCKERS; solo `check_*.sh` unchanged. |
| **v1.13.9** | 2026-07-19 | **Bugfix:** no mid-run SUMMARY / progress standings between phases (no more 34/52 + SUMMARY after urls while monpages still runs); full stand only on last phase or solo check script; `progress_finish` remains. |
| **v1.13.8** | 2026-07-19 | **Bugfix:** monpages **TOP** meta/marker failures (missing generated/version-link/taler-mon-page, empty race body) are **WARN only** — no longer self-red the run; optional `MONPAGES_TOP_STRICT=1` restores ERROR. |
| **v1.13.7** | 2026-07-19 | **Bugfix:** move optional/meta scripts under `meta/` (deprecated mail/mm units, firecuda outside-runner, legacy multi-host site-gen); production path stays host-agent + `site-gen/console_to_html.py`. |
| **v1.13.6** | 2026-07-19 | **Bugfix:** machine paths and SSH hosts only via env (`~/.config/taler-monitoring/env` / `taler-monitoring-env`); suite loads env before defaults; no `/home/hernani` or bogus host hardcodes. |
| **v1.13.5** | 2026-07-19 | **Bugfix:** progress bar — env-aware phase estimates (CHECK_BANK/LANDING, mon hosts); re-fit total after each phase (done+remaining); drop fake +12 for host-agent monpages post-check (fixes stage 42/139→46/46 snap). |
| **v1.13.4** | 2026-07-19 | **Bugfix:** monpages race-proof — atomic HTML write; rsync `--delay-updates` + settle; fetch retries; larger head windows; pre-publish soft content (`MONPAGES_PRE_PUBLISH`); bootstrap markers. |
| **v1.13.3** | 2026-07-19 | **Bugfix:** sticky **generated** time in **Europe/Zurich** (CEST/CET · MESZ/MEZ), not UTC `…Z`; ISO keeps offset for age JS. |
| **v1.13.2** | 2026-07-19 | **Bugfix:** sticky version link uses **commit** URL (`/src/commit/<sha>`) not `/src/tag/v…` (tags not yet pushed → Forgejo fat 404); tags v1.11–v1.13.1 pushed to origin. |
| **v1.13.1** | 2026-07-19 | **Bugfix:** progress bar **global** done/total — phase `summary()` no longer snaps total to phase-local N (fixes 34/34 then 38/47); final `progress_finish` snaps once; header shows 0/N. |
| **v1.13.0** | 2026-07-19 | **Feature:** phase **devtesting** / **franken** — fake-franken CHF via `rusty.taler-ops.ch` (`taler-devtesting` geniban + fake-incoming); stage host-agent can include it. |
| **v1.12.0** | 2026-07-19 | **Feature:** collapsible **monitoring env context** (agent header: env_file, suite pin, domain/hosts/flags) — `<details>` default closed; summary shows agent · domain · suite. |
| **v1.11.0** | 2026-07-19 | **Feature:** sticky **error/warn filter** — click counts to show only error or only warn log lines; sticky bar + overviews stay; `#filter-error` / `#filter-warn` (+ `#first-*` still works). |
| **v1.10.11** | 2026-07-19 | **Bugfix:** surface/aptdeploy wrappers **force** MON_HOSTS+PHASES+MONPAGES_INVENTORY (env cannot expand job to bank/exchange); wrap more mon knobs. |
| **v1.10.10** | 2026-07-19 | **Bugfix:** monpages bare code 21 is WARN again by default (slash form remains ERROR); bare redir still needs live Caddy `redir *` — set MONPAGES_BARE_STRICT=1 to enforce. |
| **v1.10.9** | 2026-07-19 | **Bugfix:** stats policy INFO no longer says bare “ERROR” (false #first-error on www.stats-01); extract_errors only from `┌ ERROR` badges. |
| **v1.10.8** | 2026-07-19 | **Docs/clarity:** only three public mon URL families (`/monitoring`, `/taler-monitoring-surface`, `/taler-monitoring-aptdeploy`) each bare+slash(+_err); suite name ≠ path; no `/taler-monitoring` page. |
| **v1.10.7** | 2026-07-19 | **Bugfix:** stop always-rsync of **stale suite-overlay** over Forgejo tip (broke stage sticky / new site-gen). Overlay only when suite missing phases. |
| **v1.10.6** | 2026-07-19 | **Fix:** stage mon HTML quality — suite update out of public log; filter git noise; stage-specific sticky (not GOA); `CHECK_BANK=0` / `MERCHANT_REQUIRED` for mytops stage scope. |
| **v1.10.5** | 2026-07-19 | **Fix:** aptdeploy monpages **job-only** (own `/taler-monitoring-aptdeploy*`); full GOA mon inventory + bare URL checks stay on **hacktivism** general run — no bank/exchange/surface monpages noise on aptdeploy HTML. |
| **v1.10.4** | 2026-07-19 | **Fix:** suite **auto-upgrade strict** (`SUITE_UPDATE_STRICT=1` default — abort on fetch fail); monpages ERROR on **bootstrap** HTML; ship mytops-stage timer/service units; host-agent refuses stale tree. |
| **v1.10.3** | 2026-07-19 | **Bugfix:** sticky ERROR only from check badges (`┌ ERROR`) / `ERROR monpages`; ignore git `HEAD is now at …` commit subjects that mention ERROR. |
| **v1.10.2** | 2026-07-19 | **Bugfix:** sticky ERROR count — classify with word-boundary `\bERROR\b` (not substring); avoids false red from git “wrote-errors” commit lines and similar. |
| **v1.10.1** | 2026-07-19 | **Bugfix:** HTML converter must not treat teed `wrote … (errors=N)` lines as ERROR (false sticky red after clean host-agent runs). |
| **v1.10.0** | 2026-07-19 | **Feature:** aptdeploy **container overview board** — inventory + per-pod matrix (state, merchant, httpd, ldd, OK/ERROR) in log and HTML sticky/top (`#container-overview`). |
| **v1.9.4** | 2026-07-19 | **Fix:** restore **taler-monitoring-aptdeploy(+_err)** as first-class GOA mon page (catalog + monpages path allow + surface docs). Mail/mattermost stay folded into surface. |
| **v1.9.3** | 2026-07-19 | **Bugfix (suite):** monpages merchant **code 21** always ERROR (bare no longer soft-WARN); apply hints use absolute `/home/hernani/koopa-caddy/…`; ship `host-agent/apply-monitoring-live.sh` (smoke array, bare check); fix ROOT-ON-KOOPA wrong `redir /path/ 302` examples. |
| **v1.9.2** | 2026-07-19 | **Bugfix:** Caddy bare-path redir footgun — inside `handle` use `redir * /path/ 302` (not `redir /path/ 302`, which becomes `Location: 302` and bare URLs fall through to merchant **code 21**). Snippet + ROOT-APPLY: absolute apply path (`/home/hernani/koopa-caddy/…`, not `~` as root). |
| **v1.9.1** | 2026-07-19 | **Bugfix:** stale path confusions in repo — sticky `pages_i3` no longer lists mail/mattermost/aptdeploy mon pages; lib.sh/android-test drop `scripts/taler-monitoring/`; monpages/host-agent hints match v1.8 layout; clarify GIT vs GUI vs CLI `*AUTOMATION-NOTES.md` (three files by design). |
| **v1.9.0** | 2026-07-19 | **monpages content:** ERROR if page missing or markers incomplete — top (sticky-bar/status-bar/generated) + bottom (footer/`taler-mon:bottom`); mid-run (progress &lt;100% / incomplete meta) relaxes bottom only. Surface required on normal GOA runs with meaningful body (not progress-only). HTML emits `taler-mon:top` / `taler-mon:bottom:complete|incomplete`. |
| **v1.8.0** | 2026-07-19 | **Simplified public pages:** only `/taler-monitoring-surface(+_err)` on `taler.hacktivism.ch` for ecosystem software/versions; **9 landing stacks** each keep `/monitoring(+_err)`. Mail (firefly.gnunet.org + anastasis.taler-systems.com) and Mattermost folded into surface job; separate mail/mattermost mon pages + timers deprecated. **nmap** OS fingerprinting on surface (`SURFACE_NMAP`); **ERROR** when packages are behind (`TALER_PKG_BEHIND=error` default) or OS fingerprint is EOL. |
| **v1.8.1** | *planned* | **L10n** packs: **fr-CH** (FrancPaysan) and **de-CH** (hacktivism / Taler CH German). |
| **v1.7.7** | 2026-07-18 | **domains:** stage mytops merchant FQDN is `stage.my.taler-ops.ch` (betel mon sites). |
| **v1.9.1** | 2026-07-19 | **Bugfix:** stale path confusions in repo — sticky `pages_i3` no longer lists mail/mattermost/aptdeploy mon pages; lib.sh/android-test drop `scripts/taler-monitoring/`; monpages/host-agent hints match v1.8 layout; clarify GIT vs GUI vs CLI `*AUTOMATION-NOTES.md` (three files by design). |
| **v1.9.0** | 2026-07-19 | **monpages content:** ERROR if page missing or markers incomplete — top (sticky-bar/status-bar/generated) + bottom (footer/`taler-mon:bottom`); mid-run (progress &lt;100% / incomplete meta) relaxes bottom only. Surface required on normal GOA runs with meaningful body (not progress-only). HTML emits `taler-mon:top` / `taler-mon:bottom:complete|incomplete`. |
| **v1.8.0** | 2026-07-18 | **monpages obligatory (ERROR):** GOA checks full suite inventory (bank/exchange/taler `/monitoring/` + surface/aptdeploy/mattermost/mail + on-disk discovery); FP only its own mon hosts; `MONPAGES_REQUIRE_PUBLIC=1` default (escape hatch `=0`). **Bugfix:** `with_timeout` no longer re-runs a phase when it exits non-zero. |
| **v1.9.1** | 2026-07-19 | **Bugfix:** stale path confusions in repo — sticky `pages_i3` no longer lists mail/mattermost/aptdeploy mon pages; lib.sh/android-test drop `scripts/taler-monitoring/`; monpages/host-agent hints match v1.8 layout; clarify GIT vs GUI vs CLI `*AUTOMATION-NOTES.md` (three files by design). |
| **v1.9.0** | 2026-07-19 | **monpages content:** ERROR if page missing or markers incomplete — top (sticky-bar/status-bar/generated) + bottom (footer/`taler-mon:bottom`); mid-run (progress &lt;100% / incomplete meta) relaxes bottom only. Surface required on normal GOA runs with meaningful body (not progress-only). HTML emits `taler-mon:top` / `taler-mon:bottom:complete|incomplete`. |
| **v1.8.0** | 2026-07-18 | **Bugfix:** monpages soft mode `MONPAGES_REQUIRE_PUBLIC=0` (FP wrappers default) — missing public HTML is WARN until vhost publish; generic publish hints. |
| **v1.7.4** | 2026-07-18 | **Bugfix:** monpages bare URLs (`/monitoring` without `/`) WARN by default (trailing slash remains ERROR); avoids false red when slash form is live. |
| **v1.7.3** | 2026-07-18 | Monitoring **pages themselves** as suite scope: README section + sticky “What this monitors” block (monpages / public HTML paths). |
| **v1.7.2** | 2026-07-18 | **Bugfix:** Caddy static handles — `root` must be host dir + matcher `/monitoring*` (not leaf dir + `/monitoring/`), else file_server returns empty 404 after apply. |
| **v1.7.1** | 2026-07-18 | Host-agent default `STRICT_EXIT=1`; monpages staging-vs-public diagnosis; re-htmlify after public check fails. |
| **v1.7.0** | 2026-07-18 | CLI `--ver` / `--version` / `-V`: suite version, git commit, repo/tag/commit URLs. |
| **v1.6.0** | 2026-07-18 | **Mail monitoring:** phase `mail` (firefly + pixel/TSA); MX/SMTP/IMAP/SPF/DMARC; `/taler-monitoring-mail*`. |
| **v1.5.0** | 2026-07-18 | **Mattermost monitoring:** phase `mattermost` for `mattermost.taler.net`; `/taler-monitoring-mattermost*`. |
| **v1.4.1** | 2026-07-18 | Generic docs (no client hostnames); canonical install path `~/src/taler-monitoring`. |
| **v1.4.0** | 2026-07-18 | **MAJOR.FEATURE.FIX** version scheme formalized (three-component tags + `VERSION` file). |
| **v1.3.4** | 2026-07-18 | Summary i18n, monpages code-21 detection, aptdeploy rsync path. |
| **v1.3.3** | 2026-07-18 | GIT notes vs GUI notes; monpages in TESTS. |
| **v1.3.2** | 2026-07-18 | Suite paths only `~/src/taler-monitoring` (no admin-log suite tree). |
| **v1.3.1** | 2026-07-18 | monpages FQDN check + deploy no longer silent-skip. |
| **v1.3.0** | 2026-07-18 | Sticky bar suite version + Forgejo tag link. |
| **v1.2.0** | 2026-07-18 | Language selection: `--lang` / `TALER_MON_LANG` (en\|fr). |
| **v1.1.0** | 2026-07-18 | i18n chrome: English default; French for FrancPaysan. |
| **v1.0.0** | 2026-07-18 | Initial standalone release. |
## Sticky bar version (v1.3.0+)
Generated HTML shows the installed release tag (e.g. `v1.8.0`) and links to Forgejo:
`https://git.hacktivism.ch/hernani/taler-monitoring/src/tag/<tag>`
Resolved via `git describe --tags` / `VERSION`.
## Public monitoring pages (v1.3.1+)
Publish under `DEPLOY_WWW_ROOT` and Caddy `handle`s. Missing handles → merchant **JSON code 21**.
See koopa-admin-log issue **I-09** / `apply-monitoring-live.sh`.
| Check | How |
|-------|-----|
| Phase | `./taler-monitoring.sh monpages` |
| CLI version | `./taler-monitoring.sh --ver` |
## Language (v1.2.0+)
| Method | Example |
|--------|---------|
| CLI | `./taler-monitoring.sh --lang fr …` |
| Env | `TALER_MON_LANG=en` or `fr` |
| Auto | `*lefrancpaysan*` → fr, else en |
Full locale packs (fr-CH / de-CH): **v1.8.0** (planned).
## Pin a release
```bash
git clone https://git.hacktivism.ch/hernani/taler-monitoring.git ~/src/taler-monitoring
cd ~/src/taler-monitoring && git checkout v1.9.1
```

11
android-test/.gitignore vendored Normal file
View file

@ -0,0 +1,11 @@
apks/*.apk
out/
*.png
out-source/
out-source-*/
out-gui/
out-source-gui/
out-gui-chain/
out-matrix-*/
out-stack-run-*/
out-survey-*/

367
android-test/GUI-AUTOMATION-NOTES.md Normal file
View file

@ -0,0 +1,367 @@
# GUI automation notes (Android wallet)
Companion docs:
- CLI / wallet-cli: [`../CLI-AUTOMATION-NOTES.md`](../CLI-AUTOMATION-NOTES.md)
- Git / tags / suite tree: [`../VERSIONS.md`](../VERSIONS.md)
Canonical notes for **graphical** Android wallet tests against GOA / stage.
---
## Minimal variant (implemented first) — status 2026-07
This is the **minimal vanilla** path we actually shipped; full dual-platform
GUI is planned via flags (below), not required to use these scripts today.
| Piece | Status | Notes |
|-------|--------|--------|
| F-Droid published APK install | **done** | `net.taler.wallet.fdroid` 1.6.1 / 854 |
| Source build + same smoke | **done** | `:wallet:assembleFdroidDebug` → `net.taler.wallet.fdroid.debug` |
| Default source branch | **done** | `dev/hernani-inference/fix-bank-withdraw-auto-exchange` (minimal GOA exchange auto-add) |
| Deep-link entry (≡ QR) | **done** | `adb` `VIEW` **withdraw then pay** (`taler://withdraw/…` → `taler://pay…`) |
| Explorer API mint (chain) | **done** | `run-goa-gui-chain.sh` (GOA + stage) |
| UI taps via uiautomator | **done** | best-effort Confirm/ToS/Pay; ANR → prefer **Wait** |
| Screenshots / logcat artifacts | **done** | under `out*`, `out-gui-chain/` (gitignored) |
| Host CLI e2e settlement | **separate** | `taler-monitoring.sh e2e` / `ladder` (not Android UI) |
| Reliable unattended Confirm on 4 GiB Linux host | **limited** | System UI ANR; use phone or more RAM |
| iOS GUI | **not in this tree** | see taler-ios `dev/hernani-inference/gui-workflows` |
| macOS full dual AVD helpers | **upstream** | taler-android `gui-workflows` (Homebrew); ported ideas here for Linux |
**Ground rule:** bring it to run; app diffs only if needed and **minimal**; prefer
extending existing `dev/hernani-inference/*` branches over new trees.
### Minimalvariante: was geht / was nicht
Einschränkungen der **aktuell implementierten** Minimalvariante (deep-link +
uiautomator best-effort). Kein voller Dual-Platform-GUI-Stack, kein Ersatz für
Host-CLI-Settlement.
#### Geht (grün)
| Fähigkeit | Wie / Hinweis |
|-----------|----------------|
| APK installieren | F-Droid `net.taler.wallet.fdroid` **oder** Source `assembleFdroidDebug` (`.debug`) |
| GOA-Exchange ohne manuelles Add | Source-Build von `fix-bank-withdraw-auto-exchange` (Default in `run-android-build-and-smoke.sh`) |
| **Abheben + Bezahlen** | Beide Beine: erst Withdraw-URI + Taps, dann Pay-URI + Taps (`DO_WITHDRAW`/`DO_PAY`) |
| Light UI-Drive | Confirm / Accept / Pay-Taps über Label-Suche (`lib_ui.py`, gui-smoke, chain) |
| Multi-Runden-Taps | `run-android-gui-smoke.sh` / `run-goa-gui-chain.sh` (best-effort) |
| Explorer-Mint (Kette) | `run-goa-gui-chain.sh` + `EXP_PW_FILE` (GOA + stage) |
| Stack-Wahl | `STACK=goa` / `stage` (Bank/Merchant-URLs) |
| Artefakte | Screenshots, UI-XML, logcat unter `out*` / `out-gui-chain/` (gitignored) |
| Hybrid-Smoke | Install + Intent + leichte Taps: `run-android-pay-smoke.sh` |
| Settlement **nachweisen** | **Host-seitig**: `taler-monitoring.sh e2e` / `ladder` (nicht Android-UI) |
| Flags abschalten | `AUTO_ANDROID=0` skip; `AUTO_GUI=0` → nur deep-link/pay-smoke |
#### Geht **nicht** oder nur eingeschränkt (rot / gelb)
| Beschränkung | Folge / Workaround |
|--------------|--------------------|
| **Unattended Confirm → settled im Emulator (~4 GiB Host)** | Häufig **System UI ANR**; Taps bleiben hängen. Prefer **Wait** auf ANR-Dialog, echtes Gerät, oder Host mit mehr RAM. Kein zuverlässiges grünes E2E nur über GUI auf dem schwachen Emulator. |
| **Vollständige Wallet-Settlement-Assertion in der GUI** | Minimalvariante prüft **nicht** „Balance final / transfer_done“ in der App-Oberfläche. Proof bleibt CLI/e2e. |
| **Kamera / physischer QR-Scan** | Absichtlich nicht automatisiert; deep-link ist der Shortcut. Landing-QR-Pfade sind **HTTP**-Thema, nicht Android-GUI. |
| **Browser-Shop-Checkout-UI** | Kein Chromium/WebView-Drive; Pay-URI kommt per Template-`POST` + Intent. |
| **Volles ToS-Scroll / alle Dialoge** | Best-effort Accept-Tap; lange ToS oder unerwartete Sheets können hängen bleiben. |
| **Published F-Droid allein auf GOA** | Kann Exchange-Add / Spinner-Probleme zeigen → **fix-branch-APK** bauen; bei Spinner-Hang optional `fix-withdraw-spinner-fallback`. |
| **iOS-GUI** | Nicht in diesem Tree; siehe taler-ios `gui-workflows`. |
| **macOS dual AVD / Homebrew-Helpers** | Upstream `gui-workflows`; hier nur Linux-taugliche Port-Ideen. |
| **`AUTO_*` in jedem Script erzwungen** | Design + teilweise verdrahtet; Wrapper-Checkliste noch offen. |
| **`run-until-done` / CLI-Wallet in GUI-Skripten** | Gehört zu CLI-Automation, nicht GUI; siehe `CLI-AUTOMATION-NOTES.md`. |
| **CI grün = „User hat bezahlt“** | Smoke = Intent + best-effort UI. Wirtschaftlicher Erfolg = Host-e2e / ladder / Explorer. |
#### Erwartetes Ergebnis pro Schicht
| Schicht | Erfolgskriterium Minimalvariante |
|---------|----------------------------------|
| A Install | APK installed, App startet |
| B Deep-link | Intent delivered, Wallet öffnet Withdraw/Pay-Flow |
| C GUI taps | Dump+Tap ohne Crash; ANR ggf. mit Wait; **kein** Garant für „Paid“ |
| D Evidence | Artefakte geschrieben |
| Settlement | **Außerhalb** GUI: Host e2e/ladder |
#### Wann Minimalvariante reicht
- Smoke nach APK-/Branch-Build (Intent kommt an, App crasht nicht).
- Manuelle Nacharbeit am Gerät nach vorbereiteten URIs.
- GOA/stage **Erreichbarkeit** von Landing/Template + Wallet-Einstieg.
#### Wann nicht reicht → nächster Schritt
- Unattended grünes Pay-E2E auf dem Emulator → mehr RAM / physisches Gerät, ggf. `fix-withdraw-spinner-fallback`.
- Beweis „Münzen da / Merchant paid“ → CLI e2e/ladder, nicht GUI-smoke.
- iOS / Dual-Platform → andere Repos/Flags, nicht diese Skripte.
```bash
cd ~/src/taler-monitoring/android-test
# Minimal hybrid (install + deep-link + light taps)
STACK=goa ./run-android-pay-smoke.sh
# GUI multi-round taps
STACK=goa ./run-android-gui-smoke.sh
STACK=stage ./run-android-gui-smoke.sh
# Explorer mint + multi withdraw/pay chain (from gui-workflows)
EXP_PW_FILE=$HOME/src/koopa/koopa-admin-secrets/koopa/host-root/taler-bank/bank-explorer-password.txt \
STACK=goa PKG=net.taler.wallet.fdroid.debug \
./run-goa-gui-chain.sh
# Build fix-branch APK then smoke (GUI=1 for gui smoke)
./run-android-build-and-smoke.sh
GUI=1 ./run-android-build-and-smoke.sh
```
---
## Platform capability flags (planned + defaults)
Automation can run on hosts that support **Android only**, **GUI tooling only**
(conceptual), or **both**. Flags keep that explicit for CI and workstations.
### Proposed env flags
| Variable | Values | Meaning |
|----------|--------|---------|
| `AUTO_ANDROID` | `0` / `1` | Run Android wallet automation (adb, APK, emulator/device) |
| `AUTO_GUI` | `0` / `1` | Drive **graphical** UI (uiautomator taps, multi-round). If `0`, deep-link/CLI-only install smoke. |
| `AUTO_PLATFORM` | `auto` / `linux` / `macos` / `ios` | Host family (optional override) |
| `EMULATOR_HEADLESS` | **`1` (default)** / `0` | No host window (`-no-window`). Server/CI-safe. |
| `WINDOWED` | `0` / `1` | Shortcut: `WINDOWED=1` ⇒ show emulator window (`EMULATOR_HEADLESS=0`) |
| `EMULATOR_GPU` | default **`swiftshader_indirect`** | Guest GLES still runs headless (layout/taps/screenshots). Override `host` only if windowed + real GPU. |
| `AUTO_START_EMULATOR` | **`1` (default)** / `0` | If no adb device, start AVD via `start-android-emulator.sh --wait` |
---
## Headless mode (default)
**Default for all GUI / pay smokes:** no window on the host. Opening an emulator
GUI on servers (or headless CI) is wrong; these scripts therefore start the AVD
with **`-no-window`** and still keep **graphics logic** in the guest:
| Piece | Headless default |
|-------|------------------|
| Host window | **off** (`-no-window`, `QT_QPA_PLATFORM=offscreen`, no `DISPLAY`) |
| Guest GLES | **on** via **SwiftShader** (`-gpu swiftshader_indirect`) |
| UI automation | **on** — `uiautomator` dump/tap + `screencap` (no X11 needed) |
| Audio / boot anim | off (`-no-audio -no-boot-anim`) |
```bash
# explicit (same as default)
./start-android-emulator.sh --wait
# or let smoke auto-start headless AVD when no phone is plugged in
STACK=goa ./run-android-gui-smoke.sh
# ops workstation: show the emulator window
WINDOWED=1 ./start-android-emulator.sh --wait
# or: EMULATOR_HEADLESS=0 EMULATOR_GPU=host ./start-android-emulator.sh --wait
```
Shared code: `lib_android_env.sh` + `start-android-emulator.sh`.
**Semantics:**
| `AUTO_ANDROID` | `AUTO_GUI` | Behaviour |
|----------------|------------|-----------|
| `1` | `0` | Android install + deep-link smoke only (no multi-round taps) |
| `1` | `1` | Android + graphical drive (vanilla GUI / chain) |
| `0` | `1` | Reserved (e.g. future desktop/web GUI); currently **no-op** with a clear message |
| `0` | `0` | Skip mobile automation |
### Defaults by host (when flags unset)
| Host (`uname -s`) | Default `AUTO_ANDROID` | Default `AUTO_GUI` | Rationale |
|-------------------|------------------------|--------------------|-----------|
| **Linux** | **`1`** | **`1`** if device/emulator present, else scripts exit 3 | This repo’s day-to-day path; **Android only** (no iOS here) |
| **Darwin (macOS)** | `1` | `1` | Can run Android emulators **and** (separately) taler-ios GUI helpers; both flags on for Android scripts; iOS is out of tree |
| **Other** | `0` | `0` | Fail closed |
**Linux default = Android** (no second mobile platform in this suite).
**macOS** may enable both ecosystems in the wider monorepo sense; for *these*
scripts only Android is implemented — set `AUTO_ANDROID=1` (default) and keep
iOS under `taler-ios`.
### Resolution helper (convention for future wrappers)
```bash
# Example for a future run-all-mobile.sh
os=$(uname -s)
: "${AUTO_PLATFORM:=auto}"
case "$AUTO_PLATFORM" in
auto) case "$os" in Linux) AUTO_PLATFORM=linux ;; Darwin) AUTO_PLATFORM=macos ;; *) AUTO_PLATFORM=other ;; esac ;;
esac
case "$AUTO_PLATFORM" in
linux)
: "${AUTO_ANDROID:=1}"
: "${AUTO_GUI:=1}"
# no iOS
;;
macos)
: "${AUTO_ANDROID:=1}"
: "${AUTO_GUI:=1}"
# optional later: AUTO_IOS=1 for taler-ios scripts
;;
*)
: "${AUTO_ANDROID:=0}"
: "${AUTO_GUI:=0}"
;;
esac
```
Scripts **today** implement Android only; they should honour:
- `AUTO_ANDROID=0` → exit 0 with “skipped (AUTO_ANDROID=0)”
- `AUTO_GUI=0` → call deep-link smoke without multi-round GUI (or set `GUI_ROUNDS=0`)
---
## Vanilla layers (detail)
| Layer | What | Tooling |
|-------|------|---------|
| **A — Install** | F-Droid or from-source APK | `adb install` |
| **B — Entry (shortcut)** | `taler://withdraw/…` / `taler://pay…` | `adb am start -a VIEW` |
| **C — Graphical UI** | Confirm / ToS / Pay taps; ANR Wait | `uiautomator` + `lib_ui.py` / chain |
| **D — Evidence** | Screenshots, XML, logcat, URIs | `out*/` |
### Legitimate shortcuts
| Shortcut | Replaces |
|----------|----------|
| Deep-link withdraw/pay | Camera QR / opening paywall |
| Host `POST` template → pay URI | Browser shop checkout UI |
| Label-list taps | Human reading button text |
| Skip full ToS scroll | Tap Accept if shown |
### Scripts
| Script | Role |
|--------|------|
| `run-android-pay-smoke.sh` | Hybrid: install + deep-link + light taps |
| `run-android-gui-smoke.sh` | Multi-round GUI drive |
| `run-goa-gui-chain.sh` | Multi mint/withdraw/pay (gui-workflows port) |
| `run-android-build-and-smoke.sh` | Build git ref (branch/tag) + smoke (`GUI=1` optional) |
| `run-android-variant-matrix.sh` | **published + stable-self + master** (compare builds) |
| `start-android-emulator.sh` | Start AVD (**headless default**, SwiftShader) |
| `lib_android_env.sh` | Flags, PATH, `android_ensure_device` |
| `lib_ui.py` | Dump/tap/ANR helpers |
### Build variants (anti–“weird local build”)
| Variant | Source | Severity | Purpose |
|---------|--------|----------|---------|
| **stable-self** | **only** rebuild of current stable tag | WARN | Self-build vs published for **one** release |
| **published** | F-Droid current APK | **≤14d → BLOCKER**, else WARN | What users install |
| **older / milestones** | F-Droid APK ~**3 / 6 / 9 / 12 mo** | **always WARN** | No self-build of old tags |
| **master** | `origin/master` rebuild | WARN | Tip — never hard-blocks |
| **fix** (optional) | inference branch | soft | GOA automation fixes |
**Self-build:** current stable only (plus master/fix if requested) — not older releases.
**Blocker window:** releases ≤ 14 days.
**Milestones:** nearest tags to 90/180/270/365d with a still-hosted F-Droid APK.
---
## Stacks under test
| Stack | Withdraw | Pay |
|-------|----------|-----|
| **GOA** | explorer mint / `demo-withdraw.json` @ bank.hacktivism.ch | goa-shop templates / Paivana pay-template |
| **stage** | explorer / demo-withdraw @ stage.bank… | fermes / jardin public templates |
### Lageübersicht GOA + stage (as-is survey · 2026-07-17 ~22:05–22:18)
**Device:** headless `emulator-5554` · host ~3.7 GiB RAM (often &lt;300 MiB free) · timeout per leg ≤420 s.
**Legs:** Abheben + Bezahlen (`DO_WITHDRAW=1 DO_PAY=1`). Artefakte: `out-survey-20260717-220527/` (gitignored).
#### Varianten
| Variante | Quelle | Bemerkung |
|----------|--------|-----------|
| **stable (published)** | F-Droid `net.taler.wallet.fdroid` **1.6.1 / 854** | as-is |
| **master as-is** | rebuild `origin/master` @ `f65a976c` → `.fdroid.debug` | as-is, no patches |
| **ältere Releases** (3/6/9/12 mon) | F-Droid APKs für `1.5.0`/`1.3.0`/`1.1.0`/`1.0.8` | **nicht auffindbar** (HTTP miss) → **SKIP** (kein Selbstbau älterer Tags) |
| **master + inference** (extra) | local branch `local/master-with-inference` @ `3ebe8b09` | merges `fix-bank-withdraw-auto-exchange` + `fix-withdraw-spinner-fallback` · **not pushed** |
#### Ergebnis-Matrix
| Variant | Stack | Install / deep-link | Withdraw URI | Pay URI | wallet-core Abheben | wallet-core Pay | ANR | Exit |
|---------|-------|---------------------|--------------|---------|---------------------|-----------------|-----|------|
| stable published | **goa** (hacktivism) | OK | OK bank.hacktivism.ch | OK paivana template | **NO** | soft/YES once* | **YES** | **10** |
| stable published | **stage** (*.lefrancpaysan) | OK | OK stage.bank… | OK fermes order | **NO** | **NO** | **YES** | **10** |
| master as-is | **goa** | OK | OK | OK paivana | **NO** | **NO** | **YES** | **10** |
| master as-is | **stage** | OK | OK | OK fermes | **NO** | **NO** | **YES** | **10** |
| master+inference | **goa** | OK | OK | OK | **NO** | **NO** | **YES** | **10** |
| master+inference | **stage** | OK | OK | OK | **NO** | **NO** | **YES** | **10** |
\*stable-goa console once reported pay wallet-core YES mid-run; final score still ANR-dominated / not reliable settlement.
#### Was pro Stack **steht** (Infra)
| Check | **goa / hacktivism** | **stage / \*.lefrancpaysan** |
|-------|----------------------|------------------------------|
| Bank `…/intro/demo-withdraw.json` → `taler://withdraw/…` | **OK** | **OK** |
| Pay entry | paivana `pay-template` **OK** | fermes `panier-legumes` POST → `taler://pay/…` **OK** |
| adb `VIEW` Intent delivery | **OK** | **OK** |
| Unattended Confirm/Abheben/Pay GUI | **FAIL (ANR)** | **FAIL (ANR)** |
| wallet-core prepare/accept withdraw in logcat | **not seen** | **not seen** |
**Fazit as-is:** Beide Stacks liefern die **Eingänge** (Withdraw- + Pay-URIs). Die **GUI/wallet-core-Strecke** ist auf diesem Emulator für **alle** getesteten App-Varianten (stable, master, inference) **gleich tot** — System UI / Android System ANR, keine brauchbaren Taps. Das ist **kein Stack-Unterschied** GOA vs stage und **kein** belastbarer App-Versions-Vergleich.
#### Inference-Patch-Versuch (lokal, **kein Push**)
Weil **nichts** der as-is-Varianten wallet-core-grün war, wurde zusätzlich gebaut:
```text
taler-android branch: local/master-with-inference # NOT pushed
base: origin/master
merged: origin/dev/hernani-inference/fix-bank-withdraw-auto-exchange
origin/dev/hernani-inference/fix-withdraw-spinner-fallback
commit: 3ebe8b091379
APK: apks/wallet-master-inference-3ebe8b091379.apk (local)
```
| Stack | Inference smoke | Besser als master/stable? |
|-------|-----------------|---------------------------|
| goa | exit 10, ANR, no withdraw core | **Nein** |
| stage | exit 10, ANR, no withdraw core | **Nein** |
**Kein grünes Inference-Signal** auf diesem Host → wir können **nicht** belegen, dass die Patches Withdraw/Pay „retten“.
Ebenso können wir **nicht** belegen, dass master ohne Patches „schuld“ ist: die Umgebung erreicht wallet-core gar nicht.
> **Wenn später auf Gerät/mehr RAM die Inference-APK plötzlich grün und master/stable rot ist:**
> das ist der **klare Fix-Hinweis** — Patches aus `dev/hernani-inference/*` upstreamen / in master holen.
> **Heute:** noch **nicht** der Fall; primärer Blocker = **Emulator/RAM/ANR**.
#### Nächste sinnvolle Schritte (später)
1. Physisches Gerät oder Host ≥6–8 GiB free RAM
2. Nur **eine** Wallet-Package-ID installiert (kein Open-with)
3. Survey wiederholen: stable · master · `local/master-with-inference` auf **goa + stage**
4. Ältere F-Droid-APKs nur wenn wieder gehostet; kein Selbstbau alter Tags
---
## Upstream branches (`taler-android` `dev/hernani-inference/*`)
| Branch | Role |
|--------|------|
| `gui-workflows` | macOS emulator helpers + original `goa-chain-emu.sh` |
| `fix/bank-withdraw-auto-exchange` | **Minimal** GOA exchange auto-add (default build branch) |
| `fix/withdraw-spinner-fallback` | Spinner + OIM; use if spinner still hangs |
| `experimental-oim*` | Optional cash UI |
---
## Checklist
- [x] Minimal variant documented
- [x] Minimalvariante: was geht / was nicht (Beschränkungen)
- [x] Two stacks (goa / stage)
- [x] Dual-stack run documented (2026-07-17: both FAIL ANR, landings OK)
- [x] Flags design (`AUTO_ANDROID` / `AUTO_GUI` / platform defaults)
- [x] **Headless emulator default** (`EMULATOR_HEADLESS=1`, SwiftShader)
- [x] Linux default = Android suite only
- [x] Inference-branch policy
- [ ] Wrapper enforces flags in every entry script (incremental)
- [ ] Reliable unattended Confirm on low-RAM Linux emulator

111
android-test/README.md Normal file
View file

@ -0,0 +1,111 @@
# Android wallet smoke (Abheben + Bezahlen)
Smokes always cover **withdraw first**, then **pay** (not pay-only).
| Leg | Default | What |
|-----|---------|------|
| **1 WITHDRAW** | on | `taler://withdraw/…` + Confirm/Abheben taps |
| **2 PAY** | on | `taler://pay…` / pay-template + Pay taps |
```bash
# needs adb device/emulator
./run-android-pay-smoke.sh
STACK=goa ./run-android-pay-smoke.sh
DO_PAY=0 ./run-android-pay-smoke.sh # withdraw only
DO_WITHDRAW=0 DO_PAY=1 … # pay only (unusual)
```
APK: F-Droid **1.6.1** (`net.taler.wallet.fdroid`).
## 2) Source build + same tests
Builds `taler-android` at a git **ref** (`:wallet:assembleFdroidDebug`), then
runs the **same** smoke (install + withdraw/pay deep-links).
```bash
# default: inference fix branch (GOA withdraw reliability)
./run-android-build-and-smoke.sh
# tip of master
BRANCH=master ./run-android-build-and-smoke.sh
# self-build of published stable **tag** (same release as F-Droid 1.6.1)
BRANCH=wallet-1.6.1 LABEL=stable-self ./run-android-build-and-smoke.sh
SKIP_PULL=1 BRANCH=master ./run-android-build-and-smoke.sh
TALER_ANDROID_SRC=/path/to/taler-android ./run-android-build-and-smoke.sh
```
Built package id: `net.taler.wallet.fdroid.debug`.
Needs `ANDROID_HOME` (e.g. `$HOME/Android/Sdk`), Java 17+, network for Maven.
## 2b) Variant matrix (published + stable self-build + master)
Runs several smokes so a single odd developer build cannot be the only sample:
| Variant | What |
|---------|------|
| Order | Variant | Severity | What |
|------|---------|----------|------|
| **1** | **stable-self** | WARN | **Only** self-build: current stable tag |
| **2** | **published** | age ≤14d → **BLOCKER**, else WARN | Current F-Droid APK |
| **3** | **older / milestones** | **always WARN** | F-Droid APKs nearest **3 / 6 / 9 / 12 months** (if still hosted) — **no self-build** |
| **4** | **master** | WARN | Rebuild `origin/master` |
**Self-build policy:** only **current published stable** (`stable-self`) + optional **master**.
Historical versions: published F-Droid APK only; skip if APK gone.
**Milestones:** `MILESTONE_DAYS="90 180 270 365"` (±50d), warn only.
**Young releases** (≤ 14 days): hard blocker on fail.
```bash
./run-android-variant-matrix.sh
STACK=goa ./run-android-variant-matrix.sh
INCLUDE_OLDER=0 VARIANTS=stable-self,published,master ./run-android-variant-matrix.sh
# custom milestones (days):
MILESTONE_DAYS="90 180 270 365" ./run-android-variant-matrix.sh
```
Artifacts + `SUMMARY.txt` under `out-matrix-<timestamp>/`.
## 3) Graphical UI (vanilla)
Deep-link entry (≡ QR) **plus** multi-round uiautomator taps:
```bash
STACK=goa ./run-android-gui-smoke.sh
STACK=stage ./run-android-gui-smoke.sh
# after source build:
GUI=1 ./run-android-build-and-smoke.sh
```
## 4) Full GOA/stage chain (from `dev/hernani-inference/gui-workflows`)
Bank API mint → withdraw deep-link → taps → template pay → taps:
```bash
# needs explorer password + device
EXP_PW_FILE=$HOME/src/koopa/koopa-admin-secrets/koopa/host-root/taler-bank/bank-explorer-password.txt \
STACK=goa PKG=net.taler.wallet.fdroid.debug \
./run-goa-gui-chain.sh
STACK=stage ./run-goa-gui-chain.sh
```
Canonical notes (minimal variant, **flags**, Linux defaults):
**[GUI-AUTOMATION-NOTES.md](./GUI-AUTOMATION-NOTES.md)**.
| Flag | Linux default | Meaning |
|------|---------------|---------|
| `AUTO_ANDROID` | `1` | Run Android adb/APK automation |
| `AUTO_GUI` | `1` | Graphical uiautomator drive (else deep-link only) |
| `AUTO_PLATFORM` | `auto` → `linux` | Host family; macOS can later pair with iOS out of tree |
| **`EMULATOR_HEADLESS`** | **`1`** | **No host window** (`-no-window`). Server-safe. |
| `EMULATOR_GPU` | `swiftshader_indirect` | Guest GLES still active for UI/taps/screenshots |
| `AUTO_START_EMULATOR` | `1` | Auto-start headless AVD if no adb device |
| `WINDOWED` | `0` | `WINDOWED=1` → show emulator window |
```bash
# headless AVD only (default path for servers)
./start-android-emulator.sh --wait
# windowed (workstations with display)
WINDOWED=1 ./start-android-emulator.sh --wait
```
See also parent [VERSIONS.md](../VERSIONS.md).

158
android-test/lib_android_env.sh Normal file
View file

@ -0,0 +1,158 @@
# shellcheck shell=bash
# Shared Android env for GUI / adb smokes.
# Source: . "$ROOT/lib_android_env.sh"
#
# Defaults favour **servers / CI**: no host window (headless emulator), while the
# guest still runs GLES via SwiftShader so UI layout / uiautomator work.
#
# See GUI-AUTOMATION-NOTES.md → Headless mode
# --- SDK / PATH ---------------------------------------------------------------
export ANDROID_HOME="${ANDROID_HOME:-${ANDROID_SDK_ROOT:-$HOME/Android/Sdk}}"
export ANDROID_SDK_ROOT="${ANDROID_SDK_ROOT:-$ANDROID_HOME}"
for d in \
"$ANDROID_HOME/platform-tools" \
"$ANDROID_HOME/emulator" \
"$ANDROID_HOME/cmdline-tools/latest/bin"
do
[ -d "$d" ] && case ":$PATH:" in *":$d:"*) ;; *) PATH="$d:$PATH" ;; esac
done
export PATH
# AVD path (some hosts keep AVDs under ~/.config/.android)
if [ -z "${ANDROID_AVD_HOME:-}" ]; then
if [ -d "$HOME/.android/avd" ]; then
export ANDROID_AVD_HOME="$HOME/.android/avd"
elif [ -d "$HOME/.config/.android/avd" ]; then
export ANDROID_AVD_HOME="$HOME/.config/.android/avd"
# emulator also looks at ~/.android/avd — soft-link if missing
if [ ! -e "$HOME/.android/avd" ]; then
mkdir -p "$HOME/.android"
ln -sfn "$HOME/.config/.android/avd" "$HOME/.android/avd" 2>/dev/null || true
fi
fi
fi
# --- Platform / feature flags -------------------------------------------------
os=$(uname -s)
case "${AUTO_PLATFORM:-auto}" in
auto)
case "$os" in
Linux) : "${AUTO_ANDROID:=1}"; : "${AUTO_GUI:=1}" ;;
Darwin) : "${AUTO_ANDROID:=1}"; : "${AUTO_GUI:=1}" ;;
*) : "${AUTO_ANDROID:=0}"; : "${AUTO_GUI:=0}" ;;
esac
;;
linux|macos) : "${AUTO_ANDROID:=1}"; : "${AUTO_GUI:=1}" ;;
*) : "${AUTO_ANDROID:=0}"; : "${AUTO_GUI:=0}" ;;
esac
# Headless = default (no X11/Wayland window on host). GUI *logic* still runs in
# the guest via software GPU. Set EMULATOR_HEADLESS=0 or WINDOWED=1 for a window.
: "${EMULATOR_HEADLESS:=1}"
if [ "${WINDOWED:-0}" = "1" ]; then
EMULATOR_HEADLESS=0
fi
export EMULATOR_HEADLESS
# Soft GLES for headless; windowed may override to host/auto if set by user.
if [ "${EMULATOR_HEADLESS}" = "1" ]; then
: "${EMULATOR_GPU:=swiftshader_indirect}"
else
: "${EMULATOR_GPU:=swiftshader_indirect}" # still safe default; host GPU: EMULATOR_GPU=host
fi
export EMULATOR_GPU
: "${EMULATOR_AVD:=TalerWallet34}"
: "${EMULATOR_MEMORY:=1536}"
: "${EMULATOR_CORES:=2}"
# If no adb device: start headless AVD (0 = require phone/manual emulator)
: "${AUTO_START_EMULATOR:=1}"
: "${EMULATOR_BOOT_TIMEOUT:=360}"
: "${EMULATOR_LOG:=/tmp/taler-android-emulator.log}"
: "${EMULATOR_NO_SNAPSHOT:=1}"
android_adb_serial() {
adb devices 2>/dev/null | awk '/\tdevice$/{print $1; exit}'
}
android_has_device() {
[ -n "$(android_adb_serial)" ]
}
# Build emulator argv for current headless/windowed mode.
# Prints args; caller exec/nohup.
android_emulator_args() {
local avd="${1:-$EMULATOR_AVD}"
local args=( -avd "$avd" -gpu "$EMULATOR_GPU" -no-audio -no-boot-anim )
if [ "${EMULATOR_HEADLESS}" = "1" ]; then
args+=( -no-window )
# Avoid Qt trying to open a display on servers
export QT_QPA_PLATFORM="${QT_QPA_PLATFORM:-offscreen}"
fi
# Broken default_boot snapshots leave adb offline forever — cold boot by default.
if [ "${EMULATOR_NO_SNAPSHOT:-1}" = "1" ]; then
args+=( -no-snapshot -no-snapshot-load -no-snapshot-save )
fi
if [ -n "${EMULATOR_MEMORY:-}" ]; then
args+=( -memory "$EMULATOR_MEMORY" )
fi
if [ -n "${EMULATOR_CORES:-}" ]; then
args+=( -cores "$EMULATOR_CORES" )
fi
# extra user flags
# shellcheck disable=SC2206
if [ -n "${EMULATOR_EXTRA_ARGS:-}" ]; then
# intentional word-split for extra flags
args+=( ${EMULATOR_EXTRA_ARGS} )
fi
printf '%s\n' "${args[@]}"
}
# Ensure an adb "device" is online. Optionally start headless emulator.
# Sets SERIAL / ANDROID_SERIAL on success. Returns 0 ok, 3 no device.
android_ensure_device() {
local serial
serial="${SERIAL:-${ANDROID_SERIAL:-}}"
if [ -z "$serial" ]; then
serial=$(android_adb_serial || true)
fi
if [ -n "$serial" ]; then
SERIAL="$serial"
export ANDROID_SERIAL="$serial"
return 0
fi
if [ "${AUTO_START_EMULATOR}" != "1" ]; then
echo "no adb device (AUTO_START_EMULATOR=0 — plug phone or start AVD manually)" >&2
return 3
fi
if ! command -v emulator >/dev/null 2>&1; then
echo "no adb device and emulator binary missing under $ANDROID_HOME" >&2
return 3
fi
echo "no adb device → starting AVD ${EMULATOR_AVD} (headless=${EMULATOR_HEADLESS} gpu=${EMULATOR_GPU})"
local starter
starter="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/start-android-emulator.sh"
if [ ! -x "$starter" ]; then
echo "missing $starter" >&2
return 3
fi
EMULATOR_HEADLESS="$EMULATOR_HEADLESS" EMULATOR_GPU="$EMULATOR_GPU" \
EMULATOR_AVD="$EMULATOR_AVD" EMULATOR_MEMORY="$EMULATOR_MEMORY" \
EMULATOR_CORES="$EMULATOR_CORES" EMULATOR_LOG="$EMULATOR_LOG" \
EMULATOR_BOOT_TIMEOUT="$EMULATOR_BOOT_TIMEOUT" \
"$starter" --wait || return 3
serial=$(android_adb_serial || true)
if [ -z "$serial" ]; then
echo "emulator started but no adb device yet — see $EMULATOR_LOG" >&2
return 3
fi
SERIAL="$serial"
export ANDROID_SERIAL="$serial"
echo "device: $SERIAL (headless=${EMULATOR_HEADLESS})"
return 0
}

133
android-test/lib_release_age.sh Normal file
View file

@ -0,0 +1,133 @@
# shellcheck shell=bash
# Release age helpers for Android wallet matrix.
# Source after ROOT/lib_android_env.sh (optional).
#
# BLOCKER only when a *released* app is within BLOCKER_MAX_AGE_DAYS (default 14).
# Older releases: try them, but failures are WARN only.
: "${BLOCKER_MAX_AGE_DAYS:=14}"
: "${TALER_ANDROID_SRC:=$HOME/taler/taler-android}"
# Days since git tag (tagger/committer date). Empty if unknown.
# Usage: release_age_days wallet-1.6.1 → e.g. 28
release_age_days() {
local tag="$1"
local src="${2:-$TALER_ANDROID_SRC}"
[ -d "$src/.git" ] || return 1
local epoch
epoch=$(git -C "$src" log -1 --format=%ct "refs/tags/${tag}" 2>/dev/null || true)
[ -n "$epoch" ] || epoch=$(git -C "$src" log -1 --format=%ct "$tag" 2>/dev/null || true)
[ -n "$epoch" ] || return 1
local now
now=$(date +%s)
echo $(( (now - epoch) / 86400 ))
}
# severity from age: blocker if age_days <= BLOCKER_MAX_AGE_DAYS, else warn
# Unknown age → warn (never accidental hard fail on mystery builds)
severity_from_age_days() {
local days="${1:-}"
if [ -z "$days" ] || ! [[ "$days" =~ ^[0-9]+$ ]]; then
echo warn
return
fi
if [ "$days" -le "${BLOCKER_MAX_AGE_DAYS}" ]; then
echo blocker
else
echo warn
fi
}
severity_for_tag() {
local tag="$1"
local days
days=$(release_age_days "$tag" 2>/dev/null || true)
severity_from_age_days "$days"
}
# List wallet-* release tags newest first (exclude rc/dev).
# Args: max count (default 5). Pass 0 or "all" for no limit.
list_wallet_release_tags() {
local n="${1:-5}"
local src="${TALER_ANDROID_SRC:-$HOME/taler/taler-android}"
[ -d "$src/.git" ] || return 1
git -C "$src" fetch --tags origin 2>/dev/null || true
local cmd=(git -C "$src" for-each-ref --sort=-creatordate --format='%(refname:short)' refs/tags)
if [ "$n" = "all" ] || [ "$n" = "0" ]; then
"${cmd[@]}" | grep -E '^wallet-[0-9]+\.[0-9]+(\.[0-9]+)?$'
else
"${cmd[@]}" | grep -E '^wallet-[0-9]+\.[0-9]+(\.[0-9]+)?$' | head -n "$n"
fi
}
# Pick one tag closest to each age milestone (days). Always intended as WARN samples.
# Default milestones: 3,6,9,12 months ≈ 90,180,270,365 days.
# Env: MILESTONE_DAYS="90 180 270 365" MILESTONE_TOLERANCE_DAYS=50
# Prints lines: TAG AGE_DAYS TARGET_DAYS (only when a tag within tolerance exists)
select_milestone_tags() {
local src="${TALER_ANDROID_SRC:-$HOME/taler/taler-android}"
local milestones="${MILESTONE_DAYS:-90 180 270 365}"
local tol="${MILESTONE_TOLERANCE_DAYS:-50}"
local skip="${1:-}" # optional tag to exclude (e.g. current STABLE_TAG)
local -a tags=()
mapfile -t tags < <(list_wallet_release_tags all 2>/dev/null || true)
[ "${#tags[@]}" -gt 0 ] || return 0
local target best_tag best_age best_delta age t delta
local -A used=()
for target in $milestones; do
best_tag=""
best_age=""
best_delta=99999
for t in "${tags[@]}"; do
[ -n "$t" ] || continue
[ "$t" = "$skip" ] && continue
[ -n "${used[$t]:-}" ] && continue
age=$(release_age_days "$t" "$src" 2>/dev/null || true)
[[ "$age" =~ ^[0-9]+$ ]] || continue
delta=$(( age > target ? age - target : target - age ))
if [ "$delta" -le "$tol" ] && [ "$delta" -lt "$best_delta" ]; then
best_delta=$delta
best_tag=$t
best_age=$age
fi
done
if [ -n "$best_tag" ]; then
used[$best_tag]=1
echo "$best_tag $best_age $target"
fi
done
}
# Guess F-Droid APK URL for a wallet version name (e.g. 1.6.1).
# Uses versionCode from tag's build.gradle if possible, else STABLE mapping.
# Prints: versionCode apk_url or returns 1
fdroid_apk_for_version_name() {
local vn="$1" # 1.6.1
local src="${TALER_ANDROID_SRC:-$HOME/taler/taler-android}"
local tag="wallet-${vn}"
local vc=""
if git -C "$src" cat-file -e "refs/tags/${tag}:wallet/build.gradle" 2>/dev/null; then
vc=$(git -C "$src" show "refs/tags/${tag}:wallet/build.gradle" 2>/dev/null \
| sed -n 's/^[[:space:]]*versionCode[[:space:]]\+\([0-9]\+\).*/\1/p' | head -1)
fi
# ABI x86_64 → +4 in override scheme 10*vc+abi (common F-Droid multi-apk)
# Prefer universal-style code used by F-Droid: for 1.6.1 → 854 = 10*85+4
local apk_vc=""
if [ -n "$vc" ]; then
apk_vc=$((10 * vc + 4))
fi
# Known overrides / suggested
case "$vn" in
1.6.1) apk_vc=854 ;;
esac
[ -n "$apk_vc" ] || return 1
echo "${apk_vc}|https://f-droid.org/repo/net.taler.wallet.fdroid_${apk_vc}.apk"
}
# Check HTTP HEAD/GET that APK exists (0 = ok)
fdroid_apk_exists() {
local url="$1"
curl -sfS -m 15 -o /dev/null -r 0-0 "$url" 2>/dev/null \
|| curl -sfS -m 15 -o /dev/null -I "$url" 2>/dev/null
}

192
android-test/lib_ui.py Executable file
View file

@ -0,0 +1,192 @@
#!/usr/bin/env python3
"""Minimal adb + uiautomator helpers for Taler wallet GUI smoke (no root).
Does not replace Maestro/Appium; uses only adb shell + uiautomator dump.
"""
from __future__ import annotations
import re
import subprocess
import time
from pathlib import Path
from typing import Iterable, List, Optional, Sequence, Tuple
Bounds = Tuple[int, int, int, int]
def adb(serial: str, *args: str, check: bool = False, timeout: Optional[float] = 60) -> subprocess.CompletedProcess:
cmd = ["adb", "-s", serial, *args]
return subprocess.run(cmd, check=check, capture_output=True, text=True, timeout=timeout)
def dump_ui(serial: str, dest: Path) -> str:
adb(serial, "shell", "uiautomator", "dump", "/sdcard/ui.xml", timeout=90)
adb(serial, "pull", "/sdcard/ui.xml", str(dest), timeout=30)
return dest.read_text(errors="replace") if dest.is_file() else ""
def screencap(serial: str, dest: Path) -> None:
p = subprocess.run(
["adb", "-s", serial, "exec-out", "screencap", "-p"],
capture_output=True,
timeout=60,
)
if p.returncode == 0 and p.stdout:
dest.write_bytes(p.stdout)
def texts(xml: str) -> List[str]:
return [t for t in re.findall(r'text="([^"]*)"', xml) if t.strip()]
def find_bounds(xml: str, label: str) -> List[Bounds]:
"""Return list of (x1,y1,x2,y2) for nodes whose text or content-desc matches label (exact)."""
out: List[Bounds] = []
esc = re.escape(label)
patterns = [
rf'(?:text|content-desc)="{esc}"[^>]*bounds="\[(\d+),(\d+)\]\[(\d+),(\d+)\]"',
rf'bounds="\[(\d+),(\d+)\]\[(\d+),(\d+)\]"[^>]*(?:text|content-desc)="{esc}"',
]
for pat in patterns:
for m in re.finditer(pat, xml, flags=re.I):
out.append(tuple(map(int, m.groups()))) # type: ignore[arg-type]
return out
def find_bounds_re(xml: str, pattern: str) -> List[Tuple[str, Bounds]]:
out: List[Tuple[str, Bounds]] = []
for m in re.finditer(
rf'(?:text|content-desc)="([^"]*{pattern}[^"]*)"[^>]*bounds="\[(\d+),(\d+)\]\[(\d+),(\d+)\]"',
xml,
flags=re.I,
):
label = m.group(1)
b = tuple(map(int, m.group(2, 3, 4, 5)))
out.append((label, b)) # type: ignore[arg-type]
return out
def center(b: Bounds) -> Tuple[int, int]:
x1, y1, x2, y2 = b
return (x1 + x2) // 2, (y1 + y2) // 2
def tap(serial: str, x: int, y: int) -> None:
adb(serial, "shell", "input", "tap", str(x), str(y))
def tap_bounds(serial: str, b: Bounds) -> None:
x, y = center(b)
tap(serial, x, y)
def dismiss_anr(serial: str, xml: str) -> bool:
"""If System UI isn't responding, prefer Wait over Close."""
if "isn't responding" not in xml and "reagiert nicht" not in xml.lower():
return False
for label in ("Wait", "Warten", "OK"):
bs = find_bounds(xml, label)
if bs:
tap_bounds(serial, bs[0])
return True
# fallback: right-ish button often Wait
adb(serial, "shell", "input", "keyevent", "22")
adb(serial, "shell", "input", "keyevent", "66")
return True
# Labels seen on wallet UI (EN/DE/FR) — expand as needed
CONFIRM_LABELS = [
"Confirm",
"Confirm withdrawal",
"Withdraw",
"Accept",
"I accept",
"Agree",
"Continue",
"Next",
"OK",
"Pay",
"Pay now",
"Bestätigen",
"Abheben",
"Akzeptieren",
"Zustimmen",
"Weiter",
"Bezahlen",
"Confirmer",
"Retirer",
"Accepter",
"Payer",
]
TOS_LABELS = [
"I accept",
"Accept",
"Agree",
"Akzeptieren",
"Zustimmen",
"Accepter",
]
def try_tap_any(serial: str, xml: str, labels: Sequence[str]) -> Optional[str]:
for label in labels:
bs = find_bounds(xml, label)
if bs:
tap_bounds(serial, bs[0])
return label
return None
def gui_drive(
serial: str,
out_dir: Path,
*,
rounds: int = 8,
sleep_s: float = 3.0,
phase: str = "gui",
) -> dict:
"""Poll UI, dismiss ANR, tap confirm/ToS-like buttons. Returns status dict."""
out_dir.mkdir(parents=True, exist_ok=True)
seen_texts: List[str] = []
taps: List[str] = []
anr = 0
for i in range(1, rounds + 1):
xml_path = out_dir / f"{phase}-{i:02d}-ui.xml"
png_path = out_dir / f"{phase}-{i:02d}.png"
try:
xml = dump_ui(serial, xml_path)
except Exception as e:
taps.append(f"dump-fail:{e}")
time.sleep(sleep_s)
continue
try:
screencap(serial, png_path)
except Exception:
pass
ts = texts(xml)
seen_texts.extend(ts[:20])
if dismiss_anr(serial, xml):
anr += 1
time.sleep(sleep_s)
continue
hit = try_tap_any(serial, xml, CONFIRM_LABELS)
if hit:
taps.append(hit)
time.sleep(sleep_s)
continue
# partial regex for amount confirm rows
for label, b in find_bounds_re(xml, r"Confirm|Withdraw|Pay|Accept|Bestätig|Abheben|Payer"):
tap_bounds(serial, b)
taps.append(f"re:{label}")
time.sleep(sleep_s)
break
else:
time.sleep(sleep_s)
return {
"taps": taps,
"anr_dismissals": anr,
"sample_texts": seen_texts[-40:],
"rounds": rounds,
}

167
android-test/run-android-build-and-smoke.sh Executable file
View file

@ -0,0 +1,167 @@
#!/usr/bin/env bash
# Build GNU Taler Android wallet from source at a git **ref**, then run the same
# adb deep-link (or GUI) smoke as run-android-pay-smoke.sh.
#
# Source (default): $HOME/taler/taler-android
# Build: ./gradlew :wallet:assembleFdroidDebug
# Package id: net.taler.wallet.fdroid.debug
#
# Typical refs:
# BRANCH=master # tip of master
# BRANCH=wallet-1.6.1 # published stable **tag** (self-build)
# BRANCH=dev/hernani-inference/fix-… # GOA automation fix branch
#
# For a multi-variant matrix (master + stable self-build + published APK):
# ./run-android-variant-matrix.sh
#
# Usage:
# ./run-android-build-and-smoke.sh
# BRANCH=master STACK=goa ./run-android-build-and-smoke.sh
# BRANCH=wallet-1.6.1 LABEL=stable-self ./run-android-build-and-smoke.sh
# SKIP_PULL=1 BRANCH=master ./run-android-build-and-smoke.sh
#
set -euo pipefail
ROOT=$(cd "$(dirname "$0")" && pwd)
SMOKE="$ROOT/run-android-pay-smoke.sh"
SRC="${TALER_ANDROID_SRC:-$HOME/taler/taler-android}"
SKIP_PULL="${SKIP_PULL:-0}"
# Prefer existing inference fix branch for GOA withdraw (spinner/exchange resolve).
# Matrix uses BRANCH=master and BRANCH=$STABLE_TAG explicitly.
BRANCH="${BRANCH:-dev/hernani-inference/fix-bank-withdraw-auto-exchange}"
# Optional human label for logs / artifact names (default = sanitized BRANCH)
LABEL="${LABEL:-}"
VARIANT="${VARIANT:-fdroidDebug}"
export ANDROID_HOME="${ANDROID_HOME:-$HOME/Android/Sdk}"
export ANDROID_SDK_ROOT="${ANDROID_SDK_ROOT:-$ANDROID_HOME}"
if [ ! -x "$SRC/gradlew" ]; then
echo "taler-android not found at $SRC (set TALER_ANDROID_SRC=…)" >&2
exit 2
fi
if [ ! -d "$ANDROID_HOME" ]; then
echo "ANDROID_HOME missing: $ANDROID_HOME" >&2
exit 2
fi
printf 'sdk.dir=%s\n' "$ANDROID_HOME" >"$SRC/local.properties"
# --- checkout REF (branch, origin/branch, or tag) --------------------------------
cd "$SRC"
if [ -d .git ]; then
git fetch origin --tags 2>/dev/null || git fetch origin 2>/dev/null || true
REF="$BRANCH"
checked=0
if [ -n "$REF" ]; then
if git rev-parse --verify "refs/tags/${REF}" >/dev/null 2>&1; then
echo "checkout tag ${REF} (stable / release self-build)"
git checkout -f "refs/tags/${REF}"
checked=1
SKIP_PULL=1
elif git rev-parse --verify "origin/${REF}" >/dev/null 2>&1; then
echo "checkout origin/${REF}"
git checkout -B "$REF" "origin/${REF}" 2>/dev/null || git checkout "$REF" 2>/dev/null || true
checked=1
elif git rev-parse --verify "${REF}" >/dev/null 2>&1; then
echo "checkout ${REF}"
git checkout -f "$REF"
checked=1
else
echo "WARN: ref '${REF}' not found locally after fetch — staying on current HEAD" >&2
fi
fi
if [ "$SKIP_PULL" != "1" ] && [ "$checked" = "1" ]; then
# only pull real branches, never tags
if git symbolic-ref -q HEAD >/dev/null 2>&1; then
br=$(git rev-parse --abbrev-ref HEAD)
git pull --ff-only origin "$br" 2>/dev/null || true
fi
fi
fi
COMMIT=$(git rev-parse --short=12 HEAD 2>/dev/null || echo unknown)
HEAD_DESC=$(git describe --tags --always 2>/dev/null || echo "$COMMIT")
BR_NOW=$(git rev-parse --abbrev-ref HEAD 2>/dev/null || echo detached)
if [ -z "$LABEL" ]; then
LABEL=$(printf '%s' "$BRANCH" | tr '/:' '--' | tr -c 'A-Za-z0-9._-' '-' | cut -c1-48)
fi
echo "source: $SRC @ $COMMIT ($HEAD_DESC) ref=$BRANCH label=$LABEL head=$BR_NOW"
echo "variant: $VARIANT"
# Map VARIANT → gradle task
case "$VARIANT" in
fdroidDebug|FdroidDebug) task=assembleFdroidDebug ;;
fdroidRelease|FdroidRelease) task=assembleFdroidRelease ;;
nightlyDebug|NightlyDebug) task=assembleNightlyDebug ;;
googleDebug|GoogleDebug) task=assembleGoogleDebug ;;
*)
task="assemble$(printf '%s' "$VARIANT" | python3 -c 'import sys; s=sys.stdin.read().strip(); print(s[:1].upper()+s[1:] if s else "FdroidDebug")')"
;;
esac
# Low-RAM hosts: cap JVM
export GRADLE_OPTS="${GRADLE_OPTS:--Xmx1280m -Dorg.gradle.daemon=false -Dorg.gradle.workers.max=1}"
echo "gradle :wallet:$task …"
./gradlew ":wallet:$task" --no-daemon --max-workers="${GRADLE_MAX_WORKERS:-1}" \
-Dorg.gradle.jvmargs="${GRADLE_JVMARGS:--Xmx1280m}"
# Prefer fdroid debug APK if multiple exist
APK=$(find "$SRC/wallet/build/outputs/apk" -path '*fdroid*debug*.apk' -type f 2>/dev/null | sort | tail -1)
if [ -z "$APK" ] || [ ! -f "$APK" ]; then
APK=$(find "$SRC/wallet/build/outputs/apk" -name '*.apk' -type f 2>/dev/null | sort | tail -1)
fi
if [ -z "$APK" ] || [ ! -f "$APK" ]; then
echo "no APK under wallet/build/outputs/apk" >&2
exit 4
fi
echo "built apk: $APK ($(wc -c <"$APK") bytes)"
PKG="${PKG:-}"
if [ -z "$PKG" ] && command -v aapt >/dev/null 2>&1; then
PKG=$(aapt dump badging "$APK" 2>/dev/null | sed -n "s/^package: name='\([^']*\)'.*/\1/p" | head -1)
fi
if [ -z "$PKG" ] && command -v aapt2 >/dev/null 2>&1; then
PKG=$(aapt2 dump badging "$APK" 2>/dev/null | sed -n "s/^package: name='\([^']*\)'.*/\1/p" | head -1)
fi
: "${PKG:=net.taler.wallet.fdroid.debug}"
mkdir -p "$ROOT/apks" "$ROOT/out"
STAMP_APK="$ROOT/apks/wallet-${LABEL}-${COMMIT}.apk"
cp -f "$APK" "$STAMP_APK"
echo "copied → $STAMP_APK"
{
echo "label=$LABEL"
echo "ref=$BRANCH"
echo "commit=$COMMIT"
echo "describe=$HEAD_DESC"
echo "apk=$STAMP_APK"
echo "pkg=$PKG"
echo "task=$task"
} | tee "$ROOT/out/build-${LABEL}.txt"
echo "$COMMIT" >"$ROOT/out/build-commit.txt"
echo "$APK" >"$ROOT/out/build-apk-path.txt"
export APK_PATH="$STAMP_APK"
export APK_NAME
APK_NAME="$(basename "$STAMP_APK")"
export PKG
export OUT_DIR="${OUT_DIR:-$ROOT/out-source-$LABEL}"
mkdir -p "$OUT_DIR"
cp -f "$ROOT/out/build-${LABEL}.txt" "$OUT_DIR/build-meta.txt" 2>/dev/null || true
SMOKE_CMD="$SMOKE"
if [ "${GUI:-0}" = "1" ]; then
SMOKE_CMD="$ROOT/run-android-gui-smoke.sh"
case "$OUT_DIR" in
*-gui) ;;
*) OUT_DIR="${OUT_DIR}-gui"; export OUT_DIR; mkdir -p "$OUT_DIR" ;;
esac
fi
echo "=== smoke label=$LABEL PKG=$PKG GUI=${GUI:-0} OUT_DIR=$OUT_DIR ==="
# Do not exec: allow matrix wrapper to run multiple variants
set +e
env APK_DIR="$ROOT/apks" APK_NAME="$APK_NAME" APK_PATH="$APK_PATH" PKG="$PKG" OUT_DIR="$OUT_DIR" \
"$SMOKE_CMD"
rc=$?
set -e
echo "=== smoke label=$LABEL exit=$rc ==="
exit "$rc"

201
android-test/run-android-gui-smoke.sh Executable file
View file

@ -0,0 +1,201 @@
#!/usr/bin/env bash
# GUI-oriented Android smoke: **withdraw (Abheben)** then **pay**, each with
# deep-link entry + multi-round uiautomator taps. Not pay-only.
#
# 1) taler://withdraw/… + Confirm/Abheben/ToS taps
# 2) taler://pay… + Pay/Confirm taps
#
# See GUI-AUTOMATION-NOTES.md
#
set -euo pipefail
ROOT=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=lib_android_env.sh
. "$ROOT/lib_android_env.sh"
[ "${AUTO_ANDROID}" = "1" ] || { echo "skipped: AUTO_ANDROID=${AUTO_ANDROID}"; exit 0; }
if [ "${AUTO_GUI}" != "1" ]; then
echo "AUTO_GUI=0 → deep-link smoke only"
exec env AUTO_ANDROID=1 "$ROOT/run-android-pay-smoke.sh"
fi
export OUT_DIR="${OUT_DIR:-$ROOT/out-gui}"
export GUI_ROUNDS="${GUI_ROUNDS:-10}"
export GUI_SLEEP="${GUI_SLEEP:-3}"
# Install + deliver intents via existing smoke, but skip its weak single-tap
# by setting GUI_MODE=1 which we implement here more fully.
# Simpler: do install/URI here by sourcing patterns from pay-smoke via env + python.
# Delegate install + URI start to pay-smoke with GUI_AFTER=1 if we patch it;
# instead run a self-contained flow calling pay-smoke pieces.
APK_DIR="${APK_DIR:-$ROOT/apks}"
APK_URL="${APK_URL:-https://f-droid.org/repo/net.taler.wallet.fdroid_854.apk}"
APK_NAME="${APK_NAME:-net.taler.wallet.fdroid_854.apk}"
APK_PATH="${APK_PATH:-}"
PKG="${PKG:-net.taler.wallet.fdroid}"
STACK="${STACK:-auto}"
SERIAL="${SERIAL:-}"
mkdir -p "$OUT_DIR"
command -v adb >/dev/null || { echo "adb missing" >&2; exit 2; }
# Headless default: no host window; guest GLES via SwiftShader (uiautomator).
android_ensure_device || { echo "no adb device (try ./start-android-emulator.sh --wait)" >&2; exit 3; }
ADB=(adb -s "$SERIAL")
echo "device: $SERIAL mode: GUI (uiautomator) headless=${EMULATOR_HEADLESS} gpu=${EMULATOR_GPU}"
case "$STACK" in
auto)
if curl -sfS -m 5 -o /dev/null https://bank.hacktivism.ch/config; then STACK=goa; else STACK=stage; fi
;;
esac
case "$STACK" in
goa)
BANK=https://bank.hacktivism.ch
MERCHANT=https://taler.hacktivism.ch
WITHDRAW_JSON="$BANK/intro/demo-withdraw.json"
;;
stage)
BANK=https://stage.bank.lefrancpaysan.ch
MERCHANT=https://stage.monnaie.lefrancpaysan.ch
WITHDRAW_JSON="$BANK/intro/demo-withdraw.json"
;;
*) echo "unknown STACK" >&2; exit 2 ;;
esac
echo "stack: $STACK"
if [ -n "$APK_PATH" ] && [ -f "$APK_PATH" ]; then
:
elif [ -f "$APK_DIR/$APK_NAME" ]; then
APK_PATH="$APK_DIR/$APK_NAME"
else
APK_PATH="$APK_DIR/$APK_NAME"
mkdir -p "$APK_DIR"
curl -fL --retry 3 -o "$APK_PATH" "$APK_URL"
fi
echo "apk: $APK_PATH pkg: $PKG"
"${ADB[@]}" wait-for-device
"${ADB[@]}" install -r "$APK_PATH"
"${ADB[@]}" shell settings put global window_animation_scale 0 || true
"${ADB[@]}" shell settings put global transition_animation_scale 0 || true
"${ADB[@]}" shell settings put global animator_duration_scale 0 || true
: "${DO_WITHDRAW:=1}"
: "${DO_PAY:=1}"
: "${SMOKE_STRICT:=0}"
WURI=""
PAY_URI=""
WITHDRAW_OK=0
PAY_OK=0
"${ADB[@]}" shell am force-stop "$PKG" 2>/dev/null || true
"${ADB[@]}" logcat -c 2>/dev/null || true
if [ "$DO_WITHDRAW" = "1" ]; then
echo
echo "======== 1 WITHDRAW (Abheben) ========"
WURI=$(curl -sfS -m 25 "$WITHDRAW_JSON" | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d.get("taler_withdraw_uri") or d.get("qr_payload") or "")')
[ -n "$WURI" ] || { echo "no withdraw uri" >&2; exit 4; }
echo "withdraw: $WURI"
echo "$WURI" >"$OUT_DIR/last-withdraw.uri"
"${ADB[@]}" shell am start -a android.intent.action.VIEW -d "$WURI" | tee "$OUT_DIR/start-withdraw.txt"
sleep 5
python3 - "$ROOT/lib_ui.py" "$SERIAL" "$OUT_DIR" "$GUI_ROUNDS" "$GUI_SLEEP" <<'PY'
import json, sys, importlib.util
from pathlib import Path
lib_path, serial, out, rounds, sleep_s = (
Path(sys.argv[1]), sys.argv[2], Path(sys.argv[3]), int(sys.argv[4]), float(sys.argv[5])
)
spec = importlib.util.spec_from_file_location("lib_ui", lib_path)
lib = importlib.util.module_from_spec(spec)
spec.loader.exec_module(lib)
st = lib.gui_drive(serial, out / "gui-withdraw", rounds=rounds, sleep_s=sleep_s, phase="wd")
(out / "gui-withdraw-status.json").write_text(json.dumps(st, indent=2))
print("gui-withdraw taps:", st.get("taps"))
print("gui-withdraw anr:", st.get("anr_dismissals"))
PY
else
echo "======== 1 WITHDRAW skipped (DO_WITHDRAW=0) ========"
fi
if [ "$DO_PAY" = "1" ]; then
echo
echo "======== 2 PAY (Bezahlen) ========"
if [ "$STACK" = "stage" ]; then
if RESP=$(curl -sfS -m 20 -X POST -H 'Content-Type: application/json' -d '{}' \
"$MERCHANT/instances/fermes-des-collines/templates/panier-legumes" 2>/dev/null); then
PAY_URI=$(printf '%s' "$RESP" | python3 -c 'import json,sys;d=json.load(sys.stdin);oid=d.get("order_id")or"";tok=d.get("token")or"";
print("taler://pay/stage.monnaie.lefrancpaysan.ch/instances/fermes-des-collines/%s/?c=%s"%(oid,tok) if oid and tok else "")')
fi
else
PAY_URI="taler://pay-template/taler.hacktivism.ch/instances/goa-shop/paivana"
fi
if [ -n "$PAY_URI" ]; then
echo "pay: $PAY_URI"
echo "$PAY_URI" >"$OUT_DIR/last-pay.uri"
"${ADB[@]}" shell am start -a android.intent.action.VIEW -d "$PAY_URI" | tee "$OUT_DIR/start-pay.txt"
sleep 5
python3 - "$ROOT/lib_ui.py" "$SERIAL" "$OUT_DIR" "$GUI_ROUNDS" "$GUI_SLEEP" <<'PY'
import json, sys, importlib.util
from pathlib import Path
lib_path, serial, out, rounds, sleep_s = (
Path(sys.argv[1]), sys.argv[2], Path(sys.argv[3]), int(sys.argv[4]), float(sys.argv[5])
)
spec = importlib.util.spec_from_file_location("lib_ui", lib_path)
lib = importlib.util.module_from_spec(spec)
spec.loader.exec_module(lib)
st = lib.gui_drive(serial, out / "gui-pay", rounds=rounds, sleep_s=sleep_s, phase="pay")
(out / "gui-pay-status.json").write_text(json.dumps(st, indent=2))
print("gui-pay taps:", st.get("taps"))
print("gui-pay anr:", st.get("anr_dismissals"))
PY
else
echo "pay: (no URI)"
fi
else
echo "======== 2 PAY skipped (DO_PAY=0) ========"
fi
"${ADB[@]}" logcat -d -t 500 >"$OUT_DIR/logcat.txt" || true
grep -iE 'taler-wallet|prepareBank|acceptWithdrawal|confirmWithdrawal|preparePay|confirmPay|Error|success|withdraw|pay' \
"$OUT_DIR/logcat.txt" | tail -100 >"$OUT_DIR/logcat-taler.txt" || true
if [ "$DO_WITHDRAW" = "1" ] && grep -qE 'prepareBankIntegratedWithdrawal|acceptWithdrawal|confirmWithdrawal' \
"$OUT_DIR/logcat.txt" 2>/dev/null; then
WITHDRAW_OK=1
fi
if [ "$DO_PAY" = "1" ] && [ -n "$PAY_URI" ] && \
grep -qiE 'preparePay|confirmPay|preparePurchase|checkPay|pay-template|PayForTemplate' \
"$OUT_DIR/logcat.txt" 2>/dev/null; then
PAY_OK=1
fi
rc=0
echo
echo "=== GUI smoke summary ($STACK) — withdraw + pay ==="
echo "package: $PKG DO_WITHDRAW=$DO_WITHDRAW DO_PAY=$DO_PAY STRICT=$SMOKE_STRICT"
if [ "$DO_WITHDRAW" = "1" ]; then
echo "withdraw URI: delivered → $OUT_DIR/gui-withdraw-status.json"
echo "withdraw wallet-core: $([ "$WITHDRAW_OK" = 1 ] && echo YES || echo NO)"
if [ "$SMOKE_STRICT" = "1" ] && [ "$WITHDRAW_OK" != "1" ]; then
echo "FAIL: strict withdraw missing"; rc=11
fi
fi
if [ "$DO_PAY" = "1" ]; then
echo "pay URI: ${PAY_URI:-none} → $OUT_DIR/gui-pay-status.json"
echo "pay wallet-core: $([ "$PAY_OK" = 1 ] && echo YES || echo NO)"
if [ "$SMOKE_STRICT" = "1" ] && [ "$PAY_OK" != "1" ]; then
echo "FAIL: strict pay missing"; [ "$rc" -eq 0 ] && rc=13
fi
fi
if grep -qiE "isn't responding|not responding" "$OUT_DIR/logcat.txt" 2>/dev/null \
|| grep -rqiE "isn't responding" "$OUT_DIR/gui-withdraw" "$OUT_DIR/gui-pay" 2>/dev/null; then
echo "FAIL: ANR"; rc=10
fi
echo "artifacts: $OUT_DIR/"
ls -la "$OUT_DIR" | sed 's/^/ /'
echo "Documented shortcuts → GUI-AUTOMATION-NOTES.md"
exit "$rc"

308
android-test/run-android-pay-smoke.sh Executable file
View file

@ -0,0 +1,308 @@
#!/usr/bin/env bash
# Android wallet smoke: install APK, then **withdraw (Abheben)** and **pay** via
# deep-link + best-effort UI taps. Both legs are first-class (not pay-only).
#
# Flow (default both on):
# 1) WITHDRAW taler://withdraw/… from bank demo-withdraw.json + confirm taps
# 2) PAY taler://pay… / pay-template + optional taps
#
# Env:
# DO_WITHDRAW=1 (default) DO_PAY=1 (default)
# SMOKE_STRICT=1 → require wallet-core withdraw (+ pay if DO_PAY) + no ANR
# WITHDRAW_GUI_ROUNDS multi-round Abheben taps (default 6)
#
# Usage:
# ./android-test/run-android-pay-smoke.sh
# STACK=goa ./android-test/run-android-pay-smoke.sh
# DO_PAY=0 ./android-test/run-android-pay-smoke.sh # withdraw-only
#
set -euo pipefail
ROOT=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=lib_android_env.sh
. "$ROOT/lib_android_env.sh"
if [ "${AUTO_ANDROID}" != "1" ]; then
echo "skipped: AUTO_ANDROID=${AUTO_ANDROID} (platform flags — see GUI-AUTOMATION-NOTES.md)"
exit 0
fi
APK_DIR="${APK_DIR:-$ROOT/apks}"
APK_URL="${APK_URL:-https://f-droid.org/repo/net.taler.wallet.fdroid_854.apk}"
APK_NAME="${APK_NAME:-net.taler.wallet.fdroid_854.apk}"
# Optional override: full path to an already-built APK (from-source)
APK_PATH="${APK_PATH:-}"
PKG="${PKG:-net.taler.wallet.fdroid}"
STACK="${STACK:-auto}" # auto | goa | stage
SERIAL="${SERIAL:-}"
OUT_DIR="${OUT_DIR:-$ROOT/out}"
mkdir -p "$APK_DIR" "$OUT_DIR"
if ! command -v adb >/dev/null 2>&1; then
echo "adb missing — install android-tools-adb or user SDK platform-tools" >&2
exit 2
fi
# Default: headless AVD (no host window). WINDOWED=1 / EMULATOR_HEADLESS=0 for UI.
if ! android_ensure_device; then
cat >&2 <<EOF
No adb device. Options:
1) Plug phone with USB debugging
2) Headless AVD (default): $ROOT/start-android-emulator.sh --wait
3) Windowed AVD: WINDOWED=1 $ROOT/start-android-emulator.sh --wait
4) AUTO_START_EMULATOR=0 disables auto-start from this smoke
EOF
exit 3
fi
ADB=(adb -s "$SERIAL")
echo "device: $SERIAL emulator_headless=${EMULATOR_HEADLESS} gpu=${EMULATOR_GPU}"
# Resolve stack endpoints
case "$STACK" in
auto)
if curl -sfS -m 5 -o /dev/null https://bank.hacktivism.ch/config 2>/dev/null; then
STACK=goa
else
STACK=stage
fi
;;
esac
case "$STACK" in
goa)
BANK=https://bank.hacktivism.ch
MERCHANT=https://taler.hacktivism.ch
WITHDRAW_JSON="$BANK/intro/demo-withdraw.json"
PAY_HINT="paivana template / goa-shop"
;;
stage)
BANK=https://stage.bank.lefrancpaysan.ch
MERCHANT=https://stage.monnaie.lefrancpaysan.ch
WITHDRAW_JSON="$BANK/intro/demo-withdraw.json"
PAY_HINT="farmer shop templates"
;;
*) echo "unknown STACK=$STACK" >&2; exit 2 ;;
esac
echo "stack: $STACK bank: $BANK"
# APK: explicit path, or download published F-Droid into APK_DIR
if [ -n "$APK_PATH" ] && [ -f "$APK_PATH" ]; then
:
elif [ -f "$APK_DIR/$APK_NAME" ]; then
APK_PATH="$APK_DIR/$APK_NAME"
else
APK_PATH="$APK_DIR/$APK_NAME"
echo "downloading published wallet APK…"
curl -fL --retry 3 -o "$APK_PATH" "$APK_URL"
fi
echo "apk: $APK_PATH ($(wc -c <"$APK_PATH") bytes)"
echo "pkg: $PKG"
: "${DO_WITHDRAW:=1}"
: "${DO_PAY:=1}"
: "${WITHDRAW_GUI_ROUNDS:=6}"
: "${PAY_GUI_ROUNDS:=4}"
: "${SMOKE_STRICT:=0}"
"${ADB[@]}" wait-for-device
"${ADB[@]}" install -r "$APK_PATH"
"${ADB[@]}" shell settings put global window_animation_scale 0 || true
"${ADB[@]}" shell settings put global transition_animation_scale 0 || true
"${ADB[@]}" shell settings put global animator_duration_scale 0 || true
"${ADB[@]}" shell am force-stop "$PKG" 2>/dev/null || true
"${ADB[@]}" logcat -c 2>/dev/null || true
WURI=""
PAY_URI=""
WITHDRAW_OK=0
PAY_OK=0
# ═══════════════════════════════════════════════════════════════
# 1) WITHDRAW / Abheben (required by default — not pay-only)
# ═══════════════════════════════════════════════════════════════
if [ "$DO_WITHDRAW" = "1" ]; then
echo
echo "======== 1 WITHDRAW (Abheben) ========"
WURI=$(curl -sfS -m 25 "$WITHDRAW_JSON" | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d.get("taler_withdraw_uri") or d.get("qr_payload") or "")')
if [ -z "$WURI" ]; then
echo "FAIL: no taler_withdraw_uri from $WITHDRAW_JSON" >&2
exit 4
fi
echo "withdraw URI: $WURI"
echo "$WURI" >"$OUT_DIR/last-withdraw.uri"
"${ADB[@]}" shell am start -a android.intent.action.VIEW -d "$WURI" 2>&1 | tee "$OUT_DIR/start-withdraw.txt"
sleep 6
"${ADB[@]}" exec-out screencap -p >"$OUT_DIR/01-after-withdraw-uri.png" || true
"${ADB[@]}" shell uiautomator dump /sdcard/ui.xml 2>/dev/null || true
"${ADB[@]}" pull /sdcard/ui.xml "$OUT_DIR/01-ui.xml" 2>/dev/null || true
# Multi-round confirm / Abheben / ToS (same idea as GUI smoke)
if [ -f "$ROOT/lib_ui.py" ]; then
python3 - "$ROOT/lib_ui.py" "$SERIAL" "$OUT_DIR" "$WITHDRAW_GUI_ROUNDS" 2 <<'PY' || true
import json, sys, importlib.util
from pathlib import Path
lib_path, serial, out, rounds, sleep_s = (
Path(sys.argv[1]), sys.argv[2], Path(sys.argv[3]), int(sys.argv[4]), float(sys.argv[5])
)
spec = importlib.util.spec_from_file_location("lib_ui", lib_path)
lib = importlib.util.module_from_spec(spec)
spec.loader.exec_module(lib)
st = lib.gui_drive(serial, out / "gui-withdraw", rounds=rounds, sleep_s=sleep_s, phase="wd")
(out / "gui-withdraw-status.json").write_text(json.dumps(st, indent=2))
print("withdraw GUI taps:", st.get("taps"), "anr:", st.get("anr_dismissals"))
PY
else
python3 - "$SERIAL" "$OUT_DIR/01-ui.xml" <<'PY' || true
import re, subprocess, sys
serial, path = sys.argv[1], sys.argv[2]
try:
xml = open(path, errors="replace").read()
except FileNotFoundError:
raise SystemExit(0)
labels = [
"Confirm", "Confirm withdrawal", "Withdraw", "Accept", "Continue", "OK",
"I accept", "Agree", "Bestätigen", "Abheben", "Akzeptieren", "Retirer",
]
for label in labels:
for pat in (
r'text="%s"[^>]*bounds="\[(\d+),(\d+)\]\[(\d+),(\d+)\]"' % re.escape(label),
r'bounds="\[(\d+),(\d+)\]\[(\d+),(\d+)\]"[^>]*text="%s"' % re.escape(label),
):
for m in re.finditer(pat, xml, flags=re.I):
x1, y1, x2, y2 = map(int, m.groups())
x, y = (x1 + x2) // 2, (y1 + y2) // 2
print(f"tap {label} @ {x},{y}")
subprocess.run(["adb", "-s", serial, "shell", "input", "tap", str(x), str(y)], check=False)
PY
sleep 8
fi
"${ADB[@]}" exec-out screencap -p >"$OUT_DIR/02-after-withdraw-taps.png" || true
# Mid-flow logcat: withdraw must register before pay
"${ADB[@]}" logcat -d -t 400 >"$OUT_DIR/logcat-after-withdraw.txt" || true
if grep -qE 'prepareBankIntegratedWithdrawal|acceptWithdrawal|confirmWithdrawal' \
"$OUT_DIR/logcat-after-withdraw.txt" 2>/dev/null; then
WITHDRAW_OK=1
echo "withdraw: wallet-core activity YES"
else
echo "withdraw: wallet-core activity NOT seen yet (onboarding/ANR/slow?)"
fi
else
echo "======== 1 WITHDRAW skipped (DO_WITHDRAW=0) ========"
fi
# ═══════════════════════════════════════════════════════════════
# 2) PAY (after withdraw; still first-class, but second leg)
# ═══════════════════════════════════════════════════════════════
if [ "$DO_PAY" = "1" ]; then
echo
echo "======== 2 PAY (Bezahlen) ========"
if [ "$STACK" = "stage" ]; then
if RESP=$(curl -sfS -m 20 -X POST -H 'Content-Type: application/json' -d '{}' \
"$MERCHANT/instances/fermes-des-collines/templates/panier-legumes" 2>/dev/null); then
PAY_URI=$(printf '%s' "$RESP" | python3 -c '
import json,sys
d=json.load(sys.stdin)
oid=d.get("order_id") or ""
tok=d.get("token") or ""
if oid and tok:
print("taler://pay/stage.monnaie.lefrancpaysan.ch/instances/fermes-des-collines/%s/?c=%s"%(oid,tok))
' 2>/dev/null || true)
fi
elif [ "$STACK" = "goa" ]; then
PAY_URI="taler://pay-template/taler.hacktivism.ch/instances/goa-shop/paivana"
fi
if [ -n "$PAY_URI" ]; then
echo "pay URI: $PAY_URI ($PAY_HINT)"
echo "$PAY_URI" >"$OUT_DIR/last-pay.uri"
"${ADB[@]}" shell am start -a android.intent.action.VIEW -d "$PAY_URI" 2>&1 | tee "$OUT_DIR/start-pay.txt"
sleep 6
if [ -f "$ROOT/lib_ui.py" ]; then
python3 - "$ROOT/lib_ui.py" "$SERIAL" "$OUT_DIR" "$PAY_GUI_ROUNDS" 2 <<'PY' || true
import json, sys, importlib.util
from pathlib import Path
lib_path, serial, out, rounds, sleep_s = (
Path(sys.argv[1]), sys.argv[2], Path(sys.argv[3]), int(sys.argv[4]), float(sys.argv[5])
)
spec = importlib.util.spec_from_file_location("lib_ui", lib_path)
lib = importlib.util.module_from_spec(spec)
spec.loader.exec_module(lib)
st = lib.gui_drive(serial, out / "gui-pay", rounds=rounds, sleep_s=sleep_s, phase="pay")
(out / "gui-pay-status.json").write_text(json.dumps(st, indent=2))
print("pay GUI taps:", st.get("taps"), "anr:", st.get("anr_dismissals"))
PY
fi
"${ADB[@]}" exec-out screencap -p >"$OUT_DIR/03-after-pay-uri.png" || true
else
echo "pay: (no public template URI for stack=$STACK)"
fi
else
echo "======== 2 PAY skipped (DO_PAY=0) ========"
fi
# Final evidence
"${ADB[@]}" logcat -d -t 500 >"$OUT_DIR/logcat.txt" || true
grep -iE 'taler-wallet|prepareBank|acceptWithdrawal|confirmWithdrawal|preparePay|confirmPay|error|Error|success|withdraw|pay' \
"$OUT_DIR/logcat.txt" | tail -100 >"$OUT_DIR/logcat-taler.txt" || true
# Re-evaluate withdraw/pay from full logcat
if [ "$DO_WITHDRAW" = "1" ]; then
if grep -qE 'prepareBankIntegratedWithdrawal|acceptWithdrawal|confirmWithdrawal' \
"$OUT_DIR/logcat.txt" 2>/dev/null; then
WITHDRAW_OK=1
fi
fi
if [ "$DO_PAY" = "1" ] && [ -n "$PAY_URI" ]; then
if grep -qiE 'preparePay|confirmPay|preparePurchase|checkPay|pay-template|PayForTemplate' \
"$OUT_DIR/logcat.txt" 2>/dev/null; then
PAY_OK=1
fi
fi
rc=0
echo
echo "=== summary (withdraw + pay) ==="
echo "package: $PKG stack: $STACK"
echo "DO_WITHDRAW=$DO_WITHDRAW DO_PAY=$DO_PAY SMOKE_STRICT=$SMOKE_STRICT"
if [ "$DO_WITHDRAW" = "1" ]; then
echo "withdraw URI: ${WURI:-(none)}"
if [ "$WITHDRAW_OK" = "1" ]; then
echo "withdraw wallet-core: YES"
else
echo "withdraw wallet-core: NO"
if [ "$SMOKE_STRICT" = "1" ]; then
echo "FAIL: SMOKE_STRICT requires withdraw (prepareBank/accept/confirm) in logcat"
rc=11
fi
fi
fi
if [ "$DO_PAY" = "1" ]; then
echo "pay URI: ${PAY_URI:-(none)}"
if [ -z "$PAY_URI" ]; then
echo "pay: no URI (template/API)"
if [ "$SMOKE_STRICT" = "1" ]; then
echo "FAIL: SMOKE_STRICT requires a pay URI when DO_PAY=1"
rc=12
fi
elif [ "$PAY_OK" = "1" ]; then
echo "pay wallet-core: YES"
else
echo "pay wallet-core: NO (intent may still have been delivered)"
if [ "$SMOKE_STRICT" = "1" ]; then
echo "FAIL: SMOKE_STRICT requires pay path activity in logcat"
[ "$rc" -eq 0 ] && rc=13
fi
fi
fi
if grep -qiE "isn't responding|isn.t responding|System UI isn" "$OUT_DIR/01-ui.xml" 2>/dev/null \
|| grep -qiE "isn't responding|not responding" "$OUT_DIR/logcat.txt" 2>/dev/null \
|| grep -rqiE "isn't responding" "$OUT_DIR/gui-withdraw" 2>/dev/null; then
echo "FAIL: System UI / app ANR"
rc=10
fi
echo "artifacts: $OUT_DIR/"
ls -la "$OUT_DIR" | sed 's/^/ /'
echo
echo "NOTE: Full unattended Abheben+Bezahlen needs responsive device (≥6–8 GiB host RAM)."
echo "Smoke legs: withdraw=$( [ "$DO_WITHDRAW" = 1 ] && echo on || echo off) pay=$( [ "$DO_PAY" = 1 ] && echo on || echo off) rc=$rc"
exit "$rc"

322
android-test/run-android-variant-matrix.sh Executable file
View file

@ -0,0 +1,322 @@
#!/usr/bin/env bash
# Run Android wallet smoke against several build / published variants.
#
# Default order:
# 1) stable-self — **only** self-build of current published stable (WARN)
# 2) published — current F-Droid APK
# 3) older — F-Droid APKs ~3/6/9/12 mo if still hosted (WARN, no self-build)
# 4) master — tip self-build (WARN)
#
# Self-build policy: current stable + master/fix only — never rebuild old releases.
#
# Severity: age ≤14d → BLOCKER; milestones/older/self-builds → WARN
#
# Env:
# STABLE_TAG, STABLE_APK_URL, STABLE_APK_NAME
# MASTER_BRANCH, FIX_BRANCH
# VARIANTS default: stable-self,published,older,master
# INCLUDE_OLDER=1 expand token "older" / "milestones" (default 1)
# MILESTONE_DAYS default "90 180 270 365" (≈ 3,6,9,12 months)
# MILESTONE_TOLERANCE_DAYS default 50 — max distance to accept a tag
# BLOCKER_MAX_AGE_DAYS default 14
# PUBLISHED_FORCE_BLOCKER=1 always treat current published as blocker
# FAIL_FAST=1, GUI=0, STACK=…
#
set -euo pipefail
ROOT=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=lib_android_env.sh
. "$ROOT/lib_android_env.sh"
# shellcheck source=lib_release_age.sh
. "$ROOT/lib_release_age.sh"
: "${STABLE_TAG:=wallet-1.6.1}"
: "${STABLE_APK_URL:=https://f-droid.org/repo/net.taler.wallet.fdroid_854.apk}"
: "${STABLE_APK_NAME:=net.taler.wallet.fdroid_854.apk}"
: "${MASTER_BRANCH:=master}"
: "${FIX_BRANCH:=dev/hernani-inference/fix-bank-withdraw-auto-exchange}"
: "${VARIANTS:=stable-self,published,older,master}"
: "${INCLUDE_OLDER:=1}"
: "${MILESTONE_DAYS:=90 180 270 365}"
: "${MILESTONE_TOLERANCE_DAYS:=50}"
: "${BLOCKER_MAX_AGE_DAYS:=14}"
: "${PUBLISHED_FORCE_BLOCKER:=0}"
: "${FAIL_FAST:=1}"
: "${GUI:=0}"
: "${STACK:=auto}"
: "${TALER_ANDROID_SRC:=$HOME/taler/taler-android}"
export TALER_ANDROID_SRC BLOCKER_MAX_AGE_DAYS MILESTONE_DAYS MILESTONE_TOLERANCE_DAYS
TS=$(date +%Y%m%d-%H%M%S)
MATRIX_OUT="${MATRIX_OUT:-$ROOT/out-matrix-$TS}"
mkdir -p "$MATRIX_OUT"
SUMMARY="$MATRIX_OUT/SUMMARY.txt"
: >"$SUMMARY"
echo "=== Android variant matrix ===" | tee -a "$SUMMARY"
echo "STACK=$STACK GUI=$GUI headless=${EMULATOR_HEADLESS} variants=$VARIANTS" | tee -a "$SUMMARY"
echo "STABLE_TAG=$STABLE_TAG BLOCKER_MAX_AGE_DAYS=$BLOCKER_MAX_AGE_DAYS" | tee -a "$SUMMARY"
echo "milestones(days)=$MILESTONE_DAYS tolerance=$MILESTONE_TOLERANCE_DAYS (always WARN)" | tee -a "$SUMMARY"
echo "out: $MATRIX_OUT" | tee -a "$SUMMARY"
android_ensure_device || {
echo "no adb device — start ./start-android-emulator.sh --wait" >&2
exit 3
}
export SERIAL ANDROID_SERIAL
echo "device: $SERIAL" | tee -a "$SUMMARY"
# --- expand VARIANTS (token "older" → concrete older-pub / older-self entries) ---
# Append one older release as WARN-only — **F-Droid APK only** (no self-build).
# Returns 0 if queued, 1 if skipped (no published APK).
append_older_warn() {
local t="$1" age="$2" target="${3:-}"
local vn meta apk_vc apk_url
vn="${t#wallet-}"
if [ -n "$target" ]; then
echo " milestone ~${target}d → $t (age=${age}d) severity=WARN" | tee -a "$SUMMARY"
else
echo " older $t (age=${age}d) severity=WARN" | tee -a "$SUMMARY"
fi
meta=$(fdroid_apk_for_version_name "$vn" 2>/dev/null || true)
if [ -n "$meta" ]; then
apk_vc="${meta%%|*}"
apk_url="${meta#*|}"
if fdroid_apk_exists "$apk_url"; then
out+=("older-pub:${vn}:${apk_vc}:warn")
return 0
fi
echo " SKIP $t — no F-Droid APK at ${apk_url:-?} (self-build only for current stable)" | tee -a "$SUMMARY"
else
echo " SKIP $t — cannot map F-Droid versionCode (self-build only for current stable)" | tee -a "$SUMMARY"
fi
return 1
}
expand_variants() {
local raw="$1"
local -a out=()
local v tag vn age sev line t target
IFS=',' read -r -a toks <<<"$raw"
for v in "${toks[@]}"; do
v=$(echo "$v" | tr -d '[:space:]')
[ -n "$v" ] || continue
if [ "$v" = "older" ] || [ "$v" = "older-releases" ] || [ "$v" = "milestones" ]; then
[ "$INCLUDE_OLDER" = "1" ] || continue
# 3/6/9/12-month samples: published APK only, always WARN.
local found=0 queued=0
while read -r t age target; do
[ -n "$t" ] || continue
[ "$t" = "$STABLE_TAG" ] && continue
found=1
if append_older_warn "$t" "$age" "$target"; then
queued=$((queued + 1))
fi
done < <(select_milestone_tags "$STABLE_TAG" 2>/dev/null || true)
if [ "$found" = "0" ]; then
echo " (no milestone tags within ±${MILESTONE_TOLERANCE_DAYS}d of $MILESTONE_DAYS)" | tee -a "$SUMMARY"
elif [ "$queued" = "0" ]; then
echo " (milestone tags found but none still on F-Droid — no older APK smokes)" | tee -a "$SUMMARY"
fi
else
out+=("$v")
fi
done
printf '%s\n' "${out[@]}"
}
mapfile -t VLIST < <(expand_variants "$VARIANTS")
echo "expanded variants: ${VLIST[*]}" | tee -a "$SUMMARY"
declare -a RESULTS=()
declare -a WARNINGS=()
overall=0
severity_for_published_current() {
if [ "${PUBLISHED_FORCE_BLOCKER}" = "1" ]; then
echo blocker
return
fi
local days
days=$(release_age_days "$STABLE_TAG" 2>/dev/null || true)
severity_from_age_days "$days"
}
run_published_apk() {
local label="$1"
local apk_url="$2"
local apk_name="$3"
local sev="$4"
local out="$MATRIX_OUT/$label"
mkdir -p "$out"
echo "" | tee -a "$SUMMARY"
echo "-------- variant: $label (F-Droid APK) [$(echo "$sev" | tr 'a-z' 'A-Z')] --------" | tee -a "$SUMMARY"
echo " url=$apk_url" | tee -a "$SUMMARY"
local smoke="$ROOT/run-android-pay-smoke.sh"
[ "$GUI" = "1" ] && smoke="$ROOT/run-android-gui-smoke.sh"
local strict=1
[ "$sev" = "warn" ] && strict="${SMOKE_STRICT_WARN:-0}"
[ "$sev" = "blocker" ] && strict="${SMOKE_STRICT:-1}"
set +e
env STACK="$STACK" \
APK_URL="$apk_url" \
APK_NAME="$apk_name" \
APK_PATH="" \
PKG=net.taler.wallet.fdroid \
OUT_DIR="$out" \
SERIAL="$SERIAL" \
AUTO_START_EMULATOR=0 \
SMOKE_STRICT="$strict" \
"$smoke"
local rc=$?
set -e
echo "$label exit=$rc severity=$sev" | tee -a "$SUMMARY"
echo "$label $rc $sev $apk_url" >>"$MATRIX_OUT/results.tsv"
return "$rc"
}
run_selfbuild() {
local label="$1"
local ref="$2"
local sev="${3:-warn}"
local out="$MATRIX_OUT/$label"
mkdir -p "$out"
echo "" | tee -a "$SUMMARY"
echo "-------- variant: $label (self-build ref=$ref) [$(echo "$sev" | tr 'a-z' 'A-Z')] --------" | tee -a "$SUMMARY"
local strict="${SMOKE_STRICT:-1}"
[ "$sev" = "warn" ] && strict="${SMOKE_STRICT_SELF:-0}"
[ "$sev" = "blocker" ] && strict="${SMOKE_STRICT:-1}"
set +e
env STACK="$STACK" \
BRANCH="$ref" \
LABEL="$label" \
GUI="$GUI" \
OUT_DIR="$out" \
SERIAL="$SERIAL" \
ANDROID_SERIAL="$SERIAL" \
AUTO_START_EMULATOR=0 \
PKG=net.taler.wallet.fdroid.debug \
SMOKE_STRICT="$strict" \
TALER_ANDROID_SRC="$TALER_ANDROID_SRC" \
"$ROOT/run-android-build-and-smoke.sh"
local rc=$?
set -e
echo "$label (ref=$ref) exit=$rc severity=$sev" | tee -a "$SUMMARY"
echo "$label $rc $ref $sev" >>"$MATRIX_OUT/results.tsv"
if [ -f "$out/build-meta.txt" ]; then
echo "build-meta:" | tee -a "$SUMMARY"
cat "$out/build-meta.txt" | tee -a "$SUMMARY"
fi
return "$rc"
}
record_result() {
local v="$1" rc="$2" sev="$3"
if [ "$rc" -eq 0 ]; then
RESULTS+=("$v:OK($sev)")
return
fi
case "$sev" in
warn)
echo "WARN: $v failed rc=$rc — not a blocker; continuing" | tee -a "$SUMMARY"
WARNINGS+=("$v:$rc")
RESULTS+=("$v:WARN($rc)")
;;
blocker)
echo "BLOCKER: $v failed rc=$rc (release within ${BLOCKER_MAX_AGE_DAYS}d window)" | tee -a "$SUMMARY"
RESULTS+=("$v:BLOCKER($rc)")
overall=1
if [ "$FAIL_FAST" = "1" ]; then
echo "FAIL_FAST: stopping after blocker $v" | tee -a "$SUMMARY"
return 2
fi
;;
soft)
echo "FAIL: $v rc=$rc" | tee -a "$SUMMARY"
RESULTS+=("$v:FAIL($rc)")
overall=1
if [ "$FAIL_FAST" = "1" ]; then
echo "FAIL_FAST: stopping after $v" | tee -a "$SUMMARY"
return 2
fi
;;
esac
return 0
}
for raw in "${VLIST[@]}"; do
v=$(echo "$raw" | tr -d '[:space:]')
[ -n "$v" ] || continue
rc=0
sev=warn
stop=0
case "$v" in
published|fdroid|stable-published)
sev=$(severity_for_published_current)
age=$(release_age_days "$STABLE_TAG" 2>/dev/null || echo "?")
echo "published current: tag=$STABLE_TAG age_days=$age → severity=$sev" | tee -a "$SUMMARY"
run_published_apk "published" "$STABLE_APK_URL" "$STABLE_APK_NAME" "$sev" || rc=$?
;;
stable-self|stable|self-stable|release-self)
# rebuild of stable: always warn (toolchain), age does not make self-build a blocker
sev=warn
run_selfbuild "stable-self" "$STABLE_TAG" warn || rc=$?
;;
master|main)
sev=warn
run_selfbuild "master" "$MASTER_BRANCH" warn || rc=$?
;;
fix|fix-branch|inference)
sev=soft
run_selfbuild "fix" "$FIX_BRANCH" soft || rc=$?
;;
older-pub:*)
# older-pub:VERSION:VC:SEV
IFS=':' read -r _ vn apk_vc sev <<<"$v"
: "${sev:=warn}"
label="older-pub-${vn}"
url="https://f-droid.org/repo/net.taler.wallet.fdroid_${apk_vc}.apk"
name="net.taler.wallet.fdroid_${apk_vc}.apk"
age=$(release_age_days "wallet-${vn}" 2>/dev/null || echo "?")
echo "older published $vn age_days=$age severity=$sev" | tee -a "$SUMMARY"
run_published_apk "$label" "$url" "$name" "$sev" || rc=$?
v="$label"
;;
older-self:*)
# Explicit older-self is disabled by policy (self-build = current stable only).
echo "SKIP $v — self-build only for current stable ($STABLE_TAG); use older-pub or published" | tee -a "$SUMMARY"
RESULTS+=("$v:SKIP(no-selfbuild-older)")
continue
;;
*)
# raw git ref: only master-like / fix via named tokens; bare wallet-* tags → no self-build
if [[ "$v" =~ ^wallet-[0-9] ]]; then
echo "SKIP self-build $v — only current stable self-build ($STABLE_TAG) is enabled" | tee -a "$SUMMARY"
RESULTS+=("$v:SKIP(no-selfbuild-older)")
continue
fi
sev=warn
run_selfbuild "ref-${v//\//-}" "$v" warn || rc=$?
;;
esac
record_result "$v" "$rc" "$sev" || stop=$?
[ "$stop" = "2" ] && break
done
echo "" | tee -a "$SUMMARY"
echo "=== matrix done (overall=$overall) ===" | tee -a "$SUMMARY"
echo "rule: age ≤ ${BLOCKER_MAX_AGE_DAYS}d → BLOCKER; 3/6/9/12mo milestones + older → WARN only" | tee -a "$SUMMARY"
for r in "${RESULTS[@]}"; do
echo " $r" | tee -a "$SUMMARY"
done
if [ "${#WARNINGS[@]}" -gt 0 ]; then
echo "warnings (non-blocking):" | tee -a "$SUMMARY"
for w in "${WARNINGS[@]}"; do
echo " WARN $w" | tee -a "$SUMMARY"
done
fi
echo ""
echo "Artifacts: $MATRIX_OUT"
exit "$overall"

366
android-test/run-goa-gui-chain.sh Executable file
View file

@ -0,0 +1,366 @@
#!/usr/bin/env bash
# GOA (and optionally stage) GUI chain on Android — adapted from
# taler-android branch **dev/hernani-inference/gui-workflows**
# (`scripts/goa-chain-emu.sh`, docs/gui-workflows.md).
#
# Linux + macOS: uses adb + uiautomator only (no Homebrew hard dependency).
# Upstream scripts are documented as macOS-only for SDK install paths.
#
# Flow:
# 1) API mint withdraw (explorer) → VIEW taler://withdraw → GUI taps
# 2) Merchant template POST → VIEW taler://pay → GUI taps
#
# Usage:
# ./run-goa-gui-chain.sh
# STACK=stage ./run-goa-gui-chain.sh
# WITHDRAW_AMOUNTS='GOA:10' PAY_LIMIT=2 ./run-goa-gui-chain.sh
# PKG=net.taler.wallet.fdroid.debug ./run-goa-gui-chain.sh --no-clear
#
# Env: ANDROID_SERIAL, PKG, EXP_PW_FILE, BANK, MERCHANT, INSTANCE,
# WITHDRAW_AMOUNTS, PRODUCTS, PAY_LIMIT, SHOTDIR, PAUSE
#
# See GUI-AUTOMATION-NOTES.md
#
set -euo pipefail
ROOT=$(cd "$(dirname "$0")" && pwd)
CLEAR=1
for a in "$@"; do
case "$a" in
--no-clear) CLEAR=0 ;;
-h|--help) sed -n '2,28p' "$0"; exit 0 ;;
esac
done
die() { echo "ERROR: $*" >&2; exit 1; }
step() { echo; echo "======== $* ========"; }
info() { echo " $*"; }
pause() { sleep "${1:-${PAUSE:-0.8}}"; }
# Platform flags + headless emulator defaults (see GUI-AUTOMATION-NOTES.md)
# shellcheck source=lib_android_env.sh
. "$ROOT/lib_android_env.sh"
[ "${AUTO_ANDROID}" = "1" ] || { echo "skipped: AUTO_ANDROID=${AUTO_ANDROID}"; exit 0; }
[ "${AUTO_GUI}" = "1" ] || { echo "skipped: AUTO_GUI=${AUTO_GUI} (chain is GUI)"; exit 0; }
command -v adb >/dev/null || die "adb not found"
command -v python3 >/dev/null || die "python3 required"
STACK="${STACK:-goa}"
case "$STACK" in
goa|hacktivism)
PKG="${PKG:-net.taler.wallet.fdroid.debug}"
BANK="${BANK:-https://bank.hacktivism.ch}"
MERCHANT="${MERCHANT:-https://taler.hacktivism.ch}"
INSTANCE=$(printf '%s' "${INSTANCE:-goa-shop}" | tr '[:upper:]' '[:lower:]')
DONATE_INSTANCE=$(printf '%s' "${DONATE_INSTANCE:-goa-demo-cp4zqk}" | tr '[:upper:]' '[:lower:]')
DONATE_TEMPLATE="${DONATE_TEMPLATE:-goa-free}"
DEFAULT_AMOUNTS=("GOA:10" "GOA:20")
DEFAULT_PAYS=("product:orbit-sticker" "product:nebula-coffee" "product:voidwave-playlist" "donate:GOA:12")
CUR_HINT=GOA
;;
stage|testpaysan)
PKG="${PKG:-net.taler.wallet.fdroid.debug}"
BANK="${BANK:-https://stage.bank.lefrancpaysan.ch}"
MERCHANT="${MERCHANT:-https://stage.monnaie.lefrancpaysan.ch}"
INSTANCE=$(printf '%s' "${INSTANCE:-fermes-des-collines}" | tr '[:upper:]' '[:lower:]')
DONATE_INSTANCE=$(printf '%s' "${DONATE_INSTANCE:-fermes-des-collines}" | tr '[:upper:]' '[:lower:]')
DONATE_TEMPLATE="${DONATE_TEMPLATE:-don-panier-libre}"
DEFAULT_AMOUNTS=("TESTPAYSAN:10" "TESTPAYSAN:20")
DEFAULT_PAYS=("product:panier-legumes" "product:fromage-chevre" "product:oeufs-6")
CUR_HINT=TESTPAYSAN
;;
*) die "STACK must be goa|stage (got $STACK)" ;;
esac
ACT="${ACT:-$PKG/net.taler.wallet.main.MainActivity}"
EXP_USER="${EXP_USER:-explorer}"
PAY_LIMIT="${PAY_LIMIT:-4}"
PAUSE="${PAUSE:-0.8}"
SHOTDIR="${SHOTDIR:-$ROOT/out-gui-chain/$(date +%Y%m%d-%H%M%S)-$STACK}"
mkdir -p "$SHOTDIR"
# Explorer password (same search as taler-monitoring ladder)
if [[ -z "${EXP_PW_FILE:-}" ]]; then
for cand in \
"${HOME}/src/koopa/koopa-admin-secrets/koopa/host-root/taler-bank/bank-explorer-password.txt" \
"${HOME}/.config/taler-landing/bank-explorer-password.txt"
do
[[ -f "$cand" ]] && EXP_PW_FILE="$cand" && break
done
fi
# stage: try stagepaysan secret via ssh if missing
if [[ ! -f "${EXP_PW_FILE:-}" && "$STACK" = "stage" ]]; then
for host in ${INSIDE_SSH:+"$INSIDE_SSH"} ${FRANCPAYSAN_SSH:+"$FRANCPAYSAN_SSH"}; do
tmp=$(mktemp)
if ssh -o BatchMode=yes -o ConnectTimeout=10 "$host" \
'tr -d "\n\r" </mnt/data/stagepaysan/bank/secrets/bank-explorer-password.txt' \
>"$tmp" 2>/dev/null && [[ -s "$tmp" ]]; then
EXP_PW_FILE="$tmp"
info "explorer password via $host"
break
fi
rm -f "$tmp"
done
fi
[[ -n "${EXP_PW_FILE:-}" && -f "$EXP_PW_FILE" ]] || die "set EXP_PW_FILE (explorer password)"
# Headless AVD by default (no host window). WINDOWED=1 for a visible emulator.
SERIAL="${SERIAL:-${ANDROID_SERIAL:-}}"
android_ensure_device || die "no adb device (./start-android-emulator.sh --wait)"
export ANDROID_SERIAL="$SERIAL"
info "ANDROID_SERIAL=$ANDROID_SERIAL STACK=$STACK PKG=$PKG SHOTDIR=$SHOTDIR headless=$EMULATOR_HEADLESS gpu=$EMULATOR_GPU"
ADB=(adb -s "$ANDROID_SERIAL")
adb_sh() { "${ADB[@]}" shell "$@"; }
shot() {
local n="$1"
adb_sh screencap -p "/sdcard/demo-$n.png" 2>/dev/null || true
"${ADB[@]}" pull "/sdcard/demo-$n.png" "$SHOTDIR/$n.png" >/dev/null 2>&1 || true
info "[shot] $SHOTDIR/$n.png"
}
tap_text() {
local label="$1"
adb_sh uiautomator dump /sdcard/ui.xml >/dev/null 2>&1 || return 1
"${ADB[@]}" pull /sdcard/ui.xml "$SHOTDIR/ui.xml" >/dev/null 2>&1 || return 1
SERIAL="$ANDROID_SERIAL" python3 - "$label" "$SHOTDIR/ui.xml" <<'PY'
import os, re, subprocess, sys
from pathlib import Path
label, path = sys.argv[1], sys.argv[2]
serial = os.environ.get("SERIAL") or os.environ.get("ANDROID_SERIAL") or ""
xml = Path(path).read_text(errors="replace")
# ANR dismiss
if "isn't responding" in xml or "reagiert nicht" in xml.lower():
for lab in ("Wait", "Warten"):
for m in re.finditer(rf'text="{lab}"[^>]*bounds="\[(\d+),(\d+)\]\[(\d+),(\d+)\]"', xml):
x = (int(m.group(1))+int(m.group(3)))//2
y = (int(m.group(2))+int(m.group(4)))//2
subprocess.run(["adb","-s",serial,"shell","input","tap",str(x),str(y)], check=False)
print(f"anr-wait @{x},{y}")
sys.exit(0)
for pat in [
rf'text="{re.escape(label)}"[^>]*bounds="\[(\d+),(\d+)\]\[(\d+),(\d+)\]"',
rf'text="[^"]*{re.escape(label)}[^"]*"[^>]*bounds="\[(\d+),(\d+)\]\[(\d+),(\d+)\]"',
rf'content-desc="[^"]*{re.escape(label)}[^"]*"[^>]*bounds="\[(\d+),(\d+)\]\[(\d+),(\d+)\]"',
]:
for m in re.finditer(pat, xml, re.I):
chunk = xml[max(0, m.start()-120):m.end()]
if 'enabled="false"' in chunk:
continue
x = (int(m.group(1))+int(m.group(3)))//2
y = (int(m.group(2))+int(m.group(4)))//2
print(f"tap {label!r} @{x},{y}")
cmd = ["adb","shell","input","tap",str(x),str(y)]
if serial:
cmd = ["adb","-s",serial,"shell","input","tap",str(x),str(y)]
subprocess.run(cmd, check=False)
sys.exit(0)
print("miss", label)
sys.exit(1)
PY
}
mint_withdraw() {
local amount="$1"
EXP_USER="$EXP_USER" EXP_PW_FILE="$EXP_PW_FILE" BANK="$BANK" \
python3 - "$amount" <<'PY'
import base64, json, os, ssl, sys, urllib.request
from pathlib import Path
amount = sys.argv[1]
bank = os.environ["BANK"].rstrip("/")
user = os.environ["EXP_USER"]
pw = Path(os.environ["EXP_PW_FILE"]).read_text().strip()
auth = "Basic " + base64.b64encode(f"{user}:{pw}".encode()).decode()
ctx = ssl.create_default_context()
def req(method, url, data=None, headers=None):
h = dict(headers or {})
body = None
if data is not None:
body = json.dumps(data).encode()
h["Content-Type"] = "application/json"
r = urllib.request.Request(url, data=body, headers=h, method=method)
with urllib.request.urlopen(r, context=ctx, timeout=25) as resp:
return json.load(resp)
tok = req(
"POST",
f"{bank}/accounts/{user}/token",
{"scope": "readwrite", "duration": {"d_us": 3600_000_000}},
{"Authorization": auth},
)["access_token"]
wd = req(
"POST",
f"{bank}/accounts/{user}/withdrawals",
{"amount": amount},
{"Authorization": f"Bearer {tok}"},
)
print(wd.get("taler_withdraw_uri") or wd.get("talerWithdrawUri") or "")
PY
}
mint_pay() {
local mode="$1"
MERCHANT="$MERCHANT" INSTANCE="$INSTANCE" \
DONATE_INSTANCE="$DONATE_INSTANCE" DONATE_TEMPLATE="$DONATE_TEMPLATE" \
python3 - "$mode" <<'PY'
import json, os, ssl, sys, urllib.error, urllib.request
mode = sys.argv[1]
merchant = os.environ["MERCHANT"].rstrip("/")
inst = os.environ["INSTANCE"]
d_inst = os.environ["DONATE_INSTANCE"]
d_tmpl = os.environ["DONATE_TEMPLATE"]
ctx = ssl.create_default_context()
def post(url, data):
body = json.dumps(data).encode()
r = urllib.request.Request(url, data=body, method="POST",
headers={"Content-Type": "application/json"})
try:
with urllib.request.urlopen(r, context=ctx, timeout=25) as resp:
return resp.status, json.load(resp)
except urllib.error.HTTPError as e:
raw = e.read() if hasattr(e, "read") else b""
try:
return getattr(e, "code", None), json.loads(raw)
except Exception as err:
raise SystemExit(f"POST {url} failed: {e} body={raw[:200]!r}") from err
if mode.startswith("product:"):
pid = mode.split(":", 1)[1]
st, d = post(f"{merchant}/instances/{inst}/templates/{pid}", {})
use_inst = inst
elif mode.startswith("donate:"):
amt = mode.split(":", 1)[1]
st, d = post(f"{merchant}/instances/{d_inst}/templates/{d_tmpl}", {"amount": amt})
use_inst = d_inst
else:
raise SystemExit(f"bad mode {mode!r}")
if "taler_pay_uri" in d:
print(d["taler_pay_uri"]); raise SystemExit(0)
oid, tok = d.get("order_id"), d.get("token")
if not oid or not tok:
raise SystemExit(f"no order: {d!r}"[:200])
host = merchant.replace("https://","").replace("http://","").split("/")[0]
print(f"taler://pay/{host}/instances/{use_inst}/{oid}/?c={tok}")
PY
}
click_through() {
local labels=("$@")
local i lab misses=0
# Prefer dismissing ANR before hunting Confirm (low-RAM hosts thrash System UI)
for i in 1 2 3 4 5 6; do
if tap_text "Wait" 2>/dev/null || tap_text "Warten" 2>/dev/null; then
info "dismissed ANR (Wait)"
pause 2.0
misses=0
continue
fi
local hit=0
for lab in "${labels[@]}"; do
if tap_text "$lab" 2>/dev/null; then
hit=1
misses=0
pause 0.9
break
fi
done
if [[ "$hit" -eq 0 ]]; then
misses=$((misses + 1))
# stop early if hierarchy has nothing useful (avoid burning order deadlines)
[[ "$misses" -ge 3 ]] && break
fi
pause 0.5
done
}
if [[ -n "${WITHDRAW_AMOUNTS:-}" ]]; then
# shellcheck disable=SC2206
AMOUNTS=($WITHDRAW_AMOUNTS)
else
AMOUNTS=("${DEFAULT_AMOUNTS[@]}")
fi
if [[ -n "${PRODUCTS:-}" ]]; then
PAYS=()
for p in $PRODUCTS; do PAYS+=("product:$p"); done
else
PAYS=("${DEFAULT_PAYS[@]}")
fi
############################
step "0 device + wallet ($STACK / $CUR_HINT)"
adb_sh input keyevent KEYCODE_WAKEUP 2>/dev/null || true
adb_sh wm dismiss-keyguard 2>/dev/null || true
if [[ "$CLEAR" -eq 1 ]]; then
info "pm clear $PKG"
adb_sh pm clear "$PKG" >/dev/null 2>&1 || true
fi
adb_sh am force-stop "$PKG" 2>/dev/null || true
pause 0.4
# Prefer explicit MainActivity (from gui-workflows); fall back to VIEW launcher
if ! adb_sh am start -n "$ACT" >/dev/null 2>&1; then
info "MainActivity path failed — monkey launch"
adb_sh monkey -p "$PKG" -c android.intent.category.LAUNCHER 1 >/dev/null 2>&1 || true
fi
pause 2.0
shot "00-start"
############################
step "2 withdraw chain"
idx=0
for AMT in "${AMOUNTS[@]}"; do
idx=$((idx + 1))
step "2.$idx withdraw $AMT"
URI="$(mint_withdraw "$AMT" | head -1)"
[[ -n "$URI" ]] || die "mint failed for $AMT"
# wallet / monitoring QR rules: strip default ports
URI=$(printf '%s' "$URI" | sed 's/:443\//\//g; s/:443?/?/g; s/:80\//\//g')
info "URI=$URI"
echo "$URI" >>"$SHOTDIR/withdraw-uris.txt"
adb_sh am start -a android.intent.action.VIEW -d "$URI" "$PKG" >/dev/null
pause 1.5
shot "w${idx}-open"
click_through "Accept" "I accept" "Accept terms" "Confirm" "Continue" "Withdraw" "Next" "OK" "Agree" "Bestätigen" "Abheben" "Akzeptieren"
pause 3.0
shot "w${idx}-after"
adb_sh am start -n "$ACT" >/dev/null 2>&1 || true
pause 0.6
tap_text "Assets" 2>/dev/null || true
pause 0.4
shot "w${idx}-assets"
done
############################
step "3 pay chain (limit=$PAY_LIMIT)"
pidx=0
for PAY in "${PAYS[@]}"; do
pidx=$((pidx + 1))
[[ "$pidx" -gt "$PAY_LIMIT" ]] && break
step "3.$pidx pay $PAY"
if ! PAYURI="$(mint_pay "$PAY" | head -1)"; then
info "mint_pay failed for $PAY — skip"
continue
fi
[[ -n "$PAYURI" ]] || { info "empty pay uri — skip"; continue; }
PAYURI=$(printf '%s' "$PAYURI" | sed 's/:443\//\//g; s/:443?/?/g; s/:80\//\//g')
info "PAYURI=$PAYURI"
echo "$PAYURI" >>"$SHOTDIR/pay-uris.txt"
adb_sh am start -a android.intent.action.VIEW -d "$PAYURI" "$PKG" >/dev/null
pause 1.5
shot "p${pidx}-open"
click_through "Pay" "Confirm" "Accept" "Continue" "OK" "Next" "Bezahlen" "Bestätigen" "Payer"
pause 1.5
shot "p${pidx}-after"
adb_sh am start -n "$ACT" >/dev/null 2>&1 || true
pause 0.5
done
step "4 done → $SHOTDIR"
ls -la "$SHOTDIR" | tail -40
adb_sh dumpsys window 2>/dev/null | grep mCurrentFocus | head -1 || true
info "PKG=$PKG amounts=${AMOUNTS[*]} pays(limit)=$PAY_LIMIT"
echo "See GUI-AUTOMATION-NOTES.md (ported from taler-android dev/hernani-inference/gui-workflows)"

90
android-test/start-android-emulator.sh Executable file
View file

@ -0,0 +1,90 @@
#!/usr/bin/env bash
# Start the Taler Android AVD. **Default = headless** (no host window).
# Guest GLES still runs via SwiftShader so uiautomator / screenshots work.
#
# Usage:
# ./start-android-emulator.sh # headless, background
# ./start-android-emulator.sh --wait # headless + wait for boot
# ./start-android-emulator.sh --windowed # show emulator window
# EMULATOR_HEADLESS=0 ./start-android-emulator.sh
# EMULATOR_GPU=host WINDOWED=1 ./start-android-emulator.sh
#
# Env: EMULATOR_AVD EMULATOR_MEMORY EMULATOR_CORES EMULATOR_GPU
# EMULATOR_HEADLESS EMULATOR_LOG EMULATOR_BOOT_TIMEOUT EMULATOR_EXTRA_ARGS
#
set -euo pipefail
ROOT=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=lib_android_env.sh
. "$ROOT/lib_android_env.sh"
WAIT=0
for a in "$@"; do
case "$a" in
--wait) WAIT=1 ;;
--windowed| --gui) EMULATOR_HEADLESS=0; export EMULATOR_HEADLESS ;;
--headless) EMULATOR_HEADLESS=1; export EMULATOR_HEADLESS ;;
-h|--help)
sed -n '2,18p' "$0"
exit 0
;;
esac
done
command -v emulator >/dev/null || {
echo "emulator not found (ANDROID_HOME=$ANDROID_HOME)" >&2
exit 2
}
command -v adb >/dev/null || {
echo "adb not found" >&2
exit 2
}
# Already booted?
if serial=$(android_adb_serial); [ -n "$serial" ]; then
boot=$(adb -s "$serial" shell getprop sys.boot_completed 2>/dev/null | tr -d '\r' || true)
if [ "$boot" = "1" ]; then
echo "already running: $serial (boot_completed=1) — headless default still applies to *new* starts"
exit 0
fi
fi
# Avoid pkill patterns that match this wrapper's argv (self-kill).
mapfile -t EMU_ARGS < <(android_emulator_args "$EMULATOR_AVD")
echo "starting: emulator ${EMU_ARGS[*]}"
echo " headless=$EMULATOR_HEADLESS gpu=$EMULATOR_GPU log=$EMULATOR_LOG"
if [ "${EMULATOR_HEADLESS}" = "1" ]; then
export QT_QPA_PLATFORM="${QT_QPA_PLATFORM:-offscreen}"
# Do not require DISPLAY
unset DISPLAY || true
fi
mkdir -p "$(dirname "$EMULATOR_LOG")"
# Background; do not use pkill -f emulator from callers matching full cmdline
nohup emulator "${EMU_ARGS[@]}" >"$EMULATOR_LOG" 2>&1 &
echo "emulator pid=$! log=$EMULATOR_LOG"
if [ "$WAIT" != "1" ]; then
echo "not waiting (pass --wait for boot). adb devices when ready."
exit 0
fi
echo "waiting for boot (timeout ${EMULATOR_BOOT_TIMEOUT}s)…"
deadline=$((SECONDS + EMULATOR_BOOT_TIMEOUT))
serial=""
while (( SECONDS < deadline )); do
adb wait-for-device 2>/dev/null || true
serial=$(android_adb_serial || true)
if [ -n "$serial" ]; then
boot=$(adb -s "$serial" shell getprop sys.boot_completed 2>/dev/null | tr -d '\r' || true)
if [ "$boot" = "1" ]; then
echo "BOOT_OK serial=$serial headless=$EMULATOR_HEADLESS gpu=$EMULATOR_GPU"
exit 0
fi
fi
sleep 2
done
echo "boot timeout — last log:" >&2
tail -30 "$EMULATOR_LOG" >&2 || true
exit 3

384
check_apt_deploy.sh Executable file
View file

@ -0,0 +1,384 @@
#!/usr/bin/env bash
# check_apt_deploy.sh — podman apt-src merchant deploy tests on koopa.
#
# Containers (default):
# Fresh (rebuild when repo versions change):
# koopa-taler-deploy-test-apt-src-trixie
# koopa-taler-deploy-test-apt-src-trixie-testing
# Upgrade track (initial install, then mytops-style apt upgrade):
# koopa-taler-deploy-test-apt-src-trixie-upgrade
# koopa-taler-deploy-test-apt-src-trixie-testing-upgrade
#
# Emits a compact container board (board · key=value) for the HTML sticky/top
# overview: which pods are active and which fail (httpd / ldd / …).
#
# Env:
# APT_DEPLOY_CONTAINERS "name:suite:mode …" mode=fresh|upgrade (optional, default fresh)
# APT_DEPLOY_SKIP=1
# APT_DEPLOY_PODMAN=podman
#
set -euo pipefail
ROOT=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=lib.sh
source "$ROOT/lib.sh"
if [ "${APT_DEPLOY_SKIP:-0}" = "1" ]; then
echo "[INFO] skip phase aptdeploy (APT_DEPLOY_SKIP=1)"
exit 0
fi
set_area aptdeploy
section "aptdeploy · podman apt-src merchant (fresh + upgrade tracks)"
PODMAN_BIN="${APT_DEPLOY_PODMAN:-podman}"
DEFAULT_LIST="\
koopa-taler-deploy-test-apt-src-trixie:trixie:fresh \
koopa-taler-deploy-test-apt-src-trixie-testing:trixie-testing:fresh \
koopa-taler-deploy-test-apt-src-trixie-upgrade:trixie:upgrade \
koopa-taler-deploy-test-apt-src-trixie-testing-upgrade:trixie-testing:upgrade"
LIST="${APT_DEPLOY_CONTAINERS:-$DEFAULT_LIST}"
if ! command -v "$PODMAN_BIN" >/dev/null 2>&1; then
err "podman" "podman binary missing ($PODMAN_BIN)"
summary
exit 1
fi
ok "podman" "$("$PODMAN_BIN" --version 2>/dev/null | head -1)"
info "host" "$(hostname 2>/dev/null || echo unknown) · whoami=$(whoami)"
fail_any=0
# board rows for end overview (human table + machine board · lines)
BOARD_ROWS=()
_short_name() {
# koopa-taler-deploy-test-apt-src-trixie-testing-upgrade → trixie-testing-upgrade
local n="$1"
n="${n#koopa-taler-deploy-test-apt-src-}"
printf '%s' "$n"
}
# Record one container for the top-of-page board (HTML parses "board · key=value").
board_record() {
local short="$1" name="$2" suite="$3" mode="$4" state="$5"
local merchant="$6" httpd="$7" ldd="$8" verdict="$9" note="${10:-}"
local kv
kv="short=${short} name=${name} suite=${suite} mode=${mode} state=${state}"
kv+=" merchant=${merchant} httpd=${httpd} ldd=${ldd} verdict=${verdict}"
# note: no spaces (HTML/parser splits on whitespace)
if [ -n "$note" ]; then
note_safe=$(printf '%s' "$note" | tr ' ' '_' | tr -d '|')
kv+=" note=${note_safe}"
fi
info "board" "$kv"
BOARD_ROWS+=("$short|$suite|$mode|$state|$merchant|$httpd|$ldd|$verdict|$note")
}
print_board_table() {
section "aptdeploy · container overview (active / errors)"
info "board-table" "| short | suite | mode | podman | merchant | httpd | ldd | verdict | note |"
info "board-table" "|-------|-------|------|--------|----------|-------|-----|---------|------|"
local r short suite mode state merchant httpd ldd verdict note
local n_ok=0 n_err=0 n_miss=0
for r in "${BOARD_ROWS[@]+"${BOARD_ROWS[@]}"}"; do
IFS='|' read -r short suite mode state merchant httpd ldd verdict note <<<"$r"
info "board-table" "| ${short} | ${suite} | ${mode} | ${state} | ${merchant} | ${httpd} | ${ldd} | ${verdict} | ${note} |"
case "$verdict" in
OK|ok) n_ok=$((n_ok + 1)) ;;
MISSING|missing) n_miss=$((n_miss + 1)) ;;
*) n_err=$((n_err + 1)) ;;
esac
done
info "board-summary" "containers=${#BOARD_ROWS[@]} ok=${n_ok} error=${n_err} missing=${n_miss}"
# one-line human skim (INFO/OK only — do not re-emit ERROR, details already counted)
for r in "${BOARD_ROWS[@]+"${BOARD_ROWS[@]}"}"; do
IFS='|' read -r short suite mode state merchant httpd ldd verdict note <<<"$r"
if [ "$verdict" = "OK" ]; then
ok "overview" "${short}: ${state} · merchant=${merchant} · httpd=${httpd} · ldd=${ldd}"
else
info "overview" "${short}: ${verdict} · state=${state} merchant=${merchant} httpd=${httpd} ldd=${ldd}${note:+ · ${note}}"
fi
done
}
print_pkg_table() {
local name=$1
local rows
rows=$("$PODMAN_BIN" exec "$name" bash -lc '
echo "| package | version |"
echo "|---------|---------|"
for p in taler-merchant libtalermerchant libtalerexchange libdonau libgnunet \
taler-merchant-webui taler-merchant-typst taler-terms-generator; do
v=$(dpkg-query -W -f="\${Version}" "$p" 2>/dev/null || echo missing)
echo "| $p | $v |"
done
' 2>/dev/null || echo "| (query failed) | |")
info "pkg-table" "$name"
while IFS= read -r line; do
[ -n "$line" ] || continue
info "pkg" "$line"
done <<<"$rows"
}
# merchant package version only (for board)
_merchant_ver() {
local name=$1
"$PODMAN_BIN" exec "$name" dpkg-query -W -f='${Version}' taler-merchant 2>/dev/null || echo "?"
}
check_ldd_deep() {
local name=$1
local out
out=$("$PODMAN_BIN" exec "$name" bash -lc '
set +e
bin=$(command -v taler-merchant-httpd)
echo "=== ldd taler-merchant-httpd ==="
ldd "$bin" 2>&1 | grep -E "libtalerutil|libdonau|libgnunet|not found" || true
echo "=== ldd libdonau (if present) ==="
for f in /usr/lib/*/libdonau.so* /usr/lib/*/libdonauutil.so*; do
[ -e "$f" ] || continue
echo "-- $f --"
ldd "$f" 2>&1 | grep -E "libtalerutil|not found" || true
done
echo "=== libtalerutil files ==="
ls -la /usr/lib/*/libtalerutil* 2>/dev/null || echo "(none)"
' 2>/dev/null || echo "ldd probe failed")
while IFS= read -r line; do
[ -n "$line" ] || continue
info "ldd" "$line"
done <<<"$out"
if echo "$out" | grep -q 'not found'; then
err "ldd" "shared library missing in $name" "see ldd lines above"
return 1
fi
return 0
}
check_systemd() {
local name=$1
local out active_httpd
out=$("$PODMAN_BIN" exec "$name" bash -lc '
set +e
echo "system: $(systemctl is-system-running 2>&1)"
echo "target_enabled: $(systemctl is-enabled taler-merchant.target 2>&1)"
echo "target_active: $(systemctl is-active taler-merchant.target 2>&1)"
echo "httpd_active: $(systemctl is-active taler-merchant-httpd.service 2>&1)"
echo "httpd_enabled: $(systemctl is-enabled taler-merchant-httpd.service 2>&1)"
systemctl start taler-merchant.target 2>&1 | tail -5
sleep 2
echo "after_start_target: $(systemctl is-active taler-merchant.target 2>&1)"
echo "after_start_httpd: $(systemctl is-active taler-merchant-httpd.service 2>&1)"
systemctl --failed --no-legend 2>/dev/null | grep -i taler || echo "failed_taler: none"
' 2>/dev/null || echo "systemd probe failed")
while IFS= read -r line; do
[ -n "$line" ] || continue
info "systemd" "$line"
done <<<"$out"
active_httpd=$(echo "$out" | sed -n 's/^after_start_httpd: //p' | tail -1)
# export for caller board
APT_LAST_HTTPD="${active_httpd:-?}"
if [ "$active_httpd" = "active" ]; then
ok "systemd httpd" "taler-merchant-httpd active after start taler-merchant.target"
return 0
fi
err "systemd httpd" "taler-merchant-httpd not active" "after_start_httpd=${active_httpd:-?} · see systemd lines"
return 1
}
check_merchant_basics() {
local name=$1
local out
out=$("$PODMAN_BIN" exec "$name" bash -lc '
set +e
echo "=== binaries ==="
command -v taler-merchant-httpd taler-merchant-dbinit taler-config 2>/dev/null
echo "=== --version ==="
taler-merchant-httpd --version 2>&1
echo "=== config probe ==="
for u in \
http://127.0.0.1:9966/config \
http://127.0.0.1:8081/config \
http://127.0.0.1/config
do
code=$(curl -sS -m 2 -o /tmp/mcfg -w "%{http_code}" "$u" 2>/dev/null || echo 000)
echo "curl $u -> $code"
[ "$code" = "200" ] && head -c 120 /tmp/mcfg && echo
done
sock=$(ls /run/taler-merchant/httpd/*.sock 2>/dev/null | head -1)
if [ -n "$sock" ]; then
echo "socket: $sock"
code=$(curl -sS -m 2 --unix-socket "$sock" -o /tmp/mcfg -w "%{http_code}" http://localhost/config 2>/dev/null || echo 000)
echo "curl --unix-socket -> $code"
[ "$code" = "200" ] && head -c 160 /tmp/mcfg && echo
else
echo "socket: (none under /run/taler-merchant/httpd/)"
fi
' 2>/dev/null || echo "basics probe failed")
while IFS= read -r line; do
[ -n "$line" ] || continue
info "basics" "$line"
done <<<"$out"
if echo "$out" | grep -qi 'error while loading shared libraries'; then
err "httpd" "shared library error on --version" "see basics/ldd"
return 1
fi
set +e
"$PODMAN_BIN" exec "$name" taler-merchant-httpd --version >/dev/null 2>&1
local ec=$?
set -e
if [ "$ec" -eq 0 ]; then
ok "httpd --version" "exit 0"
else
err "httpd --version" "exit $ec"
return 1
fi
if echo "$out" | grep -qE 'curl .* -> 200'; then
ok "merchant /config" "HTTP 200 (local)"
else
warn "merchant /config" "no local HTTP 200 (unit may need BASE_URL/db — still report version/ldd)"
fi
return 0
}
check_one() {
local name="$1" expect_suite="$2" mode="${3:-fresh}"
local st short merchant="-" httpd="-" ldd="-" verdict="OK" note="" c_fail=0
short="$(_short_name "$name")"
APT_LAST_HTTPD="-"
# group ids: aptdeploy.trixie-01 / aptdeploy.trixie-testing-upgrade-01
set_group "${expect_suite}${mode:+-$mode}"
section "aptdeploy · $name (suite=$expect_suite mode=$mode)"
if ! "$PODMAN_BIN" container exists "$name" 2>/dev/null; then
err "container" "$name missing" "run ensure-apt-deploy-test-containers.sh"
board_record "$short" "$name" "$expect_suite" "$mode" "missing" "-" "-" "-" "MISSING" "ensure containers"
fail_any=1
return
fi
st=$("$PODMAN_BIN" inspect -f '{{.State.Status}}' "$name" 2>/dev/null || echo unknown)
if [ "$st" != "running" ]; then
err "container" "$name not running" "status=$st"
board_record "$short" "$name" "$expect_suite" "$mode" "$st" "-" "-" "-" "ERROR" "not running"
fail_any=1
return
fi
ok "container" "$name running (mode=$mode)"
suite_line=$("$PODMAN_BIN" exec "$name" bash -lc \
'grep -h "^Suites:" /etc/apt/sources.list.d/*taler* 2>/dev/null | head -1' 2>/dev/null || true)
if echo "$suite_line" | grep -qE "Suites:[[:space:]]*${expect_suite}([[:space:]]|$)"; then
ok "apt suite" "$expect_suite"
else
err "apt suite" "expected $expect_suite" "got: ${suite_line:-empty}"
c_fail=1
note="suite mismatch"
fi
merchant="$(_merchant_ver "$name")"
print_pkg_table "$name"
if ! check_merchant_basics "$name"; then
c_fail=1
if ! check_ldd_deep "$name"; then
ldd="FAIL"
c_fail=1
else
ldd="OK"
fi
else
ldd_out=$("$PODMAN_BIN" exec "$name" bash -lc \
'ldd "$(command -v taler-merchant-httpd)" 2>&1 | grep -E "libtalerutil|not found" || true' 2>/dev/null || true)
if echo "$ldd_out" | grep -q 'not found'; then
err "libtalerutil" "not found" "$ldd_out"
check_ldd_deep "$name" || true
ldd="FAIL"
c_fail=1
else
ldd="OK"
info "libtalerutil" "$(echo "$ldd_out" | tr '\n' ' ')"
fi
fi
if ! check_systemd "$name"; then
c_fail=1
httpd="${APT_LAST_HTTPD:-FAIL}"
# deeper ldd if httpd failed
if [ "$ldd" != "FAIL" ]; then
check_ldd_deep "$name" || { ldd="FAIL"; c_fail=1; }
fi
else
httpd="active"
fi
# normalize httpd cell
[ -z "$httpd" ] || [ "$httpd" = "-" ] && httpd="${APT_LAST_HTTPD:-?}"
# Disk free inside this deploy-test container (+ host once, first container only)
if [ "${_APT_DEPLOY_HOST_DISK_DONE:-0}" != "1" ]; then
set_group disk
mon_disk_check_host "host" || fail_any=1
_APT_DEPLOY_HOST_DISK_DONE=1
fi
set_group disk
if ! mon_disk_check_podman "$name" "$PODMAN_BIN"; then
c_fail=1
note="${note:+$note; }disk"
fi
if [ "$c_fail" -ne 0 ]; then
verdict="ERROR"
fail_any=1
else
verdict="OK"
fi
board_record "$short" "$name" "$expect_suite" "$mode" "running" "$merchant" "$httpd" "$ldd" "$verdict" "$note"
}
# --- inventory first (what exists before checks) ---
section "aptdeploy · inventory (podman)"
info "inventory" "expected tracks: trixie/trixie-testing × fresh/upgrade (4 containers)"
for entry in $LIST; do
cname=${entry%%:*}
rest=${entry#*:}
case "$rest" in
*:*) suite=${rest%%:*}; mode=${rest#*:} ;;
*) suite=$rest; mode=fresh ;;
esac
short="$(_short_name "$cname")"
if "$PODMAN_BIN" container exists "$cname" 2>/dev/null; then
st=$("$PODMAN_BIN" inspect -f '{{.State.Status}}' "$cname" 2>/dev/null || echo unknown)
ok "inventory" "${short}: ${st} (${suite}/${mode})"
else
warn "inventory" "${short}: missing (${suite}/${mode})" "will ERROR in check"
fi
done
for entry in $LIST; do
cname=${entry%%:*}
rest=${entry#*:}
case "$rest" in
*:*)
suite=${rest%%:*}
mode=${rest#*:}
;;
*)
suite=$rest
mode=fresh
;;
esac
check_one "$cname" "$suite" "$mode"
done
# Compact matrix + overview ERROR lines (jump targets for HTML)
print_board_table
summary
if [ "$fail_any" -ne 0 ]; then
exit 1
fi
exit 0

704
check_auth401.sh Executable file
View file

@ -0,0 +1,704 @@
#!/usr/bin/env bash
# auth401 — merchant HTTP 401 / case-sensitivity matrix (WebUI-shaped flows)
#
# Exercises the same APIs the merchant SPA uses for:
# • self-provision signup (POST /instances) — mixed-case id
# • login / createAccessToken (POST …/private/token) — Basic user case
# • password change (POST …/private/auth)
# • admin-create (POST /management/instances) when admin bearer available
# • access-token + private/ with Bearer
#
# Backend fact (#11340): Basic-auth *username* is case-sensitive; URL path is
# usually not. SPA must always lowercase the username (createAccessToken, signup
# id, admin-create, pw change re-login). These tests assert the *raw API*
# behaviour so a missing SPA lowercasing shows up as 401 where 200 is wanted
# only after lowercasing.
#
# ./taler-monitoring.sh -d stage.monnaie.lefrancpaysan.ch auth401
# ./taler-monitoring.sh -d hacktivism.ch auth401
#
# Continue through all groups / collect every ERROR (optional):
# AUTH401_CONTINUE=1 ./taler-monitoring.sh -d stage… auth401
# CONTINUE_ON_ERROR=1 ./taler-monitoring.sh -d stage… auth401 # same
#
# Env: see secrets.env.example (AUTH401_*).
#
# NOTE: we intentionally do NOT use `set -e`. A single failed curl/grep under
# set -e would abort with no ERROR summary — the opposite of what monitoring
# needs. Optional CONTINUE_ON_ERROR still controls hard prereq early-exit.
set -uo pipefail
ROOT=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=lib.sh
source "$ROOT/lib.sh"
load_monitoring_secrets_env 2>/dev/null || true
tmp=$(mktemp -d)
AUTH401_FINISHED=0
_auth401_finish() {
[ "${AUTH401_FINISHED}" = "1" ] && return 0
AUTH401_FINISHED=1
# summary() returns non-zero when FAIL_N>0 — never let that skip the print
summary || true
rm -rf "${tmp:-}"
if [ "${FAIL_N:-0}" -gt 0 ]; then
exit 1
fi
exit 0
}
trap '_auth401_finish' EXIT
# ---------------------------------------------------------------------------
# Parameters
# ---------------------------------------------------------------------------
: "${AUTH401_TIMEOUT:=${TIMEOUT:-20}}"
: "${AUTH401_CREATE:=1}"
: "${AUTH401_INSTANCE:=}"
: "${AUTH401_PASSWORD:=${MERCHANT_INSTANCE_PASSWORD:-${AUTH401_PASS:-}}}"
: "${AUTH401_ID_PREFIX:=mon401}"
: "${AUTH401_SKIP_IF_MFA:=1}"
: "${AUTH401_BEARER:=${E2E_MERCHANT_TOKEN:-${MERCHANT_TOKEN:-}}}"
: "${AUTH401_CREATE_PASSWORD:=${AUTH401_PASSWORD:-Mon401-Test-Pass!}}"
: "${AUTH401_DURABLE_ID:=mon401}"
# Prefer durable alone (skip throwaway) — default 0 so full matrix always runs
: "${AUTH401_PREFER_DURABLE:=0}"
# Run expanded groups (signup/login/pwchange/admin/webui). Default 1.
: "${AUTH401_FULL:=1}"
# Admin bearer for POST /management/instances (optional)
: "${AUTH401_ADMIN_TOKEN:=${MERCHANT_ADMIN_TOKEN:-}}"
: "${AUTH401_ADMIN_PASSWORD:=}"
: "${AUTH401_ADMIN_USER:=admin}"
# 1 = do not abort on hard prereq fail; run remaining groups (default 0)
# Alias: CONTINUE_ON_ERROR=1
: "${AUTH401_CONTINUE:=${CONTINUE_ON_ERROR:-0}}"
# Never rotate password on durable account (default 1 — keep mon401.password valid)
: "${AUTH401_PWCHANGE_THROWAY_ONLY:=1}"
TIMEOUT="$AUTH401_TIMEOUT"
export TIMEOUT
_auth401_abort_or_continue() {
local why="${1:-prereq failed}"
if [ "${AUTH401_CONTINUE}" = "1" ]; then
warn "continue" "${why} (AUTH401_CONTINUE=1 / CONTINUE_ON_ERROR=1)"
return 0
fi
# hard stop — EXIT trap prints summary + all ERRORS collected so far
exit 1
}
BASE="${MERCHANT_PUBLIC%/}"
if [ -z "$BASE" ]; then
echo "MERCHANT_PUBLIC empty — set -d domain or --merchant" >&2
exit 2
fi
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
expect_http() {
local label="$1" want="$2" got="$3" detail="${4:-}"
case ",$want," in
*",$got,"*) ok "$label" "HTTP $got${detail:+ · $detail}" ;;
*) fail "$label" "HTTP $got (want $want)${detail:+ · $detail}" ;;
esac
}
to_lower() { printf '%s' "$1" | tr '[:upper:]' '[:lower:]'; }
to_upper() { printf '%s' "$1" | tr '[:lower:]' '[:upper:]'; }
# First char upper, rest lower → synthetic MIX for Basic user (always ≠ low if len>0)
synth_mix() {
python3 -c "s='''$1'''; print((s[:1].upper()+s[1:].lower()) if s else s)"
}
# First 3 upper (partial mix)
synth_partmix() {
python3 -c "s='''$1'''; print(s[:3].upper()+s[3:] if len(s)>=3 else s.upper())"
}
# Normalize curl status: "200", "000", never "200000" / "000000"
_http_norm() {
local raw="$1" dig
dig=$(printf '%s' "$raw" | tr -cd '0-9')
if [ -z "$dig" ]; then
printf '000'
else
# first 3 digits = real code (curl 200 then || echo 000 → 200000)
printf '%s' "$dig" | head -c 3
fi
}
token_http() {
local path_id="$1" user="$2" password="$3"
local b64 raw
b64=$(printf '%s' "${user}:${password}" | base64 -w0 2>/dev/null \
|| printf '%s' "${user}:${password}" | base64)
raw=$(curl -skS --max-redirs 0 -m "${TIMEOUT}" \
-o "$tmp/last_token.json" -w '%{http_code}' \
-X POST "${BASE}/instances/${path_id}/private/token" \
-H "Authorization: Basic ${b64}" \
-H 'Content-Type: application/json' \
-d '{"scope":"write","duration":{"d_us":3600000000},"refreshable":true}' \
2>/dev/null || true)
_http_norm "${raw:-}"
}
json_code_hint() {
python3 - "$tmp/last_token.json" <<'PY' 2>/dev/null || true
import json, sys
try:
d = json.load(open(sys.argv[1]))
except Exception:
print("")
raise SystemExit
c = d.get("code", "")
h = (d.get("hint") or d.get("detail") or "")[:70]
print(f"code={c} {h}".strip())
PY
}
extract_token() {
python3 - "$tmp/last_token.json" <<'PY' 2>/dev/null || true
import json, sys
try:
d = json.load(open(sys.argv[1]))
except Exception:
print("")
raise SystemExit(0)
print(d.get("access_token") or d.get("token") or "")
PY
}
instance_body() {
local id="$1" pass="$2" name="${3:-auth401 probe}"
python3 - "$id" "$pass" "$name" <<'PY'
import json, sys
print(json.dumps({
"address": {},
"auth": {"method": "token", "password": sys.argv[2]},
"default_pay_delay": {"d_us": 60000000},
"default_wire_transfer_delay": {"d_us": 30000000},
"default_refund_delay": {"d_us": 30000000},
"id": sys.argv[1],
"jurisdiction": {},
"name": sys.argv[3],
"use_stefan": True,
}))
PY
}
auth_change_http() {
local path_id="$1" bearer="$2" new_pass="$3"
local tok="$bearer" body
case "$tok" in secret-token:*) ;; *) tok="secret-token:${tok}" ;; esac
body=$(NEW_PASS="$new_pass" python3 -c 'import json,os; print(json.dumps({"method":"token","password":os.environ["NEW_PASS"]}))')
curl -skS --max-redirs 0 -m "${TIMEOUT}" \
-o "$tmp/auth_change.json" -w '%{http_code}' \
-X POST "${BASE}/instances/${path_id}/private/auth" \
-H "Authorization: Bearer ${tok}" \
-H 'Content-Type: application/json' \
-d "$body" \
2>/dev/null || echo 000
}
# ---------------------------------------------------------------------------
# Stage secrets (quiet)
# ---------------------------------------------------------------------------
_auth401_stage_secret_roots() {
printf '%s\n' \
"${FRANCPAYSAN_SECRETS:-}" \
"${HOME}/francpaysan-secrets" \
"${HOME}/src/francpaysan-secrets" \
"${HOME}/taler/src/francpaysan-secrets"
}
_auth401_resolve_secrets_quiet() {
local root inst_id pass
for root in $(_auth401_stage_secret_roots); do
[ -n "$root" ] && [ -d "$root" ] || continue
# durable ids: prefer mon401fix when mon401 was rotated by old tests
if [ -z "${DURABLE_PASS:-}" ]; then
for inst_id in "${AUTH401_DURABLE_ID}" mon401fix mon401 "${MERCHANT_INSTANCE:-}"; do
[ -n "$inst_id" ] || continue
pass=""
if [ -f "$root/stage/instances/${inst_id}.password" ]; then
pass=$(tr -d '\n\r' <"$root/stage/instances/${inst_id}.password")
fi
# skip empty password files
if [ -n "$pass" ]; then
# verify file is for this id (mon401.password may be stale copy of mon401fix)
DURABLE_PASS="$pass"
DURABLE_ID=$(to_lower "$inst_id")
AUTH401_SECRET_NOTE="durable ${inst_id} from ${root}/stage/instances/"
# only set AUTH401_PASSWORD if caller did not pin an instance
if [ -z "${AUTH401_PASSWORD}" ] && [ -z "${AUTH401_INSTANCE}" ]; then
AUTH401_PASSWORD="$pass"
fi
break
fi
done
fi
if [ -z "${AUTH401_BEARER}" ] && [ -f "$root/stage/default-instance-token.txt" ]; then
AUTH401_BEARER=$(tr -d '\n\r' <"$root/stage/default-instance-token.txt")
AUTH401_BEARER_NOTE="default-instance-token.txt"
fi
if [ -z "${AUTH401_ADMIN_TOKEN}" ] && [ -f "$root/stage/admin-instance-token.txt" ]; then
AUTH401_ADMIN_TOKEN=$(tr -d '\n\r' <"$root/stage/admin-instance-token.txt")
AUTH401_ADMIN_NOTE="admin-instance-token.txt"
fi
[ -n "${DURABLE_PASS:-}" ] && break
done
# also keep AUTH401_INSTANCE for durable-only mode
if [ "${AUTH401_PREFER_DURABLE}" = "1" ] && [ -n "${DURABLE_ID:-}" ] && [ -n "${DURABLE_PASS:-}" ]; then
AUTH401_INSTANCE="${AUTH401_INSTANCE:-$DURABLE_ID}"
AUTH401_PASSWORD="${AUTH401_PASSWORD:-$DURABLE_PASS}"
AUTH401_CREATE=0
AUTH401_MODE_NOTE="PREFER_DURABLE=1 → only durable ${AUTH401_INSTANCE}"
fi
}
_auth401_resolve_secrets_quiet
: "${AUTH401_CREATE_PASSWORD:=${AUTH401_PASSWORD:-Mon401-Test-Pass!}}"
export AUTH401_INSTANCE AUTH401_PASSWORD AUTH401_BEARER AUTH401_CREATE AUTH401_ADMIN_TOKEN
# ---------------------------------------------------------------------------
set_area auth401
section "auth401 · merchant 401 / case matrix (WebUI APIs) · ${BASE}"
if [ -n "${AUTH401_SECRET_NOTE:-}" ]; then info "stage secret" "$AUTH401_SECRET_NOTE"; fi
if [ -n "${AUTH401_BEARER_NOTE:-}" ]; then info "stage secret" "AUTH401_BEARER from ${AUTH401_BEARER_NOTE}"; fi
if [ -n "${AUTH401_ADMIN_NOTE:-}" ]; then info "stage secret" "AUTH401_ADMIN_TOKEN from ${AUTH401_ADMIN_NOTE}"; fi
if [ -n "${AUTH401_MODE_NOTE:-}" ]; then info "auth401 mode" "$AUTH401_MODE_NOTE"; fi
info "auth401 scope" "FULL=${AUTH401_FULL} CREATE=${AUTH401_CREATE} PREFER_DURABLE=${AUTH401_PREFER_DURABLE} CONTINUE=${AUTH401_CONTINUE}"
if [ "${AUTH401_CONTINUE}" = "1" ]; then
info "continue mode" "CONTINUE_ON_ERROR/AUTH401_CONTINUE=1 — hard prereqs soft; always full ERROR list via EXIT trap"
fi
# ===========================================================================
set_group config
# ===========================================================================
cfg_code=$(http_body "$BASE/config" "$tmp/mer-config.json")
expect_http "merchant /config" 200 "$cfg_code" "$BASE/config"
HAVE_SP=0
MANDATORY_TAN=0
if [ "$cfg_code" = "200" ]; then
eval "$(python3 - "$tmp/mer-config.json" <<'PY'
import json, sys
d = json.load(open(sys.argv[1]))
sp = 1 if d.get("have_self_provisioning") else 0
tans = d.get("mandatory_tan_channels") or []
print(f"HAVE_SP={sp}")
print(f"MANDATORY_TAN={1 if tans else 0}")
print(f"TAN_CH={','.join(tans) if tans else ''}")
print(f"API_VER={d.get('version','')}")
PY
)"
info "self-provision" "have_self_provisioning=${HAVE_SP} mandatory_tan=${MANDATORY_TAN}${TAN_CH:+ ($TAN_CH)} api=${API_VER:-?}"
else
fail "merchant config required for auth401" "HTTP $cfg_code"
_auth401_abort_or_continue "no /config"
fi
# ===========================================================================
set_group webui
# SPA surface: /webui/ loads + version stamp; optional toLowerCase in bundle
# ===========================================================================
if [ "${AUTH401_FULL}" = "1" ]; then
wcode=$(http_code "$BASE/webui/")
case "$wcode" in
200|301|302) ok "webui /" "HTTP $wcode" ;;
*) warn "webui /" "HTTP $wcode" ;;
esac
vcode=$(http_body "$BASE/webui/version.txt" "$tmp/spa-ver.txt" 2>/dev/null || echo 000)
if [ "$vcode" = "200" ] && [ -s "$tmp/spa-ver.txt" ]; then
ok "webui version.txt" "$(tr -d '\n\r' <"$tmp/spa-ver.txt")"
else
warn "webui version.txt" "HTTP ${vcode:-000}"
fi
# Download a slice of index.js — look for lowercasing mitigation (#11340)
jcode=$(curl -skS -m 60 -o "$tmp/spa-index.js" -w '%{http_code}' "$BASE/webui/index.js" 2>/dev/null || echo 000)
jcode=$(printf '%s' "$jcode" | tr -cd '0-9' | head -c 3)
if [ "$jcode" = "200" ] && [ -s "$tmp/spa-index.js" ]; then
ok "webui index.js" "HTTP 200 · $(wc -c <"$tmp/spa-index.js" | tr -d ' ') bytes"
if grep -q 'toLowerCase' "$tmp/spa-index.js" 2>/dev/null; then
ok "webui bundle toLowerCase" "present (SPA may mitigate #11340)"
else
warn "webui bundle toLowerCase" "not found in index.js — login MIX may 401 without lowercasing"
fi
else
warn "webui index.js" "HTTP ${jcode:-000}"
fi
fi
# ===========================================================================
set_group signup
# WebUI self-provision: POST /instances with mixed-case id (as typed)
# ===========================================================================
CREATED=0
USE_ID=""
USE_PASS=""
ID_MIX=""
ID_LOW=""
CREATE_PASS="$AUTH401_CREATE_PASSWORD"
stamp=$(date +%H%M%S)
# Mixed-case id like the SPA would send before lowercasing (or if it forgot)
ID_MIX="$(printf '%s' "${AUTH401_ID_PREFIX}" | awk '{print toupper(substr($0,1,1)) substr($0,2)}')X${stamp}"
ID_LOW=$(to_lower "$ID_MIX")
if [ "${AUTH401_FULL}" = "1" ] && [ "$AUTH401_CREATE" = "1" ] && [ "$HAVE_SP" = "1" ]; then
if [ "$MANDATORY_TAN" = "1" ] && [ "$AUTH401_SKIP_IF_MFA" = "1" ]; then
warn "signup skipped" "mandatory_tan_channels set — set AUTH401_INSTANCE+PASSWORD"
else
body=$(instance_body "$ID_MIX" "$CREATE_PASS" "auth401 signup MIX")
ccode=$(_http_norm "$(curl -skS --max-redirs 0 -m "${TIMEOUT}" \
-o "$tmp/create.json" -w '%{http_code}' \
-X POST "${BASE}/instances" \
-H 'Content-Type: application/json' \
-d "$body" 2>/dev/null || true)")
case "$ccode" in
200|204)
ok "signup POST /instances id=MIX" "HTTP $ccode id=$ID_MIX (SPA self-provision)"
CREATED=1
USE_ID="$ID_MIX"
USE_PASS="$CREATE_PASS"
# Immediate login as SPA after signup: must use *lowercase* Basic user
code=$(token_http "$ID_LOW" "$ID_LOW" "$CREATE_PASS")
expect_http "signup→login SPA-sim path=low user=low → 200" 200 "$code" "$(json_code_hint)"
# Bug path: SPA forgot toLowerCase on login after signup with MIX id typed
code=$(token_http "$ID_LOW" "$ID_MIX" "$CREATE_PASS")
expect_http "signup→login RAW user=MIX (no lower) → 401" 401 "$code" "$(json_code_hint)"
;;
202)
warn "signup needs MFA" "HTTP 202 — use AUTH401_INSTANCE"
;;
*)
fail "signup POST /instances" "HTTP $ccode $(head -c 120 "$tmp/create.json" 2>/dev/null | tr '\n' ' ')"
;;
esac
fi
elif [ -n "${AUTH401_INSTANCE:-}" ] && [ -n "${AUTH401_PASSWORD:-}" ]; then
USE_ID=$(to_lower "$AUTH401_INSTANCE")
USE_PASS="$AUTH401_PASSWORD"
ID_LOW=$(to_lower "$USE_ID")
ID_MIX=$(synth_mix "$ID_LOW")
info "signup" "using existing AUTH401_INSTANCE=$USE_ID (no POST /instances)"
ok "using existing instance" "$USE_ID"
else
warn "signup skipped" "CREATE=${AUTH401_CREATE} HAVE_SP=${HAVE_SP} (need instance for matrix)"
fi
# Fallback durable
if [ -z "$USE_ID" ] && [ -n "${DURABLE_ID:-}" ] && [ -n "${DURABLE_PASS:-}" ]; then
USE_ID="$DURABLE_ID"
USE_PASS="$DURABLE_PASS"
ID_LOW=$(to_lower "$USE_ID")
ID_MIX=$(synth_mix "$ID_LOW")
info "fallback durable" "$USE_ID"
ok "using durable instance" "$USE_ID"
fi
if [ -z "$USE_ID" ]; then
fail "no instance under test" "need signup OK or AUTH401_INSTANCE+PASSWORD or mon401.password"
_auth401_abort_or_continue "no instance under test"
# CONTINUE: synthesize dummy ids so later groups still run (expect many 404/401)
ID_LOW_USE="auth401-missing-instance"
ID_USER_MIX=$(synth_mix "$ID_LOW_USE")
ID_PATH_MIX="$ID_USER_MIX"
ID_UPPER=$(to_upper "$ID_LOW_USE")
ID_PARTMIX=$(synth_partmix "$ID_LOW_USE")
PASS_OK="invalid"
PASS_BAD='auth401-wrong-password-NOT-VALID'
USE_ID="$ID_LOW_USE"
USE_PASS="$PASS_OK"
fi
ID_LOW_USE=$(to_lower "$USE_ID")
# Always synthetic case variants (even for mon401) so MIX ≠ low
ID_USER_MIX=$(synth_mix "$ID_LOW_USE")
ID_PATH_MIX="$ID_USER_MIX"
ID_UPPER=$(to_upper "$ID_LOW_USE")
ID_PARTMIX=$(synth_partmix "$ID_LOW_USE")
PASS_OK="$USE_PASS"
PASS_BAD='auth401-wrong-password-NOT-VALID'
info "probe ids" "low=${ID_LOW_USE} MIX_user=${ID_USER_MIX} UPPER=${ID_UPPER} created=${CREATED} pass_len=${#PASS_OK}"
# ===========================================================================
set_group login
# WebUI login = POST …/private/token with Basic (username, password)
# ===========================================================================
if [ "${AUTH401_FULL}" = "1" ]; then
code=$(token_http "$ID_LOW_USE" "$ID_LOW_USE" "$PASS_OK")
expect_http "login SPA-sim user=low pwd=OK → 200" 200 "$code" "$(json_code_hint)"
code=$(token_http "$ID_LOW_USE" "$ID_USER_MIX" "$PASS_OK")
expect_http "login RAW user=MIX pwd=OK → 401 (#11340)" 401 "$code" "$(json_code_hint)"
code=$(token_http "$ID_LOW_USE" "$ID_UPPER" "$PASS_OK")
expect_http "login RAW user=UPPER pwd=OK → 401" 401 "$code" "$(json_code_hint)"
code=$(token_http "$ID_PATH_MIX" "$ID_LOW_USE" "$PASS_OK")
expect_http "login path=MIX user=low pwd=OK → 200 (path tolerant)" 200 "$code" "$(json_code_hint)"
code=$(token_http "$ID_UPPER" "$ID_LOW_USE" "$PASS_OK")
expect_http "login path=UPPER user=low pwd=OK → 200" 200 "$code" "$(json_code_hint)"
fi
# ===========================================================================
set_group case
# Full Basic case matrix (always hard expects — synthetic MIX)
# ===========================================================================
code=$(token_http "$ID_PATH_MIX" "$ID_USER_MIX" "$PASS_OK")
expect_http "path=MIX user=MIX pwd=OK → 401 (case)" 401 "$code" "$(json_code_hint)"
code=$(token_http "$ID_PATH_MIX" "$ID_LOW_USE" "$PASS_OK")
expect_http "path=MIX user=low pwd=OK → 200" 200 "$code" "$(json_code_hint)"
code=$(token_http "$ID_LOW_USE" "$ID_USER_MIX" "$PASS_OK")
expect_http "path=low user=MIX pwd=OK → 401 (case)" 401 "$code" "$(json_code_hint)"
code=$(token_http "$ID_LOW_USE" "$ID_LOW_USE" "$PASS_OK")
expect_http "path=low user=low pwd=OK → 200" 200 "$code" "$(json_code_hint)"
code=$(token_http "$ID_UPPER" "$ID_LOW_USE" "$PASS_OK")
expect_http "path=UPPER user=low pwd=OK → 200" 200 "$code" "$(json_code_hint)"
code=$(token_http "$ID_LOW_USE" "$ID_UPPER" "$PASS_OK")
expect_http "path=low user=UPPER pwd=OK → 401 (case)" 401 "$code" "$(json_code_hint)"
code=$(token_http "$ID_LOW_USE" "$ID_PARTMIX" "$PASS_OK")
expect_http "path=low user=MiX pwd=OK → 401 (case)" 401 "$code" "$(json_code_hint)"
# ===========================================================================
set_group password
# ===========================================================================
code=$(token_http "$ID_LOW_USE" "$ID_LOW_USE" "$PASS_BAD")
expect_http "pwd=BAD → 401" 401 "$code" "$(json_code_hint)"
code=$(token_http "$ID_LOW_USE" "$ID_LOW_USE" "")
expect_http "pwd=EMPTY → 401" 401 "$code" "$(json_code_hint)"
code=$(token_http "$ID_PATH_MIX" "$ID_USER_MIX" "$PASS_BAD")
expect_http "path=MIX user=MIX pwd=BAD → 401" 401 "$code" "$(json_code_hint)"
# ===========================================================================
set_group pwchange
# WebUI password change: POST private/auth then re-login
# Only on throwaway signup instances — never rotate durable mon401.password
# ===========================================================================
if [ "${AUTH401_FULL}" = "1" ]; then
if [ "${AUTH401_PWCHANGE_THROWAY_ONLY}" = "1" ] && [ "${CREATED:-0}" != "1" ]; then
info "pwchange skipped" "not a throwaway instance (CREATED=0) — refuse to rotate durable password"
else
code=$(token_http "$ID_LOW_USE" "$ID_LOW_USE" "$PASS_OK")
if [ "$code" != "200" ]; then
fail "pwchange: login for token" "HTTP $code — skip pwchange"
else
TOK=$(extract_token)
if [ -z "$TOK" ]; then
fail "pwchange: parse token" "empty access_token"
else
ok "pwchange: got bearer" "len=${#TOK}"
NEW_PASS="Auth401-New-${stamp}!"
ac=$(_http_norm "$(auth_change_http "$ID_LOW_USE" "$TOK" "$NEW_PASS")")
case "$ac" in
200|204)
ok "pwchange POST private/auth" "HTTP $ac (SPA password form)"
code=$(token_http "$ID_LOW_USE" "$ID_LOW_USE" "$PASS_OK")
expect_http "pwchange: old pwd → 401" 401 "$code" "$(json_code_hint)"
code=$(token_http "$ID_LOW_USE" "$ID_LOW_USE" "$NEW_PASS")
expect_http "pwchange: new pwd user=low → 200" 200 "$code" "$(json_code_hint)"
code=$(token_http "$ID_LOW_USE" "$ID_USER_MIX" "$NEW_PASS")
expect_http "pwchange: new pwd user=MIX → 401" 401 "$code" "$(json_code_hint)"
PASS_OK="$NEW_PASS"
USE_PASS="$NEW_PASS"
;;
202)
warn "pwchange needs MFA" "HTTP 202"
;;
*)
fail "pwchange POST private/auth" "HTTP $ac $(head -c 100 "$tmp/auth_change.json" 2>/dev/null | tr '\n' ' ')"
;;
esac
fi
fi
fi
fi
# ===========================================================================
set_group admin
# Admin UI: POST /management/instances (needs admin bearer)
# ===========================================================================
if [ "${AUTH401_FULL}" = "1" ]; then
ADMIN_TOK="${AUTH401_ADMIN_TOKEN:-}"
# Optional: login as admin user with password
if [ -z "$ADMIN_TOK" ] && [ -n "${AUTH401_ADMIN_PASSWORD}" ]; then
code=$(token_http "$AUTH401_ADMIN_USER" "$AUTH401_ADMIN_USER" "$AUTH401_ADMIN_PASSWORD")
if [ "$code" = "200" ]; then
ADMIN_TOK=$(extract_token)
ok "admin login user=${AUTH401_ADMIN_USER}" "got token"
else
# try lower only
au=$(to_lower "$AUTH401_ADMIN_USER")
code=$(token_http "$au" "$au" "$AUTH401_ADMIN_PASSWORD")
if [ "$code" = "200" ]; then
ADMIN_TOK=$(extract_token)
ok "admin login user=${au}" "got token"
else
warn "admin login" "HTTP $code — set AUTH401_ADMIN_TOKEN"
fi
fi
fi
if [ -z "$ADMIN_TOK" ]; then
info "admin-create skipped" "no AUTH401_ADMIN_TOKEN / AUTH401_ADMIN_PASSWORD (stage default is not admin)"
else
case "$ADMIN_TOK" in secret-token:*) ;; *) ADMIN_TOK="secret-token:${ADMIN_TOK}" ;; esac
AID_MIX="Adm401X${stamp}"
AID_LOW=$(to_lower "$AID_MIX")
APASS="Adm401-Pass-${stamp}!"
body=$(instance_body "$AID_MIX" "$APASS" "auth401 admin-create MIX")
acode=$(curl -skS --max-redirs 0 -m "${TIMEOUT}" \
-o "$tmp/admin_create.json" -w '%{http_code}' \
-X POST "${BASE}/management/instances" \
-H "Authorization: Bearer ${ADMIN_TOK}" \
-H 'Content-Type: application/json' \
-d "$body" 2>/dev/null || echo 000)
case "$acode" in
200|204)
ok "admin-create POST /management/instances id=MIX" "HTTP $acode id=$AID_MIX"
code=$(token_http "$AID_LOW" "$AID_LOW" "$APASS")
expect_http "admin-create→login user=low → 200" 200 "$code" "$(json_code_hint)"
code=$(token_http "$AID_LOW" "$AID_MIX" "$APASS")
expect_http "admin-create→login user=MIX → 401" 401 "$code" "$(json_code_hint)"
# admin-set auth on new instance
code=$(token_http "$AID_LOW" "$AID_LOW" "$APASS")
if [ "$code" = "200" ]; then
AT=$(extract_token)
NEW_A="Adm401-New-${stamp}!"
ch=$(auth_change_http "$AID_LOW" "$AT" "$NEW_A")
case "$ch" in
200|204)
ok "admin-created instance pwchange" "HTTP $ch"
code=$(token_http "$AID_LOW" "$AID_LOW" "$NEW_A")
expect_http "admin-created re-login new pwd → 200" 200 "$code" "$(json_code_hint)"
;;
*) warn "admin-created pwchange" "HTTP $ch" ;;
esac
fi
;;
401|403)
warn "admin-create unauthorized" "HTTP $acode — token not admin (expected on stage default)"
;;
202)
warn "admin-create MFA" "HTTP 202"
;;
*)
fail "admin-create" "HTTP $acode $(head -c 120 "$tmp/admin_create.json" 2>/dev/null | tr '\n' ' ')"
;;
esac
fi
fi
# ===========================================================================
set_group missing
# ===========================================================================
code=$(curl -skS --max-redirs 0 -m "${TIMEOUT}" \
-o "$tmp/last_token.json" -w '%{http_code}' \
-X POST "${BASE}/instances/${ID_LOW_USE}/private/token" \
-H 'Content-Type: application/json' \
-d '{"scope":"write","duration":{"d_us":3600000000},"refreshable":true}' \
2>/dev/null || echo 000)
expect_http "private/token NO_AUTH → 401" 401 "$code" "$(json_code_hint)"
code=$(token_http "$ID_LOW_USE" "auth401-not-a-real-user" "$PASS_OK")
expect_http "user=OTHER pwd=OK → 401" 401 "$code" "$(json_code_hint)"
code=$(token_http "auth401-does-not-exist-${stamp}" "$ID_LOW_USE" "$PASS_OK")
# Must not look like auth failure (401). Prefer 404; 502 from proxy is soft.
case "$code" in
404) ok "path=missing → 404 (not 401)" "HTTP 404 · $(json_code_hint)" ;;
401) fail "path=missing must not be 401" "HTTP 401 · $(json_code_hint)" ;;
502|503) warn "path=missing" "HTTP $code (want 404; proxy flake, not auth)" ;;
*) fail "path=missing → 404 (not 401)" "HTTP $code (want 404) · $(json_code_hint)" ;;
esac
# ===========================================================================
set_group bearer
# ===========================================================================
code=$(token_http "$ID_LOW_USE" "$ID_LOW_USE" "$PASS_OK")
if [ "$code" = "200" ]; then
TOKEN=$(extract_token)
if [ -n "$TOKEN" ]; then
ok "got access_token" "len=${#TOKEN}"
code=$(curl -skS --max-redirs 0 -m "${TIMEOUT}" \
-o "$tmp/priv.json" -w '%{http_code}' \
"${BASE}/instances/${ID_LOW_USE}/private/" \
-H "Authorization: Bearer ${TOKEN}" 2>/dev/null || echo 000)
expect_http "GET private/ Bearer OK → 200" 200 "$code"
code=$(curl -skS --max-redirs 0 -m "${TIMEOUT}" \
-o "$tmp/priv.json" -w '%{http_code}' \
"${BASE}/instances/${ID_LOW_USE}/private/" \
-H "Authorization: Bearer secret-token:AUTH401INVALID000" 2>/dev/null || echo 000)
expect_http "GET private/ Bearer BAD → 401" 401 "$code"
code=$(curl -skS --max-redirs 0 -m "${TIMEOUT}" \
-o "$tmp/priv.json" -w '%{http_code}' \
"${BASE}/instances/${ID_LOW_USE}/private/" 2>/dev/null || echo 000)
expect_http "GET private/ NO_AUTH → 401" 401 "$code"
else
fail "parse access_token" "HTTP $code but no token field"
fi
else
fail "token for bearer tests" "HTTP $code"
fi
code=$(curl -skS --max-redirs 0 -m "${TIMEOUT}" \
-o "$tmp/mgmt.json" -w '%{http_code}' \
"${BASE}/management/instances" 2>/dev/null || echo 000)
expect_http "GET /management/instances NO_AUTH → 401" 401 "$code"
if [ -n "${AUTH401_BEARER:-}" ]; then
if [ -z "${AUTH401_BEARER_INSTANCE:-}" ]; then
case "${AUTH401_BEARER_NOTE:-}" in
*default-instance-token*) AUTH401_BEARER_INSTANCE=default ;;
*) AUTH401_BEARER_INSTANCE="${DURABLE_ID:-default}" ;;
esac
fi
inst=$(to_lower "$AUTH401_BEARER_INSTANCE")
tok="$AUTH401_BEARER"
case "$tok" in secret-token:*) ;; *) tok="secret-token:${tok}" ;; esac
code=$(curl -skS --max-redirs 0 -m "${TIMEOUT}" \
-o "$tmp/priv2.json" -w '%{http_code}' \
"${BASE}/instances/${inst}/private/" \
-H "Authorization: Bearer ${tok}" 2>/dev/null || echo 000)
case "$code" in
200) ok "GET private/ secrets Bearer" "HTTP 200 instance=$inst" ;;
401) warn "secrets Bearer rejected" "HTTP 401 on $inst" ;;
*) warn "secrets Bearer" "HTTP $code on $inst" ;;
esac
fi
# ===========================================================================
set_group durable
# Extra: durable mon401 still works after throwaway tests (if present)
# ===========================================================================
if [ -n "${DURABLE_ID:-}" ] && [ -n "${DURABLE_PASS:-}" ]; then
dlow=$(to_lower "$DURABLE_ID")
dmix=$(synth_mix "$dlow")
code=$(token_http "$dlow" "$dlow" "$DURABLE_PASS")
expect_http "durable ${dlow} login user=low → 200" 200 "$code" "$(json_code_hint)"
code=$(token_http "$dlow" "$dmix" "$DURABLE_PASS")
expect_http "durable ${dlow} login user=MIX → 401" 401 "$code" "$(json_code_hint)"
else
info "durable" "no mon401.password — skipped"
fi
# ===========================================================================
set_group report
# ===========================================================================
info "summary matrix" "401: Basic user case≠low | bad/empty pwd | no auth | bad bearer | wrong user · 200: user=low + path any case · 404: missing instance · SPA must toLowerCase username on login/signup/admin/pw-relogin"
if [ "$CREATED" = "1" ]; then
info "throwaway left" "id=${ID_MIX:-?} low=${ID_LOW_USE} (password rotated if pwchange ran)"
fi
# fall through → EXIT trap runs _auth401_finish (summary + full ERROR list)
:

147
check_devtesting.sh Executable file
View file

@ -0,0 +1,147 @@
#!/usr/bin/env bash
# check_devtesting.sh — fake-franken (CHF) via rusty.taler-ops.ch taler-devtesting
#
# Requires SSH to Host DEVTESTING_SSH (default: taler-rusty-devtesting →
# rusty.taler-ops.ch user devtesting, forced command taler-devtesting).
#
# Probes:
# 1) CLI reachable (fake-incoming --help)
# 2) geniban → Swiss IBAN
# 3) fake-incoming CHF amount to exchange credit IBAN (nexus IN)
# Bounce "missing reserve public key" = plumbing OK (no real wallet reserve)
# Full credit needs a real reserve pub as --subject (optional later)
#
# Env:
# DEVTESTING_SSH default taler-rusty-devtesting (or user@host)
# DEVTESTING_AMOUNT default CHF:1.00
# DEVTESTING_CREDIT_PAYTO exchange IBAN payto (auto from EXCHANGE_PUBLIC/keys if empty)
# DEVTESTING_EXCHANGE_IBAN fallback CH6808573105529100001 (TOPS CHF exchange)
# DEVTESTING_SKIP=1 skip phase
# SKIP_SSH=1 skip (same as other SSH phases)
#
set -euo pipefail
ROOT=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=lib.sh
source "$ROOT/lib.sh"
set_area devtesting
section "devtesting · fake-franken CHF (rusty taler-devtesting)"
# Do NOT honor SKIP_SSH here: stage/tops profiles set SKIP_SSH=1 for container
# inside checks, but rusty.taler-ops.ch is a separate SSH hop (devtesting key).
if [ "${DEVTESTING_SKIP:-0}" = "1" ]; then
info "skip" "DEVTESTING_SKIP=1"
exit 0
fi
: "${DEVTESTING_SSH:=}"
: "${DEVTESTING_AMOUNT:=CHF:1.00}"
: "${DEVTESTING_EXCHANGE_IBAN:=CH6808573105529100001}"
: "${SSH_CONNECT_TIMEOUT:=8}"
: "${SSH_CMD_TIMEOUT:=45}"
ssh_dt() {
# Prefer Host alias; fall back to explicit user@host if config missing
local target="$DEVTESTING_SSH"
if ! ssh -G "$target" >/dev/null 2>&1; then
target="devtesting@rusty.taler-ops.ch"
fi
if command -v timeout >/dev/null 2>&1; then
timeout "${SSH_CMD_TIMEOUT}" ssh \
-o BatchMode=yes \
-o ConnectTimeout="${SSH_CONNECT_TIMEOUT}" \
-o StrictHostKeyChecking=accept-new \
"$target" "$@"
else
ssh \
-o BatchMode=yes \
-o ConnectTimeout="${SSH_CONNECT_TIMEOUT}" \
-o StrictHostKeyChecking=accept-new \
"$target" "$@"
fi
}
set_group cli
info "ssh" "DEVTESTING_SSH=${DEVTESTING_SSH} amount=${DEVTESTING_AMOUNT}"
help_out=$(ssh_dt fake-incoming --help 2>&1) || true
if printf '%s' "$help_out" | grep -q 'fake-incoming'; then
ok "cli" "taler-devtesting fake-incoming reachable via ${DEVTESTING_SSH}"
else
fail "cli" "cannot run taler-devtesting on ${DEVTESTING_SSH}" \
"$(printf '%s' "$help_out" | tr '\n' ' ' | head -c 200)"
exit 1
fi
set_group geniban
iban_out=$(ssh_dt geniban 2>&1) || true
iban=$(printf '%s\n' "$iban_out" | grep -E '^CH[0-9A-Z]+$' | tail -1 || true)
if [ -n "$iban" ]; then
ok "geniban" "synthetic IBAN $iban"
else
fail "geniban" "expected CH… IBAN" "$(printf '%s' "$iban_out" | tr '\n' ' ' | head -c 160)"
exit 1
fi
# Credit payto: rusty libeufin-nexus only accepts the *production* exchange IBAN
# configured there (DEVTESTING_EXCHANGE_IBAN). Do not use stage /keys first-account.
set_group credit-payto
credit_payto="${DEVTESTING_CREDIT_PAYTO:-}"
if [ -z "$credit_payto" ]; then
credit_payto="payto://iban/${DEVTESTING_EXCHANGE_IBAN}?receiver-name=Taler+Operations+AG"
fi
ok "credit-payto" "$credit_payto"
# Subject: not a real reserve — nexus should bounce missing reserve public key
# (proves CHF fake-incoming path). Real reserve pub would credit wirewatch.
subject="mon-fake-franken-$(date -u +%Y%m%d%H%M%S)-$$"
debit_payto="payto://iban/${iban}?receiver-name=MonDevTest"
set_group fake-incoming
info "fake-incoming" "amount=${DEVTESTING_AMOUNT} subject=${subject}"
fi_rc=0
fi_out=$(ssh_dt fake-incoming \
--amount "${DEVTESTING_AMOUNT}" \
--subject "${subject}" \
--credit-payto "${credit_payto}" \
--debit-payto "${debit_payto}" 2>&1) || fi_rc=$?
# Normalize multi-line for matching
fi_flat=$(printf '%s' "$fi_out" | tr '\n' ' ')
if printf '%s' "$fi_flat" | grep -qiE 'Permission denied|Connection refused|Could not resolve|No such command'; then
fail "fake-incoming" "SSH/CLI failure" "$(printf '%s' "$fi_flat" | head -c 200)"
exit 1
fi
if printf '%s' "$fi_flat" | grep -qiE 'Creditor must be the exchange'; then
fail "fake-incoming" "wrong credit IBAN (not exchange)" "$(printf '%s' "$fi_flat" | head -c 220)"
exit 1
fi
if printf '%s' "$fi_flat" | grep -qiE 'Missing amount|unable to read secrets|CalledProcessError'; then
# secrets warn alone is soft if we still see IN line
if ! printf '%s' "$fi_flat" | grep -qE 'libeufin-nexus - IN .* CHF'; then
fail "fake-incoming" "nexus rejected request" "$(printf '%s' "$fi_flat" | head -c 220)"
exit 1
fi
fi
if printf '%s' "$fi_flat" | grep -qE 'libeufin-nexus - IN .* CHF:[0-9]'; then
if printf '%s' "$fi_flat" | grep -qiE 'bounced.*missing reserve public key'; then
ok "fake-incoming" "CHF IN accepted by nexus · bounced (no reserve pub) — fake-franken plumbing OK"
info "note" "use real reserve pub as --subject for wirewatch credit (not required for mon probe)"
elif printf '%s' "$fi_flat" | grep -qiE 'bounced'; then
warn "fake-incoming" "CHF IN bounced (other reason)" "$(printf '%s' "$fi_flat" | grep -oiE 'bounced[^ ]* [^ ]*' | head -c 160)"
else
ok "fake-incoming" "CHF IN recorded by nexus (no bounce line)"
fi
elif [ "$fi_rc" -eq 0 ] && printf '%s' "$fi_flat" | grep -qi 'Faking incoming CHF'; then
ok "fake-incoming" "CLI completed CHF fake-incoming"
else
fail "fake-incoming" "no CHF IN line from nexus" "rc=${fi_rc} $(printf '%s' "$fi_flat" | head -c 200)"
exit 1
fi
info "done" "devtesting fake-franken probe finished"
exit 0

1710
check_e2e.sh Executable file
View file

File diff suppressed because it is too large Load diff

1356
check_goa_ladder.sh Executable file
View file

File diff suppressed because it is too large Load diff

546
check_inside.sh Executable file
View file

@ -0,0 +1,546 @@
#!/usr/bin/env bash
# Inside status for bank / exchange / merchant containers.
#
# Access modes (shown in the log as flags):
# host-podman — this host runs podman exec into containers (no SSH).
# INSIDE_PODMAN=1 or INSIDE_MODE=local-podman, or auto on GOA host.
# ssh — from a laptop/remote: SSH to KOOPA_SSH / INSIDE_SSH, then podman there.
#
# Profiles:
# koopa (default LOCAL_STACK=1) — taler-hacktivism* containers
# stage-lfp (TESTPAYSAN) — SSH INSIDE_SSH (stagepaysan), low-priv podman
set -euo pipefail
ROOT=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=lib.sh
source "$ROOT/lib.sh"
# shellcheck source=metrics.sh
source "$ROOT/metrics.sh"
set_area inside
PROFILE="${INSIDE_PROFILE:-}"
if [ -z "$PROFILE" ]; then
if [ "${LOCAL_STACK:-0}" = "1" ]; then
PROFILE=koopa
elif [ "${EXPECT_CURRENCY:-}" = "TESTPAYSAN" ] || [ -n "${INSIDE_SSH:-}" ]; then
PROFILE=stage-lfp
else
PROFILE=koopa
fi
fi
# Resolve host-podman vs ssh before first check ID
_use_local_podman=0
if [ "$PROFILE" != "stage-lfp" ]; then
if [ "${INSIDE_PODMAN:-0}" = "1" ] || [ "${INSIDE_MODE:-}" = "local-podman" ]; then
_use_local_podman=1
elif command -v podman >/dev/null 2>&1 \
&& podman ps --format '{{.Names}}' 2>/dev/null | grep -qE 'taler-hacktivism'; then
_use_local_podman=1
fi
fi
if [ "$_use_local_podman" = "1" ]; then
INSIDE_ACCESS=host-podman
set_group host
else
INSIDE_ACCESS=ssh
set_group ssh
fi
export INSIDE_ACCESS
section "inside · collect (${PROFILE} · access=${INSIDE_ACCESS})"
info "flags" "INSIDE_ACCESS=${INSIDE_ACCESS} INSIDE_PODMAN=${INSIDE_PODMAN:-0} INSIDE_MODE=${INSIDE_MODE:-} LOCAL_STACK=${LOCAL_STACK:-0} SKIP_SSH=${SKIP_SSH:-0} KOOPA_SSH=${KOOPA_SSH:-} INSIDE_SSH=${INSIDE_SSH:-}"
# ---------------------------------------------------------------------------
# stage-lfp: low-priv stagepaysan on the FP stage host (INSIDE_SSH)
# ---------------------------------------------------------------------------
if [ "$PROFILE" = "stage-lfp" ]; then
SSH_HOST="${INSIDE_SSH:-}"
BANK_CTR="${INSIDE_BANK_CTR:-stage-lfp-bank}"
EX_CTR="${INSIDE_EXCHANGE_CTR:-stage-lfp-exchange-ansible}"
MER_CTR="${INSIDE_MERCHANT_CTR:-stage-lfp-merchant}"
BANK_PORT="${INSIDE_BANK_PORT:-9032}"
EX_PORT="${INSIDE_EXCHANGE_PORT:-9031}"
MER_PORT="${INSIDE_MERCHANT_PORT:-9030}"
DNS_BANK="${INSIDE_DNS_BANK:-stage.bank.lefrancpaysan.ch}"
DNS_EX="${INSIDE_DNS_EXCHANGE:-stage.exchange.lefrancpaysan.ch}"
DNS_MER="${INSIDE_DNS_MERCHANT:-stage.monnaie.lefrancpaysan.ch}"
# Stage remote has more podman execs than koopa; allow a bit more wall time
STAGE_SSH_T="${INSIDE_SSH_TIMEOUT:-${SSH_CMD_TIMEOUT:-24}}"
if [ "${STAGE_SSH_T}" -lt 24 ] 2>/dev/null; then STAGE_SSH_T=24; fi
if ! mon_ssh_ok "$SSH_HOST"; then
err "ssh" "cannot reach ${SSH_HOST} (stagepaysan low-priv) — set INSIDE_SSH= or SKIP"
summary
exit 1
fi
ok "ssh ${SSH_HOST}" "stagepaysan (podman, no sudo)"
# Inject names/ports into remote (ssh bash -s does not inherit local env).
RAW=$(
{
printf 'BANK_CTR=%q; EX_CTR=%q; MER_CTR=%q\n' "$BANK_CTR" "$EX_CTR" "$MER_CTR"
printf 'BANK_PORT=%q; EX_PORT=%q; MER_PORT=%q\n' "$BANK_PORT" "$EX_PORT" "$MER_PORT"
printf 'DNS_BANK=%q; DNS_EX=%q; DNS_MER=%q\n' "$DNS_BANK" "$DNS_EX" "$DNS_MER"
cat <<'REMOTE'
set +e
emit() { printf 'E|%s|%s|%s|%s\n' "$1" "$2" "$3" "$(printf '%s' "${4:-}" | tr '\n\r' ' ' | head -c 200)"; }
hc() { curl -skS -m 3 -o /tmp/mb -w '%{http_code}' "$1" 2>/dev/null || echo 000; }
hasp() { podman exec "$1" pgrep -f "$2" >/dev/null 2>&1; }
BANK="${BANK_CTR}"
EX="${EX_CTR}"
MER="${MER_CTR}"
podman ps --format '{{.Names}}' 2>/dev/null | grep -qx "$BANK" || BANK=$(podman ps --format '{{.Names}}' 2>/dev/null | grep -iE 'stage.*bank|lfp-bank' | head -1)
podman ps --format '{{.Names}}' 2>/dev/null | grep -qx "$EX" || EX=$(podman ps --format '{{.Names}}' 2>/dev/null | grep -iE 'stage.*exchange|lfp-exchange' | head -1)
podman ps --format '{{.Names}}' 2>/dev/null | grep -qx "$MER" || MER=$(podman ps --format '{{.Names}}' 2>/dev/null | grep -iE 'stage.*merchant|lfp-merchant' | head -1)
check_dns() {
local comp="$1" ctr="$2" host="$3"
local line ip code
[ -n "$host" ] || { emit "$comp" WARN "dns" "empty host"; return 1; }
line=$(podman exec "$ctr" getent ahostsv4 "$host" 2>/dev/null | head -1)
[ -z "$line" ] && line=$(podman exec "$ctr" getent hosts "$host" 2>/dev/null | head -1)
ip=$(echo "$line" | awk '{print $1}')
if [ -z "$ip" ]; then
emit "$comp" WARN "dns $host" "no resolve inside container"
return 1
fi
if [ "$ip" = "127.0.0.1" ] || [ "$ip" = "::1" ]; then
emit "$comp" WARN "dns $host" "$ip (loopback)"
return 1
fi
# 169.254.x = pasta host-gateway pin is OK for stage
code=$(podman exec "$ctr" curl -skS -m 3 -o /dev/null -w '%{http_code}' "https://${host}/config" 2>/dev/null || echo 000)
if [ "$code" = "200" ]; then
emit "$comp" OK "dns $host" "→ $ip /config=$code"
else
emit "$comp" WARN "dns $host" "→ $ip /config=$code"
fi
}
if [ -z "$BANK" ]; then emit bank ERROR container "not running"
else
emit bank INFO container "$(podman ps --filter name=^${BANK}$ --format '{{.Names}} {{.Status}}' | head -1)"
emit bank INFO ports "$(podman ps --filter name=^${BANK}$ --format '{{.Ports}}' | head -1)"
hasp "$BANK" 'MainKt serve|libeufin-bank serve' && emit bank OK libeufin "running" || emit bank ERROR libeufin "not running"
podman exec "$BANK" pg_isready -q 2>/dev/null && emit bank OK postgres "ready" || emit bank WARN postgres "pg_isready failed"
c=$(hc "http://127.0.0.1:${BANK_PORT}/config")
[ "$c" = "200" ] && emit bank OK "host :${BANK_PORT}/config" "HTTP $c" || emit bank ERROR "host :${BANK_PORT}/config" "HTTP $c"
c=$(hc "http://127.0.0.1:${BANK_PORT}/taler-integration/config")
[ "$c" = "200" ] && emit bank OK "host integration" "HTTP $c" || emit bank ERROR "host integration" "HTTP $c"
check_dns bank "$BANK" "$DNS_BANK" || true
check_dns bank "$BANK" "$DNS_EX" || true
fi
if [ -z "$EX" ]; then emit exchange ERROR container "not running"
else
emit exchange INFO container "$(podman ps --filter name=^${EX}$ --format '{{.Names}} {{.Status}}' | head -1)"
c=$(hc "http://127.0.0.1:${EX_PORT}/config")
[ "$c" = "200" ] && emit exchange OK "host :${EX_PORT}/config" "HTTP $c" || emit exchange ERROR "host :${EX_PORT}/config" "HTTP $c"
c=$(curl -sS -m 5 -o /dev/null -w '%{http_code}' "http://127.0.0.1:${EX_PORT}/keys" 2>/dev/null || echo 000)
[ "$c" = "200" ] && emit exchange OK "host :${EX_PORT}/keys" "HTTP $c" || emit exchange ERROR "host :${EX_PORT}/keys" "HTTP $c"
hasp "$EX" 'taler-exchange-httpd' && emit exchange OK httpd "running" || emit exchange ERROR httpd "not running"
hasp "$EX" 'taler-exchange-wirewatch' && emit exchange OK wirewatch "running" || emit exchange ERROR wirewatch "not running"
hasp "$EX" 'taler-exchange-aggregator' && emit exchange OK aggregator "running" || emit exchange WARN aggregator "not running"
hasp "$EX" 'taler-exchange-transfer' && emit exchange OK transfer "running" || emit exchange WARN transfer "not running"
check_dns exchange "$EX" "$DNS_BANK" || true
check_dns exchange "$EX" "$DNS_EX" || true
check_dns exchange "$EX" "$DNS_MER" || true
fi
if [ -z "$MER" ]; then emit merchant ERROR container "not running"
else
emit merchant INFO container "$(podman ps --filter name=^${MER}$ --format '{{.Names}} {{.Status}}' | head -1)"
c=$(hc "http://127.0.0.1:${MER_PORT}/config")
[ "$c" = "200" ] && emit merchant OK "host :${MER_PORT}/config" "HTTP $c" || emit merchant ERROR "host :${MER_PORT}/config" "HTTP $c"
hasp "$MER" 'taler-merchant-httpd' && emit merchant OK httpd "running" || emit merchant ERROR httpd "not running"
# Stage merchant often has no wirewatch unit — INFO not WARN
hasp "$MER" 'taler-merchant-wirewatch' && emit merchant OK wirewatch "running" || emit merchant INFO wirewatch "not running (optional on stage)"
hasp "$MER" 'taler-merchant-depositcheck' && emit merchant OK depositcheck "running" || emit merchant INFO depositcheck "not running (optional on stage)"
hasp "$MER" 'taler-merchant-exchangekeyupdate' && emit merchant OK exchangekeyupdate "running" || emit merchant INFO exchangekeyupdate "not running"
check_dns merchant "$MER" "$DNS_BANK" || true
check_dns merchant "$MER" "$DNS_EX" || true
check_dns merchant "$MER" "$DNS_MER" || true
fi
# Caddy is host-wide (often root); stagepaysan can only see the process
if pgrep -x caddy >/dev/null 2>&1; then
emit caddy OK process "running (host)"
else
emit caddy WARN process "not seen as stagepaysan (may still run as root)"
fi
echo DONE
REMOTE
} | mon_ssh_bash "$SSH_HOST" "${STAGE_SSH_T}" || true
)
if [ -z "$RAW" ] || ! echo "$RAW" | grep -q '^E|'; then
err "ssh" "stage remote timed out or empty (cap ${STAGE_SSH_T}s · host=${SSH_HOST})"
summary
exit 1
fi
_last_inside_grp=""
while IFS= read -r line; do
case "$line" in
E\|*)
IFS='|' read -r _ comp level key detail <<<"$line"
case "$comp" in
bank|exchange|merchant|caddy) _g="$comp" ;;
*) _g="ssh" ;;
esac
if [ "$_g" != "$_last_inside_grp" ]; then
set_group "$_g"
_last_inside_grp="$_g"
fi
case "$level" in
OK) ok "[$comp] $key${detail:+ ($detail)}" ;;
ERROR) err "$comp" "$key" "$detail" ;;
WARN) warn "[$comp] $key" "$detail" ;;
INFO) info "[$comp] $key" "$detail" ;;
esac
;;
esac
done <<<"$RAW"
# Stats from outside (laptop) — public HTTPS, no stagepaysan write needed
set_group stats
section "inside · public stats (outside-in)"
_probe_public_stats() {
local comp="$1" url="$2"
local body code age unix now
body=$(mktemp)
code=$(curl -skS -m 8 -o "$body" -w '%{http_code}' "$url" 2>/dev/null || echo 000)
if [ "$code" != "200" ]; then
warn "[$comp] public stats.json" "HTTP $code · $url"
rm -f "$body"
return
fi
unix=$(python3 -c 'import json,sys; d=json.load(open(sys.argv[1])); print(d.get("generated_at_unix") or 0)' "$body" 2>/dev/null || echo 0)
now=$(date +%s)
if [ "${unix:-0}" -gt 0 ] 2>/dev/null; then
age=$((now - unix))
if [ "$age" -lt 0 ]; then age=0; fi
if [ "$age" -le "${STATS_STALE_SECS:-900}" ]; then
ok "[$comp] public stats.json" "HTTP 200 age=${age}s · $url"
elif [ "$age" -le "${STATS_FAIL_SECS:-3600}" ]; then
warn "[$comp] public stats.json" "stale age=${age}s · $url"
else
err "$comp" "public stats.json" "too old age=${age}s · $url"
fi
else
ok "[$comp] public stats.json" "HTTP 200 · $url"
fi
rm -f "$body"
}
_probe_public_stats bank "https://${DNS_BANK}/intro/stats.json"
_probe_public_stats exchange "https://${DNS_EX}/intro/stats.json"
_probe_public_stats merchant "https://${DNS_MER}/intro/stats.json"
# Host load as stagepaysan (no container RSS from koopa metrics)
set_group load
section "inside · load (stagepaysan host)"
LOAD_LINE=$(mon_ssh_bash "$SSH_HOST" 8 <<'EOF' || true
python3 - <<'PY'
import os
la=os.getloadavg()
print("loadavg=%.2f,%.2f,%.2f" % la)
try:
with open("/proc/meminfo") as f:
d={}
for line in f:
k,v=line.split(":")[0], line.split(":")[1].strip().split()[0]
d[k]=int(v)
total=d.get("MemTotal",0)/1024/1024
avail=d.get("MemAvailable",0)/1024/1024
used=total-avail
print("mem_used=%.2fGiB avail=%.2fGiB total=%.2fGiB" % (used, avail, total))
except Exception:
print("mem=?")
PY
EOF
)
if [ -n "$LOAD_LINE" ]; then
info "stage host" "$(echo "$LOAD_LINE" | tr '\n' ' ')"
else
info "stage host" "load probe empty"
fi
set_group disk
section "inside · disk free space (stage host + containers)"
_disk_raw=$(mon_ssh_bash "$SSH_HOST" "${STAGE_SSH_T:-24}" <<'DISK' || true
set +e
echo "###HOST###"
df -Pk / /var /home /tmp /mnt/data 2>/dev/null || df -Pk
echo "###CTRS###"
for c in $(podman ps --format '{{.Names}}' 2>/dev/null); do
echo "###CTR $c###"
podman exec "$c" df -Pk / /var /tmp 2>/dev/null || podman exec "$c" df -Pk 2>/dev/null
done
DISK
)
_host_df=$(printf '%s\n' "$_disk_raw" | sed -n '/^###HOST###$/,/^###CTRS###$/p' | sed '1d;$d')
mon_disk_check_remote_text "ssh:${SSH_HOST}" "$_host_df" || true
_ctr=""; _buf=""
while IFS= read -r _line || [ -n "$_line" ]; do
case "$_line" in
'###CTR '*)
if [ -n "$_ctr" ]; then mon_disk_check_remote_text "ctr:${_ctr}" "$_buf" || true; fi
_ctr=${_line####CTR }; _ctr=${_ctr%###}; _buf=""
;;
'###CTRS###'|'###HOST###') ;;
*) [ -n "$_ctr" ] && _buf="${_buf}${_line}"$'\n' ;;
esac
done <<<"$_disk_raw"
[ -n "$_ctr" ] && mon_disk_check_remote_text "ctr:${_ctr}" "$_buf" || true
summary
exit 0
fi
# ---------------------------------------------------------------------------
# koopa (default) — host-podman exec (on GOA host) or SSH to KOOPA_SSH (laptop)
# ---------------------------------------------------------------------------
section "inside · collect from koopa (access=${INSIDE_ACCESS})"
if [ "$_use_local_podman" = "1" ]; then
set_group host
if ! command -v podman >/dev/null 2>&1; then
err "host" "INSIDE_PODMAN/host-podman but podman missing"
summary
exit 1
fi
ok "host→container" "podman exec on this host (INSIDE_ACCESS=host-podman · no SSH)"
elif [ "${SKIP_SSH}" = "1" ] && [ "${LOCAL_STACK:-0}" != "1" ]; then
set_group ssh
warn "ssh" "SKIP_SSH=1 and not host-podman — skipped"
summary
exit 0
elif ! koopa_ssh_ok; then
set_group ssh
err "ssh" "cannot reach ${KOOPA_SSH} — from laptop use KOOPA_SSH=; on host set INSIDE_PODMAN=1"
summary
exit 1
else
set_group ssh
ok "ssh ${KOOPA_SSH}" "remote host then podman (INSIDE_ACCESS=ssh · laptop/remote)"
fi
# Collect script (file → bash local or ssh bash -s)
_INSIDE_SCRIPT=$(mktemp)
trap 'rm -f "${_INSIDE_SCRIPT:-}"' RETURN
cat >"$_INSIDE_SCRIPT" <<'REMOTE'
set +e
emit() { printf 'E|%s|%s|%s|%s\n' "$1" "$2" "$3" "$(printf '%s' "${4:-}" | tr '\n\r' ' ' | head -c 200)"; }
# quick curl
hc() { curl -skS -m 3 -o /tmp/mb -w '%{http_code}' "$1" 2>/dev/null || echo 000; }
# quick process check inside container (pgrep only)
hasp() { podman exec "$1" pgrep -f "$2" >/dev/null 2>&1; }
BANK=$(podman ps --format '{{.Names}}' 2>/dev/null | grep -iE 'hacktivism-bank|taler-bank' | head -1)
[ -z "$BANK" ] && BANK=$(podman ps --format '{{.Names}}' 2>/dev/null | grep -i bank | head -1)
EX=$(podman ps --format '{{.Names}}' 2>/dev/null | grep -i exchange | head -1)
# Exact merchant container name first — never fall through to *-bank
MER=$(podman ps --format '{{.Names}}' 2>/dev/null | grep -E '^taler-hacktivism$' | head -1)
[ -z "$MER" ] && MER=$(podman ps --format '{{.Names}}' 2>/dev/null | grep -i merchant | grep -viE 'bank|exchange' | head -1)
[ -z "$MER" ] && MER=$(podman ps --format '{{.Names}}' 2>/dev/null | grep -E 'hacktivism' | grep -viE 'bank|exchange' | head -1)
# Domain resolve inside container (wirewatch needs bank.hacktivism.ch → real IP)
# emit: comp LEVEL dns "host → ip" or fail
check_dns() {
local comp="$1" ctr="$2" host="$3"
local line ip code
# Prefer IPv4 (wirewatch/libcurl often happier; avoid dead AAAA)
line=$(podman exec "$ctr" getent ahostsv4 "$host" 2>/dev/null | head -1)
[ -z "$line" ] && line=$(podman exec "$ctr" getent hosts "$host" 2>/dev/null | head -1)
ip=$(echo "$line" | awk '{print $1}')
if [ -z "$ip" ]; then
emit "$comp" ERROR "dns $host" "no resolve — run pin-container-hosts.sh"
return 1
fi
# 127.0.0.1 is almost always wrong for public bank/exchange from inside pasta
if [ "$ip" = "127.0.0.1" ] || [ "$ip" = "::1" ]; then
emit "$comp" ERROR "dns $host" "$ip (loopback — wirewatch will fail)"
return 1
fi
code=$(podman exec "$ctr" curl -skS -m 3 -o /dev/null -w '%{http_code}' "https://${host}/config" 2>/dev/null || echo 000)
if [ "$code" = "200" ]; then
emit "$comp" OK "dns $host" "→ $ip /config=$code"
else
emit "$comp" WARN "dns $host" "→ $ip /config=$code"
fi
}
if [ -z "$BANK" ]; then emit bank ERROR container "not running"
else
emit bank INFO container "$(podman ps --filter name=$BANK --format '{{.Names}} {{.Status}}' | head -1)"
emit bank INFO ports "$(podman ps --filter name=$BANK --format '{{.Ports}}' | head -1)"
hasp "$BANK" 'MainKt serve|libeufin-bank serve' && emit bank OK libeufin "running" || emit bank ERROR libeufin "not running — API/withdraw dead"
podman exec "$BANK" pg_isready -q 2>/dev/null && emit bank OK postgres "ready" || emit bank ERROR postgres "not ready"
c=$(hc http://127.0.0.1:9012/config)
[ "$c" = "200" ] && emit bank OK "local /config" "HTTP $c" || emit bank ERROR "local /config" "HTTP $c"
c=$(hc http://127.0.0.1:9012/taler-integration/config)
[ "$c" = "200" ] && emit bank OK "local integration" "HTTP $c" || emit bank ERROR "local integration" "HTTP $c"
hasp "$BANK" 'nginx' && emit bank OK nginx ":9013" || emit bank WARN nginx "not running"
check_dns bank "$BANK" bank.hacktivism.ch || true
check_dns bank "$BANK" exchange.hacktivism.ch || true
fi
if [ -z "$EX" ]; then emit exchange ERROR container "not running"
else
emit exchange INFO container "$(podman ps --filter name=$EX --format '{{.Names}} {{.Status}}' | head -1)"
c=$(hc http://127.0.0.1:9011/config)
[ "$c" = "200" ] && emit exchange OK "local /config" "HTTP $c" || emit exchange ERROR "local /config" "HTTP $c"
c=$(curl -sS -m 5 -o /dev/null -w '%{http_code}' http://127.0.0.1:9011/keys 2>/dev/null || echo 000)
[ "$c" = "200" ] && emit exchange OK "local /keys" "HTTP $c" || emit exchange ERROR "local /keys" "HTTP $c"
hasp "$EX" 'taler-exchange-httpd' && emit exchange OK httpd "running" || emit exchange ERROR httpd "not running"
hasp "$EX" 'taler-exchange-wirewatch' && emit exchange OK wirewatch "running" || emit exchange ERROR wirewatch "not running — withdraw stuck after bank confirm"
hasp "$EX" 'taler-exchange-aggregator' && emit exchange OK aggregator "running" || emit exchange WARN aggregator "not running"
hasp "$EX" 'taler-exchange-transfer' && emit exchange OK transfer "running" || emit exchange WARN transfer "not running"
# critical for wire gateway
check_dns exchange "$EX" bank.hacktivism.ch || true
check_dns exchange "$EX" exchange.hacktivism.ch || true
check_dns exchange "$EX" taler.hacktivism.ch || true
fi
if [ -z "$MER" ]; then emit merchant ERROR container "not running"
else
emit merchant INFO container "$(podman ps --filter name=$MER --format '{{.Names}} {{.Status}}' | head -1)"
c=$(hc https://127.0.0.1:9010/config)
[ "$c" = "200" ] && emit merchant OK "local /config" "HTTP $c" || emit merchant ERROR "local /config" "HTTP $c"
hasp "$MER" 'taler-merchant-httpd' && emit merchant OK httpd "running" || emit merchant ERROR httpd "not running"
hasp "$MER" 'taler-merchant-wirewatch' && emit merchant OK wirewatch "running" || emit merchant WARN wirewatch "not running"
# depositcheck: may abort on exchange track 404 (merchant assertion) — try ensure once
if hasp "$MER" 'taler-merchant-depositcheck'; then
emit merchant OK depositcheck "running"
else
podman exec "$MER" /usr/local/bin/ensure_merchant_helpers.sh >/dev/null 2>&1 || true
sleep 1
if hasp "$MER" 'taler-merchant-depositcheck'; then
emit merchant OK depositcheck "running (after ensure_merchant_helpers)"
elif hasp "$MER" 'taler-merchant-wirewatch'; then
# Known: depositcheck can SIGABRT on exchange 404 for stale coins — not startable cleanly
emit merchant INFO depositcheck "not running (often aborts on track 404; wirewatch OK · settlement partial)"
else
emit merchant WARN depositcheck "not running"
fi
fi
check_dns merchant "$MER" bank.hacktivism.ch || true
check_dns merchant "$MER" exchange.hacktivism.ch || true
check_dns merchant "$MER" taler.hacktivism.ch || true
fi
if systemctl is-active caddy >/dev/null 2>&1 || pgrep -x caddy >/dev/null 2>&1; then
emit caddy OK process "active"
else
emit caddy ERROR process "not active"
fi
echo DONE
REMOTE
if [ "$_use_local_podman" = "1" ]; then
RAW=$(bash "$_INSIDE_SCRIPT" || true)
else
RAW=$(koopa_ssh_bash "${SSH_CMD_TIMEOUT}" <"$_INSIDE_SCRIPT" || true)
fi
rm -f "$_INSIDE_SCRIPT"
if [ -z "$RAW" ] || ! echo "$RAW" | grep -q '^E|'; then
if [ "$_use_local_podman" = "1" ]; then
err "host" "collect empty (INSIDE_ACCESS=host-podman · podman exec failed)"
else
err "ssh" "collect empty (INSIDE_ACCESS=ssh · cap ${SSH_CMD_TIMEOUT}s · host=${KOOPA_SSH})"
fi
summary
exit 1
fi
# Fallback group for non-component rows: host (podman) vs ssh (remote)
_access_grp=ssh
[ "$_use_local_podman" = "1" ] && _access_grp=host
_last_inside_grp=""
while IFS= read -r line; do
case "$line" in
E\|*)
IFS='|' read -r _ comp level key detail <<<"$line"
# Group IDs by component: inside.bank-02, inside.exchange-04; access = host|ssh
case "$comp" in
bank|exchange|merchant|caddy) _g="$comp" ;;
*) _g="$_access_grp" ;;
esac
if [ "$_g" != "$_last_inside_grp" ]; then
set_group "$_g"
_last_inside_grp="$_g"
fi
case "$level" in
OK) ok "[$comp] $key${detail:+ ($detail)}" ;;
ERROR) err "$comp" "$key" "$detail" ;;
WARN) warn "[$comp] $key" "$detail" ;;
INFO) info "[$comp] $key" "$detail" ;;
esac
;;
esac
done <<<"$RAW"
# Host loadavg + RAM + per-container RSS/CPU (same probe as e2e/ladder)
set_group load
section "inside · load / memory"
METRICS_DIR="${METRICS_DIR:-$(mktemp -d)}"
export METRICS_DIR
metrics_report_load "${METRICS_DIR}/load-inside.json" "inside" || true
# Disk free space: host + taler containers (WARN tight, ERROR full)
set_group disk
section "inside · disk free space"
_disk_ec=0
if [ "$_use_local_podman" = "1" ]; then
mon_disk_check_host "host" || _disk_ec=1
while read -r _cname; do
[ -n "$_cname" ] || continue
mon_disk_check_podman "$_cname" || _disk_ec=1
done < <(podman ps --format '{{.Names}}' 2>/dev/null | grep -iE 'hacktivism|taler-|stage-lfp|lfp-' || true)
else
# remote host via SSH: df on host + each container
_disk_raw=$(koopa_ssh_bash "${SSH_CMD_TIMEOUT:-20}" <<'DISK' || true
set +e
echo "###HOST###"
df -Pk / /var /home /tmp 2>/dev/null || df -Pk
echo "###CTRS###"
for c in $(podman ps --format '{{.Names}}' 2>/dev/null); do
echo "###CTR $c###"
podman exec "$c" df -Pk / /var /tmp 2>/dev/null || podman exec "$c" df -Pk 2>/dev/null
done
DISK
)
_host_df=$(printf '%s\n' "$_disk_raw" | sed -n '/^###HOST###$/,/^###CTRS###$/p' | sed '1d;$d')
mon_disk_check_remote_text "ssh:${KOOPA_SSH:-remote}" "$_host_df" || _disk_ec=1
_ctr=""
_buf=""
while IFS= read -r _line || [ -n "$_line" ]; do
case "$_line" in
'###CTR '*)
if [ -n "$_ctr" ]; then
mon_disk_check_remote_text "ctr:${_ctr}" "$_buf" || _disk_ec=1
fi
_ctr=${_line####CTR }
_ctr=${_ctr%###}
_buf=""
;;
'###CTRS###'|'###HOST###') ;;
*)
if [ -n "$_ctr" ]; then
_buf="${_buf}${_line}"$'\n'
fi
;;
esac
done <<<"$_disk_raw"
if [ -n "$_ctr" ]; then
mon_disk_check_remote_text "ctr:${_ctr}" "$_buf" || _disk_ec=1
fi
fi
unset _disk_raw _host_df _ctr _buf _line _cname _disk_ec
summary

291
check_mail.sh Executable file
View file

@ -0,0 +1,291 @@
#!/usr/bin/env bash
# check_mail.sh — outside-in mail (MX / SMTP / IMAP / SPF / DMARC)
#
# Catalog: mail-catalog.conf (MAIL_CATALOG=… to override)
# Covers firefly (taler.net, gnunet.org) and pixel (taler-systems.com, …).
#
# Outside-only. Phase: mail
#
set -euo pipefail
ROOT=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=lib.sh
source "$ROOT/lib.sh"
set_area mail
section "mail · MX / SMTP / IMAP / SPF / DMARC (outside-in)"
CATALOG="${MAIL_CATALOG:-$ROOT/mail-catalog.conf}"
PORT_TIMEOUT="${MAIL_PORT_TIMEOUT:-4}"
SMTP_TIMEOUT="${MAIL_SMTP_TIMEOUT:-10}"
if [ ! -f "$CATALOG" ]; then
fail "catalog" "missing $CATALOG"
exit 1
fi
# dig or host fallback
dns_mx() {
local d="$1"
if command -v dig >/dev/null 2>&1; then
dig +short +time=3 +tries=2 MX "$d" 2>/dev/null | awk '{print tolower($2)}' | sed 's/\.$//'
else
host -t MX "$d" 2>/dev/null | awk -F'in mail is |has address ' '/mail is/{print tolower($NF)}' | sed 's/\.$//'
fi
}
dns_a() {
local h="$1"
if command -v dig >/dev/null 2>&1; then
dig +short +time=3 +tries=2 A "$h" 2>/dev/null | grep -E '^[0-9.]+$' || true
dig +short +time=3 +tries=2 AAAA "$h" 2>/dev/null | grep -E ':' || true
else
getent ahosts "$h" 2>/dev/null | awk '{print $1}' | sort -u
fi
}
dns_txt() {
local name="$1"
if command -v dig >/dev/null 2>&1; then
dig +short +time=3 +tries=2 TXT "$name" 2>/dev/null | tr -d '"'
else
host -t TXT "$name" 2>/dev/null | sed 's/.*"\(.*\)"/\1/'
fi
}
tcp_open() {
local h="$1" p="$2"
if command -v timeout >/dev/null 2>&1; then
timeout "$PORT_TIMEOUT" bash -c "echo >/dev/tcp/${h}/${p}" 2>/dev/null
else
bash -c "echo >/dev/tcp/${h}/${p}" 2>/dev/null
fi
}
# SMTP: read banner first, then EHLO (avoids "protocol synchronization")
smtp_probe() {
local host="$1" port="$2"
MAIL_HOST="$host" MAIL_PORT="$port" MAIL_TO="${SMTP_TIMEOUT}" python3 - <<'PY' 2>/dev/null
import os, socket, sys
host = os.environ["MAIL_HOST"]
port = int(os.environ["MAIL_PORT"])
to = float(os.environ.get("MAIL_TO", "10"))
try:
s = socket.create_connection((host, port), to)
s.settimeout(to)
banner = s.recv(1024).decode("utf-8", "replace").strip().split("\n")[0]
if not banner.startswith("220"):
print(f"bad_banner={banner[:80]}")
sys.exit(1)
s.sendall(b"EHLO taler-monitoring.invalid\r\n")
data = b""
while True:
chunk = s.recv(4096)
if not chunk:
break
data += chunk
if b"\n" in chunk and (data.endswith(b"\r\n") or len(data) > 8000):
# multi-line 250-… ends with 250 space
lines = data.decode("utf-8", "replace").splitlines()
if any(l.startswith("250 ") for l in lines):
break
if any(l.startswith("5") for l in lines[:3]):
break
text = data.decode("utf-8", "replace")
s.sendall(b"QUIT\r\n")
try:
s.recv(256)
except Exception:
pass
s.close()
starttls = "starttls" if "STARTTLS" in text.upper() else "no_starttls"
print(f"banner={banner[:60]} · {starttls}")
sys.exit(0)
except Exception as e:
print(f"err={e}")
sys.exit(1)
PY
}
# IMAP unencrypted greeting on 143, or just TCP on 993 (TLS)
imap_probe() {
local host="$1" port="$2"
if [ "$port" = "993" ] || [ "$port" = "995" ]; then
if tcp_open "$host" "$port"; then
echo "tcp_open tls_port=$port"
return 0
fi
return 1
fi
MAIL_HOST="$host" MAIL_PORT="$port" MAIL_TO="${SMTP_TIMEOUT}" python3 - <<'PY' 2>/dev/null
import os, socket, sys
host = os.environ["MAIL_HOST"]
port = int(os.environ["MAIL_PORT"])
to = float(os.environ.get("MAIL_TO", "10"))
try:
s = socket.create_connection((host, port), to)
s.settimeout(to)
banner = s.recv(1024).decode("utf-8", "replace").strip().split("\n")[0]
s.sendall(b"a001 LOGOUT\r\n")
try:
s.recv(256)
except Exception:
pass
s.close()
if banner.upper().startswith("* OK") or "IMAP" in banner.upper():
print(f"banner={banner[:70]}")
sys.exit(0)
print(f"unexpected={banner[:70]}")
sys.exit(1)
except Exception as e:
print(f"err={e}")
sys.exit(1)
PY
}
info "catalog" "$CATALOG"
# Track probed hosts to avoid duplicate SMTP checks
declare -A HOST_DONE=()
while IFS= read -r line || [ -n "$line" ]; do
line=${line%%#*}
line=$(echo "$line" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')
[ -z "$line" ] && continue
# shellcheck disable=SC2086
set -- $line
domain=$1
expect_mx=${2:-}
mail_hosts=${3:-}
smtp_ports=${4:-25,465,587}
imap_ports=${5:-143,993}
label=${6:-$domain}
set_group "$label"
info "domain" "$domain · expected_mx=$expect_mx · hosts=$mail_hosts"
# --- MX ---
mapfile -t mx_hosts < <(dns_mx "$domain" | sed '/^$/d')
if [ "${#mx_hosts[@]}" -eq 0 ]; then
fail "mx" "$domain has no MX records"
else
ok "mx" "$domain MX → ${mx_hosts[*]}"
# expected MX match (suffix / exact)
if [ -n "$expect_mx" ]; then
matched=0
IFS=',' read -ra want_list <<<"$expect_mx"
for w in "${want_list[@]}"; do
w=$(echo "$w" | tr '[:upper:]' '[:lower:]' | sed 's/\.$//')
for m in "${mx_hosts[@]}"; do
m=$(echo "$m" | tr '[:upper:]' '[:lower:]')
case "$m" in
"$w"|"$w".*|*."$w") matched=1; break ;;
esac
[ "$m" = "$w" ] && matched=1
done
[ "$matched" = "1" ] && break
done
if [ "$matched" = "1" ]; then
ok "mx expected" "$domain MX matches catalog ($expect_mx)"
else
fail "mx expected" "$domain MX ${mx_hosts[*]} · want one of: $expect_mx"
fi
fi
fi
# --- SPF / DMARC ---
spf=$(dns_txt "$domain" | tr '\n' ' ')
if echo "$spf" | grep -qi 'v=spf1'; then
ok "spf" "$domain has SPF"
else
warn "spf" "$domain no SPF TXT (v=spf1)"
fi
dmarc=$(dns_txt "_dmarc.$domain" | tr '\n' ' ')
if echo "$dmarc" | grep -qi 'v=dmarc1'; then
ok "dmarc" "$domain has DMARC"
else
warn "dmarc" "$domain no DMARC at _dmarc.$domain"
fi
# --- mail hosts: DNS + ports + banners ---
IFS=',' read -ra hosts <<<"$mail_hosts"
for h in "${hosts[@]}"; do
h=$(echo "$h" | tr '[:upper:]' '[:lower:]' | sed 's/\.$//;s/^[[:space:]]*//;s/[[:space:]]*$//')
[ -z "$h" ] && continue
addrs=$(dns_a "$h" | tr '\n' ' ')
if [ -z "${addrs// }" ]; then
fail "dns" "$h does not resolve (A/AAAA)"
continue
fi
ok "dns" "$h → $addrs"
# skip full protocol re-probe if already done
if [ "${HOST_DONE[$h]:-}" = "1" ]; then
info "host" "$h already probed this run"
continue
fi
HOST_DONE[$h]=1
IFS=',' read -ra sports <<<"$smtp_ports"
for p in "${sports[@]}"; do
p=${p// /}
[ -z "$p" ] && continue
if ! tcp_open "$h" "$p"; then
# 587 optional on pixel
case "$p" in
587) warn "smtp port" "$h:$p closed/filtered (submission)" ;;
*) fail "smtp port" "$h:$p closed/filtered" ;;
esac
continue
fi
case "$p" in
465)
ok "smtp port" "$h:$p open (SMTPS)"
;;
25|587)
if detail=$(smtp_probe "$h" "$p"); then
ok "smtp" "$h:$p $detail"
else
# port open but banner failed — still ERROR for 25
if [ "$p" = "25" ]; then
fail "smtp" "$h:$p open but SMTP handshake failed · $detail"
else
warn "smtp" "$h:$p open but handshake failed · $detail"
fi
fi
;;
*)
ok "smtp port" "$h:$p open"
;;
esac
done
IFS=',' read -ra iports <<<"$imap_ports"
for p in "${iports[@]}"; do
p=${p// /}
[ -z "$p" ] && continue
if ! tcp_open "$h" "$p"; then
fail "imap port" "$h:$p closed/filtered"
continue
fi
case "$p" in
993|995)
ok "imap port" "$h:$p open (TLS)"
;;
143|110)
if detail=$(imap_probe "$h" "$p"); then
ok "imap" "$h:$p $detail"
else
warn "imap" "$h:$p open but greeting failed · $detail"
fi
;;
*)
ok "imap port" "$h:$p open"
;;
esac
done
done
done <"$CATALOG"
info "hint" "firefly = taler.net/gnunet.org MX; pixel = taler-systems.com (mail.anastasis.lu / mail.taler-systems.com)"
exit 0

113
check_mattermost.sh Executable file
View file

@ -0,0 +1,113 @@
#!/usr/bin/env bash
# check_mattermost.sh — outside-in checks for Taler Mattermost (chat)
#
# Default host: mattermost.taler.net
# Override: MATTERMOST_PUBLIC=https://mattermost.example.org
# MATTERMOST_HOST=mattermost.example.org
#
# Outside-only (no SSH). Phase name: mattermost
#
set -euo pipefail
ROOT=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=lib.sh
source "$ROOT/lib.sh"
set_area mattermost
section "mattermost · public chat (outside-in)"
if [ -n "${MATTERMOST_PUBLIC:-}" ]; then
BASE="${MATTERMOST_PUBLIC%/}"
elif [ -n "${MATTERMOST_HOST:-}" ]; then
BASE="https://${MATTERMOST_HOST}"
else
BASE="https://mattermost.taler.net"
fi
HOST=${BASE#https://}
HOST=${HOST#http://}
HOST=${HOST%%/*}
info "target" "$BASE"
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
# GET with follow redirects (Mattermost SPA often 302 → /)
mm_get() {
local url="$1" out="$2"
curl -skS -L --max-redirs 5 -m "${TIMEOUT:-12}" -o "$out" -w '%{http_code}' "$url" 2>/dev/null || echo 000
}
# --- DNS / HTTPS landing ---
set_group landing
code=$(mm_get "$BASE/" "$tmp/root.html")
if [ "$code" = "200" ] && head -c 512 "$tmp/root.html" | grep -qiE '<!doctype html|<html'; then
ok "landing" "$BASE/ → HTTP $code HTML"
else
fail "landing" "$BASE/ → HTTP $code (want 200 HTML Mattermost SPA)"
fi
# --- Official Mattermost health API ---
set_group api
code=$(mm_get "$BASE/api/v4/system/ping" "$tmp/ping.json")
if [ "$code" != "200" ]; then
fail "system/ping" "$BASE/api/v4/system/ping → HTTP $code (want 200 JSON)"
else
# Ping returns JSON object (fields vary by MM version)
if python3 - "$tmp/ping.json" <<'PY' 2>/dev/null
import json, sys
p = sys.argv[1]
with open(p) as f:
d = json.load(f)
if not isinstance(d, dict):
sys.exit(2)
# older builds may include status; newer still return object with backends
sys.exit(0)
PY
then
ok "system/ping" "$BASE/api/v4/system/ping → HTTP 200 JSON"
else
fail "system/ping" "$BASE/api/v4/system/ping → HTTP 200 but not valid JSON object"
fi
fi
# --- Login SPA (must be served for users) ---
set_group login
code=$(mm_get "$BASE/login" "$tmp/login.html")
if [ "$code" = "200" ] && head -c 512 "$tmp/login.html" | grep -qiE '<!doctype html|<html'; then
ok "login" "$BASE/login → HTTP $code HTML"
else
fail "login" "$BASE/login → HTTP $code (want 200 HTML)"
fi
# --- TLS certificate expiry (openssl s_client if available) ---
set_group tls
if command -v openssl >/dev/null 2>&1; then
end=$(echo | openssl s_client -servername "$HOST" -connect "${HOST}:443" 2>/dev/null \
| openssl x509 -noout -enddate 2>/dev/null | sed 's/notAfter=//')
if [ -n "$end" ]; then
# days remaining
end_epoch=$(date -d "$end" +%s 2>/dev/null || date -j -f "%b %d %T %Y %Z" "$end" +%s 2>/dev/null || echo 0)
now_epoch=$(date +%s)
if [ "$end_epoch" -gt 0 ]; then
days=$(( (end_epoch - now_epoch) / 86400 ))
if [ "$days" -lt 0 ]; then
fail "tls cert" "$HOST cert expired ($end)"
elif [ "$days" -lt 14 ]; then
warn "tls cert" "$HOST expires in ${days}d ($end)"
elif [ "$days" -lt 30 ]; then
warn "tls cert" "$HOST expires in ${days}d ($end)"
else
ok "tls cert" "$HOST valid · ${days}d left · $end"
fi
else
info "tls cert" "$HOST notAfter=$end"
fi
else
warn "tls cert" "$HOST could not read certificate"
fi
else
info "tls cert" "openssl not available — skip expiry check"
fi
info "hint" "surface catalog also lists $HOST; this phase is Mattermost-specific (SPA + /api/v4/system/ping)"
exit 0

632
check_monitoring_pages.sh Executable file
View file

@ -0,0 +1,632 @@
#!/usr/bin/env bash
# check_monitoring_pages.sh — verify public monitoring HTML pages via FQDN
#
# Outside-in: curl https://<fqdn>/<path>/ and require a real HTML monitoring page.
# Catches: merchant JSON code 21 (Caddy handle missing), WP 404, empty stubs, deploy skip.
#
# Policy (v1.7.6+):
# • monpages is obligatory → missing/wrong pages are ERROR (exit 1).
# • GOA / hacktivism: full inventory of suite monitoring sites (catalog + on-disk discovery).
# • FP: only that stack’s own monitoring hosts/paths (never GOA URLs).
# • Override soft mode only with MONPAGES_REQUIRE_PUBLIC=0 (emergency / staging).
#
# Env:
# MON_HOSTS space-separated FQDNs (default from TALER_DOMAIN)
# HTML_URL_OK path for OK page (default /monitoring/)
# HTML_URL_ERR err path; checked if present on disk or MONPAGES_CHECK_ERR=1
# MONITORING_PAGE_URLS optional full URL list (overrides inventory construction)
# MONPAGES_EXTRA_URLS additional absolute URLs (space-separated)
# MONPAGES_INVENTORY auto|full|job (default auto)
# auto/full = stack inventory; job = only MON_HOSTS+paths
# MONPAGES_REQUIRE_PUBLIC default 1 (ERROR). 0 = WARN only (escape hatch).
# MONPAGES_STAGING_BASE / HTML_OUT / DEPLOY_WWW_ROOT trees for discovery
# TIMEOUT curl timeout (from lib.sh)
#
set -euo pipefail
ROOT=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=lib.sh
source "$ROOT/lib.sh"
set_area monpages
section "monpages · public monitoring HTML via FQDN (outside-in, obligatory)"
# Obligatory: public pages missing → ERROR. Soft only if explicitly disabled.
: "${MONPAGES_REQUIRE_PUBLIC:=1}"
: "${MONPAGES_INVENTORY:=auto}"
# Default hosts from domain (mirror run-host-report.sh)
if [ -z "${MON_HOSTS:-}" ]; then
case "${TALER_DOMAIN:-}" in
hacktivism.ch|koopa)
MON_HOSTS="bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch"
;;
stage.lefrancpaysan.ch|stage.*lefrancpaysan*)
MON_HOSTS="stage.bank.lefrancpaysan.ch stage.exchange.lefrancpaysan.ch stage.monnaie.lefrancpaysan.ch"
;;
lefrancpaysan.ch)
MON_HOSTS="bank.lefrancpaysan.ch exchange.lefrancpaysan.ch monnaie.lefrancpaysan.ch"
;;
*)
MON_HOSTS="${TALER_DOMAIN:-}"
;;
esac
fi
HTML_URL_OK="${HTML_URL_OK:-/monitoring/}"
HTML_URL_ERR="${HTML_URL_ERR:-/monitoring_err/}"
# ensure leading + trailing slash
_norm_path() {
local p="$1"
case "$p" in
/*) ;;
*) p="/$p" ;;
esac
case "$p" in
*/) ;;
*) p="${p}/" ;;
esac
printf '%s' "$p"
}
HTML_URL_OK="$(_norm_path "$HTML_URL_OK")"
HTML_URL_ERR="$(_norm_path "$HTML_URL_ERR")"
# Stack family: goa | fp | other — FP never checks GOA and vice versa.
_monpages_family() {
case "${TALER_DOMAIN:-}" in
*lefrancpaysan*|*francpaysan*) printf 'fp' ;;
hacktivism.ch|koopa) printf 'goa' ;;
*)
# Infer from MON_HOSTS if domain ambiguous
case " ${MON_HOSTS:-} " in
*lefrancpaysan*) printf 'fp' ;;
*hacktivism*) printf 'goa' ;;
*) printf 'other' ;;
esac
;;
esac
}
_host_allowed() {
local h="$1" fam="$2"
case "$fam" in
goa)
case "$h" in *.hacktivism.ch) return 0 ;; *) return 1 ;; esac
;;
fp)
case "$h" in *lefrancpaysan*) return 0 ;; *) return 1 ;; esac
;;
*) return 0 ;;
esac
}
# Public mon URL families (suite name "taler-monitoring" ≠ a public path):
# 1) /monitoring(+_err) — landing hosts (bare + slash)
# 2) /taler-monitoring-surface(+_err) — taler.hacktivism.ch only
# 3) /taler-monitoring-aptdeploy(+_err) — taler.hacktivism.ch only
# No /taler-monitoring page. No /taler-monitoring-mail|mattermost pages.
_path_allowed() {
case "$1" in
monitoring|monitoring_err|\
taler-monitoring-surface|taler-monitoring-surface_err|\
taler-monitoring-aptdeploy|taler-monitoring-aptdeploy_err) return 0 ;;
# explicit deny (never invent a mon page here)
taler-monitoring|taler-monitoring-mail|taler-monitoring-mattermost) return 1 ;;
*) return 1 ;;
esac
}
# Emit https://host/path/ for each …/host/path/index.html under root (max depth 3).
_discover_tree() {
local root="$1" fam="$2"
local f rel host path
[ -n "$root" ] && [ -d "$root" ] || return 0
# layout: $root/<fqdn>/<url-path>/index.html
while IFS= read -r -d '' f; do
rel="${f#"${root%/}"/}"
host="${rel%%/*}"
path="${rel#*/}"
path="${path%/index.html}"
[ -n "$host" ] && [ -n "$path" ] || continue
_host_allowed "$host" "$fam" || continue
_path_allowed "$path" || continue
printf 'https://%s/%s/\n' "$host" "$path"
done < <(find "${root%/}" -mindepth 3 -maxdepth 3 -type f -name index.html -print0 2>/dev/null)
}
# Landing stacks (GOA 3 + FP stage 3) with /monitoring/
_landing_allowed() {
printf '%s\n' \
bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch \
stage.bank.lefrancpaysan.ch stage.exchange.lefrancpaysan.ch stage.monnaie.lefrancpaysan.ch
}
# Suite catalog of OK pages that must exist for this family.
_catalog_urls() {
local fam="$1" h
case "$fam" in
goa)
for h in bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch; do
printf 'https://%s/monitoring/\n' "$h"
done
# ecosystem surface (versions / software) + apt-src deploy mon page
printf 'https://taler.hacktivism.ch/taler-monitoring-surface/\n'
printf 'https://taler.hacktivism.ch/taler-monitoring-aptdeploy/\n'
;;
fp)
for h in $MON_HOSTS; do
[ -z "$h" ] && continue
_host_allowed "$h" fp || continue
printf 'https://%s/monitoring/\n' "$h"
done
;;
all|nine)
# GOA + FP stage landings + surface + aptdeploy (nine kept as alias for inventory)
while IFS= read -r h; do
[ -n "$h" ] || continue
printf 'https://%s/monitoring/\n' "$h"
done < <(_landing_allowed)
printf 'https://taler.hacktivism.ch/taler-monitoring-surface/\n'
printf 'https://taler.hacktivism.ch/taler-monitoring-aptdeploy/\n'
;;
esac
}
# Job-local URLs (this host-agent run’s path).
_job_urls() {
local h p
for h in $MON_HOSTS; do
[ -z "$h" ] && continue
printf 'https://%s%s\n' "$h" "$HTML_URL_OK"
if [ "${MONPAGES_CHECK_ERR:-0}" = "1" ]; then
printf 'https://%s%s\n' "$h" "$HTML_URL_ERR"
fi
done
}
build_urls() {
local fam inv
fam="$(_monpages_family)"
inv="${MONPAGES_INVENTORY:-auto}"
if [ -n "${MONITORING_PAGE_URLS:-}" ]; then
# shellcheck disable=SC2086
printf '%s\n' $MONITORING_PAGE_URLS
else
case "$inv" in
job)
# Only this host-agent job’s HTML paths (MON_HOSTS + HTML_URL_OK/_ERR).
# Specialized timers (aptdeploy, surface) must not inherit GOA full catalog
# (no bank/exchange /monitoring bare-URL noise on the aptdeploy page).
_job_urls
;;
full|auto|*)
# Always include this job’s paths (host-agent post-check).
_job_urls
# Stack inventory: GOA = landings + surface + aptdeploy; FP = only FP mon hosts.
# Full inventory belongs on the general stack run (run-hacktivism-monitoring).
if [ "$fam" = "goa" ] || [ "$fam" = "fp" ]; then
_catalog_urls "$fam"
# On-disk discovery (err pages only exist after failed runs; extras welcome)
[ -n "${DEPLOY_WWW_ROOT:-}" ] && _discover_tree "${DEPLOY_WWW_ROOT}" "$fam"
_discover_tree "${MONPAGES_STAGING_BASE:-${HTML_OUT:-${HTML_BASE:-$HOME/monitoring-sites-staging}}}" "$fam"
fi
;;
esac
fi
if [ -n "${MONPAGES_EXTRA_URLS:-}" ]; then
# shellcheck disable=SC2086
printf '%s\n' $MONPAGES_EXTRA_URLS
fi
}
# Install stub only (no console log) — not a real mon page.
is_bootstrap_html() {
local f="$1"
grep -qiE 'Monitoring page \(bootstrap\)|pill">STAGE</|install stub|bootstrap page' "$f" 2>/dev/null \
&& ! grep -qE 'taler-mon:top|id="mon-console"|class="console"' "$f" 2>/dev/null
}
# True if body looks like our monitoring HTML (not merchant API / WP 404 / bootstrap).
is_monitoring_html() {
local f="$1"
if grep -qE '"code"[[:space:]]*:[[:space:]]*21' "$f" 2>/dev/null; then
return 1
fi
if grep -qiE 'There is no endpoint defined for the URL' "$f" 2>/dev/null; then
return 1
fi
if grep -qiE 'wp-content|wordpress' "$f" 2>/dev/null \
&& grep -qiE '404|not found|page introuvable' "$f" 2>/dev/null; then
return 1
fi
# Bootstrap sticky is not enough — need suite console output
if is_bootstrap_html "$f"; then
return 1
fi
if grep -qE 'sticky-bar|version-link|taler-monitoring|class="sticky|taler-mon:top' "$f" 2>/dev/null; then
return 0
fi
if head -c 4096 "$f" | grep -qiE '<!DOCTYPE html|<html' \
&& grep -qiE 'monitoring|mon ·|host-agent' "$f" 2>/dev/null; then
return 0
fi
if head -c 512 "$f" | grep -qiE '<!DOCTYPE html|<html'; then
return 0
fi
return 1
}
# Highest progress percent in page body (0 if none). Pattern: ┌ 42%┐ or ┌100%┐
_progress_max_pct() {
local f="$1"
local m
m=$(grep -oE '┌[[:space:]]*[0-9]+%┐' "$f" 2>/dev/null \
| grep -oE '[0-9]+' | sort -n | tail -1 || true)
printf '%s' "${m:-0}"
}
# Mid-run: incomplete meta / redirect stub, or progress never reached 100% and no complete foot.
_page_mid_run() {
local f="$1"
local pct
if grep -qE 'name="taler-mon-page"[[:space:]]+content="incomplete"' "$f" 2>/dev/null \
|| grep -q 'taler-mon:bottom:incomplete' "$f" 2>/dev/null \
|| grep -qE 'http-equiv="refresh".*monitoring_err|mode-redirect' "$f" 2>/dev/null; then
return 0
fi
# finished page always has complete foot or classic footer
if grep -q 'taler-mon:bottom:complete' "$f" 2>/dev/null \
|| grep -qE 'id="mon-footer"' "$f" 2>/dev/null; then
return 1
fi
pct=$(_progress_max_pct "$f")
# max progress seen < 100% and no complete footer → still running
if [ "${pct:-0}" -gt 0 ] && [ "${pct:-0}" -lt 100 ]; then
if ! grep -qE 'Console-style render of taler-monitoring|Rendu console de taler-monitoring' "$f" 2>/dev/null; then
return 0
fi
fi
return 1
}
# Validate top/bottom markers + meaningful body (v1.9.0).
# Sets _MON_CONTENT_MSG; returns 0=ok, 1=hard fail, 2=soft/mid-run bottom skip
#
# Head windows are large (v1.13.4+): sticky CSS grew past old 12–16 KiB limits and
# caused false TOP errors while publish races were also writing the file.
validate_mon_content() {
local f="$1" url="$2"
local top_ok=1 bot_ok=1 body_ok=1 mid=0 pct
local head_top="${MONPAGES_HEAD_TOP_BYTES:-49152}"
local head_meta="${MONPAGES_HEAD_META_BYTES:-65536}"
_MON_CONTENT_MSG=""
# Tiny / empty body = likely mid-publish race (partial rsync or curl)
if [ ! -s "$f" ] || [ "$(wc -c <"$f" 2>/dev/null || echo 0)" -lt 200 ]; then
top_ok=0
_MON_CONTENT_MSG="empty or truncated body (publish race?)"
# --- top markers (always required) ---
elif ! head -c "$head_top" "$f" | grep -qE 'sticky-bar|id="status-bar"|taler-mon:top'; then
# full-file fallback (small pages / unusual layout)
if ! grep -qE 'sticky-bar|id="status-bar"|taler-mon:top' "$f" 2>/dev/null; then
top_ok=0
_MON_CONTENT_MSG="missing top marker (sticky-bar / status-bar / taler-mon:top)"
fi
elif ! head -c "$head_meta" "$f" | grep -qE 'data-generated-iso|name="generated"|version-link|taler-mon-page'; then
if ! grep -qE 'data-generated-iso|name="generated"|version-link|taler-mon-page' "$f" 2>/dev/null; then
top_ok=0
_MON_CONTENT_MSG="missing top meta (generated / version-link / taler-mon-page)"
fi
fi
# --- meaningful body: real checks, not only progress bars ---
if grep -qE 'class="line (ok|error|warn|blocker|info)"|┌ OK|┌ ERROR|┌ WARN|┌ BLOCKER|┌ INFO|class="box"' "$f" 2>/dev/null; then
body_ok=1
elif grep -qE 'taler-mon:bottom:incomplete|mode-redirect|has failures' "$f" 2>/dev/null; then
body_ok=1 # redirect stub still has sticky + box
else
# only progress / empty?
if grep -qE '┌[[:space:]]*[0-9]+%┐|█|░' "$f" 2>/dev/null \
&& ! grep -qE '┌ OK|┌ ERROR|class="line ok"' "$f" 2>/dev/null; then
body_ok=0
_MON_CONTENT_MSG="${_MON_CONTENT_MSG:+$_MON_CONTENT_MSG; }only progress bars / no check lines"
elif ! grep -qE 'console|host-agent|taler-monitoring|monitoring' "$f" 2>/dev/null; then
body_ok=0
_MON_CONTENT_MSG="${_MON_CONTENT_MSG:+$_MON_CONTENT_MSG; }no meaningful suite content"
fi
fi
# --- bottom markers ---
if grep -q 'taler-mon:bottom:complete' "$f" 2>/dev/null \
|| tail -c 8000 "$f" | grep -qE 'id="mon-footer"|</footer>|Console-style render of taler-monitoring'; then
bot_ok=1
elif grep -q 'taler-mon:bottom:incomplete' "$f" 2>/dev/null; then
bot_ok=1 # incomplete is an explicit bottom marker
mid=1
else
bot_ok=0
_MON_CONTENT_MSG="${_MON_CONTENT_MSG:+$_MON_CONTENT_MSG; }missing bottom marker (footer / taler-mon:bottom)"
fi
if _page_mid_run "$f"; then
mid=1
fi
pct=$(_progress_max_pct "$f")
if [ "$top_ok" != "1" ]; then
# TOP missing is WARN-only (v1.13.8+): mid-run races and incomplete
# /monitoring stubs used to self-red the next HTML. Hard ERROR still for
# missing page / code 21 / non-suite body. Override: MONPAGES_TOP_STRICT=1.
_MON_CONTENT_MSG="TOP: $_MON_CONTENT_MSG"
if [ "${MONPAGES_TOP_STRICT:-0}" = "1" ]; then
return 1
fi
return 2
fi
if [ "$body_ok" != "1" ]; then
_MON_CONTENT_MSG="BODY: $_MON_CONTENT_MSG"
return 1
fi
if [ "$bot_ok" != "1" ]; then
if [ "$mid" = "1" ]; then
_MON_CONTENT_MSG="BOTTOM soft (mid-run progress ${pct}%): $_MON_CONTENT_MSG"
return 2
fi
_MON_CONTENT_MSG="BOTTOM: $_MON_CONTENT_MSG"
return 1
fi
if [ "$mid" = "1" ] && [ "${pct:-0}" -gt 0 ] && [ "${pct:-0}" -lt 100 ]; then
_MON_CONTENT_MSG="mid-run progress ${pct}% (bottom markers relaxed)"
return 2
fi
_MON_CONTENT_MSG="top+body+bottom markers OK"
return 0
}
_is_bare_url() {
local url="$1"
case " ${MONPAGES_BARE_URLS:-} " in
*" $url "*) return 0 ;;
esac
case "$url" in
*/) return 1 ;;
*) return 0 ;;
esac
}
_mon_fail_or_soft() {
local label="$1" detail="$2"
if [ "${MONPAGES_REQUIRE_PUBLIC:-1}" != "1" ]; then
warn "$label" "$detail (MONPAGES_REQUIRE_PUBLIC=0)"
return 0
fi
fail "$label" "$detail"
return 1
}
# Fetch + validate one URL. Retries content failures (publish/rsync races).
# MONPAGES_PRE_PUBLISH=1 → content failures are WARN only (suite still regenerates HTML after).
# MONPAGES_FETCH_RETRIES (default 3), MONPAGES_FETCH_SLEEP_S (default 0.5).
check_one() {
local url="$1"
local body code hint soft=0 vc attempt max_try sleep_s pre_pub
body=$(mktemp)
if _is_bare_url "$url" && [ "${MONPAGES_BARE_STRICT:-0}" != "1" ]; then
soft=1
fi
pre_pub=0
[ "${MONPAGES_PRE_PUBLISH:-0}" = "1" ] && pre_pub=1
max_try="${MONPAGES_FETCH_RETRIES:-3}"
sleep_s="${MONPAGES_FETCH_SLEEP_S:-0.5}"
[ "$max_try" -lt 1 ] && max_try=1
attempt=1
while [ "$attempt" -le "$max_try" ]; do
: >"$body"
code=$(curl -skS -L --max-redirs 5 -m "${TIMEOUT}" -o "$body" -w '%{http_code}' "$url" 2>/dev/null || echo 000)
if grep -qE '"code"[[:space:]]*:[[:space:]]*21' "$body" 2>/dev/null \
|| grep -qiE 'There is no endpoint defined for the URL' "$body" 2>/dev/null; then
# Bare without trailing slash often hits merchant until Caddy has redir * /path/ 302.
if [ "$soft" = "1" ]; then
warn "public mon page bare URL" \
"$url -> HTTP $code code 21 (prefer trailing slash; Caddy: redir * /path/ 302; set MONPAGES_BARE_STRICT=1 to ERROR)"
rm -f "$body"
return 0
fi
# hard 21: retry once more (rare brief outage) then fail
if [ "$attempt" -lt "$max_try" ]; then
attempt=$((attempt + 1))
sleep "$sleep_s" 2>/dev/null || sleep 1
continue
fi
_mon_fail_or_soft "public mon page missing" \
"$url -> HTTP $code merchant/API JSON code 21 (Caddy handle/redir; use redir * /path/ 302 inside handle${APPLY_MONITORING_LIVE:+; apply: sudo $APPLY_MONITORING_LIVE})" || { rm -f "$body"; return 1; }
rm -f "$body"
return 0
fi
if [ "$code" != "200" ]; then
if [ "$soft" = "1" ]; then
warn "public mon page bare URL" "$url -> HTTP $code (prefer trailing slash form)"
rm -f "$body"
return 0
fi
if [ "$attempt" -lt "$max_try" ]; then
attempt=$((attempt + 1))
sleep "$sleep_s" 2>/dev/null || sleep 1
continue
fi
_mon_fail_or_soft "public mon page" "$url -> HTTP $code (want 200 HTML)" || { rm -f "$body"; return 1; }
rm -f "$body"
return 0
fi
if is_bootstrap_html "$body"; then
if [ "$attempt" -lt "$max_try" ]; then
attempt=$((attempt + 1))
sleep "$sleep_s" 2>/dev/null || sleep 1
continue
fi
_mon_fail_or_soft "public mon page bootstrap stub" \
"$url -> install bootstrap HTML still live (run host-agent; suite auto-update must replace stub)" || { rm -f "$body"; return 1; }
rm -f "$body"
return 0
fi
if ! is_monitoring_html "$body"; then
if [ "$attempt" -lt "$max_try" ]; then
attempt=$((attempt + 1))
sleep "$sleep_s" 2>/dev/null || sleep 1
continue
fi
hint=$(head -c 120 "$body" | tr '\n' ' ' | tr -cd '[:print:] ')
_mon_fail_or_soft "public mon page not monitoring HTML" \
"$url -> HTTP 200 body!=suite (${hint}...)" || { rm -f "$body"; return 1; }
rm -f "$body"
return 0
fi
# v1.9.0: top + bottom markers + meaningful content
set +e
validate_mon_content "$body" "$url"
vc=$?
set -e
case "$vc" in
0)
ok "public mon page" "$url -> HTTP 200 · ${_MON_CONTENT_MSG}"
rm -f "$body"
return 0
;;
2)
# soft: TOP incomplete / mid-run bottom — WARN, never hard fail
case "${_MON_CONTENT_MSG}" in
TOP:*)
warn "public mon page content" \
"$url -> ${_MON_CONTENT_MSG} (WARN only; set MONPAGES_TOP_STRICT=1 to ERROR)"
;;
*)
warn "public mon page mid-run" "$url -> ${_MON_CONTENT_MSG}"
ok "public mon page" "$url -> HTTP 200 suite HTML (mid-run bottom relaxed)"
;;
esac
rm -f "$body"
return 0
;;
*)
# Content race (truncated HTML while another agent publishes): retry
if [ "$attempt" -lt "$max_try" ]; then
attempt=$((attempt + 1))
sleep "$sleep_s" 2>/dev/null || sleep 1
continue
fi
# TOP failures should already be soft (return 2); belt-and-braces:
case "${_MON_CONTENT_MSG}" in
TOP:*)
if [ "${MONPAGES_TOP_STRICT:-0}" != "1" ]; then
warn "public mon page content" \
"$url -> ${_MON_CONTENT_MSG} (WARN only; set MONPAGES_TOP_STRICT=1 to ERROR)"
rm -f "$body"
return 0
fi
;;
esac
# Pre-publish inventory in host-agent PHASES: WARN only (post-check is hard)
if [ "$pre_pub" = "1" ] || [ "$soft" = "1" ]; then
warn "public mon page content" \
"$url -> ${_MON_CONTENT_MSG}${pre_pub:+ (pre-publish; post-check is authoritative)}"
rm -f "$body"
return 0
fi
_mon_fail_or_soft "public mon page content" \
"$url -> ${_MON_CONTENT_MSG}" || { rm -f "$body"; return 1; }
rm -f "$body"
return 0
;;
esac
done
rm -f "$body"
return 0
}
URLS=()
while IFS= read -r line; do
[ -n "$line" ] || continue
# dedupe
skip=0
for e in "${URLS[@]+"${URLS[@]}"}"; do
[ "$e" = "$line" ] && skip=1 && break
done
[ "$skip" = "1" ] && continue
URLS+=("$line")
done < <(build_urls | sort -u)
# Bare URLs without trailing slash: checked by default.
# Merchant JSON code 21 is always ERROR (Caddy/proxy not serving mon path).
# Other bare failures: WARN unless MONPAGES_BARE_STRICT=1 (then ERROR).
# MONPAGES_CHECK_BARE=0 → skip bare entirely
# MONPAGES_BARE_STRICT=1 → non-21 bare failures are ERROR too
if [ "${MONPAGES_CHECK_BARE:-1}" = "1" ]; then
_extra=()
for u in "${URLS[@]}"; do
case "$u" in
*/) _bare=${u%/}; [ -n "$_bare" ] && _extra+=("$_bare") ;;
esac
done
for u in "${_extra[@]+"${_extra[@]}"}"; do
skip=0
for e in "${URLS[@]+"${URLS[@]}"}"; do
[ "$e" = "$u" ] && skip=1 && break
done
[ "$skip" = "1" ] && continue
URLS+=("$u")
MONPAGES_BARE_URLS="${MONPAGES_BARE_URLS:-} $u"
done
export MONPAGES_BARE_URLS
fi
if [ "${#URLS[@]}" -eq 0 ]; then
fail "monpages" "no URLs to check (set MON_HOSTS or MONITORING_PAGE_URLS)"
exit 1
fi
_fam="$(_monpages_family)"
echo " family=${_fam} inventory=${MONPAGES_INVENTORY} require_public=${MONPAGES_REQUIRE_PUBLIC}"
echo " checking ${#URLS[@]} URL(s) via FQDN..."
ec=0
for u in "${URLS[@]}"; do
check_one "$u" || ec=1
done
if [ "$ec" -ne 0 ]; then
echo " hint: public monitoring HTML not served (404 / merchant code 21)"
echo " publish staging HTML + configure reverse-proxy/Caddy handle for mon paths"
echo " (set APPLY_MONITORING_LIVE in env; sudo "$APPLY_MONITORING_LIVE")"
echo " bare code 21: Caddy redir inside handle must be: redir * /path/ 302 (not redir /path/ 302)"
if [ "$_fam" = "fp" ]; then
echo " FP: only FP mon hosts are checked — wire Infomaniak vhost for /monitoring*"
fi
if [ "$_fam" = "goa" ]; then
echo " GOA: bank/exchange/taler /monitoring/ + surface + aptdeploy (v1.9.4+ layout)"
fi
# Staging vs public diagnosis (host-agent sets MONPAGES_STAGING_BASE / HTML_OUT)
_stg="${MONPAGES_STAGING_BASE:-${HTML_OUT:-${HTML_BASE:-$HOME/monitoring-sites-staging}}}"
_ok_dir="${HTML_OK_DIR:-monitoring}"
if [ -n "${MON_HOSTS:-}" ] && [ -n "$_stg" ]; then
for _h in $MON_HOSTS; do
_f="$_stg/$_h/${_ok_dir}/index.html"
if [ -f "$_f" ]; then
echo " staging OK: $_f ($(wc -c <"$_f" | tr -d " ") bytes) — not published/served"
else
echo " staging missing: $_f"
fi
done
fi
_www="${DEPLOY_WWW_ROOT:-}"
if [ -e "$_www" ] && [ ! -w "$_www" ]; then
echo " DEPLOY_WWW_ROOT=$_www not writable by $(id -un 2>/dev/null || echo user)"
fi
fi
exit "$ec"

294
check_qr_payloads.py Executable file
View file

@ -0,0 +1,294 @@
#!/usr/bin/env python3
"""Collect + validate QR-related payloads for taler-monitoring (www.qr-*).
Sources (CLI paths / URLs handled by caller writing local files):
- landing HTML (href, data-qr-url, data-qr-taler, raw text)
- demo-withdraw.json / auto-account.json (taler_withdraw_uri / qr_payload)
- optional: already-decoded lines from zbarimg (kind=decoded)
Stdout TSV: source\\tkind\\turi\\tstatus\\tdetail
kind: withdraw | pay | pay-template | refund | payto | https | other | empty
status: ok | bad
Exit 0 always (caller tallies ok/bad lines). stderr for hard parse errors only.
"""
from __future__ import annotations
import json
import re
import sys
from html.parser import HTMLParser
from pathlib import Path
from urllib.parse import unquote, urlparse
# Schemes we care about for wallet / settlement QRs
TALER_RE = re.compile(r"taler://[^\s\"'<>]+", re.I)
PAYTO_RE = re.compile(r"payto://[^\s\"'<>]+", re.I)
HTTPS_RE = re.compile(r"https://[^\s\"'<>]+", re.I)
WITHDRAW_RE = re.compile(
r"^taler://withdraw/([^/]+)/taler-integration/([0-9a-fA-F-]+)/?$"
)
# order pay / template — host may include path segments after
PAY_RE = re.compile(r"^taler://pay/([^/?#]+)/", re.I)
PAY_TEMPLATE_RE = re.compile(r"^taler://pay-template/([^/?#]+)/", re.I)
REFUND_RE = re.compile(r"^taler://refund/", re.I)
PAYTO_OK_RE = re.compile(r"^payto://[a-z0-9][a-z0-9+.-]*/", re.I)
def looks_like_real_uri(uri: str) -> bool:
"""Drop JS regex fragments, docs ellipses, incomplete placeholders."""
if not uri or len(uri) < 12:
return False
if any(c in uri for c in "[]{}<>\n\r\t"):
return False
if "…" in uri or "..." in uri:
return False
# incomplete withdraw stubs in docs
if re.match(r"^taler://withdraw/?$", uri, re.I):
return False
if re.match(r"^taler://pay/?$", uri, re.I):
return False
# trailing junk / JS regex fragments (allow ? & = in query strings)
if uri.endswith((")", "(", "\\", "^", "*")):
return False
if re.search(r"[\\^$*]", uri):
return False
# lone incomplete `taler://…?` without path
if re.match(r"^taler://[^/]+$", uri, re.I):
return False
return True
def strip_default_port_host(host: str) -> str | None:
if not host or host.startswith(":"):
return None
if ":" in host:
h, _, p = host.rpartition(":")
if not h or not p.isdigit():
return None
if p in ("443", "80"):
return None # must be stripped for wallets
return host
def classify_and_validate(uri: str) -> tuple[str, str, str]:
"""Return (kind, status, detail)."""
uri = (uri or "").strip()
if not uri:
return "empty", "bad", "empty"
# strip accidental trailing punctuation from HTML scrape
uri = uri.rstrip(").,;]")
low = uri.lower()
if low.startswith("taler://withdraw/"):
m = WITHDRAW_RE.match(uri)
if not m:
return "withdraw", "bad", "shape need taler://withdraw/HOST/taler-integration/UUID"
host = m.group(1)
if strip_default_port_host(host) is None:
return "withdraw", "bad", "host/port invalid or default :443/:80 not stripped"
return "withdraw", "ok", f"host={host} id={m.group(2)[:12]}…"
if low.startswith("taler://pay-template/"):
m = PAY_TEMPLATE_RE.match(uri)
if not m:
return "pay-template", "bad", "need taler://pay-template/HOST/…"
host = m.group(1)
if strip_default_port_host(host) is None:
return "pay-template", "bad", "host/port invalid or default port not stripped"
return "pay-template", "ok", f"host={host}"
if low.startswith("taler://pay/"):
m = PAY_RE.match(uri)
if not m:
return "pay", "bad", "need taler://pay/HOST/…"
host = m.group(1)
if strip_default_port_host(host) is None:
return "pay", "bad", "host/port invalid or default port not stripped"
if "instances" not in uri and "/orders/" not in uri:
# still often valid private order URIs
return "pay", "ok", f"host={host}"
return "pay", "ok", f"host={host}"
if low.startswith("taler://refund/"):
if not REFUND_RE.match(uri):
return "refund", "bad", "bad refund URI"
return "refund", "ok", "refund"
if low.startswith("taler://"):
return "other", "ok", "other taler:// (accepted)"
if low.startswith("payto://"):
if not PAYTO_OK_RE.match(uri):
return "payto", "bad", "need payto://method/…"
# wallet pay QR must not be payto for GOA demo-withdraw (checked elsewhere)
return "payto", "ok", uri.split("?", 1)[0][:72]
if low.startswith("https://") or low.startswith("http://"):
p = urlparse(uri)
if p.scheme not in ("http", "https") or not p.netloc:
return "https", "bad", "bad URL"
return "https", "ok", p.netloc + (p.path or "/")[:40]
return "other", "bad", "unsupported scheme"
class LinkHarvester(HTMLParser):
def __init__(self) -> None:
super().__init__()
self.uris: list[tuple[str, str]] = [] # (source_hint, uri)
def handle_starttag(self, tag, attrs):
ad = {k.lower(): (v or "") for k, v in attrs}
for key in ("href", "src", "data-qr-url", "data-qr-taler", "data-uri", "data-payto"):
if key in ad and ad[key]:
self.uris.append((f"attr:{key}", ad[key].strip()))
for k, v in ad.items():
if k.startswith("data-") and v and (
"taler://" in v.lower() or "payto://" in v.lower() or k.endswith("qr-url")
):
self.uris.append((f"attr:{k}", v.strip()))
def handle_data(self, data):
if not data or "://" not in data:
return
for m in TALER_RE.finditer(data):
self.uris.append(("text", m.group(0)))
for m in PAYTO_RE.finditer(data):
self.uris.append(("text", m.group(0)))
def harvest_html(path: Path, source: str) -> list[tuple[str, str, str]]:
raw = path.read_text(encoding="utf-8", errors="replace")
h = LinkHarvester()
try:
h.feed(raw)
except Exception:
pass
out: list[tuple[str, str, str]] = []
for hint, u in h.uris:
lu = u.lower()
if lu.startswith(("taler://", "payto://", "https://", "http://")):
if looks_like_real_uri(u):
out.append((f"{source}:{hint}", u, "html"))
# raster image paths that look like QR assets (not .js)
elif re.search(r"\.(png|jpe?g|gif)(\?|$)", lu) and "qr" in lu:
out.append((f"{source}:imgpath", u, "imgpath"))
# Prefer attribute/text harvest; light regex only for complete-looking URIs in pre/code
for m in TALER_RE.finditer(raw):
u = m.group(0).rstrip(").,;]'\"")
if looks_like_real_uri(u) and (
WITHDRAW_RE.match(u)
or PAY_RE.match(u)
or PAY_TEMPLATE_RE.match(u)
or u.lower().startswith("taler://refund/")
):
out.append((f"{source}:re", u, "html"))
for m in PAYTO_RE.finditer(raw):
u = m.group(0).rstrip(").,;]'\"")
if looks_like_real_uri(u) and PAYTO_OK_RE.match(u):
out.append((f"{source}:re", u, "html"))
return out
def harvest_json(path: Path, source: str) -> list[tuple[str, str, str]]:
try:
d = json.loads(path.read_text(encoding="utf-8", errors="replace"))
except Exception as e:
return [(source, "", f"json-error:{e}")]
out: list[tuple[str, str, str]] = []
if not isinstance(d, dict):
return out
for key in (
"taler_withdraw_uri",
"qr_payload",
"taler_pay_uri",
"payto_uri",
"payto",
"uri",
):
v = d.get(key)
if isinstance(v, str) and v.strip():
out.append((f"{source}:{key}", v.strip(), "json"))
# nested
for k, v in d.items():
if isinstance(v, str) and (
v.startswith("taler://") or v.startswith("payto://")
):
out.append((f"{source}:{k}", v.strip(), "json"))
return out
def main(argv: list[str]) -> int:
# args: pairs of source_label path
if len(argv) < 3 or len(argv) % 2 == 0:
print(
"usage: check_qr_payloads.py LABEL path [LABEL path ...]",
file=sys.stderr,
)
return 2
seen: set[str] = set()
rows: list[tuple[str, str, str, str, str]] = []
i = 1
while i < len(argv):
label, path_s = argv[i], argv[i + 1]
i += 2
path = Path(path_s)
if not path.is_file():
rows.append((label, "empty", "", "bad", f"missing file {path_s}"))
continue
if path.suffix.lower() in (".html", ".htm") or "html" in path.name:
items = harvest_html(path, label)
elif path.suffix.lower() == ".json" or path.name.endswith(".json"):
items = harvest_json(path, label)
else:
# plain text: one URI per line or raw
text = path.read_text(encoding="utf-8", errors="replace").strip()
items = []
for line in text.splitlines():
line = line.strip()
if line:
items.append((label, line, "text"))
if not items and text:
items.append((label, text, "text"))
for src, uri, _origin in items:
if not uri:
continue
# skip pure fragment / relative without scheme for validate
if uri.startswith("#") or uri == "/":
continue
lu = uri.lower()
if lu.startswith("/") or "imgpath" in src:
continue
if not looks_like_real_uri(uri):
continue
# https: app-store / wallet / explicit data-qr-* only
if lu.startswith("http"):
if "data-qr" not in src and not any(
x in lu
for x in (
"play.google",
"apps.apple",
"f-droid",
"wallet.taler",
"taler.net/wallet",
)
):
continue
if uri in seen:
continue
seen.add(uri)
kind, status, detail = classify_and_validate(uri)
rows.append((src, kind, uri, status, detail))
for src, kind, uri, status, detail in rows:
# TSV — uri may contain tabs rarely; replace
safe = uri.replace("\t", " ").replace("\n", " ")
print(f"{src}\t{kind}\t{safe}\t{status}\t{detail}")
return 0
if __name__ == "__main__":
raise SystemExit(main(sys.argv))

285
check_sanity.sh Executable file
View file

@ -0,0 +1,285 @@
#!/usr/bin/env bash
# Sanity checks for bank · exchange · merchant (public + server-side).
# Sections are independent; continues after failures.
set -euo pipefail
ROOT=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=lib.sh
source "$ROOT/lib.sh"
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
expect_code() {
local label="$1" want="$2" url="$3"
local code
code=$(http_code "$url")
case ",$want," in
*",$code,"*) ok "$label" ;;
*) fail "$label" "HTTP $code (want $want) $url" ;;
esac
}
json_currency() {
python3 -c 'import json,sys; d=json.load(open(sys.argv[1])); print(d.get("currency") or "")' "$1" 2>/dev/null || true
}
# Area sanity.* — public + optional server-side per component
# Groups: sanity.bank / sanity.exchange / sanity.merchant
set_area sanity
# ---------------------------------------------------------------------------
set_group bank
section "sanity · bank"
# ---------------------------------------------------------------------------
expect_code "bank public /config" 200 "$BANK_PUBLIC/config"
expect_code "bank public /taler-integration/config" 200 "$BANK_PUBLIC/taler-integration/config"
expect_code "bank public /webui/" 200 "$BANK_PUBLIC/webui/"
code=$(http_body "$BANK_PUBLIC/config" "$tmp/bank-config.json")
if [ "$code" = "200" ]; then
cur=$(json_currency "$tmp/bank-config.json")
[ "$cur" = "GOA" ] && ok "bank currency GOA" || fail "bank currency" "got ${cur:-?}"
# wire type / name if present
python3 - "$tmp/bank-config.json" <<'PY' 2>/dev/null && ok "bank config JSON object" || fail "bank config JSON"
import json,sys
d=json.load(open(sys.argv[1]))
sys.exit(0 if isinstance(d, dict) and d.get("currency") else 1)
PY
if json_has_alt_unit_names "$tmp/bank-config.json" >/tmp/alt-bank-s.$$ 2>&1; then
ok "bank /config alt_unit_names" "$(tr '\n' '; ' </tmp/alt-bank-s.$$ | sed 's/; $//')"
else
fail "bank /config alt_unit_names" "$(tr '\n' '; ' </tmp/alt-bank-s.$$ | sed 's/; $//')"
fi
rm -f /tmp/alt-bank-s.$$
fi
# server-side bank
if koopa_ssh_ok; then
BOUT=$(koopa_ssh_bash 40 <<'REMOTE' || true
set +e
BANK=$(podman ps --format '{{.Names}}' | grep -iE 'hacktivism-bank|taler-bank' | head -1)
[ -z "$BANK" ] && BANK=$(podman ps --format '{{.Names}}' | grep -i bank | head -1)
echo "CTR=$BANK"
if [ -z "$BANK" ]; then echo "NOCTR"; exit 0; fi
code=$(curl -sS -m 5 -o /dev/null -w '%{http_code}' http://127.0.0.1:9012/config 2>/dev/null || echo 000)
echo "LOCAL_CONFIG=$code"
code2=$(curl -sS -m 5 -o /dev/null -w '%{http_code}' http://127.0.0.1:9012/taler-integration/config 2>/dev/null || echo 000)
echo "LOCAL_INT=$code2"
if podman exec "$BANK" bash -c 'pgrep -f "MainKt serve|libeufin-bank serve" >/dev/null' 2>/dev/null; then
echo "LIBEUFIN=1"
else
echo "LIBEUFIN=0"
fi
if podman exec "$BANK" bash -c 'pg_isready -q' 2>/dev/null; then
echo "PG=1"
else
echo "PG=0"
fi
# in-container health if present
if podman exec "$BANK" test -x /usr/local/bin/check_bank-health.sh 2>/dev/null; then
podman exec "$BANK" /usr/local/bin/check_bank-health.sh 2>&1 | sed 's/^/HEALTH /' | tail -20
echo "HEALTH_EC=${PIPESTATUS[0]}"
fi
REMOTE
)
echo "$BOUT" | grep -q '^CTR=.\+' && ok "bank container $(echo "$BOUT" | sed -n 's/^CTR=//p' | head -1)" || fail "bank container" "not found"
echo "$BOUT" | grep -q 'LOCAL_CONFIG=200' && ok "bank local :9012/config" || fail "bank local :9012/config"
echo "$BOUT" | grep -q 'LOCAL_INT=200' && ok "bank local :9012/taler-integration/config" || fail "bank local integration"
echo "$BOUT" | grep -q 'LIBEUFIN=1' && ok "bank libeufin-bank process" || fail "bank libeufin-bank process"
echo "$BOUT" | grep -q 'PG=1' && ok "bank postgres ready" || warn "bank postgres" "pg_isready failed"
if echo "$BOUT" | grep -q 'HEALTH '; then
if echo "$BOUT" | grep -qE 'HEALTH_EC=0|ALL CRITICAL CHECKS PASSED'; then
ok "bank check_bank-health.sh"
else
# health script may false-fail process grep; warn not fail if local config ok
warn "bank check_bank-health.sh" "non-zero or incomplete"
fi
fi
else
warn "bank server-side" "ssh ${KOOPA_SSH} unavailable"
fi
# ---------------------------------------------------------------------------
set_group exchange
section "sanity · exchange"
# ---------------------------------------------------------------------------
expect_code "exchange public /config" 200 "$EXCHANGE_PUBLIC/config"
expect_code "exchange public /keys" 200 "$EXCHANGE_PUBLIC/keys"
expect_code "exchange public /terms" 200 "$EXCHANGE_PUBLIC/terms"
code=$(http_body "$EXCHANGE_PUBLIC/config" "$tmp/ex-config.json")
if [ "$code" = "200" ]; then
cur=$(json_currency "$tmp/ex-config.json")
[ "$cur" = "GOA" ] && ok "exchange currency GOA" || fail "exchange currency" "got ${cur:-?}"
if json_has_alt_unit_names "$tmp/ex-config.json" "GOA" >/tmp/alt-ex-s.$$ 2>&1; then
ok "exchange /config alt_unit_names" "$(tr '\n' '; ' </tmp/alt-ex-s.$$ | sed 's/; $//')"
else
fail "exchange /config alt_unit_names" "$(tr '\n' '; ' </tmp/alt-ex-s.$$ | sed 's/; $//')"
fi
rm -f /tmp/alt-ex-s.$$
fi
code=$(http_body "$EXCHANGE_PUBLIC/keys" "$tmp/ex-keys.json")
if [ "$code" = "200" ]; then
python3 - "$tmp/ex-keys.json" <<'PY'
import json,sys,time
d=json.load(open(sys.argv[1]))
sk=d.get("signkeys") or []
acc=d.get("accounts") or []
den=d.get("denominations") or []
# count denoms roughly
n=0
for g in den:
if isinstance(g, dict):
n += len(g.get("denoms") or [])
print(f"signkeys={len(sk)} accounts={len(acc)} denom_groups={len(den)} denoms~={n}")
sys.exit(0 if sk and (acc or n) else 1)
PY
ec=$?
detail=$(python3 - "$tmp/ex-keys.json" <<'PY'
import json,sys
d=json.load(open(sys.argv[1]))
sk=d.get("signkeys") or []
acc=d.get("accounts") or []
den=d.get("denominations") or []
n=sum(len(g.get("denoms") or []) for g in den if isinstance(g, dict))
print(f"signkeys={len(sk)} accounts={len(acc)} denoms~={n}")
PY
)
[ "$ec" -eq 0 ] && ok "exchange /keys usable ($detail)" || fail "exchange /keys usable" "$detail"
fi
# /wire if exposed
wcode=$(http_code "$EXCHANGE_PUBLIC/wire")
if [ "$wcode" = "200" ]; then
ok "exchange public /wire"
else
warn "exchange public /wire" "HTTP $wcode (accounts may only be in /keys)"
fi
if koopa_ssh_ok; then
EOUT=$(koopa_ssh_bash 40 <<'REMOTE' || true
set +e
EX=$(podman ps --format '{{.Names}}' | grep -i exchange | head -1)
echo "CTR=$EX"
code=$(curl -sS -m 5 -o /dev/null -w '%{http_code}' http://127.0.0.1:9011/config 2>/dev/null || echo 000)
echo "LOCAL_CONFIG=$code"
codek=$(curl -sS -m 8 -o /dev/null -w '%{http_code}' http://127.0.0.1:9011/keys 2>/dev/null || echo 000)
echo "LOCAL_KEYS=$codek"
if [ -n "$EX" ]; then
if podman exec "$EX" bash -c 'pgrep -f taler-exchange-httpd >/dev/null' 2>/dev/null; then
echo "HTTPD=1"
else
echo "HTTPD=0"
fi
for p in taler-exchange-secmod-rsa taler-exchange-secmod-eddsa taler-exchange-wirewatch taler-exchange-aggregator; do
if podman exec "$EX" bash -c "pgrep -f $p >/dev/null" 2>/dev/null; then
echo "PROC_$p=1"
else
echo "PROC_$p=0"
fi
done
if podman exec "$EX" test -x /usr/local/bin/check_exchange-health.sh 2>/dev/null; then
SKIP_ENSURE=1 podman exec -e SKIP_ENSURE=1 "$EX" /usr/local/bin/check_exchange-health.sh 2>&1 | sed 's/^/HEALTH /' | tail -25
fi
fi
REMOTE
)
echo "$EOUT" | grep -q '^CTR=.\+' && ok "exchange container $(echo "$EOUT" | sed -n 's/^CTR=//p' | head -1)" || fail "exchange container"
echo "$EOUT" | grep -q 'LOCAL_CONFIG=200' && ok "exchange local :9011/config" || fail "exchange local :9011/config"
echo "$EOUT" | grep -q 'LOCAL_KEYS=200' && ok "exchange local :9011/keys" || fail "exchange local :9011/keys"
echo "$EOUT" | grep -q 'HTTPD=1' && ok "exchange-httpd process" || warn "exchange-httpd process" "not detected"
echo "$EOUT" | grep -q 'PROC_taler-exchange-wirewatch=1' && ok "exchange wirewatch" || warn "exchange wirewatch" "not running"
echo "$EOUT" | grep -q 'PROC_taler-exchange-aggregator=1' && ok "exchange aggregator" || warn "exchange aggregator" "not running"
else
warn "exchange server-side" "ssh unavailable"
fi
# ---------------------------------------------------------------------------
set_group merchant
section "sanity · merchant"
# ---------------------------------------------------------------------------
expect_code "merchant public /config" 200 "$MERCHANT_PUBLIC/config"
expect_code "merchant public /webui/" 200 "$MERCHANT_PUBLIC/webui/"
code=$(http_body "$MERCHANT_PUBLIC/config" "$tmp/mer-config.json")
if [ "$code" = "200" ]; then
if python3 - "$tmp/mer-config.json" "$EXCHANGE_PUBLIC" <<'PY'
import json,sys
d=json.load(open(sys.argv[1]))
want=sys.argv[2].rstrip("/")
curs=list((d.get("currencies") or {}).keys())
ex=d.get("exchanges") or []
urls=[]
for e in ex:
if isinstance(e, dict):
u=e.get("base_url") or e.get("url") or e.get("exchange_base_url") or ""
if u: urls.append(u.rstrip("/"))
elif isinstance(e, str):
urls.append(e.rstrip("/"))
ok_goa = "GOA" in curs or any((e.get("currency") if isinstance(e, dict) else None)=="GOA" for e in ex)
ok_ex = any(want in u or "exchange.hacktivism.ch" in u for u in urls)
print("currencies", curs)
print("exchanges", urls[:5])
sys.exit(0 if ok_goa and ok_ex else 1)
PY
then
ok "merchant config GOA + exchange.hacktivism.ch"
else
fail "merchant config GOA + exchange" "see currencies/exchanges"
fi
if json_has_alt_unit_names "$tmp/mer-config.json" >/tmp/alt-mer-s.$$ 2>&1; then
ok "merchant currencies alt_unit_names" "$(tr '\n' '; ' </tmp/alt-mer-s.$$ | sed 's/; $//')"
else
fail "merchant currencies alt_unit_names" "$(tr '\n' '; ' </tmp/alt-mer-s.$$ | sed 's/; $//')"
fi
rm -f /tmp/alt-mer-s.$$
# Each exchange listed on merchant /config must publish alt_unit_names on its own /config
check_merchant_listed_exchanges_alt_units "$tmp/mer-config.json"
fi
# demo instance reachable?
INST="${MERCHANT_INSTANCE}"
icode=$(http_code "$MERCHANT_PUBLIC/instances/${INST}/config")
if [ "$icode" = "200" ]; then
ok "merchant instance ${INST} /config"
else
# some deployments use private only
warn "merchant instance ${INST} /config" "HTTP $icode"
fi
if koopa_ssh_ok; then
MOUT=$(koopa_ssh_bash 40 <<'REMOTE' || true
set +e
MER=$(podman ps --format '{{.Names}}' | grep -E '^taler-hacktivism$' | head -1)
[ -z "$MER" ] && MER=$(podman ps --format '{{.Names}}' | grep -iE 'merchant|hacktivism' | grep -viE 'bank|exchange' | head -1)
echo "CTR=$MER"
code=$(curl -skS -m 5 -o /dev/null -w '%{http_code}' https://127.0.0.1:9010/config 2>/dev/null || echo 000)
echo "LOCAL_CONFIG=$code"
if [ -n "$MER" ]; then
if podman exec "$MER" bash -c 'pgrep -f taler-merchant-httpd >/dev/null' 2>/dev/null; then
echo "HTTPD=1"
else
echo "HTTPD=0"
fi
if podman exec "$MER" test -x /usr/local/bin/check_merchant-health.sh 2>/dev/null; then
SKIP_ENSURE=1 podman exec -e SKIP_ENSURE=1 "$MER" /usr/local/bin/check_merchant-health.sh 2>&1 | sed 's/^/HEALTH /' | tail -30
fi
fi
REMOTE
)
echo "$MOUT" | grep -q '^CTR=.\+' && ok "merchant container $(echo "$MOUT" | sed -n 's/^CTR=//p' | head -1)" || fail "merchant container"
echo "$MOUT" | grep -q 'LOCAL_CONFIG=200' && ok "merchant local :9010/config" || fail "merchant local :9010/config"
echo "$MOUT" | grep -q 'HTTPD=1' && ok "merchant-httpd process" || warn "merchant-httpd process" "not detected"
if echo "$MOUT" | grep -q 'HEALTH '; then
if echo "$MOUT" | grep -qiE 'ALL CRITICAL|HEALTH_EC=0|\[OK\]'; then
ok "merchant check_merchant-health.sh (sample OK)"
else
warn "merchant check_merchant-health.sh" "see remote output"
fi
fi
else
warn "merchant server-side" "ssh unavailable"
fi
summary

158
check_server.sh Executable file
View file

@ -0,0 +1,158 @@
#!/usr/bin/env bash
# Server-side component checks on koopa (via SSH).
set -euo pipefail
ROOT=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=lib.sh
source "$ROOT/lib.sh"
# Area server.* — host/container ports via SSH
set_area server
set_group ssh
section "server · ssh ${KOOPA_SSH}"
if ! koopa_ssh_ok; then
fail "ssh ${KOOPA_SSH}" "unreachable within ${SSH_CONNECT_TIMEOUT}s (SKIP_SSH=1 to skip)"
summary
exit 1
fi
ok "ssh ${KOOPA_SSH}"
# Run a remote script; collect structured lines
REMOTE=$(koopa_ssh_bash "${SSH_CMD_TIMEOUT}" <<'REMOTE'
set +e
report() { printf 'R|%s|%s|%s\n' "$1" "$2" "$3"; }
# containers
for name in taler-hacktivism-bank taler-hacktivism-exchange-ansible taler-hacktivism; do
if podman ps --format '{{.Names}}' 2>/dev/null | grep -qx "$name"; then
st=$(podman ps --filter "name=^${name}$" --format '{{.Status}}' | head -1)
report OK "container $name" "$st"
else
# soft match
hit=$(podman ps --format '{{.Names}}' 2>/dev/null | grep -E "$name|bank|exchange|hacktivism" | head -3 | tr '\n' ' ')
if podman ps --format '{{.Names}}' 2>/dev/null | grep -q bank && [ "$name" = taler-hacktivism-bank ]; then
bn=$(podman ps --format '{{.Names}}' | grep -i bank | head -1)
report OK "container bank ($bn)" "$(podman ps --filter name="$bn" --format '{{.Status}}' | head -1)"
elif podman ps --format '{{.Names}}' 2>/dev/null | grep -qi exchange && echo "$name" | grep -qi exchange; then
en=$(podman ps --format '{{.Names}}' | grep -i exchange | head -1)
report OK "container exchange ($en)" "$(podman ps --filter name="$en" --format '{{.Status}}' | head -1)"
elif podman ps --format '{{.Names}}' 2>/dev/null | grep -qx taler-hacktivism && [ "$name" = taler-hacktivism ]; then
report OK "container taler-hacktivism" "$(podman inspect -f '{{.State.Status}}' taler-hacktivism 2>/dev/null)"
else
report FAIL "container $name" "not running (seen: $hit)"
fi
fi
done
# local HTTP on pasta ports
for spec in \
"bank-api|http://127.0.0.1:9012/config|200" \
"bank-integration|http://127.0.0.1:9012/taler-integration/config|200" \
"landing|http://127.0.0.1:9013/intro/|200" \
"exchange|http://127.0.0.1:9011/config|200" \
"merchant|https://127.0.0.1:9010/config|200"
do
IFS='|' read -r id url want <<<"$spec"
code=$(curl -skS -m 5 -o /tmp/mon-s.out -w '%{http_code}' "$url" 2>/dev/null || echo 000)
if [ "$code" = "$want" ]; then
report OK "local $id : ${url#*//}" "HTTP $code"
else
report FAIL "local $id" "HTTP $code want $want"
fi
done
# processes inside bank
BANK=$(podman ps --format '{{.Names}}' | grep -iE 'bank|hacktivism-bank' | head -1)
if [ -n "$BANK" ]; then
if podman exec "$BANK" bash -c 'pgrep -f "MainKt serve|libeufin-bank serve" >/dev/null' 2>/dev/null; then
report OK "libeufin-bank process" "in $BANK"
else
report FAIL "libeufin-bank process" "not running in $BANK"
fi
if podman exec "$BANK" bash -c 'pg_isready -q' 2>/dev/null; then
report OK "postgres (bank)" "ready"
else
report WARN "postgres (bank)" "pg_isready failed"
fi
if podman exec "$BANK" bash -c 'pgrep -x nginx >/dev/null' 2>/dev/null; then
report OK "nginx landing" "in $BANK"
else
report WARN "nginx landing" "not seen in $BANK"
fi
else
report FAIL "bank container" "none"
fi
# exchange process / systemd if any
EX=$(podman ps --format '{{.Names}}' | grep -i exchange | head -1)
if [ -n "$EX" ]; then
if podman exec "$EX" bash -c 'pgrep -f taler-exchange-httpd >/dev/null || systemctl is-active taler-exchange-httpd 2>/dev/null | grep -q active' 2>/dev/null; then
report OK "exchange-httpd" "in $EX"
else
# config answering is enough
report WARN "exchange-httpd process" "not detected; port check above"
fi
fi
MER=$(podman ps --format '{{.Names}}' | grep -E '^taler-hacktivism$' | head -1)
[ -z "$MER" ] && MER=$(podman ps --format '{{.Names}}' | grep -i merchant | head -1)
if [ -n "$MER" ]; then
if podman exec "$MER" bash -c 'pgrep -f taler-merchant-httpd >/dev/null || true; curl -sk -m 3 -o /dev/null -w %{http_code} https://127.0.0.1:9010/config' 2>/dev/null | grep -q 200; then
report OK "merchant-httpd" "responds in $MER"
else
report WARN "merchant-httpd" "check manually in $MER"
fi
fi
# caddy on host
if systemctl is-active caddy >/dev/null 2>&1 || pgrep -x caddy >/dev/null 2>&1; then
report OK "caddy" "active"
else
report WARN "caddy" "not detected as active"
fi
# disk free (host + containers) — lines for mon_disk_check_remote_text on laptop side
echo "###DISK_HOST###"
df -Pk / /var /home /tmp 2>/dev/null || df -Pk
echo "###DISK_CTRS###"
for c in $(podman ps --format '{{.Names}}' 2>/dev/null); do
echo "###CTR $c###"
podman exec "$c" df -Pk / /var /tmp 2>/dev/null || podman exec "$c" df -Pk 2>/dev/null
done
REMOTE
)
while IFS= read -r line; do
case "$line" in
R\|*)
IFS='|' read -r _ st label detail <<<"$line"
case "$st" in
OK) ok "$label${detail:+ ($detail)}" ;;
FAIL) fail "$label" "$detail" ;;
WARN) warn "$label" "$detail" ;;
esac
;;
esac
done <<<"$REMOTE"
set_group disk
section "server · disk free space"
_host_df=$(printf '%s\n' "$REMOTE" | sed -n '/^###DISK_HOST###$/,/^###DISK_CTRS###$/p' | sed '1d;$d')
mon_disk_check_remote_text "ssh:${KOOPA_SSH}" "$_host_df" || true
_ctr=""; _buf=""
while IFS= read -r _line || [ -n "$_line" ]; do
case "$_line" in
'###CTR '*)
if [ -n "$_ctr" ]; then mon_disk_check_remote_text "ctr:${_ctr}" "$_buf" || true; fi
_ctr=${_line####CTR }; _ctr=${_ctr%###}; _buf=""
;;
'###DISK_CTRS###'|'###DISK_HOST###') ;;
*)
# only buffer after we entered CTR section
if [ -n "$_ctr" ]; then _buf="${_buf}${_line}"$'\n'; fi
;;
esac
done <<<"$REMOTE"
[ -n "$_ctr" ] && mon_disk_check_remote_text "ctr:${_ctr}" "$_buf" || true
summary

802
check_surface.sh Executable file
View file

@ -0,0 +1,802 @@
#!/usr/bin/env bash
# check_surface.sh — REMOTE-ONLY public surface / ecosystem scan.
#
# NOT in default phases. Explicit:
# ./taler-monitoring.sh surface
# ./taler-monitoring.sh -d hacktivism.ch surface
#
# Without -d / with generic ecosystem: catalog of taler.net, gnunet.org,
# taler-systems.com, taler-ops, mattermost, …
# With -d DOMAIN: expand domains.conf stack hosts + catalog entries matching
# that domain + common subdomain guesses; port/protocol probes only remote.
#
# Rules:
# - no SSH into targets, no local podman on targets
# - ICMP/port alone never proves "down" — confirm via expected protocol
# - catalogued host unreachable via protocol → ERROR
# - version unknown after probes → WARN
# - CVE hit (OSV) for identified software+version → ERROR
#
set -euo pipefail
ROOT=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=lib.sh
source "$ROOT/lib.sh"
set_area surface
section "surface · remote ecosystem / domain inventory (outside-in only)"
CATALOG="${SURFACE_CATALOG:-$ROOT/surface-catalog.conf}"
# Common ports when scanning a host discovered without explicit list
DEFAULT_SCAN_PORTS="${SURFACE_SCAN_PORTS:-22,80,443,993,8443,9418}"
PORT_TIMEOUT="${SURFACE_PORT_TIMEOUT:-2}"
HTTP_TIMEOUT="${SURFACE_HTTP_TIMEOUT:-12}"
# CVE via OSV (public, no key). Disable: SURFACE_CVE=0
: "${SURFACE_CVE:=1}"
# nmap OS / service fingerprint (v1.8.0+). Disable: SURFACE_NMAP=0
: "${SURFACE_NMAP:=1}"
# OS detection (-O) often needs root; falls back to -sV when denied
: "${SURFACE_NMAP_OS:=1}"
# Extra ports when -d domain mode
DOMAIN_EXTRA_PORTS="${SURFACE_DOMAIN_PORTS:-22,80,443,8443}"
declare -A HOST_PORTS # host -> comma ports
declare -A HOST_PROTO # host -> expect proto
declare -A HOST_LABEL # host -> short label
declare -A HOST_EXPECT # host -> 1 if catalogued (must respond)
ORDERED_HOSTS=()
add_host() {
local h="$1" ports="${2:-}" proto="${3:-https}" label="${4:-}" expect="${5:-1}"
h=$(printf '%s' "$h" | tr 'A-Z' 'a-z' | sed 's#^https\?://##;s#/.*##;s/\.$//')
[ -n "$h" ] || return 0
if [ -z "${HOST_PORTS[$h]+x}" ]; then
ORDERED_HOSTS+=("$h")
HOST_PORTS[$h]="${ports:-443}"
HOST_PROTO[$h]="${proto:-https}"
HOST_LABEL[$h]="${label:-$h}"
HOST_EXPECT[$h]="$expect"
else
# merge ports
local p
for p in ${ports//,/ }; do
case ",${HOST_PORTS[$h]}," in
*",$p,"*) ;;
*) HOST_PORTS[$h]="${HOST_PORTS[$h]},$p" ;;
esac
done
[ "${HOST_EXPECT[$h]}" = "1" ] || HOST_EXPECT[$h]="$expect"
fi
}
load_catalog() {
local line h ports proto label
[ -f "$CATALOG" ] || {
warn "catalog" "missing $CATALOG — using empty base list"
return 0
}
while IFS= read -r line || [ -n "$line" ]; do
line=${line%%#*}
line=$(echo "$line" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')
[ -z "$line" ] && continue
# host ports proto label
h=$(echo "$line" | awk '{print $1}')
ports=$(echo "$line" | awk '{print $2}')
proto=$(echo "$line" | awk '{print $3}')
label=$(echo "$line" | awk '{print $4}')
[ "$ports" = "-" ] && ports=""
add_host "$h" "$ports" "${proto:-https}" "${label:-}" 1
done <"$CATALOG"
info "catalog" "loaded $CATALOG · ${#ORDERED_HOSTS[@]} hosts"
}
# Clear inventory (used when switching to domain-only scope)
clear_hosts() {
ORDERED_HOSTS=()
unset HOST_PORTS HOST_PROTO HOST_LABEL HOST_EXPECT
declare -gA HOST_PORTS HOST_PROTO HOST_LABEL HOST_EXPECT
}
# Expand for -d domain: ONLY that domain (remote). No whole-ecosystem catalog.
expand_domain_scope() {
local d="$1" h
d=${d#https://}; d=${d%/}
d=${d#http://}
info "scope" "domain mode −d $d (remote only · not full ecosystem catalog)"
clear_hosts
# stack endpoints from applied profile (always expected)
for h in \
"${BANK_PUBLIC:-}" \
"${EXCHANGE_PUBLIC:-}" \
"${MERCHANT_PUBLIC:-}" \
"${PAIVANA_PUBLIC:-}"
do
[ -n "$h" ] || continue
add_host "$h" "80,443,$DOMAIN_EXTRA_PORTS" https "stack" 1
done
# apex + common subdomains (expected if DNS exists — set expect after DNS in scan,
# but catalogued stack already expected; guesses start optional)
for sub in "" www bank exchange taler merchant backend shop shops stage \
git docs www2 api static media landing mon401
do
if [ -z "$sub" ]; then
h="$d"
else
h="${sub}.${d}"
fi
# stack hosts already added as expect=1; guesses optional until we promote
if [ -z "${HOST_PORTS[$h]+x}" ]; then
add_host "$h" "$DOMAIN_EXTRA_PORTS" https "guess" 0
fi
done
# also pick catalog lines that belong to this domain only
if [ -f "$CATALOG" ]; then
local line ports proto label
while IFS= read -r line || [ -n "$line" ]; do
line=${line%%#*}
line=$(echo "$line" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')
[ -z "$line" ] && continue
h=$(echo "$line" | awk '{print $1}')
h=$(printf '%s' "$h" | tr 'A-Z' 'a-z' | sed 's#^https\?://##;s#/.*##')
if [ "$h" = "$d" ] || [[ "$h" == *".$d" ]]; then
ports=$(echo "$line" | awk '{print $2}')
proto=$(echo "$line" | awk '{print $3}')
label=$(echo "$line" | awk '{print $4}')
[ "$ports" = "-" ] && ports=""
add_host "$h" "$ports" "${proto:-https}" "${label:-dom}" 1
fi
done <"$CATALOG"
fi
}
# Ecosystem mode: full catalog; all entries expected
expand_ecosystem_scope() {
info "scope" "ecosystem mode (taler.net / gnunet.org / taler-systems.com / taler-ops / mattermost, … · remote only)"
local h
for h in "${ORDERED_HOSTS[@]}"; do
HOST_EXPECT[$h]=1
done
}
# --- remote probes (no SSH) ---
dns_ok() {
local h="$1"
getent ahosts "$h" >/dev/null 2>&1 || getent hosts "$h" >/dev/null 2>&1
}
# TCP connect only (not proof of service)
tcp_open() {
local h="$1" port="$2"
if command -v timeout >/dev/null 2>&1; then
timeout "$PORT_TIMEOUT" bash -c "echo >/dev/tcp/${h}/${port}" 2>/dev/null
else
bash -c "echo >/dev/tcp/${h}/${port}" 2>/dev/null
fi
}
# ICMP optional — never sole grounds for ERROR
ping_host() {
local h="$1"
ping -c 1 -W 2 "$h" >/dev/null 2>&1 || ping -c 1 -w 2 "$h" >/dev/null 2>&1
}
# HTTPS probe: code, server header, version hints, cert subject/dates
probe_https() {
local h="$1" port="${2:-443}"
local url hdr body code server via cert_end subj san
url="https://${h}/"
[ "$port" != "443" ] && url="https://${h}:${port}/"
body=$(mktemp)
hdr=$(mktemp)
code=$(curl -skS -m "$HTTP_TIMEOUT" -D "$hdr" -o "$body" -w '%{http_code}' \
--connect-timeout "$PORT_TIMEOUT" "$url" 2>/dev/null || echo 000)
server=$(awk 'BEGIN{IGNORECASE=1} /^server:/{sub(/\r$/,""); sub(/^server:[[:space:]]*/,""); print; exit}' "$hdr" 2>/dev/null || true)
via=$(awk 'BEGIN{IGNORECASE=1} /^x-powered-by:/{sub(/\r$/,""); sub(/^[^:]+:[[:space:]]*/,""); print; exit}' "$hdr" 2>/dev/null || true)
# cert
cert_end=$(echo | openssl s_client -servername "$h" -connect "${h}:${port}" 2>/dev/null \
| openssl x509 -noout -enddate 2>/dev/null | sed 's/notAfter=//')
subj=$(echo | openssl s_client -servername "$h" -connect "${h}:${port}" 2>/dev/null \
| openssl x509 -noout -subject 2>/dev/null | head -1)
# taler version hints in body
local taler_hint
taler_hint=$(grep -oE 'taler[^"[:space:]]{0,40}|GNU Taler|libeufin|gnunet' "$body" 2>/dev/null | head -3 | tr '\n' ' ' || true)
# config JSON version if /config works
local cfg_ver=""
local ccode
ccode=$(curl -skS -m "$HTTP_TIMEOUT" -o "$body" -w '%{http_code}' \
"https://${h}/config" 2>/dev/null || echo 000)
if [ "$ccode" = "200" ]; then
cfg_ver=$(python3 -c 'import json,sys
try:
d=json.load(open(sys.argv[1]))
print(d.get("version") or d.get("name") or d.get("currency") or "config-json")
except Exception:
print("")' "$body" 2>/dev/null || true)
fi
rm -f "$hdr" "$body"
printf 'code=%s server=%s powered=%s cert_end=%s cfg=%s hint=%s' \
"$code" "${server:-}" "${via:-}" "${cert_end:-}" "${cfg_ver:-}" "${taler_hint:-}"
# return 0 if HTTP answered (any 2xx/3xx/4xx — service is up)
case "$code" in
2??|3??|4??) return 0 ;;
*) return 1 ;;
esac
}
probe_http() {
local h="$1" port="${2:-80}"
local url code
url="http://${h}/"
[ "$port" != "80" ] && url="http://${h}:${port}/"
code=$(curl -sS -m "$HTTP_TIMEOUT" -o /dev/null -w '%{http_code}' \
--connect-timeout "$PORT_TIMEOUT" "$url" 2>/dev/null || echo 000)
printf 'code=%s' "$code"
case "$code" in 2??|3??|4??) return 0 ;; *) return 1 ;; esac
}
probe_ssh_banner() {
local h="$1" port="${2:-22}"
local ban
ban=$(timeout "$PORT_TIMEOUT" bash -c "exec 3<>/dev/tcp/${h}/${port}; dd bs=256 count=1 <&3 2>/dev/null" 2>/dev/null \
| tr -d '\r' | head -1 || true)
printf 'banner=%s' "${ban:-}"
[ -n "$ban" ]
}
# Map Server header → package name guess for OSV
guess_package() {
local server="$1"
local s
s=$(printf '%s' "$server" | tr 'A-Z' 'a-z')
case "$s" in
*nginx*) echo "nginx" ;;
*apache*|*httpd*) echo "apache" ;;
*caddy*) echo "caddy" ;;
*openbsd\ httpd*) echo "openbsd-httpd" ;;
*) echo "" ;;
esac
}
extract_version() {
local server="$1"
# nginx/1.22.1 → 1.22.1
printf '%s' "$server" | sed -n 's/.*\/\([0-9][0-9.]*\).*/\1/p' | head -1
}
# OSV / CVE check for software version taken from Server headers.
#
# Important: headers only expose upstream versions (nginx/1.26.3), never the
# Debian package revision (1.26.3-3+deb13u7). Querying OSV ecosystem=Debian
# with the bare version yields massive false positives (ancient DEBIAN-CVE-*
# with introduced:0 and no fixed event, plus every package revision that
# merely *starts with* 1.26.3).
#
# Policy:
# - bare upstream version → upstream SEMVER signal only (WARN by default)
# - full Debian package version (contains '-') → Debian OSV, filtered
# - SURFACE_CVE=0 disables; SURFACE_CVE_LEVEL=error|warn (default warn for bare)
check_cves() {
local pkg="$1" ver="$2" host="$3"
[ "$SURFACE_CVE" = "1" ] || return 0
[ -n "$pkg" ] && [ -n "$ver" ] || return 0
local level out rc
# bare header versions are approximate → WARN unless overridden
if [[ "$ver" == *-* ]] || [[ "$ver" == *+* ]]; then
level="${SURFACE_CVE_LEVEL:-error}"
else
level="${SURFACE_CVE_LEVEL:-warn}"
fi
out=$(
SURFACE_CVE_PKG="$pkg" SURFACE_CVE_VER="$ver" SURFACE_CVE_HOST="$host" python3 - <<'PY' 2>/dev/null || true
import json, os, re, urllib.request
pkg = os.environ.get("SURFACE_CVE_PKG", "")
ver = os.environ.get("SURFACE_CVE_VER", "")
host = os.environ.get("SURFACE_CVE_HOST", "")
def ver_tuple(s: str):
nums = [int(x) for x in re.findall(r"\d+", s or "")[:5]]
return tuple(nums) if nums else ()
def vt_cmp(a, b):
n = max(len(a), len(b))
a = a + (0,) * (n - len(a))
b = b + (0,) * (n - len(b))
return (a > b) - (a < b)
def in_semver_events(ver, events):
"""True if ver is in [introduced, fixed) for SEMVER-like event list."""
vt = ver_tuple(ver)
if not vt:
return False
# process sequential introduced/fixed pairs
intro = None
for e in events or []:
if "introduced" in e:
intro = e.get("introduced")
elif "fixed" in e or "last_affected" in e:
fixed = e.get("fixed")
last = e.get("last_affected")
lo = ver_tuple("0" if intro in (None, "0") else str(intro))
if vt_cmp(vt, lo) < 0:
intro = None
continue
if fixed is not None:
if vt_cmp(vt, ver_tuple(str(fixed))) < 0:
return True
elif last is not None:
if vt_cmp(vt, ver_tuple(str(last))) <= 0:
return True
intro = None
# open-ended introduced without fixed → ignore (never clears; Debian noise)
return False
def osv_post(body):
req = urllib.request.Request(
"https://api.osv.dev/v1/query",
data=json.dumps(body).encode(),
headers={"Content-Type": "application/json"},
method="POST",
)
try:
with urllib.request.urlopen(req, timeout=20) as r:
return json.load(r)
except Exception:
return {}
def osv_get(vid):
try:
with urllib.request.urlopen(f"https://api.osv.dev/v1/vulns/{vid}", timeout=15) as r:
return json.load(r)
except Exception:
return {}
bare = not (("-" in ver) or ("+" in ver))
hits = []
if bare:
# Upstream signal via CVE records that carry extracted_events for this package
# (Server header has no Debian revision — do NOT use ecosystem=Debian).
seed = {
"nginx": [
"CVE-2025-23419",
"CVE-2025-53859",
"CVE-2024-7347",
"CVE-2024-34161",
"CVE-2024-32760",
"CVE-2024-31079",
"CVE-2024-24989",
"CVE-2024-24990",
],
"apache": [
"CVE-2024-38474",
"CVE-2024-38476",
"CVE-2024-38477",
"CVE-2023-31122",
"CVE-2023-43622",
],
"caddy": [
"CVE-2022-29718",
"CVE-2023-50463",
],
}.get(pkg, [])
for vid in seed:
doc = osv_get(vid)
if not doc:
continue
ok_hit = False
for a in doc.get("affected") or []:
# prefer extracted_events (human SEMVER) on GIT/nginx ranges
for rg in a.get("ranges") or []:
db = rg.get("database_specific") or {}
extracted = db.get("extracted_events") or []
if extracted and in_semver_events(ver, extracted):
# package name filter when present
cpes = db.get("cpe") or []
if isinstance(cpes, str):
cpes = [cpes]
blob = json.dumps(a).lower() + json.dumps(cpes).lower()
if pkg == "nginx" and "nginx" not in blob and "f5" not in blob:
continue
ok_hit = True
break
if rg.get("type") == "SEMVER" and in_semver_events(ver, rg.get("events") or []):
ok_hit = True
break
if ok_hit:
break
if ok_hit:
hits.append(vid)
else:
# Full package version — Debian ecosystem is meaningful
data = osv_post({"package": {"name": pkg, "ecosystem": "Debian"}, "version": ver})
for v in data.get("vulns") or []:
vid = v.get("id") or "?"
actionable = False
for a in v.get("affected") or []:
for rg in a.get("ranges") or []:
events = rg.get("events") or []
has_end = any(("fixed" in e) or ("last_affected" in e) for e in events)
if not has_end:
continue # open-ended Debian noise
if rg.get("type") in ("ECOSYSTEM", "SEMVER"):
# version string is full deb version; trust OSV query match
# but only if a fixed event exists (actionable)
actionable = True
break
if actionable:
break
# exact version listed
versions = a.get("versions") or []
if ver in versions:
actionable = True
break
if actionable:
hits.append(vid)
# de-dup preserve order
seen = set()
uniq = []
for h in hits:
if h not in seen:
seen.add(h)
uniq.append(h)
if uniq:
print(f"HIT {len(uniq)} " + ",".join(uniq[:12]))
else:
print("CLEAN")
PY
)
rc=0
case "$out" in
HIT\ *)
local n ids
n=$(printf '%s' "$out" | awk '{print $2}')
ids=$(printf '%s' "$out" | cut -d' ' -f3-)
if [ "$level" = "error" ]; then
err "cve" "$host $pkg $ver — $n actionable vuln(s)" "$ids"
rc=1
else
warn "cve" "$host $pkg $ver — $n actionable vuln(s) (header version · not Debian pkg)" "$ids"
rc=0
fi
;;
CLEAN)
if [[ "$ver" == *-* ]] || [[ "$ver" == *+* ]]; then
info "cve" "$host $pkg $ver — OSV clean (Debian package version)"
else
info "cve" "$host $pkg $ver — no actionable upstream CVE for bare Server-header version"
fi
;;
*)
info "cve" "$host $pkg $ver — CVE probe skipped/unavailable"
;;
esac
return "$rc"
}
cert_expiry_check() {
local h="$1" end="$2"
[ -n "$end" ] || return 0
local end_epoch now left days
end_epoch=$(date -d "$end" +%s 2>/dev/null || date -j -f "%b %e %T %Y %Z" "$end" +%s 2>/dev/null || echo 0)
now=$(date +%s)
[ "$end_epoch" -gt 0 ] || return 0
left=$((end_epoch - now))
days=$((left / 86400))
if [ "$left" -le 0 ]; then
err "tls" "$h certificate EXPIRED" "notAfter=$end"
return 1
fi
if [ "$days" -le 14 ]; then
warn "tls" "$h certificate expires in ${days}d" "notAfter=$end"
else
info "tls" "$h cert ok · ${days}d left · notAfter=$end"
fi
return 0
}
# nmap OS / service fingerprint (v1.8.0+).
# ERROR when fingerprint indicates an OS that is EOL / should be upgraded.
# Soft/info when nmap missing or unprivileged OS scan fails.
nmap_fingerprint_host() {
local h="$1"
local xml out osline accuracy eol=0
[ "${SURFACE_NMAP:-1}" = "1" ] || return 0
if ! command -v nmap >/dev/null 2>&1; then
warn "nmap" "$h · nmap not installed (apt install nmap) — skip OS fingerprint"
return 0
fi
xml=$(mktemp)
out=$(mktemp)
# Prefer OS detection when allowed; always try service version light
if [ "${SURFACE_NMAP_OS:-1}" = "1" ] && {
nmap -Pn -n -T4 -O --osscan-guess --max-os-tries 1 --host-timeout 45s \
-p 22,80,443,25,465,587,993 "$h" -oX "$xml" >"$out" 2>&1
}; then
:
else
nmap -Pn -n -T4 -sV --version-light --host-timeout 35s \
-p 22,80,443,25,465,587,993 "$h" -oX "$xml" >"$out" 2>&1 || true
fi
# Parse best OS match from XML
osline=$(
python3 - "$xml" <<'PY' 2>/dev/null || true
import sys, xml.etree.ElementTree as ET
path = sys.argv[1]
try:
root = ET.parse(path).getroot()
except Exception:
sys.exit(0)
best = None
best_acc = -1
for osm in root.findall(".//osmatch"):
name = osm.get("name") or ""
try:
acc = int(osm.get("accuracy") or "0")
except ValueError:
acc = 0
if name and acc >= best_acc:
best_acc = acc
best = name
if best:
print(f"{best_acc}|{best}")
# also surface product/version from service table
svcs = []
for port in root.findall(".//port"):
state = (port.find("state").get("state") if port.find("state") is not None else "")
if state != "open":
continue
svc = port.find("service")
if svc is None:
continue
prod = svc.get("product") or svc.get("name") or ""
ver = svc.get("version") or ""
if prod:
svcs.append(f"{prod} {ver}".strip())
if svcs:
print("SVC|" + "; ".join(svcs[:8]))
PY
)
rm -f "$xml" "$out"
if [ -z "$osline" ]; then
info "nmap" "$h · no OS/service fingerprint (need root for -O, or host filtered)"
return 0
fi
local osname="" svcinfo=""
while IFS= read -r line; do
case "$line" in
SVC\|*) svcinfo=${line#SVC|} ;;
*\|*)
accuracy=${line%%|*}
osname=${line#*|}
ok "nmap-os" "$h · OS guess ${accuracy}% · $osname"
;;
esac
done <<<"$osline"
[ -n "$svcinfo" ] && info "nmap-svc" "$h · $svcinfo"
# EOL / upgrade-required OS fingerprints → ERROR
case "$(printf '%s' "$osname" | tr 'A-Z' 'a-z')" in
*debian*6*|*debian*7*|*debian*8*|*debian*9*|*debian*10*|*debian*\"buster\"*|*debian*buster*)
eol=1 ;;
*ubuntu*12.04*|*ubuntu*14.04*|*ubuntu*16.04*|*ubuntu*18.04*|*ubuntu*20.04*)
eol=1 ;;
*centos*[5-7]*|*red\ hat\ enterprise*5*|*red\ hat\ enterprise*6*|*red\ hat\ enterprise*7*)
eol=1 ;;
*windows\ server\ 2008*|*windows\ server\ 2012*|*windows\ xp*|*windows\ 7*)
eol=1 ;;
*freebsd\ 1[0-2]*|*freebsd\ [0-9].*)
eol=1 ;;
esac
if [ "$eol" = "1" ]; then
err "nmap-upgrade" "$h OS fingerprint indicates EOL / upgrade required" "$osname"
return 1
fi
if [ -n "$osname" ]; then
info "nmap-upgrade" "$h OS fingerprint not on EOL list · $osname"
fi
return 0
}
scan_host() {
local h="$1"
local ports proto label expect
local p open_ports=() any_proto_ok=0 ping_ok=0 dns=0
local probe_detail server_hdr pkg ver
ports=${HOST_PORTS[$h]:-443}
proto=${HOST_PROTO[$h]:-https}
label=${HOST_LABEL[$h]:-$h}
expect=${HOST_EXPECT[$h]:-0}
set_group "$label"
if dns_ok "$h"; then
dns=1
ok "dns" "$h resolves"
else
if [ "$expect" = "1" ]; then
err "dns" "$h does not resolve (catalogued)"
else
info "dns" "$h no resolve (optional guess) — skip"
fi
return 0
fi
if ping_host "$h"; then
ping_ok=1
info "ping" "$h ICMP ok"
else
info "ping" "$h ICMP no reply (not decisive)"
fi
# port scan (unique ports)
local _seen_ports=" "
for p in ${ports//,/ }; do
[ -n "$p" ] || continue
case "$_seen_ports" in *" $p "*) continue ;; esac
_seen_ports="$_seen_ports$p "
if tcp_open "$h" "$p"; then
open_ports+=("$p")
info "port" "$h:$p open (TCP)"
else
info "port" "$h:$p closed/filtered (TCP)"
fi
done
# Protocol verification on open ports (and always try 443/80 for https/http expect)
local try_ports=("${open_ports[@]}")
if [ ${#try_ports[@]} -eq 0 ]; then
# still try expected protocol ports even if scan said closed (scan false negatives)
case "$proto" in
https) try_ports=(443) ;;
http) try_ports=(80) ;;
ssh) try_ports=(22) ;;
*) try_ports=(443 80) ;;
esac
fi
server_hdr=""
ver=""
for p in "${try_ports[@]}"; do
case "$p" in
443|8443)
if probe_detail=$(probe_https "$h" "$p"); then
any_proto_ok=1
ok "https" "$h:$p up · $probe_detail"
server_hdr=$(printf '%s' "$probe_detail" | sed -n 's/.*server=\([^ ]*\).*/\1/p')
# cert
local cend
cend=$(printf '%s' "$probe_detail" | sed -n 's/.*cert_end=\([^ ]*\).*/\1/p')
cert_expiry_check "$h" "$cend" || true
local cfg
cfg=$(printf '%s' "$probe_detail" | sed -n 's/.*cfg=\([^ ]*\).*/\1/p')
if [ -n "$cfg" ]; then
info "version" "$h config/api hint: $cfg"
ver="$cfg"
fi
else
info "https" "$h:$p no HTTP response · ${probe_detail:-}"
fi
;;
80|8080)
if probe_detail=$(probe_http "$h" "$p"); then
any_proto_ok=1
ok "http" "$h:$p up · $probe_detail"
else
info "http" "$h:$p no HTTP response"
fi
;;
22)
if probe_detail=$(probe_ssh_banner "$h" "$p"); then
any_proto_ok=1
ok "ssh" "$h:$p banner · $probe_detail"
ver=$(printf '%s' "$probe_detail" | sed 's/banner=//')
else
info "ssh" "$h:$p no SSH banner"
fi
;;
*)
if tcp_open "$h" "$p"; then
info "tcp" "$h:$p open · protocol unknown"
fi
;;
esac
done
# Version summary
if [ -n "$server_hdr" ]; then
info "server-header" "$h · $server_hdr"
pkg=$(guess_package "$server_hdr")
ver_soft=$(extract_version "$server_hdr")
if [ -n "$pkg" ] && [ -n "$ver_soft" ]; then
info "version" "$h software $pkg $ver_soft (from Server header)"
check_cves "$pkg" "$ver_soft" "$h" || true
elif [ -n "$server_hdr" ]; then
warn "version" "$h Server header present but version unknown · $server_hdr"
fi
elif [ -n "$ver" ]; then
info "version" "$h · $ver"
else
if [ "$any_proto_ok" = "1" ]; then
warn "version" "$h reachable but software version unknown"
fi
fi
# nmap OS / service fingerprint (v1.8.0+) — ERROR on EOL OS
nmap_fingerprint_host "$h" || true
# Expected service must answer protocol (not just ping/port)
# Note: ${array[*]:-x} is NOT valid default syntax (bash treats : as slice).
local ports_txt="${open_ports[*]}"
ports_txt=${ports_txt:-none}
if [ "$expect" = "1" ]; then
if [ "$any_proto_ok" = "1" ]; then
ok "reachability" "$h catalogued service OK (protocol confirmed)"
else
err "reachability" "$h catalogued but not reachable via ${proto}/protocol" \
"dns=$dns ping=$ping_ok open_ports=${ports_txt} (ICMP/TCP alone not enough; protocol failed)"
fi
else
if [ "$any_proto_ok" = "1" ]; then
info "reachability" "$h optional host responds"
else
info "reachability" "$h optional · no protocol response (ok)"
fi
fi
}
# --- main ---
# Scope (remote only):
# ./taler-monitoring.sh surface → full ecosystem catalog
# ./taler-monitoring.sh -d hacktivism.ch surface → only that domain
if [ "${SURFACE_SCOPE:-}" = "ecosystem" ]; then
load_catalog
expand_ecosystem_scope
elif [ "${SURFACE_SCOPE:-}" = "domain" ] || [ "${TALER_DOMAIN_FROM_CLI:-0}" = "1" ]; then
# domain mode: do not load whole ecosystem first
CATALOG="${SURFACE_CATALOG:-$ROOT/surface-catalog.conf}"
expand_domain_scope "${TALER_DOMAIN:-hacktivism.ch}"
elif [ "${SURFACE_SCOPE:-auto}" = "auto" ]; then
# no -d → ecosystem; with -d → domain (TALER_DOMAIN_FROM_CLI)
if [ "${TALER_DOMAIN_FROM_CLI:-0}" = "1" ]; then
CATALOG="${SURFACE_CATALOG:-$ROOT/surface-catalog.conf}"
expand_domain_scope "$TALER_DOMAIN"
else
load_catalog
expand_ecosystem_scope
fi
else
load_catalog
expand_ecosystem_scope
fi
info "inventory" "${#ORDERED_HOSTS[@]} hosts to probe (remote only · no SSH · no local podman on targets)"
info "flags" "SURFACE_CVE=${SURFACE_CVE:-1} SURFACE_NMAP=${SURFACE_NMAP:-1} SURFACE_NMAP_OS=${SURFACE_NMAP_OS:-1} PORT_TIMEOUT=${PORT_TIMEOUT:-2} HTTP_TIMEOUT=${HTTP_TIMEOUT:-12}"
for h in "${ORDERED_HOSTS[@]}"; do
[ -n "$h" ] || continue
scan_host "$h" || true
done
# mon_disk on the *runner* only (optional) — not remote servers' disks
if [ "${SURFACE_CHECK_RUNNER_DISK:-0}" = "1" ]; then
set_group disk
if declare -F mon_disk_check_host >/dev/null 2>&1; then
mon_disk_check_host "runner" || true
else
warn "disk" "mon_disk_check_host not available"
fi
fi
# summary returns non-zero when FAIL_N>0; do not trip set -e mid-script
summary || true
if [ "${FAIL_N:-0}" -eq 0 ]; then
exit 0
fi
exit 1

1607
check_urls.sh Executable file
View file

File diff suppressed because it is too large Load diff

563
check_versions.sh Executable file
View file

@ -0,0 +1,563 @@
#!/usr/bin/env bash
# Area versions-### — Taler packages vs deb.taler.net (trixie) + repo availability.
#
# Checks:
# 1) deb.taler.net apt endpoints reachable (InRelease / Packages / sample .deb)
# 2) containers can reach deb.taler.net (install path from inside)
# 3) installed taler*/libeufin*/libtaler*/libdonau* versions vs suite index
#
# Env:
# TALER_APT_SUITE=trixie
# TALER_APT_BASE=https://deb.taler.net/apt/debian
# TALER_APT_INDEX=…/dists/trixie/main/binary-amd64/Packages
# TALER_APT_TESTING_INDEX=…/dists/trixie-testing/…
# TALER_PKG_BEHIND=warn|error (default error as of v1.8.0 — any behind package is ERROR)
# SKIP_SSH=1 skip container install checks
set -euo pipefail
ROOT=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=lib.sh
source "$ROOT/lib.sh"
set_area versions
set_group outside
SUITE="${TALER_APT_SUITE:-trixie}"
APT_BASE="${TALER_APT_BASE:-https://deb.taler.net/apt/debian}"
APT_BASE="${APT_BASE%/}"
INDEX="${TALER_APT_INDEX:-${APT_BASE}/dists/${SUITE}/main/binary-amd64/Packages}"
INRELEASE="${TALER_APT_INRELEASE:-${APT_BASE}/dists/${SUITE}/InRelease}"
TESTING_INDEX="${TALER_APT_TESTING_INDEX:-${APT_BASE}/dists/${SUITE}-testing/main/binary-amd64/Packages}"
BEHIND_MODE="${TALER_PKG_BEHIND:-error}"
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
http_get() {
# usage: http_get URL [outfile] → prints http_code; body to outfile or /dev/null
local url="$1" out="${2:-/dev/null}"
curl -sS --max-time 30 -L -o "$out" -w '%{http_code}' "$url" 2>/dev/null || echo 000
}
# ---------------------------------------------------------------------------
# 1) OUTSIDE — deb.taler.net install source (runner / laptop, no SSH)
# ---------------------------------------------------------------------------
section "versions · outside · deb.taler.net (${SUITE})"
# DNS
HOST_APT="${APT_BASE#https://}"; HOST_APT="${HOST_APT#http://}"; HOST_APT="${HOST_APT%%/*}"
if getent hosts "$HOST_APT" >/dev/null 2>&1 \
|| python3 -c "import socket; socket.getaddrinfo('${HOST_APT}', 443)" >/dev/null 2>&1; then
ip=$(python3 -c "import socket; print(socket.getaddrinfo('${HOST_APT}',443)[0][4][0])" 2>/dev/null || true)
ok "outside DNS ${HOST_APT}" "${ip:-resolved}"
else
fail "outside DNS ${HOST_APT}" "unresolvable — cannot use as apt source"
fi
# HTTPS portal / apt root
code=$(http_get "https://${HOST_APT}/")
[ "$code" = "200" ] && ok "outside https://${HOST_APT}/" "HTTP $code" \
|| warn "outside https://${HOST_APT}/" "HTTP $code"
code=$(http_get "${APT_BASE}/")
case "$code" in
200|301|302) ok "outside apt base" "${APT_BASE}/ HTTP $code" ;;
*) fail "outside apt base" "${APT_BASE}/ HTTP $code" ;;
esac
# Suite metadata (what apt update needs)
code=$(http_get "$INRELEASE" "$tmp/InRelease")
if [ "$code" = "200" ] && [ -s "$tmp/InRelease" ]; then
ok "outside InRelease" "${SUITE} HTTP 200 · $(wc -c <"$tmp/InRelease" | tr -d ' ') bytes"
else
fail "outside InRelease" "HTTP $code — $INRELEASE (apt update will fail)"
fi
RELEASE_URL="${APT_BASE}/dists/${SUITE}/Release"
code=$(http_get "$RELEASE_URL" "$tmp/Release")
[ "$code" = "200" ] && ok "outside Release" "${SUITE} HTTP 200" \
|| warn "outside Release" "HTTP $code — $RELEASE_URL"
# Packages (plain) — required for version compare
code=$(http_get "$INDEX" "$tmp/Packages")
if [ "$code" = "200" ] && [ -s "$tmp/Packages" ] && grep -q '^Package: ' "$tmp/Packages"; then
n_pkg=$(grep -c '^Package: ' "$tmp/Packages" || true)
ok "outside Packages" "${SUITE} HTTP 200 · ${n_pkg} packages"
else
fail "outside Packages" "HTTP $code — $INDEX"
# still try more probes, but cannot compare versions without index
fi
# Packages.gz (apt often prefers this)
PKGZ_URL="${APT_BASE}/dists/${SUITE}/main/binary-amd64/Packages.gz"
code=$(http_get "$PKGZ_URL" "$tmp/Packages.gz")
if [ "$code" = "200" ] && [ -s "$tmp/Packages.gz" ]; then
ok "outside Packages.gz" "${SUITE} HTTP 200 · $(wc -c <"$tmp/Packages.gz" | tr -d ' ') bytes"
else
warn "outside Packages.gz" "HTTP $code — apt may still use plain Packages"
fi
# Sample pool .deb downloadable (proves packages are installable, not just listed)
if [ -s "$tmp/Packages" ]; then
DEB_PATH=$(awk '
/^Package: taler-exchange$/ { p=1 }
p && /^Filename: / { sub(/^Filename: /,""); print; exit }
p && /^$/ { p=0 }
' "$tmp/Packages")
[ -z "$DEB_PATH" ] && DEB_PATH=$(awk '
/^Package: taler-merchant$/ { p=1 }
p && /^Filename: / { sub(/^Filename: /,""); print; exit }
p && /^$/ { p=0 }
' "$tmp/Packages")
if [ -n "$DEB_PATH" ]; then
DEB_URL="${APT_BASE}/${DEB_PATH}"
code=$(curl -sS --max-time 30 -o /dev/null -w '%{http_code}' -r 0-128 "$DEB_URL" 2>/dev/null || echo 000)
case "$code" in
200|206)
ok "outside pool .deb" "$(basename "$DEB_PATH") HTTP $code"
;;
*)
fail "outside pool .deb" "HTTP $code — $DEB_URL (index ok but debs not fetchable)"
;;
esac
else
warn "outside pool .deb" "no Filename for taler-exchange/merchant in index"
fi
for pkg in taler-exchange taler-merchant libeufin-bank; do
ver=$(awk -v p="$pkg" '
$0=="Package: "p { hit=1; next }
hit && /^Version: / { sub(/^Version: /,""); print; exit }
hit && /^$/ { hit=0 }
' "$tmp/Packages")
if [ -n "$ver" ]; then
ok "outside suite offers ${pkg}" "$ver"
else
fail "outside suite offers ${pkg}" "missing from ${SUITE} Packages"
fi
done
fi
HAVE_TESTING=0
code=$(http_get "$TESTING_INDEX" "$tmp/Packages-testing")
if [ "$code" = "200" ] && [ -s "$tmp/Packages-testing" ] && grep -q '^Package: ' "$tmp/Packages-testing"; then
HAVE_TESTING=1
ok "outside ${SUITE}-testing Packages" "HTTP 200 (optional compare)"
else
info "outside ${SUITE}-testing Packages" "HTTP ${code:-000} (optional)"
fi
# TLS: certificate verify (curl default) already used above; explicit openssl probe if available
if command -v openssl >/dev/null 2>&1; then
if echo | openssl s_client -servername "$HOST_APT" -connect "${HOST_APT}:443" 2>/dev/null \
| grep -q 'Verify return code: 0'; then
ok "outside TLS ${HOST_APT}" "verify ok"
else
# curl succeeded with default CA — soft warn only
warn "outside TLS ${HOST_APT}" "openssl verify not clean (curl may still work)"
fi
fi
# Need Packages index for later compare
if [ ! -s "$tmp/Packages" ] || ! grep -q '^Package: ' "$tmp/Packages"; then
fail "outside Packages usable" "cannot continue version compare without suite index"
summary
exit 1
fi
# ---------------------------------------------------------------------------
# 2) INSIDE — containers can reach deb.taler.net (pasta / install path)
# host-podman: INSIDE_PODMAN=1 on GOA host · ssh: KOOPA_SSH / INSIDE_SSH from laptop
# ---------------------------------------------------------------------------
_VERS_LOCAL=0
_VERS_SSH_HOST=""
_VERS_BANK="taler-hacktivism-bank"
_VERS_EX="taler-hacktivism-exchange-ansible"
_VERS_MER="taler-hacktivism"
if [ "${INSIDE_PODMAN:-0}" = "1" ] || [ "${INSIDE_MODE:-}" = "local-podman" ]; then
_VERS_LOCAL=1
elif [ "${INSIDE_PROFILE:-}" = "stage-lfp" ] \
|| [ "${EXPECT_CURRENCY:-}" = "TESTPAYSAN" ] \
|| [ -n "${INSIDE_SSH:-}" ]; then
_VERS_SSH_HOST="${INSIDE_SSH:-}"
_VERS_BANK="${INSIDE_BANK_CTR:-stage-lfp-bank}"
_VERS_EX="${INSIDE_EXCHANGE_CTR:-stage-lfp-exchange-ansible}"
_VERS_MER="${INSIDE_MERCHANT_CTR:-stage-lfp-merchant}"
elif [ "${LOCAL_STACK:-0}" = "1" ] && [ "${SKIP_SSH:-0}" != "1" ]; then
# laptop → koopa via SSH (not host-podman)
_VERS_SSH_HOST="${KOOPA_SSH}"
fi
if [ "$_VERS_LOCAL" = "1" ]; then
VERS_ACCESS=host-podman
set_group host
else
VERS_ACCESS=ssh
set_group ssh
fi
section "versions · inside · containers → deb.taler.net (access=${VERS_ACCESS})"
info "flags" "VERS_ACCESS=${VERS_ACCESS} INSIDE_PODMAN=${INSIDE_PODMAN:-0} INSIDE_MODE=${INSIDE_MODE:-} LOCAL_STACK=${LOCAL_STACK:-0} SKIP_SSH=${SKIP_SSH:-0} KOOPA_SSH=${KOOPA_SSH:-} INSIDE_SSH=${INSIDE_SSH:-}"
if [ "$_VERS_LOCAL" != "1" ] && [ -z "$_VERS_SSH_HOST" ]; then
info "access" "no host-podman and no SSH target — outside-only package index"
info "outside-only" "deb.taler.net public checks completed above"
summary
exit 0
fi
if [ "$_VERS_LOCAL" = "1" ]; then
if ! command -v podman >/dev/null 2>&1; then
err "host" "INSIDE_PODMAN/host-podman but podman missing"
info "outside-only" "deb.taler.net public checks completed above"
summary
exit 1
fi
ok "host→container" "podman exec on this host (VERS_ACCESS=host-podman · no SSH)"
elif [ "$_VERS_SSH_HOST" = "${KOOPA_SSH:-}" ] || [ "${INSIDE_PROFILE:-}" = "koopa" ]; then
if ! koopa_ssh_ok; then
err "ssh" "cannot reach ${KOOPA_SSH} — from laptop set KOOPA_SSH=; on host set INSIDE_PODMAN=1"
info "outside-only" "deb.taler.net public checks completed above"
summary
exit 1
fi
ok "ssh ${KOOPA_SSH}" "remote then podman (VERS_ACCESS=ssh)"
else
if ! mon_ssh_ok "$_VERS_SSH_HOST"; then
err "ssh" "cannot reach ${_VERS_SSH_HOST} (stagepaysan) — skip container package compare"
info "outside-only" "deb.taler.net public checks completed above"
summary
exit 1
fi
ok "ssh ${_VERS_SSH_HOST}" "stagepaysan (VERS_ACCESS=ssh)"
fi
INRELEASE_URL="${APT_BASE}/dists/${SUITE}/InRelease"
# Write remote script to a file (avoids bash 3.2 parse bugs with case/;; inside $(…)<<heredoc).
{
printf 'INRELEASE_URL=%q\n' "$INRELEASE_URL"
printf 'WANT_BANK=%q; WANT_EX=%q; WANT_MER=%q\n' "$_VERS_BANK" "$_VERS_EX" "$_VERS_MER"
cat <<'REMOTE'
set +e
resolve_ctr() {
local want="$1" c
c=$(podman ps --format '{{.Names}}' 2>/dev/null | grep -Fx "$want" | head -1)
if [ -n "$c" ]; then echo "$c"; return; fi
case "$want" in
*exchange*) podman ps --format '{{.Names}}' 2>/dev/null | grep -i exchange | head -1 ;;
*bank*) podman ps --format '{{.Names}}' 2>/dev/null | grep -iE 'stage.*bank|hacktivism-bank|taler-bank|lfp-bank' | head -1 ;;
*)
c=$(podman ps --format '{{.Names}}' 2>/dev/null | grep -E '^taler-hacktivism$|^stage-lfp-merchant$' | head -1)
if [ -n "$c" ]; then echo "$c"; return; fi
podman ps --format '{{.Names}}' 2>/dev/null | grep -iE 'merchant|hacktivism' | grep -viE 'bank|exchange' | head -1
;;
esac
}
for want in "$WANT_EX" "$WANT_BANK" "$WANT_MER"; do
c=$(resolve_ctr "$want")
[ -n "$c" ] || continue
role=other
case "$c" in
*exchange*) role=exchange ;;
*bank*) role=bank ;;
*) role=merchant ;;
esac
code=$(podman exec "$c" curl -sS -m 12 -o /dev/null -w '%{http_code}' "$INRELEASE_URL" 2>/dev/null || echo 000)
echo "R|${role}|${c}|${code}"
if podman exec "$c" bash -lc 'grep -Rqs deb.taler.net /etc/apt/sources.list /etc/apt/sources.list.d 2>/dev/null'; then
echo "S|${role}|${c}|yes"
else
echo "S|${role}|${c}|no"
fi
podman exec "$c" dpkg-query -W -f '${Package}\t${Version}\n' 2>/dev/null \
| awk -v role="$role" -v ctr="$c" '
$1 ~ /^(taler-|libeufin-|libtaler|libdonau)/ {
printf "P|%s|%s|%s|%s\n", role, ctr, $1, $2
}'
done
REMOTE
} >"$tmp/remote-versions.sh"
if [ "$_VERS_LOCAL" = "1" ]; then
REMOTE_OUT=$(bash "$tmp/remote-versions.sh" 2>/dev/null || true)
elif [ "$_VERS_SSH_HOST" = "${KOOPA_SSH:-}" ] || [ "${INSIDE_PROFILE:-}" = "koopa" ]; then
REMOTE_OUT=$(koopa_ssh_bash 60 <"$tmp/remote-versions.sh" || true)
else
REMOTE_OUT=$(mon_ssh_bash "$_VERS_SSH_HOST" 60 <"$tmp/remote-versions.sh" || true)
fi
printf '%s\n' "$REMOTE_OUT" | grep -E '^R\|' >"$tmp/reach.tsv" || true
printf '%s\n' "$REMOTE_OUT" | grep -E '^S\|' >"$tmp/sources.tsv" || true
printf '%s\n' "$REMOTE_OUT" | grep -E '^P\|' >"$tmp/installed.tsv" || true
if [ -s "$tmp/reach.tsv" ]; then
while IFS='|' read -r _ role ctr code; do
if [ "$code" = "200" ]; then
ok "container ${role} → deb.taler.net" "${ctr} InRelease HTTP $code"
else
fail "container ${role} → deb.taler.net" "${ctr} InRelease HTTP $code — apt install/update will fail"
fi
done <"$tmp/reach.tsv"
else
warn "container → deb.taler.net" "no reachability rows"
fi
if [ -s "$tmp/sources.tsv" ]; then
while IFS='|' read -r _ role ctr has; do
if [ "$has" = "yes" ]; then
ok "container ${role} apt source" "${ctr} lists deb.taler.net"
else
warn "container ${role} apt source" "${ctr} no deb.taler.net in sources.list*"
fi
done <"$tmp/sources.tsv"
fi
if [ ! -s "$tmp/installed.tsv" ]; then
fail "installed packages" "none found in taler containers"
summary
exit 1
fi
n_inst=$(wc -l <"$tmp/installed.tsv" | tr -d ' ')
ok "collected installed packages" "${n_inst} rows"
# ---------------------------------------------------------------------------
# 3) compare installed vs trixie (and note testing)
# ---------------------------------------------------------------------------
set_group compare
section "versions · compare installed vs ${SUITE}"
export BEHIND_MODE HAVE_TESTING
export TALER_APT_SUITE="$SUITE"
CORE_PKGS="taler-exchange taler-exchange-database taler-merchant taler-merchant-webui libeufin-bank libeufin-common libtalerexchange libtalermerchant taler-terms-generator"
export CORE_PKGS
python3 - "$tmp/Packages" "$tmp/Packages-testing" "$tmp/installed.tsv" <<'PY' >"$tmp/py.out"
import sys, os
from collections import defaultdict
main_idx, test_idx, inst_path = sys.argv[1:4]
have_testing = os.environ.get("HAVE_TESTING", "0") == "1"
core = set(os.environ.get("CORE_PKGS", "").split())
suite = os.environ.get("TALER_APT_SUITE", "trixie")
def _ord_ch(c):
if c == "~":
return -1
if c.isdigit():
return 0
if c.isalpha():
return ord(c)
return ord(c) + 256
def _split_ver(v):
epoch = 0
if ":" in v:
e, v = v.split(":", 1)
try:
epoch = int(e)
except ValueError:
epoch = 0
if "-" in v:
upstream, deb = v.rsplit("-", 1)
else:
upstream, deb = v, ""
return epoch, upstream, deb
def _cmp_part(a, b):
ia = ib = 0
while ia < len(a) or ib < len(b):
while True:
ca = a[ia] if ia < len(a) and not a[ia].isdigit() else ""
cb = b[ib] if ib < len(b) and not b[ib].isdigit() else ""
if ca == "" and cb == "":
break
if ca == "" and cb:
return -1 if _ord_ch(cb) > 0 else (1 if cb == "~" else -1)
if cb == "" and ca:
return 1 if _ord_ch(ca) > 0 else (-1 if ca == "~" else 1)
oa, ob = _ord_ch(ca), _ord_ch(cb)
ia += 1
ib += 1
if oa != ob:
return (oa > ob) - (oa < ob)
sa = sb = ""
while ia < len(a) and a[ia].isdigit():
sa += a[ia]; ia += 1
while ib < len(b) and b[ib].isdigit():
sb += b[ib]; ib += 1
na = int(sa) if sa else 0
nb = int(sb) if sb else 0
if na != nb:
return (na > nb) - (na < nb)
return 0
def deb_cmp(a, b):
if a == b:
return 0
ea, ua, da = _split_ver(a)
eb, ub, db = _split_ver(b)
if ea != eb:
return (ea > eb) - (ea < eb)
c = _cmp_part(ua, ub)
if c:
return c
return _cmp_part(da, db)
def parse_packages(path):
pkgs = {}
if not path or not os.path.isfile(path) or os.path.getsize(path) == 0:
return pkgs
cur = ver = None
with open(path, encoding="utf-8", errors="replace") as f:
for line in f:
line = line.rstrip("\n")
if line.startswith("Package: "):
if cur and ver and (cur not in pkgs or deb_cmp(ver, pkgs[cur]) > 0):
pkgs[cur] = ver
cur = line[9:].strip()
ver = None
elif line.startswith("Version: ") and cur:
ver = line[9:].strip()
elif line == "" and cur:
if ver and (cur not in pkgs or deb_cmp(ver, pkgs[cur]) > 0):
pkgs[cur] = ver
cur = ver = None
if cur and ver and (cur not in pkgs or deb_cmp(ver, pkgs[cur]) > 0):
pkgs[cur] = ver
return pkgs
main = parse_packages(main_idx)
testing = parse_packages(test_idx) if have_testing else {}
by_pkg = defaultdict(list)
with open(inst_path, encoding="utf-8") as f:
for line in f:
line = line.strip()
if not line.startswith("P|"):
continue
parts = line.split("|")
if len(parts) < 5:
continue
_, role, ctr, pkg, ver = parts[:5]
by_pkg[pkg].append((role, ctr, ver))
for pkg in sorted(by_pkg.keys()):
versions = sorted({v for _, _, v in by_pkg[pkg]}, key=lambda v: v)
# pick "highest" via deb_cmp
inst = versions[0]
for v in versions[1:]:
if deb_cmp(v, inst) > 0:
inst = v
roles = ",".join(sorted({r for r, _, _ in by_pkg[pkg]}))
multi = len(set(versions)) > 1
trixie = main.get(pkg)
testv = testing.get(pkg)
status = "ok"
detail = ""
if multi:
status = "warn"
detail = "multiple installed: " + ", ".join(sorted(set(versions)))
if trixie is None:
if status == "ok":
status = "info"
detail = (detail + "; " if detail else "") + f"not in {suite} main index"
else:
c = deb_cmp(inst, trixie)
if c == 0:
detail = (detail + "; " if detail else "") + f"= {suite} {trixie}"
elif c < 0:
status = "behind"
detail = f"installed {inst} < {suite} {trixie}"
else:
status = "ahead"
detail = f"installed {inst} > {suite} {trixie}"
if testv and deb_cmp(inst, testv) == 0:
detail += f" (={suite}-testing)"
elif testv:
detail += f" (testing has {testv})"
core_flag = "1" if pkg in core else "0"
print(f"{status}|{pkg}|{inst}|{trixie or '-'}|{roles}|{core_flag}|{detail}")
PY
ok_n=0; ahead_n=0; behind_n=0
while IFS='|' read -r status pkg inst trixie roles core detail; do
[ -n "${status:-}" ] || continue
label="pkg ${pkg} (${roles})"
case "$status" in
ok)
ok "$label" "${inst}"
ok_n=$((ok_n + 1))
;;
ahead)
info "$label" "$detail"
ahead_n=$((ahead_n + 1))
;;
behind)
behind_n=$((behind_n + 1))
if [ "$core" = "1" ] || [ "$BEHIND_MODE" = "error" ]; then
fail "$label" "$detail"
else
warn "$label" "$detail"
fi
;;
warn)
warn "$label" "$detail"
;;
info)
info "$label" "installed ${inst}${detail:+ — $detail}"
;;
*)
info "$label" "$status $detail"
;;
esac
done <"$tmp/py.out"
# Core packages must exist somewhere
for need in taler-exchange libeufin-bank taler-merchant; do
if grep -qE "^P\|[^|]+\|[^|]+\|${need}\|" "$tmp/installed.tsv"; then
ok "core installed ${need}"
else
fail "core installed ${need}" "not present in any taler container"
fi
done
info "suite" "deb.taler.net ${SUITE} (testing_index=${HAVE_TESTING})"
info "tally" "match=${ok_n} ahead=${ahead_n} behind=${behind_n}"
# Disk free on the machine(s) we just probed (host-podman or SSH target)
set_group disk
section "versions · disk free space"
if [ "${_VERS_LOCAL:-0}" = "1" ]; then
mon_disk_check_host "host" || true
while read -r _vc; do
[ -n "$_vc" ] || continue
mon_disk_check_podman "$_vc" || true
done < <(podman ps --format '{{.Names}}' 2>/dev/null | grep -iE 'hacktivism|taler-|stage-lfp|lfp-' || true)
elif [ -n "${_VERS_SSH_HOST:-}" ]; then
_vdisk=$(mon_ssh_bash "$_VERS_SSH_HOST" 20 <<'DISK' || true
set +e
echo "###HOST###"
df -Pk / /var /home /tmp 2>/dev/null || df -Pk
echo "###CTRS###"
for c in $(podman ps --format '{{.Names}}' 2>/dev/null); do
echo "###CTR $c###"
podman exec "$c" df -Pk / /var /tmp 2>/dev/null || true
done
DISK
)
mon_disk_check_remote_text "ssh:${_VERS_SSH_HOST}" \
"$(printf '%s\n' "$_vdisk" | sed -n '/^###HOST###$/,/^###CTRS###$/p' | sed '1d;$d')" || true
_ctr=""; _buf=""
while IFS= read -r _line || [ -n "$_line" ]; do
case "$_line" in
'###CTR '*)
[ -n "$_ctr" ] && mon_disk_check_remote_text "ctr:${_ctr}" "$_buf" || true
_ctr=${_line####CTR }; _ctr=${_ctr%###}; _buf=""
;;
'###CTRS###'|'###HOST###') ;;
*) [ -n "$_ctr" ] && _buf="${_buf}${_line}"$'\n' ;;
esac
done <<<"$_vdisk"
[ -n "$_ctr" ] && mon_disk_check_remote_text "ctr:${_ctr}" "$_buf" || true
fi
summary

48
domains.conf Normal file
View file

@ -0,0 +1,48 @@
# taler-monitoring domain profiles
#
# When you add a stack, declare the three public endpoints here:
# name bank exchange merchant-backend currency local landing [canonical]
#
# - name: what -d / TALER_DOMAIN matches (aliases allowed as extra lines)
# - bank / exchange / merchant: hostname or full https:// URL
# - currency: GOA | KUDOS | CHF | … (empty = report only)
# - local: 1 = koopa SSH/inside/e2e stack; 0 = public-only
# - landing: 1 = check /intro landings + assets; 0 = skip (no GOA-style landings)
# - canonical: optional TALER_DOMAIN label after alias match
#
# CLI overrides after load: --bank --exchange --merchant --currency
# Alternate file: TALER_DOMAINS_CONF=/path/to/domains.conf
# Local GOA (koopa) — landings matter
koopa bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch GOA 1 1 hacktivism.ch
hacktivism bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch GOA 1 1 hacktivism.ch
hacktivism.ch bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch GOA 1 1
# Public demo (KUDOS) — has intro-style pages; keep soft landing checks
taler.net bank.demo.taler.net exchange.demo.taler.net backend.demo.taler.net KUDOS 0 1 demo.taler.net
demo.taler.net bank.demo.taler.net exchange.demo.taler.net backend.demo.taler.net KUDOS 0 1
# Public test stack (TESTKUDOS) — no GOA-style landings
test.taler.net bank.test.taler.net exchange.test.taler.net backend.test.taler.net TESTKUDOS 0 0
# TOPS CHF — multi-tenant merchant my.taler-ops.ch; no public GOA landings
taler-ops.ch bank.taler-ops.ch exchange.taler-ops.ch my.taler-ops.ch CHF 0 0
my.taler-ops.ch bank.taler-ops.ch exchange.taler-ops.ch my.taler-ops.ch CHF 0 0 taler-ops.ch
# Stage merchant FQDN is stage.my.taler-ops.ch (on betel); mon HTML /monitoring* stage-only
stage.taler-ops.ch bank.stage.taler-ops.ch exchange.stage.taler-ops.ch stage.my.taler-ops.ch CHF 0 0
stage.my.taler-ops.ch bank.stage.taler-ops.ch exchange.stage.taler-ops.ch stage.my.taler-ops.ch CHF 0 0 stage.taler-ops.ch
my.stage.taler-ops.ch bank.stage.taler-ops.ch exchange.stage.taler-ops.ch stage.my.taler-ops.ch CHF 0 0 stage.taler-ops.ch
# Franc Paysan (Infomaniak) — merchant host is monnaie.* (not taler./backend.)
# Prod: no monetary launch yet; stage uses TESTPAYSAN (public HTTPS landings + shops)
lefrancpaysan.ch bank.lefrancpaysan.ch exchange.lefrancpaysan.ch monnaie.lefrancpaysan.ch - 0 0
monnaie.lefrancpaysan.ch bank.lefrancpaysan.ch exchange.lefrancpaysan.ch monnaie.lefrancpaysan.ch - 0 0 lefrancpaysan.ch
bank.lefrancpaysan.ch bank.lefrancpaysan.ch exchange.lefrancpaysan.ch monnaie.lefrancpaysan.ch - 0 0 lefrancpaysan.ch
exchange.lefrancpaysan.ch bank.lefrancpaysan.ch exchange.lefrancpaysan.ch monnaie.lefrancpaysan.ch - 0 0 lefrancpaysan.ch
# Stage TESTPAYSAN — landings on / + farmer shops under stage.monnaie…/shops/
# name bank exchange merchant currency local landing [canonical]
stage.lefrancpaysan.ch stage.bank.lefrancpaysan.ch stage.exchange.lefrancpaysan.ch stage.monnaie.lefrancpaysan.ch TESTPAYSAN 0 1
stage.bank.lefrancpaysan.ch stage.bank.lefrancpaysan.ch stage.exchange.lefrancpaysan.ch stage.monnaie.lefrancpaysan.ch TESTPAYSAN 0 1 stage.lefrancpaysan.ch
stage.exchange.lefrancpaysan.ch stage.bank.lefrancpaysan.ch stage.exchange.lefrancpaysan.ch stage.monnaie.lefrancpaysan.ch TESTPAYSAN 0 1 stage.lefrancpaysan.ch
stage.monnaie.lefrancpaysan.ch stage.bank.lefrancpaysan.ch stage.exchange.lefrancpaysan.ch stage.monnaie.lefrancpaysan.ch TESTPAYSAN 0 1 stage.lefrancpaysan.ch

7
harness-live.txt Normal file
View file

@ -0,0 +1,7 @@
# Optional: taler-harness against *our* exchange (needs a harness version
# that accepts current /config JSON). Older installs often FAIL codec checks
# even when the exchange is fine — trust `./taler-monitoring.sh urls` first.
#
# id<TAB>command
lint-exchange-goa taler-harness deployment lint-exchange-url https://exchange.hacktivism.ch/

41
harness-tests.txt Normal file
View file

@ -0,0 +1,41 @@
# Real-life–relevant taler-harness integration tests
# (from: taler-harness list-integrationtests)
#
# Criteria: everyday user/merchant/bank paths — withdraw, pay, deposit, refund,
# libeufin bank, templates/paywall. Excludes KYC edge cases, experimental,
# timetravel, perf, backup, mailbox, most fault-injection.
#
# These spin up a local TESTKUDOS stack (same client code paths as production).
# For tests against *public* demos, see harness-live.txt / `./taler-monitoring.sh live`.
# --- Core path: get money + spend it (the GOA story) ---
simple-payment
withdrawal-bank-integrated
payment
# --- Withdraw variants users actually hit ---
withdrawal-manual
withdrawal-external
withdrawal-idempotent
# --- Pay variants shops use ---
payment-template
paywall-flow
payment-idempotency
payment-zero
payment-abort
otp
# --- After pay: coins / merchant ops ---
deposit
refund
refund-auto
wallet-refresh
# --- Bank stack we run (libeufin / regional) ---
libeufin-bank
bank-api
# --- Client hygiene ---
wallet-config
term-of-service-format

193
host-agent/README.md Normal file
View file

@ -0,0 +1,193 @@
# host-agent — reporting generation (GOA + FP + any stack)
**Global pipeline** lives in **`run-host-report.sh`** (all stacks share it):
| Behaviour | Default |
|-----------|---------|
| Suite refresh | **`update-suite.sh` every run** → `git fetch --tags` + hard reset to `origin/main` |
| Update strict | `SUITE_UPDATE_STRICT=1` (default): **abort** if fetch/reset fails — no silent old tree |
| Wall clock | `RUN_TIMEOUT=600` |
| Log | line-buffered `run-*.log` (continuous write) |
| HTML | always (first run / empty tree too) + commit URL; **bootstrap install stubs are monpages ERROR** |
| Timeout notice | end of log + top jump on `/monitoring_err/` |
**Rule:** public mon pages must show a **current suite tag/commit**, not install bootstrap or months-old trees. Timers exist so this keeps working without manual SSH.
Thin wrappers only set stack defaults:
| Wrapper | Stack | Typical user |
|---------|-------|----------------|
| `run-hacktivism-monitoring.sh` | GOA `hacktivism.ch` | `hernani` @ koopa |
| `run-fp-stage-monitoring.sh` | FP stage | `stagepaysan` |
| `run-host-report.sh` + `taler-monitoring-mytops-stage.{service,timer}` | mytops **stage / betel** only | `taler-monitoring` @ betel (4h) |
| `run-aptdeploy.sh` | CLI: ensure + **aptdeploy** (no HTML) | local if hostname=koopa, else **ssh $DEPLOY_SSH** |
| `run-aptdeploy-monitoring.sh` | apt-src tests → `/taler-monitoring-aptdeploy*` | `hernani` @ koopa (4h) |
| `run-surface-monitoring.sh` | **surface** → only `/taler-monitoring-surface*` (ecosystem + mail + mattermost + versions) | `hernani` @ koopa (hourly) |
Deprecated mail/mattermost wrappers: **`../meta/deprecated/`** (not installed).
### apt-src deploy tests (hacktivism `/monitoring`)
| Container | Mode |
|-----------|------|
| `koopa-taler-deploy-test-apt-src-trixie` | **fresh**: recreate when repo package fingerprint changes |
| `koopa-taler-deploy-test-apt-src-trixie-testing` | **fresh** for `trixie-testing` |
| `koopa-taler-deploy-test-apt-src-trixie-upgrade` | **upgrade**: initial install, then mytops-style `apt install` + `upgrade` |
| `koopa-taler-deploy-test-apt-src-trixie-testing-upgrade` | **upgrade** for testing suite |
```bash
# from ops workstation → triggers on koopa-external
./host-agent/run-aptdeploy.sh
./host-agent/run-aptdeploy.sh upgrade
./host-agent/run-aptdeploy.sh fresh
./host-agent/run-aptdeploy.sh check-only
# on koopa
./taler-monitoring.sh -d hacktivism.ch aptdeploy
```
Checks: package table, `httpd --version`, libtalerutil/`ldd` on failure, `systemctl` for
`taler-merchant.target`, basic local `/config` probe. No nginx/HTTPS in containers.
### Public HTML layout (v1.8.0 simplified)
| URL | Role |
|-----|------|
| `https://{bank,exchange,merchant}/monitoring/` (+ `_err`) | **Landing-stack** report — GOA 3 + FP **stage** 3 (+ mytops stage as configured) |
| `https://taler.hacktivism.ch/taler-monitoring-surface/` (+ `_err`) | **Only** ecosystem page: software/versions in the Taler universe (nmap OS, mail firefly+anastasis, mattermost, …) |
No separate public pages for mail/mattermost. Staging under `~/monitoring-sites-staging/`.
```bash
systemctl --user enable --now taler-monitoring-surface.timer
systemctl --user start taler-monitoring-surface.service
```
Env examples: **`env/*.env.example`**. FP stage: `run-fp-stage-monitoring.sh` (same `run-host-report.sh` pipeline).
## GOA (koopa) — host-podman
Runs **as user `hernani`** (systemd user, linger). **Primary** mode:
- **`INSIDE_PODMAN=1`** / **`INSIDE_MODE=local-podman`**: access mode **`host-podman`** —
`podman exec` on this host into
`taler-hacktivism-bank` / `taler-hacktivism-exchange-ansible` / `taler-hacktivism`
(IDs `inside.host-*`, not `inside.ssh-*`).
From an **ops workstation**, leave `INSIDE_PODMAN` off → access mode **`ssh`** via `KOOPA_SSH`
(`inside.ssh-*`).
- Public **`urls` + `versions`** for the stack (`-d hacktivism.ch`).
- Writes console HTML for all three sites under
`~/monitoring-sites-staging/{bank,exchange,taler}.hacktivism.ch/`.
Caddy (host root) can serve that tree at `/monitoring` — you configure Caddy.
## Latest software + config isolation
**Source of truth is always Forgejo** (`git.hacktivism.ch`), not a laptop rsync copy.
Every host-agent / aptdeploy run:
1. **`update-suite.sh`**: `git fetch` + **`SUITE_UPDATE_MODE=reset`** (default) → `origin/main`
into `SUITE_DIR` (prefer `~/src/taler-monitoring`).
2. **Re-exec** host-agent scripts from that tree so the running code matches `COMMIT_URL`.
3. Local overrides only in **`~/.config/taler-monitoring/env`** — never overwritten by git.
4. HTML footer = exact commit:
`https://git.hacktivism.ch/hernani/taler-monitoring/src/commit/<full-sha>`
5. **`RUN_TIMEOUT=600`**, line-buffered logs, HTML always written.
If fetch fails → previous tree + WARN.
**To ship features (aptdeploy, surface, …): push to `origin/main`.** Local-only edits vanish on the next reset.
If pull fails (no network / auth), the previous tree is used and a WARN is logged.
## Install (on koopa as hernani)
```bash
# once: local config (optional)
mkdir -p ~/.config/taler-monitoring
cp ~/src/taler-monitoring/host-agent/env.example \
~/.config/taler-monitoring/env
# ops workstation:
./host-agent/install-host-agent.sh --remote $DEPLOY_SSH
# or on koopa:
~/src/taler-monitoring/host-agent/install-host-agent.sh
```
Units:
| Unit | Role |
|------|------|
| `taler-monitoring-hacktivism.path` | fires on software stamp (`touch-software-stamp.sh`) |
| `taler-monitoring-hacktivism.timer` | every **4h** fallback |
| `taler-monitoring-hacktivism.service` | oneshot → `/monitoring*` (urls inside versions) |
| `taler-monitoring-surface.timer` | every **1h** |
| `taler-monitoring-surface.service` | → `/taler-monitoring-surface*` (taler only) |
| `taler-monitoring-aptdeploy.timer` | every **4h** |
| `taler-monitoring-aptdeploy.service` | → `/taler-monitoring-aptdeploy*` (taler only) |
## After package / image / landing upgrades
```bash
~/src/taler-monitoring/host-agent/touch-software-stamp.sh
# → path unit starts the service
```
## Manual
```bash
systemctl --user start taler-monitoring-hacktivism.service
systemctl --user status taler-monitoring-hacktivism.service --no-pager
journalctl --user -u taler-monitoring-hacktivism.service -n 50 --no-pager
ls ~/monitoring-sites-staging/bank.hacktivism.ch/monitoring_err/
```
## Relation to optional outside-only timer
| | host-agent (koopa) | outside-runner site-gen |
|--|--------------------|-------------------|
| Access | **podman on host** | public HTTPS only |
| Phases | urls **inside** versions | urls versions |
| Trigger | software path + 4h | *could* launchd 4h |
| Primary for GOA | **yes** | **off** (optional only) |
**outside-runner is not scheduled** right now (launchd disabled). Scripts under
`site-gen/` can still one-shot a public run if needed; see `site-gen/README.md`.
No secrets required for default phases. `secrets.env` optional for e2e later.
## Git commit on HTML pages
Header of each generated page includes a link like:
`https://git.hacktivism.ch/hernani/taler-monitoring/src/commit/<full-sha>`
That SHA is the tree **after** `update-suite.sh` for that run.
## Packages on koopa
See **[../DEPENDENCIES.md](../DEPENDENCIES.md)**.
**Default host-agent needs no new packages** if curl/python3/podman/git are present (they are).
**Root once** only for full QR checks in `urls`:
```bash
zypper in qrencode zbar
```
Without them, monitoring still runs; QR group WARNs and skips encode/decode.
Caddy `/var/www/monitoring-sites` is separate (you configured hernani write access).
## Exit codes (v1.7.1+)
Host-agent jobs default to **`STRICT_EXIT=1`**: non-zero exit if suite checks fail, `DEPLOY_WWW_ROOT` is not writable, or public monpages (FQDN) fail.
Set `STRICT_EXIT=0` only if you want timers to stay green while still logging errors (not recommended for production).
## Monitoring pages (suite)
Public sticky-bar HTML under `/monitoring*` and `/taler-monitoring-*` is part of **taler-monitoring** (not a separate product). See main [README — Monitoring pages](../README.md#monitoring-pages-part-of-the-suite). Phase **monpages** is obligatory (**ERROR** if missing): GOA checks the full suite inventory; FP only FP mon hosts.

74
host-agent/ROOT-APPLY-MONITORING.md Normal file
View file

@ -0,0 +1,74 @@
# Root: make monitoring pages live on koopa (GOA)
## Detection (v1.3.1+)
Suite phase **`monpages`** and the host-agent **post-check** curl each public
FQDN (`https://taler.hacktivism.ch/monitoring/` …). Merchant JSON **code 21**
or non-HTML is an **ERROR** until this apply is done.
## Symptom if not applied
Public URLs return **merchant/bank JSON** like:
```json
{ "code": 21, "hint": "There is no endpoint defined for the URL provided..." }
```
or a **404 HTML** page from the app reverse_proxy.
Cause: live `/etc/caddy/Caddyfile` has **no** `handle /monitoring*` (falls through to
API). Prepared config is only under **`$KOOPA_CADDY_DIR/Caddyfile`**.
## One-shot as root (recommended)
```bash
# absolute path required: as root, ~ is /root (not /home/hernani)
sudo $APPLY_MONITORING_LIVE
# or already root:
# $APPLY_MONITORING_LIVE
```
**Caddy bare-path redir (v1.9.2+):** inside a `handle` block use
`redir * /path/ 302` — never `redir /path/ 302` alone (Caddy treats the first
token as matcher and sets `Location: 302`, so bare URLs fall through to the
merchant API as JSON **code 21**).
That script:
1. `rsync` staging HTML → `/var/www/monitoring-sites/` (incl. surface + aptdeploy)
2. Makes tree writable for `hernani` (group `caddy` + ACL) so later runs self-sync
3. Installs prepared Caddyfile → `/etc/caddy/Caddyfile`, validates, reloads Caddy
4. Smoke-curls all monitoring URLs (must **not** be JSON `code:21`)
## Paths (only three families — suite name ≠ URL)
There is **no** public page `/taler-monitoring`. Each family needs **bare and slash**
(`redir * /path/ 302` inside Caddy `handle`):
| Family | Paths | Host | Source timer |
|--------|-------|------|----------------|
| **monitoring** | `/monitoring` · `/monitoring/` · `/monitoring_err` · `/monitoring_err/` | bank + exchange + taler (and other landings) | `taler-monitoring-hacktivism.timer` (4h) |
| **taler-monitoring-surface** | bare + slash + `_err` | **taler.hacktivism.ch only** | `taler-monitoring-surface.timer` (1h) |
| **taler-monitoring-aptdeploy** | bare + slash + `_err` | **taler.hacktivism.ch only** | `taler-monitoring-aptdeploy.timer` (4h) |
HTML is generated as **hernani** into
`$DEPLOY_STAGING (or $HOME/monitoring-sites-staging)/`.
## Manual equivalent
```bash
rsync -a --delete $DEPLOY_STAGING (or $HOME/monitoring-sites-staging)/ /var/www/monitoring-sites/
chown -R hernani:caddy /var/www/monitoring-sites && chmod -R g+rwX /var/www/monitoring-sites
sudo $KOOPA_CADDY_DIR/apply.sh # or apply-monitoring-live.sh
```
Expect **200** (HTML with sticky bar). Not merchant JSON `code:21`.
## After each monitoring run (if www is not hernani-writable)
```bash
rsync -a --delete $DEPLOY_STAGING (or $HOME/monitoring-sites-staging)/ /var/www/monitoring-sites/
chown -R caddy:caddy /var/www/monitoring-sites
```
Or grant `hernani` write on `/var/www/monitoring-sites` once so the host-agent can rsync itself.

91
host-agent/apply-monitoring-live.sh Executable file
View file

@ -0,0 +1,91 @@
#!/usr/bin/env bash
# ONE-SHOT as root: publish monitoring HTML + enable Caddy handles
#
# Suite copy (taler-monitoring host-agent). Live install path on koopa:
# ${KOOPA_CADDY_DIR}/apply-monitoring-live.sh
# Prefer absolute path ( ~ expands to /root when already root ):
# sudo ${KOOPA_CADDY_DIR}/apply-monitoring-live.sh
# ${KOOPA_CADDY_DIR}/apply-monitoring-live.sh # if already root
#
# After editing here: rsync to koopa ~/koopa-caddy/ before root apply.
set -euo pipefail
if [[ "$(id -u)" -ne 0 ]]; then
exec sudo -E "$0" "$@"
fi
STAGING="${DEPLOY_STAGING:-${HTML_OUT:-$HOME/monitoring-sites-staging}}"
WWW="${DEPLOY_WWW_ROOT:?set DEPLOY_WWW_ROOT in env}"
SRC="${KOOPA_CADDY_DIR:?set KOOPA_CADDY_DIR in env}/Caddyfile"
DST=/etc/caddy/Caddyfile
if [[ ! -f "$SRC" ]]; then
echo "ERROR: missing Caddyfile source: $SRC" >&2
exit 1
fi
if [[ ! -d "$STAGING" ]]; then
echo "ERROR: missing staging HTML: $STAGING" >&2
exit 1
fi
echo "== 1) HTML staging → www =="
mkdir -p "$WWW"
rsync -a --delete "$STAGING/" "$WWW/"
# allow hernani to update future runs without root
chown -R hernani:caddy "$WWW"
chmod -R g+rwX "$WWW"
find "$WWW" -type d -exec chmod g+s {} \;
# ACL default if available
if command -v setfacl >/dev/null 2>&1; then
setfacl -R -m u:hernani:rwx,g:caddy:rwx "$WWW" || true
setfacl -R -d -m u:hernani:rwx,g:caddy:rwx "$WWW" || true
fi
ls -la "$WWW" "$WWW/taler.hacktivism.ch" || true
echo "== 2) Caddyfile (backup + install) =="
ts=$(date +%Y%m%d-%H%M%S)
cp -a "$DST" "/etc/caddy/Caddyfile.bak-monitoring-${ts}"
cp -a "$SRC" "$DST"
chown root:caddy "$DST" 2>/dev/null || chown root:root "$DST"
chmod 644 "$DST"
caddy validate --config "$DST"
systemctl reload caddy
systemctl is-active caddy
echo "== 3) smoke =="
# bash: every continued line except the last must end with \
urls=(
https://taler.hacktivism.ch/monitoring/
https://bank.hacktivism.ch/monitoring/
https://exchange.hacktivism.ch/monitoring/
https://taler.hacktivism.ch/taler-monitoring-surface
https://taler.hacktivism.ch/taler-monitoring-surface/
https://taler.hacktivism.ch/taler-monitoring-surface_err/
https://taler.hacktivism.ch/taler-monitoring-aptdeploy/
https://taler.hacktivism.ch/taler-monitoring-aptdeploy_err/
)
smoke_fail=0
for url in "${urls[@]}"; do
code=$(curl -sS -o /tmp/monchk -w '%{http_code}' -L --max-redirs 3 -m 12 "$url" || echo ERR)
ct=$(file -b /tmp/monchk 2>/dev/null | head -c 40)
echo " $code $url ($ct)"
if grep -qE '"code":[[:space:]]*21' /tmp/monchk 2>/dev/null; then
echo " STILL merchant JSON code 21 — Caddy handles not active?"
smoke_fail=1
fi
done
# bare path must redirect (or land on HTML), not merchant JSON
bare_code=$(curl -sS -o /tmp/monbare -w '%{http_code}' --max-redirs 0 -m 12 \
https://taler.hacktivism.ch/taler-monitoring-surface || echo ERR)
if grep -qE '"code":[[:space:]]*21' /tmp/monbare 2>/dev/null; then
echo "ERROR: bare /taler-monitoring-surface still merchant code 21 (HTTP $bare_code)" >&2
smoke_fail=1
elif [[ "$bare_code" != "302" && "$bare_code" != "301" && "$bare_code" != "200" ]]; then
echo "WARN: bare /taler-monitoring-surface HTTP $bare_code (want 302/301/200 HTML)" >&2
fi
echo " bare_no_follow=$bare_code https://taler.hacktivism.ch/taler-monitoring-surface"
if [[ "$smoke_fail" -ne 0 ]]; then
echo "ERROR: smoke failed" >&2
exit 1
fi
echo "OK apply-monitoring-live done"

368
host-agent/ensure-apt-deploy-test-containers.sh Executable file
View file

@ -0,0 +1,368 @@
#!/usr/bin/env bash
# ensure-apt-deploy-test-containers.sh
#
# Four rootless podman containers on koopa (no nginx / no HTTPS):
#
# Fresh rebuild when repo fingerprint changes:
# koopa-taler-deploy-test-apt-src-trixie
# koopa-taler-deploy-test-apt-src-trixie-testing
#
# Persistent upgrade path (mytops upgrade.sh style):
# koopa-taler-deploy-test-apt-src-trixie-upgrade
# koopa-taler-deploy-test-apt-src-trixie-testing-upgrade
#
# Run as hernani on koopa. From elsewhere use run-aptdeploy.sh (ssh $DEPLOY_SSH or $APT_DEPLOY_SSH).
#
set -euo pipefail
PODMAN_BIN="${APT_DEPLOY_PODMAN:-podman}"
IMG="${APT_DEPLOY_BASE_IMAGE:-docker.io/library/debian:trixie}"
STATE_DIR="${XDG_STATE_HOME:-$HOME/.local/state}/taler-apt-deploy"
URI_BASE="${APT_DEPLOY_URI:-https://deb.taler.net/apt/debian}"
mkdir -p "$STATE_DIR"
# Packages we track for "new version → rebuild fresh containers"
TRACK_PKGS="${APT_DEPLOY_TRACK_PKGS:-taler-merchant libtalermerchant libtalerexchange libdonau}"
# Canonical four containers (keep). Everything else matching legacy names is removed.
CANONICAL_CONTAINERS=(
koopa-taler-deploy-test-apt-src-trixie
koopa-taler-deploy-test-apt-src-trixie-testing
koopa-taler-deploy-test-apt-src-trixie-upgrade
koopa-taler-deploy-test-apt-src-trixie-testing-upgrade
)
# Legacy / typo names from earlier iterations — always delete if present
LEGACY_CONTAINERS=(
koopa-taler-build-test-apt-trixie
koopa-taler-build-test-apt-trixie-testing
koopa-taler-deploy-test-apt-trixie
koopa-taler-deploy-test-apt-trixie-testing
koopa-taler-buidl-test-apt-trixie
koopa-taler-buidl-test-apt-trixie-testing
)
cleanup_legacy() {
local n img
echo "======== cleanup legacy containers / images ========"
for n in "${LEGACY_CONTAINERS[@]}"; do
if "$PODMAN_BIN" container exists "$n" 2>/dev/null; then
echo "remove legacy container $n"
"$PODMAN_BIN" rm -f "$n" 2>/dev/null || true
fi
# old setup images for that name
if "$PODMAN_BIN" image exists "localhost/${n}:setup" 2>/dev/null; then
echo "remove legacy image localhost/${n}:setup"
"$PODMAN_BIN" rmi -f "localhost/${n}:setup" 2>/dev/null || true
fi
done
# dangling setup images not in canonical set
while read -r img; do
[ -n "$img" ] || continue
local base keep=0
base=${img#localhost/}
base=${base%:setup}
for n in "${CANONICAL_CONTAINERS[@]}"; do
[ "$base" = "$n" ] && keep=1 && break
done
if [ "$keep" -eq 0 ] && [[ "$img" == localhost/koopa-taler-*-test-apt* ]]; then
echo "remove non-canonical image $img"
"$PODMAN_BIN" rmi -f "$img" 2>/dev/null || true
fi
done < <("$PODMAN_BIN" images --format '{{.Repository}}:{{.Tag}}' 2>/dev/null | grep -E 'localhost/koopa-taler-.*-test-apt' || true)
}
# Prefer single suite tree: ~/src/taler-monitoring vs ~/taler-monitoring
# (legacy koopa-admin-log paths are not used). Keep the newer git clone; symlink the other.
dedupe_suite_repos() {
local a="$HOME/src/taler-monitoring" b="$HOME/taler-monitoring"
local ta tb
[ "${APT_DEPLOY_DEDUPE_REPOS:-1}" = "1" ] || return 0
if [ -d "$a/.git" ] && [ -d "$b/.git" ] && [ ! -L "$a" ] && [ ! -L "$b" ] \
&& [ "$(readlink -f "$a" 2>/dev/null || echo "$a")" != "$(readlink -f "$b" 2>/dev/null || echo "$b")" ]; then
ta=$(git -C "$a" log -1 --format=%ct 2>/dev/null || echo 0)
tb=$(git -C "$b" log -1 --format=%ct 2>/dev/null || echo 0)
echo "======== dedupe suite repos (both are git clones) ========"
echo " $a HEAD_ts=$ta"
echo " $b HEAD_ts=$tb"
if [ "$ta" -ge "$tb" ]; then
echo "keep $a · remove $b · symlink $b → $a"
rm -rf "$b"
ln -sfn "$a" "$b"
else
echo "keep $b · remove $a · symlink $a → $b"
rm -rf "$a"
mkdir -p "$(dirname "$a")"
ln -sfn "$b" "$a"
fi
elif [ -d "$a/.git" ] && [ ! -e "$b" ]; then
ln -sfn "$a" "$b"
elif [ -d "$b/.git" ] && [ ! -e "$a" ]; then
mkdir -p "$(dirname "$a")"
ln -sfn "$b" "$a"
fi
}
cleanup_legacy
dedupe_suite_repos
# ---------------------------------------------------------------------------
# Repo fingerprint: Version: lines from suite Packages index
# ---------------------------------------------------------------------------
repo_pkg_version() {
local suite=$1 pkg=$2
curl -fsS --max-time 45 \
"${URI_BASE}/dists/${suite}/main/binary-amd64/Packages" 2>/dev/null \
| awk -v p="$pkg" '
$0 == "Package: " p { want=1; next }
/^Package: / { want=0 }
want && /^Version: / { print $2; exit }
'
}
repo_fingerprint() {
local suite=$1 pkg v line=""
for pkg in $TRACK_PKGS; do
v=$(repo_pkg_version "$suite" "$pkg" || true)
[ -n "$v" ] || v="?"
line="${line}${pkg}=${v};"
done
printf '%s' "$line"
}
container_fingerprint() {
local name=$1
"$PODMAN_BIN" exec "$name" bash -lc '
for p in '"$TRACK_PKGS"'; do
v=$(dpkg-query -W -f="\${Version}" "$p" 2>/dev/null || echo missing)
printf "%s=%s;" "$p" "$v"
done
' 2>/dev/null || echo "missing"
}
# ---------------------------------------------------------------------------
# Bootstrap: install base + taler source + packages (shared)
# ---------------------------------------------------------------------------
install_base_and_merchant() {
local name=$1 suite=$2 srcfile=$3
"$PODMAN_BIN" exec -u root "$name" bash -lc '
set -euo pipefail
export DEBIAN_FRONTEND=noninteractive
apt-get update -qq
apt-get install -y -qq ca-certificates curl wget gnupg systemd systemd-sysv \
dbus procps iproute2
'
"$PODMAN_BIN" exec -u root "$name" bash -lc "
set -euo pipefail
export DEBIAN_FRONTEND=noninteractive
mkdir -p /etc/apt/keyrings
wget -q -O /etc/apt/keyrings/taler-systems.gpg https://taler.net/taler-systems.gpg
cat >/etc/apt/sources.list.d/${srcfile} <<EOF
Architectures: amd64
Components: main
X-Repolib-Name: Taler
Signed-By: /etc/apt/keyrings/taler-systems.gpg
Suites: ${suite}
Types: deb
URIs: ${URI_BASE}
EOF
apt-get update -qq
apt-get install -y \
taler-merchant \
taler-merchant-typst \
taler-merchant-webui \
taler-terms-generator \
postgresql \
s-nail \
jq
"
}
commit_and_systemd() {
local name=$1
"$PODMAN_BIN" stop "$name"
"$PODMAN_BIN" commit "$name" "localhost/${name}:setup"
"$PODMAN_BIN" rm "$name"
"$PODMAN_BIN" run -d --name "$name" \
--hostname "$name" \
--systemd=always \
--cgroupns=host \
"localhost/${name}:setup" \
/lib/systemd/systemd
sleep 5
# enable merchant stack if unit exists
"$PODMAN_BIN" exec "$name" bash -lc '
set +e
systemctl daemon-reload 2>/dev/null
systemctl enable taler-merchant.target 2>/dev/null
systemctl start taler-merchant.target 2>/dev/null
sleep 2
systemctl is-active taler-merchant-httpd.service 2>/dev/null || true
' || true
}
create_from_scratch() {
local name=$1 suite=$2 srcfile=$3
echo "======== CREATE (fresh) $name Suites: $suite ========"
"$PODMAN_BIN" pull "$IMG" >/dev/null
"$PODMAN_BIN" rm -f "$name" 2>/dev/null || true
# drop previous setup image so we do not accumulate duplicates
"$PODMAN_BIN" rmi -f "localhost/${name}:setup" 2>/dev/null || true
"$PODMAN_BIN" run -d --name "$name" --hostname "$name" "$IMG" sleep infinity
install_base_and_merchant "$name" "$suite" "$srcfile"
commit_and_systemd "$name"
printf '%s\n' "$(repo_fingerprint "$suite")" >"$STATE_DIR/${name}.fp"
printf '%s\n' "$(container_fingerprint "$name")" >"$STATE_DIR/${name}.installed"
echo "created $name fp=$(cat "$STATE_DIR/${name}.fp")"
}
# ---------------------------------------------------------------------------
# Upgrade path (mytops-admin-log upgrade.sh merchant packages, no restic/nginx)
# Prints a markdown-ish table of before → after versions.
# ---------------------------------------------------------------------------
upgrade_inside() {
local name=$1
local before after pkg v0 v1
echo "======== UPGRADE $name (mytops-style apt install + upgrade) ========"
before=$("$PODMAN_BIN" exec "$name" bash -lc '
for p in taler-merchant libtalermerchant libtalerexchange libdonau libgnunet \
taler-merchant-webui taler-merchant-typst taler-terms-generator; do
v=$(dpkg-query -W -f="\${Version}" "$p" 2>/dev/null || echo "-")
printf "%s %s\n" "$p" "$v"
done
' 2>/dev/null || true)
# capture upgrade output
set +e
"$PODMAN_BIN" exec -u root "$name" bash -lc '
set -euo pipefail
export DEBIAN_FRONTEND=noninteractive
systemctl stop taler-merchant.target 2>/dev/null || true
systemctl stop taler-merchant-donaukeyupdate.service 2>/dev/null || true
apt-get update -qq
apt-get install -y \
taler-merchant \
taler-merchant-typst \
taler-merchant-webui \
taler-terms-generator \
|| true
if dpkg -l challenger-httpd 2>/dev/null | grep -q "^ii"; then
apt-get remove -y challenger-httpd || true
fi
apt-get upgrade -y
systemctl daemon-reload 2>/dev/null || true
systemctl enable taler-merchant.target 2>/dev/null || true
# try dbconfig if present (may no-op without full config)
command -v taler-merchant-dbconfig >/dev/null && taler-merchant-dbconfig 2>/dev/null || true
systemctl reset-failed "taler-merchant*" 2>/dev/null || true
systemctl start taler-merchant.target 2>/dev/null || true
sleep 2
' 2>&1 | tee "$STATE_DIR/${name}.upgrade.log" | tail -30
set -e
after=$("$PODMAN_BIN" exec "$name" bash -lc '
for p in taler-merchant libtalermerchant libtalerexchange libdonau libgnunet \
taler-merchant-webui taler-merchant-typst taler-terms-generator; do
v=$(dpkg-query -W -f="\${Version}" "$p" 2>/dev/null || echo "-")
printf "%s %s\n" "$p" "$v"
done
' 2>/dev/null || true)
echo
echo "| package | before | after | change |"
echo "|---------|--------|-------|--------|"
while read -r pkg v0; do
[ -n "${pkg:-}" ] || continue
v1=$(printf '%s\n' "$after" | awk -v p="$pkg" '$1==p {print $2; exit}')
v1=${v1:--}
if [ "$v0" = "$v1" ]; then
ch="="
else
ch="UPGRADED"
fi
echo "| $pkg | $v0 | $v1 | $ch |"
done <<<"$before"
printf '%s\n' "$(container_fingerprint "$name")" >"$STATE_DIR/${name}.installed"
}
# ---------------------------------------------------------------------------
# Fresh mode: recreate if missing / not running / repo fp changed
# ---------------------------------------------------------------------------
ensure_fresh() {
local name=$1 suite=$2 srcfile=$3
local want have running
want=$(repo_fingerprint "$suite")
echo "repo fingerprint [$suite]: $want"
if ! "$PODMAN_BIN" container exists "$name" 2>/dev/null; then
create_from_scratch "$name" "$suite" "$srcfile"
return
fi
running=$("$PODMAN_BIN" inspect -f '{{.State.Running}}' "$name" 2>/dev/null || echo false)
have=$(container_fingerprint "$name")
echo "installed in $name: $have"
if [ "$running" != "true" ] \
|| ! "$PODMAN_BIN" exec "$name" bash -lc 'command -v taler-merchant-httpd' >/dev/null 2>&1 \
|| [ "$want" != "$have" ]; then
echo "rebuild $name (running=$running fp_match=$([ "$want" = "$have" ] && echo yes || echo NO))"
create_from_scratch "$name" "$suite" "$srcfile"
else
echo "OK $name up-to-date (fresh container matches repo)"
fi
}
# ---------------------------------------------------------------------------
# Upgrade mode: create once, then always upgrade_inside
# ---------------------------------------------------------------------------
ensure_upgrade() {
local name=$1 suite=$2 srcfile=$3
local running
if ! "$PODMAN_BIN" container exists "$name" 2>/dev/null; then
create_from_scratch "$name" "$suite" "$srcfile"
echo "(initial install for upgrade-track container $name)"
return
fi
running=$("$PODMAN_BIN" inspect -f '{{.State.Running}}' "$name" 2>/dev/null || echo false)
if [ "$running" != "true" ]; then
echo "start $name"
"$PODMAN_BIN" start "$name" || {
echo "recreate broken $name"
create_from_scratch "$name" "$suite" "$srcfile"
return
}
sleep 3
fi
if ! "$PODMAN_BIN" exec "$name" bash -lc 'command -v taler-merchant-httpd' >/dev/null 2>&1; then
echo "recreate $name (no httpd after start)"
create_from_scratch "$name" "$suite" "$srcfile"
return
fi
upgrade_inside "$name"
}
MODE="${1:-all}" # all | fresh | upgrade
case "$MODE" in
all|fresh)
ensure_fresh koopa-taler-deploy-test-apt-src-trixie trixie taler.sources
ensure_fresh koopa-taler-deploy-test-apt-src-trixie-testing trixie-testing taler-testing.sources
;;&
all|upgrade)
ensure_upgrade koopa-taler-deploy-test-apt-src-trixie-upgrade trixie taler.sources
ensure_upgrade koopa-taler-deploy-test-apt-src-trixie-testing-upgrade trixie-testing taler-testing.sources
;;
*)
echo "usage: $0 [all|fresh|upgrade]" >&2
exit 2
;;
esac
echo "done ensure-apt-deploy-test-containers mode=$MODE"

26
host-agent/env.example Normal file
View file

@ -0,0 +1,26 @@
# Copy to: ~/.config/taler-monitoring/env
# This file is NOT inside the git tree and is never overwritten by update-suite.sh.
#
# Full stack examples (same reporting generation for all):
# env/hacktivism.env.example
# env/stagepaysan.env.example
#
# SUITE_GIT_URL=https://git.hacktivism.ch/hernani/taler-monitoring.git
# SUITE_GIT_REF=main
# SUITE_DIR=$HOME/src/taler-monitoring
# SUITE_UPDATE_MODE=reset
#
# TALER_DOMAIN=hacktivism.ch
# INSIDE_PODMAN=1
# INSIDE_MODE=local-podman
# CONTINUE_ON_ERROR=1
# PHASES for the *default* hacktivism timer only (urls inside versions).
# Dedicated wrappers always override PHASES + HTML paths:
# run-surface-monitoring.sh → /taler-monitoring-surface*
# run-aptdeploy-monitoring.sh → /taler-monitoring-aptdeploy*
# PHASES="urls inside versions"
# RUN_TIMEOUT=600
# HTML_OUT=$HOME/monitoring-sites-staging
# MON_HOSTS="bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch"
# SOURCE_REPO_WEB=https://git.hacktivism.ch/hernani/taler-monitoring
# TALER_MON_LANG=en # en default; fr for FrancPaysan (auto if domain *lefrancpaysan*)

26
host-agent/env/hacktivism.env.example vendored Normal file
View file

@ -0,0 +1,26 @@
# Copy to: ~/.config/taler-monitoring/env (user hernani on koopa)
# Shared reporting generation: RUN_TIMEOUT, line-buffered log, always HTML.
TALER_DOMAIN=hacktivism.ch
INSIDE_PODMAN=1
INSIDE_MODE=local-podman
LOCAL_STACK=1
CONTINUE_ON_ERROR=1
PHASES="urls inside versions"
# aptdeploy HTML: run-aptdeploy-monitoring.sh → /taler-monitoring-aptdeploy*
RUN_TIMEOUT=600
# APT_DEPLOY_ENSURE=1
# APT_DEPLOY_SKIP=0
HTML_OUT=$HOME/monitoring-sites-staging
MON_HOSTS="bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch"
SUITE_DIR=$HOME/src/taler-monitoring
SUITE_GIT_URL=https://git.hacktivism.ch/hernani/taler-monitoring.git
SUITE_GIT_REF=main
SUITE_UPDATE_MODE=reset
SOURCE_REPO_WEB=https://git.hacktivism.ch/hernani/taler-monitoring
DEPLOY_WWW_ROOT=/var/www/monitoring-sites
TALER_MON_LANG=en

35
host-agent/env/mytops-stage.env.example vendored Normal file
View file

@ -0,0 +1,35 @@
# Copy to: ~/.config/taler-monitoring/env (user taler-monitoring @ betel)
# mytops CHF stage — NOT lifeline / production
#
SUITE_GIT_URL=https://git.hacktivism.ch/hernani/taler-monitoring.git
SUITE_GIT_REF=main
SUITE_DIR=$HOME/src/taler-monitoring
TALER_DOMAIN=stage.taler-ops.ch
MON_HOSTS="stage.my.taler-ops.ch stage.taler-ops.ch"
HTML_OUT=$HOME/monitoring-sites-staging
DEPLOY_WWW_ROOT=/var/www/monitoring-sites
HTML_URL_OK=/monitoring/
HTML_URL_ERR=/monitoring_err/
HTML_OK_DIR=monitoring
HTML_ERR_DIR=monitoring_err
# urls + monpages + fake-franken CHF (rusty.taler-ops.ch)
PHASES="urls monpages devtesting"
SKIP_SSH=1
LOCAL_STACK=0
INSIDE_PODMAN=0
CONTINUE_ON_ERROR=1
RUN_TIMEOUT=600
STRICT_EXIT=1
MONPAGES_REQUIRE_PUBLIC=1
MONPAGES_INVENTORY=job
TALER_MON_LANG=en
AGENT_LABEL=mytops-stage-host-agent
STATE_NAME=taler-monitoring-mytops-stage
SUITE_UPDATE_STRICT=1
CHECK_BANK=0
MERCHANT_REQUIRED=1
CHECK_LANDING=0
# rusty devtesting (SSH Host taler-rusty-devtesting → devtesting@rusty.taler-ops.ch)
DEVTESTING_SSH=taler-rusty-devtesting
DEVTESTING_AMOUNT=CHF:1.00
# DEVTESTING_SKIP=1

30
host-agent/env/stagepaysan.env.example vendored Normal file
View file

@ -0,0 +1,30 @@
# Copy to: ~/.config/taler-monitoring/env (user stagepaysan)
# Never inside the git tree — update-suite never overwrites this file.
#
# Shared reporting generation (run-host-report.sh / run-fp-stage-monitoring.sh):
# RUN_TIMEOUT, line-buffered log, always HTML, commit link on git.hacktivism.ch
TALER_DOMAIN=stage.lefrancpaysan.ch
INSIDE_PODMAN=1
INSIDE_MODE=local-podman
INSIDE_PROFILE=stage-lfp
LOCAL_STACK=0
CONTINUE_ON_ERROR=1
PHASES="urls inside versions"
RUN_TIMEOUT=600
HTML_OUT=$HOME/monitoring-sites-staging
MON_HOSTS="stage.bank.lefrancpaysan.ch stage.exchange.lefrancpaysan.ch stage.monnaie.lefrancpaysan.ch"
# Git clone of taler-monitoring (repo root, not …/taler-monitoring (repo root))
SUITE_DIR=$HOME/src/taler-monitoring
SUITE_GIT_URL=https://git.hacktivism.ch/hernani/taler-monitoring.git
SUITE_GIT_REF=main
SUITE_UPDATE_MODE=reset
SOURCE_REPO_WEB=https://git.hacktivism.ch/hernani/taler-monitoring
# Optional public docroot if this user can write it
# DEPLOY_WWW_ROOT=/var/www/monitoring-sites
# UI language for console + HTML sticky bar (en|fr)
TALER_MON_LANG=fr

114
host-agent/install-host-agent.sh Executable file
View file

@ -0,0 +1,114 @@
#!/usr/bin/env bash
# install-host-agent.sh — install hernani user systemd units on koopa
#
# # on koopa as hernani:
# ~/src/taler-monitoring/host-agent/install-host-agent.sh
# # or from laptop:
# ./install-host-agent.sh --remote $DEPLOY_SSH
#
# Suite tree is ONLY ~/src/taler-monitoring (standalone repo). No koopa-admin-log.
#
set -euo pipefail
ROOT=$(cd "$(dirname "$0")" && pwd)
REMOTE="${1:-}"
[ "${1:-}" = "--remote" ] && REMOTE="${2:-${DEPLOY_SSH:-}}"
SUITE_URL="${SUITE_GIT_URL:-https://git.hacktivism.ch/hernani/taler-monitoring.git}"
install_local() {
local mon home_unit
mon=$(cd "$ROOT/.." && pwd)
if [ -d "$HOME/src/taler-monitoring/.git" ] || [ -d "$HOME/src/taler-monitoring" ]; then
mon="$HOME/src/taler-monitoring"
fi
home_unit="$HOME/.config/systemd/user"
mkdir -p "$home_unit" \
"$HOME/.local/state/taler-hacktivism-monitoring" \
"$HOME/monitoring-sites-staging" \
"$HOME/.config/taler-monitoring" \
"$HOME/src"
# Local env (never in git) — create from example if missing
if [ ! -f "$HOME/.config/taler-monitoring/env" ] && [ -f "$ROOT/env.example" ]; then
cp "$ROOT/env.example" "$HOME/.config/taler-monitoring/env"
echo "created ~/.config/taler-monitoring/env"
fi
# Ensure suite git clone at canonical path
if [ ! -d "$HOME/src/taler-monitoring/.git" ]; then
if [ -d "$mon/.git" ] && [ "$mon" != "$HOME/src/taler-monitoring" ]; then
mkdir -p "$HOME/src"
ln -sfn "$mon" "$HOME/src/taler-monitoring"
echo "symlink $HOME/src/taler-monitoring → $mon"
else
echo "clone $SUITE_URL → $HOME/src/taler-monitoring"
git clone "$SUITE_URL" "$HOME/src/taler-monitoring" \
|| echo "WARN: clone failed (auth/network) — using tree at $mon if present"
fi
fi
mon="$HOME/src/taler-monitoring"
# Convenience link (update-suite.sh also creates this)
ln -sfn "$mon" "$HOME/taler-monitoring" 2>/dev/null || true
# Ensure suite is executable
chmod +x "$mon"/taler-monitoring.sh "$mon"/check_*.sh 2>/dev/null || true
chmod +x "$mon/host-agent"/*.sh 2>/dev/null || true
chmod +x "$mon/site-gen"/*.sh "$mon/site-gen/console_to_html.py" 2>/dev/null || true
install -m 644 "$ROOT/taler-monitoring-hacktivism.service" "$home_unit/"
install -m 644 "$ROOT/taler-monitoring-hacktivism.path" "$home_unit/"
install -m 644 "$ROOT/taler-monitoring-hacktivism.timer" "$home_unit/"
install -m 644 "$ROOT/taler-monitoring-surface.service" "$home_unit/"
install -m 644 "$ROOT/taler-monitoring-surface.timer" "$home_unit/"
# optional aptdeploy (not a public mon page layout — v1.8.0 simplified overview)
install -m 644 "$ROOT/taler-monitoring-aptdeploy.service" "$home_unit/"
install -m 644 "$ROOT/taler-monitoring-aptdeploy.timer" "$home_unit/"
systemctl --user daemon-reload
systemctl --user enable --now taler-monitoring-hacktivism.path
systemctl --user enable --now taler-monitoring-hacktivism.timer
systemctl --user enable --now taler-monitoring-surface.timer
systemctl --user enable --now taler-monitoring-aptdeploy.timer
# Remove leftover mail/mattermost units if present (moved to meta/deprecated/)
systemctl --user disable --now taler-monitoring-mattermost.timer 2>/dev/null || true
systemctl --user disable --now taler-monitoring-mail.timer 2>/dev/null || true
rm -f "$home_unit/taler-monitoring-mattermost.service" \
"$home_unit/taler-monitoring-mattermost.timer" \
"$home_unit/taler-monitoring-mail.service" \
"$home_unit/taler-monitoring-mail.timer" 2>/dev/null || true
systemctl --user daemon-reload 2>/dev/null || true
# initial stamp so path exists
"$ROOT/touch-software-stamp.sh"
systemctl --user start taler-monitoring-hacktivism.service || true
systemctl --user start taler-monitoring-surface.service || true
systemctl --user start taler-monitoring-aptdeploy.service || true
echo "--- status ---"
systemctl --user status taler-monitoring-hacktivism.path --no-pager -l | head -15
systemctl --user status taler-monitoring-hacktivism.timer --no-pager -l | head -12
systemctl --user status taler-monitoring-surface.timer --no-pager -l | head -12
systemctl --user status taler-monitoring-aptdeploy.timer --no-pager -l | head -12
systemctl --user list-timers --all | grep taler-monitoring || true
echo "OK host-agent (v1.8.0 simplified):"
echo " · hacktivism 4h → /monitoring on bank/exchange/taler (landing stacks)"
echo " · surface 1h → /taler-monitoring-surface* (ecosystem versions/software)"
echo " · aptdeploy 4h (optional tests; not a public mon-page type)"
echo " · mail/mattermost: not installed (meta/deprecated/; content in surface)"
echo "suite: $mon"
echo "manual surface: systemctl --user start taler-monitoring-surface.service"
echo "after upgrades: $ROOT/touch-software-stamp.sh"
}
if [ -n "$REMOTE" ] && [ "$REMOTE" != "--remote" ]; then
echo "install on $REMOTE …"
# ROOT = …/taler-monitoring/host-agent → suite root is parent
rsync -az --delete \
--exclude secrets.env \
--exclude 'android-test/artifacts' \
--exclude '.git' \
"$ROOT/../" \
"${REMOTE}:src/taler-monitoring/"
ssh -o BatchMode=yes "$REMOTE" \
'bash $HOME/src/taler-monitoring/host-agent/install-host-agent.sh'
else
install_local
fi

45
host-agent/run-aptdeploy-monitoring.sh Executable file
View file

@ -0,0 +1,45 @@
#!/usr/bin/env bash
# run-aptdeploy-monitoring.sh — apt-src deploy tests → public HTML
# Public only on taler.hacktivism.ch:
# https://taler.hacktivism.ch/taler-monitoring-aptdeploy/
# https://taler.hacktivism.ch/taler-monitoring-aptdeploy_err/
#
# CLI one-shot without HTML remains: run-aptdeploy.sh [all|fresh|upgrade|check-only]
#
set -uo pipefail
AGENT_DIR=$(cd "$(dirname "$0")" && pwd)
export AGENT_LABEL="${AGENT_LABEL:-aptdeploy-host-agent}"
export STATE_NAME="${STATE_NAME:-taler-aptdeploy-monitoring}"
export TALER_DOMAIN="${TALER_DOMAIN:-hacktivism.ch}"
# containers are local on koopa
export INSIDE_PODMAN=1
export INSIDE_MODE=local-podman
export LOCAL_STACK=1
export SKIP_SSH=0
export CONTINUE_ON_ERROR="${CONTINUE_ON_ERROR:-1}"
# four containers + checks; allow long wall
export RUN_TIMEOUT="${RUN_TIMEOUT:-1800}"
# dedicated job: always aptdeploy only (force — do not inherit main-timer env)
export PHASES="aptdeploy monpages"
# Public HTML only on taler.hacktivism.ch (not bank/exchange landings)
export MON_HOSTS="taler.hacktivism.ch"
export HTML_OUT="${HTML_OUT:-$HOME/monitoring-sites-staging}"
export DEPLOY_WWW_ROOT="${DEPLOY_WWW_ROOT:-/var/www/monitoring-sites}"
export HTML_OK_DIR="taler-monitoring-aptdeploy"
export HTML_ERR_DIR="taler-monitoring-aptdeploy_err"
export HTML_URL_OK="/taler-monitoring-aptdeploy/"
export HTML_URL_ERR="/taler-monitoring-aptdeploy_err/"
export PAGE_LABEL="taler-monitoring-aptdeploy"
# monpages: only this job’s paths on taler.hacktivism.ch
export MONPAGES_INVENTORY="job"
# Ensure all four apt-src containers (fresh rebuild + upgrade track)
if [ "${APT_DEPLOY_ENSURE:-1}" = "1" ] \
&& [ -x "$AGENT_DIR/ensure-apt-deploy-test-containers.sh" ]; then
bash "$AGENT_DIR/ensure-apt-deploy-test-containers.sh" all \
|| echo "WARN: ensure-apt-deploy-test-containers failed (aptdeploy may ERROR)" >&2
fi
exec bash "$AGENT_DIR/run-host-report.sh" "$@"

150
host-agent/run-aptdeploy.sh Executable file
View file

@ -0,0 +1,150 @@
#!/usr/bin/env bash
# run-aptdeploy.sh — ensure containers + aptdeploy phase only.
#
# Keep monitoring current:
# 1) Prefer Forgejo origin/main (update-suite reset)
# 2) If Forgejo is behind (no aptdeploy yet), re-apply suite-overlay saved
# from this agent tree before reset (rsync from laptop fills that tree)
# 3) Run ensure + taler-monitoring.sh aptdeploy
#
# Host: on koopa → local; else ssh APT_DEPLOY_SSH + rsync mon tree first.
#
set -euo pipefail
AGENT_DIR=$(cd "$(dirname "$0")" && pwd)
BOOT_MON=$(cd "$AGENT_DIR/.." && pwd)
MODE="${1:-all}"
APT_DEPLOY_SSH="${APT_DEPLOY_SSH:-${DEPLOY_SSH:-}}"
OVERLAY_DIR="${XDG_CONFIG_HOME:-$HOME/.config}/taler-monitoring/suite-overlay"
on_koopa() {
local h
h=$(hostname -s 2>/dev/null || hostname 2>/dev/null || echo "")
case "$h" in koopa|koopa.*) return 0 ;; esac
[ "${KOOPA_FORCE_LOCAL:-0}" = "1" ] && return 0
return 1
}
mon_has_aptdeploy() {
local mon=$1
[ -f "$mon/taler-monitoring.sh" ] && grep -qE 'aptdeploy\|apt-deploy' "$mon/taler-monitoring.sh"
}
save_overlay_from() {
local src=$1
mon_has_aptdeploy "$src" || return 0
mkdir -p "$OVERLAY_DIR"
rsync -a --delete \
--exclude secrets.env \
--exclude 'android-test/artifacts' \
--exclude 'android-test/apks' \
--exclude 'android-test/out*' \
--exclude '__pycache__' \
"$src/" "$OVERLAY_DIR/"
echo "saved suite-overlay from $src → $OVERLAY_DIR"
}
apply_overlay_to() {
local dest=$1
[ -f "$OVERLAY_DIR/taler-monitoring.sh" ] || return 1
mon_has_aptdeploy "$OVERLAY_DIR" || return 1
mkdir -p "$dest"
rsync -a \
--exclude secrets.env \
--exclude 'android-test/artifacts' \
--exclude 'android-test/apks' \
--exclude 'android-test/out*' \
--exclude '__pycache__' \
"$OVERLAY_DIR/" "$dest/"
echo "applied suite-overlay → $dest (Forgejo lag; push origin/main when possible)"
return 0
}
pick_mon_tree() {
# Preserve working mon before reset wipes tracked files
save_overlay_from "$BOOT_MON"
# shellcheck source=update-suite.sh
source "$AGENT_DIR/update-suite.sh"
local suite_mon="$SUITE_DIR"
mkdir -p "$suite_mon"
if mon_has_aptdeploy "$suite_mon"; then
MON_DIR="$suite_mon"
echo "using Forgejo suite mon @ ${COMMIT_SHORT:-?} · $MON_DIR"
elif apply_overlay_to "$suite_mon" && mon_has_aptdeploy "$suite_mon"; then
MON_DIR="$suite_mon"
echo "using suite mon + overlay @ ${COMMIT_SHORT:-?} · $MON_DIR"
elif mon_has_aptdeploy "$BOOT_MON"; then
MON_DIR="$BOOT_MON"
echo "WARN: using boot mon $MON_DIR (overlay apply failed)" >&2
else
echo "ERROR: no mon tree with aptdeploy (Forgejo + overlay empty)" >&2
echo " rsync suite from laptop or push to git.hacktivism.ch" >&2
exit 2
fi
AGENT_DIR="$MON_DIR/host-agent"
export MON_DIR AGENT_DIR SUITE_DIR COMMIT COMMIT_SHORT COMMIT_URL
chmod +x "$AGENT_DIR"/*.sh "$MON_DIR"/taler-monitoring.sh "$MON_DIR"/check_*.sh 2>/dev/null || true
}
run_local() {
export PATH="${HOME}/.local/bin:/usr/local/bin:/usr/bin:/bin:${PATH:-}"
export PYTHONUNBUFFERED=1
export PROGRESS_OFF="${PROGRESS_OFF:-0}"
echo "======== aptdeploy local on $(hostname) mode=$MODE ========"
pick_mon_tree
if [ "$MODE" != "check-only" ]; then
bash "$AGENT_DIR/ensure-apt-deploy-test-containers.sh" "$MODE"
fi
cd "$MON_DIR"
export CONTINUE_ON_ERROR="${CONTINUE_ON_ERROR:-1}"
export INSIDE_PODMAN="${INSIDE_PODMAN:-1}"
export LOCAL_STACK="${LOCAL_STACK:-1}"
export RUN_TIMEOUT="${RUN_TIMEOUT:-600}"
echo "run: $MON_DIR/taler-monitoring.sh -d hacktivism.ch aptdeploy · suite=${COMMIT_SHORT:-?}"
./taler-monitoring.sh -d hacktivism.ch aptdeploy
}
run_remote() {
echo "======== aptdeploy remote via ssh ${APT_DEPLOY_SSH} mode=$MODE ========"
if [ "${APT_DEPLOY_RSYNC:-1}" = "1" ]; then
echo "rsync mon suite → ${APT_DEPLOY_SSH}:src/taler-monitoring/"
rsync -az \
--exclude secrets.env \
--exclude 'android-test/artifacts' \
--exclude 'android-test/apks' \
--exclude 'android-test/out*' \
--exclude '__pycache__' \
"$BOOT_MON/" \
"${APT_DEPLOY_SSH}:src/taler-monitoring/" \
|| echo "WARN: rsync failed" >&2
fi
ssh -o BatchMode=yes -o ConnectTimeout=25 "$APT_DEPLOY_SSH" bash -s -- "$MODE" <<'EOS'
set -euo pipefail
MODE=${1:-all}
export PATH="${HOME}/.local/bin:/usr/local/bin:/usr/bin:/bin:${PATH:-}"
export KOOPA_FORCE_LOCAL=1
AGENT="$HOME/src/taler-monitoring/host-agent"
if [ ! -x "$AGENT/run-aptdeploy.sh" ]; then
AGENT="$HOME/src/taler-monitoring/host-agent"
fi
if [ ! -x "$AGENT/run-aptdeploy.sh" ]; then
echo "ERROR: run-aptdeploy.sh missing on remote" >&2
exit 1
fi
echo "remote AGENT=$AGENT"
exec bash "$AGENT/run-aptdeploy.sh" "$MODE"
EOS
}
if on_koopa; then
run_local
else
run_remote
fi

27
host-agent/run-fp-stage-monitoring.sh Executable file
View file

@ -0,0 +1,27 @@
#!/usr/bin/env bash
# FrancPaysan STAGE host-agent (user stagepaysan via francpaysan-stage-user).
# Shared reporting: run-host-report.sh (timeout, line-buffered log, first-run HTML).
set -uo pipefail
# Language: en|fr (selectable). Default for this wrapper: fr
export TALER_MON_LANG="${TALER_MON_LANG:-fr}"
export TALER_MON_LANG_SET=1
AGENT_DIR=$(cd "$(dirname "$0")" && pwd)
export AGENT_LABEL="${AGENT_LABEL:-fp-stage-host-agent}"
export STATE_NAME="${STATE_NAME:-taler-monitoring-stage-lfp}"
export TALER_DOMAIN="${TALER_DOMAIN:-stage.lefrancpaysan.ch}"
export INSIDE_PODMAN="${INSIDE_PODMAN:-1}"
export INSIDE_MODE="${INSIDE_MODE:-local-podman}"
export INSIDE_PROFILE="${INSIDE_PROFILE:-stage-lfp}"
export LOCAL_STACK="${LOCAL_STACK:-0}"
export CONTINUE_ON_ERROR="${CONTINUE_ON_ERROR:-1}"
# monpages obligatory (ERROR): only FP stage mon hosts below — never GOA inventory
export MONPAGES_REQUIRE_PUBLIC="${MONPAGES_REQUIRE_PUBLIC:-1}"
export RUN_TIMEOUT="${RUN_TIMEOUT:-600}"
export PHASES="${PHASES:-urls inside versions monpages}"
export MON_HOSTS="${MON_HOSTS:-stage.bank.lefrancpaysan.ch stage.exchange.lefrancpaysan.ch stage.monnaie.lefrancpaysan.ch}"
export HTML_OUT="${HTML_OUT:-$HOME/monitoring-sites-staging}"
export DEPLOY_WWW_ROOT="${DEPLOY_WWW_ROOT:-/var/www/monitoring-sites}"
exec bash "$AGENT_DIR/run-host-report.sh" "$@"

31
host-agent/run-hacktivism-monitoring.sh Executable file
View file

@ -0,0 +1,31 @@
#!/usr/bin/env bash
# GOA / hacktivism host-agent on koopa (hernani).
# Stack defaults only — reporting pipeline is shared: run-host-report.sh
set -uo pipefail
# Language: en|fr (selectable). Default for this wrapper: en
export TALER_MON_LANG="${TALER_MON_LANG:-en}"
export TALER_MON_LANG_SET=1
AGENT_DIR=$(cd "$(dirname "$0")" && pwd)
export AGENT_LABEL="${AGENT_LABEL:-hacktivism-host-agent}"
export STATE_NAME="${STATE_NAME:-taler-hacktivism-monitoring}"
export TALER_DOMAIN="${TALER_DOMAIN:-hacktivism.ch}"
export INSIDE_PODMAN="${INSIDE_PODMAN:-1}"
export INSIDE_MODE="${INSIDE_MODE:-local-podman}"
export LOCAL_STACK="${LOCAL_STACK:-1}"
export CONTINUE_ON_ERROR="${CONTINUE_ON_ERROR:-1}"
export RUN_TIMEOUT="${RUN_TIMEOUT:-600}"
# stack checks + full monpages inventory (landings + surface + aptdeploy bare/slash).
# Specialized pages also have their own timers; full outside-in inventory lives here.
# run-aptdeploy-monitoring.sh → monpages job-only for /taler-monitoring-aptdeploy*
# run-surface-monitoring.sh → monpages job-only for /taler-monitoring-surface*
export PHASES="${PHASES:-urls inside versions monpages}"
export MON_HOSTS="${MON_HOSTS:-bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch}"
export HTML_OUT="${HTML_OUT:-$HOME/monitoring-sites-staging}"
export DEPLOY_WWW_ROOT="${DEPLOY_WWW_ROOT:-/var/www/monitoring-sites}"
export APT_DEPLOY_ENSURE=0
# full GOA mon catalog (not job-only)
export MONPAGES_INVENTORY="${MONPAGES_INVENTORY:-auto}"
exec bash "$AGENT_DIR/run-host-report.sh" "$@"

495
host-agent/run-host-report.sh Executable file
View file

@ -0,0 +1,495 @@
#!/usr/bin/env bash
# run-host-report.sh — global reporting generation for all stacks
# (GOA/hacktivism, FP stage, mytops stage, …).
#
# Shared behaviour (always):
# 1) update-suite → latest git.hacktivism.ch (local ~/.config never overwritten)
# 2) line-buffered run log (written continuously)
# 3) RUN_TIMEOUT wall clock for taler-monitoring.sh (default 600s)
# 4) always write HTML (first run / empty staging too) + Forgejo commit link
# 5) rsync to DEPLOY_WWW_ROOT (ERROR if not writable — v1.3.1)
# 6) monpages post-check: public FQDN must serve monitoring HTML
# 7) exit code: STRICT_EXIT default 1 (v1.7.1) — unit fails if publish/public pages fail
#
# Configure via env file (not in git):
# ${XDG_CONFIG_HOME:-$HOME/.config}/taler-monitoring/env
# or TALER_MONITORING_ENV=/path/to/env
#
# Thin wrappers only set stack defaults then exec this script:
# run-hacktivism-monitoring.sh
# run-fp-stage-monitoring.sh
#
set -uo pipefail
export PATH="${HOME}/.local/bin:/usr/local/bin:/usr/bin:/bin:${PATH:-}"
export PYTHONUNBUFFERED=1
AGENT_DIR=$(cd "$(dirname "$0")" && pwd)
AGENT_MON=$(cd "$AGENT_DIR/.." && pwd)
CFG_DIR="${XDG_CONFIG_HOME:-$HOME/.config}/taler-monitoring"
ENV_FILE="${TALER_MONITORING_ENV:-$CFG_DIR/env}"
mkdir -p "$CFG_DIR"
# Thin wrappers (surface, aptdeploy-only, …) export stack knobs before exec.
# Snapshot them so ~/.config/taler-monitoring/env cannot clobber dedicated jobs
# (env often has PHASES="urls inside versions aptdeploy" for the main timer).
_wrap_keys=(
PHASES HTML_OUT HTML_OK_DIR HTML_ERR_DIR HTML_URL_OK HTML_URL_ERR PAGE_LABEL
RUN_TIMEOUT STATE_NAME AGENT_LABEL MON_HOSTS TALER_DOMAIN DEPLOY_WWW_ROOT
INSIDE_PODMAN INSIDE_MODE LOCAL_STACK SKIP_SSH CONTINUE_ON_ERROR AUTH401_CONTINUE
APT_DEPLOY_ENSURE SURFACE_SCOPE TALER_DOMAIN_FROM_CLI STRICT_EXIT SOURCE_SUITE_PATH
MONPAGES_INVENTORY MONPAGES_CHECK_BARE MONPAGES_BARE_STRICT MONPAGES_REQUIRE_PUBLIC
CHECK_BANK MERCHANT_REQUIRED CHECK_LANDING
)
for _k in "${_wrap_keys[@]}"; do
if [ -n "${!_k+x}" ]; then
printf -v "_WRAP_SET_${_k}" '%s' 1
printf -v "_WRAP_VAL_${_k}" '%s' "${!_k}"
fi
done
unset _k
if [ -f "$ENV_FILE" ]; then
# shellcheck disable=SC1090
set -a
# shellcheck source=/dev/null
source "$ENV_FILE"
set +a
fi
# Tell update-suite.sh not to re-source env (would clobber wrapper PHASES etc.)
export _TALER_MON_ENV_LOADED=1
# Restore wrapper overrides (dedicated jobs win over shared env file)
for _k in "${_wrap_keys[@]}"; do
_set_var="_WRAP_SET_${_k}"
_val_var="_WRAP_VAL_${_k}"
if [ "${!_set_var:-}" = "1" ]; then
printf -v "$_k" '%s' "${!_val_var}"
export "$_k"
fi
done
unset _k _set_var _val_var
# --- global report defaults (env file / wrapper wins if already set) ---
export CONTINUE_ON_ERROR="${CONTINUE_ON_ERROR:-1}"
export AUTH401_CONTINUE="${AUTH401_CONTINUE:-${CONTINUE_ON_ERROR}}"
export RUN_TIMEOUT="${RUN_TIMEOUT:-600}"
export TALER_DOMAIN="${TALER_DOMAIN:-hacktivism.ch}"
# UI language: selectable via TALER_MON_LANG / --lang (en|fr).
# If unset, auto: FrancPaysan domains → fr, else en.
if [ "${TALER_MON_LANG_SET:-0}" != "1" ] && [ -z "${TALER_MON_LANG:-}" ]; then
case "$TALER_DOMAIN" in
*lefrancpaysan*|*francpaysan*) export TALER_MON_LANG=fr ;;
*) export TALER_MON_LANG=en ;;
esac
elif [ -n "${TALER_MON_LANG:-}" ]; then
case "$TALER_MON_LANG" in
fr|FR|fra|french) export TALER_MON_LANG=fr ;;
*) export TALER_MON_LANG=en ;;
esac
export TALER_MON_LANG_SET=1
fi
export TALER_MON_LANG
PHASES="${PHASES:-urls inside versions monpages}"
HTML_BASE="${HTML_OUT:-$HOME/monitoring-sites-staging}"
MON_HOSTS="${MON_HOSTS:-}"
SUITE_PATH="${SOURCE_SUITE_PATH:-.}"
AGENT_LABEL="${AGENT_LABEL:-host-report}"
STATE_NAME="${STATE_NAME:-taler-monitoring-${TALER_DOMAIN}}"
STATE_DIR="${XDG_STATE_HOME:-$HOME/.local/state}/${STATE_NAME}"
LOG_DIR="$STATE_DIR/logs"
DEPLOY_WWW="${DEPLOY_WWW_ROOT:-/var/www/monitoring-sites}"
# HTML path layout under each host (monitoring / surface / …)
HTML_OK_DIR="${HTML_OK_DIR:-monitoring}"
HTML_ERR_DIR="${HTML_ERR_DIR:-monitoring_err}"
HTML_URL_OK="${HTML_URL_OK:-/monitoring/}"
HTML_URL_ERR="${HTML_URL_ERR:-/monitoring_err/}"
PAGE_LABEL="${PAGE_LABEL:-monitoring}"
SUITE_VERSION="${SUITE_VERSION:-}"
SUITE_VERSION_URL="${SUITE_VERSION_URL:-}"
if [ -z "$SUITE_VERSION" ] && [ -d "${ROOT:-$AGENT_MON}/.git" ]; then
_vd="${ROOT:-$AGENT_MON}"
SUITE_VERSION=$(git -C "$_vd" describe --tags --exact-match 2>/dev/null || git -C "$_vd" describe --tags --abbrev=0 2>/dev/null || true)
fi
if [ -z "$SUITE_VERSION" ] && [ -f "${ROOT:-$AGENT_MON}/VERSION" ]; then
SUITE_VERSION="v$(tr -d '[:space:]' <"${ROOT:-$AGENT_MON}/VERSION" | sed 's/^v//')"
fi
[ -z "$SUITE_VERSION" ] && SUITE_VERSION="unknown"
case "$SUITE_VERSION" in unknown|v*) ;; *) SUITE_VERSION="v$SUITE_VERSION" ;; esac
# Prefer commit URL over /src/tag/… (tags lag push → sticky version 404 on Forgejo).
if [ -z "$SUITE_VERSION_URL" ]; then
_web="${SOURCE_REPO_WEB:-https://git.hacktivism.ch/hernani/taler-monitoring}"
if [ -n "${COMMIT:-}" ] && [ "$COMMIT" != "unknown" ]; then
SUITE_VERSION_URL="${_web}/src/commit/${COMMIT}"
elif [ "$SUITE_VERSION" != "unknown" ]; then
SUITE_VERSION_URL="${_web}/src/tag/${SUITE_VERSION}"
fi
fi
export SUITE_VERSION SUITE_VERSION_URL
# Sensible MON_HOSTS from domain if not set
if [ -z "$MON_HOSTS" ]; then
case "$TALER_DOMAIN" in
hacktivism.ch|koopa)
MON_HOSTS="bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch"
;;
stage.lefrancpaysan.ch|stage.*lefrancpaysan*)
MON_HOSTS="stage.bank.lefrancpaysan.ch stage.exchange.lefrancpaysan.ch stage.monnaie.lefrancpaysan.ch"
;;
lefrancpaysan.ch)
MON_HOSTS="bank.lefrancpaysan.ch exchange.lefrancpaysan.ch monnaie.lefrancpaysan.ch"
;;
*)
MON_HOSTS="${TALER_DOMAIN}"
;;
esac
fi
mkdir -p "$LOG_DIR" "$HTML_BASE"
STAMP=$(date +%Y%m%d-%H%M%S)
LOG="$LOG_DIR/run-${STAMP}.log"
UPD_LOG="$LOG_DIR/suite-update-${STAMP}.log"
: >"$LOG"
# --- suite auto-upgrade BEFORE the HTML log (git noise must not pollute mon pages) ---
OVERLAY_DIR="${XDG_CONFIG_HOME:-$HOME/.config}/taler-monitoring/suite-overlay"
if [ -f "$AGENT_MON/taler-monitoring.sh" ]; then
mkdir -p "$OVERLAY_DIR"
rsync -a \
--exclude secrets.env --exclude 'android-test/artifacts' --exclude 'android-test/apks' \
--exclude 'android-test/out*' --exclude '__pycache__' \
"$AGENT_MON/" "$OVERLAY_DIR/" 2>/dev/null || true
fi
# shellcheck source=update-suite.sh
if ! source "$AGENT_DIR/update-suite.sh" >"$UPD_LOG" 2>&1; then
echo "ERROR: suite auto-update failed — refusing to run with stale tree" >&2
echo " detail: $UPD_LOG" >&2
tail -n 30 "$UPD_LOG" >&2 || true
echo " fix git/network or set SUITE_UPDATE_STRICT=0 only as emergency" >&2
exit 1
fi
if [ -d "${SUITE_DIR:-}/host-agent" ]; then
AGENT_DIR="${SUITE_DIR}/host-agent"
fi
# Now open the public HTML log (clean agent header + suite summary only)
if command -v stdbuf >/dev/null 2>&1; then
exec > >(stdbuf -oL -eL tee -a "$LOG") 2>&1
else
exec > >(tee -a "$LOG") 2>&1
fi
echo "======== $(date -Iseconds 2>/dev/null || date) ${AGENT_LABEL} ========"
echo "env_file=$ENV_FILE"
echo "domain=$TALER_DOMAIN · phases=$PHASES · RUN_TIMEOUT=${RUN_TIMEOUT}s"
echo "hosts=$MON_HOSTS"
echo "flags INSIDE_PODMAN=${INSIDE_PODMAN:-} INSIDE_MODE=${INSIDE_MODE:-} LOCAL_STACK=${LOCAL_STACK:-} SKIP_SSH=${SKIP_SSH:-} INSIDE_PROFILE=${INSIDE_PROFILE:-}"
# One-line suite pin (no git M/checkout noise)
echo "suite ${SUITE_VERSION:-?} @ ${COMMIT_SHORT:-?} ${COMMIT_URL:-}"
echo " dir=${SUITE_DIR:-}"
SUITE_MON="${SUITE_DIR}"
ROOT="$SUITE_MON"
_need_overlay=0
# Detect missing features in Forgejo tree (untracked helper files alone are not enough —
# taler-monitoring.sh must accept the phase name).
case " $PHASES " in
*" aptdeploy "*|*" apt-deploy "*|*" apt_src "*)
if ! grep -qE 'aptdeploy\|apt-deploy' "$SUITE_MON/taler-monitoring.sh" 2>/dev/null; then
_need_overlay=1
fi
;;
esac
case " $PHASES " in
*" surface "*|*" ecosystem "*)
if ! grep -qE 'surface\|ecosystem' "$SUITE_MON/taler-monitoring.sh" 2>/dev/null \
|| [ ! -f "$SUITE_MON/check_surface.sh" ]; then
_need_overlay=1
fi
;;
esac
# Wrapper-preserve + HTML path knobs may live only in local overlay until pushed
if [ -f "$OVERLAY_DIR/host-agent/run-host-report.sh" ] \
&& ! grep -q '_wrap_keys\|_WRAP_SET_' "$SUITE_MON/host-agent/run-host-report.sh" 2>/dev/null \
&& grep -q '_wrap_keys\|_WRAP_SET_' "$OVERLAY_DIR/host-agent/run-host-report.sh" 2>/dev/null; then
_need_overlay=1
fi
if [ "$_need_overlay" = "1" ] && [ -f "$OVERLAY_DIR/taler-monitoring.sh" ]; then
echo "WARN: Forgejo @ ${COMMIT_SHORT:-?} lagging local suite-overlay (phases=[$PHASES])" >&2
echo " re-applying suite-overlay → $SUITE_MON" >&2
mkdir -p "$SUITE_MON"
rsync -a \
--exclude secrets.env --exclude 'android-test/artifacts' --exclude 'android-test/apks' \
--exclude 'android-test/out*' --exclude '__pycache__' \
"$OVERLAY_DIR/" "$SUITE_MON/"
ROOT="$SUITE_MON"
elif [ "$_need_overlay" = "1" ]; then
echo "WARN: Forgejo missing features and no suite-overlay — using agent mon $AGENT_MON" >&2
ROOT="$AGENT_MON"
else
echo "using suite mon @ ${COMMIT_SHORT:-?} · $ROOT"
# Do NOT rsync suite-overlay over a fresh Forgejo tree — that reintroduced
# stale site-gen/host-agent (bootstrap sticky, old monpages) on stage/betel.
fi
unset _need_overlay
if [ -f "$ROOT/site-gen/console_to_html.py" ]; then
SITE_GEN="$ROOT/site-gen"
elif [ -f "$AGENT_MON/site-gen/console_to_html.py" ]; then
SITE_GEN="$AGENT_MON/site-gen"
echo "note: using agent site-gen at $SITE_GEN"
elif [ -f "$SUITE_MON/site-gen/console_to_html.py" ]; then
SITE_GEN="$SUITE_MON/site-gen"
else
SITE_GEN=""
fi
if [ ! -x "$ROOT/taler-monitoring.sh" ]; then
if [ -x "$AGENT_MON/taler-monitoring.sh" ]; then
ROOT="$AGENT_MON"
echo "note: using agent mon tree $ROOT"
else
echo "ERROR: suite missing at $ROOT" >&2
COMMIT="${COMMIT:-unknown}"
COMMIT_SHORT="${COMMIT_SHORT:-unknown}"
COMMIT_URL="${COMMIT_URL:-}"
fi
fi
echo "commit=${COMMIT_SHORT:-?} · ${COMMIT_URL:-}"
ec=1
if [ -x "${ROOT:-}/taler-monitoring.sh" ]; then
cd "$ROOT" || exit 1
chmod +x taler-monitoring.sh check_*.sh host-agent/*.sh site-gen/*.sh site-gen/*.py 2>/dev/null || true
# Overlay newer inside/versions host-podman bits if suite clone is stale
for f in check_inside.sh check_versions.sh taler-monitoring.sh lib.sh; do
if [ -f "$AGENT_MON/$f" ] && [ -f "$ROOT/$f" ]; then
if ! grep -q 'host-podman\|INSIDE_ACCESS\|RUN_TIMEOUT' "$ROOT/$f" 2>/dev/null \
&& grep -q 'host-podman\|INSIDE_ACCESS\|RUN_TIMEOUT' "$AGENT_MON/$f" 2>/dev/null; then
echo "note: overlay $f from agent install"
cp -f "$AGENT_MON/$f" "$ROOT/$f"
chmod +x "$ROOT/$f" 2>/dev/null || true
fi
fi
done
# monpages inside PHASES runs *before* htmlify/publish — content failures would
# race with concurrent publishes or reflect last run's incomplete redirect stub.
# Soft-content there; post-check after rsync stays hard (authoritative).
case " $PHASES " in
*" monpages "*|*" pages "*) export MONPAGES_PRE_PUBLISH=1 ;;
*) unset MONPAGES_PRE_PUBLISH 2>/dev/null || true ;;
esac
# shellcheck disable=SC2086
./taler-monitoring.sh -d "$TALER_DOMAIN" $PHASES
ec=$?
unset MONPAGES_PRE_PUBLISH 2>/dev/null || true
fi
# --- always generate HTML (first run / missing pages / every run) ---
write_bootstrap_html() {
local out="$1" host="$2" mode="$3"
mkdir -p "$(dirname "$out")"
local cshort="${COMMIT_SHORT:-unknown}"
local curl="${COMMIT_URL:-}"
local clink="$cshort"
local gen iso
# Local wall clock (CEST/CET · MESZ/MEZ on CH/DE hosts), not UTC Z
if TZ=Europe/Zurich date +"%Y-%m-%d %H:%M:%S %Z" >/dev/null 2>&1; then
gen=$(TZ=Europe/Zurich date +"%Y-%m-%d %H:%M:%S %Z")
iso=$(TZ=Europe/Zurich date +"%Y-%m-%dT%H:%M:%S%z" | sed 's/\([+-][0-9][0-9]\)\([0-9][0-9]\)$/\1:\2/')
else
gen=$(date +"%Y-%m-%d %H:%M:%S %Z" 2>/dev/null || date)
iso=$(date -Iseconds 2>/dev/null || date)
fi
if [ -n "$curl" ]; then
clink="<a href=\"$(printf '%s' "$curl" | sed 's/&/\&amp;/g')\">$cshort</a>"
fi
local tmp
tmp=$(mktemp "${out}.XXXXXX")
cat >"$tmp" <<EOF
<!DOCTYPE html>
<html lang="en"><head><meta charset="utf-8"/>
<meta name="generated" content="${iso}"/>
<meta name="taler-mon-page" content="incomplete"/>
<title>monitoring ${mode} · ${host}</title>
<style>
body{margin:0;background:#0c0c0c;color:#c8c8c8;font-family:ui-monospace,monospace}
a{color:#61afef}.err{color:#ff5c5c}
.sticky-bar{position:sticky;top:0;z-index:100;padding:10px 14px;background:#2a1010f2;border-bottom:2px solid #a33}
.sticky-bar .pill{font-weight:800;color:#ff5c5c;margin-right:8px}
.version-link{color:#61afef}
main{padding:1.5rem}
</style></head><body>
<!-- taler-mon:top -->
<div class="sticky-bar" id="status-bar">
<span class="pill">BOOTSTRAP</span>
${PAGE_LABEL:-monitoring} · ${host} ·
<span class="version-link muted">bootstrap</span> ·
<span data-generated-iso="${iso}">generated ${gen}</span>
· source ${clink}
</div>
<main>
<p class="err">Bootstrap page (converter missing or first install). See host-agent log.</p>
<pre style="white-space:pre-wrap;font-size:12px">$(tail -n 80 "$LOG" 2>/dev/null | sed 's/&/\&amp;/g;s/</\&lt;/g' || true)</pre>
</main>
<!-- taler-mon:bottom:incomplete -->
</body></html>
EOF
mv -f "$tmp" "$out"
echo "bootstrap html → $out"
}
htmlify_host() {
local host="$1"
local mon="$HTML_BASE/$host/${HTML_OK_DIR}/index.html"
local mon_err="$HTML_BASE/$host/${HTML_ERR_DIR}/index.html"
mkdir -p "$HTML_BASE/$host/${HTML_OK_DIR}" "$HTML_BASE/$host/${HTML_ERR_DIR}"
if [ -n "$SITE_GEN" ] && [ -f "$SITE_GEN/console_to_html.py" ]; then
python3 "$SITE_GEN/console_to_html.py" \
--lang "${TALER_MON_LANG:-en}" \
--log "$LOG" \
--out "$mon_err" \
--hostname "$host" \
--mode err \
--commit "${COMMIT:-}" \
--commit-url "${COMMIT_URL:-}" \
--suite-version "${SUITE_VERSION:-unknown}" \
--suite-version-url "${SUITE_VERSION_URL:-}" \
--suite-path "$SUITE_PATH" \
--page-label "$PAGE_LABEL" \
--path-err "$HTML_URL_ERR" \
--link-other "$HTML_URL_OK"
if [ "$ec" -eq 0 ]; then
python3 "$SITE_GEN/console_to_html.py" \
--lang "${TALER_MON_LANG:-en}" \
--log "$LOG" \
--out "$mon" \
--hostname "$host" \
--mode ok \
--commit "${COMMIT:-}" \
--commit-url "${COMMIT_URL:-}" \
--suite-version "${SUITE_VERSION:-unknown}" \
--suite-version-url "${SUITE_VERSION_URL:-}" \
--suite-path "$SUITE_PATH" \
--page-label "$PAGE_LABEL" \
--path-err "$HTML_URL_ERR" \
--link-other ""
rm -rf "$HTML_BASE/$host/${HTML_ERR_DIR}"
echo "html $host → ${HTML_URL_OK} only (clean · ${COMMIT_SHORT:-?})"
else
python3 "$SITE_GEN/console_to_html.py" \
--lang "${TALER_MON_LANG:-en}" \
--log "$LOG" \
--out "$mon" \
--hostname "$host" \
--mode redirect \
--commit "${COMMIT:-}" \
--commit-url "${COMMIT_URL:-}" \
--suite-version "${SUITE_VERSION:-unknown}" \
--suite-version-url "${SUITE_VERSION_URL:-}" \
--suite-path "$SUITE_PATH" \
--page-label "$PAGE_LABEL" \
--path-err "$HTML_URL_ERR"
echo "html $host → ${HTML_URL_OK} stub + ${HTML_URL_ERR} (ec=$ec · ${COMMIT_SHORT:-?})"
fi
else
echo "WARN: console_to_html.py missing — bootstrap pages"
write_bootstrap_html "$mon_err" "$host" "err"
if [ "$ec" -eq 0 ]; then
write_bootstrap_html "$mon" "$host" "ok"
rm -rf "$HTML_BASE/$host/${HTML_ERR_DIR}"
else
write_bootstrap_html "$mon" "$host" "redirect"
fi
fi
if [ ! -f "$mon" ]; then
write_bootstrap_html "$mon" "$host" "err"
fi
printf '%s\n' "${COMMIT:-unknown}" >"$HTML_BASE/$host/COMMIT"
printf '%s\n' "${COMMIT_URL:-}" >"$HTML_BASE/$host/COMMIT_URL"
}
for host in $MON_HOSTS; do
htmlify_host "$host"
echo "public URL (expected): https://${host}${HTML_URL_OK}"
done
# Publish to live web root when possible (v1.3.1+: never silent-skip)
if [ -n "$DEPLOY_WWW" ]; then
if [ -w "$DEPLOY_WWW" ] 2>/dev/null; then
# --delay-updates: write to temp names then rename (less partial-file exposure)
if rsync -a --delete --delay-updates "$HTML_BASE/" "$DEPLOY_WWW/"; then
echo "synced → $DEPLOY_WWW/ (delay-updates)"
# brief settle so concurrent monpages curls do not hit mid-rename FS views
sleep "${MONPAGES_PUBLISH_SETTLE_S:-0.3}" 2>/dev/null || sleep 1
else
echo "ERROR: rsync to DEPLOY_WWW_ROOT=$DEPLOY_WWW failed" >&2
ec=1
fi
elif [ -e "$DEPLOY_WWW" ]; then
echo "ERROR: DEPLOY_WWW_ROOT=$DEPLOY_WWW exists but is not writable by $(id -un)" >&2
echo " public FQDN pages will stay missing/stale — fix ownership or run:" >&2
echo " sudo ${APPLY_MONITORING_LIVE}" >&2
ec=1
else
echo "ERROR: DEPLOY_WWW_ROOT=$DEPLOY_WWW missing — public pages not published" >&2
echo " create tree + Caddy handles: sudo ${APPLY_MONITORING_LIVE}" >&2
ec=1
fi
else
echo "WARN: DEPLOY_WWW_ROOT empty — skipping live publish (staging only: $HTML_BASE)" >&2
fi
# Post-publish: monpages obligatory (v1.7.6+) — GOA full inventory / FP only FP hosts
_monpages_failed=0
if [ "${MONPAGES_POSTCHECK:-1}" = "1" ] && [ -x "$ROOT/check_monitoring_pages.sh" ]; then
echo "--- monpages post-check (public FQDN, obligatory) ---"
export MON_HOSTS HTML_URL_OK HTML_URL_ERR TALER_DOMAIN HTML_OUT HTML_BASE
export MONPAGES_STAGING_BASE="${HTML_BASE}"
export DEPLOY_WWW_ROOT="${DEPLOY_WWW:-${DEPLOY_WWW_ROOT:-}}"
export MONPAGES_REQUIRE_PUBLIC="${MONPAGES_REQUIRE_PUBLIC:-1}"
# auto: GOA = all suite mon sites; FP = only FP mon hosts; job paths always included
export MONPAGES_INVENTORY="${MONPAGES_INVENTORY:-auto}"
# Hard check (not pre-publish soft)
unset MONPAGES_PRE_PUBLISH 2>/dev/null || true
export MONPAGES_PRE_PUBLISH=0
if ! bash "$ROOT/check_monitoring_pages.sh"; then
echo "ERROR: public monitoring page(s) missing or not suite HTML via FQDN" >&2
echo " staging HTML_OUT=${HTML_BASE} · DEPLOY_WWW=${DEPLOY_WWW:-}" >&2
ec=1
_monpages_failed=1
fi
unset MONPAGES_PRE_PUBLISH 2>/dev/null || true
fi
# If public check failed after first HTML write, rewrite pages so sticky bar shows monpages ERRORs
if [ "$_monpages_failed" = "1" ]; then
echo "--- re-htmlify after monpages failure (include public outage in report) ---"
# append marker to log so converter sees ERROR lines if monpages was post-only
{
echo ""
echo "ERROR monpages: public FQDN monitoring HTML missing or merchant JSON code 21"
echo "ERROR monpages: fix with sudo ${APPLY_MONITORING_LIVE} (www + Caddy handles)"
} >>"$LOG"
for host in $MON_HOSTS; do
htmlify_host "$host"
done
fi
ls -1t "$LOG_DIR"/run-*.log 2>/dev/null | tail -n +30 | xargs rm -f 2>/dev/null || true
echo "======== done ec=$ec · log=$LOG · ${COMMIT_URL:-} ========"
# v1.7.1: default STRICT_EXIT=1 so systemd/timer reflects publish/public failures
if [ "${STRICT_EXIT:-1}" = "1" ]; then
exit "$ec"
fi
exit 0

51
host-agent/run-surface-monitoring.sh Executable file
View file

@ -0,0 +1,51 @@
#!/usr/bin/env bash
# run-surface-monitoring.sh — ecosystem surface report (v1.8.0+ simplified)
#
# Public HTML only on taler.hacktivism.ch:
# https://taler.hacktivism.ch/taler-monitoring-surface/
# https://taler.hacktivism.ch/taler-monitoring-surface_err/
#
# This page covers remote ecosystem inventory: host/port probes, nmap OS
# fingerprint, Mattermost, mail (firefly + anastasis), package/version signals.
# There are NO separate public pages for mail / mattermost (folded here).
# Apt-src deploy tests keep their own page + timer:
# run-aptdeploy-monitoring.sh → /taler-monitoring-aptdeploy(+_err)/
#
# Landing-stack reports stay on each of the 9 fronts as /monitoring(/_err).
#
set -uo pipefail
AGENT_DIR=$(cd "$(dirname "$0")" && pwd)
export AGENT_LABEL="${AGENT_LABEL:-surface-host-agent}"
export STATE_NAME="${STATE_NAME:-taler-surface-monitoring}"
export TALER_DOMAIN="${TALER_DOMAIN:-hacktivism.ch}"
# surface is primarily remote; versions may use host podman when available
export INSIDE_PODMAN="${INSIDE_PODMAN:-1}"
export LOCAL_STACK="${LOCAL_STACK:-0}"
export SKIP_SSH="${SKIP_SSH:-1}"
export INSIDE_MODE="${INSIDE_MODE:-local-podman}"
export CONTINUE_ON_ERROR="${CONTINUE_ON_ERROR:-1}"
export RUN_TIMEOUT="${RUN_TIMEOUT:-2400}"
# All ecosystem checks → one surface HTML report (force — do not inherit main-timer env)
export PHASES="surface mattermost mail versions monpages"
export MON_HOSTS="taler.hacktivism.ch"
export HTML_OUT="${HTML_OUT:-$HOME/monitoring-sites-staging}"
export DEPLOY_WWW_ROOT="${DEPLOY_WWW_ROOT:-/var/www/monitoring-sites}"
export HTML_OK_DIR="taler-monitoring-surface"
export HTML_ERR_DIR="taler-monitoring-surface_err"
export HTML_URL_OK="/taler-monitoring-surface/"
export HTML_URL_ERR="/taler-monitoring-surface_err/"
export PAGE_LABEL="taler-monitoring-surface"
export SURFACE_SCOPE="${SURFACE_SCOPE:-ecosystem}"
export TALER_DOMAIN_FROM_CLI=0
export APT_DEPLOY_ENSURE=0
# monpages job-only for surface paths on taler.hacktivism.ch
export MONPAGES_INVENTORY="job"
# package behind → ERROR (v1.8.0)
export TALER_PKG_BEHIND="${TALER_PKG_BEHIND:-error}"
exec bash "$AGENT_DIR/run-host-report.sh" "$@"

18
host-agent/taler-monitoring-aptdeploy.service Normal file
View file

@ -0,0 +1,18 @@
[Unit]
Description=Taler monitoring aptdeploy (apt-src container deploy tests)
Documentation=file:%h/src/taler-monitoring/host-agent/README.md
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin
Environment=CONTINUE_ON_ERROR=1
Environment=RUN_TIMEOUT=1800
WorkingDirectory=%h/src/taler-monitoring
ExecStart=%h/src/taler-monitoring/host-agent/run-aptdeploy-monitoring.sh
Nice=10
TimeoutStartSec=45min
[Install]
WantedBy=default.target

14
host-agent/taler-monitoring-aptdeploy.timer Normal file
View file

@ -0,0 +1,14 @@
[Unit]
Description=Periodic taler-monitoring aptdeploy (apt-src deploy tests)
Documentation=file:%h/src/taler-monitoring/host-agent/README.md
[Timer]
OnBootSec=35min
OnUnitActiveSec=4h
AccuracySec=5min
Persistent=true
RandomizedDelaySec=10min
Unit=taler-monitoring-aptdeploy.service
[Install]
WantedBy=timers.target

15
host-agent/taler-monitoring-hacktivism.path Normal file
View file

@ -0,0 +1,15 @@
[Unit]
Description=Watch hacktivism software/config changes → re-run monitoring
Documentation=file:%h/src/taler-monitoring/host-agent/README.md
[Path]
# Explicit stamp after package/image upgrades (touch-software-stamp.sh)
# Landing/caddy ops live outside this suite (host admin-log) — touch the stamp there.
PathChanged=%h/.local/state/taler-hacktivism-monitoring/software.stamp
PathModified=%h/.local/state/taler-hacktivism-monitoring/software.stamp
# Debounce: wait for quiet period before firing
TriggerLimitIntervalSec=120
TriggerLimitBurst=3
[Install]
WantedBy=default.target

21
host-agent/taler-monitoring-hacktivism.service Normal file
View file

@ -0,0 +1,21 @@
[Unit]
Description=Taler monitoring GOA/hacktivism (host-podman inside + public urls)
Documentation=file:%h/src/taler-monitoring/host-agent/README.md
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
# Primary: host can podman-exec into taler-hacktivism* containers
Environment=INSIDE_PODMAN=1
Environment=CONTINUE_ON_ERROR=1
Environment=RUN_TIMEOUT=600
Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin
WorkingDirectory=%h/src/taler-monitoring
ExecStart=%h/src/taler-monitoring/host-agent/run-hacktivism-monitoring.sh
Nice=10
# Suite budget is RUN_TIMEOUT (10m) + git/HTML overhead
TimeoutStartSec=15min
[Install]
WantedBy=default.target

14
host-agent/taler-monitoring-hacktivism.timer Normal file
View file

@ -0,0 +1,14 @@
[Unit]
Description=Periodic hacktivism monitoring (fallback if no path events)
Documentation=file:%h/src/taler-monitoring/host-agent/README.md
[Timer]
# Every 4h + boot
OnBootSec=10min
OnUnitActiveSec=4h
AccuracySec=5min
Persistent=true
Unit=taler-monitoring-hacktivism.service
[Install]
WantedBy=timers.target

20
host-agent/taler-monitoring-mytops-stage.service Normal file
View file

@ -0,0 +1,20 @@
[Unit]
Description=taler-monitoring mytops stage (betel) — auto-update suite + urls/monpages
Documentation=file:%h/src/taler-monitoring/host-agent/README.md
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
# Env: ~/.config/taler-monitoring/env (SUITE_GIT_*, MON_HOSTS, PHASES=urls monpages, …)
EnvironmentFile=-%h/.config/taler-monitoring/env
Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin
Environment=SUITE_UPDATE_STRICT=1
WorkingDirectory=%h/src/taler-monitoring
# run-host-report sources update-suite.sh first (strict fetch/reset to main)
ExecStart=%h/src/taler-monitoring/host-agent/run-host-report.sh
Nice=10
TimeoutStartSec=20min
[Install]
WantedBy=default.target

11
host-agent/taler-monitoring-mytops-stage.timer Normal file
View file

@ -0,0 +1,11 @@
[Unit]
Description=taler-monitoring mytops stage (betel) every 4h — suite auto-upgrade + mon HTML
[Timer]
OnBootSec=5min
OnUnitActiveSec=4h
Persistent=true
Unit=taler-monitoring-mytops-stage.service
[Install]
WantedBy=timers.target

18
host-agent/taler-monitoring-surface.service Normal file
View file

@ -0,0 +1,18 @@
[Unit]
Description=Taler monitoring surface (remote ecosystem inventory)
Documentation=file:%h/src/taler-monitoring/host-agent/README.md
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin
Environment=CONTINUE_ON_ERROR=1
Environment=RUN_TIMEOUT=2400
WorkingDirectory=%h/src/taler-monitoring
ExecStart=%h/src/taler-monitoring/host-agent/run-surface-monitoring.sh
Nice=10
TimeoutStartSec=50min
[Install]
WantedBy=default.target

14
host-agent/taler-monitoring-surface.timer Normal file
View file

@ -0,0 +1,14 @@
[Unit]
Description=Hourly taler-monitoring surface scan (ecosystem)
Documentation=file:%h/src/taler-monitoring/host-agent/README.md
[Timer]
OnBootSec=20min
OnUnitActiveSec=1h
AccuracySec=2min
Persistent=true
RandomizedDelaySec=5min
Unit=taler-monitoring-surface.service
[Install]
WantedBy=timers.target

17
host-agent/touch-software-stamp.sh Executable file
View file

@ -0,0 +1,17 @@
#!/usr/bin/env bash
# Call after upgrading hacktivism packages / redeploying landings / image rebuilds.
# systemd path unit watches this stamp → re-runs monitoring.
set -euo pipefail
STATE="${XDG_STATE_HOME:-$HOME/.local/state}/taler-hacktivism-monitoring"
mkdir -p "$STATE"
STAMP="$STATE/software.stamp"
date -Iseconds >"$STAMP"
# also record container image ids for debugging
if command -v podman >/dev/null 2>&1; then
{
echo "# images $(date -Iseconds)"
podman images --format '{{.Repository}}:{{.Tag}} {{.ID}} {{.Digest}}' 2>/dev/null | grep -i hacktivism || true
podman ps --format '{{.Names}} {{.ImageID}} {{.Status}}' 2>/dev/null | grep -i hacktivism || true
} >"$STATE/software-images.txt" || true
fi
echo "touched $STAMP"

132
host-agent/update-suite.sh Executable file
View file

@ -0,0 +1,132 @@
#!/usr/bin/env bash
# update-suite.sh — pull latest taler-monitoring from Forgejo (standalone repo root).
#
# Sourced by run-host-report.sh / wrappers. Every host-agent run must land on
# current origin/$SUITE_GIT_REF (default main). Stale clones are not acceptable.
#
# Env:
# SUITE_GIT_URL default https://git.hacktivism.ch/hernani/taler-monitoring.git
# SUITE_GIT_REF default main
# SUITE_DIR default ~/src/taler-monitoring
# SUITE_UPDATE_MODE reset (default) | pull
# SUITE_UPDATE_STRICT 1 (default) = fetch/reset failure aborts the agent run
# 0 = warn and continue on old tree (emergency only)
#
set -uo pipefail
CFG_DIR="${XDG_CONFIG_HOME:-$HOME/.config}/taler-monitoring"
ENV_FILE="${TALER_MONITORING_ENV:-$CFG_DIR/env}"
if [ -f "$ENV_FILE" ] && [ "${_TALER_MON_ENV_LOADED:-0}" != "1" ]; then
set -a; source "$ENV_FILE"; set +a
fi
SUITE_GIT_URL="${SUITE_GIT_URL:-https://git.hacktivism.ch/hernani/taler-monitoring.git}"
SUITE_GIT_REF="${SUITE_GIT_REF:-main}"
SUITE_UPDATE_MODE="${SUITE_UPDATE_MODE:-reset}"
SUITE_UPDATE_STRICT="${SUITE_UPDATE_STRICT:-1}"
if [ -z "${SUITE_DIR:-}" ]; then
if [ -d "$HOME/src/taler-monitoring/.git" ]; then SUITE_DIR="$HOME/src/taler-monitoring"
elif [ -d "$HOME/taler-monitoring/.git" ]; then SUITE_DIR="$HOME/taler-monitoring"
else SUITE_DIR="$HOME/src/taler-monitoring"; fi
fi
export SUITE_DIR
_upd_die() {
echo "ERROR suite-update: $*" >&2
if [ "${SUITE_UPDATE_STRICT}" = "1" ]; then
return 1 2>/dev/null || exit 1
fi
echo "WARN suite-update: continuing on existing tree (SUITE_UPDATE_STRICT=0)" >&2
return 0
}
mkdir -p "$(dirname "$SUITE_DIR")" "$CFG_DIR"
if [ ! -d "$SUITE_DIR/.git" ]; then
echo "clone $SUITE_GIT_URL → $SUITE_DIR"
if ! git clone --branch "$SUITE_GIT_REF" "$SUITE_GIT_URL" "$SUITE_DIR" \
&& ! git clone "$SUITE_GIT_URL" "$SUITE_DIR"; then
_upd_die "git clone failed"
return 1 2>/dev/null || exit 1
fi
fi
cd "$SUITE_DIR" || { _upd_die "cd $SUITE_DIR failed"; return 1 2>/dev/null || exit 1; }
git remote set-url origin "$SUITE_GIT_URL" 2>/dev/null || true
_before=$(git rev-parse --short=12 HEAD 2>/dev/null || echo none)
_before_ver=$(git describe --tags --exact-match 2>/dev/null \
|| git describe --tags --abbrev=0 2>/dev/null \
|| echo unknown)
echo "fetch origin ($SUITE_GIT_REF) + tags …"
if ! git fetch --tags --force --prune origin 2>&1; then
_upd_die "git fetch failed — suite would stay at ${_before} (${_before_ver})"
# fall through only if STRICT=0
else
case "$SUITE_UPDATE_MODE" in
reset)
if ! git checkout -B "$SUITE_GIT_REF" "origin/$SUITE_GIT_REF" 2>/dev/null \
&& ! git checkout -B "$SUITE_GIT_REF" FETCH_HEAD; then
_upd_die "git checkout $SUITE_GIT_REF failed"
fi
if ! git reset --hard "origin/$SUITE_GIT_REF" 2>/dev/null \
&& ! git reset --hard FETCH_HEAD; then
_upd_die "git reset --hard origin/$SUITE_GIT_REF failed"
fi
;;
*)
if ! git pull --ff-only origin "$SUITE_GIT_REF" 2>&1; then
_upd_die "git pull --ff-only failed"
fi
;;
esac
fi
COMMIT=$(git rev-parse HEAD)
COMMIT_SHORT=$(git rev-parse --short=12 HEAD)
SOURCE_REPO_WEB="${SOURCE_REPO_WEB:-https://git.hacktivism.ch/hernani/taler-monitoring}"
COMMIT_URL="${SOURCE_REPO_WEB}/src/commit/${COMMIT}"
export COMMIT COMMIT_SHORT COMMIT_URL SOURCE_REPO_WEB
# Release tag for sticky bar (exact tag if HEAD is tagged, else nearest)
SUITE_VERSION=""
if command -v git >/dev/null 2>&1; then
SUITE_VERSION=$(git describe --tags --exact-match 2>/dev/null || true)
if [ -z "$SUITE_VERSION" ]; then
SUITE_VERSION=$(git describe --tags --abbrev=0 2>/dev/null || true)
fi
fi
if [ -z "$SUITE_VERSION" ] && [ -f "$SUITE_DIR/VERSION" ]; then
SUITE_VERSION="v$(tr -d '[:space:]' <"$SUITE_DIR/VERSION" | sed 's/^v//')"
fi
[ -z "$SUITE_VERSION" ] && SUITE_VERSION="unknown"
case "$SUITE_VERSION" in
unknown|v*) ;;
*) SUITE_VERSION="v${SUITE_VERSION}" ;;
esac
# Sticky version badge: prefer commit URL (exact tree of this run).
# Tag URLs (/src/tag/vX.Y.Z) 404 when the tag is not yet pushed to Forgejo;
# commit URLs work as soon as main (or the tree) is on the remote.
SUITE_VERSION_URL=""
if [ -n "$SOURCE_REPO_WEB" ] && [ -n "${COMMIT:-}" ] && [ "$COMMIT" != "unknown" ]; then
SUITE_VERSION_URL="${SOURCE_REPO_WEB}/src/commit/${COMMIT}"
elif [ -n "$SOURCE_REPO_WEB" ] && [ "$SUITE_VERSION" != "unknown" ]; then
SUITE_VERSION_URL="${SOURCE_REPO_WEB}/src/tag/${SUITE_VERSION}"
fi
export SUITE_VERSION SUITE_VERSION_URL
if [ "$_before" != "$COMMIT_SHORT" ]; then
echo " upgraded ${_before} (${_before_ver}) → ${COMMIT_SHORT} (${SUITE_VERSION})"
else
echo " already at ${COMMIT_SHORT} (${SUITE_VERSION})"
fi
echo " version=$SUITE_VERSION ${SUITE_VERSION_URL:+· $SUITE_VERSION_URL}"
echo "suite @ $COMMIT_SHORT $COMMIT_URL"
echo " dir=$SUITE_DIR"
# Sanity: must have suite entrypoint after update
if [ ! -x "$SUITE_DIR/taler-monitoring.sh" ] && [ ! -f "$SUITE_DIR/taler-monitoring.sh" ]; then
_upd_die "taler-monitoring.sh missing after update"
return 1 2>/dev/null || exit 1
fi
ln -sfn "$SUITE_DIR" "$HOME/taler-monitoring" 2>/dev/null || true
chmod +x taler-monitoring.sh check_*.sh host-agent/*.sh site-gen/*.sh site-gen/*.py 2>/dev/null || true
return 0 2>/dev/null || true

104
i18n.sh Normal file
View file

@ -0,0 +1,104 @@
#!/usr/bin/env bash
# i18n.sh — language for taler-monitoring console + HTML chrome
#
# TALER_MON_LANG=en|fr (default en)
# Auto: domains *lefrancpaysan* → fr unless TALER_MON_LANG is set explicitly.
: "${TALER_MON_LANG:=}"
: "${TALER_MON_LANG_SET:=0}"
i18n_init() {
if [ -n "${TALER_MON_LANG:-}" ]; then
TALER_MON_LANG_SET=1
fi
if [ "${TALER_MON_LANG_SET}" != "1" ]; then
case "${TALER_DOMAIN:-}" in
*lefrancpaysan*|*francpaysan*)
TALER_MON_LANG=fr
;;
*)
TALER_MON_LANG=en
;;
esac
fi
case "${TALER_MON_LANG}" in
fr|FR|fra|french) TALER_MON_LANG=fr ;;
*) TALER_MON_LANG=en ;;
esac
export TALER_MON_LANG
}
i18n_tag() {
local t="$1"
if [ "${TALER_MON_LANG:-en}" != "fr" ]; then
printf '%s' "$t"
return 0
fi
case "$t" in
OK) printf 'OK' ;;
ERROR) printf 'ERREUR' ;;
WARN) printf 'AVERT' ;;
INFO) printf 'INFO' ;;
BLOCKER) printf 'BLOCAGE' ;;
BLOCK) printf 'BLOC' ;;
SUMMARY) printf 'RESUME' ;;
PROG|PROGRESS) printf 'PROG' ;;
*) printf '%s' "$t" ;;
esac
}
i18n_text() {
local s="$*"
if [ "${TALER_MON_LANG:-en}" != "fr" ] || [ -z "$s" ]; then
printf '%s' "$s"
return 0
fi
case "$s" in
"numbered checks this run:") printf 'controles numerotes de cette execution :' ;;
"numbered checks:") printf 'controles numerotes :' ;;
"(global done/total)") printf '(fait/total global)' ;;
"global checks:") printf 'controles globaux :' ;;
"failed — see list above"|"failed - see list above") printf 'echec — voir la liste ci-dessus' ;;
"BLOCKERS · pay/withdraw cannot finish") printf 'BLOCAGES · paiement/retrait impossible' ;;
"BLOCKERS (pay/withdraw cannot finish)") printf 'BLOCAGES (paiement/retrait impossible)' ;;
"ERRORS · failed checks") printf 'ERREURS · controles en echec' ;;
"ERRORS (failed checks)") printf 'ERREURS (controles en echec)' ;;
"pay/withdraw cannot finish") printf 'paiement/retrait impossible a terminer' ;;
"SUMMARY") printf 'RESUME' ;;
"totals:") printf 'totaux :' ;;
"container") printf 'conteneur' ;;
"disk") printf 'disque' ;;
"package") printf 'paquet' ;;
"version") printf 'version' ;;
"reachability") printf 'accessibilite' ;;
"server-header") printf 'en-tete Server' ;;
"inventory") printf 'inventaire' ;;
"flags") printf 'options' ;;
"target domain") printf 'domaine cible' ;;
"currency") printf 'devise' ;;
"phases") printf 'phases' ;;
"access") printf 'acces' ;;
*)
local out="$s"
out=${out//shared library missing/bibliotheque partagee manquante}
out=${out//shared library error/erreur de bibliotheque partagee}
out=${out//not active/non actif}
out=${out//after_start/apres_demarrage}
out=${out//catalogued but not reachable/catalogue mais inaccessible}
out=${out//does not resolve/ne resout pas}
out=${out//protocol confirmed/protocole confirme}
out=${out//catalogued service OK/service catalogue OK}
out=${out//from Server header/depuis en-tete Server}
out=${out//OSV clean/OSV sans vulnerabilite}
out=${out//actionable vuln/vuln. actionnable}
out=${out//header version · not Debian pkg/version en-tete · pas paquet Debian}
out=${out//RUN_TIMEOUT exceeded/RUN_TIMEOUT depasse}
out=${out//public HTTPS/HTTPS public}
out=${out//container status/etat du conteneur}
out=${out//remote-only/uniquement distant}
out=${out//no SSH/sans SSH}
out=${out//failed — see list above/echec — voir la liste ci-dessus}
printf '%s' "$out"
;;
esac
}

1413
lib.sh Executable file
View file

File diff suppressed because it is too large Load diff

18
mail-catalog.conf Normal file
View file

@ -0,0 +1,18 @@
# mail-catalog.conf — domains and mail hosts for phase `mail` (v1.8.0+)
#
# Format (whitespace-separated; # comments):
# domain expected_mx (comma) mail_hosts (comma) smtp_ports imap_ports label
#
# Only these mail planes (no hacktivism / ad-hoc MX):
# • firefly.gnunet.org — taler.net + gnunet.org
# • anastasis.taler-systems.com — Anastasis / TSA mail
#
# Override path: MAIL_CATALOG=/path/to/file
# --- firefly.gnunet.org ---
taler.net firefly.gnunet.org firefly.gnunet.org 25,465,587 143,993 firefly-taler
gnunet.org firefly.gnunet.org firefly.gnunet.org 25,465,587 143,993 firefly-gnunet
# --- anastasis.taler-systems.com ---
anastasis.taler-systems.com anastasis.taler-systems.com anastasis.taler-systems.com 25,465,587 143,993 anastasis-tsa
taler-systems.com anastasis.taler-systems.com anastasis.taler-systems.com 25,465,587 143,993 tsa-via-anastasis

21
meta/README.md Normal file
View file

@ -0,0 +1,21 @@
# meta/ — optional & deprecated tooling
Not required for the normal **host-agent** monitoring pipeline
(`host-agent/run-*.sh` + `site-gen/console_to_html.py`).
| Subdir | Contents |
|--------|----------|
| **`deprecated/`** | Pre-v1.8 mail/mattermost host-agent wrappers + user systemd units (folded into **surface**) |
| **`firecuda/`** | Outside-runner install (launchd) + pull helpers — optional; prefer koopa host-agent |
| **`site-gen-legacy/`** | Multi-host `generate-monitoring-sites` / `deploy-monitoring-sites` (older laptop/outside path) |
## Production path (keep using)
```text
host-agent/run-host-report.sh
host-agent/run-{hacktivism,surface,aptdeploy,fp-*}.sh
host-agent/update-suite.sh
site-gen/console_to_html.py
```
Machine-local paths/SSH: **`taler-monitoring-env`** → `~/.config/taler-monitoring/env`.

20
meta/deprecated/run-mail-monitoring.sh Executable file
View file

@ -0,0 +1,20 @@
#!/usr/bin/env bash
# DEPRECATED (v1.8.0): mail is part of taler-monitoring-surface.
# Kept as a thin alias so old timers/docs do not break.
#
# Prefer: run-surface-monitoring.sh
# Public page: https://taler.hacktivism.ch/taler-monitoring-surface/
#
set -uo pipefail
AGENT_DIR=$(cd "$(dirname "$0")" && pwd)
echo "NOTE: run-mail-monitoring.sh is deprecated (v1.8.0) — folding into surface" >&2
export PHASES="${PHASES:-mail surface monpages}"
# force surface HTML paths (no /taler-monitoring-mail*)
export HTML_OK_DIR="taler-monitoring-surface"
export HTML_ERR_DIR="taler-monitoring-surface_err"
export HTML_URL_OK="/taler-monitoring-surface/"
export HTML_URL_ERR="/taler-monitoring-surface_err/"
export PAGE_LABEL="taler-monitoring-surface"
export MON_HOSTS="${MON_HOSTS:-taler.hacktivism.ch}"
export MONPAGES_INVENTORY=job
exec bash "$AGENT_DIR/run-surface-monitoring.sh" "$@"

19
meta/deprecated/run-mattermost-monitoring.sh Executable file
View file

@ -0,0 +1,19 @@
#!/usr/bin/env bash
# DEPRECATED (v1.8.0): mattermost is part of taler-monitoring-surface.
# Kept as a thin alias so old timers/docs do not break.
#
# Prefer: run-surface-monitoring.sh
# Public page: https://taler.hacktivism.ch/taler-monitoring-surface/
#
set -uo pipefail
AGENT_DIR=$(cd "$(dirname "$0")" && pwd)
echo "NOTE: run-mattermost-monitoring.sh is deprecated (v1.8.0) — folding into surface" >&2
export PHASES="${PHASES:-mattermost surface monpages}"
export HTML_OK_DIR="taler-monitoring-surface"
export HTML_ERR_DIR="taler-monitoring-surface_err"
export HTML_URL_OK="/taler-monitoring-surface/"
export HTML_URL_ERR="/taler-monitoring-surface_err/"
export PAGE_LABEL="taler-monitoring-surface"
export MON_HOSTS="${MON_HOSTS:-taler.hacktivism.ch}"
export MONPAGES_INVENTORY=job
exec bash "$AGENT_DIR/run-surface-monitoring.sh" "$@"

18
meta/deprecated/taler-monitoring-mail.service Normal file
View file

@ -0,0 +1,18 @@
[Unit]
Description=Taler monitoring mail (firefly/pixel MX SMTP IMAP)
Documentation=file:%h/src/taler-monitoring/host-agent/README.md
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin
Environment=CONTINUE_ON_ERROR=1
Environment=RUN_TIMEOUT=600
WorkingDirectory=%h/src/taler-monitoring
ExecStart=%h/src/taler-monitoring/meta/deprecated/run-mail-monitoring.sh
Nice=10
TimeoutStartSec=20min
[Install]
WantedBy=default.target

14
meta/deprecated/taler-monitoring-mail.timer Normal file
View file

@ -0,0 +1,14 @@
[Unit]
Description=Periodic taler-monitoring mail (MX/SMTP/IMAP)
Documentation=file:%h/src/taler-monitoring/host-agent/README.md
[Timer]
OnBootSec=30min
OnUnitActiveSec=4h
AccuracySec=5min
Persistent=true
RandomizedDelaySec=10min
Unit=taler-monitoring-mail.service
[Install]
WantedBy=timers.target

18
meta/deprecated/taler-monitoring-mattermost.service Normal file
View file

@ -0,0 +1,18 @@
[Unit]
Description=Taler monitoring Mattermost (mattermost.taler.net outside-in)
Documentation=file:%h/src/taler-monitoring/host-agent/README.md
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin
Environment=CONTINUE_ON_ERROR=1
Environment=RUN_TIMEOUT=300
WorkingDirectory=%h/src/taler-monitoring
ExecStart=%h/src/taler-monitoring/meta/deprecated/run-mattermost-monitoring.sh
Nice=10
TimeoutStartSec=15min
[Install]
WantedBy=default.target

14
meta/deprecated/taler-monitoring-mattermost.timer Normal file
View file

@ -0,0 +1,14 @@
[Unit]
Description=Periodic taler-monitoring Mattermost (chat health)
Documentation=file:%h/src/taler-monitoring/host-agent/README.md
[Timer]
OnBootSec=25min
OnUnitActiveSec=1h
AccuracySec=2min
Persistent=true
RandomizedDelaySec=5min
Unit=taler-monitoring-mattermost.service
[Install]
WantedBy=timers.target

34
meta/firecuda/com.hacktivism.taler-monitoring-sites.plist Normal file
View file

@ -0,0 +1,34 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<!--
launchd user agent — firecuda (max): every 4 hours
Install: ./install-firecuda-timer.sh
-->
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.hacktivism.taler-monitoring-sites</string>
<key>ProgramArguments</key>
<array>
<string>/bin/bash</string>
<string>__SITE_GEN__/run-on-firecuda.sh</string>
</array>
<key>StartInterval</key>
<integer>14400</integer>
<key>RunAtLoad</key>
<true/>
<key>StandardOutPath</key>
<string>__HOME__/Library/Logs/taler-monitoring-sites/launchd.out.log</string>
<key>StandardErrorPath</key>
<string>__HOME__/Library/Logs/taler-monitoring-sites/launchd.err.log</string>
<key>EnvironmentVariables</key>
<dict>
<key>PATH</key>
<string>/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin</string>
<key>SKIP_SSH</key>
<string>1</string>
<key>CONTINUE_ON_ERROR</key>
<string>1</string>
</dict>
</dict>
</plist>

85
meta/firecuda/install-firecuda-timer.sh Executable file
View file

@ -0,0 +1,85 @@
#!/usr/bin/env bash
# install-firecuda-timer.sh — OPTIONAL: rsync suite to firecuda + launchd every 4h
#
# POLICY (2026-07): firecuda launchd is **disabled**. Prefer koopa host-agent.
# This script remains for documentation / future re-enable only.
# Do not run unless you intentionally want a second outside-only scheduler.
#
# ./install-firecuda-timer.sh
# ./install-firecuda-timer.sh --run-now
#
set -euo pipefail
ROOT=$(cd "$(dirname "$0")" && pwd)
MON_ROOT=$(cd "$ROOT/../.." && pwd)
FIRECUDA="${FIRECUDA_SSH:-}"
if [ -z "$FIRECUDA" ]; then
echo "Set FIRECUDA_SSH to your outside-runner SSH host alias" >&2
exit 2
fi
REMOTE_BASE="${FIRECUDA_INSTALL_DIR:-taler-monitoring-site-gen}"
RUN_NOW=0
[ "${1:-}" = "--run-now" ] && RUN_NOW=1
echo "install → $FIRECUDA:~/$REMOTE_BASE (outside-only, every 4h)"
ssh -o BatchMode=yes -o ConnectTimeout=20 "$FIRECUDA" \
"mkdir -p \"\$HOME/$REMOTE_BASE\" \"\$HOME/Library/Logs/taler-monitoring-sites\" \"\$HOME/var/taler-monitoring-sites-work\" \"\$HOME/Library/LaunchAgents\""
# Sync full taler-monitoring suite (no secrets.env)
rsync -az --delete \
--exclude secrets.env \
--exclude 'android-test/artifacts' \
--exclude '.git' \
--exclude 'site-gen/settings.conf' \
"$MON_ROOT/" \
"${FIRECUDA}:$REMOTE_BASE/"
# settings on firecuda: outside only, run locally on firecuda, deploy to koopa
ssh -o BatchMode=yes "$FIRECUDA" "cat > \"\$HOME/$REMOTE_BASE/site-gen/settings.conf\" <<'EOF'
# auto-installed by install-firecuda-timer.sh — no secrets
RUNNER_SSH=
SKIP_SSH=1
CONTINUE_ON_ERROR=1
PHASES_ERR=urls versions
PHASES_OK=urls versions
DEPLOY_SSH=${DEPLOY_SSH:-}
DEPLOY_STAGING=${DEPLOY_STAGING:-$HOME/monitoring-sites-staging}
WORK_ROOT=
MONITORING_ROOT=..
SOURCE_REPO_WEB=https://git.hacktivism.ch/hernani/taler-monitoring
SOURCE_COMMIT_URL_TMPL={repo}/src/commit/{commit}
SOURCE_SUITE_PATH=.
EOF
# WORK_ROOT expanded on remote
sed -i.bak \"s|^WORK_ROOT=.*|WORK_ROOT=\$HOME/var/taler-monitoring-sites-work|\" \"\$HOME/$REMOTE_BASE/site-gen/settings.conf\" 2>/dev/null || \
sed -i '' \"s|^WORK_ROOT=.*|WORK_ROOT=\$HOME/var/taler-monitoring-sites-work|\" \"\$HOME/$REMOTE_BASE/site-gen/settings.conf\"
"
# launchd plist with absolute paths
ssh -o BatchMode=yes "$FIRECUDA" bash -s <<REMOTE
set -euo pipefail
HOME_DIR=\$HOME
FIRECUDA_META="\$HOME_DIR/$REMOTE_BASE/meta/firecuda"
SITE_GEN="\$HOME_DIR/$REMOTE_BASE/site-gen"
PLIST_SRC="\$FIRECUDA_META/com.hacktivism.taler-monitoring-sites.plist"
PLIST_DST="\$HOME_DIR/Library/LaunchAgents/com.hacktivism.taler-monitoring-sites.plist"
sed -e "s|__SITE_GEN__|\${FIRECUDA_META}|g" -e "s|__HOME__|\${HOME_DIR}|g" "\$PLIST_SRC" > "\$PLIST_DST"
chmod +x "\$FIRECUDA_META"/*.sh "\$SITE_GEN"/console_to_html.py 2>/dev/null || true
chmod +x "\$HOME_DIR/$REMOTE_BASE/meta/site-gen-legacy"/*.sh 2>/dev/null || true
chmod +x "\$FIRECUDA_META/run-on-firecuda.sh"
launchctl bootout "gui/\$(id -u)/com.hacktivism.taler-monitoring-sites" 2>/dev/null || true
launchctl unload "\$PLIST_DST" 2>/dev/null || true
launchctl bootstrap "gui/\$(id -u)" "\$PLIST_DST" 2>/dev/null || launchctl load "\$PLIST_DST"
launchctl enable "gui/\$(id -u)/com.hacktivism.taler-monitoring-sites" 2>/dev/null || true
launchctl print "gui/\$(id -u)/com.hacktivism.taler-monitoring-sites" 2>/dev/null | head -20 || launchctl list | grep taler-monitoring || true
echo "OK: launchd com.hacktivism.taler-monitoring-sites (StartInterval=14400)"
REMOTE
if [ "$RUN_NOW" = "1" ]; then
echo "run once now…"
ssh -o BatchMode=yes "$FIRECUDA" "\"\$HOME/$REMOTE_BASE/meta/firecuda/run-on-firecuda.sh\"" || true
fi
echo "done. logs: $FIRECUDA:~/Library/Logs/taler-monitoring-sites/"
echo "manual: ssh $FIRECUDA '~/taler-monitoring-site-gen/meta/firecuda/run-on-firecuda.sh'"

19
meta/firecuda/pull-firecuda-to-koopa.sh Executable file
View file

@ -0,0 +1,19 @@
#!/usr/bin/env bash
# pull-firecuda-to-koopa.sh — copy HTML from firecuda work dir → koopa staging
# Run on laptop (hernani) where both SSH aliases work.
set -euo pipefail
FIRECUDA="${FIRECUDA_SSH:-}"
if [ -z "$FIRECUDA" ]; then
echo "Set FIRECUDA_SSH to your outside-runner SSH host alias" >&2
exit 2
fi
KOOPA="${DEPLOY_SSH:?set DEPLOY_SSH in env}"
REMOTE_HTML="${FIRECUDA_HTML:-var/taler-monitoring-sites-work/html/}"
STAGE="${DEPLOY_STAGING:-${DEPLOY_STAGING:-$HOME/monitoring-sites-staging}}"
echo "firecuda:$REMOTE_HTML → $KOOPA:$STAGE"
ssh -o BatchMode=yes -o ConnectTimeout=20 "$KOOPA" "mkdir -p '$STAGE'"
rsync -az --delete \
"${FIRECUDA}:${REMOTE_HTML}" \
"${KOOPA}:${STAGE}/"
echo "OK staged. Root: rsync to (see ROOT-ON-KOOPA.md)"

59
meta/firecuda/run-on-firecuda.sh Executable file
View file

@ -0,0 +1,59 @@
#!/usr/bin/env bash
# run-on-firecuda.sh — entrypoint for launchd/cron on outside-runner.
# Outside-only monitoring (SKIP_SSH=1), all 9 sites, then stage HTML to koopa.
#
# Lives in meta/firecuda/ (optional). Prefer koopa host-agent for production mon.
#
set -uo pipefail
export PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:${PATH:-}"
HERE=$(cd "$(dirname "$0")" && pwd)
SUITE_ROOT=$(cd "$HERE/../.." && pwd)
LEGACY="$SUITE_ROOT/meta/site-gen-legacy"
SITE_GEN="$SUITE_ROOT/site-gen"
LOG_DIR="${FIRECUDA_LOG_DIR:-$HOME/Library/Logs/taler-monitoring-sites}"
mkdir -p "$LOG_DIR"
STAMP=$(date +%Y%m%d-%H%M%S)
LOG="$LOG_DIR/run-${STAMP}.log"
exec >>"$LOG" 2>&1
echo "======== $(date -Iseconds 2>/dev/null || date) outside-runner monitoring sites ========"
export SKIP_SSH=1
export CONTINUE_ON_ERROR=1
export AUTH401_CONTINUE=1
export RUNNER_SSH=
export PHASES_ERR="${PHASES_ERR:-urls versions}"
export PHASES_OK="${PHASES_OK:-urls versions}"
export WORK_ROOT="${WORK_ROOT:-$HOME/var/taler-monitoring-sites-work}"
export MONITORING_ROOT="${MONITORING_ROOT:-$SUITE_ROOT}"
export DEPLOY_SSH="${DEPLOY_SSH:-}"
export DEPLOY_STAGING="${DEPLOY_STAGING:-$HOME/monitoring-sites-staging}"
export SITE_GEN_DIR="$SITE_GEN"
unset ONLY_HOSTS
cd "$LEGACY" || exit 1
chmod +x generate-monitoring-sites.sh deploy-monitoring-sites.sh 2>/dev/null || true
chmod +x "$SITE_GEN/console_to_html.py" 2>/dev/null || true
./generate-monitoring-sites.sh
ec=$?
if [ -n "${DEPLOY_SSH:-}" ] && [ "${DEPLOY_FROM_FIRECUDA:-0}" = "1" ]; then
if [ -x ./deploy-monitoring-sites.sh ]; then
./deploy-monitoring-sites.sh || echo "WARN: deploy staging failed"
fi
else
echo "note: HTML at ${WORK_ROOT}/html — pull from laptop:"
echo " rsync -az outside-runner:var/taler-monitoring-sites-work/html/ \\"
echo " \${DEPLOY_SSH}:monitoring-sites-staging/"
echo " # or: $HERE/pull-firecuda-to-koopa.sh"
fi
ls -1t "$LOG_DIR"/run-*.log 2>/dev/null | tail -n +21 | xargs rm -f 2>/dev/null || true
echo "======== done ec=$ec ========"
exit "$ec"

47
meta/site-gen-legacy/deploy-monitoring-sites.sh Executable file
View file

@ -0,0 +1,47 @@
#!/usr/bin/env bash
# deploy-monitoring-sites.sh — rsync generated HTML to DEPLOY_WWW_ROOT on koopa.
# Does NOT reload Caddy (needs root — see ROOT-ON-KOOPA.md).
set -uo pipefail
HERE=$(cd "$(dirname "$0")" && pwd)
SUITE_ROOT=$(cd "$HERE/../.." && pwd)
SITE_GEN="${SITE_GEN_DIR:-$SUITE_ROOT/site-gen}"
ROOT="$HERE"
SETTINGS="${SITE_GEN_SETTINGS:-$SITE_GEN/settings.conf}"
[ -f "$SETTINGS" ] || SETTINGS="$SITE_GEN/settings.conf.example"
# shellcheck disable=SC1090
set -a
source <(grep -v '^\s*#' "$SETTINGS" | grep -v '^\s*$' | sed 's/\r$//')
set +a
WORK_ROOT="${WORK_ROOT:-/tmp/taler-monitoring-sites-work}"
HTML_DIR="$WORK_ROOT/html"
DEPLOY_SSH="${DEPLOY_SSH:-}"
DEPLOY_WWW_ROOT="${DEPLOY_WWW_ROOT:-}"
if [ ! -d "$HTML_DIR" ]; then
echo "no HTML_DIR $HTML_DIR — run generate-monitoring-sites.sh first" >&2
exit 1
fi
echo "deploy $HTML_DIR → ${DEPLOY_SSH}:${DEPLOY_WWW_ROOT}/"
# Prefer writing to a hernani-owned staging dir if /var/www not writable
STAGE_REMOTE="${DEPLOY_STAGING:-${DEPLOY_STAGING:-$HOME/monitoring-sites-staging}}"
ssh -o BatchMode=yes -o ConnectTimeout=20 "$DEPLOY_SSH" \
"mkdir -p '$STAGE_REMOTE' '$DEPLOY_WWW_ROOT' 2>/dev/null || mkdir -p '$STAGE_REMOTE'"
rsync -az --delete "$HTML_DIR/" "${DEPLOY_SSH}:${STAGE_REMOTE}/"
echo "staged on $DEPLOY_SSH:$STAGE_REMOTE"
echo
echo "If $DEPLOY_WWW_ROOT is root-owned, as root on koopa:"
echo " mkdir -p $DEPLOY_WWW_ROOT"
echo " rsync -a --delete $STAGE_REMOTE/ $DEPLOY_WWW_ROOT/"
echo " chown -R caddy:caddy $DEPLOY_WWW_ROOT # or root:caddy — match other www"
echo " # then Caddyfile handles + reload — see ROOT-ON-KOOPA.md"
echo
# try direct rsync to DEPLOY_WWW_ROOT if writable
if ssh -o BatchMode=yes "$DEPLOY_SSH" "test -w '$DEPLOY_WWW_ROOT' 2>/dev/null || test -w \"\$(dirname '$DEPLOY_WWW_ROOT')\""; then
rsync -az --delete "$HTML_DIR/" "${DEPLOY_SSH}:${DEPLOY_WWW_ROOT}/" && \
echo "also synced directly to $DEPLOY_WWW_ROOT"
fi

277
meta/site-gen-legacy/generate-monitoring-sites.sh Executable file
View file

@ -0,0 +1,277 @@
#!/usr/bin/env bash
# generate-monitoring-sites.sh — run taler-monitoring, build console HTML for
# /monitoring and /monitoring_err (static files for host Caddy later).
#
# Usage:
# ./generate-monitoring-sites.sh # all SITES (or ONLY_HOSTS)
# ONLY_HOSTS="bank.hacktivism.ch taler.hacktivism.ch" ./generate-monitoring-sites.sh
# ./generate-monitoring-sites.sh --dry-run # HTML from empty/missing logs only
# ./generate-monitoring-sites.sh --skip-run # only convert existing logs
#
# Settings: settings.conf (from settings.conf.example). No secrets in settings.
# Secrets for e2e: ../secrets.env (optional; default phases are urls versions).
#
set -uo pipefail
export PYTHONUNBUFFERED=1
# Global reporting defaults (same as host-agent run-host-report.sh)
: "${RUN_TIMEOUT:=600}"
export RUN_TIMEOUT
HERE=$(cd "$(dirname "$0")" && pwd)
SUITE_ROOT=$(cd "$HERE/../.." && pwd)
SITE_GEN="${SITE_GEN_DIR:-$SUITE_ROOT/site-gen}"
ROOT="$HERE"
SETTINGS="${SITE_GEN_SETTINGS:-$SITE_GEN/settings.conf}"
EXAMPLE="${SITE_GEN}/settings.conf.example"
[ -f "$EXAMPLE" ] || EXAMPLE="$HERE/settings.conf.example"
if [ ! -f "$SETTINGS" ]; then
if [ -f "$EXAMPLE" ]; then
echo "note: no $SETTINGS — using example defaults (copy to settings.conf to customize)"
SETTINGS="$EXAMPLE"
else
echo "missing settings: $EXAMPLE" >&2
exit 2
fi
fi
# Load KEY=value lines; allow SITES already set in environment to win.
# Multiline SITES in the conf file: use SITES_FILE= or env SITES= instead.
while IFS= read -r _line || [ -n "$_line" ]; do
_line=${_line%%#*}
_line=$(echo "$_line" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')
[ -z "$_line" ] && continue
case "$_line" in
SITES=*|SITES_FILE=*) continue ;; # handled below / env
*=*)
_k=${_line%%=*}
_v=${_line#*=}
# do not override pre-set env
if [ -z "${!_k+x}" ] 2>/dev/null || [ -z "${!_k}" ]; then
export "$_k=$_v"
fi
;;
esac
done < "$SETTINGS"
# SITES from dedicated file or keep env; else parse heredoc-style from example list
if [ -n "${SITES_FILE:-}" ] && [ -f "$SITES_FILE" ]; then
SITES=$(grep -v '^\s*#' "$SITES_FILE" | grep -v '^\s*$')
export SITES
elif [ -z "${SITES:-}" ]; then
# default 9 fronts
SITES="bank.hacktivism.ch|hacktivism.ch
exchange.hacktivism.ch|hacktivism.ch
taler.hacktivism.ch|hacktivism.ch
stage.bank.lefrancpaysan.ch|stage.lefrancpaysan.ch
stage.exchange.lefrancpaysan.ch|stage.lefrancpaysan.ch
stage.monnaie.lefrancpaysan.ch|stage.lefrancpaysan.ch
bank.lefrancpaysan.ch|lefrancpaysan.ch
exchange.lefrancpaysan.ch|lefrancpaysan.ch
monnaie.lefrancpaysan.ch|lefrancpaysan.ch"
export SITES
fi
SKIP_RUN=0
DRY=0
while [ $# -gt 0 ]; do
case "$1" in
--skip-run) SKIP_RUN=1; shift ;;
--dry-run) DRY=1; SKIP_RUN=1; shift ;;
-h|--help) sed -n '1,20p' "$0"; exit 0 ;;
*) echo "unknown arg: $1" >&2; exit 2 ;;
esac
done
MONITORING_ROOT=$(cd "${MONITORING_ROOT:-$SUITE_ROOT}" && pwd)
WORK_ROOT="${WORK_ROOT:-/tmp/taler-monitoring-sites-work}"
LOG_DIR="$WORK_ROOT/logs"
HTML_DIR="$WORK_ROOT/html"
mkdir -p "$LOG_DIR" "$HTML_DIR"
# commit of monitoring suite used for this run
if git -C "$MONITORING_ROOT/../.." rev-parse HEAD >/dev/null 2>&1; then
REPO_ROOT=$(cd "$MONITORING_ROOT/../.." && pwd)
elif git -C "$MONITORING_ROOT" rev-parse HEAD >/dev/null 2>&1; then
REPO_ROOT=$(cd "$MONITORING_ROOT" && pwd)
else
REPO_ROOT=$(cd "$ROOT/../../.." && pwd)
fi
COMMIT=$(git -C "$REPO_ROOT" rev-parse HEAD 2>/dev/null || echo unknown)
COMMIT_SHORT=$(git -C "$REPO_ROOT" rev-parse --short=12 HEAD 2>/dev/null || echo unknown)
SOURCE_REPO_WEB="${SOURCE_REPO_WEB:-https://git.hacktivism.ch/hernani/taler-monitoring}"
TMPL="${SOURCE_COMMIT_URL_TMPL:-\{repo\}/src/commit/\{commit\}}"
COMMIT_URL=$(printf '%s' "$TMPL" | sed "s|{repo}|$SOURCE_REPO_WEB|g; s|{commit}|$COMMIT|g")
SUITE_PATH="${SOURCE_SUITE_PATH:-.}"
# Outside-only public checks (no SSH into koopa/stage containers)
PHASES_ERR="${PHASES_ERR:-urls versions}"
PHASES_OK="${PHASES_OK:-urls versions}"
CONTINUE_ON_ERROR="${CONTINUE_ON_ERROR:-1}"
SKIP_SSH="${SKIP_SSH:-1}"
# Drop SSH phases if someone passes "all" / inside / server / e2e by mistake
filter_phases_outside() {
local out=() p
for p in "$@"; do
case "$p" in
inside|server|e2e|ladder|goa-ladder|auth401) continue ;;
all) out+=(urls versions);;
*) out+=("$p") ;;
esac
done
printf '%s\n' "${out[@]}"
}
run_mon() {
local domain="$1" phases="$2" logfile="$3" cont="$4"
# shellcheck disable=SC2206
local ph=( $phases )
mapfile -t ph < <(filter_phases_outside "${ph[@]}")
[ "${#ph[@]}" -gt 0 ] || ph=(urls versions)
: >"$logfile"
echo "+ -d $domain · phases=${ph[*]} · CONTINUE=$cont · SKIP_SSH=$SKIP_SSH · RUN_TIMEOUT=$RUN_TIMEOUT"
if [ -n "${RUNNER_SSH:-}" ]; then
echo "+ via ssh $RUNNER_SSH"
rsync -az --delete \
--exclude secrets.env \
--exclude 'android-test/artifacts' \
--exclude '.git' \
"$MONITORING_ROOT/" \
"${RUNNER_SSH}:${RUNNER_REMOTE_WORKDIR}/" \
|| return 1
# shellcheck disable=SC2029
if command -v stdbuf >/dev/null 2>&1; then
ssh -o BatchMode=yes -o ConnectTimeout=30 "$RUNNER_SSH" \
"cd '${RUNNER_REMOTE_WORKDIR}' && chmod +x taler-monitoring.sh check_*.sh 2>/dev/null; \
export CONTINUE_ON_ERROR='$cont' AUTH401_CONTINUE='$cont' SKIP_SSH='$SKIP_SSH' RUN_TIMEOUT='$RUN_TIMEOUT'; \
stdbuf -oL -eL ./taler-monitoring.sh -d '$domain' ${ph[*]}" \
2>&1 | stdbuf -oL -eL tee -a "$logfile"
else
ssh -o BatchMode=yes -o ConnectTimeout=30 "$RUNNER_SSH" \
"cd '${RUNNER_REMOTE_WORKDIR}' && chmod +x taler-monitoring.sh check_*.sh 2>/dev/null; \
export CONTINUE_ON_ERROR='$cont' AUTH401_CONTINUE='$cont' SKIP_SSH='$SKIP_SSH' RUN_TIMEOUT='$RUN_TIMEOUT'; \
./taler-monitoring.sh -d '$domain' ${ph[*]}" \
2>&1 | tee -a "$logfile"
fi
return "${PIPESTATUS[0]}"
fi
if command -v stdbuf >/dev/null 2>&1; then
( cd "$MONITORING_ROOT" && \
CONTINUE_ON_ERROR="$cont" AUTH401_CONTINUE="$cont" SKIP_SSH="$SKIP_SSH" \
RUN_TIMEOUT="$RUN_TIMEOUT" \
stdbuf -oL -eL ./taler-monitoring.sh -d "$domain" "${ph[@]}" ) \
2>&1 | stdbuf -oL -eL tee -a "$logfile"
else
( cd "$MONITORING_ROOT" && \
CONTINUE_ON_ERROR="$cont" AUTH401_CONTINUE="$cont" SKIP_SSH="$SKIP_SSH" \
RUN_TIMEOUT="$RUN_TIMEOUT" \
./taler-monitoring.sh -d "$domain" "${ph[@]}" ) \
2>&1 | tee -a "$logfile"
fi
return "${PIPESTATUS[0]}"
}
htmlify() {
local host="$1" mode="$2" log="$3" out="$4" other="$5"
mkdir -p "$(dirname "$out")"
if [ ! -f "$SITE_GEN/console_to_html.py" ]; then
echo "WARN: console_to_html.py missing — bootstrap $out"
{
echo "<!DOCTYPE html><html><head><meta charset=utf-8><title>$host $mode</title></head>"
echo "<body><h1>$host · $mode</h1><p>commit <a href=\"$COMMIT_URL\">$COMMIT_SHORT</a></p>"
echo "<pre>"
tail -n 100 "$log" 2>/dev/null | sed 's/&/\&amp;/g;s/</\&lt;/g' || true
echo "</pre></body></html>"
} >"$out"
return 0
fi
python3 "$SITE_GEN/console_to_html.py" \
--log "$log" \
--out "$out" \
--hostname "$host" \
--mode "$mode" \
--commit "$COMMIT" \
--commit-url "$COMMIT_URL" \
--suite-path "$SUITE_PATH" \
--link-other "$other"
}
# Parse SITES
mapfile -t SITE_LINES < <(printf '%s\n' "$SITES" | sed '/^\s*$/d' | sed 's/#.*//')
ONLY="${ONLY_HOSTS:-}"
for line in "${SITE_LINES[@]}"; do
line=$(echo "$line" | tr -d ' \t')
[ -z "$line" ] && continue
host=${line%%|*}
domain=${line#*|}
if [ -n "$ONLY" ]; then
echo " $ONLY " | grep -q " $host " || continue
fi
echo "======== $host (domain=$domain) commit=$COMMIT_SHORT ========"
base_html="$HTML_DIR/$host"
mkdir -p "$base_html/monitoring" "$base_html/monitoring_err"
log_err="$LOG_DIR/${host}.err.log"
log_ok="$LOG_DIR/${host}.ok.log"
ec_err=1
ec_ok=1
if [ "$SKIP_RUN" != "1" ]; then
run_mon "$domain" "$PHASES_ERR" "$log_err" "$CONTINUE_ON_ERROR"
ec_err=$?
# ok-run without continue (strict)
run_mon "$domain" "$PHASES_OK" "$log_ok" "0"
ec_ok=$?
else
[ -f "$log_err" ] || : >"$log_err"
[ -f "$log_ok" ] || : >"$log_ok"
# infer exit from log if present
if grep -q '\[ ERROR \]' "$log_err" 2>/dev/null || grep -q 'ERROR' "$log_err" 2>/dev/null; then
ec_err=1
else
ec_err=0
fi
ec_ok=$ec_err
fi
# Always write pages (first run / empty logs too)
[ -f "$log_err" ] || : >"$log_err"
[ -f "$log_ok" ] || : >"$log_ok"
htmlify "$host" "err" "$log_err" \
"$base_html/monitoring_err/index.html" "/monitoring/"
if [ "$ec_ok" -eq 0 ] && [ "$ec_err" -eq 0 ]; then
htmlify "$host" "ok" "$log_ok" \
"$base_html/monitoring/index.html" "/monitoring_err/"
echo " → /monitoring (clean) + /monitoring_err"
else
htmlify "$host" "redirect" "$log_err" \
"$base_html/monitoring/index.html" "/monitoring_err/"
echo " → /monitoring (redirect stub) + /monitoring_err (errors) ec_err=$ec_err ec_ok=$ec_ok"
fi
# first-run safety
if [ ! -f "$base_html/monitoring/index.html" ]; then
htmlify "$host" "err" "$log_err" \
"$base_html/monitoring/index.html" "/monitoring_err/"
fi
printf '%s\n' "$COMMIT" >"$base_html/COMMIT"
printf '%s\n' "$COMMIT_URL" >"$base_html/COMMIT_URL"
printf 'host=%s domain=%s ec_err=%s ec_ok=%s commit=%s run_timeout=%s\n' \
"$host" "$domain" "$ec_err" "$ec_ok" "$COMMIT" "$RUN_TIMEOUT" | tee "$base_html/STATUS.txt"
done
echo
echo "HTML under $HTML_DIR"
echo "commit $COMMIT_SHORT $COMMIT_URL"
echo "RUN_TIMEOUT=${RUN_TIMEOUT}s (global; 0=unlimited)"
echo "Deploy (as hernani, may need sudo for /var/www):"
echo " $HERE/deploy-monitoring-sites.sh"
echo "Root on koopa: see ROOT-ON-KOOPA.md"

1431
metrics.sh Normal file
View file

File diff suppressed because it is too large Load diff

96
secrets.env.example Normal file
View file

@ -0,0 +1,96 @@
# taler-monitoring — local secrets template
#
# Copy to secrets.env (gitignored) and fill values, OR export the same vars:
# cp secrets.env.example secrets.env
# $EDITOR secrets.env
#
# Never commit secrets.env. Prefer file paths under koopa-admin-secrets
# for long-term storage; this file is for workstation one-shots / overrides.
#
# Load order: env already set > secrets.env > SECRETS_ROOT files > SSH koopa
# --- Preferred: point at secrets tree (no passwords in this file) ---
# Sibling of admin-log on this machine:
# KOOPA_ADMIN_SECRETS=$HOME/src/koopa/koopa-admin-secrets
# Or host-root only:
# SECRETS_ROOT=$HOME/src/koopa/koopa-admin-secrets/koopa/host-root
# --- Or paste passwords / tokens (local GOA e2e) ---
# Bank admin (libeufin-bank admin) — funds / creates accounts in e2e
# E2E_BANK_ADMIN_PASS=
# BANK_ADMIN_PASS= # alias, same meaning
# Merchant instance token for private orders (goa-demo-cp4zqk on hacktivism)
# Full "secret-token:…" or raw secret (Bearer prefix added by e2e)
# E2E_MERCHANT_TOKEN=
# MERCHANT_TOKEN=
# MERCHANT_INSTANCE_PASSWORD=
# --- FrancPaysan STAGE (TESTPAYSAN) e2e ---
# Default phases for -d stage.lefrancpaysan.ch: urls + inside + versions + e2e
# inside uses francpaysan-stage-user (stagepaysan); stats from public HTTPS.
# Admin credit (required for ATM withdraw ladder on stage):
# E2E_BANK_ADMIN_PASS=… # or auto-load via SSH
# FRANCPAYSAN_SSH=
# FRANCPAYSAN_SECRETS=$HOME/francpaysan-secrets
# Public shop pays use E2E_USE_TEMPLATES=1 (default on TESTPAYSAN) — no merchant token.
# Optional overrides
# MERCHANT_INSTANCE=goa-demo-cp4zqk
# EXPECT_CURRENCY=GOA
# E2E_TIMEOUT=340
# PROGRESS_TOTAL=120
# PROGRESS_SHOW_EVERY=8
# PROGRESS_OFF=1
# --- Merchant SPA /webui/ fingerprints (urls phase, group www.webui) ---
# EXPECT_WEBUI_VERSION=1.6.11
# EXPECT_WEBUI_OVERLAY=selfbuild-v1.6.11
# WEBUI_OVERLAY_DENY=c778af04|master-11340
# CHECK_WEBUI_SPA=0
# --- auth401 phase (WebUI signup/login/pwchange/admin + case matrix) ---
# Default FULL=1 creates throwaway MIX instance + runs all groups.
# Stage also loads durable mon401 for extra durable.* checks:
# AUTH401_FULL=1
# AUTH401_CONTINUE=1 # optional: hard prereqs soft; full ERROR list always via EXIT trap
# CONTINUE_ON_ERROR=1 # alias for AUTH401_CONTINUE
# (auth401 no longer uses set -e — silent mid-run abort fixed)
# AUTH401_CREATE=1
# AUTH401_PREFER_DURABLE=0 # 1 = only mon401, skip throwaway signup
# AUTH401_INSTANCE=mon401
# AUTH401_PASSWORD=
# AUTH401_DURABLE_ID=mon401
# AUTH401_CREATE_PASSWORD=Mon401-Test-Pass!
# AUTH401_SKIP_IF_MFA=1
# AUTH401_BEARER= # default-instance-token (not admin)
# AUTH401_ADMIN_TOKEN= # required for admin-create group
# AUTH401_ADMIN_PASSWORD= # or login as AUTH401_ADMIN_USER=admin
# AUTH401_ADMIN_USER=admin
# FRANCPAYSAN_SECRETS=$HOME/francpaysan-secrets
# --- Devtesting / fake-franken CHF (rusty.taler-ops.ch) ---
# Phase: devtesting | franken | fake-franken
# SSH Host taler-rusty-devtesting (user devtesting, forced taler-devtesting CLI).
# DEVTESTING_SSH=taler-rusty-devtesting
# DEVTESTING_AMOUNT=CHF:1.00
# DEVTESTING_CREDIT_PAYTO= # optional; auto from EXCHANGE_PUBLIC/keys or TOPS IBAN
# DEVTESTING_EXCHANGE_IBAN=CH6808573105529100001
# DEVTESTING_SKIP=1 # skip phase
# --- Ladder (GOA withdraw ladder · explorer pool) ---
# Prefer SECRETS_ROOT / KOOPA_ADMIN_SECRETS (bank-explorer-password.txt).
# Overrides only if needed:
# EXP_USER=explorer
# EXP_PW=
# EXP_PW_FILE=$HOME/src/koopa/koopa-admin-secrets/koopa/host-root/taler-bank/bank-explorer-password.txt
# CLI_JS=$HOME/taler/src/taler-typescript-core/packages/taler-wallet-cli/bin/taler-wallet-cli.mjs
# Values that typically appear on a full stack (for orientation only):
# bank-admin-password.txt
# bank-explorer-password.txt
# bank-exchange-password.txt
# bank-goa-demo-cp4zqk-password.txt
# merchant-goa-demo-cp4zqk-password.txt
# exchange-attribute-encryption.secret.conf
# (see ../SECRETS.md and koopa-admin-secrets/MAP.md)

20
site-gen/README.md Normal file
View file

@ -0,0 +1,20 @@
# site-gen — HTML converter + Caddy notes
**Required for host-agent:** `console_to_html.py` (log → sticky HTML).
| File | Role |
|------|------|
| `console_to_html.py` | Convert suite console log → public mon HTML |
| `caddy-monitoring-handles.snippet` | Caddy `handle` / bare redir notes |
| `ROOT-ON-KOOPA.md` | Publish staging → `/var/www` (paths from env) |
| `settings.conf.example` | Optional settings for **legacy** multi-host generator |
## Optional / moved
Legacy multi-host generate/deploy and firecuda launchd tooling live under
**`../meta/`** (see `meta/README.md`).
```bash
# normal publish is host-agent on the mon host, not these scripts
../host-agent/run-hacktivism-monitoring.sh # etc.
```

97
site-gen/ROOT-ON-KOOPA.md Normal file
View file

@ -0,0 +1,97 @@
# Root on koopa (hacktivism monitoring sites)
Host Caddy runs as **root/systemd** (`caddy` user). Static monitoring HTML is
**not** put into Taler containers — only host paths + Caddy `handle`.
After host-agent (or optional `meta/site-gen-legacy/generate-monitoring-sites.sh`) + deploy, files live in:
```text
$DEPLOY_STAGING (or $HOME/monitoring-sites-staging)/<hostname>/monitoring/index.html
$DEPLOY_STAGING (or $HOME/monitoring-sites-staging)/<hostname>/monitoring_err/index.html
```
## One-shot as root (preferred)
```bash
# absolute path — as root, ~ is /root (script not found under ~/koopa-caddy/…)
sudo $APPLY_MONITORING_LIVE
# same script is versioned in the suite:
# host-agent/apply-monitoring-live.sh → install/sync to $KOOPA_CADDY_DIR/
```
That rsyncs staging → `/var/www/monitoring-sites`, installs the prepared
Caddyfile, reloads Caddy, and smokes mon URLs (including bare surface).
## Manual equivalent
```bash
install -d -o hernani -g caddy -m 755 /var/www/monitoring-sites
rsync -a --delete $DEPLOY_STAGING (or $HOME/monitoring-sites-staging)/ /var/www/monitoring-sites/
chown -R hernani:caddy /var/www/monitoring-sites
install -m 644 $KOOPA_CADDY_DIR/Caddyfile /etc/caddy/Caddyfile
caddy validate --config /etc/caddy/Caddyfile
systemctl reload caddy
systemctl is-active caddy
```
## Snippet per hacktivism site (inside each `*.hacktivism.ch { }` block)
Place **before** the catch-all `reverse_proxy`. Canonical copy:
`site-gen/caddy-monitoring-handles.snippet`.
Disk layout (path on disk matches URL under host root):
```text
/var/www/monitoring-sites/bank.hacktivism.ch/monitoring/index.html
/var/www/monitoring-sites/bank.hacktivism.ch/monitoring_err/index.html
```
```caddy
# bare → slash (MUST use redir * /path/ — see footgun below)
handle /monitoring {
redir * /monitoring/ 302
}
handle /monitoring* {
root * /var/www/monitoring-sites/{host}
file_server
}
handle /monitoring_err {
redir * /monitoring_err/ 302
}
handle /monitoring_err* {
root * /var/www/monitoring-sites/{host}
file_server
}
```
### Caddy redir footgun (merchant code 21)
```caddy
# WRONG — parsed as matcher=/monitoring/ to="302" → Location: 302
handle /monitoring {
redir /monitoring/ 302
}
# RIGHT
handle /monitoring {
redir * /monitoring/ 302
}
```
Symptom: `https://…/monitoring` or `…/taler-monitoring-surface` returns merchant
JSON `{"code":21,…}` while the slash form serves HTML.
## Check
```bash
curl -sS -o /dev/null -w '%{http_code}\n' https://bank.hacktivism.ch/monitoring/
curl -sS -D- -o /dev/null https://taler.hacktivism.ch/taler-monitoring-surface | head -15
# bare: expect 302 Location: /taler-monitoring-surface/ — not JSON code 21
curl -sS -o /dev/null -w '%{http_code}\n' https://taler.hacktivism.ch/taler-monitoring-surface/
```
## Not needed as root
- Running `taler-monitoring` (use **optional outside runner** / laptop)
- Writing into podman Taler containers

72
site-gen/caddy-monitoring-handles.snippet Normal file
View file

@ -0,0 +1,72 @@
# Host Caddy (root service). Place BEFORE catch-all reverse_proxy in each site block.
#
# CRITICAL: bare paths (no trailing slash) must redirect to slash, or the request
# falls through to merchant/bank/exchange API → Taler JSON code 21.
#
# CORRECT (site-level named matcher — preferred):
# @mon_bare path /monitoring /monitoring_err \
# /taler-monitoring-surface /taler-monitoring-surface_err \
# /taler-monitoring-aptdeploy /taler-monitoring-aptdeploy_err
# redir @mon_bare {path}/ 302
#
# ALSO OK inside handle (matcher must be `*`):
# handle /taler-monitoring-surface {
# redir * /taler-monitoring-surface/ 302
# }
#
# WRONG (Caddy footgun — causes merchant code 21 on bare URLs):
# redir /taler-monitoring-surface/ 302
# → parsed as matcher=/taler-monitoring-surface/ to="302"
# → Location: 302, or no-op then fallthrough to reverse_proxy
#
# root must be the *host directory* (…/monitoring-sites/{host}), and the
# file_server matcher a path *prefix* (`/monitoring*`), so
# /monitoring/index.html maps to {root}/monitoring/index.html.
#
# Disk layout:
# /var/www/monitoring-sites/{host}/monitoring/index.html
# /var/www/monitoring-sites/taler.hacktivism.ch/taler-monitoring-*/index.html
#
# Live apply on koopa (absolute path — ~ as root is /root):
# sudo ${APPLY_MONITORING_LIVE}
# Runtime-only (no sudo; hernani admin API):
# curl -g -X POST http://[::1]:2019/load -H 'Content-Type: text/caddyfile' \
# --data-binary @${KOOPA_CADDY_DIR}/Caddyfile
# --- only inside taler.hacktivism.ch { ... } ---
@mon_bare path /monitoring /monitoring_err \
/taler-monitoring-surface /taler-monitoring-surface_err \
/taler-monitoring-aptdeploy /taler-monitoring-aptdeploy_err \
/taler-monitoring-mail /taler-monitoring-mail_err \
/taler-monitoring-mattermost /taler-monitoring-mattermost_err
redir @mon_bare {path}/ 302
handle /taler-monitoring-surface_err* {
root * /var/www/monitoring-sites/taler.hacktivism.ch
file_server
}
handle /taler-monitoring-surface* {
root * /var/www/monitoring-sites/taler.hacktivism.ch
file_server
}
handle /taler-monitoring-aptdeploy_err* {
root * /var/www/monitoring-sites/taler.hacktivism.ch
file_server
}
handle /taler-monitoring-aptdeploy* {
root * /var/www/monitoring-sites/taler.hacktivism.ch
file_server
}
# --- bank + exchange + taler (each site block; set root host dir) ---
@mon_bare path /monitoring /monitoring_err
redir @mon_bare {path}/ 302
handle /monitoring_err* {
root * /var/www/monitoring-sites/{host}
file_server
}
handle /monitoring* {
root * /var/www/monitoring-sites/{host}
file_server
}

2072
site-gen/console_to_html.py Executable file
View file

File diff suppressed because it is too large Load diff

55
site-gen/settings.conf.example Normal file
View file

@ -0,0 +1,55 @@
# monitoring site-gen — non-secret settings (safe as example)
# Copy: cp settings.conf.example settings.conf
# Never put passwords/tokens here.
# On an outside runner the suite runs *locally* (RUNNER_SSH empty). From a laptop you
# can set RUNNER_SSH=<outside-runner-ssh-alias> to drive a one-off remote run instead.
RUNNER_SSH=
RUNNER_SSH_FALLBACKS=
RUNNER_REMOTE_WORKDIR=/tmp/taler-monitoring-site-gen
MONITORING_ROOT=..
# Outside-only: never SSH into bank/exchange/merchant containers
SKIP_SSH=1
# Deploy staging on koopa (host Caddy later — root copies to /var/www)
DEPLOY_SSH= # set SSH host for deploy
DEPLOY_SSH_FALLBACKS=koopa
# Static files for Caddy file_server (created per hostname)
DEPLOY_WWW_ROOT= # e.g. /var/www/monitoring-sites
DEPLOY_SUDO=sudo
CADDY_CONFIG=/etc/caddy/Caddyfile
CADDY_MIRROR_REL=configs/caddy/Caddyfile
# Working tree on deploy host for apply (optional)
KOOPA_CADDY_DIR= # absolute path to Caddy tree on mon host
# Source link for HTML footer (Forgejo)
SOURCE_REPO_WEB=https://git.hacktivism.ch/hernani/taler-monitoring
SOURCE_COMMIT_URL_TMPL={repo}/src/commit/{commit}
SOURCE_SUITE_PATH=.
CONTINUE_ON_ERROR=1
PHASES_ERR=urls versions
PHASES_OK=urls versions
SKIP_E2E=1
# Local work dir for logs + HTML
WORK_ROOT=/tmp/taler-monitoring-sites-work
# Sites: hostname|domain_profile (9 fronts)
# Start with hacktivism; stage + prod LFP follow
SITES="
bank.hacktivism.ch|hacktivism.ch
exchange.hacktivism.ch|hacktivism.ch
taler.hacktivism.ch|hacktivism.ch
stage.bank.lefrancpaysan.ch|stage.lefrancpaysan.ch
stage.exchange.lefrancpaysan.ch|stage.lefrancpaysan.ch
stage.monnaie.lefrancpaysan.ch|stage.lefrancpaysan.ch
bank.lefrancpaysan.ch|lefrancpaysan.ch
exchange.lefrancpaysan.ch|lefrancpaysan.ch
monnaie.lefrancpaysan.ch|lefrancpaysan.ch
"
# Only process this subset (space-separated hostnames). Empty = all SITES.
# Example for first deploy: ONLY_HOSTS=bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch
ONLY_HOSTS=

56
surface-catalog.conf Normal file
View file

@ -0,0 +1,56 @@
# surface-catalog.conf — known public Taler / GNUnet / Taler Systems hosts
# Format (whitespace-separated; # comments):
# host ports expect_proto label
# expect_proto: https | http | ssh | tcp | any
# ports: comma-separated (empty = defaults for proto)
#
# This list is expanded at runtime (DNS discovery, cert SANs, -d domain profile).
# Phase `surface` is NOT in default phases — run explicitly:
# ./taler-monitoring.sh surface
# ./taler-monitoring.sh -d hacktivism.ch surface
# --- taler.net ecosystem ---
www.taler.net 443 https taler-www
docs.taler.net 443 https taler-docs
git.taler.net 443,22 https taler-git
bugs.taler.net 443 https taler-bugs
lists.taler.net 443 https taler-lists
tutorials.taler.net 443 https taler-tutorials
mattermost.taler.net 443 https mattermost
bank.demo.taler.net 443 https demo-bank
exchange.demo.taler.net 443 https demo-exchange
backend.demo.taler.net 443 https demo-merchant
shop.demo.taler.net 443 https demo-shop
bank.test.taler.net 443 https test-bank
exchange.test.taler.net 443 https test-exchange
backend.test.taler.net 443 https test-merchant
# --- taler-ops / TOPS ---
www.taler-ops.ch 443 https tops-www
taler-ops.ch 443 https tops-apex
bank.taler-ops.ch 443 https tops-bank
exchange.taler-ops.ch 443 https tops-exchange
my.taler-ops.ch 443 https tops-merchant
stage.taler-ops.ch 443 https tops-stage-www
stage.my.taler-ops.ch 443 https tops-stage-merchant
exchange.stage.taler-ops.ch 443 https tops-stage-exchange
nexus.stage.taler-ops.ch 443 https tops-stage-nexus
shops.taler-ops.ch 443 https tops-shops
# --- gnunet.org ---
www.gnunet.org 443,80 https gnunet-www
gnunet.org 443,80 https gnunet-apex
git.gnunet.org 443,22 https gnunet-git
bugs.gnunet.org 443 https gnunet-bugs
# --- taler-systems.com ---
www.taler-systems.com 443 https tsa-www
taler-systems.com 443 https tsa-apex
anastasis.taler-systems.com 443,25,465,587,143,993 https anastasis-mail
# --- mail (also covered by phase mail; listed for surface nmap/ports) ---
firefly.gnunet.org 443,25,465,587,143,993 https firefly-mail
# --- common infra hostnames (remote only; may 404/redirect) ---
deb.taler.net 443 https deb-taler
ftp.gnu.org 443 https gnu-ftp

679
taler-monitoring.sh Executable file
View file

@ -0,0 +1,679 @@
#!/usr/bin/env bash
# taler-monitoring — public URL / stack checks for a Taler domain
#
# ./taler-monitoring.sh # local GOA (urls + inside + e2e)
# ./taler-monitoring.sh -d taler.net urls # public demo, no SSH
# ./taler-monitoring.sh --domain taler-ops.ch # public ops, no SSH
# TALER_DOMAIN=demo.taler.net ./taler-monitoring.sh urls
#
# Tags: [OK] [INFO] [WARN] [ERROR] [BLOCKER]
# Exit 0 only if every selected phase exits 0.
set -euo pipefail
ROOT=$(cd "$(dirname "$0")" && pwd)
# --- suite identity (also: --ver / --version / -V) ---
SOURCE_REPO_WEB="${SOURCE_REPO_WEB:-https://git.hacktivism.ch/hernani/taler-monitoring}"
SOURCE_REPO_GIT="${SOURCE_REPO_GIT:-https://git.hacktivism.ch/hernani/taler-monitoring.git}"
print_version() {
local ver tag commit commit_full dirty branch
ver="unknown"
if [ -f "$ROOT/VERSION" ]; then
ver=$(tr -d "[:space:]" <"$ROOT/VERSION" | sed "s/^v//")
fi
tag=""
commit="unknown"
commit_full=""
branch=""
dirty=""
if git -C "$ROOT" rev-parse --git-dir >/dev/null 2>&1; then
tag=$(git -C "$ROOT" describe --tags --exact-match 2>/dev/null || true)
if [ -z "$tag" ]; then
tag=$(git -C "$ROOT" describe --tags --abbrev=0 2>/dev/null || true)
fi
commit=$(git -C "$ROOT" rev-parse --short=12 HEAD 2>/dev/null || echo unknown)
commit_full=$(git -C "$ROOT" rev-parse HEAD 2>/dev/null || true)
branch=$(git -C "$ROOT" rev-parse --abbrev-ref HEAD 2>/dev/null || true)
if [ -n "$(git -C "$ROOT" status --porcelain 2>/dev/null)" ]; then
dirty=" (dirty)"
fi
fi
if [ -n "$tag" ] && [ "$tag" = "v$ver" ]; then
printf "taler-monitoring %s\n" "$tag"
else
printf "taler-monitoring v%s\n" "$ver"
fi
printf "version: %s\n" "$ver"
if [ -n "$tag" ]; then
if [ "$tag" = "v$ver" ]; then
printf "tag: %s\n" "$tag"
else
printf "git-tag: %s (tree; VERSION file is %s)\n" "$tag" "$ver"
fi
fi
printf "commit: %s%s\n" "$commit" "$dirty"
[ -n "$branch" ] && [ "$branch" != "HEAD" ] && printf "branch: %s\n" "$branch"
printf "repo: %s\n" "$SOURCE_REPO_WEB"
printf "clone: %s\n" "$SOURCE_REPO_GIT"
if [ -n "$commit_full" ]; then
printf "commit-url: %s/src/commit/%s\n" "$SOURCE_REPO_WEB" "$commit_full"
fi
if [ -n "$tag" ]; then
printf "tag-url: %s/src/tag/%s\n" "$SOURCE_REPO_WEB" "$tag"
fi
}
# Fast path: version before loading lib.sh / phases
case "${1:-}" in
--ver|--version|-V)
print_version
exit 0
;;
esac
# Line-buffered stdout/stderr so redirected logs (tee/host-agent) flush each line.
if [ "${_TALER_MON_STDBUF:-0}" != "1" ] && command -v stdbuf >/dev/null 2>&1; then
export _TALER_MON_STDBUF=1
exec stdbuf -oL -eL bash "$0" "$@"
fi
usage() {
cat <<'EOF'
taler-monitoring — bank / exchange / merchant checks
Usage:
./taler-monitoring.sh [options] [phases...]
Phases:
urls public HTTPS (no SSH) ← default with --domain
(includes merchant /webui/ SPA fingerprints: version.txt, overlay, assets)
inside container status via SSH (local stack only)
versions taler packages vs deb.taler.net trixie + repo availability
sanity public + optional server
server server-side only (SSH)
e2e withdraw + pay (small amounts; remote aborts on login/KYC)
ladder withdraw/pay amount ladder (GOA ceiling or stage TESTPAYSAN max_wire)
auth401 merchant Basic-auth / case matrix (HTTP 401 paths; may create throwaway instance)
aptdeploy koopa podman apt-src smoke: taler-merchant in
koopa-taler-deploy-test-apt-src-trixie{,-testing}
surface REMOTE-ONLY public inventory (NOT in default/all/full):
ecosystem hosts (taler.net, gnunet.org, taler-systems.com, mattermost, …)
or -d DOMAIN → that domain’s surface; port/protocol/TLS/CVE (OSV)
mattermost Mattermost chat health (default mattermost.taler.net; SPA + /api/v4/system/ping)
mail MX/SMTP/IMAP for Taler mail (firefly, pixel/TSA, catalogued domains)
monpages public monitoring HTML via FQDN (obligatory ERROR; GOA full inventory, FP only FP)
uses MON_HOSTS + HTML_URL_OK (same as host-agent); v1.3.1+
devtesting fake-franken CHF via rusty.taler-ops.ch (taler-devtesting)
geniban + fake-incoming; needs SSH Host DEVTESTING_SSH (key to devtesting@rusty)
franken|fake-franken aliases for devtesting
all urls + inside + versions + sanity + e2e (SSH phases only on koopa)
full all + server + ladder + auth401 (maximum; long-running, needs secrets)
NOTE: surface is never included in all/full — pass it explicitly
Options:
-d, --domain DOMAIN load profile from domains.conf (bank/exchange/merchant)
presets: koopa | hacktivism.ch | taler.net | taler-ops.ch
| my.taler-ops.ch | stage.taler-ops.ch
unknown domain → heuristic hosts (see README)
local (koopa) allows SSH; others public-only
--bank URL bank base (hostname or https://…) — overrides profile
--exchange URL exchange base — overrides profile
--merchant URL merchant-backend base — overrides profile
--currency CODE expected currency (GOA, KUDOS, CHF, …); empty = report only
--no-probe do not probe alternate hosts (unknown domains only)
--lang en|fr UI/console language (sticky bar + badges)
overrides auto (FrancPaysan→fr, else en)
--ver, --version, -V print suite version, git commit, and repo URLs
-h, --help
Add a stack: edit domains.conf (name + bank + exchange + merchant + currency).
Examples:
./taler-monitoring.sh -d taler.net
./taler-monitoring.sh -d taler-ops.ch urls
./taler-monitoring.sh -d my.taler-ops.ch urls
./taler-monitoring.sh -d taler-ops.ch --merchant https://my.taler-ops.ch urls
./taler-monitoring.sh -d demo.taler.net --currency KUDOS
./taler-monitoring.sh --exchange https://exchange.taler-ops.ch \
--merchant https://my.taler-ops.ch --bank https://bank.taler-ops.ch urls
Env (same meaning):
TALER_DOMAIN BANK_PUBLIC EXCHANGE_PUBLIC MERCHANT_PUBLIC EXPECT_CURRENCY
TALER_DOMAINS_CONF SKIP_SSH=1
NO_COLOR=1 / CLICOLOR=0 disable green/yellow/red tags (default: coloured)
SKIP_SSH=1 NO_COLOR=1
RUN_TIMEOUT=600 whole-run wall clock seconds (default 600; 0=unlimited)
long phases (ladder/full/e2e) need a higher value or 0
DISK_WARN_USED_PCT=85 disk free: WARN when used ≥ this %
DISK_ERR_USED_PCT=95 disk free: ERROR when used ≥ this % (or avail=0 / 100%)
PERF_WARN_MS PERF_FAIL_MS (urls latency; default 8000 / 20000)
QR_CHECK=0 skip QR form + qrencode/zbarimg (urls phase)
QR_ECC=M qrencode ECC level (default M)
KOOPA_SSH KOOPA_SSH_FALLBACKS (set KOOPA_SSH / KOOPA_SSH_FALLBACKS in env)
METRICS_LOAD=0 skip host/container RAM/CPU probes (e2e/ladder/inside)
EXPECT_WEBUI_VERSION=1.6.11 pin merchant /webui/version.txt (urls)
EXPECT_WEBUI_OVERLAY=selfbuild require substr in version-overlay.txt
WEBUI_OVERLAY_DENY='master-11340' fail if overlay matches regex
CHECK_WEBUI_SPA=0 skip SPA fingerprint block in urls
AUTH401_* see check_auth401.sh / secrets.env.example
AUTH401_CONTINUE=1 / CONTINUE_ON_ERROR=1
auth401: run all groups, collect every ERROR (no mid-run halt)
APT_DEPLOY_SKIP=1 skip aptdeploy phase
APT_DEPLOY_CONTAINERS="name:suite …" default trixie + trixie-testing deploy-test pods
SURFACE_CVE=0 disable CVE queries in surface phase
SURFACE_CVE_LEVEL=warn|error bare Server-header versions default warn
(Debian package versions default error)
SURFACE_CATALOG=path override surface-catalog.conf
TALER_MON_LANG=en|fr same as --lang (default en; auto fr for *lefrancpaysan*)
Full load (GOA / hacktivism):
./taler-monitoring.sh -d hacktivism.ch full
# or explicit:
./taler-monitoring.sh -d hacktivism.ch urls inside versions sanity server e2e ladder auth401
# hacktivism host-agent also runs aptdeploy (apt-src merchant containers on koopa)
Remote surface / ecosystem (explicit only — never default):
./taler-monitoring.sh surface
./taler-monitoring.sh -d hacktivism.ch surface
./taler-monitoring.sh -d lefrancpaysan.ch surface
./taler-monitoring.sh mattermost # mattermost.taler.net chat health
./taler-monitoring.sh mail # firefly + pixel MX/SMTP/IMAP
SPA pin after selfbuild:
EXPECT_WEBUI_VERSION=1.6.11 EXPECT_WEBUI_OVERLAY=selfbuild-v1.6.11 \\
WEBUI_OVERLAY_DENY='c778af04|master-11340' \\
./taler-monitoring.sh -d hacktivism.ch urls
EOF
}
# Machine-local paths/hosts from taler-monitoring-env install (no hardcodes in suite).
# Install: ~/src/taler-monitoring-env/bin/install-env.sh <machine>
if [ "${_TALER_MON_ENV_LOADED:-0}" != "1" ]; then
_mon_cfg="${TALER_MONITORING_ENV:-${XDG_CONFIG_HOME:-$HOME/.config}/taler-monitoring/env}"
if [ -f "$_mon_cfg" ]; then
# shellcheck disable=SC1090
set -a
# shellcheck disable=SC1091
. "$_mon_cfg"
set +a
fi
unset _mon_cfg
export _TALER_MON_ENV_LOADED=1
fi
# shellcheck source=lib.sh
source "$ROOT/lib.sh"
# ensure i18n after domain may change later
i18n_init 2>/dev/null || true
# Optional gitignored secrets.env next to this suite (or MONITORING_SECRETS_ENV=)
load_monitoring_secrets_env 2>/dev/null || true
# Persist #NNN + progress across check_*.sh child processes
TALER_MON_STATE=$(mktemp "${TMPDIR:-/tmp}/taler-mon-state.XXXXXX")
export TALER_MON_STATE
trap 'rm -f "${TALER_MON_STATE:-}"' EXIT
PHASES=()
DOMAIN_SET=0
BANK_OVERRIDE=""
EXCHANGE_OVERRIDE=""
MERCHANT_OVERRIDE=""
CURRENCY_OVERRIDE=""
NO_PROBE=0
while [ $# -gt 0 ]; do
case "$1" in
--lang)
[ $# -ge 2 ] || { echo "error: --lang needs en|fr" >&2; exit 2; }
_lang="$2"
case "$_lang" in
fr|FR|fra|french) TALER_MON_LANG=fr ;;
en|EN|eng|english) TALER_MON_LANG=en ;;
*) echo "error: unknown language '$_lang' (use en or fr)" >&2; exit 2 ;;
esac
TALER_MON_LANG_SET=1
export TALER_MON_LANG TALER_MON_LANG_SET
shift 2
i18n_init 2>/dev/null || true
;;
-h|--help) usage; exit 0 ;;
--ver|--version|-V) print_version; exit 0 ;;
-d|--domain)
[ $# -ge 2 ] || { echo "missing arg for $1" >&2; exit 2; }
apply_taler_domain "$2"
DOMAIN_SET=1
TALER_DOMAIN_FROM_CLI=1
shift 2
;;
--bank)
[ $# -ge 2 ] || { echo "missing arg for $1" >&2; exit 2; }
BANK_OVERRIDE="${2%/}"; shift 2
;;
--exchange)
[ $# -ge 2 ] || { echo "missing arg for $1" >&2; exit 2; }
EXCHANGE_OVERRIDE="${2%/}"; shift 2
;;
--merchant)
[ $# -ge 2 ] || { echo "missing arg for $1" >&2; exit 2; }
MERCHANT_OVERRIDE="${2%/}"; shift 2
;;
--currency)
[ $# -ge 2 ] || { echo "missing arg for $1" >&2; exit 2; }
CURRENCY_OVERRIDE="$2"; shift 2
;;
--no-probe) NO_PROBE=1; shift ;;
urls|inside|versions|sanity|server|e2e|ladder|goa-ladder|auth401|aptdeploy|apt-deploy|apt_src|surface|ecosystem|mattermost|mail|monpages|pages|devtesting|franken|fake-franken|fake_franken|all|full) PHASES+=("$1"); shift ;;
*)
# bare domain shorthand: ./taler-monitoring.sh taler.net
if [[ "$1" == *.* && "$1" != *://* && "$1" != -* ]]; then
apply_taler_domain "$1"
DOMAIN_SET=1
TALER_DOMAIN_FROM_CLI=1
shift
else
echo "Unknown: $1" >&2; usage >&2; exit 2
fi
;;
esac
done
if [ "$NO_PROBE" = "1" ]; then
TALER_DOMAIN_PROBE=0
fi
if [ -n "$CURRENCY_OVERRIDE" ]; then
EXPECT_CURRENCY="$CURRENCY_OVERRIDE"
fi
if [ -n "$BANK_OVERRIDE" ]; then BANK_PUBLIC="$BANK_OVERRIDE"; fi
if [ -n "$EXCHANGE_OVERRIDE" ]; then EXCHANGE_PUBLIC="$EXCHANGE_OVERRIDE"; fi
if [ -n "$MERCHANT_OVERRIDE" ]; then MERCHANT_PUBLIC="$MERCHANT_OVERRIDE"; fi
# Only koopa may use SSH. Remote domains: public + optional e2e (no SSH).
if [ "${LOCAL_STACK:-1}" != "1" ]; then
SKIP_SSH=1
fi
# Remote: no SSH; ATM withdraw ladder set in check_e2e (smaller notes)
if [ "${LOCAL_STACK}" != "1" ]; then
E2E_FAKE_INCOMING=0
E2E_REMOTE=1
: "${E2E_WITHDRAW_VALUES:=10 20 50}"
: "${E2E_PAY_VALUES:=0.01 0.05 0.1 1}"
fi
# Export so check_*.sh (re-source lib) see the same targets via env
export TALER_DOMAIN BANK_PUBLIC EXCHANGE_PUBLIC MERCHANT_PUBLIC
export EXPECT_CURRENCY SKIP_SSH LOCAL_STACK TALER_DOMAIN_PROBE CHECK_LANDING
export TALER_DOMAIN_FROM_CLI="${TALER_DOMAIN_FROM_CLI:-0}"
export DOMAIN_SET="${DOMAIN_SET:-0}"
export WITHDRAW_AMT PAY_AMT CREDIT_AMT MERCHANT_INSTANCE
export E2E_FAKE_INCOMING E2E_REMOTE E2E_VARIABLE E2E_ATM_MAX
export E2E_WITHDRAW_VALUES E2E_PAY_VALUES E2E_USE_TEMPLATES E2E_TEMPLATE_MAP
export PAIVANA_PUBLIC E2E_PAIVANA E2E_PAIVANA_TEMPLATE E2E_PAIVANA_AMOUNT E2E_PAIVANA_INSTANCE
export INSIDE_PROFILE INSIDE_SSH
export INSIDE_BANK_CTR INSIDE_EXCHANGE_CTR INSIDE_MERCHANT_CTR
export INSIDE_BANK_PORT INSIDE_EXCHANGE_PORT INSIDE_MERCHANT_PORT
export INSIDE_DNS_BANK INSIDE_DNS_EXCHANGE INSIDE_DNS_MERCHANT
# Ladder: withdraw then pay — 0 + random mids + max-1 + max (see check_goa_ladder.sh).
# Defaults so set -u export is safe when vars were never set by caller.
: "${LADDER_STEPS:=23}"
: "${LADDER_MIN_AMOUNT:=0.000001}"
: "${LADDER_CONFIRM_POLLS:=40}"
: "${LADDER_MAX_RUNGS:=99}"
: "${LADDER_TIMEOUT_S:=3600}"
: "${LADDER_REPORT_DIR:=}"
: "${LADDER_SETTLE_ROUNDS:=18}"
: "${LADDER_SETTLE_SLEEP:=2}"
: "${LADDER_MAX_AMOUNT:=4503599627370496}"
: "${LADDER_INCLUDE_ZERO:=1}"
: "${LADDER_INCLUDE_MAX:=1}"
: "${LADDER_HIGH_FROM:=1000000}"
: "${LADDER_HIGH_RUNGS:=12}"
: "${LADDER_LOAD:=1}"
: "${LADDER_PAY:=1}"
: "${LADDER_WITHDRAW_SCALE:=1.5}"
: "${LADDER_PAY_SETTLE_ROUNDS:=6}"
export LADDER_STEPS LADDER_MIN_AMOUNT LADDER_CONFIRM_POLLS LADDER_MAX_RUNGS LADDER_TIMEOUT_S LADDER_REPORT_DIR
export LADDER_SETTLE_ROUNDS LADDER_SETTLE_SLEEP EXP_PW_FILE EXP_USER
export LADDER_MAX_AMOUNT LADDER_INCLUDE_ZERO LADDER_INCLUDE_MAX
export LADDER_HIGH_FROM LADDER_HIGH_RUNGS LADDER_LOAD
export LADDER_PAY LADDER_WITHDRAW_SCALE LADDER_PAY_SETTLE_ROUNDS
export TALER_DOMAIN_APPLIED=1
i18n_init 2>/dev/null || true
# Optional: auth401 and other phases may honor CONTINUE_ON_ERROR
export CONTINUE_ON_ERROR="${CONTINUE_ON_ERROR:-}"
export AUTH401_CONTINUE="${AUTH401_CONTINUE:-${CONTINUE_ON_ERROR:-}}"
# Default phases
# - local GOA (koopa): urls + inside + versions + e2e
# - FrancPaysan STAGE: urls + inside (stagepaysan) + versions + e2e
# - other remote: urls only (pass e2e/inside explicitly)
if [ "${#PHASES[@]}" -eq 0 ]; then
if [ "${LOCAL_STACK}" = "1" ]; then
PHASES=(urls inside versions e2e)
elif [ "${EXPECT_CURRENCY:-}" = "TESTPAYSAN" ] \
|| [[ "${TALER_DOMAIN:-}" == stage.*lefrancpaysan* ]] \
|| [[ "${TALER_DOMAIN:-}" == *stage.lefrancpaysan* ]]; then
# Like hacktivism breadth: public urls + inside (stagepaysan) + versions + e2e
PHASES=(urls inside versions e2e)
else
PHASES=(urls)
fi
fi
OUT_PHASES=()
for p in "${PHASES[@]}"; do
if [ "$p" = "all" ]; then
if [ "${LOCAL_STACK}" = "1" ]; then
OUT_PHASES+=(urls inside versions sanity e2e)
elif [ -n "${INSIDE_SSH:-}" ] || [ "${INSIDE_PROFILE:-}" = "stage-lfp" ]; then
OUT_PHASES+=(urls inside versions e2e)
else
# remote without inside SSH: public + optional e2e
OUT_PHASES+=(urls versions e2e)
fi
elif [ "$p" = "full" ]; then
# Maximum coverage. Long. Needs secrets for e2e/ladder/auth401.
if [ "${LOCAL_STACK}" = "1" ]; then
OUT_PHASES+=(urls inside versions sanity server e2e ladder auth401)
elif [ -n "${INSIDE_SSH:-}" ] || [ "${INSIDE_PROFILE:-}" = "stage-lfp" ]; then
OUT_PHASES+=(urls inside versions e2e ladder auth401)
else
OUT_PHASES+=(urls versions e2e auth401)
fi
else
OUT_PHASES+=("$p")
fi
done
PHASES=()
seen=" "
for p in "${OUT_PHASES[@]}"; do
# server = koopa-only. inside allowed on koopa OR stagepaysan (INSIDE_SSH).
case "$p" in
server)
if [ "${LOCAL_STACK}" != "1" ]; then
echo "[INFO] skip phase 'server' (koopa only)" >&2
continue
fi
;;
inside)
if [ "${LOCAL_STACK}" != "1" ] && [ -z "${INSIDE_SSH:-}" ] && [ "${INSIDE_PROFILE:-}" != "stage-lfp" ]; then
echo "[INFO] skip phase 'inside' (no INSIDE_SSH / not local)" >&2
continue
fi
;;
esac
case "$seen" in *" $p "*) ;; *) PHASES+=("$p"); seen="$seen$p " ;; esac
done
if [ "${#PHASES[@]}" -eq 0 ]; then
PHASES=(urls)
fi
# Global expected check count for progress bar done/total (override: PROGRESS_TOTAL=N).
# Auto estimate is env-aware (CHECK_BANK=0 etc.) and re-fit after each phase so stage
# jobs do not sit at 42/139 then snap to 46/46. Phase summary() must NOT snap mid-run.
# Host-agent monpages post-check is outside this process — never counted here.
_progress_estimate_phase() {
local p="$1" n=0 hosts=1
hosts=$(printf '%s' "${MON_HOSTS:-x}" | wc -w)
[ "$hosts" -lt 1 ] && hosts=1
case "$p" in
urls)
# Core: exchange + merchant + webui + perf + terms (~28–40). Bank/landing optional.
# Stage (CHECK_BANK=0 CHECK_LANDING=0) lands near ~34 numbered lines — keep close.
n=28
[ "${CHECK_BANK:-1}" != "0" ] && n=$((n + 24))
[ "${CHECK_LANDING:-1}" != "0" ] && n=$((n + 18)) # intro + stats.json + qr
n=$((n + 6)) # headroom (SPA / terms / privacy / alt paths)
;;
inside) n=30 ;;
versions) n=35 ;;
sanity) n=35 ;;
server) n=20 ;;
# e2e: ATM ladder emits many INFO lines (coins before/after each note)
e2e) n=240 ;;
ladder|goa-ladder) n=120 ;;
auth401) n=70 ;;
aptdeploy) n=40 ;;
surface|ecosystem) n=120 ;;
monpages|pages)
# slash + bare (+ optional _err) per inventory host; GOA full suite ≈20 URLs
case "${TALER_DOMAIN:-}" in
*hacktivism*) n=24 ;;
*lefrancpaysan*|*francpaysan*) n=$((hosts * 2 + 6)); [ "$n" -lt 12 ] && n=12 ;;
*) n=$((hosts * 2 + 4)); [ "$n" -lt 8 ] && n=8 ;;
esac
[ "$n" -gt 48 ] && n=48
;;
mattermost) n=25 ;;
mail) n=40 ;;
devtesting|franken|fake-franken|fake_franken) n=10 ;;
*) n=15 ;;
esac
printf '%s' "$n"
}
# Recompute PROGRESS_TOTAL = done + estimate(remaining phases). Allows shrink after short phases.
_progress_refit_remaining() {
[ "${PROGRESS_TOTAL_AUTO:-0}" = "1" ] || return 0
local rem=0 rp
for rp in "$@"; do
rem=$((rem + $(_progress_estimate_phase "$rp")))
done
if type _mon_state_load >/dev/null 2>&1; then
_mon_state_load
fi
set_progress_total "$(( ${PROGRESS_DONE:-0} + rem ))"
}
PROGRESS_TOTAL_AUTO=0
if [ "${PROGRESS_TOTAL:-0}" = "0" ] || [ -z "${PROGRESS_TOTAL:-}" ]; then
PROGRESS_TOTAL_AUTO=1
_pt=0
for p in "${PHASES[@]}"; do
_pt=$((_pt + $(_progress_estimate_phase "$p")))
done
set_progress_total "$_pt"
unset _pt
fi
export PROGRESS_TOTAL_AUTO
# Whole-run wall clock (default 10 min). Override: RUN_TIMEOUT=0 (unlimited).
: "${RUN_TIMEOUT:=600}"
export RUN_TIMEOUT
MON_T0=$(date +%s)
export MON_T0
RUN_TIMED_OUT=0
RUN_TIMEOUT_AT_PHASE=""
RUN_SKIPPED_PHASES=()
mon_seconds_left() {
if [ "${RUN_TIMEOUT:-0}" -eq 0 ]; then
printf '%s' "999999"
return 0
fi
local now left
now=$(date +%s)
left=$((RUN_TIMEOUT - (now - MON_T0)))
[ "$left" -lt 0 ] && left=0
printf '%s' "$left"
}
# Run one phase script under remaining RUN_TIMEOUT budget.
# Exit 124 (timeout utility) → mark RUN_TIMED_OUT.
run_phase() {
local phase="$1" script="$2" left rc
left=$(mon_seconds_left)
if [ "${RUN_TIMEOUT:-0}" -gt 0 ] && [ "$left" -le 0 ]; then
RUN_TIMED_OUT=1
RUN_TIMEOUT_AT_PHASE="${RUN_TIMEOUT_AT_PHASE:-$phase}"
return 1
fi
if [ "${RUN_TIMEOUT:-0}" -eq 0 ]; then
"$script"
return $?
fi
# Keep at least 2s so timeout(1) can start the child.
[ "$left" -lt 2 ] && left=2
set +e
with_timeout "$left" "$script"
rc=$?
set -e
if [ "$rc" -eq 124 ]; then
RUN_TIMED_OUT=1
RUN_TIMEOUT_AT_PHASE="$phase"
return 1
fi
return "$rc"
}
# How inside/versions will reach containers (host-podman vs ssh)
_INSIDE_ACCESS_HINT=ssh
if [ "${INSIDE_PODMAN:-0}" = "1" ] || [ "${INSIDE_MODE:-}" = "local-podman" ]; then
_INSIDE_ACCESS_HINT=host-podman
elif [ "${SKIP_SSH:-0}" = "1" ]; then
_INSIDE_ACCESS_HINT=skipped
elif [ "${LOCAL_STACK:-0}" = "1" ] && command -v podman >/dev/null 2>&1 \
&& podman ps --format '{{.Names}}' 2>/dev/null | grep -qE 'taler-hacktivism'; then
_INSIDE_ACCESS_HINT=host-podman
fi
printf 'target domain=%s\n' "${TALER_DOMAIN}"
printf ' bank %s\n' "$BANK_PUBLIC"
printf ' exchange %s\n' "$EXCHANGE_PUBLIC"
printf ' merchant %s\n' "$MERCHANT_PUBLIC"
printf ' currency expect=%s\n' "${EXPECT_CURRENCY:-any}"
printf ' phases %s\n' "${PHASES[*]}"
printf ' flags LOCAL_STACK=%s SKIP_SSH=%s INSIDE_PODMAN=%s INSIDE_MODE=%s\n' \
"${LOCAL_STACK:-}" "${SKIP_SSH:-0}" "${INSIDE_PODMAN:-0}" "${INSIDE_MODE:-}"
printf ' flags KOOPA_SSH=%s INSIDE_SSH=%s INSIDE_PROFILE=%s\n' \
"${KOOPA_SSH:-}" "${INSIDE_SSH:-}" "${INSIDE_PROFILE:-}"
printf ' access inside/versions → %s' "$_INSIDE_ACCESS_HINT"
case "$_INSIDE_ACCESS_HINT" in
host-podman) printf ' (podman exec on this host; IDs inside.host-*)\n' ;;
ssh) printf ' (SSH then podman; IDs inside.ssh-*)\n' ;;
skipped) printf ' (SKIP_SSH=1)\n' ;;
*) printf '\n' ;;
esac
if [ "${RUN_TIMEOUT:-0}" -eq 0 ]; then
printf ' run_timeout unlimited (RUN_TIMEOUT=0)\n'
else
printf ' run_timeout %ss wall clock (RUN_TIMEOUT=; 0=unlimited)\n' "$RUN_TIMEOUT"
fi
printf ' progress %s/%s (global done/total · set PROGRESS_TOTAL= to override; PROGRESS_OFF=1 to hide)\n' \
"${PROGRESS_DONE:-0}" "${PROGRESS_TOTAL:-0}"
unset _INSIDE_ACCESS_HINT
chmod +x "$ROOT"/check_*.sh 2>/dev/null || true
ec=0
_phase_idx=0
_n_phases=${#PHASES[@]}
# Suppress all phase SUMMARY / progress standings when more than one phase;
# parent prints one global SUMMARY after progress_finish (v1.13.10+).
if [ "$_n_phases" -gt 1 ]; then
export TALER_MON_MULTI_PHASE=1
else
export TALER_MON_MULTI_PHASE=0
fi
_MON_WAS_MULTI="${TALER_MON_MULTI_PHASE}"
while [ "$_phase_idx" -lt "$_n_phases" ]; do
p="${PHASES[$_phase_idx]}"
if [ "$RUN_TIMED_OUT" = "1" ]; then
RUN_SKIPPED_PHASES+=("$p")
_phase_idx=$((_phase_idx + 1))
continue
fi
left=$(mon_seconds_left)
if [ "${RUN_TIMEOUT:-0}" -gt 0 ] && [ "$left" -le 0 ]; then
RUN_TIMED_OUT=1
RUN_TIMEOUT_AT_PHASE="${RUN_TIMEOUT_AT_PHASE:-$p}"
RUN_SKIPPED_PHASES+=("$p")
_phase_idx=$((_phase_idx + 1))
continue
fi
case "$p" in
urls) run_phase urls "$ROOT/check_urls.sh" || ec=1 ;;
inside) run_phase inside "$ROOT/check_inside.sh" || ec=1 ;;
versions) run_phase versions "$ROOT/check_versions.sh" || ec=1 ;;
sanity) run_phase sanity "$ROOT/check_sanity.sh" || ec=1 ;;
server) run_phase server "$ROOT/check_server.sh" || ec=1 ;;
e2e) run_phase e2e "$ROOT/check_e2e.sh" || ec=1 ;;
ladder|goa-ladder) run_phase ladder "$ROOT/check_goa_ladder.sh" || ec=1 ;;
auth401) run_phase auth401 "$ROOT/check_auth401.sh" || ec=1 ;;
aptdeploy|apt-deploy|apt_src) run_phase aptdeploy "$ROOT/check_apt_deploy.sh" || ec=1 ;;
surface|ecosystem) run_phase surface "$ROOT/check_surface.sh" || ec=1 ;;
mattermost) run_phase mattermost "$ROOT/check_mattermost.sh" || ec=1 ;;
mail) run_phase mail "$ROOT/check_mail.sh" || ec=1 ;;
monpages|pages) run_phase monpages "$ROOT/check_monitoring_pages.sh" || ec=1 ;;
devtesting|franken|fake-franken|fake_franken)
run_phase devtesting "$ROOT/check_devtesting.sh" || ec=1
;;
esac
# After each phase: refit total = actual done + estimate(remaining). Fixes
# stage over-estimate (e.g. 42/139 while only ~4 checks left).
if [ "${PROGRESS_TOTAL_AUTO:-0}" = "1" ]; then
_rem_args=()
_j=$((_phase_idx + 1))
while [ "$_j" -lt "$_n_phases" ]; do
_rem_args+=("${PHASES[$_j]}")
_j=$((_j + 1))
done
_progress_refit_remaining "${_rem_args[@]+"${_rem_args[@]}"}"
unset _rem_args _j
fi
_phase_idx=$((_phase_idx + 1))
done
unset _phase_idx _n_phases
unset PROGRESS_FINAL TALER_MON_MULTI_PHASE 2>/dev/null || true
# Extraordinary run-budget failure: always report at end; HTML links top → here.
if [ "$RUN_TIMED_OUT" = "1" ]; then
ec=1
elapsed=$(( $(date +%s) - MON_T0 ))
skipped="${RUN_SKIPPED_PHASES[*]:-}"
# Stable id for jump links (console HTML + err-top banner)
printf '\n'
printf '╔══════════════════════════════════════════════════════════╗\n'
printf '║ RUN TIMEOUT · extraordinary (see jump target below) ║\n'
printf '╚══════════════════════════════════════════════════════════╝\n'
printf '┌ ERROR ┐ #run.timeout-01 RUN_TIMEOUT exceeded · budget %ss · elapsed %ss\n' \
"$RUN_TIMEOUT" "$elapsed"
printf ' detail: wall-clock limit hit'
if [ -n "${RUN_TIMEOUT_AT_PHASE:-}" ]; then
printf ' during/after phase "%s"' "$RUN_TIMEOUT_AT_PHASE"
fi
if [ -n "$skipped" ]; then
printf ' · skipped: %s' "$skipped"
fi
printf '\n'
printf ' id=run.timeout-01\n'
printf ' hint: raise RUN_TIMEOUT= (seconds) or set RUN_TIMEOUT=0 for unlimited\n'
printf -- '┌ RUN TIMEOUT · extraordinary ┐\n'
printf -- ' • run.timeout-01 [run] RUN_TIMEOUT=%ss exceeded (elapsed %ss)%s%s\n' \
"$RUN_TIMEOUT" "$elapsed" \
"${RUN_TIMEOUT_AT_PHASE:+ · phase $RUN_TIMEOUT_AT_PHASE}" \
"${skipped:+ · skipped $skipped}"
printf -- '--- ERRORS (failed checks) ---\n'
printf -- ' • run.timeout-01 [run] RUN_TIMEOUT=%ss exceeded (elapsed %ss)\n' \
"$RUN_TIMEOUT" "$elapsed"
fi
# Snap progress bar to real global count (done/total) once for the whole run.
# Reload TALER_MON_STATE — check_*.sh update done/total + GLOBAL_*_N in a separate process.
if type _mon_state_load >/dev/null 2>&1; then
_mon_state_load
fi
if type progress_finish >/dev/null 2>&1; then
printf '\n'
progress_finish
fi
# Multi-phase: one global SUMMARY (OK/ERROR/WARN/INFO/BLOCK across all phases).
# Single-phase already printed phase-local SUMMARY via check_*.sh.
if [ "${_MON_WAS_MULTI:-0}" = "1" ] && type summary_global >/dev/null 2>&1; then
summary_global || true
fi
unset _MON_WAS_MULTI 2>/dev/null || true
exit "$ec"