diff --git a/2026/2026-07-16--taler-package-upgrade-goa.md b/2026/2026-07-16--taler-package-upgrade-goa.md new file mode 100644 index 0000000..91a412f --- /dev/null +++ b/2026/2026-07-16--taler-package-upgrade-goa.md @@ -0,0 +1,145 @@ +# 2026-07-16 — GOA container package upgrade (trixie debs) + +## Why + +`taler-monitoring.sh versions` reported packages **behind** `deb.taler.net` **trixie**: + +| Component | Package | Was | Target (trixie index) | +|-----------|---------|-----|------------------------| +| bank | libeufin-bank / common | 1.6.6 | **1.6.7** | +| exchange | taler-exchange* / libtalerexchange | 1.6.6 | **1.6.7~dev2** | +| merchant | taler-merchant-webui | 1.6.9 | **1.6.11** | +| merchant | taler-merchant / libtalermerchant | 1.6.9 | still 1.6.9 (no newer in suite for those) | + +Host is openSUSE Tumbleweed — **irrelevant** for these versions. Debs live **inside** Debian-based podman images. + +## Containers + +| Role | Name | +|------|------| +| bank | `taler-hacktivism-bank` | +| exchange | `taler-hacktivism-exchange-ansible` | +| merchant | `taler-hacktivism` | + +**No systemd as PID 1** in these containers. Do **not** rely on `systemctl restart` after apt (policy-rc.d / no bus). Restart with: + +```bash +# bank +podman exec -u root taler-hacktivism-bank \ + runuser -u libeufin-bank -- /usr/local/bin/start_bank.sh --restart + +# merchant +podman exec -u root taler-hacktivism \ + runuser -u taler-merchant-httpd -- /usr/local/bin/start_merchant.sh --restart + +# exchange: base (root) then start_exchange as httpd user +podman exec -u root taler-hacktivism-exchange-ansible \ + bash -c '/root/start_base_services_for_taler_exchange.sh --no-shell 2>/dev/null; \ + runuser -u taler-exchange-httpd -- /usr/local/bin/start_exchange.sh --restart' +``` + +## Procedure (manual) + +```bash +# as root inside each container (example bank) +export DEBIAN_FRONTEND=noninteractive +apt-get update -qq +apt-get install -y libeufin-bank libeufin-common # bank +# exchange: taler-exchange taler-exchange-database libtalerexchange … +# merchant: taler-merchant-webui (and stack as needed) +``` + +Then restart with `start_*.sh` as above. + +**Scripted (host):** `scripts/taler-shared/upgrade-goa-debs.sh` +Copy to koopa or run from a checkout: `./upgrade-goa-debs.sh` / `bank|exchange|merchant`. + +## Lessons from 2026-07-16 run + +1. **`apt-get install` as root works** (`podman exec -u root`); suite `trixie` (+ merchant may also have `trixie-testing`). +2. **`taler-exchange-dbinit` as root fails** (`role "root" does not exist`) — run as `taler-exchange-httpd` if needed. +3. **Bank after upgrade:** if postgres socket was down, bank dies with pool init error; `pg_ctlcluster 17 main start` then `start_bank.sh --restart`. +4. **Merchant `taler-merchant-dbinit` / merchant-0041.sql** can noise-fail (psql cluster path); after `start_merchant.sh --restart`, health check can still be green — verify `https://127.0.0.1:9010/config` and public `taler.hacktivism.ch`. +5. Exchange **postinst** may warn about missing SPA files under `/usr/share/taler-exchange/{aml,kyc}-spa/`; packages `taler-exchange-aml-webui` / `kyc-webui` pull in SPAs — re-check if AML UI is used. +6. **Images are live-writable** (not immutable rebuild): upgraded debs are in the running container layers until next image rebuild/snapshot. + +## Smoke after upgrade + +```bash +curl -sS -o /dev/null -w "%{http_code}\n" http://127.0.0.1:9012/config +curl -sS -o /dev/null -w "%{http_code}\n" http://127.0.0.1:9011/config +curl -skS -o /dev/null -w "%{http_code}\n" https://127.0.0.1:9010/config +# public +curl -skS -o /dev/null -w "%{http_code}\n" https://bank.hacktivism.ch/config +curl -skS -o /dev/null -w "%{http_code}\n" https://exchange.hacktivism.ch/config +curl -skS -o /dev/null -w "%{http_code}\n" https://taler.hacktivism.ch/config +``` + +Laptop: `./scripts/taler-monitoring/taler-monitoring.sh versions` (or `urls`). + +## Result (this day) + +- bank **1.6.7**, exchange **1.6.7~dev2**, merchant-webui **1.6.11** +- public `/config` **200** for bank / exchange / merchant after restarts +- `taler-monitoring.sh -d hacktivism.ch versions` → **behind=0** (all package rows OK) + +## Specific residual problems (re-checked after upgrade) + +Severity: **P1** = breaks settlement/ops soon · **P2** = noise / tooling · **P3** = hygiene. + +### P1 — none known after restarts + +Settlement path: bank/exchange/merchant `/config` 200; merchant health can pass including wirewatch (see below). + +### P2 — `taler-merchant-dbinit` fails when run by hand + +```text +WARNING Could not run PSQL on file …/global_procedures.sql: psql exit code was 1 +ERROR Failed to initialize tables +``` + +- **Schema is actually current:** `_v.patches` includes **`merchant-0041`** (applied 2026-07-16, by role `postgres` during upgrade noise). +- **Service works:** `check_merchant-health.sh` → ALL CRITICAL CHECKS PASSED when helpers are up. +- **Cause (likely):** service-user `psql` / cluster path (`Error: Invalid data directory for cluster 17 main` when run poorly), not missing migrations. +- **Action:** do not treat hand-run `dbinit` exit≠0 as deploy failure; verify `_v.patches` + `/config`. Optional follow-up: fix `pg_wrapper`/`.postgresqlrc` for `taler-merchant-httpd` so dbinit is clean. + +### P2 — merchant health vs wirewatch (race / detect) + +- `check_merchant-health.sh` uses `pgrep` for `taler-merchant-wirewatch`. +- Wirewatch is supervised by **`taler-merchant-wirewatch-supervise.sh`** (restarts on PG NOTIFY exit). +- During restart windows the helper can be absent for seconds → **false FAIL**. +- When wirewatch + supervise are both live, health reports **OK**. +- **Action:** re-run health after 5s; ensure supervise is started after package upgrades (start_merchant / ensure_helpers). + +### P2 — `check_exchange-health.sh` missing in live exchange container + +```text +exec: "/usr/local/bin/check_exchange-health.sh": no such file or directory +``` + +- Script exists in **admin-log** (`scripts/taler-exchange/check_exchange-health.sh`) but was **not installed** into `taler-hacktivism-exchange-ansible`. +- **Action:** copy into image/live `/usr/local/bin/` on next deploy (same as bank/merchant health scripts). + +### P3 — zombie processes inside bank + merchant containers + +- **bank:** defunct `java`, `python3`, occasional `postgres` / `dpkg-preconfigu`. +- **merchant:** many old `taler-merchant-*` defunct (pre-restart leftovers). +- **Cause:** no proper init/reaper (not systemd PID 1); supervise/start scripts leave zombies. +- **Impact:** mostly cosmetic / PID table clutter unless extreme. +- **Action:** periodic container restart or install a tiny reaper; not urgent. + +### P3 — package skew (informational) + +| Package | Installed | Note | +|---------|-----------|------| +| taler-merchant / libtalermerchant / typst | 1.6.9 | no newer in trixie index at check time | +| taler-merchant-webui | 1.6.11 | intentionally newer SPA | +| exchange aml/kyc webui | 1.6.8~dev3 | pulled with exchange upgrade | + +Monitoring does **not** ERROR on this skew when suite index matches installed. + +### Resolved during upgrade (do not re-open without evidence) + +- Version **behind** bank/exchange/webui — fixed. +- Exchange postinst “missing aml-spa/forms.json” — paths **`/usr/share/taler-exchange/{aml,kyc}-spa`** now present after webui packages. +- Bank down after apt — fixed with postgres socket + `start_bank.sh --restart`. diff --git a/scripts/README.md b/scripts/README.md index 34ef2bc..341260a 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -7,6 +7,7 @@ | `taler-hacktivism-bank/` | podman `taler-hacktivism-bank`: `/root`, `/usr/local/bin` | | `taler-sanity/` | host root checks (stack, settlement, helpers) | | `taler-monitoring/` | **outside-in** public URL walk (`/config` → keys/terms/integration/webui) | +| `taler-shared/` | shared helpers (`upgrade-goa-debs.sh` = apt upgrade **inside** bank/exchange/merchant containers) | | `monitoring/` | host `/home/hernani/scripts` (tor relay stats) | | `taler-wallet-cli/` | thin wrappers; **benchmarks live in** `../benchmarks/` | | `castopod/` | host `hernani` podman-compose `~/koopa-castopod` — see `castopod/README.md` | @@ -44,4 +45,8 @@ Exchange one-shots (root, offline / wire): `runuser -u USER -- bash` (never `-u` with `-s` on util-linux). +### Package upgrades (trixie debs inside containers) + +Containers are **not** systemd. After `apt-get install` of Taler/libeufin packages, restart with `start_*.sh` only — see **`taler-shared/upgrade-goa-debs.sh`** and day note `2026/2026-07-16--taler-package-upgrade-goa.md`. Host `zypper` does not update these. + SMS helper symlinks into `/var/taler-src/...` are not copied (merchant only). diff --git a/scripts/taler-monitoring/README.md b/scripts/taler-monitoring/README.md index 35edc67..170bf46 100644 --- a/scripts/taler-monitoring/README.md +++ b/scripts/taler-monitoring/README.md @@ -33,16 +33,67 @@ End of each phase: totals + list of **BLOCKERS** and **ERRORS**. # Other domains — public HTTPS only, never SSH ./taler-monitoring.sh -d taler.net ./taler-monitoring.sh --domain taler-ops.ch +./taler-monitoring.sh -d my.taler-ops.ch urls ./taler-monitoring.sh -d demo.taler.net urls ./taler-monitoring.sh taler.net # bare domain = same as -d + +# Explicit endpoints (any mix; override profile or skip -d) +./taler-monitoring.sh --bank https://bank.demo.taler.net \ + --exchange https://exchange.demo.taler.net \ + --merchant https://backend.demo.taler.net --currency KUDOS urls ``` -| Domain | Bank | Exchange | Merchant | Currency | -|--------|------|----------|----------|----------| -| `hacktivism.ch` (default) | bank.hacktivism.ch | exchange.hacktivism.ch | taler.hacktivism.ch | GOA | +## Domain profiles (`domains.conf`) + +Each monitored stack must say **what is bank, exchange, and merchant-backend**. + +Edit **[domains.conf](./domains.conf)** (or set `TALER_DOMAINS_CONF=`): + +```text +# name bank exchange merchant-backend currency local [canonical] +taler-ops.ch bank.taler-ops.ch exchange.taler-ops.ch my.taler-ops.ch CHF 0 +``` + +| Field | Meaning | +|-------|---------| +| **name** | `-d NAME` / `TALER_DOMAIN` match (add alias lines as needed) | +| **bank** | libeufin/bank public host or `https://…` | +| **exchange** | exchange public host or URL | +| **merchant** | merchant **backend** (SPA/API), e.g. `my.taler-ops.ch` | +| **currency** | `GOA` / `KUDOS` / `CHF` / … | +| **local** | `1` = koopa SSH stack; `0` = public only | +| **landing** | `1` = check `/intro` landings + assets; `0` = skip (TOPS / mytops) | +| **canonical** | optional label after alias (e.g. `my.taler-ops.ch` → `taler-ops.ch`) | + +Then: + +```bash +./taler-monitoring.sh -d taler-ops.ch urls +``` + +Overrides (always win after profile load): + +```bash +./taler-monitoring.sh -d taler-ops.ch \ + --merchant https://my.taler-ops.ch \ + --exchange https://exchange.taler-ops.ch \ + --bank https://bank.taler-ops.ch urls +``` + +Helpers in `lib.sh`: `set_taler_stack`, `load_domain_profile`, `apply_taler_domain`. + +### Built-in profiles (see `domains.conf`) + +| Domain / alias | Bank | Exchange | Merchant | Currency | +|----------------|------|----------|----------|----------| +| `hacktivism.ch` / `koopa` | bank.hacktivism.ch | exchange.hacktivism.ch | taler.hacktivism.ch | GOA | | `taler.net` / `demo.taler.net` | bank.demo.taler.net | exchange.demo.taler.net | backend.demo.taler.net | KUDOS | -| `taler-ops.ch` | bank.* (probe) | exchange.taler-ops.ch | backend.* (probe) | CHF | -| other | bank.DOMAIN | exchange.DOMAIN | backend/taler/merchant (probe) | any | +| `test.taler.net` | bank.test… | exchange.test… | backend.test… | **TESTKUDOS** · no landings | +| `taler-ops.ch` / `my.taler-ops.ch` | bank.taler-ops.ch | exchange.taler-ops.ch | **my.taler-ops.ch** | CHF · **no landings** | +| `stage.taler-ops.ch` | bank.stage… | exchange.stage… | **my.stage…** | CHF · **no landings** | +| unknown | bank.DOMAIN | exchange.DOMAIN | my.DOMAIN then probe | any · no landings | + +**Landings:** only GOA (and optionally demo) use public `/intro` pages. **taler-ops.ch does not** — `CHECK_LANDING=0` in the profile skips intro crawl, shop assets, and demo-withdraw checks. **SSH only for koopa** (`hacktivism.ch` / `-d koopa`). @@ -52,8 +103,9 @@ Other domains: never SSH. Optional **e2e** aborts cleanly on login/KYC. | | Local (koopa) | Remote | |--|---------------|--------| -| **ATM withdraw** | 20 · 50 · 100 · 200 | 10 · 20 · 50 | +| **ATM withdraw** | 20 · 50 · 100 · 200 · **4200** (paivana) | 10 · 20 · 50 | | **Pay ladder** | 0.01 … 10 | 0.01 … 1 | +| **Paivana** | HTTP paywall + template pay **GOA:4200** (`goa-shop` / `paivana`) | skipped | ```bash ./taler-monitoring.sh e2e @@ -66,6 +118,10 @@ E2E_VARIABLE=0 WITHDRAW_AMT=GOA:50 PAY_AMT=GOA:1 ./taler-monitoring.sh e2e # s # E2E_SHOP_PRODUCTS lines: id|Product name|GOA:amount # E2E_SHOP_PICK_N=2 # (landing QR = taler://pay-template/…/{id}; popup = live taler://pay after POST templates/{id}) +# Paivana paywall (local GOA only): +# E2E_PAIVANA=1 (default) PAIVANA_PUBLIC=https://paivana.hacktivism.ch +# E2E_PAIVANA_TEMPLATE=paivana E2E_PAIVANA_AMOUNT=GOA:4200 E2E_PAIVANA_INSTANCE=goa-shop +# E2E_PAIVANA=0 # skip # remote secrets: # E2E_BANK_ADMIN_PASS=… E2E_MERCHANT_TOKEN=… ``` diff --git a/scripts/taler-monitoring/TESTS.md b/scripts/taler-monitoring/TESTS.md index 51686d4..2774dac 100644 --- a/scripts/taler-monitoring/TESTS.md +++ b/scripts/taler-monitoring/TESTS.md @@ -43,12 +43,13 @@ IDs are assigned **in run order** within the area (`set_area` resets the counter | www-… | **merchant `/terms`** body (dual-currency notice) | | www-… | **merchant `/privacy`** body (must not be `not configured`) | | www-… | merchant `/terms/` redirect | -| www-… | **landing exposed links** (bank / merchant / exchange): parse each `/intro/` HTML, probe every own-stack `https://` + root-relative `href`/`src`/`content`, soft-check external stores/docs | -| www-… | landing static: `qrcode.min.js`, `og-goa-shop.png`, `qr-logo.png`, shop-pay.js/css | -| www-… | cross-links between bank ↔ merchant ↔ exchange intros (local stack) | +| www-… | **landing (aggregated per site)**: one line for bank/merchant/exchange (`/intro` + assets + own/external link counts); failures listed as samples; soft external/shop assets compact | +| www-… | cross-links: one line (4 intros) on local stack | +| www-… | bank demo-withdraw + shop assets: one line (+ soft shop-pay if missing) | | www-… | **bank `/intro/demo-withdraw.json`** → `taler://withdraw/HOST:PORT/taler-integration/…` + integration op HTTP 200 | | www-… | bank `/intro/auto-account.json` (earlier) → same withdraw shape, **no payto_uri**, login at `/webui/` | -| www-… | **performance** (outside-in): public HTTPS RTT for bank `/config`, `/taler-integration/config`, `/webui/`, `/intro/`, `stats.json`; exchange `/config`, `/keys`, `/intro/`; merchant `/config`, `/webui/`, `/intro/` — report ms; WARN ≥ `PERF_WARN_MS` (default 8000); **ERROR ≥ `PERF_FAIL_MS` (default 20000)** | +| www-… | **performance** (outside-in RTT): bank/exchange/merchant `/config`, keys, webui, intro — ms; WARN ≥ `PERF_WARN_MS` (8000); ERROR ≥ `PERF_FAIL_MS` (20000) | +| www-… | **landing load stats**: `/intro/stats.json` → loadavg, container RSS, in-container probe ms (public; SSH container cat fallback); warn if stale/missing | **Legal docs rule:** HTTP 200, non-empty body, not plain `not configured`, not merchant API JSON `code:21`. On local stack, optional content needle (terms/privacy/FADP/GOA…). @@ -135,6 +136,8 @@ Without SSH (`SKIP_SSH=1` or remote domain): still runs outside-in repo checks; ## e2e — payment path (`./taler-monitoring.sh e2e`) +Local GOA also: **ATM includes GOA:4200**, then **paivana** (HTTP 302 on `PAIVANA_PUBLIC` + public template pay `paivana` / GOA:4200 on `goa-shop`). Disable with `E2E_PAIVANA=0`. + | ID | Step (approx.) | |----|----------------| | e2e-001 | budget info | diff --git a/scripts/taler-shared/upgrade-goa-debs.sh b/scripts/taler-shared/upgrade-goa-debs.sh new file mode 100755 index 0000000..0022510 --- /dev/null +++ b/scripts/taler-shared/upgrade-goa-debs.sh @@ -0,0 +1,126 @@ +#!/usr/bin/env bash +# Upgrade GNU Taler .deb packages *inside* the three GOA containers on koopa. +# Host openSUSE packages are unrelated (zypper) — do not confuse with this. +# +# Containers (no systemd as PID 1): after apt, restart with start_*.sh only. +# +# Run on koopa host as a user that can `podman exec` the containers: +# ./upgrade-goa-debs.sh +# ./upgrade-goa-debs.sh bank|exchange|merchant +# +# Mirror: scripts/taler-shared/ in koopa-admin-log. +# Live note: 2026/2026-07-16--taler-package-upgrade-goa.md +set -euo pipefail + +BANK_CTR="${BANK_CTR:-taler-hacktivism-bank}" +EXCHANGE_CTR="${EXCHANGE_CTR:-taler-hacktivism-exchange-ansible}" +MERCHANT_CTR="${MERCHANT_CTR:-taler-hacktivism}" + +only="${1:-all}" + +pod_root() { + local ctr="$1" + shift + podman exec -u root "$ctr" bash -c "$*" +} + +upgrade_bank() { + echo "======== bank ($BANK_CTR) ========" + pod_root "$BANK_CTR" ' + set -e + export DEBIAN_FRONTEND=noninteractive + apt-get update -qq + apt-get install -y libeufin-bank libeufin-common + dpkg -l libeufin-bank libeufin-common | awk "/^ii/{print \$2,\$3}" + ' + # restart app only (postgres via start_base if needed) + pod_root "$BANK_CTR" ' + set -e + if ! runuser -u postgres -- psql -c "SELECT 1" >/dev/null 2>&1; then + pg_ctlcluster 17 main start || true + sleep 2 + fi + mkdir -p /var/log/libeufin-bank + chown libeufin-bank:libeufin-bank /var/log/libeufin-bank 2>/dev/null || true + runuser -u libeufin-bank -- /usr/local/bin/start_bank.sh --restart + ' +} + +upgrade_exchange() { + echo "======== exchange ($EXCHANGE_CTR) ========" + pod_root "$EXCHANGE_CTR" ' + set -e + export DEBIAN_FRONTEND=noninteractive + apt-get update -qq + apt-get install -y \ + taler-exchange taler-exchange-database taler-exchange-offline \ + taler-exchange-typst libtalerexchange taler-terms-generator \ + taler-exchange-aml-webui taler-exchange-kyc-webui || true + dpkg -l "taler-exchange*" "libtalerexchange" "taler-terms-generator" 2>/dev/null \ + | awk "/^ii/{print \$2,\$3}" + ' + # do not run dbinit as root (role "root" does not exist) + pod_root "$EXCHANGE_CTR" ' + set -e + if id taler-exchange-httpd >/dev/null 2>&1; then + runuser -u taler-exchange-httpd -- taler-exchange-dbinit 2>&1 | tail -20 || true + fi + if [ -x /root/start_base_services_for_taler_exchange.sh ]; then + /root/start_base_services_for_taler_exchange.sh --no-shell 2>/dev/null || true + fi + if [ -x /usr/local/bin/start_exchange.sh ]; then + runuser -u taler-exchange-httpd -- /usr/local/bin/start_exchange.sh --restart 2>&1 | tail -30 || true + fi + pgrep -a taler-exchange-httpd | head -3 || true + ' +} + +upgrade_merchant() { + echo "======== merchant ($MERCHANT_CTR) ========" + pod_root "$MERCHANT_CTR" ' + set -e + export DEBIAN_FRONTEND=noninteractive + apt-get update -qq + apt-get install -y \ + taler-merchant taler-merchant-webui taler-merchant-typst \ + libtalermerchant taler-terms-generator + dpkg -l "taler-merchant*" "libtalermerchant" 2>/dev/null | awk "/^ii/{print \$2,\$3}" + ' + pod_root "$MERCHANT_CTR" ' + set -e + # schema: prefer tools as service user; failures may be noisy — check /config after + if id taler-merchant-httpd >/dev/null 2>&1; then + runuser -u taler-merchant-httpd -- taler-merchant-dbinit 2>&1 | tail -20 || true + fi + runuser -u taler-merchant-httpd -- /usr/local/bin/start_merchant.sh --restart + ' +} + +smoke() { + echo "======== smoke (host loopback) ========" + curl -sS -m 8 -o /dev/null -w "bank %{http_code} :9012/config\n" http://127.0.0.1:9012/config || true + curl -sS -m 8 -o /dev/null -w "exchange %{http_code} :9011/config\n" http://127.0.0.1:9011/config || true + curl -skS -m 8 -o /dev/null -w "merchant %{http_code} :9010/config\n" https://127.0.0.1:9010/config || true +} + +case "$only" in + all) + upgrade_bank + upgrade_exchange + upgrade_merchant + smoke + ;; + bank) upgrade_bank; smoke ;; + exchange) upgrade_exchange; smoke ;; + merchant) upgrade_merchant; smoke ;; + -h|--help|help) + sed -n '2,20p' "$0" + exit 0 + ;; + *) + echo "usage: $0 [all|bank|exchange|merchant]" >&2 + exit 2 + ;; +esac + +echo "DONE — re-run taler-monitoring versions/urls from a laptop if desired."