docs: GOA deb upgrades + monitoring notes

2026/2026-07-16--taler-package-upgrade-goa.md

@@ -0,0 +1,145 @@
+# 2026-07-16 — GOA container package upgrade (trixie debs)
+
+## Why
+
+`taler-monitoring.sh versions` reported packages **behind** `deb.taler.net` **trixie**:
+
+| Component | Package | Was | Target (trixie index) |
+|-----------|---------|-----|------------------------|
+| bank | libeufin-bank / common | 1.6.6 | **1.6.7** |
+| exchange | taler-exchange* / libtalerexchange | 1.6.6 | **1.6.7~dev2** |
+| merchant | taler-merchant-webui | 1.6.9 | **1.6.11** |
+| merchant | taler-merchant / libtalermerchant | 1.6.9 | still 1.6.9 (no newer in suite for those) |
+
+Host is openSUSE Tumbleweed — **irrelevant** for these versions. Debs live **inside** Debian-based podman images.
+
+## Containers
+
+| Role | Name |
+|------|------|
+| bank | `taler-hacktivism-bank` |
+| exchange | `taler-hacktivism-exchange-ansible` |
+| merchant | `taler-hacktivism` |
+
+**No systemd as PID 1** in these containers. Do **not** rely on `systemctl restart` after apt (policy-rc.d / no bus). Restart with:
+
+```bash
+# bank
+podman exec -u root taler-hacktivism-bank \
+  runuser -u libeufin-bank -- /usr/local/bin/start_bank.sh --restart
+
+# merchant
+podman exec -u root taler-hacktivism \
+  runuser -u taler-merchant-httpd -- /usr/local/bin/start_merchant.sh --restart
+
+# exchange: base (root) then start_exchange as httpd user
+podman exec -u root taler-hacktivism-exchange-ansible \
+  bash -c '/root/start_base_services_for_taler_exchange.sh --no-shell 2>/dev/null; \
+    runuser -u taler-exchange-httpd -- /usr/local/bin/start_exchange.sh --restart'
+```
+
+## Procedure (manual)
+
+```bash
+# as root inside each container (example bank)
+export DEBIAN_FRONTEND=noninteractive
+apt-get update -qq
+apt-get install -y libeufin-bank libeufin-common   # bank
+# exchange: taler-exchange taler-exchange-database libtalerexchange …
+# merchant: taler-merchant-webui (and stack as needed)
+```
+
+Then restart with `start_*.sh` as above.
+
+**Scripted (host):** `scripts/taler-shared/upgrade-goa-debs.sh`  
+Copy to koopa or run from a checkout: `./upgrade-goa-debs.sh` / `bank|exchange|merchant`.
+
+## Lessons from 2026-07-16 run
+
+1. **`apt-get install` as root works** (`podman exec -u root`); suite `trixie` (+ merchant may also have `trixie-testing`).
+2. **`taler-exchange-dbinit` as root fails** (`role "root" does not exist`) — run as `taler-exchange-httpd` if needed.
+3. **Bank after upgrade:** if postgres socket was down, bank dies with pool init error; `pg_ctlcluster 17 main start` then `start_bank.sh --restart`.
+4. **Merchant `taler-merchant-dbinit` / merchant-0041.sql** can noise-fail (psql cluster path); after `start_merchant.sh --restart`, health check can still be green — verify `https://127.0.0.1:9010/config` and public `taler.hacktivism.ch`.
+5. Exchange **postinst** may warn about missing SPA files under `/usr/share/taler-exchange/{aml,kyc}-spa/`; packages `taler-exchange-aml-webui` / `kyc-webui` pull in SPAs — re-check if AML UI is used.
+6. **Images are live-writable** (not immutable rebuild): upgraded debs are in the running container layers until next image rebuild/snapshot.
+
+## Smoke after upgrade
+
+```bash
+curl -sS -o /dev/null -w "%{http_code}\n" http://127.0.0.1:9012/config
+curl -sS -o /dev/null -w "%{http_code}\n" http://127.0.0.1:9011/config
+curl -skS -o /dev/null -w "%{http_code}\n" https://127.0.0.1:9010/config
+# public
+curl -skS -o /dev/null -w "%{http_code}\n" https://bank.hacktivism.ch/config
+curl -skS -o /dev/null -w "%{http_code}\n" https://exchange.hacktivism.ch/config
+curl -skS -o /dev/null -w "%{http_code}\n" https://taler.hacktivism.ch/config
+```
+
+Laptop: `./scripts/taler-monitoring/taler-monitoring.sh versions` (or `urls`).
+
+## Result (this day)
+
+- bank **1.6.7**, exchange **1.6.7~dev2**, merchant-webui **1.6.11**
+- public `/config` **200** for bank / exchange / merchant after restarts
+- `taler-monitoring.sh -d hacktivism.ch versions` → **behind=0** (all package rows OK)
+
+## Specific residual problems (re-checked after upgrade)
+
+Severity: **P1** = breaks settlement/ops soon · **P2** = noise / tooling · **P3** = hygiene.
+
+### P1 — none known after restarts
+
+Settlement path: bank/exchange/merchant `/config` 200; merchant health can pass including wirewatch (see below).
+
+### P2 — `taler-merchant-dbinit` fails when run by hand
+
+```text
+WARNING Could not run PSQL on file …/global_procedures.sql: psql exit code was 1
+ERROR Failed to initialize tables
+```
+
+- **Schema is actually current:** `_v.patches` includes **`merchant-0041`** (applied 2026-07-16, by role `postgres` during upgrade noise).
+- **Service works:** `check_merchant-health.sh` → ALL CRITICAL CHECKS PASSED when helpers are up.
+- **Cause (likely):** service-user `psql` / cluster path (`Error: Invalid data directory for cluster 17 main` when run poorly), not missing migrations.
+- **Action:** do not treat hand-run `dbinit` exit≠0 as deploy failure; verify `_v.patches` + `/config`. Optional follow-up: fix `pg_wrapper`/`.postgresqlrc` for `taler-merchant-httpd` so dbinit is clean.
+
+### P2 — merchant health vs wirewatch (race / detect)
+
+- `check_merchant-health.sh` uses `pgrep` for `taler-merchant-wirewatch`.
+- Wirewatch is supervised by **`taler-merchant-wirewatch-supervise.sh`** (restarts on PG NOTIFY exit).
+- During restart windows the helper can be absent for seconds → **false FAIL**.
+- When wirewatch + supervise are both live, health reports **OK**.
+- **Action:** re-run health after 5s; ensure supervise is started after package upgrades (start_merchant / ensure_helpers).
+
+### P2 — `check_exchange-health.sh` missing in live exchange container
+
+```text
+exec: "/usr/local/bin/check_exchange-health.sh": no such file or directory
+```
+
+- Script exists in **admin-log** (`scripts/taler-exchange/check_exchange-health.sh`) but was **not installed** into `taler-hacktivism-exchange-ansible`.
+- **Action:** copy into image/live `/usr/local/bin/` on next deploy (same as bank/merchant health scripts).
+
+### P3 — zombie processes inside bank + merchant containers
+
+- **bank:** defunct `java`, `python3`, occasional `postgres` / `dpkg-preconfigu`.
+- **merchant:** many old `taler-merchant-*` defunct (pre-restart leftovers).
+- **Cause:** no proper init/reaper (not systemd PID 1); supervise/start scripts leave zombies.
+- **Impact:** mostly cosmetic / PID table clutter unless extreme.
+- **Action:** periodic container restart or install a tiny reaper; not urgent.
+
+### P3 — package skew (informational)
+
+| Package | Installed | Note |
+|---------|-----------|------|
+| taler-merchant / libtalermerchant / typst | 1.6.9 | no newer in trixie index at check time |
+| taler-merchant-webui | 1.6.11 | intentionally newer SPA |
+| exchange aml/kyc webui | 1.6.8~dev3 | pulled with exchange upgrade |
+
+Monitoring does **not** ERROR on this skew when suite index matches installed.
+
+### Resolved during upgrade (do not re-open without evidence)
+
+- Version **behind** bank/exchange/webui — fixed.
+- Exchange postinst “missing aml-spa/forms.json” — paths **`/usr/share/taler-exchange/{aml,kyc}-spa`** now present after webui packages.
+- Bank down after apt — fixed with postgres socket + `start_bank.sh --restart`.

scripts/README.md

@@ -7,6 +7,7 @@
 | `taler-hacktivism-bank/` | podman `taler-hacktivism-bank`: `/root`, `/usr/local/bin` |
 | `taler-sanity/` | host root checks (stack, settlement, helpers) |
 | `taler-monitoring/` | **outside-in** public URL walk (`/config` → keys/terms/integration/webui) |
+| `taler-shared/` | shared helpers (`upgrade-goa-debs.sh` = apt upgrade **inside** bank/exchange/merchant containers) |
 | `monitoring/` | host `/home/hernani/scripts` (tor relay stats) |
 | `taler-wallet-cli/` | thin wrappers; **benchmarks live in** `../benchmarks/` |
 | `castopod/` | host `hernani` podman-compose `~/koopa-castopod` — see `castopod/README.md` |
@@ -44,4 +45,8 @@ Exchange one-shots (root, offline / wire):
 
 `runuser -u USER -- bash` (never `-u` with `-s` on util-linux).
 
+### Package upgrades (trixie debs inside containers)
+
+Containers are **not** systemd. After `apt-get install` of Taler/libeufin packages, restart with `start_*.sh` only — see **`taler-shared/upgrade-goa-debs.sh`** and day note `2026/2026-07-16--taler-package-upgrade-goa.md`. Host `zypper` does not update these.
+
 SMS helper symlinks into `/var/taler-src/...` are not copied (merchant only).

scripts/taler-monitoring/README.md

@@ -33,16 +33,67 @@ End of each phase: totals + list of **BLOCKERS** and **ERRORS**.
 # Other domains — public HTTPS only, never SSH
 ./taler-monitoring.sh -d taler.net
 ./taler-monitoring.sh --domain taler-ops.ch
+./taler-monitoring.sh -d my.taler-ops.ch urls
 ./taler-monitoring.sh -d demo.taler.net urls
 ./taler-monitoring.sh taler.net    # bare domain = same as -d
+
+# Explicit endpoints (any mix; override profile or skip -d)
+./taler-monitoring.sh --bank https://bank.demo.taler.net \
+  --exchange https://exchange.demo.taler.net \
+  --merchant https://backend.demo.taler.net --currency KUDOS urls
 ```
 
-| Domain | Bank | Exchange | Merchant | Currency |
+## Domain profiles (`domains.conf`)
-|--------|------|----------|----------|----------|
+
-| `hacktivism.ch` (default) | bank.hacktivism.ch | exchange.hacktivism.ch | taler.hacktivism.ch | GOA |
+Each monitored stack must say **what is bank, exchange, and merchant-backend**.
+
+Edit **[domains.conf](./domains.conf)** (or set `TALER_DOMAINS_CONF=`):
+
+```text
+# name   bank   exchange   merchant-backend   currency   local   [canonical]
+taler-ops.ch  bank.taler-ops.ch  exchange.taler-ops.ch  my.taler-ops.ch  CHF  0
+```
+
+| Field | Meaning |
+|-------|---------|
+| **name** | `-d NAME` / `TALER_DOMAIN` match (add alias lines as needed) |
+| **bank** | libeufin/bank public host or `https://…` |
+| **exchange** | exchange public host or URL |
+| **merchant** | merchant **backend** (SPA/API), e.g. `my.taler-ops.ch` |
+| **currency** | `GOA` / `KUDOS` / `CHF` / … |
+| **local** | `1` = koopa SSH stack; `0` = public only |
+| **landing** | `1` = check `/intro` landings + assets; `0` = skip (TOPS / mytops) |
+| **canonical** | optional label after alias (e.g. `my.taler-ops.ch` → `taler-ops.ch`) |
+
+Then:
+
+```bash
+./taler-monitoring.sh -d taler-ops.ch urls
+```
+
+Overrides (always win after profile load):
+
+```bash
+./taler-monitoring.sh -d taler-ops.ch \
+  --merchant https://my.taler-ops.ch \
+  --exchange https://exchange.taler-ops.ch \
+  --bank https://bank.taler-ops.ch urls
+```
+
+Helpers in `lib.sh`: `set_taler_stack`, `load_domain_profile`, `apply_taler_domain`.
+
+### Built-in profiles (see `domains.conf`)
+
+| Domain / alias | Bank | Exchange | Merchant | Currency |
+|----------------|------|----------|----------|----------|
+| `hacktivism.ch` / `koopa` | bank.hacktivism.ch | exchange.hacktivism.ch | taler.hacktivism.ch | GOA |
 | `taler.net` / `demo.taler.net` | bank.demo.taler.net | exchange.demo.taler.net | backend.demo.taler.net | KUDOS |
-| `taler-ops.ch` | bank.* (probe) | exchange.taler-ops.ch | backend.* (probe) | CHF |
+| `test.taler.net` | bank.test… | exchange.test… | backend.test… | **TESTKUDOS** · no landings |
-| other | bank.DOMAIN | exchange.DOMAIN | backend/taler/merchant (probe) | any |
+| `taler-ops.ch` / `my.taler-ops.ch` | bank.taler-ops.ch | exchange.taler-ops.ch | **my.taler-ops.ch** | CHF · **no landings** |
+| `stage.taler-ops.ch` | bank.stage… | exchange.stage… | **my.stage…** | CHF · **no landings** |
+| unknown | bank.DOMAIN | exchange.DOMAIN | my.DOMAIN then probe | any · no landings |
+
+**Landings:** only GOA (and optionally demo) use public `/intro` pages. **taler-ops.ch does not** — `CHECK_LANDING=0` in the profile skips intro crawl, shop assets, and demo-withdraw checks.
 
 **SSH only for koopa** (`hacktivism.ch` / `-d koopa`).
 
@@ -52,8 +103,9 @@ Other domains: never SSH. Optional **e2e** aborts cleanly on login/KYC.
 
 | | Local (koopa) | Remote |
 |--|---------------|--------|
-| **ATM withdraw** | 20 · 50 · 100 · 200 | 10 · 20 · 50 |
+| **ATM withdraw** | 20 · 50 · 100 · 200 · **4200** (paivana) | 10 · 20 · 50 |
 | **Pay ladder** | 0.01 … 10 | 0.01 … 1 |
+| **Paivana** | HTTP paywall + template pay **GOA:4200** (`goa-shop` / `paivana`) | skipped |
 
 ```bash
 ./taler-monitoring.sh e2e
@@ -66,6 +118,10 @@ E2E_VARIABLE=0 WITHDRAW_AMT=GOA:50 PAY_AMT=GOA:1 ./taler-monitoring.sh e2e   # s
 #   E2E_SHOP_PRODUCTS lines: id|Product name|GOA:amount
 #   E2E_SHOP_PICK_N=2
 #   (landing QR = taler://pay-template/…/{id}; popup = live taler://pay after POST templates/{id})
+# Paivana paywall (local GOA only):
+#   E2E_PAIVANA=1 (default)  PAIVANA_PUBLIC=https://paivana.hacktivism.ch
+#   E2E_PAIVANA_TEMPLATE=paivana  E2E_PAIVANA_AMOUNT=GOA:4200  E2E_PAIVANA_INSTANCE=goa-shop
+#   E2E_PAIVANA=0            # skip
 # remote secrets:
 #   E2E_BANK_ADMIN_PASS=…  E2E_MERCHANT_TOKEN=…
 ```

scripts/taler-monitoring/TESTS.md

@@ -43,12 +43,13 @@ IDs are assigned **in run order** within the area (`set_area` resets the counter
 | www-… | **merchant `/terms`** body (dual-currency notice) |
 | www-… | **merchant `/privacy`** body (must not be `not configured`) |
 | www-… | merchant `/terms/` redirect |
-| www-… | **landing exposed links** (bank / merchant / exchange): parse each `/intro/` HTML, probe every own-stack `https://` + root-relative `href`/`src`/`content`, soft-check external stores/docs |
+| www-… | **landing (aggregated per site)**: one line for bank/merchant/exchange (`/intro` + assets + own/external link counts); failures listed as samples; soft external/shop assets compact |
-| www-… | landing static: `qrcode.min.js`, `og-goa-shop.png`, `qr-logo.png`, shop-pay.js/css |
+| www-… | cross-links: one line (4 intros) on local stack |
-| www-… | cross-links between bank ↔ merchant ↔ exchange intros (local stack) |
+| www-… | bank demo-withdraw + shop assets: one line (+ soft shop-pay if missing) |
 | www-… | **bank `/intro/demo-withdraw.json`** → `taler://withdraw/HOST:PORT/taler-integration/…` + integration op HTTP 200 |
 | www-… | bank `/intro/auto-account.json` (earlier) → same withdraw shape, **no payto_uri**, login at `/webui/` |
-| www-… | **performance** (outside-in): public HTTPS RTT for bank `/config`, `/taler-integration/config`, `/webui/`, `/intro/`, `stats.json`; exchange `/config`, `/keys`, `/intro/`; merchant `/config`, `/webui/`, `/intro/` — report ms; WARN ≥ `PERF_WARN_MS` (default 8000); **ERROR ≥ `PERF_FAIL_MS` (default 20000)** |
+| www-… | **performance** (outside-in RTT): bank/exchange/merchant `/config`, keys, webui, intro — ms; WARN ≥ `PERF_WARN_MS` (8000); ERROR ≥ `PERF_FAIL_MS` (20000) |
+| www-… | **landing load stats**: `/intro/stats.json` → loadavg, container RSS, in-container probe ms (public; SSH container cat fallback); warn if stale/missing |
 
 **Legal docs rule:** HTTP 200, non-empty body, not plain `not configured`, not merchant API JSON `code:21`. On local stack, optional content needle (terms/privacy/FADP/GOA…).
 
@@ -135,6 +136,8 @@ Without SSH (`SKIP_SSH=1` or remote domain): still runs outside-in repo checks;
 
 ## e2e — payment path (`./taler-monitoring.sh e2e`)
 
+Local GOA also: **ATM includes GOA:4200**, then **paivana** (HTTP 302 on `PAIVANA_PUBLIC` + public template pay `paivana` / GOA:4200 on `goa-shop`). Disable with `E2E_PAIVANA=0`.
+
 | ID | Step (approx.) |
 |----|----------------|
 | e2e-001 | budget info |

scripts/taler-shared/upgrade-goa-debs.sh

@@ -0,0 +1,126 @@
+#!/usr/bin/env bash
+# Upgrade GNU Taler .deb packages *inside* the three GOA containers on koopa.
+# Host openSUSE packages are unrelated (zypper) — do not confuse with this.
+#
+# Containers (no systemd as PID 1): after apt, restart with start_*.sh only.
+#
+# Run on koopa host as a user that can `podman exec` the containers:
+#   ./upgrade-goa-debs.sh
+#   ./upgrade-goa-debs.sh bank|exchange|merchant
+#
+# Mirror: scripts/taler-shared/ in koopa-admin-log.
+# Live note: 2026/2026-07-16--taler-package-upgrade-goa.md
+set -euo pipefail
+
+BANK_CTR="${BANK_CTR:-taler-hacktivism-bank}"
+EXCHANGE_CTR="${EXCHANGE_CTR:-taler-hacktivism-exchange-ansible}"
+MERCHANT_CTR="${MERCHANT_CTR:-taler-hacktivism}"
+
+only="${1:-all}"
+
+pod_root() {
+  local ctr="$1"
+  shift
+  podman exec -u root "$ctr" bash -c "$*"
+}
+
+upgrade_bank() {
+  echo "======== bank ($BANK_CTR) ========"
+  pod_root "$BANK_CTR" '
+    set -e
+    export DEBIAN_FRONTEND=noninteractive
+    apt-get update -qq
+    apt-get install -y libeufin-bank libeufin-common
+    dpkg -l libeufin-bank libeufin-common | awk "/^ii/{print \$2,\$3}"
+  '
+  # restart app only (postgres via start_base if needed)
+  pod_root "$BANK_CTR" '
+    set -e
+    if ! runuser -u postgres -- psql -c "SELECT 1" >/dev/null 2>&1; then
+      pg_ctlcluster 17 main start || true
+      sleep 2
+    fi
+    mkdir -p /var/log/libeufin-bank
+    chown libeufin-bank:libeufin-bank /var/log/libeufin-bank 2>/dev/null || true
+    runuser -u libeufin-bank -- /usr/local/bin/start_bank.sh --restart
+  '
+}
+
+upgrade_exchange() {
+  echo "======== exchange ($EXCHANGE_CTR) ========"
+  pod_root "$EXCHANGE_CTR" '
+    set -e
+    export DEBIAN_FRONTEND=noninteractive
+    apt-get update -qq
+    apt-get install -y \
+      taler-exchange taler-exchange-database taler-exchange-offline \
+      taler-exchange-typst libtalerexchange taler-terms-generator \
+      taler-exchange-aml-webui taler-exchange-kyc-webui || true
+    dpkg -l "taler-exchange*" "libtalerexchange" "taler-terms-generator" 2>/dev/null \
+      | awk "/^ii/{print \$2,\$3}"
+  '
+  # do not run dbinit as root (role "root" does not exist)
+  pod_root "$EXCHANGE_CTR" '
+    set -e
+    if id taler-exchange-httpd >/dev/null 2>&1; then
+      runuser -u taler-exchange-httpd -- taler-exchange-dbinit 2>&1 | tail -20 || true
+    fi
+    if [ -x /root/start_base_services_for_taler_exchange.sh ]; then
+      /root/start_base_services_for_taler_exchange.sh --no-shell 2>/dev/null || true
+    fi
+    if [ -x /usr/local/bin/start_exchange.sh ]; then
+      runuser -u taler-exchange-httpd -- /usr/local/bin/start_exchange.sh --restart 2>&1 | tail -30 || true
+    fi
+    pgrep -a taler-exchange-httpd | head -3 || true
+  '
+}
+
+upgrade_merchant() {
+  echo "======== merchant ($MERCHANT_CTR) ========"
+  pod_root "$MERCHANT_CTR" '
+    set -e
+    export DEBIAN_FRONTEND=noninteractive
+    apt-get update -qq
+    apt-get install -y \
+      taler-merchant taler-merchant-webui taler-merchant-typst \
+      libtalermerchant taler-terms-generator
+    dpkg -l "taler-merchant*" "libtalermerchant" 2>/dev/null | awk "/^ii/{print \$2,\$3}"
+  '
+  pod_root "$MERCHANT_CTR" '
+    set -e
+    # schema: prefer tools as service user; failures may be noisy — check /config after
+    if id taler-merchant-httpd >/dev/null 2>&1; then
+      runuser -u taler-merchant-httpd -- taler-merchant-dbinit 2>&1 | tail -20 || true
+    fi
+    runuser -u taler-merchant-httpd -- /usr/local/bin/start_merchant.sh --restart
+  '
+}
+
+smoke() {
+  echo "======== smoke (host loopback) ========"
+  curl -sS -m 8 -o /dev/null -w "bank    %{http_code}  :9012/config\n" http://127.0.0.1:9012/config || true
+  curl -sS -m 8 -o /dev/null -w "exchange %{http_code}  :9011/config\n" http://127.0.0.1:9011/config || true
+  curl -skS -m 8 -o /dev/null -w "merchant %{http_code}  :9010/config\n" https://127.0.0.1:9010/config || true
+}
+
+case "$only" in
+  all)
+    upgrade_bank
+    upgrade_exchange
+    upgrade_merchant
+    smoke
+    ;;
+  bank) upgrade_bank; smoke ;;
+  exchange) upgrade_exchange; smoke ;;
+  merchant) upgrade_merchant; smoke ;;
+  -h|--help|help)
+    sed -n '2,20p' "$0"
+    exit 0
+    ;;
+  *)
+    echo "usage: $0 [all|bank|exchange|merchant]" >&2
+    exit 2
+    ;;
+esac
+
+echo "DONE — re-run taler-monitoring versions/urls from a laptop if desired."