koopa-admin-log/2026/2026-07-16--taler-package-upgrade-goa.md

2026-07-16 — GOA container package upgrade (trixie debs)

Why

taler-monitoring.sh versions reported packages behind deb.taler.net trixie:

Component	Package	Was	Target (trixie index)
bank	libeufin-bank / common	1.6.6	1.6.7
exchange	taler-exchange* / libtalerexchange	1.6.6	1.6.7~dev2
merchant	taler-merchant-webui	1.6.9	1.6.11
merchant	taler-merchant / libtalermerchant	1.6.9	still 1.6.9 (no newer in suite for those)

Host is openSUSE Tumbleweed — irrelevant for these versions. Debs live inside Debian-based podman images.

Containers

Role	Name
bank	taler-hacktivism-bank
exchange	taler-hacktivism-exchange-ansible
merchant	taler-hacktivism

No systemd as PID 1 in these containers. Do not rely on systemctl restart after apt (policy-rc.d / no bus). Restart with:

# bank
podman exec -u root taler-hacktivism-bank \
  runuser -u libeufin-bank -- /usr/local/bin/start_bank.sh --restart

# merchant
podman exec -u root taler-hacktivism \
  runuser -u taler-merchant-httpd -- /usr/local/bin/start_merchant.sh --restart

# exchange: base (root) then start_exchange as httpd user
podman exec -u root taler-hacktivism-exchange-ansible \
  bash -c '/root/start_base_services_for_taler_exchange.sh --no-shell 2>/dev/null; \
    runuser -u taler-exchange-httpd -- /usr/local/bin/start_exchange.sh --restart'

Procedure (manual)

# as root inside each container (example bank)
export DEBIAN_FRONTEND=noninteractive
apt-get update -qq
apt-get install -y libeufin-bank libeufin-common   # bank
# exchange: taler-exchange taler-exchange-database libtalerexchange …
# merchant: taler-merchant-webui (and stack as needed)

Then restart with start_*.sh as above.

Scripted (host): scripts/taler-shared/upgrade-goa-debs.sh
Copy to koopa or run from a checkout: ./upgrade-goa-debs.sh / bank|exchange|merchant.

Lessons from 2026-07-16 run

1. apt-get install as root works (podman exec -u root); suite trixie (+ merchant may also have trixie-testing).
2. taler-exchange-dbinit as root fails (role "root" does not exist) — run as taler-exchange-httpd if needed.
3. Bank after upgrade: if postgres socket was down, bank dies with pool init error; pg_ctlcluster 17 main start then start_bank.sh --restart.
4. Merchant taler-merchant-dbinit / merchant-0041.sql can noise-fail (psql cluster path); after start_merchant.sh --restart, health check can still be green — verify https://127.0.0.1:9010/config and public taler.hacktivism.ch.
5. Exchange postinst may warn about missing SPA files under /usr/share/taler-exchange/{aml,kyc}-spa/; packages taler-exchange-aml-webui / kyc-webui pull in SPAs — re-check if AML UI is used.
6. Images are live-writable (not immutable rebuild): upgraded debs are in the running container layers until next image rebuild/snapshot.

Smoke after upgrade

curl -sS -o /dev/null -w "%{http_code}\n" http://127.0.0.1:9012/config
curl -sS -o /dev/null -w "%{http_code}\n" http://127.0.0.1:9011/config
curl -skS -o /dev/null -w "%{http_code}\n" https://127.0.0.1:9010/config
# public
curl -skS -o /dev/null -w "%{http_code}\n" https://bank.hacktivism.ch/config
curl -skS -o /dev/null -w "%{http_code}\n" https://exchange.hacktivism.ch/config
curl -skS -o /dev/null -w "%{http_code}\n" https://taler.hacktivism.ch/config

Laptop: ./scripts/taler-monitoring/taler-monitoring.sh versions (or urls).

Result (this day)

• bank 1.6.7, exchange 1.6.7~dev2, merchant-webui 1.6.11
• public /config 200 for bank / exchange / merchant after restarts
• taler-monitoring.sh -d hacktivism.ch versions → behind=0 (all package rows OK)

Specific residual problems (re-checked after upgrade)

Severity: P1 = breaks settlement/ops soon · P2 = noise / tooling · P3 = hygiene.

P1 — none known after restarts

Settlement path: bank/exchange/merchant /config 200; merchant health can pass including wirewatch (see below).

P2 — taler-merchant-dbinit fails when run by hand

WARNING Could not run PSQL on file …/global_procedures.sql: psql exit code was 1
ERROR Failed to initialize tables

• Schema is actually current: _v.patches includes merchant-0041 (applied 2026-07-16, by role postgres during upgrade noise).
• Service works: check_merchant-health.sh → ALL CRITICAL CHECKS PASSED when helpers are up.
• Cause (likely): service-user psql / cluster path (Error: Invalid data directory for cluster 17 main when run poorly), not missing migrations.
• Action: do not treat hand-run dbinit exit≠0 as deploy failure; verify _v.patches + /config. Optional follow-up: fix pg_wrapper/.postgresqlrc for taler-merchant-httpd so dbinit is clean.

P2 — merchant health vs wirewatch (race / detect)

• check_merchant-health.sh uses pgrep for taler-merchant-wirewatch.
• Wirewatch is supervised by taler-merchant-wirewatch-supervise.sh (restarts on PG NOTIFY exit).
• During restart windows the helper can be absent for seconds → false FAIL.
• When wirewatch + supervise are both live, health reports OK.
• Action: re-run health after 5s; ensure supervise is started after package upgrades (start_merchant / ensure_helpers).

P2 — check_exchange-health.sh missing in live exchange container

exec: "/usr/local/bin/check_exchange-health.sh": no such file or directory

• Script exists in admin-log (scripts/taler-exchange/check_exchange-health.sh) but was not installed into taler-hacktivism-exchange-ansible.
• Action: copy into image/live /usr/local/bin/ on next deploy (same as bank/merchant health scripts).

P3 — zombie processes inside bank + merchant containers

• bank: defunct java, python3, occasional postgres / dpkg-preconfigu.
• merchant: many old taler-merchant-* defunct (pre-restart leftovers).
• Cause: no proper init/reaper (not systemd PID 1); supervise/start scripts leave zombies.
• Impact: mostly cosmetic / PID table clutter unless extreme.
• Action: periodic container restart or install a tiny reaper; not urgent.

P3 — package skew (informational)

Package	Installed	Note
taler-merchant / libtalermerchant / typst	1.6.9	no newer in trixie index at check time
taler-merchant-webui	1.6.11	intentionally newer SPA
exchange aml/kyc webui	1.6.8~dev3	pulled with exchange upgrade

Monitoring does not ERROR on this skew when suite index matches installed.

Resolved during upgrade (do not re-open without evidence)

• Version behind bank/exchange/webui — fixed.
• Exchange postinst “missing aml-spa/forms.json” — paths /usr/share/taler-exchange/{aml,kyc}-spa now present after webui packages.
• Bank down after apt — fixed with postgres socket + start_bank.sh --restart.