# 2026-07-16 — GOA container package upgrade (trixie debs)

## Why

`taler-monitoring.sh versions` reported packages **behind** `deb.taler.net` **trixie**:

| Component | Package | Was | Target (trixie index) |
|-----------|---------|-----|------------------------|
| bank | libeufin-bank / common | 1.6.6 | **1.6.7** |
| exchange | taler-exchange* / libtalerexchange | 1.6.6 | **1.6.7~dev2** |
| merchant | taler-merchant-webui | 1.6.9 | **1.6.11** |
| merchant | taler-merchant / libtalermerchant | 1.6.9 | still 1.6.9 (no newer in suite for those) |

Host is openSUSE Tumbleweed — **irrelevant** for these versions. Debs live **inside** Debian-based podman images.

## Containers

| Role | Name |
|------|------|
| bank | `taler-hacktivism-bank` |
| exchange | `taler-hacktivism-exchange-ansible` |
| merchant | `taler-hacktivism` |

**No systemd as PID 1** in these containers. Do **not** rely on `systemctl restart` after apt (policy-rc.d / no bus). Restart with:

```bash
# bank
podman exec -u root taler-hacktivism-bank \
  runuser -u libeufin-bank -- /usr/local/bin/start_bank.sh --restart

# merchant
podman exec -u root taler-hacktivism \
  runuser -u taler-merchant-httpd -- /usr/local/bin/start_merchant.sh --restart

# exchange: base (root) then start_exchange as httpd user
podman exec -u root taler-hacktivism-exchange-ansible \
  bash -c '/root/start_base_services_for_taler_exchange.sh --no-shell 2>/dev/null; \
    runuser -u taler-exchange-httpd -- /usr/local/bin/start_exchange.sh --restart'
```

## Procedure (manual)

```bash
# as root inside each container (example bank)
export DEBIAN_FRONTEND=noninteractive
apt-get update -qq
apt-get install -y libeufin-bank libeufin-common   # bank
# exchange: taler-exchange taler-exchange-database libtalerexchange …
# merchant: taler-merchant-webui (and stack as needed)
```

Then restart with `start_*.sh` as above.

**Scripted (host):** `scripts/taler-shared/upgrade-goa-debs.sh`  
Copy to koopa or run from a checkout: `./upgrade-goa-debs.sh` / `bank|exchange|merchant`.

## Lessons from 2026-07-16 run

1. **`apt-get install` as root works** (`podman exec -u root`); suite `trixie` (+ merchant may also have `trixie-testing`).
2. **`taler-exchange-dbinit` as root fails** (`role "root" does not exist`) — run as `taler-exchange-httpd` if needed.
3. **Bank after upgrade:** if postgres socket was down, bank dies with pool init error; `pg_ctlcluster 17 main start` then `start_bank.sh --restart`.
4. **Merchant `taler-merchant-dbinit` / merchant-0041.sql** can noise-fail (psql cluster path); after `start_merchant.sh --restart`, health check can still be green — verify `https://127.0.0.1:9010/config` and public `taler.hacktivism.ch`.
5. Exchange **postinst** may warn about missing SPA files under `/usr/share/taler-exchange/{aml,kyc}-spa/`; packages `taler-exchange-aml-webui` / `kyc-webui` pull in SPAs — re-check if AML UI is used.
6. **Images are live-writable** (not immutable rebuild): upgraded debs are in the running container layers until next image rebuild/snapshot.

## Smoke after upgrade

```bash
curl -sS -o /dev/null -w "%{http_code}\n" http://127.0.0.1:9012/config
curl -sS -o /dev/null -w "%{http_code}\n" http://127.0.0.1:9011/config
curl -skS -o /dev/null -w "%{http_code}\n" https://127.0.0.1:9010/config
# public
curl -skS -o /dev/null -w "%{http_code}\n" https://bank.hacktivism.ch/config
curl -skS -o /dev/null -w "%{http_code}\n" https://exchange.hacktivism.ch/config
curl -skS -o /dev/null -w "%{http_code}\n" https://taler.hacktivism.ch/config
```

Laptop: `./scripts/taler-monitoring/taler-monitoring.sh versions` (or `urls`).

## Result (this day)

- bank **1.6.7**, exchange **1.6.7~dev2**, merchant-webui **1.6.11**
- public `/config` **200** for bank / exchange / merchant after restarts
- `taler-monitoring.sh -d hacktivism.ch versions` → **behind=0** (all package rows OK)

## Specific residual problems (re-checked after upgrade)

Severity: **P1** = breaks settlement/ops soon · **P2** = noise / tooling · **P3** = hygiene.

### P1 — none known after restarts

Settlement path: bank/exchange/merchant `/config` 200; merchant health can pass including wirewatch (see below).

### P2 — `taler-merchant-dbinit` fails when run by hand

```text
WARNING Could not run PSQL on file …/global_procedures.sql: psql exit code was 1
ERROR Failed to initialize tables
```

- **Schema is actually current:** `_v.patches` includes **`merchant-0041`** (applied 2026-07-16, by role `postgres` during upgrade noise).
- **Service works:** `check_merchant-health.sh` → ALL CRITICAL CHECKS PASSED when helpers are up.
- **Cause (likely):** service-user `psql` / cluster path (`Error: Invalid data directory for cluster 17 main` when run poorly), not missing migrations.
- **Action:** do not treat hand-run `dbinit` exit≠0 as deploy failure; verify `_v.patches` + `/config`. Optional follow-up: fix `pg_wrapper`/`.postgresqlrc` for `taler-merchant-httpd` so dbinit is clean.

### P2 — merchant health vs wirewatch (race / detect)

- `check_merchant-health.sh` uses `pgrep` for `taler-merchant-wirewatch`.
- Wirewatch is supervised by **`taler-merchant-wirewatch-supervise.sh`** (restarts on PG NOTIFY exit).
- During restart windows the helper can be absent for seconds → **false FAIL**.
- When wirewatch + supervise are both live, health reports **OK**.
- **Action:** re-run health after 5s; ensure supervise is started after package upgrades (start_merchant / ensure_helpers).

### P2 — `check_exchange-health.sh` missing in live exchange container

```text
exec: "/usr/local/bin/check_exchange-health.sh": no such file or directory
```

- Script exists in **admin-log** (`scripts/taler-exchange/check_exchange-health.sh`) but was **not installed** into `taler-hacktivism-exchange-ansible`.
- **Action:** copy into image/live `/usr/local/bin/` on next deploy (same as bank/merchant health scripts).

### P3 — zombie processes inside bank + merchant containers

- **bank:** defunct `java`, `python3`, occasional `postgres` / `dpkg-preconfigu`.
- **merchant:** many old `taler-merchant-*` defunct (pre-restart leftovers).
- **Cause:** no proper init/reaper (not systemd PID 1); supervise/start scripts leave zombies.
- **Impact:** mostly cosmetic / PID table clutter unless extreme.
- **Action:** periodic container restart or install a tiny reaper; not urgent.

### P3 — package skew (informational)

| Package | Installed | Note |
|---------|-----------|------|
| taler-merchant / libtalermerchant / typst | 1.6.9 | no newer in trixie index at check time |
| taler-merchant-webui | 1.6.11 | intentionally newer SPA |
| exchange aml/kyc webui | 1.6.8~dev3 | pulled with exchange upgrade |

Monitoring does **not** ERROR on this skew when suite index matches installed.

### Resolved during upgrade (do not re-open without evidence)

- Version **behind** bank/exchange/webui — fixed.
- Exchange postinst “missing aml-spa/forms.json” — paths **`/usr/share/taler-exchange/{aml,kyc}-spa`** now present after webui packages.
- Bank down after apt — fixed with postgres socket + `start_bank.sh --restart`.