145 lines
7 KiB
Markdown
145 lines
7 KiB
Markdown
# 2026-07-16 — GOA container package upgrade (trixie debs)
|
|
|
|
## Why
|
|
|
|
`taler-monitoring.sh versions` reported packages **behind** `deb.taler.net` **trixie**:
|
|
|
|
| Component | Package | Was | Target (trixie index) |
|
|
|-----------|---------|-----|------------------------|
|
|
| bank | libeufin-bank / common | 1.6.6 | **1.6.7** |
|
|
| exchange | taler-exchange* / libtalerexchange | 1.6.6 | **1.6.7~dev2** |
|
|
| merchant | taler-merchant-webui | 1.6.9 | **1.6.11** |
|
|
| merchant | taler-merchant / libtalermerchant | 1.6.9 | still 1.6.9 (no newer in suite for those) |
|
|
|
|
Host is openSUSE Tumbleweed — **irrelevant** for these versions. Debs live **inside** Debian-based podman images.
|
|
|
|
## Containers
|
|
|
|
| Role | Name |
|
|
|------|------|
|
|
| bank | `taler-hacktivism-bank` |
|
|
| exchange | `taler-hacktivism-exchange-ansible` |
|
|
| merchant | `taler-hacktivism` |
|
|
|
|
**No systemd as PID 1** in these containers. Do **not** rely on `systemctl restart` after apt (policy-rc.d / no bus). Restart with:
|
|
|
|
```bash
|
|
# bank
|
|
podman exec -u root taler-hacktivism-bank \
|
|
runuser -u libeufin-bank -- /usr/local/bin/start_bank.sh --restart
|
|
|
|
# merchant
|
|
podman exec -u root taler-hacktivism \
|
|
runuser -u taler-merchant-httpd -- /usr/local/bin/start_merchant.sh --restart
|
|
|
|
# exchange: base (root) then start_exchange as httpd user
|
|
podman exec -u root taler-hacktivism-exchange-ansible \
|
|
bash -c '/root/start_base_services_for_taler_exchange.sh --no-shell 2>/dev/null; \
|
|
runuser -u taler-exchange-httpd -- /usr/local/bin/start_exchange.sh --restart'
|
|
```
|
|
|
|
## Procedure (manual)
|
|
|
|
```bash
|
|
# as root inside each container (example bank)
|
|
export DEBIAN_FRONTEND=noninteractive
|
|
apt-get update -qq
|
|
apt-get install -y libeufin-bank libeufin-common # bank
|
|
# exchange: taler-exchange taler-exchange-database libtalerexchange …
|
|
# merchant: taler-merchant-webui (and stack as needed)
|
|
```
|
|
|
|
Then restart with `start_*.sh` as above.
|
|
|
|
**Scripted (host):** `scripts/taler-shared/upgrade-goa-debs.sh`
|
|
Copy to koopa or run from a checkout: `./upgrade-goa-debs.sh` / `bank|exchange|merchant`.
|
|
|
|
## Lessons from 2026-07-16 run
|
|
|
|
1. **`apt-get install` as root works** (`podman exec -u root`); suite `trixie` (+ merchant may also have `trixie-testing`).
|
|
2. **`taler-exchange-dbinit` as root fails** (`role "root" does not exist`) — run as `taler-exchange-httpd` if needed.
|
|
3. **Bank after upgrade:** if postgres socket was down, bank dies with pool init error; `pg_ctlcluster 17 main start` then `start_bank.sh --restart`.
|
|
4. **Merchant `taler-merchant-dbinit` / merchant-0041.sql** can noise-fail (psql cluster path); after `start_merchant.sh --restart`, health check can still be green — verify `https://127.0.0.1:9010/config` and public `taler.hacktivism.ch`.
|
|
5. Exchange **postinst** may warn about missing SPA files under `/usr/share/taler-exchange/{aml,kyc}-spa/`; packages `taler-exchange-aml-webui` / `kyc-webui` pull in SPAs — re-check if AML UI is used.
|
|
6. **Images are live-writable** (not immutable rebuild): upgraded debs are in the running container layers until next image rebuild/snapshot.
|
|
|
|
## Smoke after upgrade
|
|
|
|
```bash
|
|
curl -sS -o /dev/null -w "%{http_code}\n" http://127.0.0.1:9012/config
|
|
curl -sS -o /dev/null -w "%{http_code}\n" http://127.0.0.1:9011/config
|
|
curl -skS -o /dev/null -w "%{http_code}\n" https://127.0.0.1:9010/config
|
|
# public
|
|
curl -skS -o /dev/null -w "%{http_code}\n" https://bank.hacktivism.ch/config
|
|
curl -skS -o /dev/null -w "%{http_code}\n" https://exchange.hacktivism.ch/config
|
|
curl -skS -o /dev/null -w "%{http_code}\n" https://taler.hacktivism.ch/config
|
|
```
|
|
|
|
Laptop: `./scripts/taler-monitoring/taler-monitoring.sh versions` (or `urls`).
|
|
|
|
## Result (this day)
|
|
|
|
- bank **1.6.7**, exchange **1.6.7~dev2**, merchant-webui **1.6.11**
|
|
- public `/config` **200** for bank / exchange / merchant after restarts
|
|
- `taler-monitoring.sh -d hacktivism.ch versions` → **behind=0** (all package rows OK)
|
|
|
|
## Specific residual problems (re-checked after upgrade)
|
|
|
|
Severity: **P1** = breaks settlement/ops soon · **P2** = noise / tooling · **P3** = hygiene.
|
|
|
|
### P1 — none known after restarts
|
|
|
|
Settlement path: bank/exchange/merchant `/config` 200; merchant health can pass including wirewatch (see below).
|
|
|
|
### P2 — `taler-merchant-dbinit` fails when run by hand
|
|
|
|
```text
|
|
WARNING Could not run PSQL on file …/global_procedures.sql: psql exit code was 1
|
|
ERROR Failed to initialize tables
|
|
```
|
|
|
|
- **Schema is actually current:** `_v.patches` includes **`merchant-0041`** (applied 2026-07-16, by role `postgres` during upgrade noise).
|
|
- **Service works:** `check_merchant-health.sh` → ALL CRITICAL CHECKS PASSED when helpers are up.
|
|
- **Cause (likely):** service-user `psql` / cluster path (`Error: Invalid data directory for cluster 17 main` when run poorly), not missing migrations.
|
|
- **Action:** do not treat hand-run `dbinit` exit≠0 as deploy failure; verify `_v.patches` + `/config`. Optional follow-up: fix `pg_wrapper`/`.postgresqlrc` for `taler-merchant-httpd` so dbinit is clean.
|
|
|
|
### P2 — merchant health vs wirewatch (race / detect)
|
|
|
|
- `check_merchant-health.sh` uses `pgrep` for `taler-merchant-wirewatch`.
|
|
- Wirewatch is supervised by **`taler-merchant-wirewatch-supervise.sh`** (restarts on PG NOTIFY exit).
|
|
- During restart windows the helper can be absent for seconds → **false FAIL**.
|
|
- When wirewatch + supervise are both live, health reports **OK**.
|
|
- **Action:** re-run health after 5s; ensure supervise is started after package upgrades (start_merchant / ensure_helpers).
|
|
|
|
### P2 — `check_exchange-health.sh` missing in live exchange container
|
|
|
|
```text
|
|
exec: "/usr/local/bin/check_exchange-health.sh": no such file or directory
|
|
```
|
|
|
|
- Script exists in **admin-log** (`scripts/taler-exchange/check_exchange-health.sh`) but was **not installed** into `taler-hacktivism-exchange-ansible`.
|
|
- **Action:** copy into image/live `/usr/local/bin/` on next deploy (same as bank/merchant health scripts).
|
|
|
|
### P3 — zombie processes inside bank + merchant containers
|
|
|
|
- **bank:** defunct `java`, `python3`, occasional `postgres` / `dpkg-preconfigu`.
|
|
- **merchant:** many old `taler-merchant-*` defunct (pre-restart leftovers).
|
|
- **Cause:** no proper init/reaper (not systemd PID 1); supervise/start scripts leave zombies.
|
|
- **Impact:** mostly cosmetic / PID table clutter unless extreme.
|
|
- **Action:** periodic container restart or install a tiny reaper; not urgent.
|
|
|
|
### P3 — package skew (informational)
|
|
|
|
| Package | Installed | Note |
|
|
|---------|-----------|------|
|
|
| taler-merchant / libtalermerchant / typst | 1.6.9 | no newer in trixie index at check time |
|
|
| taler-merchant-webui | 1.6.11 | intentionally newer SPA |
|
|
| exchange aml/kyc webui | 1.6.8~dev3 | pulled with exchange upgrade |
|
|
|
|
Monitoring does **not** ERROR on this skew when suite index matches installed.
|
|
|
|
### Resolved during upgrade (do not re-open without evidence)
|
|
|
|
- Version **behind** bank/exchange/webui — fixed.
|
|
- Exchange postinst “missing aml-spa/forms.json” — paths **`/usr/share/taler-exchange/{aml,kyc}-spa`** now present after webui packages.
|
|
- Bank down after apt — fixed with postgres socket + `start_bank.sh --restart`.
|