hernani/koopa-admin-log
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(paivana): clear postmortem — three failures that broke paywall

Browse source 
Document boot race (upstream unit exit 125), start-before-merchant 502,
and pasta AAAA/timeout needing 172.17.0.1 pin. Healthy smoke stays 302/402.
This commit is contained in:
Hernâni Marques 2026-07-17 21:28:14 +02:00
parent 80f0219f11
commit 4b729663b3
No known key found for this signature in database
3 changed files with 81 additions and 21 deletions
Download patch file Download diff file Expand all files Collapse all files

86
configs/paivana/README.md
View file

@ -37,6 +37,65 @@ GNU Taler **paivana-httpd** reverse-proxy paywall (DD 95 / DD 76 style), GOA pay
Mirror notes: `koopa-admin-secrets` → `koopa/home-hernani/koopa-paivana/secrets/` (paths only / examples).
## Was kaputt war (klar · 2026-07)
Drei **unabhängige** Defekte; Symptom oft gleich (Paywall 502 / tot / kein Template).
Stand: behoben auf koopa; nicht wiederholen.
### 1) Boot-Race: extra systemd-Unit nur für Upstream
| | |
|--|--|
| **Symptom** | Nach Reboot: `container-koopa-paivana-upstream.service` **failed** / start-limit-hit; Upstream fehlt oder Compose und Unit kämpfen. |
| **Ursache** | Separate User-Unit tat nur `podman start koopa-paivana-upstream` **ohne** dass Compose die Container schon angelegt hatte → **exit 125**. Compose-Unit startet **beide** Services (`paivana` + `upstream`) ohnehin. |
| **Fix** | **Nur** `container-koopa-paivana.service` (compose up). Separate Upstream-Unit: `disable --now`. Nicht neu anlegen. |
| **Nicht verwechseln mit** | Problem 2 (Merchant noch down → 502 trotz laufender Container). |
### 2) Start-Reihenfolge: Paivana vor Merchant-Apps → **502**
| | |
|--|--|
| **Symptom** | `paivana.hacktivism.ch` / `:9025` **502** nach Boot; Container laufen, Paywall/Template-Flow tot. |
| **Ursache** | Merchant-Container ist hoch, aber **In-App** (`taler-merchant-httpd` etc.) startet erst über `taler-merchant-apps.service` / `ensure-taler-apps`. Paivana braucht die **private Merchant-API** (Template `paivana` / goa-shop). Zu früh → 502. |
| **Fix** | Drop-in `configs/systemd/user/container-koopa-paivana.service.d/order.conf`: `After=` / `Wants=` `taler-merchant-apps.service` (+ ggf. merchant-Container). Siehe `2026/2026-07-16--reboot-recovery.md`. |
| **Check** | Merchant `https://taler.hacktivism.ch/config` **200**, dann Paivana smoke (unten). |
### 3) Pasta / DNS: Container erreicht Merchant nicht (Timeout / AAAA)
| | |
|--|--|
| **Symptom** | Paivana läuft, aber Template-Laden / Merchant-Calls **hängen oder scheitern**; Logs: Timeouts Richtung `taler.hacktivism.ch`. |
| **Ursache** | Rootless **pasta** hairpint öffentliche DNS nicht sauber. `extra_hosts: …:host-gateway` allein reichte nicht: oft gewinnt noch die **öffentliche AAAA**, Connection timeout. Merchant sitzt am Host (**~:9010**), erreichbar über Pasta-IPv4 **`172.17.0.1`**. |
| **Fix** | In `compose.yml` feste IPv4-Hosts (nicht nur host-gateway): |
```yaml
extra_hosts:
- "taler.hacktivism.ch:172.17.0.1"
- "bank.hacktivism.ch:172.17.0.1"
- "exchange.hacktivism.ch:172.17.0.1"
```
| | |
|--|--|
| **Commit** | `f26bf6a` (pin Taler hosts to pasta IPv4) |
### Kurz: was *nicht* das Problem war
- Caddy-Vhost an sich (leitet auf **9025**).
- Fehlen des Templates im Merchant (Template `paivana` / GOA:4200 war angelegt; ohne Merchant-Apps oder ohne Routing wirkt es so, als fehlte es).
- „Nur nginx-Upstream“ — Upstream ist unprivileged nginx **:8080**; ohne Problem 1/2/3 ist das stabil.
### Erwartet gesund
| Check | Expect |
|-------|--------|
| `curl -si http://127.0.0.1:9025/` | **302** → `/.well-known/paivana/templates/paivana#…` |
| `https://paivana.hacktivism.ch/` | **402** paywall HTML + `paivana: taler://pay-template/…` |
| monitoring `urls` | `www.paivana` OK |
| e2e (GOA, lokal) | `e2e.paivana` Template-Pay **GOA:4200** (wenn `E2E_PAIVANA=1`) |
---
## Ops
```bash
@ -54,35 +113,22 @@ Autostart (**only** the compose unit — starts **both** `koopa-paivana` and
```bash
mkdir -p ~/.config/systemd/user
cp ~/koopa-paivana/container-koopa-paivana.service ~/.config/systemd/user/
# optional: After= merchant (configs/systemd/user/container-koopa-paivana.service.d/)
# required on koopa: After= merchant apps
# configs/systemd/user/container-koopa-paivana.service.d/order.conf
systemctl --user daemon-reload
systemctl --user enable --now container-koopa-paivana.service
```
**Do not** enable a separate `container-koopa-paivana-upstream.service` that only
runs `podman start koopa-paivana-upstream`. At boot that unit races compose
(container not created yet → exit 125 → start-limit-hit). Compose already
starts the upstream. If such a unit exists:
runs `podman start koopa-paivana-upstream` — see failure **#1** above.
If such a unit exists:
`systemctl --user disable --now container-koopa-paivana-upstream.service`.
Healthy smoke:
| Check | Expect |
|-------|--------|
| `curl -si http://127.0.0.1:9025/` | **302** → `/.well-known/paivana/templates/paivana#…` |
| template URL / public `https://paivana.hacktivism.ch/` | **402** paywall HTML + `paivana: taler://pay-template/…` |
| monitoring | `./scripts/taler-monitoring/taler-monitoring.sh -d hacktivism.ch urls` → `www.paivana` OK |
## Network notes
Rootless pasta cannot hairpin public DNS for the merchant. Compose sets:
```yaml
extra_hosts:
- "taler.hacktivism.ch:host-gateway"
```
so `paivana-httpd` can load templates from the merchant API.
See failure **#3**. Live compose pins Taler hostnames to pasta IPv4
`172.17.0.1` so `paivana-httpd` reaches the host merchant private API without
public AAAA timeouts.
## Caddy