ops: GOA merchant-first 2026-09-15 daylog + lab-hosts + APPLY driver

2026-09-16 02:48:57 +02:00 · 2026-09-16 02:48:57 +02:00 · 0d881cd726
commit 0d881cd726
parent 2eb44c95d9
3 changed files with 468 additions and 0 deletions
--- a/2026/2026-09-15--goa-ansible-merchant-first.md
+++ b/2026/2026-09-15--goa-ansible-merchant-first.md
@ -0,0 +1,124 @@
+# 2026-09-15 — GOA Ansible Merchant-first
+
+**Freigabe:** Live-Stop / Caddy = nein bis Chat-OK.  
+**Feed:** `taler-deployment` `origin/dev/hernani/regio-goa-explorational-currency`  
+**Work branch:** `dev/hernani/goa-regio-ng-vanilla`
+
+## Naming
+
+| | |
+|--|--|
+| **vanilla** | Ansible Vollkomposition (Merchant+Exchange+Bank+nginx/pg); GOA-Ersatz-Kandidat |
+| **local** | Lab only: loopback / `goa.local` |
+
+`configs/lab-local/goa-regio-ng` = **vanilla** @ **local**. Nie `REGIO_PUBLISH_KOOPA_PORTS=1` neben Live.
+
+## Ist (lab green)
+
+```bash
+ssh hernani@koopa 'podman ps --format "{{.Names}} {{.Status}} {{.Ports}}" | grep -E "goa-regio|taler-hacktivism"'
+# taler-hacktivism* Up  9010–9015
+# goa-regio-ng      Up  127.0.0.1:9180→80, 127.0.0.1:9322→22
+
+ssh hernani@koopa "podman exec goa-regio-ng sh -c '
+  for u in nginx postgresql libeufin-bank taler-merchant-httpd taler-exchange-httpd taler-merchant.target; do
+    printf \"%s=\"; systemctl is-active \$u
+  done'"
+# all=active
+
+ssh hernani@koopa 'for h in bank.goa.local exchange.goa.local backend.goa.local; do
+  curl -sS -o /dev/null -w "$h %{http_code}\n" -H "Host: $h" http://127.0.0.1:9180/config
+done'
+# all 200
+
+ssh hernani@koopa 'curl -sS -H "Host: exchange.goa.local" http://127.0.0.1:9180/keys' \
+  | python3 -c 'import sys,json;d=json.load(sys.stdin);print(d["master_public_key"],d["currency"],len(d.get("denominations",[])))'
+# TW6K5… GOA 27
+```
+
+| Piece | Live | Lab vanilla@local |
+|-------|------|-------------------|
+| Merchant | :9010 | **active** · instance `default` |
+| Exchange | :9011 | **active** · master TW6K5 · 27 denoms |
+| Bank | :9012 | libeufin-bank OK |
+| MASTER | TW6K5… | sha256 match enc `current/exchange-offline/master.priv` |
+
+Playbook: `ok=92 changed=20 failed=0` after `/etc/hosts` pin (bank.goa.local NXDOMAIN once).
+
+## Lab hosts (ephemeral)
+
+Container recreate drops hosts. Re-pin:
+
+```bash
+cd $HOME/git/admin-logs/local/koopa-admin-log/configs/lab-local
+ansible-playbook -i goa-regio-ng-koopa.yml lab-hosts.yml
+# unsafe_writes — /etc/hosts EBUSY on atomic rename inside podman
+```
+
+## Merchant token (lab)
+
+SoT: `koopa-admin-secrets-enc/backups/goa/lab-local/merchant/secrets/merchant-default-access-token`  
+passwd = PLAIN; Bearer = file value as-is (`secret-token:…`). `/instances/default/private/products` **200**.  
+`/config` `"currency":"CHF"` = tip dual-merchant; GOA under `currencies.GOA`.
+
+## t2 — wire + order + withdraw + pay (lab) **done**
+
+Bank acct `lab-merchant` via `libeufin-bank create-account`; secrets under `…/lab-local/merchant/secrets/`. Wire `x-taler-bank`.
+
+```bash
+# POST /instances/default/private/accounts → 200 salt/h_wire
+# POST orders GOA:1 → 200 unpaid
+# create-token labwallet; POST withdrawals; wallet-cli withdraw; POST confirm → 204
+# handle-uri --yes "$taler_pay_uri"
+```
+
+| | |
+|--|--|
+| WID | `1eab9b0d-a3a8-41a8-b070-d7645b25ab25` |
+| withdraw | `txn:withdrawal:2EBFWF7GK415A151KP56MWCVK2CYMDHMR55Q9BG11VT6NR55YC80` final |
+| pay | `txn:payment:D4H75B5YDNJ52G8T6ADBDCXVWXCVZBG5HVSGE7G4S0XBQ6K8H3Y0` done |
+| order | `2026.257-6E75WGDBR32M4` **paid** |
+| wallet | GOA:9 (GOA:10 − GOA:1); CLI 1.6.43 **inside** `goa-regio-ng` |
+
+## Zielbild
+
+Ansible = SoT. Zwei Formen, gleiche Playbooks: **Full-stack Podman** · **Recreate-Fabrik**.  
+Branch: **master = stage**; anderer = **prod**. Kein `STAGE_ENV`.  
+**Reihenfolge:** **1 GOA** → **2 FP/Stage** (zwei Stacks) → **3 mytops-***. Merchant-Probleme → mytops vorziehen.  
+Gate: elelmon moderate → purse/withdraw → lab harden → Live-Merchant cutover nur wenn 1–3 grün → restore-back. Greenfield `*.2.*` PARK.
+
+## APPLY 20260915T0013Z
+
+BACKUP+APT+COMMIT OK. APPLY_SMOKE_FAIL was `sm=$(smoke)` capture; driver now `if smoke; then` → **APPLY_OK**. Smoke endpoints 200. Live bank/exchange/merchant Up; lab `goa-regio-ng` Up. Images retagged `:live` / `:landing`.
+
+## Arbeit 1 — live confirm overflow (2026-09-16) **FAIL**
+
+Smoke: `$HOME/git/llm/llm-work-input/goa-confirm-smoke.sh` (fresh scratch wallet; WCLI `$HOME/.local/bin/taler-wallet-cli.mjs`).
+
+| | |
+|--|--|
+| user | `cfm18923` 2026-09-16T00:35:23Z |
+| max_debt | acct#1=1e12 · **#2=1e8** · #3=1e12 |
+| credit | GOA:4798.5 **200** |
+| TOS | `exchanges add` + `update` + `accept-tos` **OK** |
+| GOA:20 WID | `062e3112-e235-4fea-95d1-644773100224` |
+| create | body `amount` + `exchange_url` → **200** |
+| force-select | rpub + epayto from `/keys` → **200** `selected` |
+| confirm | POST `/accounts/{user}/withdrawals/{wid}/confirm` → **500** code **5110** P0001 `amount value overflowed` |
+| GOA:4200 | same scratch rpub → **409** code **5114** Reserve pub already used (script; skip until GOA:20 green) |
+
+`CONFIRM_SMOKE_FAIL`. Select+TOS+RPUB green for GOA:20; overflow still at confirm. Next: dump `make_incoming` / `bank_wire_transfer` for that WID; lower remaining 1e12 if dump confirms.
+
+## Pointers
+
+| | |
+|--|--|
+| inventory | `configs/lab-local/goa-regio-ng-koopa.yml` |
+| lab hosts | `configs/lab-local/lab-hosts.yml` |
+| Lab README | `configs/lab-local/README.md` |
+| backup daylog | `2026/2026-09-04--goa-backup-regio-ng.md` |
+| APPLY driver | `scripts/taler-shared/goa-live-backup-upgrade.sh` |
+| dumps | koopa `~/backups/goa/current` → `20260904T1238Z` |
+| Caddy | taler→9010 · exchange→9011 · bank→9012 |
+
+Leave-off: t1+t2 lab green. Live confirm **5110**. No Live-Stop / no Caddy.
--- a/configs/lab-local/lab-hosts.yml
+++ b/configs/lab-local/lab-hosts.yml
@ -0,0 +1,19 @@
+---
+# lab-only: pin *.goa.local → 127.0.0.1 inside goa-regio-ng
+# ephemeral on container recreate — re-run after recreate / before site.yml wait-on-bank
+# inventory: goa-regio-ng-koopa.yml (same SSH ProxyCommand)
+#
+#   ansible-playbook -i goa-regio-ng-koopa.yml lab-hosts.yml
+
+- name: lab hosts goa.local
+  hosts: regional_currency
+  gather_facts: false
+  tasks:
+    - name: /etc/hosts *.goa.local → 127.0.0.1
+      ansible.builtin.blockinfile:
+        path: /etc/hosts
+        marker: "# {mark} ANSIBLE LAB goa.local"
+        # podman systemd: /etc/hosts often EBUSY on atomic rename
+        unsafe_writes: true
+        block: |
+          127.0.0.1 bank.goa.local exchange.goa.local backend.goa.local goa.local
--- a/scripts/taler-shared/goa-live-backup-upgrade.sh
+++ b/scripts/taler-shared/goa-live-backup-upgrade.sh
@ -0,0 +1,325 @@
+#!/usr/bin/env bash
+# Live GOA: dry-run → backup → apt upgrade (bank/exchange/merchant) → podman commit.
+# Host: koopa (hernani, podman). No Freigabe-bypass — caller must have Freigabe.
+#
+#   ./goa-live-backup-upgrade.sh dry-run
+#   ./goa-live-backup-upgrade.sh apply          # backup + apt + commit + smoke
+#   ./goa-live-backup-upgrade.sh status
+#
+# State/log: ~/backups/goa/upgrade-runs/<stamp>/
+# Mirror: koopa-admin-log/scripts/taler-shared/
+set -euo pipefail
+
+BANK_CTR="${BANK_CTR:-taler-hacktivism-bank}"
+EXCHANGE_CTR="${EXCHANGE_CTR:-taler-hacktivism-exchange-ansible}"
+MERCHANT_CTR="${MERCHANT_CTR:-taler-hacktivism}"
+
+BANK_IMG_LIVE="${BANK_IMG_LIVE:-localhost/taler-hacktivism-banking:live}"
+EXCHANGE_IMG_LIVE="${EXCHANGE_IMG_LIVE:-localhost/taler-hacktivism-exchange-ansible:landing}"
+MERCHANT_IMG_LIVE="${MERCHANT_IMG_LIVE:-localhost/taler-hacktivism-live:landing}"
+
+STAMP="${STAMP:-$(date -u +%Y%m%dT%H%MZ)}"
+ROOT="${ROOT:-$HOME/backups/goa}"
+RUN_DIR="${RUN_DIR:-$ROOT/upgrade-runs/$STAMP}"
+MODE="${1:-dry-run}"
+
+mkdir -p "$RUN_DIR"
+LOG="$RUN_DIR/run.log"
+STATUS="$RUN_DIR/STATUS"
+exec > >(tee -a "$LOG") 2>&1
+
+log() { printf '%s %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*"; }
+set_status() {
+  printf 'STATUS=%s\nSTAMP=%s\nWHEN=%s\nMODE=%s\n' "$1" "$STAMP" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$MODE" >"$STATUS"
+  shift || true
+  for kv in "$@"; do printf '%s\n' "$kv" >>"$STATUS"; done
+  ln -sfn "$STAMP" "$ROOT/upgrade-runs/current" 2>/dev/null || true
+  ln -sfn "upgrade-runs/$STAMP" "$ROOT/upgrade-current-run" 2>/dev/null || true
+}
+
+pod_root() {
+  local ctr="$1"; shift
+  podman exec -u root "$ctr" bash -lc "$*"
+}
+
+dry_one() {
+  local name="$1" ctr="$2" pkgs="$3"
+  log "=== DRY $name ($ctr) ==="
+  pod_root "$ctr" "
+    set -e
+    export DEBIAN_FRONTEND=noninteractive
+    apt-get update -qq
+    echo '--- installed ---'
+    dpkg -l $pkgs 2>/dev/null | awk '/^ii/{print \$2,\$3}' || true
+    echo '--- candidates (policy sample) ---'
+    for p in $pkgs; do
+      apt-cache policy \"\$p\" 2>/dev/null | awk -v p=\"\$p\" 'NR<=6{print p\": \"\$0}'
+    done
+    echo '--- simulate install ---'
+    apt-get -s install -y $pkgs 2>&1 | egrep -i '^(Inst |Remv |Conf |The following packages)' || true
+  "
+}
+
+cmd_dry_run() {
+  set_status DRY_RUN_RUNNING
+  log "RUN_DIR=$RUN_DIR"
+  df -h "$HOME" | tee "$RUN_DIR/df.txt"
+  podman ps --format '{{.Names}} {{.Status}} {{.Image}}' | grep -E 'taler-hacktivism|goa-regio' | tee "$RUN_DIR/ctrs.txt" || true
+
+  dry_one bank "$BANK_CTR" "libeufin-bank libeufin-common"
+  dry_one exchange "$EXCHANGE_CTR" "taler-exchange taler-exchange-database taler-exchange-offline libtalerexchange taler-terms-generator"
+  dry_one merchant "$MERCHANT_CTR" "taler-merchant taler-merchant-webui libtalermerchant taler-terms-generator"
+
+  cat >"$RUN_DIR/PLAN.txt" <<EOF
+PLAN (apply):
+1. Backup stamp $STAMP under $ROOT/$STAMP + current symlink
+   - bank: etc-libeufin + PG dumps (libeufinbank/libeufin)
+   - exchange: etc-taler-exchange + offline/master.priv + taler-exchange.dump
+   - merchant: etc-taler + merchant DB dump
+2. apt-get install (real) via upgrade-goa-debs logic + start_*.sh restarts
+3. podman commit each ctr → :upgraded-$STAMP then retag live/landing
+   NOTE: no Containerfile rebuild (would wipe overlay). Commit = new image with newer debs.
+4. smoke /config on :9010 :9011 :9012 + public FQDNs
+5. leave STATUS=APPLY_OK for laptop elelmon recheck
+EOF
+  log "PLAN written $RUN_DIR/PLAN.txt"
+  set_status DRY_RUN_OK "NEXT=apply" "LOG=$LOG"
+  log "DRY_RUN_OK"
+}
+
+backup_ctr_bank() {
+  local dest="$1/bank"
+  mkdir -p "$dest"
+  pod_root "$BANK_CTR" '
+    set -e
+    mkdir -p /tmp/goa-bak
+    rm -rf /tmp/goa-bak/*
+    mkdir -p /tmp/goa-bak/etc-libeufin /tmp/goa-bak/var-lib-libeufin-bank
+    cp -a /etc/libeufin/* /tmp/goa-bak/etc-libeufin/ 2>/dev/null || true
+    cp -a /var/lib/libeufin-bank/. /tmp/goa-bak/var-lib-libeufin-bank/ 2>/dev/null || true
+    runuser -u postgres -- psql -lAt > /tmp/goa-bak/pg-databases.txt || true
+    runuser -u postgres -- pg_dump -Fc libeufinbank > /tmp/goa-bak/libeufinbank.dump || \
+      runuser -u postgres -- pg_dump -Fc libeufinbank > /tmp/goa-bak/libeufinbank.dump
+    runuser -u postgres -- pg_dump -Fc libeufin > /tmp/goa-bak/libeufin.dump 2>/dev/null || true
+    du -sh /tmp/goa-bak/* > /tmp/goa-bak/du.txt || true
+    tar -C /tmp/goa-bak -czf /tmp/goa-bank-bak.tgz .
+  '
+  podman cp "$BANK_CTR:/tmp/goa-bank-bak.tgz" "$dest/goa-bank-bak.tgz"
+  pod_root "$BANK_CTR" 'rm -rf /tmp/goa-bak /tmp/goa-bank-bak.tgz'
+}
+
+backup_ctr_exchange() {
+  local dest="$1/exchange"
+  mkdir -p "$dest"
+  pod_root "$EXCHANGE_CTR" '
+    set -e
+    mkdir -p /tmp/goa-bak
+    rm -rf /tmp/goa-bak/*
+    mkdir -p /tmp/goa-bak/etc-taler-exchange /tmp/goa-bak/offline
+    cp -a /etc/taler-exchange/. /tmp/goa-bak/etc-taler-exchange/ 2>/dev/null || true
+    cp -a /var/lib/taler-exchange/offline/. /tmp/goa-bak/offline/ 2>/dev/null || true
+    runuser -u postgres -- psql -lAt > /tmp/goa-bak/pg-databases.txt || true
+    DB=$(runuser -u postgres -- psql -lAt | awk -F"|" "/taler/ {print \$1}" | head -1 | tr -d " ")
+    [ -n "$DB" ] || DB=taler-exchange
+    runuser -u postgres -- pg_dump -Fc "$DB" > /tmp/goa-bak/taler-exchange.dump
+    tar -C /tmp/goa-bak -czf /tmp/goa-exchange-bak.tgz .
+  '
+  podman cp "$EXCHANGE_CTR:/tmp/goa-exchange-bak.tgz" "$dest/goa-exchange-bak.tgz"
+  pod_root "$EXCHANGE_CTR" 'rm -rf /tmp/goa-bak /tmp/goa-exchange-bak.tgz'
+}
+
+backup_ctr_merchant() {
+  local dest="$1/merchant"
+  mkdir -p "$dest"
+  pod_root "$MERCHANT_CTR" '
+    set -e
+    mkdir -p /tmp/goa-bak
+    rm -rf /tmp/goa-bak/*
+    mkdir -p /tmp/goa-bak/etc-taler /tmp/goa-bak/etc-taler-merchant
+    cp -a /etc/taler/. /tmp/goa-bak/etc-taler/ 2>/dev/null || true
+    cp -a /etc/taler-merchant/. /tmp/goa-bak/etc-taler-merchant/ 2>/dev/null || true
+    runuser -u postgres -- psql -lAt > /tmp/goa-bak/pg-databases.txt || true
+    DB=$(runuser -u postgres -- psql -lAt | awk -F"|" "/merchant|taler/ {print \$1}" | head -1 | tr -d " ")
+    [ -n "$DB" ] || DB=taler-merchant
+    runuser -u postgres -- pg_dump -Fc "$DB" > /tmp/goa-bak/taler-merchant.dump
+    tar -C /tmp/goa-bak -czf /tmp/goa-merchant-bak.tgz .
+  '
+  podman cp "$MERCHANT_CTR:/tmp/goa-merchant-bak.tgz" "$dest/goa-merchant-bak.tgz"
+  pod_root "$MERCHANT_CTR" 'rm -rf /tmp/goa-bak /tmp/goa-merchant-bak.tgz'
+}
+
+cmd_backup() {
+  local bak="$ROOT/$STAMP"
+  mkdir -p "$bak/meta"
+  log "BACKUP → $bak"
+  {
+    date -R
+    hostname
+    podman ps -a --format '{{.Names}} | {{.Image}} | {{.Status}}'
+  } >"$bak/meta/inventory.txt"
+  backup_ctr_bank "$bak"
+  backup_ctr_exchange "$bak"
+  backup_ctr_merchant "$bak"
+  du -sh "$bak"/* >"$bak/meta/du.txt" || true
+  find "$bak" -type f -printf '%s %p\n' >"$bak/meta/manifest-sizes.txt" || true
+  ln -sfn "$STAMP" "$ROOT/current"
+  log "BACKUP_OK current→$STAMP"
+  echo "$bak"
+}
+
+upgrade_bank() {
+  log "======== apt bank ========"
+  pod_root "$BANK_CTR" '
+    set -e
+    export DEBIAN_FRONTEND=noninteractive
+    apt-get update -qq
+    apt-get install -y libeufin-bank libeufin-common
+    dpkg -l libeufin-bank libeufin-common | awk "/^ii/{print \$2,\$3}"
+  '
+  pod_root "$BANK_CTR" '
+    set -e
+    if ! runuser -u postgres -- psql -c "SELECT 1" >/dev/null 2>&1; then
+      pg_ctlcluster 17 main start || true
+      sleep 2
+    fi
+    mkdir -p /var/log/libeufin-bank
+    chown libeufin-bank:libeufin-bank /var/log/libeufin-bank 2>/dev/null || true
+    runuser -u libeufin-bank -- /usr/local/bin/start_bank.sh --restart
+  '
+}
+
+upgrade_exchange() {
+  log "======== apt exchange ========"
+  pod_root "$EXCHANGE_CTR" '
+    set -e
+    export DEBIAN_FRONTEND=noninteractive
+    apt-get update -qq
+    apt-get install -y \
+      taler-exchange taler-exchange-database taler-exchange-offline \
+      taler-exchange-typst libtalerexchange taler-terms-generator \
+      taler-exchange-aml-webui taler-exchange-kyc-webui || true
+    dpkg -l "taler-exchange*" "libtalerexchange" "taler-terms-generator" 2>/dev/null \
+      | awk "/^ii/{print \$2,\$3}"
+  '
+  pod_root "$EXCHANGE_CTR" '
+    set -e
+    if id taler-exchange-httpd >/dev/null 2>&1; then
+      runuser -u taler-exchange-httpd -- taler-exchange-dbinit 2>&1 | tail -20 || true
+    fi
+    if [ -x /root/start_base_services_for_taler_exchange.sh ]; then
+      /root/start_base_services_for_taler_exchange.sh --no-shell 2>/dev/null || true
+    fi
+    if [ -x /usr/local/bin/start_exchange.sh ]; then
+      runuser -u taler-exchange-httpd -- /usr/local/bin/start_exchange.sh --restart 2>&1 | tail -30 || true
+    fi
+    pgrep -a taler-exchange-httpd | head -3 || true
+  '
+}
+
+upgrade_merchant() {
+  log "======== apt merchant ========"
+  pod_root "$MERCHANT_CTR" '
+    set -e
+    export DEBIAN_FRONTEND=noninteractive
+    apt-get update -qq
+    apt-get install -y \
+      taler-merchant taler-merchant-webui taler-merchant-typst \
+      libtalermerchant taler-terms-generator
+    dpkg -l "taler-merchant*" "libtalermerchant" 2>/dev/null | awk "/^ii/{print \$2,\$3}"
+  '
+  pod_root "$MERCHANT_CTR" '
+    set -e
+    if id taler-merchant-httpd >/dev/null 2>&1; then
+      runuser -u taler-merchant-httpd -- taler-merchant-dbinit 2>&1 | tail -20 || true
+    fi
+    runuser -u taler-merchant-httpd -- /usr/local/bin/start_merchant.sh --restart
+  '
+}
+
+commit_images() {
+  log "======== podman commit ========"
+  local tag="upgraded-$STAMP"
+  podman commit "$BANK_CTR" "localhost/taler-hacktivism-banking:$tag"
+  podman tag "localhost/taler-hacktivism-banking:$tag" "$BANK_IMG_LIVE"
+  podman commit "$EXCHANGE_CTR" "localhost/taler-hacktivism-exchange-ansible:$tag"
+  podman tag "localhost/taler-hacktivism-exchange-ansible:$tag" "$EXCHANGE_IMG_LIVE"
+  podman commit "$MERCHANT_CTR" "localhost/taler-hacktivism-live:$tag"
+  podman tag "localhost/taler-hacktivism-live:$tag" "$MERCHANT_IMG_LIVE"
+  podman images --format '{{.Repository}}:{{.Tag}} {{.ID}} {{.Size}}' \
+    | grep -E "upgraded-$STAMP|:live|:landing" | tee "$RUN_DIR/images-after.txt" || true
+}
+
+smoke() {
+  # Return 0=ok / 1=fail. Do NOT echo ok into stdout for capture:
+  # with exec>tee, sm=$(smoke) would swallow log lines and never equal "1".
+  log "======== smoke ========"
+  local ok=1
+  for pair in "bank:9012:http" "exchange:9011:http" "merchant:9010:https"; do
+    IFS=: read -r name port sch <<<"$pair"
+    code=$(curl -skS -m 10 -o /dev/null -w '%{http_code}' "${sch}://127.0.0.1:${port}/config" || echo 000)
+    log "loopback $name :$port → $code"
+    [ "$code" = "200" ] || ok=0
+  done
+  for url in \
+    https://bank.hacktivism.ch/config \
+    https://exchange.hacktivism.ch/config \
+    https://taler.hacktivism.ch/config; do
+    code=$(curl -skS -m 15 -o /dev/null -w '%{http_code}' "$url" || echo 000)
+    log "public $url → $code"
+    [ "$code" = "200" ] || ok=0
+  done
+  [ "$ok" = "1" ]
+}
+
+versions_snap() {
+  log "======== versions after ========"
+  pod_root "$BANK_CTR" 'dpkg -l libeufin-bank libeufin-common | awk "/^ii/{print \$2,\$3}"' | tee "$RUN_DIR/ver-bank.txt"
+  pod_root "$EXCHANGE_CTR" 'dpkg -l "taler-exchange" "libtalerexchange" 2>/dev/null | awk "/^ii/{print \$2,\$3}"' | tee "$RUN_DIR/ver-exchange.txt"
+  pod_root "$MERCHANT_CTR" 'dpkg -l "taler-merchant" "taler-merchant-webui" "libtalermerchant" 2>/dev/null | awk "/^ii/{print \$2,\$3}"' | tee "$RUN_DIR/ver-merchant.txt"
+}
+
+cmd_apply() {
+  set_status APPLY_RUNNING
+  cmd_backup
+  set_status APPLY_BACKUP_OK "BAK=$ROOT/$STAMP"
+  upgrade_bank
+  upgrade_exchange
+  upgrade_merchant
+  set_status APPLY_APT_OK
+  commit_images
+  set_status APPLY_COMMIT_OK
+  versions_snap
+  if smoke; then
+    set_status APPLY_OK "NEXT=elelmon-moderate" "BAK=$ROOT/$STAMP" "LOG=$LOG"
+    log "APPLY_OK — next: elelmon moderate from laptop"
+  else
+    set_status APPLY_SMOKE_FAIL "NEXT=triage-smoke" "BAK=$ROOT/$STAMP" "LOG=$LOG"
+    log "APPLY_SMOKE_FAIL"
+    exit 1
+  fi
+}
+
+cmd_status() {
+  local cur="$ROOT/upgrade-runs/current"
+  if [ -L "$cur" ] || [ -d "$cur" ]; then
+    echo "CURRENT_RUN=$(readlink -f "$ROOT/upgrade-runs/current" 2>/dev/null || echo "$cur")"
+    cat "$ROOT/upgrade-runs/current/STATUS" 2>/dev/null || cat "$cur/STATUS" 2>/dev/null || echo 'no STATUS'
+  else
+    echo 'NO_RUN'
+  fi
+  ls -la "$ROOT/upgrade-runs/" 2>/dev/null | tail -10 || true
+}
+
+case "$MODE" in
+  dry-run|dry) cmd_dry_run ;;
+  apply) cmd_apply ;;
+  status) cmd_status ;;
+  -h|--help|help)
+    sed -n '2,16p' "$0"
+    exit 0
+    ;;
+  *)
+    echo "usage: $0 dry-run|apply|status" >&2
+    exit 2
+    ;;
+esac