release 1.9.3: monpages code 21 always ERROR; suite apply script + absolute paths

README.md

@@ -414,7 +414,7 @@ Soft mode only with `MONPAGES_REQUIRE_PUBLIC=0` (escape hatch).
 ./taler-monitoring.sh -d lefrancpaysan.ch monpages
 ```
 
-If pages are missing publicly: `sudo ~/koopa-caddy/apply-monitoring-live.sh` on koopa
+If pages are missing publicly: `sudo /home/hernani/koopa-caddy/apply-monitoring-live.sh` on koopa
 (FP: Infomaniak vhost for `/monitoring*`).
 
 ## Version on monitoring pages (v1.3+ / sticky bar)

VERSION

@@ -1 +1 @@
-1.9.2
+1.9.3

VERSIONS.md

@@ -17,6 +17,7 @@ Git tags: `vMAJOR.FEATURE.FIX` (e.g. `v1.8.0`). File `VERSION` omits the `v` pre
 
 | Tag | Date (UTC) | Notes |
 |-----|------------|--------|
+| **v1.9.3** | 2026-07-19 | **Bugfix (suite):** monpages merchant **code 21** always ERROR (bare no longer soft-WARN); apply hints use absolute `/home/hernani/koopa-caddy/…`; ship `host-agent/apply-monitoring-live.sh` (smoke array, bare check); fix ROOT-ON-KOOPA wrong `redir /path/ 302` examples. |
 | **v1.9.2** | 2026-07-19 | **Bugfix:** Caddy bare-path redir footgun — inside `handle` use `redir * /path/ 302` (not `redir /path/ 302`, which becomes `Location: 302` and bare URLs fall through to merchant **code 21**). Snippet + ROOT-APPLY: absolute apply path (`/home/hernani/koopa-caddy/…`, not `~` as root). |
 | **v1.9.1** | 2026-07-19 | **Bugfix:** stale path confusions in repo — sticky `pages_i3` no longer lists mail/mattermost/aptdeploy mon pages; lib.sh/android-test drop `scripts/taler-monitoring/`; monpages/host-agent hints match v1.8 layout; clarify GIT vs GUI vs CLI `*AUTOMATION-NOTES.md` (three files by design). |
 | **v1.9.0** | 2026-07-19 | **monpages content:** ERROR if page missing or markers incomplete — top (sticky-bar/status-bar/generated) + bottom (footer/`taler-mon:bottom`); mid-run (progress <100% / incomplete meta) relaxes bottom only. Surface required on normal GOA runs with meaningful body (not progress-only). HTML emits `taler-mon:top` / `taler-mon:bottom:complete|incomplete`. |

check_monitoring_pages.sh

@@ -380,14 +380,10 @@ check_one() {
   fi
   if grep -qE '"code"[[:space:]]*:[[:space:]]*21' "$body" 2>/dev/null \
     || grep -qiE 'There is no endpoint defined for the URL' "$body" 2>/dev/null; then
-    if [ "$soft" = "1" ]; then
+    # Merchant code 21 is always ERROR (even bare): slash-only soft mode used to hide
-      warn "public mon page bare URL" \
+    # Caddy redir footgun (redir /path/ 302 → Location:302). Bare must 302/200 HTML.
-        "$url -> HTTP $code code 21 (prefer trailing slash; set MONPAGES_BARE_STRICT=1 to ERROR)"
-      rm -f "$body"
-      return 0
-    fi
     _mon_fail_or_soft "public mon page missing" \
-      "$url -> HTTP $code merchant/API JSON code 21 (publish HTML + reverse-proxy handle /monitoring*)" || { rm -f "$body"; return 1; }
+      "$url -> HTTP $code merchant/API JSON code 21 (Caddy handle/redir; use redir * /path/ 302 inside handle; apply: sudo /home/hernani/koopa-caddy/apply-monitoring-live.sh)" || { rm -f "$body"; return 1; }
     rm -f "$body"
     return 0
   fi
@@ -447,10 +443,11 @@ while IFS= read -r line; do
   URLS+=("$line")
 done < <(build_urls | sort -u)
 
-# Bare URLs without trailing slash: optional soft check (default on).
+# Bare URLs without trailing slash: checked by default.
-# Many reverse proxies only redirect /monitoring → /monitoring/; bare may hit the app (code 21).
+# Merchant JSON code 21 is always ERROR (Caddy/proxy not serving mon path).
+# Other bare failures: WARN unless MONPAGES_BARE_STRICT=1 (then ERROR).
 # MONPAGES_CHECK_BARE=0  → skip bare entirely
-# MONPAGES_BARE_STRICT=1 → bare failures are ERROR (default: WARN if slash form exists)
+# MONPAGES_BARE_STRICT=1 → non-21 bare failures are ERROR too
 if [ "${MONPAGES_CHECK_BARE:-1}" = "1" ]; then
   _extra=()
   for u in "${URLS[@]}"; do
@@ -486,7 +483,8 @@ done
 if [ "$ec" -ne 0 ]; then
   echo "  hint: public monitoring HTML not served (404 / merchant code 21)"
   echo "        publish staging HTML + configure reverse-proxy/Caddy handle for mon paths"
-  echo "        (on koopa: sudo ~/koopa-caddy/apply-monitoring-live.sh)"
+  echo "        (on koopa: sudo /home/hernani/koopa-caddy/apply-monitoring-live.sh)"
+  echo "        bare code 21: Caddy redir inside handle must be: redir * /path/ 302 (not redir /path/ 302)"
   if [ "$_fam" = "fp" ]; then
     echo "        FP: only FP mon hosts are checked — wire Infomaniak vhost for /monitoring*"
   fi

host-agent/apply-monitoring-live.sh

@@ -0,0 +1,91 @@
+#!/usr/bin/env bash
+# ONE-SHOT as root: publish monitoring HTML + enable Caddy handles
+#
+# Suite copy (taler-monitoring host-agent). Live install path on koopa:
+#   /home/hernani/koopa-caddy/apply-monitoring-live.sh
+# Prefer absolute path ( ~ expands to /root when already root ):
+#   sudo /home/hernani/koopa-caddy/apply-monitoring-live.sh
+#   /home/hernani/koopa-caddy/apply-monitoring-live.sh   # if already root
+#
+# After editing here: rsync to koopa ~/koopa-caddy/ before root apply.
+set -euo pipefail
+if [[ "$(id -u)" -ne 0 ]]; then
+  exec sudo -E "$0" "$@"
+fi
+
+STAGING=/home/hernani/monitoring-sites-staging
+WWW=/var/www/monitoring-sites
+SRC=/home/hernani/koopa-caddy/Caddyfile
+DST=/etc/caddy/Caddyfile
+
+if [[ ! -f "$SRC" ]]; then
+  echo "ERROR: missing Caddyfile source: $SRC" >&2
+  exit 1
+fi
+if [[ ! -d "$STAGING" ]]; then
+  echo "ERROR: missing staging HTML: $STAGING" >&2
+  exit 1
+fi
+
+echo "== 1) HTML staging → www =="
+mkdir -p "$WWW"
+rsync -a --delete "$STAGING/" "$WWW/"
+# allow hernani to update future runs without root
+chown -R hernani:caddy "$WWW"
+chmod -R g+rwX "$WWW"
+find "$WWW" -type d -exec chmod g+s {} \;
+# ACL default if available
+if command -v setfacl >/dev/null 2>&1; then
+  setfacl -R -m u:hernani:rwx,g:caddy:rwx "$WWW" || true
+  setfacl -R -d -m u:hernani:rwx,g:caddy:rwx "$WWW" || true
+fi
+ls -la "$WWW" "$WWW/taler.hacktivism.ch" || true
+
+echo "== 2) Caddyfile (backup + install) =="
+ts=$(date +%Y%m%d-%H%M%S)
+cp -a "$DST" "/etc/caddy/Caddyfile.bak-monitoring-${ts}"
+cp -a "$SRC" "$DST"
+chown root:caddy "$DST" 2>/dev/null || chown root:root "$DST"
+chmod 644 "$DST"
+caddy validate --config "$DST"
+systemctl reload caddy
+systemctl is-active caddy
+
+echo "== 3) smoke =="
+# bash: every continued line except the last must end with \
+urls=(
+  https://taler.hacktivism.ch/monitoring/
+  https://bank.hacktivism.ch/monitoring/
+  https://exchange.hacktivism.ch/monitoring/
+  https://taler.hacktivism.ch/taler-monitoring-surface
+  https://taler.hacktivism.ch/taler-monitoring-surface/
+  https://taler.hacktivism.ch/taler-monitoring-surface_err/
+  https://taler.hacktivism.ch/taler-monitoring-aptdeploy/
+  https://taler.hacktivism.ch/taler-monitoring-aptdeploy_err/
+)
+smoke_fail=0
+for url in "${urls[@]}"; do
+  code=$(curl -sS -o /tmp/monchk -w '%{http_code}' -L --max-redirs 3 -m 12 "$url" || echo ERR)
+  ct=$(file -b /tmp/monchk 2>/dev/null | head -c 40)
+  echo "  $code  $url  ($ct)"
+  if grep -qE '"code":[[:space:]]*21' /tmp/monchk 2>/dev/null; then
+    echo "    STILL merchant JSON code 21 — Caddy handles not active?"
+    smoke_fail=1
+  fi
+done
+# bare path must redirect (or land on HTML), not merchant JSON
+bare_code=$(curl -sS -o /tmp/monbare -w '%{http_code}' --max-redirs 0 -m 12 \
+  https://taler.hacktivism.ch/taler-monitoring-surface || echo ERR)
+if grep -qE '"code":[[:space:]]*21' /tmp/monbare 2>/dev/null; then
+  echo "ERROR: bare /taler-monitoring-surface still merchant code 21 (HTTP $bare_code)" >&2
+  smoke_fail=1
+elif [[ "$bare_code" != "302" && "$bare_code" != "301" && "$bare_code" != "200" ]]; then
+  echo "WARN: bare /taler-monitoring-surface HTTP $bare_code (want 302/301/200 HTML)" >&2
+fi
+echo "  bare_no_follow=$bare_code  https://taler.hacktivism.ch/taler-monitoring-surface"
+
+if [[ "$smoke_fail" -ne 0 ]]; then
+  echo "ERROR: smoke failed" >&2
+  exit 1
+fi
+echo "OK apply-monitoring-live done"

host-agent/run-host-report.sh

@@ -401,11 +401,11 @@ if [ -n "$DEPLOY_WWW" ]; then
   elif [ -e "$DEPLOY_WWW" ]; then
     echo "ERROR: DEPLOY_WWW_ROOT=$DEPLOY_WWW exists but is not writable by $(id -un)" >&2
     echo "       public FQDN pages will stay missing/stale — fix ownership or run:" >&2
-    echo "       sudo ~/koopa-caddy/apply-monitoring-live.sh" >&2
+    echo "       sudo /home/hernani/koopa-caddy/apply-monitoring-live.sh" >&2
     ec=1
   else
     echo "ERROR: DEPLOY_WWW_ROOT=$DEPLOY_WWW missing — public pages not published" >&2
-    echo "       create tree + Caddy handles: sudo ~/koopa-caddy/apply-monitoring-live.sh" >&2
+    echo "       create tree + Caddy handles: sudo /home/hernani/koopa-caddy/apply-monitoring-live.sh" >&2
     ec=1
   fi
 else
@@ -437,7 +437,7 @@ if [ "$_monpages_failed" = "1" ]; then
   {
     echo ""
     echo "ERROR monpages: public FQDN monitoring HTML missing or merchant JSON code 21"
-    echo "ERROR monpages: fix with sudo ~/koopa-caddy/apply-monitoring-live.sh (www + Caddy handles)"
+    echo "ERROR monpages: fix with sudo /home/hernani/koopa-caddy/apply-monitoring-live.sh (www + Caddy handles)"
   } >>"$LOG"
   for host in $MON_HOSTS; do
     htmlify_host "$host"

site-gen/ROOT-ON-KOOPA.md

@@ -3,31 +3,33 @@
 Host Caddy runs as **root/systemd** (`caddy` user). Static monitoring HTML is
 **not** put into Taler containers — only host paths + Caddy `handle`.
 
-After `generate-monitoring-sites.sh` + `deploy-monitoring-sites.sh` (as hernani),
+After host-agent / `generate-monitoring-sites.sh` + deploy (as hernani), files live in:
-files live in:
 
 ```text
 /home/hernani/monitoring-sites-staging/<hostname>/monitoring/index.html
 /home/hernani/monitoring-sites-staging/<hostname>/monitoring_err/index.html
 ```
 
-## One-shot as root (copy + Caddy)
+## One-shot as root (preferred)
 
 ```bash
-# on koopa (koopa-external), as root:
+# absolute path — as root, ~ is /root (script not found under ~/koopa-caddy/…)
+sudo /home/hernani/koopa-caddy/apply-monitoring-live.sh
+# same script is versioned in the suite:
+#   host-agent/apply-monitoring-live.sh  → install/sync to /home/hernani/koopa-caddy/
+```
 
-# 1) web root
+That rsyncs staging → `/var/www/monitoring-sites`, installs the prepared
-install -d -o caddy -g caddy -m 755 /var/www/monitoring-sites
+Caddyfile, reloads Caddy, and smokes mon URLs (including bare surface).
+
+## Manual equivalent
+
+```bash
+install -d -o hernani -g caddy -m 755 /var/www/monitoring-sites
 rsync -a --delete /home/hernani/monitoring-sites-staging/ /var/www/monitoring-sites/
-chown -R caddy:caddy /var/www/monitoring-sites
+chown -R hernani:caddy /var/www/monitoring-sites
-
-# 2) Caddyfile — from ~/koopa-caddy (host ops), then:
-#    (either edit /etc/caddy/Caddyfile by hand using snippet below,
-#     or copy full mirror after review)
-#
-#    install -m 644 /home/hernani/koopa-caddy/Caddyfile /etc/caddy/Caddyfile
-#    # or: ~/koopa-caddy/apply.sh  after syncing Caddyfile into ~/koopa-caddy/
 
+install -m 644 /home/hernani/koopa-caddy/Caddyfile /etc/caddy/Caddyfile
 caddy validate --config /etc/caddy/Caddyfile
 systemctl reload caddy
 systemctl is-active caddy
@@ -35,23 +37,10 @@ systemctl is-active caddy
 
 ## Snippet per hacktivism site (inside each `*.hacktivism.ch { }` block)
 
-Place **before** the catch-all `reverse_proxy` (same idea as `/intro*`):
+Place **before** the catch-all `reverse_proxy`. Canonical copy:
+`site-gen/caddy-monitoring-handles.snippet`.
 
-```caddy
+Disk layout (path on disk matches URL under host root):
-    # Public taler-monitoring console HTML (static; host only)
-    handle /monitoring_err* {
-        root * /var/www/monitoring-sites/{host}
-        rewrite * /monitoring_err{uri}
-        # uri is /monitoring_err or /monitoring_err/ → serve directory index
-        file_server
-    }
-    handle /monitoring* {
-        root * /var/www/monitoring-sites/{host}
-        file_server
-    }
-```
-
-Simpler layout (recommended): path on disk matches URL under host root:
 
 ```text
 /var/www/monitoring-sites/bank.hacktivism.ch/monitoring/index.html
@@ -59,34 +48,50 @@ Simpler layout (recommended): path on disk matches URL under host root:
 ```
 
 ```caddy
-    handle_path /monitoring_err/* {
+    # bare → slash (MUST use redir * /path/ — see footgun below)
-        root * /var/www/monitoring-sites/{host}/monitoring_err
+    handle /monitoring {
+        redir * /monitoring/ 302
+    }
+    handle /monitoring* {
+        root * /var/www/monitoring-sites/{host}
         file_server
     }
     handle /monitoring_err {
-        redir /monitoring_err/ 302
+        redir * /monitoring_err/ 302
     }
-    handle_path /monitoring/* {
+    handle /monitoring_err* {
-        root * /var/www/monitoring-sites/{host}/monitoring
+        root * /var/www/monitoring-sites/{host}
         file_server
     }
-    handle /monitoring {
-        redir /monitoring/ 302
-    }
 ```
 
-Canonical snippet is also in `caddy-monitoring-handles.snippet` and applied
+### Caddy redir footgun (merchant code 21)
-in `configs/caddy/Caddyfile` for the three GOA hosts.
+
+```caddy
+# WRONG — parsed as matcher=/monitoring/ to="302" → Location: 302
+handle /monitoring {
+    redir /monitoring/ 302
+}
+
+# RIGHT
+handle /monitoring {
+    redir * /monitoring/ 302
+}
+```
+
+Symptom: `https://…/monitoring` or `…/taler-monitoring-surface` returns merchant
+JSON `{"code":21,…}` while the slash form serves HTML.
 
 ## Check
 
 ```bash
 curl -sS -o /dev/null -w '%{http_code}\n' https://bank.hacktivism.ch/monitoring/
-curl -sS -o /dev/null -w '%{http_code}\n' https://bank.hacktivism.ch/monitoring_err/
+curl -sS -D- -o /dev/null https://taler.hacktivism.ch/taler-monitoring-surface | head -15
-curl -sS https://bank.hacktivism.ch/monitoring_err/ | head
+# bare: expect 302 Location: /taler-monitoring-surface/  — not JSON code 21
+curl -sS -o /dev/null -w '%{http_code}\n' https://taler.hacktivism.ch/taler-monitoring-surface/
 ```
 
 ## Not needed as root
 
-- Running `taler-monitoring` (use **optional outside runner** (SSH alias via `FIRECUDA_SSH`) / laptop)
+- Running `taler-monitoring` (use **optional outside runner** / laptop)
 - Writing into podman Taler containers