release 1.7.6: monpages obligatory ERROR with stack inventory

GOA monpages checks the full suite catalog (bank/exchange/taler
/monitoring/ plus surface/aptdeploy/mattermost/mail) and on-disk
discovery; FP only its own mon hosts. Default MONPAGES_REQUIRE_PUBLIC=1.
Also fix with_timeout re-running failed phases via || fallback.

README.md

@@ -31,7 +31,7 @@ The **public HTML reports** are not an external addon — they are a first-class
 
 Related:
 
-- **Phase `monpages`:** outside-in check that those FQDNs serve suite HTML (sticky-bar), not merchant JSON code 21.
+- **Phase `monpages`:** obligatory outside-in check that public mon HTML is live (sticky-bar), not merchant JSON code 21 — GOA full inventory, FP only FP hosts.
 - **Sticky “What this monitors”:** each page lists its checks **and** a dedicated block *Monitoring pages themselves (suite)*.
 - **Publish:** staging `~/monitoring-sites-staging` → `DEPLOY_WWW_ROOT` + Caddy handles (`site-gen/caddy-monitoring-handles.snippet`, v1.7.2+).
 
@@ -401,17 +401,25 @@ LADDER_LOAD=0 ./taler-monitoring.sh ladder
 Default **English**. Select language with `--lang en|fr` or `TALER_MON_LANG`.
 FrancPaysan stacks auto-select **French** (`*lefrancpaysan*`). See `VERSIONS.md` and `i18n.sh`.
 
-## Public monitoring pages via FQDN (v1.3.1+)
+## Public monitoring pages via FQDN (v1.3.1+ / obligatory v1.7.6+)
 
-Phase **`monpages`** checks that each host’s monitoring URL is live HTML
+Phase **`monpages`** is **obligatory**: missing or non-suite HTML is **ERROR**
-(not merchant JSON code 21):
+(not merchant JSON code 21, not 404). Soft mode only with
+`MONPAGES_REQUIRE_PUBLIC=0` (escape hatch).
+
+| Stack | What is checked |
+|-------|-----------------|
+| **GOA / hacktivism** | Full suite inventory: `bank`/`exchange`/`taler` `/monitoring/` plus `/taler-monitoring-{surface,aptdeploy,mattermost,mail}/` and any extra pages found under staging/`DEPLOY_WWW_ROOT` |
+| **FrancPaysan** | Only that stack’s mon hosts (`bank`/`exchange`/`monnaie` · stage variants) — never GOA URLs |
 
 ```bash
 ./taler-monitoring.sh -d hacktivism.ch monpages
+./taler-monitoring.sh -d lefrancpaysan.ch monpages
 # host-agent runs monpages by default and again after HTML publish
 ```
 
-If pages are missing publicly: `sudo ~/koopa-caddy/apply-monitoring-live.sh` on koopa.
+If pages are missing publicly: `sudo ~/koopa-caddy/apply-monitoring-live.sh` on koopa
+(FP: Infomaniak vhost for `/monitoring*`).
 
 ## Version on monitoring pages (v1.3+ / sticky bar)
 

TESTS.md

@@ -20,7 +20,7 @@ Every check line has a **global** run number and a **grouped** id:
 | **versions** | `check_versions.sh` | `outside` `inside` `compare` |
 | **aptdeploy** | `check_apt_deploy.sh` | `trixie` `trixie-testing` (koopa podman apt-src merchant smoke) |
 | **surface** | `check_surface.sh` | **remote-only** ecosystem / `-d` domain inventory (not in default/all/full) |
-| **monpages** | `check_monitoring_pages.sh` | public monitoring HTML via FQDN (`https://host/monitoring/`, not merchant code 21) |
+| **monpages** | `check_monitoring_pages.sh` | public monitoring HTML via FQDN — **obligatory ERROR**; GOA full inventory; FP only FP mon hosts |
 | **sanity** | `check_sanity.sh` | `bank` `exchange` `merchant` |
 | **server** | `check_server.sh` | (flat `server-NN` or host groups) |
 | **e2e** | `check_e2e.sh` | `prereq` `load` `bank` `wallet` `atm` `settle` `pay` `shop` `paivana` `dig` `report` |

VERSION

@@ -1 +1 @@
-1.7.5
+1.7.6

VERSIONS.md

@@ -18,6 +18,7 @@ Git tags: `vMAJOR.FEATURE.FIX` (e.g. `v1.7.5`). File `VERSION` omits the `v` pre
 | Tag | Date (UTC) | Notes |
 |-----|------------|--------|
 | **v1.8.0** | *planned* | **L10n** packs for user-facing strings: **fr-CH** (FrancPaysan) and **de-CH** (hacktivism / Taler CH German). |
+| **v1.7.6** | 2026-07-18 | **monpages obligatory (ERROR):** GOA checks full suite inventory (bank/exchange/taler `/monitoring/` + surface/aptdeploy/mattermost/mail + on-disk discovery); FP only its own mon hosts; `MONPAGES_REQUIRE_PUBLIC=1` default (escape hatch `=0`). **Bugfix:** `with_timeout` no longer re-runs a phase when it exits non-zero. |
 | **v1.7.5** | 2026-07-18 | **Bugfix:** monpages soft mode `MONPAGES_REQUIRE_PUBLIC=0` (FP wrappers default) — missing public HTML is WARN until vhost publish; generic publish hints. |
 | **v1.7.4** | 2026-07-18 | **Bugfix:** monpages bare URLs (`/monitoring` without `/`) WARN by default (trailing slash remains ERROR); avoids false red when slash form is live. |
 | **v1.7.3** | 2026-07-18 | Monitoring **pages themselves** as suite scope: README section + sticky “What this monitors” block (monpages / public HTML paths). |
@@ -39,7 +40,7 @@ Git tags: `vMAJOR.FEATURE.FIX` (e.g. `v1.7.5`). File `VERSION` omits the `v` pre
 
 ## Sticky bar version (v1.3.0+)
 
-Generated HTML shows the installed release tag (e.g. `v1.7.5`) and links to Forgejo:
+Generated HTML shows the installed release tag (e.g. `v1.7.6`) and links to Forgejo:
 
 `https://git.hacktivism.ch/hernani/taler-monitoring/src/tag/<tag>`
 
@@ -69,5 +70,5 @@ Full locale packs (fr-CH / de-CH): **v1.8.0** (planned).
 
 ```bash
 git clone https://git.hacktivism.ch/hernani/taler-monitoring.git ~/src/taler-monitoring
-cd ~/src/taler-monitoring && git checkout v1.7.5
+cd ~/src/taler-monitoring && git checkout v1.7.6
 ```

check_monitoring_pages.sh

@@ -4,15 +4,23 @@
 # Outside-in: curl https://<fqdn>/<path>/ and require a real HTML monitoring page.
 # Catches: merchant JSON code 21 (Caddy handle missing), WP 404, empty stubs, deploy skip.
 #
-# Env (same knobs as host-agent HTML layout):
+# Policy (v1.7.6+):
+#   • monpages is obligatory → missing/wrong pages are ERROR (exit 1).
+#   • GOA / hacktivism: full inventory of suite monitoring sites (catalog + on-disk discovery).
+#   • FP: only that stack’s own monitoring hosts/paths (never GOA URLs).
+#   • Override soft mode only with MONPAGES_REQUIRE_PUBLIC=0 (emergency / staging).
+#
+# Env:
 #   MON_HOSTS             space-separated FQDNs (default from TALER_DOMAIN)
 #   HTML_URL_OK           path for OK page (default /monitoring/)
-#   HTML_URL_ERR          optional err path; checked if MONPAGES_CHECK_ERR=1
+#   HTML_URL_ERR          err path; checked if present on disk or MONPAGES_CHECK_ERR=1
-#   MONITORING_PAGE_URLS  optional full URL list (overrides host+path construction)
+#   MONITORING_PAGE_URLS  optional full URL list (overrides inventory construction)
 #   MONPAGES_EXTRA_URLS   additional absolute URLs (space-separated)
+#   MONPAGES_INVENTORY    auto|full|job  (default auto)
+#                         auto/full = stack inventory; job = only MON_HOSTS+paths
+#   MONPAGES_REQUIRE_PUBLIC  default 1 (ERROR). 0 = WARN only (escape hatch).
+#   MONPAGES_STAGING_BASE / HTML_OUT / DEPLOY_WWW_ROOT  trees for discovery
 #   TIMEOUT               curl timeout (from lib.sh)
-#   MONPAGES_STAGING_BASE / HTML_OUT  optional staging root for diagnosis
-#   HTML_OK_DIR           staging subdir (default monitoring)
 #
 set -euo pipefail
 ROOT=$(cd "$(dirname "$0")" && pwd)
@@ -20,10 +28,11 @@ ROOT=$(cd "$(dirname "$0")" && pwd)
 source "$ROOT/lib.sh"
 
 set_area monpages
-section "monpages · public monitoring HTML via FQDN (outside-in)"
+section "monpages · public monitoring HTML via FQDN (outside-in, obligatory)"
 
-# MONPAGES_REQUIRE_PUBLIC=0 → missing public pages are WARN (staging-only stacks / FP until vhost)
+# Obligatory: public pages missing → ERROR. Soft only if explicitly disabled.
 : "${MONPAGES_REQUIRE_PUBLIC:=1}"
+: "${MONPAGES_INVENTORY:=auto}"
 
 # Default hosts from domain (mirror run-host-report.sh)
 if [ -z "${MON_HOSTS:-}" ]; then
@@ -46,31 +55,131 @@ fi
 HTML_URL_OK="${HTML_URL_OK:-/monitoring/}"
 HTML_URL_ERR="${HTML_URL_ERR:-/monitoring_err/}"
 # ensure leading + trailing slash
-case "$HTML_URL_OK" in
+_norm_path() {
+  local p="$1"
+  case "$p" in
     /*) ;;
-  *) HTML_URL_OK="/$HTML_URL_OK" ;;
+    *) p="/$p" ;;
-esac
+  esac
-case "$HTML_URL_OK" in
+  case "$p" in
     */) ;;
-  *) HTML_URL_OK="${HTML_URL_OK}/" ;;
+    *) p="${p}/" ;;
-esac
+  esac
+  printf '%s' "$p"
+}
+HTML_URL_OK="$(_norm_path "$HTML_URL_OK")"
+HTML_URL_ERR="$(_norm_path "$HTML_URL_ERR")"
 
-build_urls() {
+# Stack family: goa | fp | other — FP never checks GOA and vice versa.
+_monpages_family() {
+  case "${TALER_DOMAIN:-}" in
+    *lefrancpaysan*|*francpaysan*) printf 'fp' ;;
+    hacktivism.ch|koopa) printf 'goa' ;;
+    *)
+      # Infer from MON_HOSTS if domain ambiguous
+      case " ${MON_HOSTS:-} " in
+        *lefrancpaysan*) printf 'fp' ;;
+        *hacktivism*) printf 'goa' ;;
+        *) printf 'other' ;;
+      esac
+      ;;
+  esac
+}
+
+_host_allowed() {
+  local h="$1" fam="$2"
+  case "$fam" in
+    goa)
+      case "$h" in *.hacktivism.ch) return 0 ;; *) return 1 ;; esac
+      ;;
+    fp)
+      case "$h" in *lefrancpaysan*) return 0 ;; *) return 1 ;; esac
+      ;;
+    *) return 0 ;;
+  esac
+}
+
+# Emit https://host/path/ for each …/host/path/index.html under root (max depth 3).
+_discover_tree() {
+  local root="$1" fam="$2"
+  local f rel host path
+  [ -n "$root" ] && [ -d "$root" ] || return 0
+  # layout: $root/<fqdn>/<url-path>/index.html
+  while IFS= read -r -d '' f; do
+    rel="${f#"${root%/}"/}"
+    host="${rel%%/*}"
+    path="${rel#*/}"
+    path="${path%/index.html}"
+    [ -n "$host" ] && [ -n "$path" ] || continue
+    _host_allowed "$host" "$fam" || continue
+    printf 'https://%s/%s/\n' "$host" "$path"
+  done < <(find "${root%/}" -mindepth 3 -maxdepth 3 -type f -name index.html -print0 2>/dev/null)
+}
+
+# Suite catalog of OK pages that must exist for this family.
+_catalog_urls() {
+  local fam="$1" h
+  case "$fam" in
+    goa)
+      for h in bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch; do
+        printf 'https://%s/monitoring/\n' "$h"
+      done
+      for h in \
+        taler-monitoring-surface \
+        taler-monitoring-aptdeploy \
+        taler-monitoring-mattermost \
+        taler-monitoring-mail
+      do
+        printf 'https://taler.hacktivism.ch/%s/\n' "$h"
+      done
+      ;;
+    fp)
+      for h in $MON_HOSTS; do
+        [ -z "$h" ] && continue
+        _host_allowed "$h" fp || continue
+        printf 'https://%s/monitoring/\n' "$h"
+      done
+      ;;
+  esac
+}
+
+# Job-local URLs (this host-agent run’s path).
+_job_urls() {
   local h p
-  if [ -n "${MONITORING_PAGE_URLS:-}" ]; then
-    # shellcheck disable=SC2086
-    printf '%s\n' $MONITORING_PAGE_URLS
-  else
   for h in $MON_HOSTS; do
     [ -z "$h" ] && continue
     printf 'https://%s%s\n' "$h" "$HTML_URL_OK"
     if [ "${MONPAGES_CHECK_ERR:-0}" = "1" ]; then
-        p="$HTML_URL_ERR"
+      printf 'https://%s%s\n' "$h" "$HTML_URL_ERR"
-        case "$p" in /*) ;; *) p="/$p" ;; esac
-        case "$p" in */) ;; *) p="${p}/" ;; esac
-        printf 'https://%s%s\n' "$h" "$p"
     fi
   done
+}
+
+build_urls() {
+  local fam inv
+  fam="$(_monpages_family)"
+  inv="${MONPAGES_INVENTORY:-auto}"
+
+  if [ -n "${MONITORING_PAGE_URLS:-}" ]; then
+    # shellcheck disable=SC2086
+    printf '%s\n' $MONITORING_PAGE_URLS
+  else
+    case "$inv" in
+      job)
+        _job_urls
+        ;;
+      full|auto|*)
+        # Always include this job’s paths (host-agent post-check).
+        _job_urls
+        # Stack inventory: GOA = all suite mon sites; FP = only FP mon hosts.
+        if [ "$fam" = "goa" ] || [ "$fam" = "fp" ]; then
+          _catalog_urls "$fam"
+          # On-disk discovery (err pages only exist after failed runs; extras welcome)
+          _discover_tree "${DEPLOY_WWW_ROOT:-/var/www/monitoring-sites}" "$fam"
+          _discover_tree "${MONPAGES_STAGING_BASE:-${HTML_OUT:-${HTML_BASE:-$HOME/monitoring-sites-staging}}}" "$fam"
+        fi
+        ;;
+    esac
   fi
   if [ -n "${MONPAGES_EXTRA_URLS:-}" ]; then
     # shellcheck disable=SC2086
@@ -115,7 +224,6 @@ _is_bare_url() {
   esac
 }
 
-
 _mon_fail_or_soft() {
   local label="$1" detail="$2"
   if [ "${MONPAGES_REQUIRE_PUBLIC:-1}" != "1" ]; then
@@ -182,7 +290,7 @@ while IFS= read -r line; do
   done
   [ "$skip" = "1" ] && continue
   URLS+=("$line")
-done < <(build_urls)
+done < <(build_urls | sort -u)
 
 # Bare URLs without trailing slash: optional soft check (default on).
 # Many reverse proxies only redirect /monitoring → /monitoring/; bare may hit the app (code 21).
@@ -202,7 +310,6 @@ if [ "${MONPAGES_CHECK_BARE:-1}" = "1" ]; then
     done
     [ "$skip" = "1" ] && continue
     URLS+=("$u")
-    # mark bare for soft handling via env list
     MONPAGES_BARE_URLS="${MONPAGES_BARE_URLS:-} $u"
   done
   export MONPAGES_BARE_URLS
@@ -213,6 +320,8 @@ if [ "${#URLS[@]}" -eq 0 ]; then
   exit 1
 fi
 
+_fam="$(_monpages_family)"
+echo "  family=${_fam} inventory=${MONPAGES_INVENTORY} require_public=${MONPAGES_REQUIRE_PUBLIC}"
 echo "  checking ${#URLS[@]} URL(s) via FQDN..."
 ec=0
 for u in "${URLS[@]}"; do
@@ -221,8 +330,14 @@ done
 
 if [ "$ec" -ne 0 ]; then
   echo "  hint: public monitoring HTML not served (404 / merchant code 21)"
-  echo "        publish staging HTML + configure reverse-proxy/Caddy handle for /monitoring*"
+  echo "        publish staging HTML + configure reverse-proxy/Caddy handle for mon paths"
   echo "        (on koopa: sudo ~/koopa-caddy/apply-monitoring-live.sh)"
+  if [ "$_fam" = "fp" ]; then
+    echo "        FP: only FP mon hosts are checked — wire Infomaniak vhost for /monitoring*"
+  fi
+  if [ "$_fam" = "goa" ]; then
+    echo "        GOA: full inventory (bank/exchange/taler /monitoring/ + surface/aptdeploy/mattermost/mail)"
+  fi
   # Staging vs public diagnosis (host-agent sets MONPAGES_STAGING_BASE / HTML_OUT)
   _stg="${MONPAGES_STAGING_BASE:-${HTML_OUT:-${HTML_BASE:-$HOME/monitoring-sites-staging}}}"
   _ok_dir="${HTML_OK_DIR:-monitoring}"

host-agent/README.md

@@ -195,4 +195,4 @@ Set `STRICT_EXIT=0` only if you want timers to stay green while still logging er
 
 ## Monitoring pages (suite)
 
-Public sticky-bar HTML under `/monitoring*` and `/taler-monitoring-*` is part of **taler-monitoring** (not a separate product). See main [README — Monitoring pages](../README.md#monitoring-pages-part-of-the-suite). Phase **monpages** verifies they are live.
+Public sticky-bar HTML under `/monitoring*` and `/taler-monitoring-*` is part of **taler-monitoring** (not a separate product). See main [README — Monitoring pages](../README.md#monitoring-pages-part-of-the-suite). Phase **monpages** is obligatory (**ERROR** if missing): GOA checks the full suite inventory; FP only FP mon hosts.

host-agent/run-fp-prod-monitoring.sh

@@ -15,8 +15,8 @@ export INSIDE_PODMAN="${INSIDE_PODMAN:-1}"
 export INSIDE_MODE="${INSIDE_MODE:-local-podman}"
 export LOCAL_STACK="${LOCAL_STACK:-0}"
 export CONTINUE_ON_ERROR="${CONTINUE_ON_ERROR:-1}"
-# Public /monitoring* not wired on Infomaniak yet — monpages WARN only
+# monpages obligatory (ERROR): only FP mon hosts below — never GOA inventory
-export MONPAGES_REQUIRE_PUBLIC="${MONPAGES_REQUIRE_PUBLIC:-0}"
+export MONPAGES_REQUIRE_PUBLIC="${MONPAGES_REQUIRE_PUBLIC:-1}"
 export RUN_TIMEOUT="${RUN_TIMEOUT:-600}"
 export PHASES="${PHASES:-urls inside versions monpages}"
 export MON_HOSTS="${MON_HOSTS:-bank.lefrancpaysan.ch exchange.lefrancpaysan.ch monnaie.lefrancpaysan.ch}"

host-agent/run-fp-stage-monitoring.sh

@@ -16,8 +16,8 @@ export INSIDE_MODE="${INSIDE_MODE:-local-podman}"
 export INSIDE_PROFILE="${INSIDE_PROFILE:-stage-lfp}"
 export LOCAL_STACK="${LOCAL_STACK:-0}"
 export CONTINUE_ON_ERROR="${CONTINUE_ON_ERROR:-1}"
-# Public /monitoring* not wired on Infomaniak yet — monpages WARN only
+# monpages obligatory (ERROR): only FP stage mon hosts below — never GOA inventory
-export MONPAGES_REQUIRE_PUBLIC="${MONPAGES_REQUIRE_PUBLIC:-0}"
+export MONPAGES_REQUIRE_PUBLIC="${MONPAGES_REQUIRE_PUBLIC:-1}"
 export RUN_TIMEOUT="${RUN_TIMEOUT:-600}"
 export PHASES="${PHASES:-urls inside versions monpages}"
 export MON_HOSTS="${MON_HOSTS:-stage.bank.lefrancpaysan.ch stage.exchange.lefrancpaysan.ch stage.monnaie.lefrancpaysan.ch}"

host-agent/run-host-report.sh

@@ -412,12 +412,16 @@ else
   echo "WARN: DEPLOY_WWW_ROOT empty — skipping live publish (staging only: $HTML_BASE)" >&2
 fi
 
-# Post-publish: verify this job's pages via FQDN (same paths as monpages phase)
+# Post-publish: monpages obligatory (v1.7.6+) — GOA full inventory / FP only FP hosts
 _monpages_failed=0
 if [ "${MONPAGES_POSTCHECK:-1}" = "1" ] && [ -x "$ROOT/check_monitoring_pages.sh" ]; then
-  echo "--- monpages post-check (public FQDN) ---"
+  echo "--- monpages post-check (public FQDN, obligatory) ---"
   export MON_HOSTS HTML_URL_OK HTML_URL_ERR TALER_DOMAIN HTML_OUT HTML_BASE
   export MONPAGES_STAGING_BASE="${HTML_BASE}"
+  export DEPLOY_WWW_ROOT="${DEPLOY_WWW:-${DEPLOY_WWW_ROOT:-}}"
+  export MONPAGES_REQUIRE_PUBLIC="${MONPAGES_REQUIRE_PUBLIC:-1}"
+  # auto: GOA = all suite mon sites; FP = only FP mon hosts; job paths always included
+  export MONPAGES_INVENTORY="${MONPAGES_INVENTORY:-auto}"
   if ! bash "$ROOT/check_monitoring_pages.sh"; then
     echo "ERROR: public monitoring page(s) missing or not suite HTML via FQDN" >&2
     echo "       staging HTML_OUT=${HTML_BASE} · DEPLOY_WWW=${DEPLOY_WWW:-}" >&2

lib.sh

@@ -323,14 +323,25 @@ SSH_BASE_OPTS=(
 # Hard wall-clock timeout so ssh/curl never block the monitoring run forever.
 with_timeout() {
   local secs="$1"; shift
+  local rc
   if command -v gtimeout >/dev/null 2>&1; then
     gtimeout --kill-after=2 "$secs" "$@"
     return $?
   fi
   if command -v timeout >/dev/null 2>&1; then
-    timeout -k 2 "$secs" "$@" 2>/dev/null || timeout --kill-after=2 "$secs" "$@"
+    # Prefer GNU -k; if that flag form is unsupported, fall back once.
+    # Do NOT use `cmd || cmd` on the child — a phase exit 1 would re-run it.
+    set +e
+    timeout -k 2 "$secs" "$@" 2>/dev/null
+    rc=$?
+    set -e
+    # 125 = timeout(1) itself failed (bad options); 126/127 = not executable
+    if [ "$rc" -eq 125 ] || [ "$rc" -eq 126 ] || [ "$rc" -eq 127 ]; then
+      timeout --kill-after=2 "$secs" "$@"
       return $?
     fi
+    return "$rc"
+  fi
   # Portable: perl alarm + process group kill
   perl -e '
     use strict; use warnings;

taler-monitoring.sh

@@ -102,7 +102,7 @@ Phases:
              or -d DOMAIN → that domain’s surface; port/protocol/TLS/CVE (OSV)
   mattermost  Mattermost chat health (default mattermost.taler.net; SPA + /api/v4/system/ping)
   mail     MX/SMTP/IMAP for Taler mail (firefly, pixel/TSA, catalogued domains)
-  monpages public monitoring HTML via FQDN (HTTP 200 + suite page, not code 21)
+  monpages public monitoring HTML via FQDN (obligatory ERROR; GOA full inventory, FP only FP)
              uses MON_HOSTS + HTML_URL_OK (same as host-agent); v1.3.1+
   all      urls + inside + versions + sanity + e2e  (SSH phases only on koopa)
   full     all + server + ladder + auth401  (maximum; long-running, needs secrets)