release 1.8.0: simplified mon pages + surface nmap + mail hosts

Public layout: only taler-monitoring-surface(+_err) for ecosystem
software/versions; nine landing stacks keep /monitoring(+_err).
Fold mail (firefly.gnunet.org, anastasis.taler-systems.com) and
mattermost into the surface job; deprecate separate mon pages/timers.
nmap OS fingerprinting; package-behind defaults to ERROR.

DEPENDENCIES.md

@@ -8,6 +8,7 @@
 |--------|------------------|------|
 | `bash` | (base) | runner |
 | `curl` | `curl` | HTTPS/API probes + **monpages** FQDN checks |
+| `nmap` | `nmap` | **surface** OS/service fingerprint (`SURFACE_NMAP=1`, v1.8.0+; OS detect may need root) |
 | `python3` | `python313-base` (provides `/usr/bin/python3`) | JSON, HTML site-gen, metrics |
 | `podman` | `podman` | **inside**: `podman exec` into taler-hacktivism* |
 | `git` | `git` | pin commit SHA in monitoring HTML footer |

README.md

@@ -6,34 +6,31 @@ Checks for **GNU Taler** stacks (bank, exchange, merchant; optional surface inve
 |------|------|
 | **Outside-only** | Public HTTPS from anywhere — no SSH, no host-agent. Phases such as `urls`, `monpages`, `surface`, … |
 | **Inside / local** | Where you operate the stack: podman/SSH (`inside`, `versions`, `e2e`, `ladder`, …) |
-| **Host-agent + HTML** | Optional: timers run the suite and publish sticky-bar HTML under `…/monitoring/` (and surface/aptdeploy paths) |
+| **Host-agent + HTML** | Optional: timers run the suite and publish sticky-bar HTML (see simplified page layout below) |
 
 Outside-only example: `./taler-monitoring.sh -d demo.taler.net urls`  
-Mattermost chat: `./taler-monitoring.sh mattermost` (default `https://mattermost.taler.net`)  
+Surface (ecosystem + mail + mattermost + versions): `./taler-monitoring.sh surface mattermost mail versions`  
-Mail (firefly/pixel): `./taler-monitoring.sh mail`  
 Full local GOA example: `./taler-monitoring.sh -d hacktivism.ch urls inside versions`
 
 Repo: https://git.hacktivism.ch/hernani/taler-monitoring · tree: `~/src/taler-monitoring`  
 Version: `./taler-monitoring.sh --ver` (or `--version` / `-V`)
 
 
-## Monitoring pages (part of the suite)
+## Monitoring pages — simplified overview (v1.8.0+)
 
-The **public HTML reports** are not an external addon — they are a first-class part of **taler-monitoring**:
+Public HTML is **not** an external addon. Layout is intentionally small:
 
-| Kind | Public path(s) | Produced by |
+| Kind | Where | Purpose |
-|------|----------------|-------------|
+|------|--------|---------|
-| Stack host report | `https://{bank,exchange,merchant}/monitoring/` | host-agent + `site-gen/console_to_html.py` |
+| **Landing-stack report** | `https://{host}/monitoring/` and `/monitoring_err/` | Per **landing** front (9 stacks): GOA bank/exchange/taler · FP stage bank/exchange/monnaie · FP prod bank/exchange/monnaie |
-| Surface inventory | `https://taler.hacktivism.ch/taler-monitoring-surface/` | `run-surface-monitoring.sh` |
+| **Ecosystem surface** | **only** `https://taler.hacktivism.ch/taler-monitoring-surface/` and `…/taler-monitoring-surface_err/` | Software/versions found in the Taler universe (ports, nmap OS fingerprint, Mattermost, mail firefly+anastasis, package signals) |
-| apt-src deploy tests | `https://taler.hacktivism.ch/taler-monitoring-aptdeploy/` | `run-aptdeploy-monitoring.sh` |
+
-| Mattermost | `https://taler.hacktivism.ch/taler-monitoring-mattermost/` | `run-mattermost-monitoring.sh` |
+There are **no** first-class public pages for `/taler-monitoring-mail*`, `/taler-monitoring-mattermost*`, or similar — those checks run **inside the surface job** and appear on the surface page.
-| Mail (MX/SMTP/IMAP) | `https://taler.hacktivism.ch/taler-monitoring-mail/` | `run-mail-monitoring.sh` |
 
 Related:
 
-- **Phase `monpages`:** obligatory outside-in check that public mon HTML is live (sticky-bar), not merchant JSON code 21 — GOA full inventory, FP only FP hosts.
+- **Phase `monpages`:** obligatory ERROR if expected pages are missing (not merchant JSON code 21).
-- **Sticky “What this monitors”:** each page lists its checks **and** a dedicated block *Monitoring pages themselves (suite)*.
+- **Publish:** staging `~/monitoring-sites-staging` → `DEPLOY_WWW_ROOT` + reverse-proxy handles.
-- **Publish:** staging `~/monitoring-sites-staging` → `DEPLOY_WWW_ROOT` + Caddy handles (`site-gen/caddy-monitoring-handles.snippet`, v1.7.2+).
 
 ```bash
 ./taler-monitoring.sh monpages
@@ -401,21 +398,20 @@ LADDER_LOAD=0 ./taler-monitoring.sh ladder
 Default **English**. Select language with `--lang en|fr` or `TALER_MON_LANG`.
 FrancPaysan stacks auto-select **French** (`*lefrancpaysan*`). See `VERSIONS.md` and `i18n.sh`.
 
-## Public monitoring pages via FQDN (v1.3.1+ / obligatory v1.7.6+)
+## Public monitoring pages via FQDN (v1.3.1+ / obligatory v1.7.6+ / layout v1.8.0+)
 
-Phase **`monpages`** is **obligatory**: missing or non-suite HTML is **ERROR**
+Phase **`monpages`** is **obligatory**: missing or non-suite HTML is **ERROR**.
-(not merchant JSON code 21, not 404). Soft mode only with
+Soft mode only with `MONPAGES_REQUIRE_PUBLIC=0` (escape hatch).
-`MONPAGES_REQUIRE_PUBLIC=0` (escape hatch).
 
-| Stack | What is checked |
+| Stack | What monpages expects (v1.8.0) |
-|-------|-----------------|
+|-------|--------------------------------|
-| **GOA / hacktivism** | Full suite inventory: `bank`/`exchange`/`taler` `/monitoring/` plus `/taler-monitoring-{surface,aptdeploy,mattermost,mail}/` and any extra pages found under staging/`DEPLOY_WWW_ROOT` |
+| **GOA** | `bank`/`exchange`/`taler` `/monitoring/` + `taler.hacktivism.ch/taler-monitoring-surface/` |
-| **FrancPaysan** | Only that stack’s mon hosts (`bank`/`exchange`/`monnaie` · stage variants) — never GOA URLs |
+| **FrancPaysan** | That stack’s three mon hosts `/monitoring/` only |
+| **Nine landings (model)** | Each of the 9 landing fronts has `/monitoring` + `/monitoring_err` (when a failed run exists) |
 
 ```bash
 ./taler-monitoring.sh -d hacktivism.ch monpages
 ./taler-monitoring.sh -d lefrancpaysan.ch monpages
-# host-agent runs monpages by default and again after HTML publish
 ```
 
 If pages are missing publicly: `sudo ~/koopa-caddy/apply-monitoring-live.sh` on koopa

TESTS.md

@@ -20,7 +20,8 @@ Every check line has a **global** run number and a **grouped** id:
 | **versions** | `check_versions.sh` | `outside` `inside` `compare` |
 | **aptdeploy** | `check_apt_deploy.sh` | `trixie` `trixie-testing` (koopa podman apt-src merchant smoke) |
 | **surface** | `check_surface.sh` | **remote-only** ecosystem / `-d` domain inventory (not in default/all/full) |
-| **monpages** | `check_monitoring_pages.sh` | public monitoring HTML via FQDN — **obligatory ERROR**; GOA full inventory; FP only FP mon hosts |
+| **monpages** | `check_monitoring_pages.sh` | public HTML via FQDN — **obligatory ERROR**; GOA: 3×`/monitoring/` + surface; FP: stack mon only (v1.8.0) |
+| **surface** | `check_surface.sh` | ecosystem inventory + **nmap** OS fingerprint; mail/mattermost content reported via surface page |
 | **sanity** | `check_sanity.sh` | `bank` `exchange` `merchant` |
 | **server** | `check_server.sh` | (flat `server-NN` or host groups) |
 | **e2e** | `check_e2e.sh` | `prereq` `load` `bank` `wallet` `atm` `settle` `pay` `shop` `paivana` `dig` `report` |

VERSION

@@ -1 +1 @@
-1.7.7
+1.8.0

VERSIONS.md

@@ -13,14 +13,15 @@ Tags and `VERSION` use **MAJOR.FEATURE.FIX** (three components), introduced in *
 | **FEATURE** | New capabilities (phases, HTML features, i18n packs, …) |
 | **FIX** | Bugfixes and docs that do not add a feature |
 
-Git tags: `vMAJOR.FEATURE.FIX` (e.g. `v1.7.5`). File `VERSION` omits the `v` prefix.
+Git tags: `vMAJOR.FEATURE.FIX` (e.g. `v1.8.0`). File `VERSION` omits the `v` prefix.
 
 | Tag | Date (UTC) | Notes |
 |-----|------------|--------|
-| **v1.7.7** | 2026-07-18 | **domains:** stage mytops merchant FQDN is `stage.my.taler-ops.ch` (betel mon sites).
+| **v1.8.0** | 2026-07-19 | **Simplified public pages:** only `/taler-monitoring-surface(+_err)` on `taler.hacktivism.ch` for ecosystem software/versions; **9 landing stacks** each keep `/monitoring(+_err)`. Mail (firefly.gnunet.org + anastasis.taler-systems.com) and Mattermost folded into surface job; separate mail/mattermost mon pages + timers deprecated. **nmap** OS fingerprinting on surface (`SURFACE_NMAP`); **ERROR** when packages are behind (`TALER_PKG_BEHIND=error` default) or OS fingerprint is EOL. |
-| **v1.8.0** | *planned* | **L10n** packs for user-facing strings: **fr-CH** (FrancPaysan) and **de-CH** (hacktivism / Taler CH German). |
+| **v1.8.1** | *planned* | **L10n** packs: **fr-CH** (FrancPaysan) and **de-CH** (hacktivism / Taler CH German). |
-| **v1.7.6** | 2026-07-18 | **monpages obligatory (ERROR):** GOA checks full suite inventory (bank/exchange/taler `/monitoring/` + surface/aptdeploy/mattermost/mail + on-disk discovery); FP only its own mon hosts; `MONPAGES_REQUIRE_PUBLIC=1` default (escape hatch `=0`). **Bugfix:** `with_timeout` no longer re-runs a phase when it exits non-zero. |
+| **v1.7.7** | 2026-07-18 | **domains:** stage mytops merchant FQDN is `stage.my.taler-ops.ch` (betel mon sites). |
-| **v1.7.5** | 2026-07-18 | **Bugfix:** monpages soft mode `MONPAGES_REQUIRE_PUBLIC=0` (FP wrappers default) — missing public HTML is WARN until vhost publish; generic publish hints. |
+| **v1.8.0** | 2026-07-18 | **monpages obligatory (ERROR):** GOA checks full suite inventory (bank/exchange/taler `/monitoring/` + surface/aptdeploy/mattermost/mail + on-disk discovery); FP only its own mon hosts; `MONPAGES_REQUIRE_PUBLIC=1` default (escape hatch `=0`). **Bugfix:** `with_timeout` no longer re-runs a phase when it exits non-zero. |
+| **v1.8.0** | 2026-07-18 | **Bugfix:** monpages soft mode `MONPAGES_REQUIRE_PUBLIC=0` (FP wrappers default) — missing public HTML is WARN until vhost publish; generic publish hints. |
 | **v1.7.4** | 2026-07-18 | **Bugfix:** monpages bare URLs (`/monitoring` without `/`) WARN by default (trailing slash remains ERROR); avoids false red when slash form is live. |
 | **v1.7.3** | 2026-07-18 | Monitoring **pages themselves** as suite scope: README section + sticky “What this monitors” block (monpages / public HTML paths). |
 | **v1.7.2** | 2026-07-18 | **Bugfix:** Caddy static handles — `root` must be host dir + matcher `/monitoring*` (not leaf dir + `/monitoring/`), else file_server returns empty 404 after apply. |
@@ -41,7 +42,7 @@ Git tags: `vMAJOR.FEATURE.FIX` (e.g. `v1.7.5`). File `VERSION` omits the `v` pre
 
 ## Sticky bar version (v1.3.0+)
 
-Generated HTML shows the installed release tag (e.g. `v1.7.6`) and links to Forgejo:
+Generated HTML shows the installed release tag (e.g. `v1.8.0`) and links to Forgejo:
 
 `https://git.hacktivism.ch/hernani/taler-monitoring/src/tag/<tag>`
 
@@ -71,5 +72,5 @@ Full locale packs (fr-CH / de-CH): **v1.8.0** (planned).
 
 ```bash
 git clone https://git.hacktivism.ch/hernani/taler-monitoring.git ~/src/taler-monitoring
-cd ~/src/taler-monitoring && git checkout v1.7.6
+cd ~/src/taler-monitoring && git checkout v1.8.0
 ```

check_monitoring_pages.sh

@@ -99,6 +99,20 @@ _host_allowed() {
   esac
 }
 
+# Public page layout (v1.8.0+ simplified):
+#   • 9 landing stacks → /monitoring/ (+ /monitoring_err/ when present)
+#   • ecosystem software/versions inventory → only
+#       https://taler.hacktivism.ch/taler-monitoring-surface/
+#       https://taler.hacktivism.ch/taler-monitoring-surface_err/
+# Legacy /taler-monitoring-{mail,mattermost,aptdeploy}* are NOT first-class pages.
+
+_path_allowed() {
+  case "$1" in
+    monitoring|monitoring_err|taler-monitoring-surface|taler-monitoring-surface_err) return 0 ;;
+    *) return 1 ;;
+  esac
+}
+
 # Emit https://host/path/ for each …/host/path/index.html under root (max depth 3).
 _discover_tree() {
   local root="$1" fam="$2"
@@ -112,10 +126,19 @@ _discover_tree() {
     path="${path%/index.html}"
     [ -n "$host" ] && [ -n "$path" ] || continue
     _host_allowed "$host" "$fam" || continue
+    _path_allowed "$path" || continue
     printf 'https://%s/%s/\n' "$host" "$path"
   done < <(find "${root%/}" -mindepth 3 -maxdepth 3 -type f -name index.html -print0 2>/dev/null)
 }
 
+# Nine landing stacks (GOA 3 + FP stage 3 + FP prod 3) with /monitoring/
+_landing_nine() {
+  printf '%s\n' \
+    bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch \
+    stage.bank.lefrancpaysan.ch stage.exchange.lefrancpaysan.ch stage.monnaie.lefrancpaysan.ch \
+    bank.lefrancpaysan.ch exchange.lefrancpaysan.ch monnaie.lefrancpaysan.ch
+}
+
 # Suite catalog of OK pages that must exist for this family.
 _catalog_urls() {
   local fam="$1" h
@@ -124,14 +147,8 @@ _catalog_urls() {
       for h in bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch; do
         printf 'https://%s/monitoring/\n' "$h"
       done
-      for h in \
+      # ecosystem surface only (versions / software found in Taler universe)
-        taler-monitoring-surface \
+      printf 'https://taler.hacktivism.ch/taler-monitoring-surface/\n'
-        taler-monitoring-aptdeploy \
-        taler-monitoring-mattermost \
-        taler-monitoring-mail
-      do
-        printf 'https://taler.hacktivism.ch/%s/\n' "$h"
-      done
       ;;
     fp)
       for h in $MON_HOSTS; do
@@ -140,6 +157,14 @@ _catalog_urls() {
         printf 'https://%s/monitoring/\n' "$h"
       done
       ;;
+    all|nine)
+      # optional: all nine landings + surface
+      while IFS= read -r h; do
+        [ -n "$h" ] || continue
+        printf 'https://%s/monitoring/\n' "$h"
+      done < <(_landing_nine)
+      printf 'https://taler.hacktivism.ch/taler-monitoring-surface/\n'
+      ;;
   esac
 }
 

check_surface.sh

@@ -32,6 +32,10 @@ PORT_TIMEOUT="${SURFACE_PORT_TIMEOUT:-2}"
 HTTP_TIMEOUT="${SURFACE_HTTP_TIMEOUT:-12}"
 # CVE via OSV (public, no key). Disable: SURFACE_CVE=0
 : "${SURFACE_CVE:=1}"
+# nmap OS / service fingerprint (v1.8.0+). Disable: SURFACE_NMAP=0
+: "${SURFACE_NMAP:=1}"
+# OS detection (-O) often needs root; falls back to -sV when denied
+: "${SURFACE_NMAP_OS:=1}"
 # Extra ports when -d domain mode
 DOMAIN_EXTRA_PORTS="${SURFACE_DOMAIN_PORTS:-22,80,443,8443}"
 
@@ -496,6 +500,109 @@ cert_expiry_check() {
   return 0
 }
 
+# nmap OS / service fingerprint (v1.8.0+).
+# ERROR when fingerprint indicates an OS that is EOL / should be upgraded.
+# Soft/info when nmap missing or unprivileged OS scan fails.
+nmap_fingerprint_host() {
+  local h="$1"
+  local xml out osline accuracy eol=0
+  [ "${SURFACE_NMAP:-1}" = "1" ] || return 0
+  if ! command -v nmap >/dev/null 2>&1; then
+    warn "nmap" "$h · nmap not installed (apt install nmap) — skip OS fingerprint"
+    return 0
+  fi
+  xml=$(mktemp)
+  out=$(mktemp)
+  # Prefer OS detection when allowed; always try service version light
+  if [ "${SURFACE_NMAP_OS:-1}" = "1" ] && {
+       nmap -Pn -n -T4 -O --osscan-guess --max-os-tries 1 --host-timeout 45s \
+         -p 22,80,443,25,465,587,993 "$h" -oX "$xml" >"$out" 2>&1
+     }; then
+    :
+  else
+    nmap -Pn -n -T4 -sV --version-light --host-timeout 35s \
+      -p 22,80,443,25,465,587,993 "$h" -oX "$xml" >"$out" 2>&1 || true
+  fi
+  # Parse best OS match from XML
+  osline=$(
+    python3 - "$xml" <<'PY' 2>/dev/null || true
+import sys, xml.etree.ElementTree as ET
+path = sys.argv[1]
+try:
+    root = ET.parse(path).getroot()
+except Exception:
+    sys.exit(0)
+best = None
+best_acc = -1
+for osm in root.findall(".//osmatch"):
+    name = osm.get("name") or ""
+    try:
+        acc = int(osm.get("accuracy") or "0")
+    except ValueError:
+        acc = 0
+    if name and acc >= best_acc:
+        best_acc = acc
+        best = name
+if best:
+    print(f"{best_acc}|{best}")
+# also surface product/version from service table
+svcs = []
+for port in root.findall(".//port"):
+    state = (port.find("state").get("state") if port.find("state") is not None else "")
+    if state != "open":
+        continue
+    svc = port.find("service")
+    if svc is None:
+        continue
+    prod = svc.get("product") or svc.get("name") or ""
+    ver = svc.get("version") or ""
+    if prod:
+        svcs.append(f"{prod} {ver}".strip())
+if svcs:
+    print("SVC|" + "; ".join(svcs[:8]))
+PY
+  )
+  rm -f "$xml" "$out"
+  if [ -z "$osline" ]; then
+    info "nmap" "$h · no OS/service fingerprint (need root for -O, or host filtered)"
+    return 0
+  fi
+  local osname="" svcinfo=""
+  while IFS= read -r line; do
+    case "$line" in
+      SVC\|*) svcinfo=${line#SVC|} ;;
+      *\|*)
+        accuracy=${line%%|*}
+        osname=${line#*|}
+        ok "nmap-os" "$h · OS guess ${accuracy}% · $osname"
+        ;;
+    esac
+  done <<<"$osline"
+  [ -n "$svcinfo" ] && info "nmap-svc" "$h · $svcinfo"
+
+  # EOL / upgrade-required OS fingerprints → ERROR
+  case "$(printf '%s' "$osname" | tr 'A-Z' 'a-z')" in
+    *debian*6*|*debian*7*|*debian*8*|*debian*9*|*debian*10*|*debian*\"buster\"*|*debian*buster*)
+      eol=1 ;;
+    *ubuntu*12.04*|*ubuntu*14.04*|*ubuntu*16.04*|*ubuntu*18.04*|*ubuntu*20.04*)
+      eol=1 ;;
+    *centos*[5-7]*|*red\ hat\ enterprise*5*|*red\ hat\ enterprise*6*|*red\ hat\ enterprise*7*)
+      eol=1 ;;
+    *windows\ server\ 2008*|*windows\ server\ 2012*|*windows\ xp*|*windows\ 7*)
+      eol=1 ;;
+    *freebsd\ 1[0-2]*|*freebsd\ [0-9].*)
+      eol=1 ;;
+  esac
+  if [ "$eol" = "1" ]; then
+    err "nmap-upgrade" "$h OS fingerprint indicates EOL / upgrade required" "$osname"
+    return 1
+  fi
+  if [ -n "$osname" ]; then
+    info "nmap-upgrade" "$h OS fingerprint not on EOL list · $osname"
+  fi
+  return 0
+}
+
 scan_host() {
   local h="$1"
   local ports proto label expect
@@ -621,6 +728,9 @@ scan_host() {
     fi
   fi
 
+  # nmap OS / service fingerprint (v1.8.0+) — ERROR on EOL OS
+  nmap_fingerprint_host "$h" || true
+
   # Expected service must answer protocol (not just ping/port)
   # Note: ${array[*]:-x} is NOT valid default syntax (bash treats : as slice).
   local ports_txt="${open_ports[*]}"
@@ -667,7 +777,7 @@ else
 fi
 
 info "inventory" "${#ORDERED_HOSTS[@]} hosts to probe (remote only · no SSH · no local podman on targets)"
-info "flags" "SURFACE_CVE=${SURFACE_CVE:-1} PORT_TIMEOUT=${PORT_TIMEOUT:-2} HTTP_TIMEOUT=${HTTP_TIMEOUT:-12}"
+info "flags" "SURFACE_CVE=${SURFACE_CVE:-1} SURFACE_NMAP=${SURFACE_NMAP:-1} SURFACE_NMAP_OS=${SURFACE_NMAP_OS:-1} PORT_TIMEOUT=${PORT_TIMEOUT:-2} HTTP_TIMEOUT=${HTTP_TIMEOUT:-12}"
 
 for h in "${ORDERED_HOSTS[@]}"; do
   [ -n "$h" ] || continue

check_versions.sh

@@ -11,7 +11,7 @@
 #   TALER_APT_BASE=https://deb.taler.net/apt/debian
 #   TALER_APT_INDEX=…/dists/trixie/main/binary-amd64/Packages
 #   TALER_APT_TESTING_INDEX=…/dists/trixie-testing/…
-#   TALER_PKG_BEHIND=warn|error   (default: core packages ERROR if behind, else warn)
+#   TALER_PKG_BEHIND=warn|error   (default error as of v1.8.0 — any behind package is ERROR)
 #   SKIP_SSH=1                    skip container install checks
 set -euo pipefail
 ROOT=$(cd "$(dirname "$0")" && pwd)
@@ -27,7 +27,7 @@ APT_BASE="${APT_BASE%/}"
 INDEX="${TALER_APT_INDEX:-${APT_BASE}/dists/${SUITE}/main/binary-amd64/Packages}"
 INRELEASE="${TALER_APT_INRELEASE:-${APT_BASE}/dists/${SUITE}/InRelease}"
 TESTING_INDEX="${TALER_APT_TESTING_INDEX:-${APT_BASE}/dists/${SUITE}-testing/main/binary-amd64/Packages}"
-BEHIND_MODE="${TALER_PKG_BEHIND:-warn}"
+BEHIND_MODE="${TALER_PKG_BEHIND:-error}"
 
 tmp=$(mktemp -d)
 trap 'rm -rf "$tmp"' EXIT

host-agent/README.md

@@ -18,8 +18,9 @@ Thin wrappers only set stack defaults:
 | `run-fp-stage-monitoring.sh` | FP stage | `stagepaysan` |
 | `run-fp-prod-monitoring.sh` | FP prod | `francpaysan` |
 | `run-aptdeploy.sh` | CLI: ensure + **aptdeploy** (no HTML) | local if hostname=koopa, else **ssh koopa-external** |
-| `run-aptdeploy-monitoring.sh` | **aptdeploy** → `/taler-monitoring-aptdeploy*` | `hernani` @ koopa (4h timer, taler only) |
+| `run-aptdeploy-monitoring.sh` | optional apt-src tests (not a public mon-page type) | `hernani` @ koopa (4h) |
-| `run-surface-monitoring.sh` | **surface** → `/taler-monitoring-surface*` | `hernani` @ koopa (hourly, taler only) |
+| `run-surface-monitoring.sh` | **surface** → only `/taler-monitoring-surface*` (ecosystem + mail + mattermost + versions) | `hernani` @ koopa (hourly) |
+| `run-mail-monitoring.sh` / `run-mattermost-monitoring.sh` | **deprecated v1.8.0** — aliases of surface | disabled timers |
 
 ### apt-src deploy tests (hacktivism `/monitoring`)
 
@@ -44,24 +45,19 @@ Thin wrappers only set stack defaults:
 Checks: package table, `httpd --version`, libtalerutil/`ldd` on failure, `systemctl` for
 `taler-merchant.target`, basic local `/config` probe. No nginx/HTTPS in containers.
 
-### surface + aptdeploy HTML (taler.hacktivism.ch only)
+### Public HTML layout (v1.8.0 simplified)
 
-| URL | Role | Timer |
+| URL | Role |
-|-----|------|-------|
+|-----|------|
-| `…/taler-monitoring-surface/` | ecosystem surface scan | 1h |
+| `https://{bank,exchange,merchant}/monitoring/` (+ `_err`) | **Landing-stack** report — each of the **9** landing fronts (GOA 3 + FP stage 3 + FP prod 3) |
-| `…/taler-monitoring-surface_err/` | errors + jump list | ″ |
+| `https://taler.hacktivism.ch/taler-monitoring-surface/` (+ `_err`) | **Only** ecosystem page: software/versions in the Taler universe (nmap OS, mail firefly+anastasis, mattermost, …) |
-| `…/taler-monitoring-aptdeploy/` | apt-src container deploy tests | 4h |
-| `…/taler-monitoring-aptdeploy_err/` | errors + jump list | ″ |
 
-Not on bank/exchange. Staging:
+No separate public pages for mail/mattermost. Staging under `~/monitoring-sites-staging/`.
-`~/monitoring-sites-staging/taler.hacktivism.ch/taler-monitoring-{surface,aptdeploy}*`.
 
 ```bash
 systemctl --user enable --now taler-monitoring-surface.timer
-systemctl --user enable --now taler-monitoring-aptdeploy.timer
 systemctl --user start taler-monitoring-surface.service
-systemctl --user start taler-monitoring-aptdeploy.service
+# mail/mattermost timers stay disabled (v1.8.0)
-# Caddy: handles only in taler.hacktivism.ch block (root)
 ```
 
 Env examples: **`env/*.env.example`**. FP: `run-fp-prod-monitoring.sh` / `run-fp-stage-monitoring.sh` (same `run-host-report.sh` pipeline).

host-agent/install-host-agent.sh

@@ -60,8 +60,10 @@ install_local() {
   install -m 644 "$ROOT/taler-monitoring-hacktivism.timer" "$home_unit/"
   install -m 644 "$ROOT/taler-monitoring-surface.service" "$home_unit/"
   install -m 644 "$ROOT/taler-monitoring-surface.timer" "$home_unit/"
+  # optional aptdeploy (not a public mon page layout — v1.8.0 simplified overview)
   install -m 644 "$ROOT/taler-monitoring-aptdeploy.service" "$home_unit/"
   install -m 644 "$ROOT/taler-monitoring-aptdeploy.timer" "$home_unit/"
+  # units still installed for migration; timers disabled (folded into surface)
   install -m 644 "$ROOT/taler-monitoring-mattermost.service" "$home_unit/"
   install -m 644 "$ROOT/taler-monitoring-mattermost.timer" "$home_unit/"
   install -m 644 "$ROOT/taler-monitoring-mail.service" "$home_unit/"
@@ -72,31 +74,28 @@ install_local() {
   systemctl --user enable --now taler-monitoring-hacktivism.timer
   systemctl --user enable --now taler-monitoring-surface.timer
   systemctl --user enable --now taler-monitoring-aptdeploy.timer
-  systemctl --user enable --now taler-monitoring-mattermost.timer
+  # v1.8.0: mail + mattermost folded into surface — stop separate timers
-  systemctl --user enable --now taler-monitoring-mail.timer
+  systemctl --user disable --now taler-monitoring-mattermost.timer 2>/dev/null || true
+  systemctl --user disable --now taler-monitoring-mail.timer 2>/dev/null || true
   # initial stamp so path exists
   "$ROOT/touch-software-stamp.sh"
   systemctl --user start taler-monitoring-hacktivism.service || true
   systemctl --user start taler-monitoring-surface.service || true
   systemctl --user start taler-monitoring-aptdeploy.service || true
-  systemctl --user start taler-monitoring-mattermost.service || true
-  systemctl --user start taler-monitoring-mail.service || true
 
   echo "--- status ---"
   systemctl --user status taler-monitoring-hacktivism.path --no-pager -l | head -15
   systemctl --user status taler-monitoring-hacktivism.timer --no-pager -l | head -12
   systemctl --user status taler-monitoring-surface.timer --no-pager -l | head -12
   systemctl --user status taler-monitoring-aptdeploy.timer --no-pager -l | head -12
-  systemctl --user status taler-monitoring-mattermost.timer --no-pager -l | head -12
-  systemctl --user status taler-monitoring-mail.timer --no-pager -l | head -12
   systemctl --user list-timers --all | grep taler-monitoring || true
-  echo "OK host-agent (hacktivism 4h + surface 1h + aptdeploy 4h + mattermost 1h + mail 4h)"
+  echo "OK host-agent (v1.8.0 simplified):"
+  echo "  · hacktivism 4h → /monitoring on bank/exchange/taler (landing stacks)"
+  echo "  · surface 1h → /taler-monitoring-surface* (ecosystem versions/software)"
+  echo "  · aptdeploy 4h (optional tests; not a public mon-page type)"
+  echo "  · mail/mattermost timers disabled (content in surface)"
   echo "suite: $mon"
-  echo "manual: systemctl --user start taler-monitoring-hacktivism.service"
   echo "manual surface: systemctl --user start taler-monitoring-surface.service"
-  echo "manual aptdeploy: systemctl --user start taler-monitoring-aptdeploy.service"
-  echo "manual mattermost: systemctl --user start taler-monitoring-mattermost.service"
-  echo "manual mail: systemctl --user start taler-monitoring-mail.service"
   echo "after upgrades: $ROOT/touch-software-stamp.sh"
 }
 

host-agent/run-mail-monitoring.sh

@@ -1,32 +1,20 @@
 #!/usr/bin/env bash
-# run-mail-monitoring.sh — mail MX/SMTP/IMAP (firefly + pixel, …) → public HTML
+# DEPRECATED (v1.8.0): mail is part of taler-monitoring-surface.
-# Public only on taler.hacktivism.ch:
+# Kept as a thin alias so old timers/docs do not break.
-#   https://taler.hacktivism.ch/taler-monitoring-mail/
+#
-#   https://taler.hacktivism.ch/taler-monitoring-mail_err/
+# Prefer: run-surface-monitoring.sh
+# Public page: https://taler.hacktivism.ch/taler-monitoring-surface/
 #
 set -uo pipefail
 AGENT_DIR=$(cd "$(dirname "$0")" && pwd)
-
+echo "NOTE: run-mail-monitoring.sh is deprecated (v1.8.0) — folding into surface" >&2
-export AGENT_LABEL="${AGENT_LABEL:-mail-host-agent}"
+export PHASES="${PHASES:-mail surface monpages}"
-export STATE_NAME="${STATE_NAME:-taler-mail-monitoring}"
+# force surface HTML paths (no /taler-monitoring-mail*)
-export TALER_DOMAIN="${TALER_DOMAIN:-hacktivism.ch}"
+export HTML_OK_DIR="taler-monitoring-surface"
-export INSIDE_PODMAN=0
+export HTML_ERR_DIR="taler-monitoring-surface_err"
-export LOCAL_STACK=0
+export HTML_URL_OK="/taler-monitoring-surface/"
-export SKIP_SSH=1
+export HTML_URL_ERR="/taler-monitoring-surface_err/"
-export INSIDE_MODE="${INSIDE_MODE:-none}"
+export PAGE_LABEL="taler-monitoring-surface"
-export CONTINUE_ON_ERROR="${CONTINUE_ON_ERROR:-1}"
-export RUN_TIMEOUT="${RUN_TIMEOUT:-600}"
-export PHASES="${PHASES:-mail monpages}"
 export MON_HOSTS="${MON_HOSTS:-taler.hacktivism.ch}"
-export HTML_OUT="${HTML_OUT:-$HOME/monitoring-sites-staging}"
+export MONPAGES_INVENTORY=job
-export DEPLOY_WWW_ROOT="${DEPLOY_WWW_ROOT:-/var/www/monitoring-sites}"
+exec bash "$AGENT_DIR/run-surface-monitoring.sh" "$@"
-
-export HTML_OK_DIR="taler-monitoring-mail"
-export HTML_ERR_DIR="taler-monitoring-mail_err"
-export HTML_URL_OK="/taler-monitoring-mail/"
-export HTML_URL_ERR="/taler-monitoring-mail_err/"
-export PAGE_LABEL="taler-monitoring-mail"
-
-export APT_DEPLOY_ENSURE=0
-
-exec bash "$AGENT_DIR/run-host-report.sh" "$@"

host-agent/run-mattermost-monitoring.sh

@@ -1,35 +1,19 @@
 #!/usr/bin/env bash
-# run-mattermost-monitoring.sh — Mattermost (mattermost.taler.net) → public HTML
+# DEPRECATED (v1.8.0): mattermost is part of taler-monitoring-surface.
-# Public only on taler.hacktivism.ch:
+# Kept as a thin alias so old timers/docs do not break.
-#   https://taler.hacktivism.ch/taler-monitoring-mattermost/
+#
-#   https://taler.hacktivism.ch/taler-monitoring-mattermost_err/
+# Prefer: run-surface-monitoring.sh
+# Public page: https://taler.hacktivism.ch/taler-monitoring-surface/
 #
 set -uo pipefail
 AGENT_DIR=$(cd "$(dirname "$0")" && pwd)
-
+echo "NOTE: run-mattermost-monitoring.sh is deprecated (v1.8.0) — folding into surface" >&2
-export AGENT_LABEL="${AGENT_LABEL:-mattermost-host-agent}"
+export PHASES="${PHASES:-mattermost surface monpages}"
-export STATE_NAME="${STATE_NAME:-taler-mattermost-monitoring}"
+export HTML_OK_DIR="taler-monitoring-surface"
-export TALER_DOMAIN="${TALER_DOMAIN:-hacktivism.ch}"
+export HTML_ERR_DIR="taler-monitoring-surface_err"
-# remote public service — no SSH / podman
+export HTML_URL_OK="/taler-monitoring-surface/"
-export INSIDE_PODMAN=0
+export HTML_URL_ERR="/taler-monitoring-surface_err/"
-export LOCAL_STACK=0
+export PAGE_LABEL="taler-monitoring-surface"
-export SKIP_SSH=1
-export INSIDE_MODE="${INSIDE_MODE:-none}"
-export CONTINUE_ON_ERROR="${CONTINUE_ON_ERROR:-1}"
-export RUN_TIMEOUT="${RUN_TIMEOUT:-300}"
-# dedicated job: mattermost + public page check
-export PHASES="${PHASES:-mattermost monpages}"
 export MON_HOSTS="${MON_HOSTS:-taler.hacktivism.ch}"
-export HTML_OUT="${HTML_OUT:-$HOME/monitoring-sites-staging}"
+export MONPAGES_INVENTORY=job
-export DEPLOY_WWW_ROOT="${DEPLOY_WWW_ROOT:-/var/www/monitoring-sites}"
+exec bash "$AGENT_DIR/run-surface-monitoring.sh" "$@"
-
-export HTML_OK_DIR="taler-monitoring-mattermost"
-export HTML_ERR_DIR="taler-monitoring-mattermost_err"
-export HTML_URL_OK="/taler-monitoring-mattermost/"
-export HTML_URL_ERR="/taler-monitoring-mattermost_err/"
-export PAGE_LABEL="taler-monitoring-mattermost"
-
-export MATTERMOST_PUBLIC="${MATTERMOST_PUBLIC:-https://mattermost.taler.net}"
-export APT_DEPLOY_ENSURE=0
-
-exec bash "$AGENT_DIR/run-host-report.sh" "$@"

host-agent/run-surface-monitoring.sh

@@ -1,26 +1,33 @@
 #!/usr/bin/env bash
-# run-surface-monitoring.sh — hourly remote-only ecosystem surface scan
+# run-surface-monitoring.sh — ecosystem surface report (v1.8.0+ simplified)
-# Public only on taler.hacktivism.ch:
+#
+# Public HTML only on taler.hacktivism.ch:
 #   https://taler.hacktivism.ch/taler-monitoring-surface/
 #   https://taler.hacktivism.ch/taler-monitoring-surface_err/
 #
+# This single page covers remote ecosystem inventory: host/port probes, nmap OS
+# fingerprint, Mattermost, mail (firefly + anastasis), package/version signals.
+# There are NO separate public pages for mail / mattermost / aptdeploy.
+#
+# Landing-stack reports stay on each of the 9 fronts as /monitoring(/_err).
+#
 set -uo pipefail
 AGENT_DIR=$(cd "$(dirname "$0")" && pwd)
 
 export AGENT_LABEL="${AGENT_LABEL:-surface-host-agent}"
 export STATE_NAME="${STATE_NAME:-taler-surface-monitoring}"
 export TALER_DOMAIN="${TALER_DOMAIN:-hacktivism.ch}"
-# surface is remote-only — force flags (ignore shared env)
+# surface is primarily remote; versions may use host podman when available
-export INSIDE_PODMAN=0
+export INSIDE_PODMAN="${INSIDE_PODMAN:-1}"
-export LOCAL_STACK=0
+export LOCAL_STACK="${LOCAL_STACK:-0}"
-export SKIP_SSH=1
+export SKIP_SSH="${SKIP_SSH:-1}"
-export INSIDE_MODE="${INSIDE_MODE:-none}"
+export INSIDE_MODE="${INSIDE_MODE:-local-podman}"
 export CONTINUE_ON_ERROR="${CONTINUE_ON_ERROR:-1}"
-# many hosts; allow up to 45 min wall (phase also has RUN_TIMEOUT)
 export RUN_TIMEOUT="${RUN_TIMEOUT:-2400}"
-# dedicated job: always surface only (never inherit PHASES from shared env)
+
-export PHASES="${PHASES:-surface monpages}"
+# All ecosystem checks → one surface HTML report
-# Only merchant public front — not bank/exchange
+export PHASES="${PHASES:-surface mattermost mail versions monpages}"
+
 export MON_HOSTS="${MON_HOSTS:-taler.hacktivism.ch}"
 export HTML_OUT="${HTML_OUT:-$HOME/monitoring-sites-staging}"
 export DEPLOY_WWW_ROOT="${DEPLOY_WWW_ROOT:-/var/www/monitoring-sites}"
@@ -31,11 +38,12 @@ export HTML_URL_OK="/taler-monitoring-surface/"
 export HTML_URL_ERR="/taler-monitoring-surface_err/"
 export PAGE_LABEL="taler-monitoring-surface"
 
-# ecosystem catalog by default (not domain-only)
 export SURFACE_SCOPE="${SURFACE_SCOPE:-ecosystem}"
 export TALER_DOMAIN_FROM_CLI=0
-
-# Do not run apt-deploy ensure for surface-only jobs
 export APT_DEPLOY_ENSURE=0
+# monpages job-only for this path (surface HTML); not the 9 landings
+export MONPAGES_INVENTORY="${MONPAGES_INVENTORY:-job}"
+# package behind → ERROR (v1.8.0)
+export TALER_PKG_BEHIND="${TALER_PKG_BEHIND:-error}"
 
 exec bash "$AGENT_DIR/run-host-report.sh" "$@"

mail-catalog.conf

@@ -1,22 +1,18 @@
-# mail-catalog.conf — domains and mail hosts for phase `mail`
+# mail-catalog.conf — domains and mail hosts for phase `mail` (v1.8.0+)
 #
 # Format (whitespace-separated; # comments):
 #   domain   expected_mx (comma)   mail_hosts (comma)   smtp_ports   imap_ports   label
 #
-# - expected_mx: MX exchange hostnames that should appear (any match = OK if multiple)
+# Only these mail planes (no hacktivism / ad-hoc MX):
-# - mail_hosts: servers to probe for SMTP/IMAP (A/AAAA + ports + optional banner)
+#   • firefly.gnunet.org  — taler.net + gnunet.org
-# - ports: comma-separated; empty = skip that protocol family
+#   • anastasis.taler-systems.com — Anastasis / TSA mail
 #
 # Override path: MAIL_CATALOG=/path/to/file
-#
-# GNU Taler / GNUnet mail (firefly) + TSA (pixel / mail.anastasis.lu)
 
-# --- firefly (primary for taler.net + gnunet.org) ---
+# --- firefly.gnunet.org ---
-taler.net            firefly.gnunet.org                 firefly.gnunet.org                         25,465,587  143,993  taler-net
+taler.net                     firefly.gnunet.org              firefly.gnunet.org              25,465,587  143,993  firefly-taler
-gnunet.org           firefly.gnunet.org                 firefly.gnunet.org                         25,465,587  143,993  gnunet
+gnunet.org                    firefly.gnunet.org              firefly.gnunet.org              25,465,587  143,993  firefly-gnunet
 
-# --- taler-systems.com / pixel (MX mail.anastasis.lu → pixel) ---
+# --- anastasis.taler-systems.com ---
-taler-systems.com    mail.anastasis.lu,pixel.taler-systems.com   pixel.taler-systems.com,mail.taler-systems.com,mail.anastasis.lu  25,465  143,993  tsa-pixel
+anastasis.taler-systems.com   anastasis.taler-systems.com     anastasis.taler-systems.com     25,465,587  143,993  anastasis-tsa
-
+taler-systems.com             anastasis.taler-systems.com     anastasis.taler-systems.com     25,465,587  143,993  tsa-via-anastasis
-# Optional related (same pixel path; keep expected MX loose)
-# anastasis.lu         mail.anastasis.lu                 mail.anastasis.lu,pixel.taler-systems.com  25,465  143,993  anastasis

surface-catalog.conf

@@ -46,6 +46,10 @@ bugs.gnunet.org               443     https   gnunet-bugs
 # --- taler-systems.com ---
 www.taler-systems.com         443     https   tsa-www
 taler-systems.com             443     https   tsa-apex
+anastasis.taler-systems.com   443,25,465,587,143,993  https   anastasis-mail
+
+# --- mail (also covered by phase mail; listed for surface nmap/ports) ---
+firefly.gnunet.org            443,25,465,587,143,993  https   firefly-mail
 
 # --- common infra hostnames (remote only; may 404/redirect) ---
 deb.taler.net                 443     https   deb-taler