From e7a3cd558f7784171274392ca8c60fde07be9743 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Hern=C3=A2ni=20Marques?= Date: Sun, 19 Jul 2026 00:29:49 +0200 Subject: [PATCH] release 1.8.0: simplified mon pages + surface nmap + mail hosts Public layout: only taler-monitoring-surface(+_err) for ecosystem software/versions; nine landing stacks keep /monitoring(+_err). Fold mail (firefly.gnunet.org, anastasis.taler-systems.com) and mattermost into the surface job; deprecate separate mon pages/timers. nmap OS fingerprinting; package-behind defaults to ERROR. --- DEPENDENCIES.md | 1 + README.md | 44 +++++----- TESTS.md | 3 +- VERSION | 2 +- VERSIONS.md | 15 ++-- check_monitoring_pages.sh | 41 +++++++-- check_surface.sh | 112 +++++++++++++++++++++++- check_versions.sh | 4 +- host-agent/README.md | 24 +++-- host-agent/install-host-agent.sh | 21 +++-- host-agent/run-mail-monitoring.sh | 42 ++++----- host-agent/run-mattermost-monitoring.sh | 44 +++------- host-agent/run-surface-monitoring.sh | 36 +++++--- mail-catalog.conf | 24 +++-- surface-catalog.conf | 4 + 15 files changed, 263 insertions(+), 154 deletions(-) diff --git a/DEPENDENCIES.md b/DEPENDENCIES.md index 8051e9f..8823098 100644 --- a/DEPENDENCIES.md +++ b/DEPENDENCIES.md @@ -8,6 +8,7 @@ |--------|------------------|------| | `bash` | (base) | runner | | `curl` | `curl` | HTTPS/API probes + **monpages** FQDN checks | +| `nmap` | `nmap` | **surface** OS/service fingerprint (`SURFACE_NMAP=1`, v1.8.0+; OS detect may need root) | | `python3` | `python313-base` (provides `/usr/bin/python3`) | JSON, HTML site-gen, metrics | | `podman` | `podman` | **inside**: `podman exec` into taler-hacktivism* | | `git` | `git` | pin commit SHA in monitoring HTML footer | diff --git a/README.md b/README.md index a53781d..5142486 100644 --- a/README.md +++ b/README.md @@ -6,34 +6,31 @@ Checks for **GNU Taler** stacks (bank, exchange, merchant; optional surface inve |------|------| | **Outside-only** | Public HTTPS from anywhere — no SSH, no host-agent. Phases such as `urls`, `monpages`, `surface`, … | | **Inside / local** | Where you operate the stack: podman/SSH (`inside`, `versions`, `e2e`, `ladder`, …) | -| **Host-agent + HTML** | Optional: timers run the suite and publish sticky-bar HTML under `…/monitoring/` (and surface/aptdeploy paths) | +| **Host-agent + HTML** | Optional: timers run the suite and publish sticky-bar HTML (see simplified page layout below) | Outside-only example: `./taler-monitoring.sh -d demo.taler.net urls` -Mattermost chat: `./taler-monitoring.sh mattermost` (default `https://mattermost.taler.net`) -Mail (firefly/pixel): `./taler-monitoring.sh mail` +Surface (ecosystem + mail + mattermost + versions): `./taler-monitoring.sh surface mattermost mail versions` Full local GOA example: `./taler-monitoring.sh -d hacktivism.ch urls inside versions` Repo: https://git.hacktivism.ch/hernani/taler-monitoring · tree: `~/src/taler-monitoring` Version: `./taler-monitoring.sh --ver` (or `--version` / `-V`) -## Monitoring pages (part of the suite) +## Monitoring pages — simplified overview (v1.8.0+) -The **public HTML reports** are not an external addon — they are a first-class part of **taler-monitoring**: +Public HTML is **not** an external addon. Layout is intentionally small: -| Kind | Public path(s) | Produced by | -|------|----------------|-------------| -| Stack host report | `https://{bank,exchange,merchant}/monitoring/` | host-agent + `site-gen/console_to_html.py` | -| Surface inventory | `https://taler.hacktivism.ch/taler-monitoring-surface/` | `run-surface-monitoring.sh` | -| apt-src deploy tests | `https://taler.hacktivism.ch/taler-monitoring-aptdeploy/` | `run-aptdeploy-monitoring.sh` | -| Mattermost | `https://taler.hacktivism.ch/taler-monitoring-mattermost/` | `run-mattermost-monitoring.sh` | -| Mail (MX/SMTP/IMAP) | `https://taler.hacktivism.ch/taler-monitoring-mail/` | `run-mail-monitoring.sh` | +| Kind | Where | Purpose | +|------|--------|---------| +| **Landing-stack report** | `https://{host}/monitoring/` and `/monitoring_err/` | Per **landing** front (9 stacks): GOA bank/exchange/taler · FP stage bank/exchange/monnaie · FP prod bank/exchange/monnaie | +| **Ecosystem surface** | **only** `https://taler.hacktivism.ch/taler-monitoring-surface/` and `…/taler-monitoring-surface_err/` | Software/versions found in the Taler universe (ports, nmap OS fingerprint, Mattermost, mail firefly+anastasis, package signals) | + +There are **no** first-class public pages for `/taler-monitoring-mail*`, `/taler-monitoring-mattermost*`, or similar — those checks run **inside the surface job** and appear on the surface page. Related: -- **Phase `monpages`:** obligatory outside-in check that public mon HTML is live (sticky-bar), not merchant JSON code 21 — GOA full inventory, FP only FP hosts. -- **Sticky “What this monitors”:** each page lists its checks **and** a dedicated block *Monitoring pages themselves (suite)*. -- **Publish:** staging `~/monitoring-sites-staging` → `DEPLOY_WWW_ROOT` + Caddy handles (`site-gen/caddy-monitoring-handles.snippet`, v1.7.2+). +- **Phase `monpages`:** obligatory ERROR if expected pages are missing (not merchant JSON code 21). +- **Publish:** staging `~/monitoring-sites-staging` → `DEPLOY_WWW_ROOT` + reverse-proxy handles. ```bash ./taler-monitoring.sh monpages @@ -401,21 +398,20 @@ LADDER_LOAD=0 ./taler-monitoring.sh ladder Default **English**. Select language with `--lang en|fr` or `TALER_MON_LANG`. FrancPaysan stacks auto-select **French** (`*lefrancpaysan*`). See `VERSIONS.md` and `i18n.sh`. -## Public monitoring pages via FQDN (v1.3.1+ / obligatory v1.7.6+) +## Public monitoring pages via FQDN (v1.3.1+ / obligatory v1.7.6+ / layout v1.8.0+) -Phase **`monpages`** is **obligatory**: missing or non-suite HTML is **ERROR** -(not merchant JSON code 21, not 404). Soft mode only with -`MONPAGES_REQUIRE_PUBLIC=0` (escape hatch). +Phase **`monpages`** is **obligatory**: missing or non-suite HTML is **ERROR**. +Soft mode only with `MONPAGES_REQUIRE_PUBLIC=0` (escape hatch). -| Stack | What is checked | -|-------|-----------------| -| **GOA / hacktivism** | Full suite inventory: `bank`/`exchange`/`taler` `/monitoring/` plus `/taler-monitoring-{surface,aptdeploy,mattermost,mail}/` and any extra pages found under staging/`DEPLOY_WWW_ROOT` | -| **FrancPaysan** | Only that stack’s mon hosts (`bank`/`exchange`/`monnaie` · stage variants) — never GOA URLs | +| Stack | What monpages expects (v1.8.0) | +|-------|--------------------------------| +| **GOA** | `bank`/`exchange`/`taler` `/monitoring/` + `taler.hacktivism.ch/taler-monitoring-surface/` | +| **FrancPaysan** | That stack’s three mon hosts `/monitoring/` only | +| **Nine landings (model)** | Each of the 9 landing fronts has `/monitoring` + `/monitoring_err` (when a failed run exists) | ```bash ./taler-monitoring.sh -d hacktivism.ch monpages ./taler-monitoring.sh -d lefrancpaysan.ch monpages -# host-agent runs monpages by default and again after HTML publish ``` If pages are missing publicly: `sudo ~/koopa-caddy/apply-monitoring-live.sh` on koopa diff --git a/TESTS.md b/TESTS.md index bbf4ad9..cc5680e 100644 --- a/TESTS.md +++ b/TESTS.md @@ -20,7 +20,8 @@ Every check line has a **global** run number and a **grouped** id: | **versions** | `check_versions.sh` | `outside` `inside` `compare` | | **aptdeploy** | `check_apt_deploy.sh` | `trixie` `trixie-testing` (koopa podman apt-src merchant smoke) | | **surface** | `check_surface.sh` | **remote-only** ecosystem / `-d` domain inventory (not in default/all/full) | -| **monpages** | `check_monitoring_pages.sh` | public monitoring HTML via FQDN — **obligatory ERROR**; GOA full inventory; FP only FP mon hosts | +| **monpages** | `check_monitoring_pages.sh` | public HTML via FQDN — **obligatory ERROR**; GOA: 3×`/monitoring/` + surface; FP: stack mon only (v1.8.0) | +| **surface** | `check_surface.sh` | ecosystem inventory + **nmap** OS fingerprint; mail/mattermost content reported via surface page | | **sanity** | `check_sanity.sh` | `bank` `exchange` `merchant` | | **server** | `check_server.sh` | (flat `server-NN` or host groups) | | **e2e** | `check_e2e.sh` | `prereq` `load` `bank` `wallet` `atm` `settle` `pay` `shop` `paivana` `dig` `report` | diff --git a/VERSION b/VERSION index 91c74a5..27f9cd3 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.7.7 +1.8.0 diff --git a/VERSIONS.md b/VERSIONS.md index 4cbc37e..0b74e72 100644 --- a/VERSIONS.md +++ b/VERSIONS.md @@ -13,14 +13,15 @@ Tags and `VERSION` use **MAJOR.FEATURE.FIX** (three components), introduced in * | **FEATURE** | New capabilities (phases, HTML features, i18n packs, …) | | **FIX** | Bugfixes and docs that do not add a feature | -Git tags: `vMAJOR.FEATURE.FIX` (e.g. `v1.7.5`). File `VERSION` omits the `v` prefix. +Git tags: `vMAJOR.FEATURE.FIX` (e.g. `v1.8.0`). File `VERSION` omits the `v` prefix. | Tag | Date (UTC) | Notes | |-----|------------|--------| -| **v1.7.7** | 2026-07-18 | **domains:** stage mytops merchant FQDN is `stage.my.taler-ops.ch` (betel mon sites). -| **v1.8.0** | *planned* | **L10n** packs for user-facing strings: **fr-CH** (FrancPaysan) and **de-CH** (hacktivism / Taler CH German). | -| **v1.7.6** | 2026-07-18 | **monpages obligatory (ERROR):** GOA checks full suite inventory (bank/exchange/taler `/monitoring/` + surface/aptdeploy/mattermost/mail + on-disk discovery); FP only its own mon hosts; `MONPAGES_REQUIRE_PUBLIC=1` default (escape hatch `=0`). **Bugfix:** `with_timeout` no longer re-runs a phase when it exits non-zero. | -| **v1.7.5** | 2026-07-18 | **Bugfix:** monpages soft mode `MONPAGES_REQUIRE_PUBLIC=0` (FP wrappers default) — missing public HTML is WARN until vhost publish; generic publish hints. | +| **v1.8.0** | 2026-07-19 | **Simplified public pages:** only `/taler-monitoring-surface(+_err)` on `taler.hacktivism.ch` for ecosystem software/versions; **9 landing stacks** each keep `/monitoring(+_err)`. Mail (firefly.gnunet.org + anastasis.taler-systems.com) and Mattermost folded into surface job; separate mail/mattermost mon pages + timers deprecated. **nmap** OS fingerprinting on surface (`SURFACE_NMAP`); **ERROR** when packages are behind (`TALER_PKG_BEHIND=error` default) or OS fingerprint is EOL. | +| **v1.8.1** | *planned* | **L10n** packs: **fr-CH** (FrancPaysan) and **de-CH** (hacktivism / Taler CH German). | +| **v1.7.7** | 2026-07-18 | **domains:** stage mytops merchant FQDN is `stage.my.taler-ops.ch` (betel mon sites). | +| **v1.8.0** | 2026-07-18 | **monpages obligatory (ERROR):** GOA checks full suite inventory (bank/exchange/taler `/monitoring/` + surface/aptdeploy/mattermost/mail + on-disk discovery); FP only its own mon hosts; `MONPAGES_REQUIRE_PUBLIC=1` default (escape hatch `=0`). **Bugfix:** `with_timeout` no longer re-runs a phase when it exits non-zero. | +| **v1.8.0** | 2026-07-18 | **Bugfix:** monpages soft mode `MONPAGES_REQUIRE_PUBLIC=0` (FP wrappers default) — missing public HTML is WARN until vhost publish; generic publish hints. | | **v1.7.4** | 2026-07-18 | **Bugfix:** monpages bare URLs (`/monitoring` without `/`) WARN by default (trailing slash remains ERROR); avoids false red when slash form is live. | | **v1.7.3** | 2026-07-18 | Monitoring **pages themselves** as suite scope: README section + sticky “What this monitors” block (monpages / public HTML paths). | | **v1.7.2** | 2026-07-18 | **Bugfix:** Caddy static handles — `root` must be host dir + matcher `/monitoring*` (not leaf dir + `/monitoring/`), else file_server returns empty 404 after apply. | @@ -41,7 +42,7 @@ Git tags: `vMAJOR.FEATURE.FIX` (e.g. `v1.7.5`). File `VERSION` omits the `v` pre ## Sticky bar version (v1.3.0+) -Generated HTML shows the installed release tag (e.g. `v1.7.6`) and links to Forgejo: +Generated HTML shows the installed release tag (e.g. `v1.8.0`) and links to Forgejo: `https://git.hacktivism.ch/hernani/taler-monitoring/src/tag/` @@ -71,5 +72,5 @@ Full locale packs (fr-CH / de-CH): **v1.8.0** (planned). ```bash git clone https://git.hacktivism.ch/hernani/taler-monitoring.git ~/src/taler-monitoring -cd ~/src/taler-monitoring && git checkout v1.7.6 +cd ~/src/taler-monitoring && git checkout v1.8.0 ``` diff --git a/check_monitoring_pages.sh b/check_monitoring_pages.sh index 1f55881..6d478e1 100755 --- a/check_monitoring_pages.sh +++ b/check_monitoring_pages.sh @@ -99,6 +99,20 @@ _host_allowed() { esac } +# Public page layout (v1.8.0+ simplified): +# • 9 landing stacks → /monitoring/ (+ /monitoring_err/ when present) +# • ecosystem software/versions inventory → only +# https://taler.hacktivism.ch/taler-monitoring-surface/ +# https://taler.hacktivism.ch/taler-monitoring-surface_err/ +# Legacy /taler-monitoring-{mail,mattermost,aptdeploy}* are NOT first-class pages. + +_path_allowed() { + case "$1" in + monitoring|monitoring_err|taler-monitoring-surface|taler-monitoring-surface_err) return 0 ;; + *) return 1 ;; + esac +} + # Emit https://host/path/ for each …/host/path/index.html under root (max depth 3). _discover_tree() { local root="$1" fam="$2" @@ -112,10 +126,19 @@ _discover_tree() { path="${path%/index.html}" [ -n "$host" ] && [ -n "$path" ] || continue _host_allowed "$host" "$fam" || continue + _path_allowed "$path" || continue printf 'https://%s/%s/\n' "$host" "$path" done < <(find "${root%/}" -mindepth 3 -maxdepth 3 -type f -name index.html -print0 2>/dev/null) } +# Nine landing stacks (GOA 3 + FP stage 3 + FP prod 3) with /monitoring/ +_landing_nine() { + printf '%s\n' \ + bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch \ + stage.bank.lefrancpaysan.ch stage.exchange.lefrancpaysan.ch stage.monnaie.lefrancpaysan.ch \ + bank.lefrancpaysan.ch exchange.lefrancpaysan.ch monnaie.lefrancpaysan.ch +} + # Suite catalog of OK pages that must exist for this family. _catalog_urls() { local fam="$1" h @@ -124,14 +147,8 @@ _catalog_urls() { for h in bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch; do printf 'https://%s/monitoring/\n' "$h" done - for h in \ - taler-monitoring-surface \ - taler-monitoring-aptdeploy \ - taler-monitoring-mattermost \ - taler-monitoring-mail - do - printf 'https://taler.hacktivism.ch/%s/\n' "$h" - done + # ecosystem surface only (versions / software found in Taler universe) + printf 'https://taler.hacktivism.ch/taler-monitoring-surface/\n' ;; fp) for h in $MON_HOSTS; do @@ -140,6 +157,14 @@ _catalog_urls() { printf 'https://%s/monitoring/\n' "$h" done ;; + all|nine) + # optional: all nine landings + surface + while IFS= read -r h; do + [ -n "$h" ] || continue + printf 'https://%s/monitoring/\n' "$h" + done < <(_landing_nine) + printf 'https://taler.hacktivism.ch/taler-monitoring-surface/\n' + ;; esac } diff --git a/check_surface.sh b/check_surface.sh index 1452d3b..bbecb07 100755 --- a/check_surface.sh +++ b/check_surface.sh @@ -32,6 +32,10 @@ PORT_TIMEOUT="${SURFACE_PORT_TIMEOUT:-2}" HTTP_TIMEOUT="${SURFACE_HTTP_TIMEOUT:-12}" # CVE via OSV (public, no key). Disable: SURFACE_CVE=0 : "${SURFACE_CVE:=1}" +# nmap OS / service fingerprint (v1.8.0+). Disable: SURFACE_NMAP=0 +: "${SURFACE_NMAP:=1}" +# OS detection (-O) often needs root; falls back to -sV when denied +: "${SURFACE_NMAP_OS:=1}" # Extra ports when -d domain mode DOMAIN_EXTRA_PORTS="${SURFACE_DOMAIN_PORTS:-22,80,443,8443}" @@ -496,6 +500,109 @@ cert_expiry_check() { return 0 } +# nmap OS / service fingerprint (v1.8.0+). +# ERROR when fingerprint indicates an OS that is EOL / should be upgraded. +# Soft/info when nmap missing or unprivileged OS scan fails. +nmap_fingerprint_host() { + local h="$1" + local xml out osline accuracy eol=0 + [ "${SURFACE_NMAP:-1}" = "1" ] || return 0 + if ! command -v nmap >/dev/null 2>&1; then + warn "nmap" "$h · nmap not installed (apt install nmap) — skip OS fingerprint" + return 0 + fi + xml=$(mktemp) + out=$(mktemp) + # Prefer OS detection when allowed; always try service version light + if [ "${SURFACE_NMAP_OS:-1}" = "1" ] && { + nmap -Pn -n -T4 -O --osscan-guess --max-os-tries 1 --host-timeout 45s \ + -p 22,80,443,25,465,587,993 "$h" -oX "$xml" >"$out" 2>&1 + }; then + : + else + nmap -Pn -n -T4 -sV --version-light --host-timeout 35s \ + -p 22,80,443,25,465,587,993 "$h" -oX "$xml" >"$out" 2>&1 || true + fi + # Parse best OS match from XML + osline=$( + python3 - "$xml" <<'PY' 2>/dev/null || true +import sys, xml.etree.ElementTree as ET +path = sys.argv[1] +try: + root = ET.parse(path).getroot() +except Exception: + sys.exit(0) +best = None +best_acc = -1 +for osm in root.findall(".//osmatch"): + name = osm.get("name") or "" + try: + acc = int(osm.get("accuracy") or "0") + except ValueError: + acc = 0 + if name and acc >= best_acc: + best_acc = acc + best = name +if best: + print(f"{best_acc}|{best}") +# also surface product/version from service table +svcs = [] +for port in root.findall(".//port"): + state = (port.find("state").get("state") if port.find("state") is not None else "") + if state != "open": + continue + svc = port.find("service") + if svc is None: + continue + prod = svc.get("product") or svc.get("name") or "" + ver = svc.get("version") or "" + if prod: + svcs.append(f"{prod} {ver}".strip()) +if svcs: + print("SVC|" + "; ".join(svcs[:8])) +PY + ) + rm -f "$xml" "$out" + if [ -z "$osline" ]; then + info "nmap" "$h · no OS/service fingerprint (need root for -O, or host filtered)" + return 0 + fi + local osname="" svcinfo="" + while IFS= read -r line; do + case "$line" in + SVC\|*) svcinfo=${line#SVC|} ;; + *\|*) + accuracy=${line%%|*} + osname=${line#*|} + ok "nmap-os" "$h · OS guess ${accuracy}% · $osname" + ;; + esac + done <<<"$osline" + [ -n "$svcinfo" ] && info "nmap-svc" "$h · $svcinfo" + + # EOL / upgrade-required OS fingerprints → ERROR + case "$(printf '%s' "$osname" | tr 'A-Z' 'a-z')" in + *debian*6*|*debian*7*|*debian*8*|*debian*9*|*debian*10*|*debian*\"buster\"*|*debian*buster*) + eol=1 ;; + *ubuntu*12.04*|*ubuntu*14.04*|*ubuntu*16.04*|*ubuntu*18.04*|*ubuntu*20.04*) + eol=1 ;; + *centos*[5-7]*|*red\ hat\ enterprise*5*|*red\ hat\ enterprise*6*|*red\ hat\ enterprise*7*) + eol=1 ;; + *windows\ server\ 2008*|*windows\ server\ 2012*|*windows\ xp*|*windows\ 7*) + eol=1 ;; + *freebsd\ 1[0-2]*|*freebsd\ [0-9].*) + eol=1 ;; + esac + if [ "$eol" = "1" ]; then + err "nmap-upgrade" "$h OS fingerprint indicates EOL / upgrade required" "$osname" + return 1 + fi + if [ -n "$osname" ]; then + info "nmap-upgrade" "$h OS fingerprint not on EOL list · $osname" + fi + return 0 +} + scan_host() { local h="$1" local ports proto label expect @@ -621,6 +728,9 @@ scan_host() { fi fi + # nmap OS / service fingerprint (v1.8.0+) — ERROR on EOL OS + nmap_fingerprint_host "$h" || true + # Expected service must answer protocol (not just ping/port) # Note: ${array[*]:-x} is NOT valid default syntax (bash treats : as slice). local ports_txt="${open_ports[*]}" @@ -667,7 +777,7 @@ else fi info "inventory" "${#ORDERED_HOSTS[@]} hosts to probe (remote only · no SSH · no local podman on targets)" -info "flags" "SURFACE_CVE=${SURFACE_CVE:-1} PORT_TIMEOUT=${PORT_TIMEOUT:-2} HTTP_TIMEOUT=${HTTP_TIMEOUT:-12}" +info "flags" "SURFACE_CVE=${SURFACE_CVE:-1} SURFACE_NMAP=${SURFACE_NMAP:-1} SURFACE_NMAP_OS=${SURFACE_NMAP_OS:-1} PORT_TIMEOUT=${PORT_TIMEOUT:-2} HTTP_TIMEOUT=${HTTP_TIMEOUT:-12}" for h in "${ORDERED_HOSTS[@]}"; do [ -n "$h" ] || continue diff --git a/check_versions.sh b/check_versions.sh index 8fc7c66..071d9f8 100755 --- a/check_versions.sh +++ b/check_versions.sh @@ -11,7 +11,7 @@ # TALER_APT_BASE=https://deb.taler.net/apt/debian # TALER_APT_INDEX=…/dists/trixie/main/binary-amd64/Packages # TALER_APT_TESTING_INDEX=…/dists/trixie-testing/… -# TALER_PKG_BEHIND=warn|error (default: core packages ERROR if behind, else warn) +# TALER_PKG_BEHIND=warn|error (default error as of v1.8.0 — any behind package is ERROR) # SKIP_SSH=1 skip container install checks set -euo pipefail ROOT=$(cd "$(dirname "$0")" && pwd) @@ -27,7 +27,7 @@ APT_BASE="${APT_BASE%/}" INDEX="${TALER_APT_INDEX:-${APT_BASE}/dists/${SUITE}/main/binary-amd64/Packages}" INRELEASE="${TALER_APT_INRELEASE:-${APT_BASE}/dists/${SUITE}/InRelease}" TESTING_INDEX="${TALER_APT_TESTING_INDEX:-${APT_BASE}/dists/${SUITE}-testing/main/binary-amd64/Packages}" -BEHIND_MODE="${TALER_PKG_BEHIND:-warn}" +BEHIND_MODE="${TALER_PKG_BEHIND:-error}" tmp=$(mktemp -d) trap 'rm -rf "$tmp"' EXIT diff --git a/host-agent/README.md b/host-agent/README.md index c37cb5f..b6e978b 100644 --- a/host-agent/README.md +++ b/host-agent/README.md @@ -18,8 +18,9 @@ Thin wrappers only set stack defaults: | `run-fp-stage-monitoring.sh` | FP stage | `stagepaysan` | | `run-fp-prod-monitoring.sh` | FP prod | `francpaysan` | | `run-aptdeploy.sh` | CLI: ensure + **aptdeploy** (no HTML) | local if hostname=koopa, else **ssh koopa-external** | -| `run-aptdeploy-monitoring.sh` | **aptdeploy** → `/taler-monitoring-aptdeploy*` | `hernani` @ koopa (4h timer, taler only) | -| `run-surface-monitoring.sh` | **surface** → `/taler-monitoring-surface*` | `hernani` @ koopa (hourly, taler only) | +| `run-aptdeploy-monitoring.sh` | optional apt-src tests (not a public mon-page type) | `hernani` @ koopa (4h) | +| `run-surface-monitoring.sh` | **surface** → only `/taler-monitoring-surface*` (ecosystem + mail + mattermost + versions) | `hernani` @ koopa (hourly) | +| `run-mail-monitoring.sh` / `run-mattermost-monitoring.sh` | **deprecated v1.8.0** — aliases of surface | disabled timers | ### apt-src deploy tests (hacktivism `/monitoring`) @@ -44,24 +45,19 @@ Thin wrappers only set stack defaults: Checks: package table, `httpd --version`, libtalerutil/`ldd` on failure, `systemctl` for `taler-merchant.target`, basic local `/config` probe. No nginx/HTTPS in containers. -### surface + aptdeploy HTML (taler.hacktivism.ch only) +### Public HTML layout (v1.8.0 simplified) -| URL | Role | Timer | -|-----|------|-------| -| `…/taler-monitoring-surface/` | ecosystem surface scan | 1h | -| `…/taler-monitoring-surface_err/` | errors + jump list | ″ | -| `…/taler-monitoring-aptdeploy/` | apt-src container deploy tests | 4h | -| `…/taler-monitoring-aptdeploy_err/` | errors + jump list | ″ | +| URL | Role | +|-----|------| +| `https://{bank,exchange,merchant}/monitoring/` (+ `_err`) | **Landing-stack** report — each of the **9** landing fronts (GOA 3 + FP stage 3 + FP prod 3) | +| `https://taler.hacktivism.ch/taler-monitoring-surface/` (+ `_err`) | **Only** ecosystem page: software/versions in the Taler universe (nmap OS, mail firefly+anastasis, mattermost, …) | -Not on bank/exchange. Staging: -`~/monitoring-sites-staging/taler.hacktivism.ch/taler-monitoring-{surface,aptdeploy}*`. +No separate public pages for mail/mattermost. Staging under `~/monitoring-sites-staging/`. ```bash systemctl --user enable --now taler-monitoring-surface.timer -systemctl --user enable --now taler-monitoring-aptdeploy.timer systemctl --user start taler-monitoring-surface.service -systemctl --user start taler-monitoring-aptdeploy.service -# Caddy: handles only in taler.hacktivism.ch block (root) +# mail/mattermost timers stay disabled (v1.8.0) ``` Env examples: **`env/*.env.example`**. FP: `run-fp-prod-monitoring.sh` / `run-fp-stage-monitoring.sh` (same `run-host-report.sh` pipeline). diff --git a/host-agent/install-host-agent.sh b/host-agent/install-host-agent.sh index 2a1fb93..0cea74c 100755 --- a/host-agent/install-host-agent.sh +++ b/host-agent/install-host-agent.sh @@ -60,8 +60,10 @@ install_local() { install -m 644 "$ROOT/taler-monitoring-hacktivism.timer" "$home_unit/" install -m 644 "$ROOT/taler-monitoring-surface.service" "$home_unit/" install -m 644 "$ROOT/taler-monitoring-surface.timer" "$home_unit/" + # optional aptdeploy (not a public mon page layout — v1.8.0 simplified overview) install -m 644 "$ROOT/taler-monitoring-aptdeploy.service" "$home_unit/" install -m 644 "$ROOT/taler-monitoring-aptdeploy.timer" "$home_unit/" + # units still installed for migration; timers disabled (folded into surface) install -m 644 "$ROOT/taler-monitoring-mattermost.service" "$home_unit/" install -m 644 "$ROOT/taler-monitoring-mattermost.timer" "$home_unit/" install -m 644 "$ROOT/taler-monitoring-mail.service" "$home_unit/" @@ -72,31 +74,28 @@ install_local() { systemctl --user enable --now taler-monitoring-hacktivism.timer systemctl --user enable --now taler-monitoring-surface.timer systemctl --user enable --now taler-monitoring-aptdeploy.timer - systemctl --user enable --now taler-monitoring-mattermost.timer - systemctl --user enable --now taler-monitoring-mail.timer + # v1.8.0: mail + mattermost folded into surface — stop separate timers + systemctl --user disable --now taler-monitoring-mattermost.timer 2>/dev/null || true + systemctl --user disable --now taler-monitoring-mail.timer 2>/dev/null || true # initial stamp so path exists "$ROOT/touch-software-stamp.sh" systemctl --user start taler-monitoring-hacktivism.service || true systemctl --user start taler-monitoring-surface.service || true systemctl --user start taler-monitoring-aptdeploy.service || true - systemctl --user start taler-monitoring-mattermost.service || true - systemctl --user start taler-monitoring-mail.service || true echo "--- status ---" systemctl --user status taler-monitoring-hacktivism.path --no-pager -l | head -15 systemctl --user status taler-monitoring-hacktivism.timer --no-pager -l | head -12 systemctl --user status taler-monitoring-surface.timer --no-pager -l | head -12 systemctl --user status taler-monitoring-aptdeploy.timer --no-pager -l | head -12 - systemctl --user status taler-monitoring-mattermost.timer --no-pager -l | head -12 - systemctl --user status taler-monitoring-mail.timer --no-pager -l | head -12 systemctl --user list-timers --all | grep taler-monitoring || true - echo "OK host-agent (hacktivism 4h + surface 1h + aptdeploy 4h + mattermost 1h + mail 4h)" + echo "OK host-agent (v1.8.0 simplified):" + echo " · hacktivism 4h → /monitoring on bank/exchange/taler (landing stacks)" + echo " · surface 1h → /taler-monitoring-surface* (ecosystem versions/software)" + echo " · aptdeploy 4h (optional tests; not a public mon-page type)" + echo " · mail/mattermost timers disabled (content in surface)" echo "suite: $mon" - echo "manual: systemctl --user start taler-monitoring-hacktivism.service" echo "manual surface: systemctl --user start taler-monitoring-surface.service" - echo "manual aptdeploy: systemctl --user start taler-monitoring-aptdeploy.service" - echo "manual mattermost: systemctl --user start taler-monitoring-mattermost.service" - echo "manual mail: systemctl --user start taler-monitoring-mail.service" echo "after upgrades: $ROOT/touch-software-stamp.sh" } diff --git a/host-agent/run-mail-monitoring.sh b/host-agent/run-mail-monitoring.sh index b939c28..cd13132 100755 --- a/host-agent/run-mail-monitoring.sh +++ b/host-agent/run-mail-monitoring.sh @@ -1,32 +1,20 @@ #!/usr/bin/env bash -# run-mail-monitoring.sh — mail MX/SMTP/IMAP (firefly + pixel, …) → public HTML -# Public only on taler.hacktivism.ch: -# https://taler.hacktivism.ch/taler-monitoring-mail/ -# https://taler.hacktivism.ch/taler-monitoring-mail_err/ +# DEPRECATED (v1.8.0): mail is part of taler-monitoring-surface. +# Kept as a thin alias so old timers/docs do not break. +# +# Prefer: run-surface-monitoring.sh +# Public page: https://taler.hacktivism.ch/taler-monitoring-surface/ # set -uo pipefail AGENT_DIR=$(cd "$(dirname "$0")" && pwd) - -export AGENT_LABEL="${AGENT_LABEL:-mail-host-agent}" -export STATE_NAME="${STATE_NAME:-taler-mail-monitoring}" -export TALER_DOMAIN="${TALER_DOMAIN:-hacktivism.ch}" -export INSIDE_PODMAN=0 -export LOCAL_STACK=0 -export SKIP_SSH=1 -export INSIDE_MODE="${INSIDE_MODE:-none}" -export CONTINUE_ON_ERROR="${CONTINUE_ON_ERROR:-1}" -export RUN_TIMEOUT="${RUN_TIMEOUT:-600}" -export PHASES="${PHASES:-mail monpages}" +echo "NOTE: run-mail-monitoring.sh is deprecated (v1.8.0) — folding into surface" >&2 +export PHASES="${PHASES:-mail surface monpages}" +# force surface HTML paths (no /taler-monitoring-mail*) +export HTML_OK_DIR="taler-monitoring-surface" +export HTML_ERR_DIR="taler-monitoring-surface_err" +export HTML_URL_OK="/taler-monitoring-surface/" +export HTML_URL_ERR="/taler-monitoring-surface_err/" +export PAGE_LABEL="taler-monitoring-surface" export MON_HOSTS="${MON_HOSTS:-taler.hacktivism.ch}" -export HTML_OUT="${HTML_OUT:-$HOME/monitoring-sites-staging}" -export DEPLOY_WWW_ROOT="${DEPLOY_WWW_ROOT:-/var/www/monitoring-sites}" - -export HTML_OK_DIR="taler-monitoring-mail" -export HTML_ERR_DIR="taler-monitoring-mail_err" -export HTML_URL_OK="/taler-monitoring-mail/" -export HTML_URL_ERR="/taler-monitoring-mail_err/" -export PAGE_LABEL="taler-monitoring-mail" - -export APT_DEPLOY_ENSURE=0 - -exec bash "$AGENT_DIR/run-host-report.sh" "$@" +export MONPAGES_INVENTORY=job +exec bash "$AGENT_DIR/run-surface-monitoring.sh" "$@" diff --git a/host-agent/run-mattermost-monitoring.sh b/host-agent/run-mattermost-monitoring.sh index 4986c3c..991f68c 100755 --- a/host-agent/run-mattermost-monitoring.sh +++ b/host-agent/run-mattermost-monitoring.sh @@ -1,35 +1,19 @@ #!/usr/bin/env bash -# run-mattermost-monitoring.sh — Mattermost (mattermost.taler.net) → public HTML -# Public only on taler.hacktivism.ch: -# https://taler.hacktivism.ch/taler-monitoring-mattermost/ -# https://taler.hacktivism.ch/taler-monitoring-mattermost_err/ +# DEPRECATED (v1.8.0): mattermost is part of taler-monitoring-surface. +# Kept as a thin alias so old timers/docs do not break. +# +# Prefer: run-surface-monitoring.sh +# Public page: https://taler.hacktivism.ch/taler-monitoring-surface/ # set -uo pipefail AGENT_DIR=$(cd "$(dirname "$0")" && pwd) - -export AGENT_LABEL="${AGENT_LABEL:-mattermost-host-agent}" -export STATE_NAME="${STATE_NAME:-taler-mattermost-monitoring}" -export TALER_DOMAIN="${TALER_DOMAIN:-hacktivism.ch}" -# remote public service — no SSH / podman -export INSIDE_PODMAN=0 -export LOCAL_STACK=0 -export SKIP_SSH=1 -export INSIDE_MODE="${INSIDE_MODE:-none}" -export CONTINUE_ON_ERROR="${CONTINUE_ON_ERROR:-1}" -export RUN_TIMEOUT="${RUN_TIMEOUT:-300}" -# dedicated job: mattermost + public page check -export PHASES="${PHASES:-mattermost monpages}" +echo "NOTE: run-mattermost-monitoring.sh is deprecated (v1.8.0) — folding into surface" >&2 +export PHASES="${PHASES:-mattermost surface monpages}" +export HTML_OK_DIR="taler-monitoring-surface" +export HTML_ERR_DIR="taler-monitoring-surface_err" +export HTML_URL_OK="/taler-monitoring-surface/" +export HTML_URL_ERR="/taler-monitoring-surface_err/" +export PAGE_LABEL="taler-monitoring-surface" export MON_HOSTS="${MON_HOSTS:-taler.hacktivism.ch}" -export HTML_OUT="${HTML_OUT:-$HOME/monitoring-sites-staging}" -export DEPLOY_WWW_ROOT="${DEPLOY_WWW_ROOT:-/var/www/monitoring-sites}" - -export HTML_OK_DIR="taler-monitoring-mattermost" -export HTML_ERR_DIR="taler-monitoring-mattermost_err" -export HTML_URL_OK="/taler-monitoring-mattermost/" -export HTML_URL_ERR="/taler-monitoring-mattermost_err/" -export PAGE_LABEL="taler-monitoring-mattermost" - -export MATTERMOST_PUBLIC="${MATTERMOST_PUBLIC:-https://mattermost.taler.net}" -export APT_DEPLOY_ENSURE=0 - -exec bash "$AGENT_DIR/run-host-report.sh" "$@" +export MONPAGES_INVENTORY=job +exec bash "$AGENT_DIR/run-surface-monitoring.sh" "$@" diff --git a/host-agent/run-surface-monitoring.sh b/host-agent/run-surface-monitoring.sh index cf67f6b..f760565 100755 --- a/host-agent/run-surface-monitoring.sh +++ b/host-agent/run-surface-monitoring.sh @@ -1,26 +1,33 @@ #!/usr/bin/env bash -# run-surface-monitoring.sh — hourly remote-only ecosystem surface scan -# Public only on taler.hacktivism.ch: +# run-surface-monitoring.sh — ecosystem surface report (v1.8.0+ simplified) +# +# Public HTML only on taler.hacktivism.ch: # https://taler.hacktivism.ch/taler-monitoring-surface/ # https://taler.hacktivism.ch/taler-monitoring-surface_err/ # +# This single page covers remote ecosystem inventory: host/port probes, nmap OS +# fingerprint, Mattermost, mail (firefly + anastasis), package/version signals. +# There are NO separate public pages for mail / mattermost / aptdeploy. +# +# Landing-stack reports stay on each of the 9 fronts as /monitoring(/_err). +# set -uo pipefail AGENT_DIR=$(cd "$(dirname "$0")" && pwd) export AGENT_LABEL="${AGENT_LABEL:-surface-host-agent}" export STATE_NAME="${STATE_NAME:-taler-surface-monitoring}" export TALER_DOMAIN="${TALER_DOMAIN:-hacktivism.ch}" -# surface is remote-only — force flags (ignore shared env) -export INSIDE_PODMAN=0 -export LOCAL_STACK=0 -export SKIP_SSH=1 -export INSIDE_MODE="${INSIDE_MODE:-none}" +# surface is primarily remote; versions may use host podman when available +export INSIDE_PODMAN="${INSIDE_PODMAN:-1}" +export LOCAL_STACK="${LOCAL_STACK:-0}" +export SKIP_SSH="${SKIP_SSH:-1}" +export INSIDE_MODE="${INSIDE_MODE:-local-podman}" export CONTINUE_ON_ERROR="${CONTINUE_ON_ERROR:-1}" -# many hosts; allow up to 45 min wall (phase also has RUN_TIMEOUT) export RUN_TIMEOUT="${RUN_TIMEOUT:-2400}" -# dedicated job: always surface only (never inherit PHASES from shared env) -export PHASES="${PHASES:-surface monpages}" -# Only merchant public front — not bank/exchange + +# All ecosystem checks → one surface HTML report +export PHASES="${PHASES:-surface mattermost mail versions monpages}" + export MON_HOSTS="${MON_HOSTS:-taler.hacktivism.ch}" export HTML_OUT="${HTML_OUT:-$HOME/monitoring-sites-staging}" export DEPLOY_WWW_ROOT="${DEPLOY_WWW_ROOT:-/var/www/monitoring-sites}" @@ -31,11 +38,12 @@ export HTML_URL_OK="/taler-monitoring-surface/" export HTML_URL_ERR="/taler-monitoring-surface_err/" export PAGE_LABEL="taler-monitoring-surface" -# ecosystem catalog by default (not domain-only) export SURFACE_SCOPE="${SURFACE_SCOPE:-ecosystem}" export TALER_DOMAIN_FROM_CLI=0 - -# Do not run apt-deploy ensure for surface-only jobs export APT_DEPLOY_ENSURE=0 +# monpages job-only for this path (surface HTML); not the 9 landings +export MONPAGES_INVENTORY="${MONPAGES_INVENTORY:-job}" +# package behind → ERROR (v1.8.0) +export TALER_PKG_BEHIND="${TALER_PKG_BEHIND:-error}" exec bash "$AGENT_DIR/run-host-report.sh" "$@" diff --git a/mail-catalog.conf b/mail-catalog.conf index 69bc819..7a67578 100644 --- a/mail-catalog.conf +++ b/mail-catalog.conf @@ -1,22 +1,18 @@ -# mail-catalog.conf — domains and mail hosts for phase `mail` +# mail-catalog.conf — domains and mail hosts for phase `mail` (v1.8.0+) # # Format (whitespace-separated; # comments): # domain expected_mx (comma) mail_hosts (comma) smtp_ports imap_ports label # -# - expected_mx: MX exchange hostnames that should appear (any match = OK if multiple) -# - mail_hosts: servers to probe for SMTP/IMAP (A/AAAA + ports + optional banner) -# - ports: comma-separated; empty = skip that protocol family +# Only these mail planes (no hacktivism / ad-hoc MX): +# • firefly.gnunet.org — taler.net + gnunet.org +# • anastasis.taler-systems.com — Anastasis / TSA mail # # Override path: MAIL_CATALOG=/path/to/file -# -# GNU Taler / GNUnet mail (firefly) + TSA (pixel / mail.anastasis.lu) -# --- firefly (primary for taler.net + gnunet.org) --- -taler.net firefly.gnunet.org firefly.gnunet.org 25,465,587 143,993 taler-net -gnunet.org firefly.gnunet.org firefly.gnunet.org 25,465,587 143,993 gnunet +# --- firefly.gnunet.org --- +taler.net firefly.gnunet.org firefly.gnunet.org 25,465,587 143,993 firefly-taler +gnunet.org firefly.gnunet.org firefly.gnunet.org 25,465,587 143,993 firefly-gnunet -# --- taler-systems.com / pixel (MX mail.anastasis.lu → pixel) --- -taler-systems.com mail.anastasis.lu,pixel.taler-systems.com pixel.taler-systems.com,mail.taler-systems.com,mail.anastasis.lu 25,465 143,993 tsa-pixel - -# Optional related (same pixel path; keep expected MX loose) -# anastasis.lu mail.anastasis.lu mail.anastasis.lu,pixel.taler-systems.com 25,465 143,993 anastasis +# --- anastasis.taler-systems.com --- +anastasis.taler-systems.com anastasis.taler-systems.com anastasis.taler-systems.com 25,465,587 143,993 anastasis-tsa +taler-systems.com anastasis.taler-systems.com anastasis.taler-systems.com 25,465,587 143,993 tsa-via-anastasis diff --git a/surface-catalog.conf b/surface-catalog.conf index ad1bdc3..cae2a14 100644 --- a/surface-catalog.conf +++ b/surface-catalog.conf @@ -46,6 +46,10 @@ bugs.gnunet.org 443 https gnunet-bugs # --- taler-systems.com --- www.taler-systems.com 443 https tsa-www taler-systems.com 443 https tsa-apex +anastasis.taler-systems.com 443,25,465,587,143,993 https anastasis-mail + +# --- mail (also covered by phase mail; listed for surface nmap/ports) --- +firefly.gnunet.org 443,25,465,587,143,993 https firefly-mail # --- common infra hostnames (remote only; may 404/redirect) --- deb.taler.net 443 https deb-taler