From c5034c0bb5f5c768a927726555ac29b1f3b6153b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Hern=C3=A2ni=20Marques?= Date: Sun, 19 Jul 2026 02:05:00 +0200 Subject: [PATCH] release 1.7.6: monpages obligatory ERROR with stack inventory GOA monpages checks the full suite catalog (bank/exchange/taler /monitoring/ plus surface/aptdeploy/mattermost/mail) and on-disk discovery; FP only its own mon hosts. Default MONPAGES_REQUIRE_PUBLIC=1. Also fix with_timeout re-running failed phases via || fallback. --- README.md | 18 ++- TESTS.md | 2 +- VERSION | 2 +- VERSIONS.md | 5 +- check_monitoring_pages.sh | 175 +++++++++++++++++++++----- host-agent/README.md | 2 +- host-agent/run-fp-prod-monitoring.sh | 4 +- host-agent/run-fp-stage-monitoring.sh | 4 +- host-agent/run-host-report.sh | 8 +- lib.sh | 15 ++- taler-monitoring.sh | 2 +- 11 files changed, 188 insertions(+), 49 deletions(-) diff --git a/README.md b/README.md index 796da73..a53781d 100644 --- a/README.md +++ b/README.md @@ -31,7 +31,7 @@ The **public HTML reports** are not an external addon — they are a first-class Related: -- **Phase `monpages`:** outside-in check that those FQDNs serve suite HTML (sticky-bar), not merchant JSON code 21. +- **Phase `monpages`:** obligatory outside-in check that public mon HTML is live (sticky-bar), not merchant JSON code 21 — GOA full inventory, FP only FP hosts. - **Sticky “What this monitors”:** each page lists its checks **and** a dedicated block *Monitoring pages themselves (suite)*. - **Publish:** staging `~/monitoring-sites-staging` → `DEPLOY_WWW_ROOT` + Caddy handles (`site-gen/caddy-monitoring-handles.snippet`, v1.7.2+). @@ -401,17 +401,25 @@ LADDER_LOAD=0 ./taler-monitoring.sh ladder Default **English**. Select language with `--lang en|fr` or `TALER_MON_LANG`. FrancPaysan stacks auto-select **French** (`*lefrancpaysan*`). See `VERSIONS.md` and `i18n.sh`. -## Public monitoring pages via FQDN (v1.3.1+) +## Public monitoring pages via FQDN (v1.3.1+ / obligatory v1.7.6+) -Phase **`monpages`** checks that each host’s monitoring URL is live HTML -(not merchant JSON code 21): +Phase **`monpages`** is **obligatory**: missing or non-suite HTML is **ERROR** +(not merchant JSON code 21, not 404). Soft mode only with +`MONPAGES_REQUIRE_PUBLIC=0` (escape hatch). + +| Stack | What is checked | +|-------|-----------------| +| **GOA / hacktivism** | Full suite inventory: `bank`/`exchange`/`taler` `/monitoring/` plus `/taler-monitoring-{surface,aptdeploy,mattermost,mail}/` and any extra pages found under staging/`DEPLOY_WWW_ROOT` | +| **FrancPaysan** | Only that stack’s mon hosts (`bank`/`exchange`/`monnaie` · stage variants) — never GOA URLs | ```bash ./taler-monitoring.sh -d hacktivism.ch monpages +./taler-monitoring.sh -d lefrancpaysan.ch monpages # host-agent runs monpages by default and again after HTML publish ``` -If pages are missing publicly: `sudo ~/koopa-caddy/apply-monitoring-live.sh` on koopa. +If pages are missing publicly: `sudo ~/koopa-caddy/apply-monitoring-live.sh` on koopa +(FP: Infomaniak vhost for `/monitoring*`). ## Version on monitoring pages (v1.3+ / sticky bar) diff --git a/TESTS.md b/TESTS.md index 85fe7e8..bbf4ad9 100644 --- a/TESTS.md +++ b/TESTS.md @@ -20,7 +20,7 @@ Every check line has a **global** run number and a **grouped** id: | **versions** | `check_versions.sh` | `outside` `inside` `compare` | | **aptdeploy** | `check_apt_deploy.sh` | `trixie` `trixie-testing` (koopa podman apt-src merchant smoke) | | **surface** | `check_surface.sh` | **remote-only** ecosystem / `-d` domain inventory (not in default/all/full) | -| **monpages** | `check_monitoring_pages.sh` | public monitoring HTML via FQDN (`https://host/monitoring/`, not merchant code 21) | +| **monpages** | `check_monitoring_pages.sh` | public monitoring HTML via FQDN — **obligatory ERROR**; GOA full inventory; FP only FP mon hosts | | **sanity** | `check_sanity.sh` | `bank` `exchange` `merchant` | | **server** | `check_server.sh` | (flat `server-NN` or host groups) | | **e2e** | `check_e2e.sh` | `prereq` `load` `bank` `wallet` `atm` `settle` `pay` `shop` `paivana` `dig` `report` | diff --git a/VERSION b/VERSION index 5849151..d263485 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.7.5 \ No newline at end of file +1.7.6 \ No newline at end of file diff --git a/VERSIONS.md b/VERSIONS.md index a788b0d..6eccfb7 100644 --- a/VERSIONS.md +++ b/VERSIONS.md @@ -18,6 +18,7 @@ Git tags: `vMAJOR.FEATURE.FIX` (e.g. `v1.7.5`). File `VERSION` omits the `v` pre | Tag | Date (UTC) | Notes | |-----|------------|--------| | **v1.8.0** | *planned* | **L10n** packs for user-facing strings: **fr-CH** (FrancPaysan) and **de-CH** (hacktivism / Taler CH German). | +| **v1.7.6** | 2026-07-18 | **monpages obligatory (ERROR):** GOA checks full suite inventory (bank/exchange/taler `/monitoring/` + surface/aptdeploy/mattermost/mail + on-disk discovery); FP only its own mon hosts; `MONPAGES_REQUIRE_PUBLIC=1` default (escape hatch `=0`). **Bugfix:** `with_timeout` no longer re-runs a phase when it exits non-zero. | | **v1.7.5** | 2026-07-18 | **Bugfix:** monpages soft mode `MONPAGES_REQUIRE_PUBLIC=0` (FP wrappers default) — missing public HTML is WARN until vhost publish; generic publish hints. | | **v1.7.4** | 2026-07-18 | **Bugfix:** monpages bare URLs (`/monitoring` without `/`) WARN by default (trailing slash remains ERROR); avoids false red when slash form is live. | | **v1.7.3** | 2026-07-18 | Monitoring **pages themselves** as suite scope: README section + sticky “What this monitors” block (monpages / public HTML paths). | @@ -39,7 +40,7 @@ Git tags: `vMAJOR.FEATURE.FIX` (e.g. `v1.7.5`). File `VERSION` omits the `v` pre ## Sticky bar version (v1.3.0+) -Generated HTML shows the installed release tag (e.g. `v1.7.5`) and links to Forgejo: +Generated HTML shows the installed release tag (e.g. `v1.7.6`) and links to Forgejo: `https://git.hacktivism.ch/hernani/taler-monitoring/src/tag/` @@ -69,5 +70,5 @@ Full locale packs (fr-CH / de-CH): **v1.8.0** (planned). ```bash git clone https://git.hacktivism.ch/hernani/taler-monitoring.git ~/src/taler-monitoring -cd ~/src/taler-monitoring && git checkout v1.7.5 +cd ~/src/taler-monitoring && git checkout v1.7.6 ``` diff --git a/check_monitoring_pages.sh b/check_monitoring_pages.sh index 5a118a5..1f55881 100755 --- a/check_monitoring_pages.sh +++ b/check_monitoring_pages.sh @@ -4,15 +4,23 @@ # Outside-in: curl https://// and require a real HTML monitoring page. # Catches: merchant JSON code 21 (Caddy handle missing), WP 404, empty stubs, deploy skip. # -# Env (same knobs as host-agent HTML layout): +# Policy (v1.7.6+): +# • monpages is obligatory → missing/wrong pages are ERROR (exit 1). +# • GOA / hacktivism: full inventory of suite monitoring sites (catalog + on-disk discovery). +# • FP: only that stack’s own monitoring hosts/paths (never GOA URLs). +# • Override soft mode only with MONPAGES_REQUIRE_PUBLIC=0 (emergency / staging). +# +# Env: # MON_HOSTS space-separated FQDNs (default from TALER_DOMAIN) # HTML_URL_OK path for OK page (default /monitoring/) -# HTML_URL_ERR optional err path; checked if MONPAGES_CHECK_ERR=1 -# MONITORING_PAGE_URLS optional full URL list (overrides host+path construction) +# HTML_URL_ERR err path; checked if present on disk or MONPAGES_CHECK_ERR=1 +# MONITORING_PAGE_URLS optional full URL list (overrides inventory construction) # MONPAGES_EXTRA_URLS additional absolute URLs (space-separated) +# MONPAGES_INVENTORY auto|full|job (default auto) +# auto/full = stack inventory; job = only MON_HOSTS+paths +# MONPAGES_REQUIRE_PUBLIC default 1 (ERROR). 0 = WARN only (escape hatch). +# MONPAGES_STAGING_BASE / HTML_OUT / DEPLOY_WWW_ROOT trees for discovery # TIMEOUT curl timeout (from lib.sh) -# MONPAGES_STAGING_BASE / HTML_OUT optional staging root for diagnosis -# HTML_OK_DIR staging subdir (default monitoring) # set -euo pipefail ROOT=$(cd "$(dirname "$0")" && pwd) @@ -20,10 +28,11 @@ ROOT=$(cd "$(dirname "$0")" && pwd) source "$ROOT/lib.sh" set_area monpages -section "monpages · public monitoring HTML via FQDN (outside-in)" +section "monpages · public monitoring HTML via FQDN (outside-in, obligatory)" -# MONPAGES_REQUIRE_PUBLIC=0 → missing public pages are WARN (staging-only stacks / FP until vhost) +# Obligatory: public pages missing → ERROR. Soft only if explicitly disabled. : "${MONPAGES_REQUIRE_PUBLIC:=1}" +: "${MONPAGES_INVENTORY:=auto}" # Default hosts from domain (mirror run-host-report.sh) if [ -z "${MON_HOSTS:-}" ]; then @@ -46,31 +55,131 @@ fi HTML_URL_OK="${HTML_URL_OK:-/monitoring/}" HTML_URL_ERR="${HTML_URL_ERR:-/monitoring_err/}" # ensure leading + trailing slash -case "$HTML_URL_OK" in - /*) ;; - *) HTML_URL_OK="/$HTML_URL_OK" ;; -esac -case "$HTML_URL_OK" in - */) ;; - *) HTML_URL_OK="${HTML_URL_OK}/" ;; -esac +_norm_path() { + local p="$1" + case "$p" in + /*) ;; + *) p="/$p" ;; + esac + case "$p" in + */) ;; + *) p="${p}/" ;; + esac + printf '%s' "$p" +} +HTML_URL_OK="$(_norm_path "$HTML_URL_OK")" +HTML_URL_ERR="$(_norm_path "$HTML_URL_ERR")" + +# Stack family: goa | fp | other — FP never checks GOA and vice versa. +_monpages_family() { + case "${TALER_DOMAIN:-}" in + *lefrancpaysan*|*francpaysan*) printf 'fp' ;; + hacktivism.ch|koopa) printf 'goa' ;; + *) + # Infer from MON_HOSTS if domain ambiguous + case " ${MON_HOSTS:-} " in + *lefrancpaysan*) printf 'fp' ;; + *hacktivism*) printf 'goa' ;; + *) printf 'other' ;; + esac + ;; + esac +} + +_host_allowed() { + local h="$1" fam="$2" + case "$fam" in + goa) + case "$h" in *.hacktivism.ch) return 0 ;; *) return 1 ;; esac + ;; + fp) + case "$h" in *lefrancpaysan*) return 0 ;; *) return 1 ;; esac + ;; + *) return 0 ;; + esac +} + +# Emit https://host/path/ for each …/host/path/index.html under root (max depth 3). +_discover_tree() { + local root="$1" fam="$2" + local f rel host path + [ -n "$root" ] && [ -d "$root" ] || return 0 + # layout: $root///index.html + while IFS= read -r -d '' f; do + rel="${f#"${root%/}"/}" + host="${rel%%/*}" + path="${rel#*/}" + path="${path%/index.html}" + [ -n "$host" ] && [ -n "$path" ] || continue + _host_allowed "$host" "$fam" || continue + printf 'https://%s/%s/\n' "$host" "$path" + done < <(find "${root%/}" -mindepth 3 -maxdepth 3 -type f -name index.html -print0 2>/dev/null) +} + +# Suite catalog of OK pages that must exist for this family. +_catalog_urls() { + local fam="$1" h + case "$fam" in + goa) + for h in bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch; do + printf 'https://%s/monitoring/\n' "$h" + done + for h in \ + taler-monitoring-surface \ + taler-monitoring-aptdeploy \ + taler-monitoring-mattermost \ + taler-monitoring-mail + do + printf 'https://taler.hacktivism.ch/%s/\n' "$h" + done + ;; + fp) + for h in $MON_HOSTS; do + [ -z "$h" ] && continue + _host_allowed "$h" fp || continue + printf 'https://%s/monitoring/\n' "$h" + done + ;; + esac +} + +# Job-local URLs (this host-agent run’s path). +_job_urls() { + local h p + for h in $MON_HOSTS; do + [ -z "$h" ] && continue + printf 'https://%s%s\n' "$h" "$HTML_URL_OK" + if [ "${MONPAGES_CHECK_ERR:-0}" = "1" ]; then + printf 'https://%s%s\n' "$h" "$HTML_URL_ERR" + fi + done +} build_urls() { - local h p + local fam inv + fam="$(_monpages_family)" + inv="${MONPAGES_INVENTORY:-auto}" + if [ -n "${MONITORING_PAGE_URLS:-}" ]; then # shellcheck disable=SC2086 printf '%s\n' $MONITORING_PAGE_URLS else - for h in $MON_HOSTS; do - [ -z "$h" ] && continue - printf 'https://%s%s\n' "$h" "$HTML_URL_OK" - if [ "${MONPAGES_CHECK_ERR:-0}" = "1" ]; then - p="$HTML_URL_ERR" - case "$p" in /*) ;; *) p="/$p" ;; esac - case "$p" in */) ;; *) p="${p}/" ;; esac - printf 'https://%s%s\n' "$h" "$p" - fi - done + case "$inv" in + job) + _job_urls + ;; + full|auto|*) + # Always include this job’s paths (host-agent post-check). + _job_urls + # Stack inventory: GOA = all suite mon sites; FP = only FP mon hosts. + if [ "$fam" = "goa" ] || [ "$fam" = "fp" ]; then + _catalog_urls "$fam" + # On-disk discovery (err pages only exist after failed runs; extras welcome) + _discover_tree "${DEPLOY_WWW_ROOT:-/var/www/monitoring-sites}" "$fam" + _discover_tree "${MONPAGES_STAGING_BASE:-${HTML_OUT:-${HTML_BASE:-$HOME/monitoring-sites-staging}}}" "$fam" + fi + ;; + esac fi if [ -n "${MONPAGES_EXTRA_URLS:-}" ]; then # shellcheck disable=SC2086 @@ -115,7 +224,6 @@ _is_bare_url() { esac } - _mon_fail_or_soft() { local label="$1" detail="$2" if [ "${MONPAGES_REQUIRE_PUBLIC:-1}" != "1" ]; then @@ -182,7 +290,7 @@ while IFS= read -r line; do done [ "$skip" = "1" ] && continue URLS+=("$line") -done < <(build_urls) +done < <(build_urls | sort -u) # Bare URLs without trailing slash: optional soft check (default on). # Many reverse proxies only redirect /monitoring → /monitoring/; bare may hit the app (code 21). @@ -202,7 +310,6 @@ if [ "${MONPAGES_CHECK_BARE:-1}" = "1" ]; then done [ "$skip" = "1" ] && continue URLS+=("$u") - # mark bare for soft handling via env list MONPAGES_BARE_URLS="${MONPAGES_BARE_URLS:-} $u" done export MONPAGES_BARE_URLS @@ -213,6 +320,8 @@ if [ "${#URLS[@]}" -eq 0 ]; then exit 1 fi +_fam="$(_monpages_family)" +echo " family=${_fam} inventory=${MONPAGES_INVENTORY} require_public=${MONPAGES_REQUIRE_PUBLIC}" echo " checking ${#URLS[@]} URL(s) via FQDN..." ec=0 for u in "${URLS[@]}"; do @@ -221,8 +330,14 @@ done if [ "$ec" -ne 0 ]; then echo " hint: public monitoring HTML not served (404 / merchant code 21)" - echo " publish staging HTML + configure reverse-proxy/Caddy handle for /monitoring*" + echo " publish staging HTML + configure reverse-proxy/Caddy handle for mon paths" echo " (on koopa: sudo ~/koopa-caddy/apply-monitoring-live.sh)" + if [ "$_fam" = "fp" ]; then + echo " FP: only FP mon hosts are checked — wire Infomaniak vhost for /monitoring*" + fi + if [ "$_fam" = "goa" ]; then + echo " GOA: full inventory (bank/exchange/taler /monitoring/ + surface/aptdeploy/mattermost/mail)" + fi # Staging vs public diagnosis (host-agent sets MONPAGES_STAGING_BASE / HTML_OUT) _stg="${MONPAGES_STAGING_BASE:-${HTML_OUT:-${HTML_BASE:-$HOME/monitoring-sites-staging}}}" _ok_dir="${HTML_OK_DIR:-monitoring}" diff --git a/host-agent/README.md b/host-agent/README.md index a01c91b..c37cb5f 100644 --- a/host-agent/README.md +++ b/host-agent/README.md @@ -195,4 +195,4 @@ Set `STRICT_EXIT=0` only if you want timers to stay green while still logging er ## Monitoring pages (suite) -Public sticky-bar HTML under `/monitoring*` and `/taler-monitoring-*` is part of **taler-monitoring** (not a separate product). See main [README — Monitoring pages](../README.md#monitoring-pages-part-of-the-suite). Phase **monpages** verifies they are live. +Public sticky-bar HTML under `/monitoring*` and `/taler-monitoring-*` is part of **taler-monitoring** (not a separate product). See main [README — Monitoring pages](../README.md#monitoring-pages-part-of-the-suite). Phase **monpages** is obligatory (**ERROR** if missing): GOA checks the full suite inventory; FP only FP mon hosts. diff --git a/host-agent/run-fp-prod-monitoring.sh b/host-agent/run-fp-prod-monitoring.sh index 20dfd71..3dad0d6 100755 --- a/host-agent/run-fp-prod-monitoring.sh +++ b/host-agent/run-fp-prod-monitoring.sh @@ -15,8 +15,8 @@ export INSIDE_PODMAN="${INSIDE_PODMAN:-1}" export INSIDE_MODE="${INSIDE_MODE:-local-podman}" export LOCAL_STACK="${LOCAL_STACK:-0}" export CONTINUE_ON_ERROR="${CONTINUE_ON_ERROR:-1}" -# Public /monitoring* not wired on Infomaniak yet — monpages WARN only -export MONPAGES_REQUIRE_PUBLIC="${MONPAGES_REQUIRE_PUBLIC:-0}" +# monpages obligatory (ERROR): only FP mon hosts below — never GOA inventory +export MONPAGES_REQUIRE_PUBLIC="${MONPAGES_REQUIRE_PUBLIC:-1}" export RUN_TIMEOUT="${RUN_TIMEOUT:-600}" export PHASES="${PHASES:-urls inside versions monpages}" export MON_HOSTS="${MON_HOSTS:-bank.lefrancpaysan.ch exchange.lefrancpaysan.ch monnaie.lefrancpaysan.ch}" diff --git a/host-agent/run-fp-stage-monitoring.sh b/host-agent/run-fp-stage-monitoring.sh index 6bf4b5b..91db2da 100755 --- a/host-agent/run-fp-stage-monitoring.sh +++ b/host-agent/run-fp-stage-monitoring.sh @@ -16,8 +16,8 @@ export INSIDE_MODE="${INSIDE_MODE:-local-podman}" export INSIDE_PROFILE="${INSIDE_PROFILE:-stage-lfp}" export LOCAL_STACK="${LOCAL_STACK:-0}" export CONTINUE_ON_ERROR="${CONTINUE_ON_ERROR:-1}" -# Public /monitoring* not wired on Infomaniak yet — monpages WARN only -export MONPAGES_REQUIRE_PUBLIC="${MONPAGES_REQUIRE_PUBLIC:-0}" +# monpages obligatory (ERROR): only FP stage mon hosts below — never GOA inventory +export MONPAGES_REQUIRE_PUBLIC="${MONPAGES_REQUIRE_PUBLIC:-1}" export RUN_TIMEOUT="${RUN_TIMEOUT:-600}" export PHASES="${PHASES:-urls inside versions monpages}" export MON_HOSTS="${MON_HOSTS:-stage.bank.lefrancpaysan.ch stage.exchange.lefrancpaysan.ch stage.monnaie.lefrancpaysan.ch}" diff --git a/host-agent/run-host-report.sh b/host-agent/run-host-report.sh index 84d0d5e..f5ea6ce 100755 --- a/host-agent/run-host-report.sh +++ b/host-agent/run-host-report.sh @@ -412,12 +412,16 @@ else echo "WARN: DEPLOY_WWW_ROOT empty — skipping live publish (staging only: $HTML_BASE)" >&2 fi -# Post-publish: verify this job's pages via FQDN (same paths as monpages phase) +# Post-publish: monpages obligatory (v1.7.6+) — GOA full inventory / FP only FP hosts _monpages_failed=0 if [ "${MONPAGES_POSTCHECK:-1}" = "1" ] && [ -x "$ROOT/check_monitoring_pages.sh" ]; then - echo "--- monpages post-check (public FQDN) ---" + echo "--- monpages post-check (public FQDN, obligatory) ---" export MON_HOSTS HTML_URL_OK HTML_URL_ERR TALER_DOMAIN HTML_OUT HTML_BASE export MONPAGES_STAGING_BASE="${HTML_BASE}" + export DEPLOY_WWW_ROOT="${DEPLOY_WWW:-${DEPLOY_WWW_ROOT:-}}" + export MONPAGES_REQUIRE_PUBLIC="${MONPAGES_REQUIRE_PUBLIC:-1}" + # auto: GOA = all suite mon sites; FP = only FP mon hosts; job paths always included + export MONPAGES_INVENTORY="${MONPAGES_INVENTORY:-auto}" if ! bash "$ROOT/check_monitoring_pages.sh"; then echo "ERROR: public monitoring page(s) missing or not suite HTML via FQDN" >&2 echo " staging HTML_OUT=${HTML_BASE} · DEPLOY_WWW=${DEPLOY_WWW:-}" >&2 diff --git a/lib.sh b/lib.sh index 1ae5b6a..bdd4887 100755 --- a/lib.sh +++ b/lib.sh @@ -323,13 +323,24 @@ SSH_BASE_OPTS=( # Hard wall-clock timeout so ssh/curl never block the monitoring run forever. with_timeout() { local secs="$1"; shift + local rc if command -v gtimeout >/dev/null 2>&1; then gtimeout --kill-after=2 "$secs" "$@" return $? fi if command -v timeout >/dev/null 2>&1; then - timeout -k 2 "$secs" "$@" 2>/dev/null || timeout --kill-after=2 "$secs" "$@" - return $? + # Prefer GNU -k; if that flag form is unsupported, fall back once. + # Do NOT use `cmd || cmd` on the child — a phase exit 1 would re-run it. + set +e + timeout -k 2 "$secs" "$@" 2>/dev/null + rc=$? + set -e + # 125 = timeout(1) itself failed (bad options); 126/127 = not executable + if [ "$rc" -eq 125 ] || [ "$rc" -eq 126 ] || [ "$rc" -eq 127 ]; then + timeout --kill-after=2 "$secs" "$@" + return $? + fi + return "$rc" fi # Portable: perl alarm + process group kill perl -e ' diff --git a/taler-monitoring.sh b/taler-monitoring.sh index e149fd7..e3e342b 100755 --- a/taler-monitoring.sh +++ b/taler-monitoring.sh @@ -102,7 +102,7 @@ Phases: or -d DOMAIN → that domain’s surface; port/protocol/TLS/CVE (OSV) mattermost Mattermost chat health (default mattermost.taler.net; SPA + /api/v4/system/ping) mail MX/SMTP/IMAP for Taler mail (firefly, pixel/TSA, catalogued domains) - monpages public monitoring HTML via FQDN (HTTP 200 + suite page, not code 21) + monpages public monitoring HTML via FQDN (obligatory ERROR; GOA full inventory, FP only FP) uses MON_HOSTS + HTML_URL_OK (same as host-agent); v1.3.1+ all urls + inside + versions + sanity + e2e (SSH phases only on koopa) full all + server + ladder + auth401 (maximum; long-running, needs secrets)