diff --git a/scripts/taler-monitoring/site-gen/README.md b/scripts/taler-monitoring/site-gen/README.md new file mode 100644 index 0000000..82e52d2 --- /dev/null +++ b/scripts/taler-monitoring/site-gen/README.md @@ -0,0 +1,81 @@ +# monitoring site-gen + +Build **console-style HTML** from `taler-monitoring` runs and stage them for +**host Caddy** (`/monitoring`, `/monitoring_err`). + +Same **global reporting defaults** as host-agents (`run-host-report.sh`): + +- **`RUN_TIMEOUT=600`** on each suite run +- **line-buffered** logs (continuous write) +- **always** write HTML (first run / empty logs too) +- Forgejo **commit link** in the page header + +Applies to all nine fronts (GOA + FP stage + FP prod) in the default `SITES` list. + +| Host | Role | +|------|------| +| **firecuda-external** | optional runner (public DNS) | +| **koopa-external** | deploy staging + Caddy (root service) | +| **host-agents** | primary: GOA + FP on-box (`../host-agent/`) | + +No secrets in `settings.conf` — only hosts, paths, site list. + +## firecuda-external (outside-only) — **disabled for now** + +**firecuda-external could** run public-only monitoring (`SKIP_SSH=1`, phases +`urls versions`) on a schedule (launchd every 4h) and write HTML under +`~/var/taler-monitoring-sites-work/html/`. Scripts for that still live here +(`install-firecuda-timer.sh`, `run-on-firecuda.sh`, plist template). + +**Current policy: do not run a real launchd job on firecuda.** +Primary GOA monitoring is **koopa host-agent** (`host-agent/`, user `hernani`, +local `podman exec`). Re-enable firecuda only if you explicitly want a second, +outside-only view. + +```bash +# optional one-shot (no timer): +# ssh firecuda-external '~/taler-monitoring-site-gen/site-gen/run-on-firecuda.sh' + +# if a timer was installed earlier, ensure it is off: +# ssh firecuda-external 'launchctl list | grep taler-monitoring || echo off' +``` + +Disabled plists may remain as `*.plist.disabled-*` under `~/Library/LaunchAgents/`. + +## Quick start (laptop) + +```bash +cd scripts/taler-monitoring/site-gen +cp settings.conf.example settings.conf # optional +chmod +x generate-monitoring-sites.sh deploy-monitoring-sites.sh console_to_html.py + +ONLY_HOSTS="bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch" \ + SKIP_SSH=1 RUNNER_SSH= \ + ./generate-monitoring-sites.sh + +./deploy-monitoring-sites.sh +``` + +Then on koopa **as root**: see **[ROOT-ON-KOOPA.md](./ROOT-ON-KOOPA.md)**. + +## URL rules + +| Path | When | +|------|------| +| `/monitoring_err/` | always: full console log + error list on top (CONTINUE_ON_ERROR run) | +| `/monitoring/` | **clean** full log if last strict run exit 0; else **stub** linking to `/monitoring_err/` | + +Footer links **Forgejo commit** of the admin-log tree used for that run. + +## Layout + +```text +site-gen/ + settings.conf.example + generate-monitoring-sites.sh + deploy-monitoring-sites.sh + console_to_html.py + caddy-monitoring-handles.snippet + ROOT-ON-KOOPA.md + README.md +``` diff --git a/scripts/taler-monitoring/site-gen/ROOT-ON-KOOPA.md b/scripts/taler-monitoring/site-gen/ROOT-ON-KOOPA.md new file mode 100644 index 0000000..604b997 --- /dev/null +++ b/scripts/taler-monitoring/site-gen/ROOT-ON-KOOPA.md @@ -0,0 +1,92 @@ +# Root on koopa (hacktivism monitoring sites) + +Host Caddy runs as **root/systemd** (`caddy` user). Static monitoring HTML is +**not** put into Taler containers — only host paths + Caddy `handle`. + +After `generate-monitoring-sites.sh` + `deploy-monitoring-sites.sh` (as hernani), +files live in: + +```text +/home/hernani/monitoring-sites-staging//monitoring/index.html +/home/hernani/monitoring-sites-staging//monitoring_err/index.html +``` + +## One-shot as root (copy + Caddy) + +```bash +# on koopa (koopa-external), as root: + +# 1) web root +install -d -o caddy -g caddy -m 755 /var/www/monitoring-sites +rsync -a --delete /home/hernani/monitoring-sites-staging/ /var/www/monitoring-sites/ +chown -R caddy:caddy /var/www/monitoring-sites + +# 2) Caddyfile — merge handles from admin-log mirror, then: +# (either edit /etc/caddy/Caddyfile by hand using snippet below, +# or copy full mirror after review) +# +# install -m 644 /home/hernani/koopa-admin-log/configs/caddy/Caddyfile /etc/caddy/Caddyfile +# # or: ~/koopa-caddy/apply.sh after syncing Caddyfile into ~/koopa-caddy/ + +caddy validate --config /etc/caddy/Caddyfile +systemctl reload caddy +systemctl is-active caddy +``` + +## Snippet per hacktivism site (inside each `*.hacktivism.ch { }` block) + +Place **before** the catch-all `reverse_proxy` (same idea as `/intro*`): + +```caddy + # Public taler-monitoring console HTML (static; host only) + handle /monitoring_err* { + root * /var/www/monitoring-sites/{host} + rewrite * /monitoring_err{uri} + # uri is /monitoring_err or /monitoring_err/ → serve directory index + file_server + } + handle /monitoring* { + root * /var/www/monitoring-sites/{host} + file_server + } +``` + +Simpler layout (recommended): path on disk matches URL under host root: + +```text +/var/www/monitoring-sites/bank.hacktivism.ch/monitoring/index.html +/var/www/monitoring-sites/bank.hacktivism.ch/monitoring_err/index.html +``` + +```caddy + handle_path /monitoring_err/* { + root * /var/www/monitoring-sites/{host}/monitoring_err + file_server + } + handle /monitoring_err { + redir /monitoring_err/ 302 + } + handle_path /monitoring/* { + root * /var/www/monitoring-sites/{host}/monitoring + file_server + } + handle /monitoring { + redir /monitoring/ 302 + } +``` + +Canonical snippet is also in `caddy-monitoring-handles.snippet` and applied +in `configs/caddy/Caddyfile` for the three GOA hosts. + +## Check + +```bash +curl -sS -o /dev/null -w '%{http_code}\n' https://bank.hacktivism.ch/monitoring/ +curl -sS -o /dev/null -w '%{http_code}\n' https://bank.hacktivism.ch/monitoring_err/ +curl -sS https://bank.hacktivism.ch/monitoring_err/ | head +``` + +## Not needed as root + +- Running `taler-monitoring` (use **firecuda-external** / laptop) +- Writing into podman Taler containers diff --git a/scripts/taler-monitoring/site-gen/caddy-monitoring-handles.snippet b/scripts/taler-monitoring/site-gen/caddy-monitoring-handles.snippet new file mode 100644 index 0000000..1c09531 --- /dev/null +++ b/scripts/taler-monitoring/site-gen/caddy-monitoring-handles.snippet @@ -0,0 +1,57 @@ +# Host Caddy on koopa (root service). Paste BEFORE catch-all reverse_proxy. +# +# Disk layout: +# /var/www/monitoring-sites/{host}/monitoring/index.html +# /var/www/monitoring-sites/{host}/monitoring_err/index.html +# /var/www/monitoring-sites/taler.hacktivism.ch/taler-monitoring-surface*/index.html +# /var/www/monitoring-sites/taler.hacktivism.ch/taler-monitoring-aptdeploy*/index.html +# +# /monitoring* → bank + exchange + taler.hacktivism.ch +# /taler-monitoring-surface* → taler.hacktivism.ch only +# /taler-monitoring-aptdeploy* → taler.hacktivism.ch only + +# --- only inside taler.hacktivism.ch { ... } --- + handle /taler-monitoring-surface_err { + redir /taler-monitoring-surface_err/ 302 + } + handle /taler-monitoring-surface_err/ { + root * /var/www/monitoring-sites/taler.hacktivism.ch/taler-monitoring-surface_err + file_server + } + handle /taler-monitoring-surface { + redir /taler-monitoring-surface/ 302 + } + handle /taler-monitoring-surface/ { + root * /var/www/monitoring-sites/taler.hacktivism.ch/taler-monitoring-surface + file_server + } + handle /taler-monitoring-aptdeploy_err { + redir /taler-monitoring-aptdeploy_err/ 302 + } + handle /taler-monitoring-aptdeploy_err/ { + root * /var/www/monitoring-sites/taler.hacktivism.ch/taler-monitoring-aptdeploy_err + file_server + } + handle /taler-monitoring-aptdeploy { + redir /taler-monitoring-aptdeploy/ 302 + } + handle /taler-monitoring-aptdeploy/ { + root * /var/www/monitoring-sites/taler.hacktivism.ch/taler-monitoring-aptdeploy + file_server + } + +# --- bank + exchange + taler (each site block) --- + handle /monitoring_err { + redir /monitoring_err/ 302 + } + handle /monitoring_err/ { + root * /var/www/monitoring-sites/{host}/monitoring_err + file_server + } + handle /monitoring { + redir /monitoring/ 302 + } + handle /monitoring/ { + root * /var/www/monitoring-sites/{host}/monitoring + file_server + } diff --git a/scripts/taler-monitoring/site-gen/com.hacktivism.taler-monitoring-sites.plist b/scripts/taler-monitoring/site-gen/com.hacktivism.taler-monitoring-sites.plist new file mode 100644 index 0000000..0b6f68b --- /dev/null +++ b/scripts/taler-monitoring/site-gen/com.hacktivism.taler-monitoring-sites.plist @@ -0,0 +1,34 @@ + + + + + + Label + com.hacktivism.taler-monitoring-sites + ProgramArguments + + /bin/bash + __SITE_GEN__/run-on-firecuda.sh + + StartInterval + 14400 + RunAtLoad + + StandardOutPath + __HOME__/Library/Logs/taler-monitoring-sites/launchd.out.log + StandardErrorPath + __HOME__/Library/Logs/taler-monitoring-sites/launchd.err.log + EnvironmentVariables + + PATH + /opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin + SKIP_SSH + 1 + CONTINUE_ON_ERROR + 1 + + + diff --git a/scripts/taler-monitoring/site-gen/console_to_html.py b/scripts/taler-monitoring/site-gen/console_to_html.py new file mode 100755 index 0000000..14aef2a --- /dev/null +++ b/scripts/taler-monitoring/site-gen/console_to_html.py @@ -0,0 +1,935 @@ +#!/usr/bin/env python3 +"""Convert taler-monitoring console log to a console-feel HTML page.""" +from __future__ import annotations + +import argparse +import html +import re +from datetime import datetime, timezone +from pathlib import Path + +# ANSI strip +ANSI_RE = re.compile(r"\x1b\[[0-9;]*m") +ERR_BULLET_RE = re.compile(r"^\s*[•·*-]\s+(?P.+)$") +TID_RE = re.compile(r"\b(?P[a-z0-9_.-]+-\d+)\b", re.I) + + +def strip_ansi(s: str) -> str: + return ANSI_RE.sub("", s) + + +def classify(line: str) -> str: + u = line.upper() + # section / summary headers before badge matching (they contain the word ERROR) + if "SUMMARY" in u or "--- ERRORS" in u or "ERRORS ·" in u or "ERRORS (" in u: + return "meta" + if "numbered checks" in line.lower() or "totals:" in line.lower(): + return "meta" + if "BLOCKER" in u or "┌ BLOCKER" in u: + return "blocker" + if "ERROR" in u or "┌ ERROR" in u: + return "error" + if "WARN" in u or "┌ WARN" in u: + return "warn" + if "INFO" in u or "┌ INFO" in u: + return "info" + if " OK" in u or "[OK" in u or "┌ OK" in u or u.strip().startswith("OK"): + return "ok" + if u.strip().startswith("-- ") or u.strip().startswith("=="): + return "section" + return "plain" + + +def is_run_timeout_text(text: str) -> bool: + u = text.upper() + return "RUN.TIMEOUT" in u or "RUN_TIMEOUT" in u or "RUN TIMEOUT" in u + + +def is_summary_error_line(ln: str) -> bool: + """Skip summary/header lines that contain ERROR but are not checks.""" + low = ln.lower() + if "failed — see" in low or "failed - see" in low: + return True + if "errors · failed" in low or "errors (failed" in low: + return True + if "--- errors" in low: + return True + if "┌ summary" in low or "[ summary" in low: + return True + if "totals:" in low: + return True + # header-ish RUN_TIMEOUT= without error badge + if is_run_timeout_text(ln) and "ERROR" not in ln.upper() and "run.timeout" not in low: + return True + return False + + +def extract_errors(lines: list[str]) -> list[str]: + errs: list[str] = [] + timeout_detail = "" + in_block = False + for ln in lines: + if "--- ERRORS" in ln or "ERRORS (failed" in ln or "RUN TIMEOUT · extraordinary" in ln: + in_block = True + continue + if in_block: + if ln.strip().startswith("---") or "[ SUMMARY" in ln or "totals:" in ln: + in_block = False + continue + m = ERR_BULLET_RE.match(ln) + if m: + body = m.group("body").strip() + if is_run_timeout_text(body): + timeout_detail = body + else: + errs.append(body) + elif "ERROR" in ln.upper() and ln.strip() and not is_run_timeout_text(ln): + errs.append(ln.strip()) + if re.search(r"\bERROR\b", ln, re.I) and "--- ERRORS" not in ln: + if is_run_timeout_text(ln): + if "run.timeout-01" in ln.lower() or "RUN_TIMEOUT exceeded" in ln: + timeout_detail = re.sub(r"\s+", " ", strip_ansi(ln)).strip() + continue + if is_summary_error_line(ln): + continue + body = re.sub(r"^.*\bERROR\b\s*\]?\s*", "", ln, flags=re.I).strip(" ·") + if body and body not in errs and "failed — see" not in body.lower(): + if re.search( + r"#\d+|www\.|e2e\.|auth401\.|versions\.|inside\.|surface\.|aptdeploy\.", + body, + ): + errs.append(body) + seen: set[str] = set() + out: list[str] = [] + for e in errs: + if e in seen or is_run_timeout_text(e): + continue + seen.add(e) + out.append(e) + if timeout_detail or any( + is_run_timeout_text(l) and ("ERROR" in l.upper() or "run.timeout" in l.lower()) + for l in lines + ): + # only if real timeout ERROR badge, not mere RUN_TIMEOUT= header + if timeout_detail or any( + "run.timeout-01" in l.lower() or "RUN_TIMEOUT exceeded" in l for l in lines + ): + canon = timeout_detail or "run.timeout-01 [run] RUN_TIMEOUT exceeded" + out = [canon] + out + return out + + +def count_status(lines: list[str]) -> tuple[int, int, int | None, int | None]: + """Return (error_count, warn_count, first_error_idx, first_warn_idx).""" + n_err = 0 + n_warn = 0 + first_err: int | None = None + first_warn: int | None = None + for i, ln in enumerate(lines): + kind = classify(ln) + if kind in ("error", "blocker"): + if is_summary_error_line(ln): + continue + n_err += 1 + if first_err is None: + first_err = i + elif kind == "warn": + n_warn += 1 + if first_warn is None: + first_warn = i + return n_err, n_warn, first_err, first_warn + + +def slug_error(i: int, text: str) -> str: + if is_run_timeout_text(text): + return "err-run-timeout" + m = TID_RE.search(text) + if m: + return "err-" + re.sub(r"[^a-z0-9_-]+", "-", m.group("tid").lower()) + return f"err-{i:03d}" + + +def render_line(ln: str, err_slugs: dict[str, str], extra_ids: list[str] | None = None) -> str: + kind = classify(ln) + esc = html.escape(ln) + for tid, slug in err_slugs.items(): + if tid in ln: + esc = esc.replace( + html.escape(tid), + f'{html.escape(tid)}', + ) + id_attr = "" + if extra_ids: + id_attr = f' id="{" ".join(extra_ids)}"' # invalid multi-id; use first only + # HTML allows one id — join carefully + id_attr = f' id="{html.escape(extra_ids[0])}"' + if len(extra_ids) > 1: + # secondary anchors as empty spans prepended + spans = "".join(f'' for x in extra_ids[1:]) + return f'{spans}{esc}\n' + return f'{esc}\n' + + +def status_level(n_err: int, n_warn: int) -> str: + if n_err > 0: + return "red" + if n_warn > 0: + return "yellow" + return "green" + + +def extract_phases(log_text: str) -> str: + """Best-effort phases string from host-agent / taler-monitoring log header.""" + for pat in ( + r"phases=([a-z0-9 _.-]+)", + r"· phases=([a-z0-9 _.-]+)", + r"phases\s{2,}([a-z0-9 _.-]+)", + r"^\s*phases\s+([a-z0-9 _.-]+)\s*$", + ): + m = re.search(pat, log_text, re.I | re.M) + if m: + return re.sub(r"\s+", " ", m.group(1)).strip(" ·") + return "" + + +def monitoring_scope( + page_label: str, + hostname: str, + log_text: str = "", +) -> dict[str, object]: + """ + Human-readable description of what this monitoring page covers. + Returned keys: title, summary, items (list[str]), kind + """ + label = (page_label or "monitoring").lower().replace("_", "-") + phases = extract_phases(log_text) + host = hostname or "?" + + if "surface" in label: + return { + "kind": "surface", + "title": "Remote ecosystem surface inventory", + "summary": "Public catalog hosts · DNS / TCP / HTTPS / TLS · Server version · CVE", + "items": [ + "Outside-in only: no SSH and no podman exec on remote targets", + "Catalog (surface-catalog.conf): taler.net, demo/test, gnunet.org, " + "taler-ops.ch, taler-systems.com, deb.taler.net, ftp.gnu.org, …", + "Per host: DNS resolve, ICMP (info), open ports, protocol probes, " + "TLS cert expiry, Server-header software version, optional CVE check", + f"HTML published only on taler.hacktivism.ch " + f"(this page host: {host})", + "Timer: taler-monitoring-surface.timer (hourly)", + ], + } + + if "aptdeploy" in label or "apt-deploy" in label or "apt_src" in label: + return { + "kind": "aptdeploy", + "title": "apt-src merchant deploy tests (podman on koopa)", + "summary": "4 containers · fresh + upgrade tracks · trixie & trixie-testing", + "items": [ + "koopa-taler-deploy-test-apt-src-trixie (fresh, suite trixie)", + "koopa-taler-deploy-test-apt-src-trixie-testing (fresh, trixie-testing)", + "koopa-taler-deploy-test-apt-src-trixie-upgrade (upgrade track)", + "koopa-taler-deploy-test-apt-src-trixie-testing-upgrade (upgrade track)", + "Checks: packages, taler-merchant-httpd --version / ldd, systemd unit, " + "optional local /config probe — not public nginx/HTTPS", + f"HTML published only on taler.hacktivism.ch (this page host: {host})", + "Timer: taler-monitoring-aptdeploy.timer (4h)", + ], + } + + # Default: stack /monitoring pages (bank, exchange, merchant) + phase_txt = phases or "urls · inside · versions" + role = "stack" + if host.startswith("bank."): + role = "bank" + focus = "Libeufin bank public HTTPS + container inside checks" + elif host.startswith("exchange.") or host.startswith("stage.exchange."): + role = "exchange" + focus = "Exchange public HTTPS + container inside checks" + elif host.startswith("taler.") or "merchant" in host or host.startswith("monnaie."): + role = "merchant" + focus = "Merchant backend / SPA public HTTPS + container inside checks" + else: + focus = "Taler stack public + inside checks" + + items = [ + f"Focus: {focus}", + f"Phases this run: {phase_txt}", + "Typical: public URLs (HTTPS, QR, perf), inside (podman/ssh), package versions", + f"HTML host: {host}", + "Timer: taler-monitoring-hacktivism (path + 4h) · /monitoring on bank, exchange, taler", + ] + if phases: + items.insert(2, f"Log header phases= {phases}") + + return { + "kind": role, + "title": f"Stack monitoring · {host}", + "summary": f"{phase_txt} · public + inside", + "items": items, + } + + +def sticky_bar_html( + *, + generated_at: str, + generated_iso: str, + n_err: int, + n_warn: int, + hostname: str, + page_label: str, + mode: str, + commit_html: str, + suite_path: str, + link_other: str | None, + first_err_href: str | None, + first_warn_href: str | None, + scope: dict[str, object] | None = None, +) -> str: + level = status_level(n_err, n_warn) + if level == "red": + status_txt = "ERRORS" + elif level == "yellow": + status_txt = "WARNINGS" + else: + status_txt = "OK" + + if first_err_href and n_err: + err_stat = ( + f'{n_err} error{"s" if n_err != 1 else ""}' + ) + else: + err_stat = f'{n_err} errors' + + if first_warn_href and n_warn: + warn_stat = ( + f'{n_warn} warning{"s" if n_warn != 1 else ""}' + ) + else: + warn_stat = f'{n_warn} warnings' + + other = "" + if link_other: + other = ( + f'' + f"{html.escape(link_other)}" + ) + + if scope is None: + scope = monitoring_scope(page_label, hostname, "") + scope_title = html.escape(str(scope.get("title") or "Monitoring")) + scope_summary = html.escape(str(scope.get("summary") or "")) + scope_kind = html.escape(str(scope.get("kind") or "monitoring")) + lis = "\n".join( + f"
  • {html.escape(str(it))}
    • " for it in (scope.get("items") or []) + ) + suite_bit = ( + f" · {html.escape(suite_path)}" if suite_path else "" + ) + + # Compact sticky row always visible; "Was geprüft wird" expands inside sticky-bar + return f""" + +""" + + +STICKY_CSS = """ +.sticky-bar { + position: sticky; top: 0; z-index: 100; + border-bottom: 2px solid var(--border); + padding: 10px 14px 8px; + backdrop-filter: blur(8px); + -webkit-backdrop-filter: blur(8px); + box-shadow: 0 2px 12px #0008; +} +.sticky-bar.sticky-green { + background: linear-gradient(180deg, #0d1f14f2, #0a1410ee); + border-bottom-color: #1f6b3a; +} +.sticky-bar.sticky-yellow { + background: linear-gradient(180deg, #2a2210f2, #1a160cee); + border-bottom-color: #a68b2d; +} +.sticky-bar.sticky-red { + background: linear-gradient(180deg, #2a1010f2, #1a0808ee); + border-bottom-color: #a33; +} +.sticky-bar-row { + display: flex; flex-wrap: wrap; align-items: center; gap: 6px 10px; + max-width: 1200px; +} +.sticky-bar-row.secondary { + margin-top: 8px; font-size: 12px; color: var(--dim); +} +.sticky-bar-row.secondary a { color: var(--info); text-decoration: none; } +.sticky-bar-row.secondary a:hover { text-decoration: underline; } +.status-pill { + display: inline-block; font-size: 11px; font-weight: 800; + letter-spacing: 0.06em; padding: 2px 10px; border-radius: 3px; + border: 1px solid; +} +.status-pill.sticky-green { color: var(--ok); border-color: #1f6b3a; background: #0a1a10; } +.status-pill.sticky-yellow { color: var(--warn); border-color: #a68b2d; background: #1a1608; } +.status-pill.sticky-red { color: var(--err); border-color: #a33; background: #1a0a0a; } +.host { font-weight: 600; color: #eee; font-size: 13px; } +.sep { color: var(--dim); } +.stat { + font-weight: 700; font-size: 12px; text-decoration: none; + padding: 1px 8px; border-radius: 3px; border: 1px solid transparent; +} +.stat.err { color: var(--err); border-color: #522; background: #1a0a0a; } +.stat.warn { color: var(--warn); border-color: #664; background: #1a1608; } +.stat.muted { opacity: 0.55; font-weight: 600; } +a.stat:hover { text-decoration: underline; filter: brightness(1.15); } +.generated { color: var(--dim); font-size: 12px; } +.generated time { color: #bbb; } +.generated .age { color: #888; margin-left: 4px; } +.generated .age:not(:empty)::before { content: "("; } +.generated .age:not(:empty)::after { content: ")"; } +.badge { + display: inline-block; padding: 1px 8px; border-radius: 3px; + font-size: 11px; font-weight: 700; margin-right: 4px; + border: 1px solid var(--border); +} +.badge.mode-err { color: var(--err); border-color: #522; background: #1a0a0a; } +.badge.mode-ok { color: var(--ok); border-color: #143; background: #0a1a10; } +.badge.mode-redirect { color: var(--warn); border-color: #664; background: #1a1608; } +/* collapsible block inside sticky-bar */ +.scope-toggle { + margin-left: auto; + display: inline-flex; + align-items: center; + gap: 8px; + max-width: min(42rem, 100%); + cursor: pointer; + font: inherit; + font-size: 12px; + color: #ddd; + background: #0006; + border: 1px solid #444; + border-radius: 4px; + padding: 4px 10px; + text-align: left; +} +.scope-toggle:hover { border-color: #888; background: #111a; color: #fff; } +.scope-toggle:focus-visible { outline: 2px solid var(--info); outline-offset: 2px; } +.scope-toggle[aria-expanded="true"] { + border-color: #777; + background: #141414; +} +.scope-chevron { + display: inline-block; + width: 0; height: 0; + border-left: 5px solid #aaa; + border-top: 4px solid transparent; + border-bottom: 4px solid transparent; + transition: transform 0.12s ease; + flex-shrink: 0; +} +.scope-toggle[aria-expanded="true"] .scope-chevron { + transform: rotate(90deg); + border-left-color: #eee; +} +.scope-toggle-label { + font-weight: 700; + letter-spacing: 0.03em; + text-transform: uppercase; + font-size: 11px; + white-space: nowrap; +} +.scope-toggle-hint { + color: #888; + overflow: hidden; + text-overflow: ellipsis; + white-space: nowrap; + min-width: 0; +} +.sticky-scope-panel { + margin-top: 8px; + max-width: 1200px; + border: 1px solid #333; + border-radius: 4px; + background: #080808e6; + padding: 8px 10px 10px; +} +.sticky-scope-panel[hidden] { display: none !important; } +.scope-body { padding: 4px 4px 2px 6px; } +.scope-title { + margin: 4px 0 4px; + font-size: 13px; + font-weight: 600; + color: #eee; +} +.scope-lead { + margin: 0 0 6px; + font-size: 12px; + color: #999; +} +.scope-list { + margin: 0; + padding-left: 1.15em; + color: #b0b0b0; + font-size: 12px; + line-height: 1.5; +} +.scope-list li { margin: 3px 0; } +#first-error, #first-warn, +[id^="err-"][id$="-line"] { + scroll-margin-top: 120px; +} +@media (max-width: 720px) { + .scope-toggle { margin-left: 0; width: 100%; } + .scope-toggle-hint { display: none; } +} +""" + +STICKY_JS = """ + +""" + + +def build_html( + *, + title: str, + hostname: str, + mode: str, + log_text: str, + commit: str, + commit_url: str, + suite_path: str, + generated_at: str, + generated_iso: str, + link_other: str | None, + page_label: str = "monitoring", +) -> str: + raw_lines = [strip_ansi(l).rstrip("\n") for l in log_text.splitlines()] + while raw_lines and not raw_lines[-1].strip(): + raw_lines.pop() + + n_err, n_warn, first_err_i, first_warn_i = count_status(raw_lines) + errors = extract_errors(raw_lines) + + err_slugs: dict[str, str] = {} + for i, e in enumerate(errors, 1): + slug = slug_error(i, e) + err_slugs[e] = slug + m = TID_RE.search(e) + if m: + err_slugs[m.group("tid")] = slug + + tid_to_slug = { + k: v for k, v in err_slugs.items() if re.match(r"^[a-z0-9_.-]+-\d+$", k, re.I) + } + + # Build body with anchors for first error/warn + per-error slugs + body_lines: list[str] = [] + claimed_err: set[str] = set() + for idx, ln in enumerate(raw_lines): + extra: list[str] = [] + if first_err_i is not None and idx == first_err_i: + extra.append("first-error") + if first_warn_i is not None and idx == first_warn_i: + extra.append("first-warn") + # per-error jump targets (all modes) + if classify(ln) in ("error", "blocker") and not is_summary_error_line(ln): + for i, e in enumerate(errors, 1): + slug = slug_error(i, e) + if slug in claimed_err: + continue + tid_m = TID_RE.search(e) + if is_run_timeout_text(e): + if "run.timeout-01" in ln and "ERROR" in ln.upper(): + extra.append(f"{slug}-line") + claimed_err.add(slug) + break + continue + key = tid_m.group(1) if tid_m else e[:40] + if key and key in ln: + extra.append(f"{slug}-line") + claimed_err.add(slug) + break + body_lines.append(render_line(ln, tid_to_slug, extra or None)) + + err_nav = "" + if errors and mode == "err": + items = [] + timeout_items = [] + for i, e in enumerate(errors, 1): + slug = slug_error(i, e) + li = ( + f'
  • ' + f'{html.escape(e)}
    • ' + ) + if is_run_timeout_text(e): + timeout_items.append(li) + else: + items.append(li) + if timeout_items: + err_nav += ( + '
    \n' + "

    EXTRAORDINARY · RUN TIMEOUT

    \n" + "

    Whole-run wall clock exceeded. " + 'Jump to detail →

    \n' + '
      \n' + + "\n".join(timeout_items[:1]) + + "\n
    \n
    \n" + ) + rest = items if timeout_items else (timeout_items + items) + if rest or not timeout_items: + err_nav += ( + '
    \n' + f"

    ERRORS ({len(errors)}) — jump to detail

    \n" + '
      \n' + + "\n".join(rest if rest else items) + + "\n
    \n
    \n" + ) + + commit_short = commit[:12] if commit else "unknown" + commit_html = ( + f'{html.escape(commit_short)}' + if commit_url + else html.escape(commit_short) + ) + + first_err_href = "#first-error" if first_err_i is not None else None + first_warn_href = "#first-warn" if first_warn_i is not None else None + scope = monitoring_scope(page_label, hostname, log_text) + + bar = sticky_bar_html( + generated_at=generated_at, + generated_iso=generated_iso, + n_err=n_err, + n_warn=n_warn, + hostname=hostname, + page_label=page_label, + mode=mode, + commit_html=commit_html, + suite_path=suite_path, + link_other=link_other, + first_err_href=first_err_href, + first_warn_href=first_warn_href, + scope=scope, + ) + + return f""" + + + + + +{html.escape(title)} + + + +{bar} +
    +{err_nav} +
    +{"".join(body_lines)} +
    +
    + Console-style render of taler-monitoring output. + Sticky bar: green = clean · yellow = warnings · red = errors. + Commit pins the exact tree used for this run. +
    +
    +{STICKY_JS} + + +""" + + +def build_redirect_html( + *, + hostname: str, + commit: str, + commit_url: str, + path_err: str = "/monitoring_err/", + page_label: str = "monitoring", + generated_at: str = "", + generated_iso: str = "", + n_err: int = 0, + n_warn: int = 0, + log_text: str = "", +) -> str: + commit_short = commit[:12] if commit else "unknown" + link = path_err if path_err.endswith("/") else path_err + "/" + c = ( + f'{html.escape(commit_short)}' + if commit_url + else html.escape(commit_short) + ) + if not generated_at: + now = datetime.now(timezone.utc) + generated_iso = now.strftime("%Y-%m-%dT%H:%M:%SZ") + generated_at = now.strftime("%Y-%m-%d %H:%M:%SZ") + # Redirect pages exist only when the run failed → treat as red if n_err unknown + if n_err <= 0: + n_err = max(n_err, 1) + scope = monitoring_scope(page_label, hostname, log_text) + bar = sticky_bar_html( + generated_at=generated_at, + generated_iso=generated_iso, + n_err=n_err, + n_warn=n_warn, + hostname=hostname, + page_label=page_label, + mode="redirect", + commit_html=c, + suite_path="", + link_other=link, + first_err_href=link, + first_warn_href=None, + scope=scope, + ) + return f""" + + + + + + +{html.escape(page_label)} · errors · {html.escape(hostname)} + + + +{bar} +
    +

    {html.escape(page_label)} has failures.

    +

    See {html.escape(link)} for the full console log, + error index, and sticky status bar.

    +

    generated {html.escape(generated_at)} · source {c}

    +
    +{STICKY_JS} + + +""" + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--log", required=True, type=Path) + ap.add_argument("--out", required=True, type=Path) + ap.add_argument("--hostname", required=True) + ap.add_argument("--mode", choices=("ok", "err", "redirect"), required=True) + ap.add_argument("--commit", default="") + ap.add_argument("--commit-url", default="") + ap.add_argument("--suite-path", default="scripts/taler-monitoring") + ap.add_argument("--title", default="") + ap.add_argument("--link-other", default="") + ap.add_argument( + "--path-err", + default="/monitoring_err/", + help="URL path for error page (redirect target)", + ) + ap.add_argument( + "--page-label", + default="monitoring", + help="Short name shown in titles/redirect box", + ) + args = ap.parse_args() + + log_text = args.log.read_text(errors="replace") if args.log.is_file() else "" + now = datetime.now(timezone.utc) + generated_iso = now.strftime("%Y-%m-%dT%H:%M:%SZ") + generated_at = now.strftime("%Y-%m-%d %H:%M:%SZ") + title = args.title or f"{args.page_label} {args.mode} · {args.hostname}" + + raw_for_counts = [strip_ansi(l).rstrip("\n") for l in log_text.splitlines()] + n_err, n_warn, _, _ = count_status(raw_for_counts) + + if args.mode == "redirect": + html_out = build_redirect_html( + hostname=args.hostname, + commit=args.commit, + commit_url=args.commit_url, + path_err=args.path_err, + page_label=args.page_label, + generated_at=generated_at, + generated_iso=generated_iso, + n_err=n_err, + n_warn=n_warn, + log_text=log_text, + ) + else: + html_out = build_html( + title=title, + hostname=args.hostname, + mode=args.mode, + log_text=log_text, + commit=args.commit, + commit_url=args.commit_url, + suite_path=args.suite_path, + generated_at=generated_at, + generated_iso=generated_iso, + link_other=args.link_other or None, + page_label=args.page_label, + ) + + args.out.parent.mkdir(parents=True, exist_ok=True) + args.out.write_text(html_out, encoding="utf-8") + print(f"wrote {args.out} (errors={n_err} warnings={n_warn} level={status_level(n_err, n_warn)})") + + +if __name__ == "__main__": + main() diff --git a/scripts/taler-monitoring/site-gen/deploy-monitoring-sites.sh b/scripts/taler-monitoring/site-gen/deploy-monitoring-sites.sh new file mode 100755 index 0000000..c5718e1 --- /dev/null +++ b/scripts/taler-monitoring/site-gen/deploy-monitoring-sites.sh @@ -0,0 +1,44 @@ +#!/usr/bin/env bash +# deploy-monitoring-sites.sh — rsync generated HTML to DEPLOY_WWW_ROOT on koopa. +# Does NOT reload Caddy (needs root — see ROOT-ON-KOOPA.md). +set -uo pipefail + +ROOT=$(cd "$(dirname "$0")" && pwd) +SETTINGS="${SITE_GEN_SETTINGS:-$ROOT/settings.conf}" +[ -f "$SETTINGS" ] || SETTINGS="$ROOT/settings.conf.example" +# shellcheck disable=SC1090 +set -a +source <(grep -v '^\s*#' "$SETTINGS" | grep -v '^\s*$' | sed 's/\r$//') +set +a + +WORK_ROOT="${WORK_ROOT:-/tmp/taler-monitoring-sites-work}" +HTML_DIR="$WORK_ROOT/html" +DEPLOY_SSH="${DEPLOY_SSH:-koopa-external}" +DEPLOY_WWW_ROOT="${DEPLOY_WWW_ROOT:-/var/www/monitoring-sites}" + +if [ ! -d "$HTML_DIR" ]; then + echo "no HTML_DIR $HTML_DIR — run generate-monitoring-sites.sh first" >&2 + exit 1 +fi + +echo "deploy $HTML_DIR → ${DEPLOY_SSH}:${DEPLOY_WWW_ROOT}/" +# Prefer writing to a hernani-owned staging dir if /var/www not writable +STAGE_REMOTE="${DEPLOY_STAGING:-/home/hernani/monitoring-sites-staging}" + +ssh -o BatchMode=yes -o ConnectTimeout=20 "$DEPLOY_SSH" \ + "mkdir -p '$STAGE_REMOTE' '$DEPLOY_WWW_ROOT' 2>/dev/null || mkdir -p '$STAGE_REMOTE'" + +rsync -az --delete "$HTML_DIR/" "${DEPLOY_SSH}:${STAGE_REMOTE}/" +echo "staged on $DEPLOY_SSH:$STAGE_REMOTE" +echo +echo "If $DEPLOY_WWW_ROOT is root-owned, as root on koopa:" +echo " mkdir -p $DEPLOY_WWW_ROOT" +echo " rsync -a --delete $STAGE_REMOTE/ $DEPLOY_WWW_ROOT/" +echo " chown -R caddy:caddy $DEPLOY_WWW_ROOT # or root:caddy — match other www" +echo " # then Caddyfile handles + reload — see ROOT-ON-KOOPA.md" +echo +# try direct rsync to DEPLOY_WWW_ROOT if writable +if ssh -o BatchMode=yes "$DEPLOY_SSH" "test -w '$DEPLOY_WWW_ROOT' 2>/dev/null || test -w \"\$(dirname '$DEPLOY_WWW_ROOT')\""; then + rsync -az --delete "$HTML_DIR/" "${DEPLOY_SSH}:${DEPLOY_WWW_ROOT}/" && \ + echo "also synced directly to $DEPLOY_WWW_ROOT" +fi diff --git a/scripts/taler-monitoring/site-gen/generate-monitoring-sites.sh b/scripts/taler-monitoring/site-gen/generate-monitoring-sites.sh new file mode 100755 index 0000000..f45e5bd --- /dev/null +++ b/scripts/taler-monitoring/site-gen/generate-monitoring-sites.sh @@ -0,0 +1,273 @@ +#!/usr/bin/env bash +# generate-monitoring-sites.sh — run taler-monitoring, build console HTML for +# /monitoring and /monitoring_err (static files for host Caddy later). +# +# Usage: +# ./generate-monitoring-sites.sh # all SITES (or ONLY_HOSTS) +# ONLY_HOSTS="bank.hacktivism.ch taler.hacktivism.ch" ./generate-monitoring-sites.sh +# ./generate-monitoring-sites.sh --dry-run # HTML from empty/missing logs only +# ./generate-monitoring-sites.sh --skip-run # only convert existing logs +# +# Settings: settings.conf (from settings.conf.example). No secrets in settings. +# Secrets for e2e: ../secrets.env (optional; default phases are urls versions). +# +set -uo pipefail + +export PYTHONUNBUFFERED=1 +# Global reporting defaults (same as host-agent run-host-report.sh) +: "${RUN_TIMEOUT:=600}" +export RUN_TIMEOUT + +ROOT=$(cd "$(dirname "$0")" && pwd) +SETTINGS="${SITE_GEN_SETTINGS:-$ROOT/settings.conf}" +EXAMPLE="$ROOT/settings.conf.example" + +if [ ! -f "$SETTINGS" ]; then + if [ -f "$EXAMPLE" ]; then + echo "note: no $SETTINGS — using example defaults (copy to settings.conf to customize)" + SETTINGS="$EXAMPLE" + else + echo "missing settings: $EXAMPLE" >&2 + exit 2 + fi +fi + +# Load KEY=value lines; allow SITES already set in environment to win. +# Multiline SITES in the conf file: use SITES_FILE= or env SITES= instead. +while IFS= read -r _line || [ -n "$_line" ]; do + _line=${_line%%#*} + _line=$(echo "$_line" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//') + [ -z "$_line" ] && continue + case "$_line" in + SITES=*|SITES_FILE=*) continue ;; # handled below / env + *=*) + _k=${_line%%=*} + _v=${_line#*=} + # do not override pre-set env + if [ -z "${!_k+x}" ] 2>/dev/null || [ -z "${!_k}" ]; then + export "$_k=$_v" + fi + ;; + esac +done < "$SETTINGS" + +# SITES from dedicated file or keep env; else parse heredoc-style from example list +if [ -n "${SITES_FILE:-}" ] && [ -f "$SITES_FILE" ]; then + SITES=$(grep -v '^\s*#' "$SITES_FILE" | grep -v '^\s*$') + export SITES +elif [ -z "${SITES:-}" ]; then + # default 9 fronts + SITES="bank.hacktivism.ch|hacktivism.ch +exchange.hacktivism.ch|hacktivism.ch +taler.hacktivism.ch|hacktivism.ch +stage.bank.lefrancpaysan.ch|stage.lefrancpaysan.ch +stage.exchange.lefrancpaysan.ch|stage.lefrancpaysan.ch +stage.monnaie.lefrancpaysan.ch|stage.lefrancpaysan.ch +bank.lefrancpaysan.ch|lefrancpaysan.ch +exchange.lefrancpaysan.ch|lefrancpaysan.ch +monnaie.lefrancpaysan.ch|lefrancpaysan.ch" + export SITES +fi + +SKIP_RUN=0 +DRY=0 +while [ $# -gt 0 ]; do + case "$1" in + --skip-run) SKIP_RUN=1; shift ;; + --dry-run) DRY=1; SKIP_RUN=1; shift ;; + -h|--help) sed -n '1,20p' "$0"; exit 0 ;; + *) echo "unknown arg: $1" >&2; exit 2 ;; + esac +done + +MONITORING_ROOT=$(cd "$ROOT/${MONITORING_ROOT:-..}" && pwd) +WORK_ROOT="${WORK_ROOT:-/tmp/taler-monitoring-sites-work}" +LOG_DIR="$WORK_ROOT/logs" +HTML_DIR="$WORK_ROOT/html" +mkdir -p "$LOG_DIR" "$HTML_DIR" + +# commit of monitoring suite used for this run +if git -C "$MONITORING_ROOT/../.." rev-parse HEAD >/dev/null 2>&1; then + REPO_ROOT=$(cd "$MONITORING_ROOT/../.." && pwd) +elif git -C "$MONITORING_ROOT" rev-parse HEAD >/dev/null 2>&1; then + REPO_ROOT=$(cd "$MONITORING_ROOT" && pwd) +else + REPO_ROOT=$(cd "$ROOT/../../.." && pwd) +fi +COMMIT=$(git -C "$REPO_ROOT" rev-parse HEAD 2>/dev/null || echo unknown) +COMMIT_SHORT=$(git -C "$REPO_ROOT" rev-parse --short=12 HEAD 2>/dev/null || echo unknown) +SOURCE_REPO_WEB="${SOURCE_REPO_WEB:-https://git.hacktivism.ch/hernani/koopa-admin-log}" +TMPL="${SOURCE_COMMIT_URL_TMPL:-\{repo\}/src/commit/\{commit\}}" +COMMIT_URL=$(printf '%s' "$TMPL" | sed "s|{repo}|$SOURCE_REPO_WEB|g; s|{commit}|$COMMIT|g") +SUITE_PATH="${SOURCE_SUITE_PATH:-scripts/taler-monitoring}" + +# Outside-only public checks (no SSH into koopa/stage containers) +PHASES_ERR="${PHASES_ERR:-urls versions}" +PHASES_OK="${PHASES_OK:-urls versions}" +CONTINUE_ON_ERROR="${CONTINUE_ON_ERROR:-1}" +SKIP_SSH="${SKIP_SSH:-1}" +# Drop SSH phases if someone passes "all" / inside / server / e2e by mistake +filter_phases_outside() { + local out=() p + for p in "$@"; do + case "$p" in + inside|server|e2e|ladder|goa-ladder|auth401) continue ;; + all) out+=(urls versions);; + *) out+=("$p") ;; + esac + done + printf '%s\n' "${out[@]}" +} + +run_mon() { + local domain="$1" phases="$2" logfile="$3" cont="$4" + # shellcheck disable=SC2206 + local ph=( $phases ) + mapfile -t ph < <(filter_phases_outside "${ph[@]}") + [ "${#ph[@]}" -gt 0 ] || ph=(urls versions) + + : >"$logfile" + echo "+ -d $domain · phases=${ph[*]} · CONTINUE=$cont · SKIP_SSH=$SKIP_SSH · RUN_TIMEOUT=$RUN_TIMEOUT" + + if [ -n "${RUNNER_SSH:-}" ]; then + echo "+ via ssh $RUNNER_SSH" + rsync -az --delete \ + --exclude secrets.env \ + --exclude 'android-test/artifacts' \ + --exclude '.git' \ + "$MONITORING_ROOT/" \ + "${RUNNER_SSH}:${RUNNER_REMOTE_WORKDIR}/" \ + || return 1 + # shellcheck disable=SC2029 + if command -v stdbuf >/dev/null 2>&1; then + ssh -o BatchMode=yes -o ConnectTimeout=30 "$RUNNER_SSH" \ + "cd '${RUNNER_REMOTE_WORKDIR}' && chmod +x taler-monitoring.sh check_*.sh 2>/dev/null; \ + export CONTINUE_ON_ERROR='$cont' AUTH401_CONTINUE='$cont' SKIP_SSH='$SKIP_SSH' RUN_TIMEOUT='$RUN_TIMEOUT'; \ + stdbuf -oL -eL ./taler-monitoring.sh -d '$domain' ${ph[*]}" \ + 2>&1 | stdbuf -oL -eL tee -a "$logfile" + else + ssh -o BatchMode=yes -o ConnectTimeout=30 "$RUNNER_SSH" \ + "cd '${RUNNER_REMOTE_WORKDIR}' && chmod +x taler-monitoring.sh check_*.sh 2>/dev/null; \ + export CONTINUE_ON_ERROR='$cont' AUTH401_CONTINUE='$cont' SKIP_SSH='$SKIP_SSH' RUN_TIMEOUT='$RUN_TIMEOUT'; \ + ./taler-monitoring.sh -d '$domain' ${ph[*]}" \ + 2>&1 | tee -a "$logfile" + fi + return "${PIPESTATUS[0]}" + fi + + if command -v stdbuf >/dev/null 2>&1; then + ( cd "$MONITORING_ROOT" && \ + CONTINUE_ON_ERROR="$cont" AUTH401_CONTINUE="$cont" SKIP_SSH="$SKIP_SSH" \ + RUN_TIMEOUT="$RUN_TIMEOUT" \ + stdbuf -oL -eL ./taler-monitoring.sh -d "$domain" "${ph[@]}" ) \ + 2>&1 | stdbuf -oL -eL tee -a "$logfile" + else + ( cd "$MONITORING_ROOT" && \ + CONTINUE_ON_ERROR="$cont" AUTH401_CONTINUE="$cont" SKIP_SSH="$SKIP_SSH" \ + RUN_TIMEOUT="$RUN_TIMEOUT" \ + ./taler-monitoring.sh -d "$domain" "${ph[@]}" ) \ + 2>&1 | tee -a "$logfile" + fi + return "${PIPESTATUS[0]}" +} + +htmlify() { + local host="$1" mode="$2" log="$3" out="$4" other="$5" + mkdir -p "$(dirname "$out")" + if [ ! -f "$ROOT/console_to_html.py" ]; then + echo "WARN: console_to_html.py missing — bootstrap $out" + { + echo "$host $mode" + echo "

    $host · $mode

    commit $COMMIT_SHORT

    " + echo "
    "
+      tail -n 100 "$log" 2>/dev/null | sed 's/&/\&/g;s/"
+    } >"$out"
+    return 0
+  fi
+  python3 "$ROOT/console_to_html.py" \
+    --log "$log" \
+    --out "$out" \
+    --hostname "$host" \
+    --mode "$mode" \
+    --commit "$COMMIT" \
+    --commit-url "$COMMIT_URL" \
+    --suite-path "$SUITE_PATH" \
+    --link-other "$other"
+}
+
+# Parse SITES
+mapfile -t SITE_LINES < <(printf '%s\n' "$SITES" | sed '/^\s*$/d' | sed 's/#.*//')
+ONLY="${ONLY_HOSTS:-}"
+
+for line in "${SITE_LINES[@]}"; do
+  line=$(echo "$line" | tr -d ' \t')
+  [ -z "$line" ] && continue
+  host=${line%%|*}
+  domain=${line#*|}
+  if [ -n "$ONLY" ]; then
+    echo " $ONLY " | grep -q " $host " || continue
+  fi
+
+  echo "======== $host  (domain=$domain)  commit=$COMMIT_SHORT ========"
+  base_html="$HTML_DIR/$host"
+  mkdir -p "$base_html/monitoring" "$base_html/monitoring_err"
+  log_err="$LOG_DIR/${host}.err.log"
+  log_ok="$LOG_DIR/${host}.ok.log"
+  ec_err=1
+  ec_ok=1
+
+  if [ "$SKIP_RUN" != "1" ]; then
+    run_mon "$domain" "$PHASES_ERR" "$log_err" "$CONTINUE_ON_ERROR"
+    ec_err=$?
+    # ok-run without continue (strict)
+    run_mon "$domain" "$PHASES_OK" "$log_ok" "0"
+    ec_ok=$?
+  else
+    [ -f "$log_err" ] || : >"$log_err"
+    [ -f "$log_ok" ] || : >"$log_ok"
+    # infer exit from log if present
+    if grep -q '\[ ERROR \]' "$log_err" 2>/dev/null || grep -q 'ERROR' "$log_err" 2>/dev/null; then
+      ec_err=1
+    else
+      ec_err=0
+    fi
+    ec_ok=$ec_err
+  fi
+
+  # Always write pages (first run / empty logs too)
+  [ -f "$log_err" ] || : >"$log_err"
+  [ -f "$log_ok" ] || : >"$log_ok"
+
+  htmlify "$host" "err" "$log_err" \
+    "$base_html/monitoring_err/index.html" "/monitoring/"
+
+  if [ "$ec_ok" -eq 0 ] && [ "$ec_err" -eq 0 ]; then
+    htmlify "$host" "ok" "$log_ok" \
+      "$base_html/monitoring/index.html" "/monitoring_err/"
+    echo "  → /monitoring (clean) + /monitoring_err"
+  else
+    htmlify "$host" "redirect" "$log_err" \
+      "$base_html/monitoring/index.html" "/monitoring_err/"
+    echo "  → /monitoring (redirect stub) + /monitoring_err (errors)  ec_err=$ec_err ec_ok=$ec_ok"
+  fi
+
+  # first-run safety
+  if [ ! -f "$base_html/monitoring/index.html" ]; then
+    htmlify "$host" "err" "$log_err" \
+      "$base_html/monitoring/index.html" "/monitoring_err/"
+  fi
+
+  printf '%s\n' "$COMMIT" >"$base_html/COMMIT"
+  printf '%s\n' "$COMMIT_URL" >"$base_html/COMMIT_URL"
+  printf 'host=%s domain=%s ec_err=%s ec_ok=%s commit=%s run_timeout=%s\n' \
+    "$host" "$domain" "$ec_err" "$ec_ok" "$COMMIT" "$RUN_TIMEOUT" | tee "$base_html/STATUS.txt"
+done
+
+echo
+echo "HTML under $HTML_DIR"
+echo "commit $COMMIT_SHORT  $COMMIT_URL"
+echo "RUN_TIMEOUT=${RUN_TIMEOUT}s (global; 0=unlimited)"
+echo "Deploy (as hernani, may need sudo for /var/www):"
+echo "  ./deploy-monitoring-sites.sh"
+echo "Root on koopa: see ROOT-ON-KOOPA.md"
diff --git a/scripts/taler-monitoring/site-gen/install-firecuda-timer.sh b/scripts/taler-monitoring/site-gen/install-firecuda-timer.sh
new file mode 100755
index 0000000..9615b96
--- /dev/null
+++ b/scripts/taler-monitoring/site-gen/install-firecuda-timer.sh
@@ -0,0 +1,79 @@
+#!/usr/bin/env bash
+# install-firecuda-timer.sh — OPTIONAL: rsync suite to firecuda + launchd every 4h
+#
+# POLICY (2026-07): firecuda launchd is **disabled**. Prefer koopa host-agent.
+# This script remains for documentation / future re-enable only.
+# Do not run unless you intentionally want a second outside-only scheduler.
+#
+#   ./install-firecuda-timer.sh
+#   ./install-firecuda-timer.sh --run-now
+#
+set -euo pipefail
+
+ROOT=$(cd "$(dirname "$0")" && pwd)
+MON_ROOT=$(cd "$ROOT/.." && pwd)
+FIRECUDA="${FIRECUDA_SSH:-firecuda-external}"
+REMOTE_BASE="${FIRECUDA_INSTALL_DIR:-taler-monitoring-site-gen}"
+RUN_NOW=0
+[ "${1:-}" = "--run-now" ] && RUN_NOW=1
+
+echo "install → $FIRECUDA:~/$REMOTE_BASE (outside-only, every 4h)"
+
+ssh -o BatchMode=yes -o ConnectTimeout=20 "$FIRECUDA" \
+  "mkdir -p \"\$HOME/$REMOTE_BASE\" \"\$HOME/Library/Logs/taler-monitoring-sites\" \"\$HOME/var/taler-monitoring-sites-work\" \"\$HOME/Library/LaunchAgents\""
+
+# Sync full taler-monitoring suite (no secrets.env)
+rsync -az --delete \
+  --exclude secrets.env \
+  --exclude 'android-test/artifacts' \
+  --exclude '.git' \
+  --exclude 'site-gen/settings.conf' \
+  "$MON_ROOT/" \
+  "${FIRECUDA}:$REMOTE_BASE/"
+
+# settings on firecuda: outside only, run locally on firecuda, deploy to koopa
+ssh -o BatchMode=yes "$FIRECUDA" "cat > \"\$HOME/$REMOTE_BASE/site-gen/settings.conf\" <<'EOF'
+# auto-installed by install-firecuda-timer.sh — no secrets
+RUNNER_SSH=
+SKIP_SSH=1
+CONTINUE_ON_ERROR=1
+PHASES_ERR=urls versions
+PHASES_OK=urls versions
+DEPLOY_SSH=koopa-external
+DEPLOY_STAGING=/home/hernani/monitoring-sites-staging
+WORK_ROOT=
+MONITORING_ROOT=..
+SOURCE_REPO_WEB=https://git.hacktivism.ch/hernani/koopa-admin-log
+SOURCE_COMMIT_URL_TMPL={repo}/src/commit/{commit}
+SOURCE_SUITE_PATH=scripts/taler-monitoring
+EOF
+# WORK_ROOT expanded on remote
+sed -i.bak \"s|^WORK_ROOT=.*|WORK_ROOT=\$HOME/var/taler-monitoring-sites-work|\" \"\$HOME/$REMOTE_BASE/site-gen/settings.conf\" 2>/dev/null || \
+  sed -i '' \"s|^WORK_ROOT=.*|WORK_ROOT=\$HOME/var/taler-monitoring-sites-work|\" \"\$HOME/$REMOTE_BASE/site-gen/settings.conf\"
+"
+
+# launchd plist with absolute paths
+ssh -o BatchMode=yes "$FIRECUDA" bash -s < "\$PLIST_DST"
+chmod +x "\$SITE_GEN"/*.sh "\$SITE_GEN"/console_to_html.py 2>/dev/null || true
+chmod +x "\$SITE_GEN/run-on-firecuda.sh"
+launchctl bootout "gui/\$(id -u)/com.hacktivism.taler-monitoring-sites" 2>/dev/null || true
+launchctl unload "\$PLIST_DST" 2>/dev/null || true
+launchctl bootstrap "gui/\$(id -u)" "\$PLIST_DST" 2>/dev/null || launchctl load "\$PLIST_DST"
+launchctl enable "gui/\$(id -u)/com.hacktivism.taler-monitoring-sites" 2>/dev/null || true
+launchctl print "gui/\$(id -u)/com.hacktivism.taler-monitoring-sites" 2>/dev/null | head -20 || launchctl list | grep taler-monitoring || true
+echo "OK: launchd com.hacktivism.taler-monitoring-sites (StartInterval=14400)"
+REMOTE
+
+if [ "$RUN_NOW" = "1" ]; then
+  echo "run once now…"
+  ssh -o BatchMode=yes "$FIRECUDA" "\"\$HOME/$REMOTE_BASE/site-gen/run-on-firecuda.sh\"" || true
+fi
+
+echo "done. logs: $FIRECUDA:~/Library/Logs/taler-monitoring-sites/"
+echo "manual: ssh $FIRECUDA '~/taler-monitoring-site-gen/site-gen/run-on-firecuda.sh'"
diff --git a/scripts/taler-monitoring/site-gen/pull-firecuda-to-koopa.sh b/scripts/taler-monitoring/site-gen/pull-firecuda-to-koopa.sh
new file mode 100755
index 0000000..b1b10ec
--- /dev/null
+++ b/scripts/taler-monitoring/site-gen/pull-firecuda-to-koopa.sh
@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+# pull-firecuda-to-koopa.sh — copy HTML from firecuda work dir → koopa staging
+# Run on laptop (hernani) where both SSH aliases work.
+set -euo pipefail
+FIRECUDA="${FIRECUDA_SSH:-firecuda-external}"
+KOOPA="${DEPLOY_SSH:-koopa-external}"
+REMOTE_HTML="${FIRECUDA_HTML:-var/taler-monitoring-sites-work/html/}"
+STAGE="${DEPLOY_STAGING:-/home/hernani/monitoring-sites-staging}"
+
+echo "firecuda:$REMOTE_HTML → $KOOPA:$STAGE"
+ssh -o BatchMode=yes -o ConnectTimeout=20 "$KOOPA" "mkdir -p '$STAGE'"
+rsync -az --delete \
+  "${FIRECUDA}:${REMOTE_HTML}" \
+  "${KOOPA}:${STAGE}/"
+echo "OK staged. Root: rsync to /var/www/monitoring-sites (see ROOT-ON-KOOPA.md)"
diff --git a/scripts/taler-monitoring/site-gen/run-on-firecuda.sh b/scripts/taler-monitoring/site-gen/run-on-firecuda.sh
new file mode 100755
index 0000000..6f77de6
--- /dev/null
+++ b/scripts/taler-monitoring/site-gen/run-on-firecuda.sh
@@ -0,0 +1,59 @@
+#!/usr/bin/env bash
+# run-on-firecuda.sh — entrypoint for launchd/cron on firecuda-external.
+# Outside-only monitoring (SKIP_SSH=1), all 9 sites, then stage HTML to koopa.
+#
+# Installed path on firecuda (default):
+#   ~/taler-monitoring-site-gen/site-gen/run-on-firecuda.sh
+#
+set -uo pipefail
+
+export PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:${PATH:-}"
+
+ROOT=$(cd "$(dirname "$0")" && pwd)
+LOG_DIR="${FIRECUDA_LOG_DIR:-$HOME/Library/Logs/taler-monitoring-sites}"
+mkdir -p "$LOG_DIR"
+STAMP=$(date +%Y%m%d-%H%M%S)
+LOG="$LOG_DIR/run-${STAMP}.log"
+exec >>"$LOG" 2>&1
+
+echo "======== $(date -Iseconds 2>/dev/null || date) firecuda monitoring sites ========"
+
+# Always: public DNS only, no SSH checks into stacks
+export SKIP_SSH=1
+export CONTINUE_ON_ERROR=1
+export AUTH401_CONTINUE=1
+export RUNNER_SSH=
+export PHASES_ERR="${PHASES_ERR:-urls versions}"
+export PHASES_OK="${PHASES_OK:-urls versions}"
+export WORK_ROOT="${WORK_ROOT:-$HOME/var/taler-monitoring-sites-work}"
+export MONITORING_ROOT="${MONITORING_ROOT:-$ROOT/..}"
+export DEPLOY_SSH="${DEPLOY_SSH:-koopa-external}"
+export DEPLOY_STAGING="${DEPLOY_STAGING:-/home/hernani/monitoring-sites-staging}"
+
+# Prefer full 9-site list from settings or defaults inside generate script
+unset ONLY_HOSTS
+
+cd "$ROOT" || exit 1
+chmod +x generate-monitoring-sites.sh deploy-monitoring-sites.sh console_to_html.py 2>/dev/null || true
+
+./generate-monitoring-sites.sh
+ec=$?
+
+# HTML stays on firecuda under WORK_ROOT/html.
+# Optional push to koopa if this host can SSH there (often only from laptop):
+if [ -n "${DEPLOY_SSH:-}" ] && [ "${DEPLOY_FROM_FIRECUDA:-0}" = "1" ]; then
+  if [ -x ./deploy-monitoring-sites.sh ]; then
+    ./deploy-monitoring-sites.sh || echo "WARN: deploy staging failed"
+  fi
+else
+  echo "note: HTML at ${WORK_ROOT}/html — pull from laptop:"
+  echo "  rsync -az firecuda-external:var/taler-monitoring-sites-work/html/ \\"
+  echo "    koopa-external:monitoring-sites-staging/"
+  echo "  # or: ./pull-firecuda-to-koopa.sh"
+fi
+
+# keep last 20 logs
+ls -1t "$LOG_DIR"/run-*.log 2>/dev/null | tail -n +21 | xargs rm -f 2>/dev/null || true
+
+echo "======== done ec=$ec ========"
+exit "$ec"
diff --git a/scripts/taler-monitoring/site-gen/settings.conf.example b/scripts/taler-monitoring/site-gen/settings.conf.example
new file mode 100644
index 0000000..3d14fc2
--- /dev/null
+++ b/scripts/taler-monitoring/site-gen/settings.conf.example
@@ -0,0 +1,55 @@
+# monitoring site-gen — non-secret settings (safe as example)
+# Copy:  cp settings.conf.example settings.conf
+# Never put passwords/tokens here.
+
+# On firecuda the suite runs *locally* (RUNNER_SSH empty). From a laptop you
+# can set RUNNER_SSH=firecuda-external to drive a one-off remote run instead.
+RUNNER_SSH=
+RUNNER_SSH_FALLBACKS=
+RUNNER_REMOTE_WORKDIR=/tmp/taler-monitoring-site-gen
+MONITORING_ROOT=..
+
+# Outside-only: never SSH into bank/exchange/merchant containers
+SKIP_SSH=1
+
+# Deploy staging on koopa (host Caddy later — root copies to /var/www)
+DEPLOY_SSH=koopa-external
+DEPLOY_SSH_FALLBACKS=koopa
+# Static files for Caddy file_server (created per hostname)
+DEPLOY_WWW_ROOT=/var/www/monitoring-sites
+DEPLOY_SUDO=sudo
+CADDY_CONFIG=/etc/caddy/Caddyfile
+CADDY_MIRROR_REL=configs/caddy/Caddyfile
+# Working tree on deploy host for apply (optional)
+KOOPA_CADDY_DIR=/home/hernani/koopa-caddy
+
+# Source link for HTML footer (Forgejo)
+SOURCE_REPO_WEB=https://git.hacktivism.ch/hernani/koopa-admin-log
+SOURCE_COMMIT_URL_TMPL={repo}/src/commit/{commit}
+SOURCE_SUITE_PATH=scripts/taler-monitoring
+
+CONTINUE_ON_ERROR=1
+PHASES_ERR=urls versions
+PHASES_OK=urls versions
+SKIP_E2E=1
+
+# Local work dir for logs + HTML
+WORK_ROOT=/tmp/taler-monitoring-sites-work
+
+# Sites: hostname|domain_profile  (9 fronts)
+# Start with hacktivism; stage + prod LFP follow
+SITES="
+bank.hacktivism.ch|hacktivism.ch
+exchange.hacktivism.ch|hacktivism.ch
+taler.hacktivism.ch|hacktivism.ch
+stage.bank.lefrancpaysan.ch|stage.lefrancpaysan.ch
+stage.exchange.lefrancpaysan.ch|stage.lefrancpaysan.ch
+stage.monnaie.lefrancpaysan.ch|stage.lefrancpaysan.ch
+bank.lefrancpaysan.ch|lefrancpaysan.ch
+exchange.lefrancpaysan.ch|lefrancpaysan.ch
+monnaie.lefrancpaysan.ch|lefrancpaysan.ch
+"
+
+# Only process this subset (space-separated hostnames). Empty = all SITES.
+# Example for first deploy: ONLY_HOSTS=bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch
+ONLY_HOSTS=