taler-monitoring v1.0: standalone suite
This commit is contained in:
Hernâni Marques
commit
a2afe690b7
73 changed files with 18268 additions and 0 deletions
81
site-gen/README.md
Normal file
81
site-gen/README.md
Normal file
|
|
@ -0,0 +1,81 @@
|
|||
# monitoring site-gen
|
||||
|
||||
Build **console-style HTML** from `taler-monitoring` runs and stage them for
|
||||
**host Caddy** (`/monitoring`, `/monitoring_err`).
|
||||
|
||||
Same **global reporting defaults** as host-agents (`run-host-report.sh`):
|
||||
|
||||
- **`RUN_TIMEOUT=600`** on each suite run
|
||||
- **line-buffered** logs (continuous write)
|
||||
- **always** write HTML (first run / empty logs too)
|
||||
- Forgejo **commit link** in the page header
|
||||
|
||||
Applies to all nine fronts (GOA + FP stage + FP prod) in the default `SITES` list.
|
||||
|
||||
| Host | Role |
|
||||
|------|------|
|
||||
| **firecuda-external** | optional runner (public DNS) |
|
||||
| **koopa-external** | deploy staging + Caddy (root service) |
|
||||
| **host-agents** | primary: GOA + FP on-box (`../host-agent/`) |
|
||||
|
||||
No secrets in `settings.conf` — only hosts, paths, site list.
|
||||
|
||||
## firecuda-external (outside-only) — **disabled for now**
|
||||
|
||||
**firecuda-external could** run public-only monitoring (`SKIP_SSH=1`, phases
|
||||
`urls versions`) on a schedule (launchd every 4h) and write HTML under
|
||||
`~/var/taler-monitoring-sites-work/html/`. Scripts for that still live here
|
||||
(`install-firecuda-timer.sh`, `run-on-firecuda.sh`, plist template).
|
||||
|
||||
**Current policy: do not run a real launchd job on firecuda.**
|
||||
Primary GOA monitoring is **koopa host-agent** (`host-agent/`, user `hernani`,
|
||||
local `podman exec`). Re-enable firecuda only if you explicitly want a second,
|
||||
outside-only view.
|
||||
|
||||
```bash
|
||||
# optional one-shot (no timer):
|
||||
# ssh firecuda-external '~/taler-monitoring-site-gen/site-gen/run-on-firecuda.sh'
|
||||
|
||||
# if a timer was installed earlier, ensure it is off:
|
||||
# ssh firecuda-external 'launchctl list | grep taler-monitoring || echo off'
|
||||
```
|
||||
|
||||
Disabled plists may remain as `*.plist.disabled-*` under `~/Library/LaunchAgents/`.
|
||||
|
||||
## Quick start (laptop)
|
||||
|
||||
```bash
|
||||
cd site-gen
|
||||
cp settings.conf.example settings.conf # optional
|
||||
chmod +x generate-monitoring-sites.sh deploy-monitoring-sites.sh console_to_html.py
|
||||
|
||||
ONLY_HOSTS="bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch" \
|
||||
SKIP_SSH=1 RUNNER_SSH= \
|
||||
./generate-monitoring-sites.sh
|
||||
|
||||
./deploy-monitoring-sites.sh
|
||||
```
|
||||
|
||||
Then on koopa **as root**: see **[ROOT-ON-KOOPA.md](./ROOT-ON-KOOPA.md)**.
|
||||
|
||||
## URL rules
|
||||
|
||||
| Path | When |
|
||||
|------|------|
|
||||
| `/monitoring_err/` | always: full console log + error list on top (CONTINUE_ON_ERROR run) |
|
||||
| `/monitoring/` | **clean** full log if last strict run exit 0; else **stub** linking to `/monitoring_err/` |
|
||||
|
||||
Footer links **Forgejo commit** of the admin-log tree used for that run.
|
||||
|
||||
## Layout
|
||||
|
||||
```text
|
||||
site-gen/
|
||||
settings.conf.example
|
||||
generate-monitoring-sites.sh
|
||||
deploy-monitoring-sites.sh
|
||||
console_to_html.py
|
||||
caddy-monitoring-handles.snippet
|
||||
ROOT-ON-KOOPA.md
|
||||
README.md
|
||||
```
|
||||
92
site-gen/ROOT-ON-KOOPA.md
Normal file
92
site-gen/ROOT-ON-KOOPA.md
Normal file
|
|
@ -0,0 +1,92 @@
|
|||
# Root on koopa (hacktivism monitoring sites)
|
||||
|
||||
Host Caddy runs as **root/systemd** (`caddy` user). Static monitoring HTML is
|
||||
**not** put into Taler containers — only host paths + Caddy `handle`.
|
||||
|
||||
After `generate-monitoring-sites.sh` + `deploy-monitoring-sites.sh` (as hernani),
|
||||
files live in:
|
||||
|
||||
```text
|
||||
/home/hernani/monitoring-sites-staging/<hostname>/monitoring/index.html
|
||||
/home/hernani/monitoring-sites-staging/<hostname>/monitoring_err/index.html
|
||||
```
|
||||
|
||||
## One-shot as root (copy + Caddy)
|
||||
|
||||
```bash
|
||||
# on koopa (koopa-external), as root:
|
||||
|
||||
# 1) web root
|
||||
install -d -o caddy -g caddy -m 755 /var/www/monitoring-sites
|
||||
rsync -a --delete /home/hernani/monitoring-sites-staging/ /var/www/monitoring-sites/
|
||||
chown -R caddy:caddy /var/www/monitoring-sites
|
||||
|
||||
# 2) Caddyfile — merge handles from admin-log mirror, then:
|
||||
# (either edit /etc/caddy/Caddyfile by hand using snippet below,
|
||||
# or copy full mirror after review)
|
||||
#
|
||||
# install -m 644 /home/hernani/koopa-admin-log/configs/caddy/Caddyfile /etc/caddy/Caddyfile
|
||||
# # or: ~/koopa-caddy/apply.sh after syncing Caddyfile into ~/koopa-caddy/
|
||||
|
||||
caddy validate --config /etc/caddy/Caddyfile
|
||||
systemctl reload caddy
|
||||
systemctl is-active caddy
|
||||
```
|
||||
|
||||
## Snippet per hacktivism site (inside each `*.hacktivism.ch { }` block)
|
||||
|
||||
Place **before** the catch-all `reverse_proxy` (same idea as `/intro*`):
|
||||
|
||||
```caddy
|
||||
# Public taler-monitoring console HTML (static; host only)
|
||||
handle /monitoring_err* {
|
||||
root * /var/www/monitoring-sites/{host}
|
||||
rewrite * /monitoring_err{uri}
|
||||
# uri is /monitoring_err or /monitoring_err/ → serve directory index
|
||||
file_server
|
||||
}
|
||||
handle /monitoring* {
|
||||
root * /var/www/monitoring-sites/{host}
|
||||
file_server
|
||||
}
|
||||
```
|
||||
|
||||
Simpler layout (recommended): path on disk matches URL under host root:
|
||||
|
||||
```text
|
||||
/var/www/monitoring-sites/bank.hacktivism.ch/monitoring/index.html
|
||||
/var/www/monitoring-sites/bank.hacktivism.ch/monitoring_err/index.html
|
||||
```
|
||||
|
||||
```caddy
|
||||
handle_path /monitoring_err/* {
|
||||
root * /var/www/monitoring-sites/{host}/monitoring_err
|
||||
file_server
|
||||
}
|
||||
handle /monitoring_err {
|
||||
redir /monitoring_err/ 302
|
||||
}
|
||||
handle_path /monitoring/* {
|
||||
root * /var/www/monitoring-sites/{host}/monitoring
|
||||
file_server
|
||||
}
|
||||
handle /monitoring {
|
||||
redir /monitoring/ 302
|
||||
}
|
||||
```
|
||||
|
||||
Canonical snippet is also in `caddy-monitoring-handles.snippet` and applied
|
||||
in `configs/caddy/Caddyfile` for the three GOA hosts.
|
||||
|
||||
## Check
|
||||
|
||||
```bash
|
||||
curl -sS -o /dev/null -w '%{http_code}\n' https://bank.hacktivism.ch/monitoring/
|
||||
curl -sS -o /dev/null -w '%{http_code}\n' https://bank.hacktivism.ch/monitoring_err/
|
||||
curl -sS https://bank.hacktivism.ch/monitoring_err/ | head
|
||||
```
|
||||
|
||||
## Not needed as root
|
||||
|
||||
- Running `taler-monitoring` (use **firecuda-external** / laptop)
|
||||
- Writing into podman Taler containers
|
||||
57
site-gen/caddy-monitoring-handles.snippet
Normal file
57
site-gen/caddy-monitoring-handles.snippet
Normal file
|
|
@ -0,0 +1,57 @@
|
|||
# Host Caddy on koopa (root service). Paste BEFORE catch-all reverse_proxy.
|
||||
#
|
||||
# Disk layout:
|
||||
# /var/www/monitoring-sites/{host}/monitoring/index.html
|
||||
# /var/www/monitoring-sites/{host}/monitoring_err/index.html
|
||||
# /var/www/monitoring-sites/taler.hacktivism.ch/taler-monitoring-surface*/index.html
|
||||
# /var/www/monitoring-sites/taler.hacktivism.ch/taler-monitoring-aptdeploy*/index.html
|
||||
#
|
||||
# /monitoring* → bank + exchange + taler.hacktivism.ch
|
||||
# /taler-monitoring-surface* → taler.hacktivism.ch only
|
||||
# /taler-monitoring-aptdeploy* → taler.hacktivism.ch only
|
||||
|
||||
# --- only inside taler.hacktivism.ch { ... } ---
|
||||
handle /taler-monitoring-surface_err {
|
||||
redir /taler-monitoring-surface_err/ 302
|
||||
}
|
||||
handle /taler-monitoring-surface_err/ {
|
||||
root * /var/www/monitoring-sites/taler.hacktivism.ch/taler-monitoring-surface_err
|
||||
file_server
|
||||
}
|
||||
handle /taler-monitoring-surface {
|
||||
redir /taler-monitoring-surface/ 302
|
||||
}
|
||||
handle /taler-monitoring-surface/ {
|
||||
root * /var/www/monitoring-sites/taler.hacktivism.ch/taler-monitoring-surface
|
||||
file_server
|
||||
}
|
||||
handle /taler-monitoring-aptdeploy_err {
|
||||
redir /taler-monitoring-aptdeploy_err/ 302
|
||||
}
|
||||
handle /taler-monitoring-aptdeploy_err/ {
|
||||
root * /var/www/monitoring-sites/taler.hacktivism.ch/taler-monitoring-aptdeploy_err
|
||||
file_server
|
||||
}
|
||||
handle /taler-monitoring-aptdeploy {
|
||||
redir /taler-monitoring-aptdeploy/ 302
|
||||
}
|
||||
handle /taler-monitoring-aptdeploy/ {
|
||||
root * /var/www/monitoring-sites/taler.hacktivism.ch/taler-monitoring-aptdeploy
|
||||
file_server
|
||||
}
|
||||
|
||||
# --- bank + exchange + taler (each site block) ---
|
||||
handle /monitoring_err {
|
||||
redir /monitoring_err/ 302
|
||||
}
|
||||
handle /monitoring_err/ {
|
||||
root * /var/www/monitoring-sites/{host}/monitoring_err
|
||||
file_server
|
||||
}
|
||||
handle /monitoring {
|
||||
redir /monitoring/ 302
|
||||
}
|
||||
handle /monitoring/ {
|
||||
root * /var/www/monitoring-sites/{host}/monitoring
|
||||
file_server
|
||||
}
|
||||
34
site-gen/com.hacktivism.taler-monitoring-sites.plist
Normal file
34
site-gen/com.hacktivism.taler-monitoring-sites.plist
Normal file
|
|
@ -0,0 +1,34 @@
|
|||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||
<!--
|
||||
launchd user agent — firecuda (max): every 4 hours
|
||||
Install: ./install-firecuda-timer.sh
|
||||
-->
|
||||
<plist version="1.0">
|
||||
<dict>
|
||||
<key>Label</key>
|
||||
<string>com.hacktivism.taler-monitoring-sites</string>
|
||||
<key>ProgramArguments</key>
|
||||
<array>
|
||||
<string>/bin/bash</string>
|
||||
<string>__SITE_GEN__/run-on-firecuda.sh</string>
|
||||
</array>
|
||||
<key>StartInterval</key>
|
||||
<integer>14400</integer>
|
||||
<key>RunAtLoad</key>
|
||||
<true/>
|
||||
<key>StandardOutPath</key>
|
||||
<string>__HOME__/Library/Logs/taler-monitoring-sites/launchd.out.log</string>
|
||||
<key>StandardErrorPath</key>
|
||||
<string>__HOME__/Library/Logs/taler-monitoring-sites/launchd.err.log</string>
|
||||
<key>EnvironmentVariables</key>
|
||||
<dict>
|
||||
<key>PATH</key>
|
||||
<string>/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin</string>
|
||||
<key>SKIP_SSH</key>
|
||||
<string>1</string>
|
||||
<key>CONTINUE_ON_ERROR</key>
|
||||
<string>1</string>
|
||||
</dict>
|
||||
</dict>
|
||||
</plist>
|
||||
935
site-gen/console_to_html.py
Executable file
935
site-gen/console_to_html.py
Executable file
|
|
@ -0,0 +1,935 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Convert taler-monitoring console log to a console-feel HTML page."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import html
|
||||
import re
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
# ANSI strip
|
||||
ANSI_RE = re.compile(r"\x1b\[[0-9;]*m")
|
||||
ERR_BULLET_RE = re.compile(r"^\s*[•·*-]\s+(?P<body>.+)$")
|
||||
TID_RE = re.compile(r"\b(?P<tid>[a-z0-9_.-]+-\d+)\b", re.I)
|
||||
|
||||
|
||||
def strip_ansi(s: str) -> str:
|
||||
return ANSI_RE.sub("", s)
|
||||
|
||||
|
||||
def classify(line: str) -> str:
|
||||
u = line.upper()
|
||||
# section / summary headers before badge matching (they contain the word ERROR)
|
||||
if "SUMMARY" in u or "--- ERRORS" in u or "ERRORS ·" in u or "ERRORS (" in u:
|
||||
return "meta"
|
||||
if "numbered checks" in line.lower() or "totals:" in line.lower():
|
||||
return "meta"
|
||||
if "BLOCKER" in u or "┌ BLOCKER" in u:
|
||||
return "blocker"
|
||||
if "ERROR" in u or "┌ ERROR" in u:
|
||||
return "error"
|
||||
if "WARN" in u or "┌ WARN" in u:
|
||||
return "warn"
|
||||
if "INFO" in u or "┌ INFO" in u:
|
||||
return "info"
|
||||
if " OK" in u or "[OK" in u or "┌ OK" in u or u.strip().startswith("OK"):
|
||||
return "ok"
|
||||
if u.strip().startswith("-- ") or u.strip().startswith("=="):
|
||||
return "section"
|
||||
return "plain"
|
||||
|
||||
|
||||
def is_run_timeout_text(text: str) -> bool:
|
||||
u = text.upper()
|
||||
return "RUN.TIMEOUT" in u or "RUN_TIMEOUT" in u or "RUN TIMEOUT" in u
|
||||
|
||||
|
||||
def is_summary_error_line(ln: str) -> bool:
|
||||
"""Skip summary/header lines that contain ERROR but are not checks."""
|
||||
low = ln.lower()
|
||||
if "failed — see" in low or "failed - see" in low:
|
||||
return True
|
||||
if "errors · failed" in low or "errors (failed" in low:
|
||||
return True
|
||||
if "--- errors" in low:
|
||||
return True
|
||||
if "┌ summary" in low or "[ summary" in low:
|
||||
return True
|
||||
if "totals:" in low:
|
||||
return True
|
||||
# header-ish RUN_TIMEOUT= without error badge
|
||||
if is_run_timeout_text(ln) and "ERROR" not in ln.upper() and "run.timeout" not in low:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def extract_errors(lines: list[str]) -> list[str]:
|
||||
errs: list[str] = []
|
||||
timeout_detail = ""
|
||||
in_block = False
|
||||
for ln in lines:
|
||||
if "--- ERRORS" in ln or "ERRORS (failed" in ln or "RUN TIMEOUT · extraordinary" in ln:
|
||||
in_block = True
|
||||
continue
|
||||
if in_block:
|
||||
if ln.strip().startswith("---") or "[ SUMMARY" in ln or "totals:" in ln:
|
||||
in_block = False
|
||||
continue
|
||||
m = ERR_BULLET_RE.match(ln)
|
||||
if m:
|
||||
body = m.group("body").strip()
|
||||
if is_run_timeout_text(body):
|
||||
timeout_detail = body
|
||||
else:
|
||||
errs.append(body)
|
||||
elif "ERROR" in ln.upper() and ln.strip() and not is_run_timeout_text(ln):
|
||||
errs.append(ln.strip())
|
||||
if re.search(r"\bERROR\b", ln, re.I) and "--- ERRORS" not in ln:
|
||||
if is_run_timeout_text(ln):
|
||||
if "run.timeout-01" in ln.lower() or "RUN_TIMEOUT exceeded" in ln:
|
||||
timeout_detail = re.sub(r"\s+", " ", strip_ansi(ln)).strip()
|
||||
continue
|
||||
if is_summary_error_line(ln):
|
||||
continue
|
||||
body = re.sub(r"^.*\bERROR\b\s*\]?\s*", "", ln, flags=re.I).strip(" ·")
|
||||
if body and body not in errs and "failed — see" not in body.lower():
|
||||
if re.search(
|
||||
r"#\d+|www\.|e2e\.|auth401\.|versions\.|inside\.|surface\.|aptdeploy\.",
|
||||
body,
|
||||
):
|
||||
errs.append(body)
|
||||
seen: set[str] = set()
|
||||
out: list[str] = []
|
||||
for e in errs:
|
||||
if e in seen or is_run_timeout_text(e):
|
||||
continue
|
||||
seen.add(e)
|
||||
out.append(e)
|
||||
if timeout_detail or any(
|
||||
is_run_timeout_text(l) and ("ERROR" in l.upper() or "run.timeout" in l.lower())
|
||||
for l in lines
|
||||
):
|
||||
# only if real timeout ERROR badge, not mere RUN_TIMEOUT= header
|
||||
if timeout_detail or any(
|
||||
"run.timeout-01" in l.lower() or "RUN_TIMEOUT exceeded" in l for l in lines
|
||||
):
|
||||
canon = timeout_detail or "run.timeout-01 [run] RUN_TIMEOUT exceeded"
|
||||
out = [canon] + out
|
||||
return out
|
||||
|
||||
|
||||
def count_status(lines: list[str]) -> tuple[int, int, int | None, int | None]:
|
||||
"""Return (error_count, warn_count, first_error_idx, first_warn_idx)."""
|
||||
n_err = 0
|
||||
n_warn = 0
|
||||
first_err: int | None = None
|
||||
first_warn: int | None = None
|
||||
for i, ln in enumerate(lines):
|
||||
kind = classify(ln)
|
||||
if kind in ("error", "blocker"):
|
||||
if is_summary_error_line(ln):
|
||||
continue
|
||||
n_err += 1
|
||||
if first_err is None:
|
||||
first_err = i
|
||||
elif kind == "warn":
|
||||
n_warn += 1
|
||||
if first_warn is None:
|
||||
first_warn = i
|
||||
return n_err, n_warn, first_err, first_warn
|
||||
|
||||
|
||||
def slug_error(i: int, text: str) -> str:
|
||||
if is_run_timeout_text(text):
|
||||
return "err-run-timeout"
|
||||
m = TID_RE.search(text)
|
||||
if m:
|
||||
return "err-" + re.sub(r"[^a-z0-9_-]+", "-", m.group("tid").lower())
|
||||
return f"err-{i:03d}"
|
||||
|
||||
|
||||
def render_line(ln: str, err_slugs: dict[str, str], extra_ids: list[str] | None = None) -> str:
|
||||
kind = classify(ln)
|
||||
esc = html.escape(ln)
|
||||
for tid, slug in err_slugs.items():
|
||||
if tid in ln:
|
||||
esc = esc.replace(
|
||||
html.escape(tid),
|
||||
f'<a class="jump" href="#{html.escape(slug)}">{html.escape(tid)}</a>',
|
||||
)
|
||||
id_attr = ""
|
||||
if extra_ids:
|
||||
id_attr = f' id="{" ".join(extra_ids)}"' # invalid multi-id; use first only
|
||||
# HTML allows one id — join carefully
|
||||
id_attr = f' id="{html.escape(extra_ids[0])}"'
|
||||
if len(extra_ids) > 1:
|
||||
# secondary anchors as empty spans prepended
|
||||
spans = "".join(f'<span id="{html.escape(x)}"></span>' for x in extra_ids[1:])
|
||||
return f'{spans}<div{id_attr} class="line {kind}">{esc}</div>\n'
|
||||
return f'<div{id_attr} class="line {kind}">{esc}</div>\n'
|
||||
|
||||
|
||||
def status_level(n_err: int, n_warn: int) -> str:
|
||||
if n_err > 0:
|
||||
return "red"
|
||||
if n_warn > 0:
|
||||
return "yellow"
|
||||
return "green"
|
||||
|
||||
|
||||
def extract_phases(log_text: str) -> str:
|
||||
"""Best-effort phases string from host-agent / taler-monitoring log header."""
|
||||
for pat in (
|
||||
r"phases=([a-z0-9 _.-]+)",
|
||||
r"· phases=([a-z0-9 _.-]+)",
|
||||
r"phases\s{2,}([a-z0-9 _.-]+)",
|
||||
r"^\s*phases\s+([a-z0-9 _.-]+)\s*$",
|
||||
):
|
||||
m = re.search(pat, log_text, re.I | re.M)
|
||||
if m:
|
||||
return re.sub(r"\s+", " ", m.group(1)).strip(" ·")
|
||||
return ""
|
||||
|
||||
|
||||
def monitoring_scope(
|
||||
page_label: str,
|
||||
hostname: str,
|
||||
log_text: str = "",
|
||||
) -> dict[str, object]:
|
||||
"""
|
||||
Human-readable description of what this monitoring page covers.
|
||||
Returned keys: title, summary, items (list[str]), kind
|
||||
"""
|
||||
label = (page_label or "monitoring").lower().replace("_", "-")
|
||||
phases = extract_phases(log_text)
|
||||
host = hostname or "?"
|
||||
|
||||
if "surface" in label:
|
||||
return {
|
||||
"kind": "surface",
|
||||
"title": "Remote ecosystem surface inventory",
|
||||
"summary": "Public catalog hosts · DNS / TCP / HTTPS / TLS · Server version · CVE",
|
||||
"items": [
|
||||
"Outside-in only: no SSH and no podman exec on remote targets",
|
||||
"Catalog (surface-catalog.conf): taler.net, demo/test, gnunet.org, "
|
||||
"taler-ops.ch, taler-systems.com, deb.taler.net, ftp.gnu.org, …",
|
||||
"Per host: DNS resolve, ICMP (info), open ports, protocol probes, "
|
||||
"TLS cert expiry, Server-header software version, optional CVE check",
|
||||
f"HTML published only on taler.hacktivism.ch "
|
||||
f"(this page host: {host})",
|
||||
"Timer: taler-monitoring-surface.timer (hourly)",
|
||||
],
|
||||
}
|
||||
|
||||
if "aptdeploy" in label or "apt-deploy" in label or "apt_src" in label:
|
||||
return {
|
||||
"kind": "aptdeploy",
|
||||
"title": "apt-src merchant deploy tests (podman on koopa)",
|
||||
"summary": "4 containers · fresh + upgrade tracks · trixie & trixie-testing",
|
||||
"items": [
|
||||
"koopa-taler-deploy-test-apt-src-trixie (fresh, suite trixie)",
|
||||
"koopa-taler-deploy-test-apt-src-trixie-testing (fresh, trixie-testing)",
|
||||
"koopa-taler-deploy-test-apt-src-trixie-upgrade (upgrade track)",
|
||||
"koopa-taler-deploy-test-apt-src-trixie-testing-upgrade (upgrade track)",
|
||||
"Checks: packages, taler-merchant-httpd --version / ldd, systemd unit, "
|
||||
"optional local /config probe — not public nginx/HTTPS",
|
||||
f"HTML published only on taler.hacktivism.ch (this page host: {host})",
|
||||
"Timer: taler-monitoring-aptdeploy.timer (4h)",
|
||||
],
|
||||
}
|
||||
|
||||
# Default: stack /monitoring pages (bank, exchange, merchant)
|
||||
phase_txt = phases or "urls · inside · versions"
|
||||
role = "stack"
|
||||
if host.startswith("bank."):
|
||||
role = "bank"
|
||||
focus = "Libeufin bank public HTTPS + container inside checks"
|
||||
elif host.startswith("exchange.") or host.startswith("stage.exchange."):
|
||||
role = "exchange"
|
||||
focus = "Exchange public HTTPS + container inside checks"
|
||||
elif host.startswith("taler.") or "merchant" in host or host.startswith("monnaie."):
|
||||
role = "merchant"
|
||||
focus = "Merchant backend / SPA public HTTPS + container inside checks"
|
||||
else:
|
||||
focus = "Taler stack public + inside checks"
|
||||
|
||||
items = [
|
||||
f"Focus: {focus}",
|
||||
f"Phases this run: {phase_txt}",
|
||||
"Typical: public URLs (HTTPS, QR, perf), inside (podman/ssh), package versions",
|
||||
f"HTML host: {host}",
|
||||
"Timer: taler-monitoring-hacktivism (path + 4h) · /monitoring on bank, exchange, taler",
|
||||
]
|
||||
if phases:
|
||||
items.insert(2, f"Log header phases= {phases}")
|
||||
|
||||
return {
|
||||
"kind": role,
|
||||
"title": f"Stack monitoring · {host}",
|
||||
"summary": f"{phase_txt} · public + inside",
|
||||
"items": items,
|
||||
}
|
||||
|
||||
|
||||
def sticky_bar_html(
|
||||
*,
|
||||
generated_at: str,
|
||||
generated_iso: str,
|
||||
n_err: int,
|
||||
n_warn: int,
|
||||
hostname: str,
|
||||
page_label: str,
|
||||
mode: str,
|
||||
commit_html: str,
|
||||
suite_path: str,
|
||||
link_other: str | None,
|
||||
first_err_href: str | None,
|
||||
first_warn_href: str | None,
|
||||
scope: dict[str, object] | None = None,
|
||||
) -> str:
|
||||
level = status_level(n_err, n_warn)
|
||||
if level == "red":
|
||||
status_txt = "ERRORS"
|
||||
elif level == "yellow":
|
||||
status_txt = "WARNINGS"
|
||||
else:
|
||||
status_txt = "OK"
|
||||
|
||||
if first_err_href and n_err:
|
||||
err_stat = (
|
||||
f'<a class="stat err" href="{html.escape(first_err_href)}" '
|
||||
f'title="Jump to first error">{n_err} error{"s" if n_err != 1 else ""}</a>'
|
||||
)
|
||||
else:
|
||||
err_stat = f'<span class="stat err muted">{n_err} errors</span>'
|
||||
|
||||
if first_warn_href and n_warn:
|
||||
warn_stat = (
|
||||
f'<a class="stat warn" href="{html.escape(first_warn_href)}" '
|
||||
f'title="Jump to first warning">{n_warn} warning{"s" if n_warn != 1 else ""}</a>'
|
||||
)
|
||||
else:
|
||||
warn_stat = f'<span class="stat warn muted">{n_warn} warnings</span>'
|
||||
|
||||
other = ""
|
||||
if link_other:
|
||||
other = (
|
||||
f'<a class="nav-link" href="{html.escape(link_other)}">'
|
||||
f"{html.escape(link_other)}</a>"
|
||||
)
|
||||
|
||||
if scope is None:
|
||||
scope = monitoring_scope(page_label, hostname, "")
|
||||
scope_title = html.escape(str(scope.get("title") or "Monitoring"))
|
||||
scope_summary = html.escape(str(scope.get("summary") or ""))
|
||||
scope_kind = html.escape(str(scope.get("kind") or "monitoring"))
|
||||
lis = "\n".join(
|
||||
f"<li>{html.escape(str(it))}</li>" for it in (scope.get("items") or [])
|
||||
)
|
||||
suite_bit = (
|
||||
f" · <code>{html.escape(suite_path)}</code>" if suite_path else ""
|
||||
)
|
||||
|
||||
# Compact sticky row always visible; "Was geprüft wird" expands inside sticky-bar
|
||||
return f"""
|
||||
<div class="sticky-bar sticky-{level}" id="status-bar" role="status"
|
||||
data-errors="{n_err}" data-warnings="{n_warn}" data-level="{level}"
|
||||
data-scope-kind="{scope_kind}">
|
||||
<div class="sticky-bar-row primary">
|
||||
<span class="status-pill sticky-{level}">{html.escape(status_txt)}</span>
|
||||
<span class="host">{html.escape(page_label)} · {html.escape(hostname)}</span>
|
||||
<span class="sep">·</span>
|
||||
{err_stat}
|
||||
<span class="sep">·</span>
|
||||
{warn_stat}
|
||||
<span class="sep">·</span>
|
||||
<span class="generated"
|
||||
data-generated-iso="{html.escape(generated_iso)}"
|
||||
title="{html.escape(generated_at)}">
|
||||
generated <time datetime="{html.escape(generated_iso)}">{html.escape(generated_at)}</time>
|
||||
<span class="age" id="generated-age"></span>
|
||||
</span>
|
||||
<button type="button" class="scope-toggle" id="scope-toggle"
|
||||
aria-expanded="false" aria-controls="sticky-scope-panel"
|
||||
title="Aufklappen: was diese Seite prüft">
|
||||
<span class="scope-chevron" aria-hidden="true"></span>
|
||||
<span class="scope-toggle-label">Was geprüft wird</span>
|
||||
<span class="scope-toggle-hint">{scope_summary}</span>
|
||||
</button>
|
||||
</div>
|
||||
<div class="sticky-scope-panel" id="sticky-scope-panel" hidden>
|
||||
<div class="sticky-bar-row secondary">
|
||||
<span class="badge mode-{html.escape(mode)}">{html.escape(mode.upper())}</span>
|
||||
source {commit_html}{suite_bit}
|
||||
{(" · " + other) if other else ""}
|
||||
</div>
|
||||
<div class="scope-body">
|
||||
<h3 class="scope-title">{scope_title}</h3>
|
||||
<p class="scope-lead">{scope_summary}</p>
|
||||
<ul class="scope-list">
|
||||
{lis}
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
"""
|
||||
|
||||
|
||||
STICKY_CSS = """
|
||||
.sticky-bar {
|
||||
position: sticky; top: 0; z-index: 100;
|
||||
border-bottom: 2px solid var(--border);
|
||||
padding: 10px 14px 8px;
|
||||
backdrop-filter: blur(8px);
|
||||
-webkit-backdrop-filter: blur(8px);
|
||||
box-shadow: 0 2px 12px #0008;
|
||||
}
|
||||
.sticky-bar.sticky-green {
|
||||
background: linear-gradient(180deg, #0d1f14f2, #0a1410ee);
|
||||
border-bottom-color: #1f6b3a;
|
||||
}
|
||||
.sticky-bar.sticky-yellow {
|
||||
background: linear-gradient(180deg, #2a2210f2, #1a160cee);
|
||||
border-bottom-color: #a68b2d;
|
||||
}
|
||||
.sticky-bar.sticky-red {
|
||||
background: linear-gradient(180deg, #2a1010f2, #1a0808ee);
|
||||
border-bottom-color: #a33;
|
||||
}
|
||||
.sticky-bar-row {
|
||||
display: flex; flex-wrap: wrap; align-items: center; gap: 6px 10px;
|
||||
max-width: 1200px;
|
||||
}
|
||||
.sticky-bar-row.secondary {
|
||||
margin-top: 8px; font-size: 12px; color: var(--dim);
|
||||
}
|
||||
.sticky-bar-row.secondary a { color: var(--info); text-decoration: none; }
|
||||
.sticky-bar-row.secondary a:hover { text-decoration: underline; }
|
||||
.status-pill {
|
||||
display: inline-block; font-size: 11px; font-weight: 800;
|
||||
letter-spacing: 0.06em; padding: 2px 10px; border-radius: 3px;
|
||||
border: 1px solid;
|
||||
}
|
||||
.status-pill.sticky-green { color: var(--ok); border-color: #1f6b3a; background: #0a1a10; }
|
||||
.status-pill.sticky-yellow { color: var(--warn); border-color: #a68b2d; background: #1a1608; }
|
||||
.status-pill.sticky-red { color: var(--err); border-color: #a33; background: #1a0a0a; }
|
||||
.host { font-weight: 600; color: #eee; font-size: 13px; }
|
||||
.sep { color: var(--dim); }
|
||||
.stat {
|
||||
font-weight: 700; font-size: 12px; text-decoration: none;
|
||||
padding: 1px 8px; border-radius: 3px; border: 1px solid transparent;
|
||||
}
|
||||
.stat.err { color: var(--err); border-color: #522; background: #1a0a0a; }
|
||||
.stat.warn { color: var(--warn); border-color: #664; background: #1a1608; }
|
||||
.stat.muted { opacity: 0.55; font-weight: 600; }
|
||||
a.stat:hover { text-decoration: underline; filter: brightness(1.15); }
|
||||
.generated { color: var(--dim); font-size: 12px; }
|
||||
.generated time { color: #bbb; }
|
||||
.generated .age { color: #888; margin-left: 4px; }
|
||||
.generated .age:not(:empty)::before { content: "("; }
|
||||
.generated .age:not(:empty)::after { content: ")"; }
|
||||
.badge {
|
||||
display: inline-block; padding: 1px 8px; border-radius: 3px;
|
||||
font-size: 11px; font-weight: 700; margin-right: 4px;
|
||||
border: 1px solid var(--border);
|
||||
}
|
||||
.badge.mode-err { color: var(--err); border-color: #522; background: #1a0a0a; }
|
||||
.badge.mode-ok { color: var(--ok); border-color: #143; background: #0a1a10; }
|
||||
.badge.mode-redirect { color: var(--warn); border-color: #664; background: #1a1608; }
|
||||
/* collapsible block inside sticky-bar */
|
||||
.scope-toggle {
|
||||
margin-left: auto;
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
max-width: min(42rem, 100%);
|
||||
cursor: pointer;
|
||||
font: inherit;
|
||||
font-size: 12px;
|
||||
color: #ddd;
|
||||
background: #0006;
|
||||
border: 1px solid #444;
|
||||
border-radius: 4px;
|
||||
padding: 4px 10px;
|
||||
text-align: left;
|
||||
}
|
||||
.scope-toggle:hover { border-color: #888; background: #111a; color: #fff; }
|
||||
.scope-toggle:focus-visible { outline: 2px solid var(--info); outline-offset: 2px; }
|
||||
.scope-toggle[aria-expanded="true"] {
|
||||
border-color: #777;
|
||||
background: #141414;
|
||||
}
|
||||
.scope-chevron {
|
||||
display: inline-block;
|
||||
width: 0; height: 0;
|
||||
border-left: 5px solid #aaa;
|
||||
border-top: 4px solid transparent;
|
||||
border-bottom: 4px solid transparent;
|
||||
transition: transform 0.12s ease;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
.scope-toggle[aria-expanded="true"] .scope-chevron {
|
||||
transform: rotate(90deg);
|
||||
border-left-color: #eee;
|
||||
}
|
||||
.scope-toggle-label {
|
||||
font-weight: 700;
|
||||
letter-spacing: 0.03em;
|
||||
text-transform: uppercase;
|
||||
font-size: 11px;
|
||||
white-space: nowrap;
|
||||
}
|
||||
.scope-toggle-hint {
|
||||
color: #888;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
min-width: 0;
|
||||
}
|
||||
.sticky-scope-panel {
|
||||
margin-top: 8px;
|
||||
max-width: 1200px;
|
||||
border: 1px solid #333;
|
||||
border-radius: 4px;
|
||||
background: #080808e6;
|
||||
padding: 8px 10px 10px;
|
||||
}
|
||||
.sticky-scope-panel[hidden] { display: none !important; }
|
||||
.scope-body { padding: 4px 4px 2px 6px; }
|
||||
.scope-title {
|
||||
margin: 4px 0 4px;
|
||||
font-size: 13px;
|
||||
font-weight: 600;
|
||||
color: #eee;
|
||||
}
|
||||
.scope-lead {
|
||||
margin: 0 0 6px;
|
||||
font-size: 12px;
|
||||
color: #999;
|
||||
}
|
||||
.scope-list {
|
||||
margin: 0;
|
||||
padding-left: 1.15em;
|
||||
color: #b0b0b0;
|
||||
font-size: 12px;
|
||||
line-height: 1.5;
|
||||
}
|
||||
.scope-list li { margin: 3px 0; }
|
||||
#first-error, #first-warn,
|
||||
[id^="err-"][id$="-line"] {
|
||||
scroll-margin-top: 120px;
|
||||
}
|
||||
@media (max-width: 720px) {
|
||||
.scope-toggle { margin-left: 0; width: 100%; }
|
||||
.scope-toggle-hint { display: none; }
|
||||
}
|
||||
"""
|
||||
|
||||
STICKY_JS = """
|
||||
<script>
|
||||
(function () {
|
||||
function fmtAge(sec) {
|
||||
if (sec < 0) sec = 0;
|
||||
if (sec < 60) return sec + "s ago";
|
||||
if (sec < 3600) return Math.floor(sec / 60) + "m ago";
|
||||
if (sec < 86400) return Math.floor(sec / 3600) + "h ago";
|
||||
return Math.floor(sec / 86400) + "d ago";
|
||||
}
|
||||
function tick() {
|
||||
var el = document.querySelector("[data-generated-iso]");
|
||||
var age = document.getElementById("generated-age");
|
||||
if (!el || !age) return;
|
||||
var iso = el.getAttribute("data-generated-iso");
|
||||
if (!iso) return;
|
||||
var t = Date.parse(iso);
|
||||
if (isNaN(t)) return;
|
||||
var sec = Math.floor((Date.now() - t) / 1000);
|
||||
age.textContent = fmtAge(sec);
|
||||
age.title = "page age · updates every second";
|
||||
}
|
||||
tick();
|
||||
setInterval(tick, 1000);
|
||||
|
||||
// Sticky-bar collapsible: what this page monitors
|
||||
var btn = document.getElementById("scope-toggle");
|
||||
var panel = document.getElementById("sticky-scope-panel");
|
||||
if (btn && panel) {
|
||||
var key = "taler-mon-scope-open:" + (location.pathname || "");
|
||||
function setOpen(open) {
|
||||
btn.setAttribute("aria-expanded", open ? "true" : "false");
|
||||
if (open) panel.removeAttribute("hidden");
|
||||
else panel.setAttribute("hidden", "");
|
||||
try { localStorage.setItem(key, open ? "1" : "0"); } catch (e) {}
|
||||
}
|
||||
btn.addEventListener("click", function () {
|
||||
setOpen(btn.getAttribute("aria-expanded") !== "true");
|
||||
});
|
||||
try {
|
||||
if (localStorage.getItem(key) === "1") setOpen(true);
|
||||
} catch (e) {}
|
||||
}
|
||||
})();
|
||||
</script>
|
||||
"""
|
||||
|
||||
|
||||
def build_html(
|
||||
*,
|
||||
title: str,
|
||||
hostname: str,
|
||||
mode: str,
|
||||
log_text: str,
|
||||
commit: str,
|
||||
commit_url: str,
|
||||
suite_path: str,
|
||||
generated_at: str,
|
||||
generated_iso: str,
|
||||
link_other: str | None,
|
||||
page_label: str = "monitoring",
|
||||
) -> str:
|
||||
raw_lines = [strip_ansi(l).rstrip("\n") for l in log_text.splitlines()]
|
||||
while raw_lines and not raw_lines[-1].strip():
|
||||
raw_lines.pop()
|
||||
|
||||
n_err, n_warn, first_err_i, first_warn_i = count_status(raw_lines)
|
||||
errors = extract_errors(raw_lines)
|
||||
|
||||
err_slugs: dict[str, str] = {}
|
||||
for i, e in enumerate(errors, 1):
|
||||
slug = slug_error(i, e)
|
||||
err_slugs[e] = slug
|
||||
m = TID_RE.search(e)
|
||||
if m:
|
||||
err_slugs[m.group("tid")] = slug
|
||||
|
||||
tid_to_slug = {
|
||||
k: v for k, v in err_slugs.items() if re.match(r"^[a-z0-9_.-]+-\d+$", k, re.I)
|
||||
}
|
||||
|
||||
# Build body with anchors for first error/warn + per-error slugs
|
||||
body_lines: list[str] = []
|
||||
claimed_err: set[str] = set()
|
||||
for idx, ln in enumerate(raw_lines):
|
||||
extra: list[str] = []
|
||||
if first_err_i is not None and idx == first_err_i:
|
||||
extra.append("first-error")
|
||||
if first_warn_i is not None and idx == first_warn_i:
|
||||
extra.append("first-warn")
|
||||
# per-error jump targets (all modes)
|
||||
if classify(ln) in ("error", "blocker") and not is_summary_error_line(ln):
|
||||
for i, e in enumerate(errors, 1):
|
||||
slug = slug_error(i, e)
|
||||
if slug in claimed_err:
|
||||
continue
|
||||
tid_m = TID_RE.search(e)
|
||||
if is_run_timeout_text(e):
|
||||
if "run.timeout-01" in ln and "ERROR" in ln.upper():
|
||||
extra.append(f"{slug}-line")
|
||||
claimed_err.add(slug)
|
||||
break
|
||||
continue
|
||||
key = tid_m.group(1) if tid_m else e[:40]
|
||||
if key and key in ln:
|
||||
extra.append(f"{slug}-line")
|
||||
claimed_err.add(slug)
|
||||
break
|
||||
body_lines.append(render_line(ln, tid_to_slug, extra or None))
|
||||
|
||||
err_nav = ""
|
||||
if errors and mode == "err":
|
||||
items = []
|
||||
timeout_items = []
|
||||
for i, e in enumerate(errors, 1):
|
||||
slug = slug_error(i, e)
|
||||
li = (
|
||||
f'<li id="{html.escape(slug)}">'
|
||||
f'<a href="#{html.escape(slug)}-line">{html.escape(e)}</a></li>'
|
||||
)
|
||||
if is_run_timeout_text(e):
|
||||
timeout_items.append(li)
|
||||
else:
|
||||
items.append(li)
|
||||
if timeout_items:
|
||||
err_nav += (
|
||||
'<section class="err-top extraordinary" id="run-timeout-top">\n'
|
||||
"<h2>EXTRAORDINARY · RUN TIMEOUT</h2>\n"
|
||||
"<p>Whole-run wall clock exceeded. "
|
||||
'<a href="#err-run-timeout-line">Jump to detail →</a></p>\n'
|
||||
'<ol class="err-list">\n'
|
||||
+ "\n".join(timeout_items[:1])
|
||||
+ "\n</ol>\n</section>\n"
|
||||
)
|
||||
rest = items if timeout_items else (timeout_items + items)
|
||||
if rest or not timeout_items:
|
||||
err_nav += (
|
||||
'<section class="err-top">\n'
|
||||
f"<h2>ERRORS ({len(errors)}) — jump to detail</h2>\n"
|
||||
'<ol class="err-list">\n'
|
||||
+ "\n".join(rest if rest else items)
|
||||
+ "\n</ol>\n</section>\n"
|
||||
)
|
||||
|
||||
commit_short = commit[:12] if commit else "unknown"
|
||||
commit_html = (
|
||||
f'<a href="{html.escape(commit_url)}">{html.escape(commit_short)}</a>'
|
||||
if commit_url
|
||||
else html.escape(commit_short)
|
||||
)
|
||||
|
||||
first_err_href = "#first-error" if first_err_i is not None else None
|
||||
first_warn_href = "#first-warn" if first_warn_i is not None else None
|
||||
scope = monitoring_scope(page_label, hostname, log_text)
|
||||
|
||||
bar = sticky_bar_html(
|
||||
generated_at=generated_at,
|
||||
generated_iso=generated_iso,
|
||||
n_err=n_err,
|
||||
n_warn=n_warn,
|
||||
hostname=hostname,
|
||||
page_label=page_label,
|
||||
mode=mode,
|
||||
commit_html=commit_html,
|
||||
suite_path=suite_path,
|
||||
link_other=link_other,
|
||||
first_err_href=first_err_href,
|
||||
first_warn_href=first_warn_href,
|
||||
scope=scope,
|
||||
)
|
||||
|
||||
return f"""<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8"/>
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1"/>
|
||||
<meta name="generated" content="{html.escape(generated_iso)}"/>
|
||||
<title>{html.escape(title)}</title>
|
||||
<style>
|
||||
:root {{
|
||||
--bg: #0c0c0c;
|
||||
--fg: #c8c8c8;
|
||||
--dim: #666;
|
||||
--ok: #3ddc84;
|
||||
--err: #ff5c5c;
|
||||
--warn: #e6c07b;
|
||||
--info: #61afef;
|
||||
--blocker: #c678dd;
|
||||
--sec: #56b6c2;
|
||||
--border: #222;
|
||||
--panel: #121212;
|
||||
font-family: "JetBrains Mono", "Fira Code", "Cascadia Code", ui-monospace, Menlo, Consolas, monospace;
|
||||
}}
|
||||
* {{ box-sizing: border-box; }}
|
||||
body {{
|
||||
margin: 0; background: var(--bg); color: var(--fg);
|
||||
font-size: 13px; line-height: 1.45;
|
||||
}}
|
||||
{STICKY_CSS}
|
||||
main {{ padding: 12px 14px 48px; max-width: 1200px; }}
|
||||
.err-top {{
|
||||
background: var(--panel); border: 1px solid #422;
|
||||
padding: 10px 12px; margin-bottom: 14px; border-radius: 4px;
|
||||
}}
|
||||
.err-top.extraordinary {{
|
||||
border-color: #a44; background: #1a0808;
|
||||
box-shadow: 0 0 0 1px #622 inset;
|
||||
}}
|
||||
.err-top.extraordinary h2 {{ color: #ff8a8a; }}
|
||||
.err-top.extraordinary p {{ margin: 0 0 8px; color: var(--fg); font-size: 12px; }}
|
||||
.err-top.extraordinary p a {{ color: #ff8a8a; font-weight: 600; }}
|
||||
.err-top h2 {{ margin: 0 0 8px; font-size: 13px; color: var(--err); }}
|
||||
.err-list {{ margin: 0; padding-left: 1.2em; }}
|
||||
.err-list li {{ margin: 4px 0; }}
|
||||
.err-list a {{ color: var(--err); text-decoration: none; }}
|
||||
.err-list a:hover {{ text-decoration: underline; }}
|
||||
#err-run-timeout-line {{
|
||||
outline: 1px solid #a44; background: #1a0808;
|
||||
padding: 4px 6px; margin: 4px 0; border-radius: 3px;
|
||||
}}
|
||||
.console {{
|
||||
background: #0a0a0a; border: 1px solid var(--border);
|
||||
border-radius: 4px; padding: 8px 10px;
|
||||
white-space: pre-wrap; word-break: break-word;
|
||||
}}
|
||||
.line {{ padding: 1px 0; }}
|
||||
.line.ok {{ color: var(--ok); }}
|
||||
.line.error {{ color: var(--err); }}
|
||||
.line.warn {{ color: var(--warn); }}
|
||||
.line.info {{ color: var(--info); }}
|
||||
.line.blocker {{ color: var(--blocker); }}
|
||||
.line.section {{ color: var(--sec); margin-top: 8px; font-weight: 600; }}
|
||||
.line.meta {{ color: var(--dim); }}
|
||||
.line.plain {{ color: var(--fg); }}
|
||||
a.jump {{ color: inherit; text-decoration: underline dotted; }}
|
||||
footer {{
|
||||
margin-top: 18px; color: var(--dim); font-size: 12px;
|
||||
border-top: 1px solid var(--border); padding-top: 10px;
|
||||
}}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
{bar}
|
||||
<main>
|
||||
{err_nav}
|
||||
<div class="console">
|
||||
{"".join(body_lines)}
|
||||
</div>
|
||||
<footer>
|
||||
Console-style render of taler-monitoring output.
|
||||
Sticky bar: green = clean · yellow = warnings · red = errors.
|
||||
Commit pins the exact tree used for this run.
|
||||
</footer>
|
||||
</main>
|
||||
{STICKY_JS}
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
|
||||
|
||||
def build_redirect_html(
|
||||
*,
|
||||
hostname: str,
|
||||
commit: str,
|
||||
commit_url: str,
|
||||
path_err: str = "/monitoring_err/",
|
||||
page_label: str = "monitoring",
|
||||
generated_at: str = "",
|
||||
generated_iso: str = "",
|
||||
n_err: int = 0,
|
||||
n_warn: int = 0,
|
||||
log_text: str = "",
|
||||
) -> str:
|
||||
commit_short = commit[:12] if commit else "unknown"
|
||||
link = path_err if path_err.endswith("/") else path_err + "/"
|
||||
c = (
|
||||
f'<a href="{html.escape(commit_url)}">{html.escape(commit_short)}</a>'
|
||||
if commit_url
|
||||
else html.escape(commit_short)
|
||||
)
|
||||
if not generated_at:
|
||||
now = datetime.now(timezone.utc)
|
||||
generated_iso = now.strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
generated_at = now.strftime("%Y-%m-%d %H:%M:%SZ")
|
||||
# Redirect pages exist only when the run failed → treat as red if n_err unknown
|
||||
if n_err <= 0:
|
||||
n_err = max(n_err, 1)
|
||||
scope = monitoring_scope(page_label, hostname, log_text)
|
||||
bar = sticky_bar_html(
|
||||
generated_at=generated_at,
|
||||
generated_iso=generated_iso,
|
||||
n_err=n_err,
|
||||
n_warn=n_warn,
|
||||
hostname=hostname,
|
||||
page_label=page_label,
|
||||
mode="redirect",
|
||||
commit_html=c,
|
||||
suite_path="",
|
||||
link_other=link,
|
||||
first_err_href=link,
|
||||
first_warn_href=None,
|
||||
scope=scope,
|
||||
)
|
||||
return f"""<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8"/>
|
||||
<meta http-equiv="refresh" content="2; url={html.escape(link)}"/>
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1"/>
|
||||
<meta name="generated" content="{html.escape(generated_iso)}"/>
|
||||
<title>{html.escape(page_label)} · errors · {html.escape(hostname)}</title>
|
||||
<style>
|
||||
:root {{
|
||||
--bg: #0c0c0c; --fg: #c8c8c8; --dim: #666;
|
||||
--ok: #3ddc84; --err: #ff5c5c; --warn: #e6c07b; --info: #61afef; --border: #222;
|
||||
font-family: ui-monospace, Menlo, Consolas, monospace;
|
||||
}}
|
||||
body {{ margin: 0; background: var(--bg); color: var(--fg); font-size: 13px; }}
|
||||
{STICKY_CSS}
|
||||
.box {{
|
||||
max-width: 520px; margin: 12vh auto; padding: 20px;
|
||||
border: 1px solid #422; background: #121212; border-radius: 4px;
|
||||
}}
|
||||
.box a {{ color: var(--err); }}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
{bar}
|
||||
<div class="box">
|
||||
<p><strong>{html.escape(page_label)}</strong> has failures.</p>
|
||||
<p>See <a href="{html.escape(link)}">{html.escape(link)}</a> for the full console log,
|
||||
error index, and sticky status bar.</p>
|
||||
<p style="color:#666;font-size:12px">generated {html.escape(generated_at)} · source {c}</p>
|
||||
</div>
|
||||
{STICKY_JS}
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--log", required=True, type=Path)
|
||||
ap.add_argument("--out", required=True, type=Path)
|
||||
ap.add_argument("--hostname", required=True)
|
||||
ap.add_argument("--mode", choices=("ok", "err", "redirect"), required=True)
|
||||
ap.add_argument("--commit", default="")
|
||||
ap.add_argument("--commit-url", default="")
|
||||
ap.add_argument("--suite-path", default=".")
|
||||
ap.add_argument("--title", default="")
|
||||
ap.add_argument("--link-other", default="")
|
||||
ap.add_argument(
|
||||
"--path-err",
|
||||
default="/monitoring_err/",
|
||||
help="URL path for error page (redirect target)",
|
||||
)
|
||||
ap.add_argument(
|
||||
"--page-label",
|
||||
default="monitoring",
|
||||
help="Short name shown in titles/redirect box",
|
||||
)
|
||||
args = ap.parse_args()
|
||||
|
||||
log_text = args.log.read_text(errors="replace") if args.log.is_file() else ""
|
||||
now = datetime.now(timezone.utc)
|
||||
generated_iso = now.strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
generated_at = now.strftime("%Y-%m-%d %H:%M:%SZ")
|
||||
title = args.title or f"{args.page_label} {args.mode} · {args.hostname}"
|
||||
|
||||
raw_for_counts = [strip_ansi(l).rstrip("\n") for l in log_text.splitlines()]
|
||||
n_err, n_warn, _, _ = count_status(raw_for_counts)
|
||||
|
||||
if args.mode == "redirect":
|
||||
html_out = build_redirect_html(
|
||||
hostname=args.hostname,
|
||||
commit=args.commit,
|
||||
commit_url=args.commit_url,
|
||||
path_err=args.path_err,
|
||||
page_label=args.page_label,
|
||||
generated_at=generated_at,
|
||||
generated_iso=generated_iso,
|
||||
n_err=n_err,
|
||||
n_warn=n_warn,
|
||||
log_text=log_text,
|
||||
)
|
||||
else:
|
||||
html_out = build_html(
|
||||
title=title,
|
||||
hostname=args.hostname,
|
||||
mode=args.mode,
|
||||
log_text=log_text,
|
||||
commit=args.commit,
|
||||
commit_url=args.commit_url,
|
||||
suite_path=args.suite_path,
|
||||
generated_at=generated_at,
|
||||
generated_iso=generated_iso,
|
||||
link_other=args.link_other or None,
|
||||
page_label=args.page_label,
|
||||
)
|
||||
|
||||
args.out.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.out.write_text(html_out, encoding="utf-8")
|
||||
print(f"wrote {args.out} (errors={n_err} warnings={n_warn} level={status_level(n_err, n_warn)})")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
44
site-gen/deploy-monitoring-sites.sh
Executable file
44
site-gen/deploy-monitoring-sites.sh
Executable file
|
|
@ -0,0 +1,44 @@
|
|||
#!/usr/bin/env bash
|
||||
# deploy-monitoring-sites.sh — rsync generated HTML to DEPLOY_WWW_ROOT on koopa.
|
||||
# Does NOT reload Caddy (needs root — see ROOT-ON-KOOPA.md).
|
||||
set -uo pipefail
|
||||
|
||||
ROOT=$(cd "$(dirname "$0")" && pwd)
|
||||
SETTINGS="${SITE_GEN_SETTINGS:-$ROOT/settings.conf}"
|
||||
[ -f "$SETTINGS" ] || SETTINGS="$ROOT/settings.conf.example"
|
||||
# shellcheck disable=SC1090
|
||||
set -a
|
||||
source <(grep -v '^\s*#' "$SETTINGS" | grep -v '^\s*$' | sed 's/\r$//')
|
||||
set +a
|
||||
|
||||
WORK_ROOT="${WORK_ROOT:-/tmp/taler-monitoring-sites-work}"
|
||||
HTML_DIR="$WORK_ROOT/html"
|
||||
DEPLOY_SSH="${DEPLOY_SSH:-koopa-external}"
|
||||
DEPLOY_WWW_ROOT="${DEPLOY_WWW_ROOT:-/var/www/monitoring-sites}"
|
||||
|
||||
if [ ! -d "$HTML_DIR" ]; then
|
||||
echo "no HTML_DIR $HTML_DIR — run generate-monitoring-sites.sh first" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "deploy $HTML_DIR → ${DEPLOY_SSH}:${DEPLOY_WWW_ROOT}/"
|
||||
# Prefer writing to a hernani-owned staging dir if /var/www not writable
|
||||
STAGE_REMOTE="${DEPLOY_STAGING:-/home/hernani/monitoring-sites-staging}"
|
||||
|
||||
ssh -o BatchMode=yes -o ConnectTimeout=20 "$DEPLOY_SSH" \
|
||||
"mkdir -p '$STAGE_REMOTE' '$DEPLOY_WWW_ROOT' 2>/dev/null || mkdir -p '$STAGE_REMOTE'"
|
||||
|
||||
rsync -az --delete "$HTML_DIR/" "${DEPLOY_SSH}:${STAGE_REMOTE}/"
|
||||
echo "staged on $DEPLOY_SSH:$STAGE_REMOTE"
|
||||
echo
|
||||
echo "If $DEPLOY_WWW_ROOT is root-owned, as root on koopa:"
|
||||
echo " mkdir -p $DEPLOY_WWW_ROOT"
|
||||
echo " rsync -a --delete $STAGE_REMOTE/ $DEPLOY_WWW_ROOT/"
|
||||
echo " chown -R caddy:caddy $DEPLOY_WWW_ROOT # or root:caddy — match other www"
|
||||
echo " # then Caddyfile handles + reload — see ROOT-ON-KOOPA.md"
|
||||
echo
|
||||
# try direct rsync to DEPLOY_WWW_ROOT if writable
|
||||
if ssh -o BatchMode=yes "$DEPLOY_SSH" "test -w '$DEPLOY_WWW_ROOT' 2>/dev/null || test -w \"\$(dirname '$DEPLOY_WWW_ROOT')\""; then
|
||||
rsync -az --delete "$HTML_DIR/" "${DEPLOY_SSH}:${DEPLOY_WWW_ROOT}/" && \
|
||||
echo "also synced directly to $DEPLOY_WWW_ROOT"
|
||||
fi
|
||||
273
site-gen/generate-monitoring-sites.sh
Executable file
273
site-gen/generate-monitoring-sites.sh
Executable file
|
|
@ -0,0 +1,273 @@
|
|||
#!/usr/bin/env bash
|
||||
# generate-monitoring-sites.sh — run taler-monitoring, build console HTML for
|
||||
# /monitoring and /monitoring_err (static files for host Caddy later).
|
||||
#
|
||||
# Usage:
|
||||
# ./generate-monitoring-sites.sh # all SITES (or ONLY_HOSTS)
|
||||
# ONLY_HOSTS="bank.hacktivism.ch taler.hacktivism.ch" ./generate-monitoring-sites.sh
|
||||
# ./generate-monitoring-sites.sh --dry-run # HTML from empty/missing logs only
|
||||
# ./generate-monitoring-sites.sh --skip-run # only convert existing logs
|
||||
#
|
||||
# Settings: settings.conf (from settings.conf.example). No secrets in settings.
|
||||
# Secrets for e2e: ../secrets.env (optional; default phases are urls versions).
|
||||
#
|
||||
set -uo pipefail
|
||||
|
||||
export PYTHONUNBUFFERED=1
|
||||
# Global reporting defaults (same as host-agent run-host-report.sh)
|
||||
: "${RUN_TIMEOUT:=600}"
|
||||
export RUN_TIMEOUT
|
||||
|
||||
ROOT=$(cd "$(dirname "$0")" && pwd)
|
||||
SETTINGS="${SITE_GEN_SETTINGS:-$ROOT/settings.conf}"
|
||||
EXAMPLE="$ROOT/settings.conf.example"
|
||||
|
||||
if [ ! -f "$SETTINGS" ]; then
|
||||
if [ -f "$EXAMPLE" ]; then
|
||||
echo "note: no $SETTINGS — using example defaults (copy to settings.conf to customize)"
|
||||
SETTINGS="$EXAMPLE"
|
||||
else
|
||||
echo "missing settings: $EXAMPLE" >&2
|
||||
exit 2
|
||||
fi
|
||||
fi
|
||||
|
||||
# Load KEY=value lines; allow SITES already set in environment to win.
|
||||
# Multiline SITES in the conf file: use SITES_FILE= or env SITES= instead.
|
||||
while IFS= read -r _line || [ -n "$_line" ]; do
|
||||
_line=${_line%%#*}
|
||||
_line=$(echo "$_line" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')
|
||||
[ -z "$_line" ] && continue
|
||||
case "$_line" in
|
||||
SITES=*|SITES_FILE=*) continue ;; # handled below / env
|
||||
*=*)
|
||||
_k=${_line%%=*}
|
||||
_v=${_line#*=}
|
||||
# do not override pre-set env
|
||||
if [ -z "${!_k+x}" ] 2>/dev/null || [ -z "${!_k}" ]; then
|
||||
export "$_k=$_v"
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
done < "$SETTINGS"
|
||||
|
||||
# SITES from dedicated file or keep env; else parse heredoc-style from example list
|
||||
if [ -n "${SITES_FILE:-}" ] && [ -f "$SITES_FILE" ]; then
|
||||
SITES=$(grep -v '^\s*#' "$SITES_FILE" | grep -v '^\s*$')
|
||||
export SITES
|
||||
elif [ -z "${SITES:-}" ]; then
|
||||
# default 9 fronts
|
||||
SITES="bank.hacktivism.ch|hacktivism.ch
|
||||
exchange.hacktivism.ch|hacktivism.ch
|
||||
taler.hacktivism.ch|hacktivism.ch
|
||||
stage.bank.lefrancpaysan.ch|stage.lefrancpaysan.ch
|
||||
stage.exchange.lefrancpaysan.ch|stage.lefrancpaysan.ch
|
||||
stage.monnaie.lefrancpaysan.ch|stage.lefrancpaysan.ch
|
||||
bank.lefrancpaysan.ch|lefrancpaysan.ch
|
||||
exchange.lefrancpaysan.ch|lefrancpaysan.ch
|
||||
monnaie.lefrancpaysan.ch|lefrancpaysan.ch"
|
||||
export SITES
|
||||
fi
|
||||
|
||||
SKIP_RUN=0
|
||||
DRY=0
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--skip-run) SKIP_RUN=1; shift ;;
|
||||
--dry-run) DRY=1; SKIP_RUN=1; shift ;;
|
||||
-h|--help) sed -n '1,20p' "$0"; exit 0 ;;
|
||||
*) echo "unknown arg: $1" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
MONITORING_ROOT=$(cd "$ROOT/${MONITORING_ROOT:-..}" && pwd)
|
||||
WORK_ROOT="${WORK_ROOT:-/tmp/taler-monitoring-sites-work}"
|
||||
LOG_DIR="$WORK_ROOT/logs"
|
||||
HTML_DIR="$WORK_ROOT/html"
|
||||
mkdir -p "$LOG_DIR" "$HTML_DIR"
|
||||
|
||||
# commit of monitoring suite used for this run
|
||||
if git -C "$MONITORING_ROOT/../.." rev-parse HEAD >/dev/null 2>&1; then
|
||||
REPO_ROOT=$(cd "$MONITORING_ROOT/../.." && pwd)
|
||||
elif git -C "$MONITORING_ROOT" rev-parse HEAD >/dev/null 2>&1; then
|
||||
REPO_ROOT=$(cd "$MONITORING_ROOT" && pwd)
|
||||
else
|
||||
REPO_ROOT=$(cd "$ROOT/../../.." && pwd)
|
||||
fi
|
||||
COMMIT=$(git -C "$REPO_ROOT" rev-parse HEAD 2>/dev/null || echo unknown)
|
||||
COMMIT_SHORT=$(git -C "$REPO_ROOT" rev-parse --short=12 HEAD 2>/dev/null || echo unknown)
|
||||
SOURCE_REPO_WEB="${SOURCE_REPO_WEB:-https://git.hacktivism.ch/hernani/taler-monitoring}"
|
||||
TMPL="${SOURCE_COMMIT_URL_TMPL:-\{repo\}/src/commit/\{commit\}}"
|
||||
COMMIT_URL=$(printf '%s' "$TMPL" | sed "s|{repo}|$SOURCE_REPO_WEB|g; s|{commit}|$COMMIT|g")
|
||||
SUITE_PATH="${SOURCE_SUITE_PATH:-.}"
|
||||
|
||||
# Outside-only public checks (no SSH into koopa/stage containers)
|
||||
PHASES_ERR="${PHASES_ERR:-urls versions}"
|
||||
PHASES_OK="${PHASES_OK:-urls versions}"
|
||||
CONTINUE_ON_ERROR="${CONTINUE_ON_ERROR:-1}"
|
||||
SKIP_SSH="${SKIP_SSH:-1}"
|
||||
# Drop SSH phases if someone passes "all" / inside / server / e2e by mistake
|
||||
filter_phases_outside() {
|
||||
local out=() p
|
||||
for p in "$@"; do
|
||||
case "$p" in
|
||||
inside|server|e2e|ladder|goa-ladder|auth401) continue ;;
|
||||
all) out+=(urls versions);;
|
||||
*) out+=("$p") ;;
|
||||
esac
|
||||
done
|
||||
printf '%s\n' "${out[@]}"
|
||||
}
|
||||
|
||||
run_mon() {
|
||||
local domain="$1" phases="$2" logfile="$3" cont="$4"
|
||||
# shellcheck disable=SC2206
|
||||
local ph=( $phases )
|
||||
mapfile -t ph < <(filter_phases_outside "${ph[@]}")
|
||||
[ "${#ph[@]}" -gt 0 ] || ph=(urls versions)
|
||||
|
||||
: >"$logfile"
|
||||
echo "+ -d $domain · phases=${ph[*]} · CONTINUE=$cont · SKIP_SSH=$SKIP_SSH · RUN_TIMEOUT=$RUN_TIMEOUT"
|
||||
|
||||
if [ -n "${RUNNER_SSH:-}" ]; then
|
||||
echo "+ via ssh $RUNNER_SSH"
|
||||
rsync -az --delete \
|
||||
--exclude secrets.env \
|
||||
--exclude 'android-test/artifacts' \
|
||||
--exclude '.git' \
|
||||
"$MONITORING_ROOT/" \
|
||||
"${RUNNER_SSH}:${RUNNER_REMOTE_WORKDIR}/" \
|
||||
|| return 1
|
||||
# shellcheck disable=SC2029
|
||||
if command -v stdbuf >/dev/null 2>&1; then
|
||||
ssh -o BatchMode=yes -o ConnectTimeout=30 "$RUNNER_SSH" \
|
||||
"cd '${RUNNER_REMOTE_WORKDIR}' && chmod +x taler-monitoring.sh check_*.sh 2>/dev/null; \
|
||||
export CONTINUE_ON_ERROR='$cont' AUTH401_CONTINUE='$cont' SKIP_SSH='$SKIP_SSH' RUN_TIMEOUT='$RUN_TIMEOUT'; \
|
||||
stdbuf -oL -eL ./taler-monitoring.sh -d '$domain' ${ph[*]}" \
|
||||
2>&1 | stdbuf -oL -eL tee -a "$logfile"
|
||||
else
|
||||
ssh -o BatchMode=yes -o ConnectTimeout=30 "$RUNNER_SSH" \
|
||||
"cd '${RUNNER_REMOTE_WORKDIR}' && chmod +x taler-monitoring.sh check_*.sh 2>/dev/null; \
|
||||
export CONTINUE_ON_ERROR='$cont' AUTH401_CONTINUE='$cont' SKIP_SSH='$SKIP_SSH' RUN_TIMEOUT='$RUN_TIMEOUT'; \
|
||||
./taler-monitoring.sh -d '$domain' ${ph[*]}" \
|
||||
2>&1 | tee -a "$logfile"
|
||||
fi
|
||||
return "${PIPESTATUS[0]}"
|
||||
fi
|
||||
|
||||
if command -v stdbuf >/dev/null 2>&1; then
|
||||
( cd "$MONITORING_ROOT" && \
|
||||
CONTINUE_ON_ERROR="$cont" AUTH401_CONTINUE="$cont" SKIP_SSH="$SKIP_SSH" \
|
||||
RUN_TIMEOUT="$RUN_TIMEOUT" \
|
||||
stdbuf -oL -eL ./taler-monitoring.sh -d "$domain" "${ph[@]}" ) \
|
||||
2>&1 | stdbuf -oL -eL tee -a "$logfile"
|
||||
else
|
||||
( cd "$MONITORING_ROOT" && \
|
||||
CONTINUE_ON_ERROR="$cont" AUTH401_CONTINUE="$cont" SKIP_SSH="$SKIP_SSH" \
|
||||
RUN_TIMEOUT="$RUN_TIMEOUT" \
|
||||
./taler-monitoring.sh -d "$domain" "${ph[@]}" ) \
|
||||
2>&1 | tee -a "$logfile"
|
||||
fi
|
||||
return "${PIPESTATUS[0]}"
|
||||
}
|
||||
|
||||
htmlify() {
|
||||
local host="$1" mode="$2" log="$3" out="$4" other="$5"
|
||||
mkdir -p "$(dirname "$out")"
|
||||
if [ ! -f "$ROOT/console_to_html.py" ]; then
|
||||
echo "WARN: console_to_html.py missing — bootstrap $out"
|
||||
{
|
||||
echo "<!DOCTYPE html><html><head><meta charset=utf-8><title>$host $mode</title></head>"
|
||||
echo "<body><h1>$host · $mode</h1><p>commit <a href=\"$COMMIT_URL\">$COMMIT_SHORT</a></p>"
|
||||
echo "<pre>"
|
||||
tail -n 100 "$log" 2>/dev/null | sed 's/&/\&/g;s/</\</g' || true
|
||||
echo "</pre></body></html>"
|
||||
} >"$out"
|
||||
return 0
|
||||
fi
|
||||
python3 "$ROOT/console_to_html.py" \
|
||||
--log "$log" \
|
||||
--out "$out" \
|
||||
--hostname "$host" \
|
||||
--mode "$mode" \
|
||||
--commit "$COMMIT" \
|
||||
--commit-url "$COMMIT_URL" \
|
||||
--suite-path "$SUITE_PATH" \
|
||||
--link-other "$other"
|
||||
}
|
||||
|
||||
# Parse SITES
|
||||
mapfile -t SITE_LINES < <(printf '%s\n' "$SITES" | sed '/^\s*$/d' | sed 's/#.*//')
|
||||
ONLY="${ONLY_HOSTS:-}"
|
||||
|
||||
for line in "${SITE_LINES[@]}"; do
|
||||
line=$(echo "$line" | tr -d ' \t')
|
||||
[ -z "$line" ] && continue
|
||||
host=${line%%|*}
|
||||
domain=${line#*|}
|
||||
if [ -n "$ONLY" ]; then
|
||||
echo " $ONLY " | grep -q " $host " || continue
|
||||
fi
|
||||
|
||||
echo "======== $host (domain=$domain) commit=$COMMIT_SHORT ========"
|
||||
base_html="$HTML_DIR/$host"
|
||||
mkdir -p "$base_html/monitoring" "$base_html/monitoring_err"
|
||||
log_err="$LOG_DIR/${host}.err.log"
|
||||
log_ok="$LOG_DIR/${host}.ok.log"
|
||||
ec_err=1
|
||||
ec_ok=1
|
||||
|
||||
if [ "$SKIP_RUN" != "1" ]; then
|
||||
run_mon "$domain" "$PHASES_ERR" "$log_err" "$CONTINUE_ON_ERROR"
|
||||
ec_err=$?
|
||||
# ok-run without continue (strict)
|
||||
run_mon "$domain" "$PHASES_OK" "$log_ok" "0"
|
||||
ec_ok=$?
|
||||
else
|
||||
[ -f "$log_err" ] || : >"$log_err"
|
||||
[ -f "$log_ok" ] || : >"$log_ok"
|
||||
# infer exit from log if present
|
||||
if grep -q '\[ ERROR \]' "$log_err" 2>/dev/null || grep -q 'ERROR' "$log_err" 2>/dev/null; then
|
||||
ec_err=1
|
||||
else
|
||||
ec_err=0
|
||||
fi
|
||||
ec_ok=$ec_err
|
||||
fi
|
||||
|
||||
# Always write pages (first run / empty logs too)
|
||||
[ -f "$log_err" ] || : >"$log_err"
|
||||
[ -f "$log_ok" ] || : >"$log_ok"
|
||||
|
||||
htmlify "$host" "err" "$log_err" \
|
||||
"$base_html/monitoring_err/index.html" "/monitoring/"
|
||||
|
||||
if [ "$ec_ok" -eq 0 ] && [ "$ec_err" -eq 0 ]; then
|
||||
htmlify "$host" "ok" "$log_ok" \
|
||||
"$base_html/monitoring/index.html" "/monitoring_err/"
|
||||
echo " → /monitoring (clean) + /monitoring_err"
|
||||
else
|
||||
htmlify "$host" "redirect" "$log_err" \
|
||||
"$base_html/monitoring/index.html" "/monitoring_err/"
|
||||
echo " → /monitoring (redirect stub) + /monitoring_err (errors) ec_err=$ec_err ec_ok=$ec_ok"
|
||||
fi
|
||||
|
||||
# first-run safety
|
||||
if [ ! -f "$base_html/monitoring/index.html" ]; then
|
||||
htmlify "$host" "err" "$log_err" \
|
||||
"$base_html/monitoring/index.html" "/monitoring_err/"
|
||||
fi
|
||||
|
||||
printf '%s\n' "$COMMIT" >"$base_html/COMMIT"
|
||||
printf '%s\n' "$COMMIT_URL" >"$base_html/COMMIT_URL"
|
||||
printf 'host=%s domain=%s ec_err=%s ec_ok=%s commit=%s run_timeout=%s\n' \
|
||||
"$host" "$domain" "$ec_err" "$ec_ok" "$COMMIT" "$RUN_TIMEOUT" | tee "$base_html/STATUS.txt"
|
||||
done
|
||||
|
||||
echo
|
||||
echo "HTML under $HTML_DIR"
|
||||
echo "commit $COMMIT_SHORT $COMMIT_URL"
|
||||
echo "RUN_TIMEOUT=${RUN_TIMEOUT}s (global; 0=unlimited)"
|
||||
echo "Deploy (as hernani, may need sudo for /var/www):"
|
||||
echo " ./deploy-monitoring-sites.sh"
|
||||
echo "Root on koopa: see ROOT-ON-KOOPA.md"
|
||||
79
site-gen/install-firecuda-timer.sh
Executable file
79
site-gen/install-firecuda-timer.sh
Executable file
|
|
@ -0,0 +1,79 @@
|
|||
#!/usr/bin/env bash
|
||||
# install-firecuda-timer.sh — OPTIONAL: rsync suite to firecuda + launchd every 4h
|
||||
#
|
||||
# POLICY (2026-07): firecuda launchd is **disabled**. Prefer koopa host-agent.
|
||||
# This script remains for documentation / future re-enable only.
|
||||
# Do not run unless you intentionally want a second outside-only scheduler.
|
||||
#
|
||||
# ./install-firecuda-timer.sh
|
||||
# ./install-firecuda-timer.sh --run-now
|
||||
#
|
||||
set -euo pipefail
|
||||
|
||||
ROOT=$(cd "$(dirname "$0")" && pwd)
|
||||
MON_ROOT=$(cd "$ROOT/.." && pwd)
|
||||
FIRECUDA="${FIRECUDA_SSH:-firecuda-external}"
|
||||
REMOTE_BASE="${FIRECUDA_INSTALL_DIR:-taler-monitoring-site-gen}"
|
||||
RUN_NOW=0
|
||||
[ "${1:-}" = "--run-now" ] && RUN_NOW=1
|
||||
|
||||
echo "install → $FIRECUDA:~/$REMOTE_BASE (outside-only, every 4h)"
|
||||
|
||||
ssh -o BatchMode=yes -o ConnectTimeout=20 "$FIRECUDA" \
|
||||
"mkdir -p \"\$HOME/$REMOTE_BASE\" \"\$HOME/Library/Logs/taler-monitoring-sites\" \"\$HOME/var/taler-monitoring-sites-work\" \"\$HOME/Library/LaunchAgents\""
|
||||
|
||||
# Sync full taler-monitoring suite (no secrets.env)
|
||||
rsync -az --delete \
|
||||
--exclude secrets.env \
|
||||
--exclude 'android-test/artifacts' \
|
||||
--exclude '.git' \
|
||||
--exclude 'site-gen/settings.conf' \
|
||||
"$MON_ROOT/" \
|
||||
"${FIRECUDA}:$REMOTE_BASE/"
|
||||
|
||||
# settings on firecuda: outside only, run locally on firecuda, deploy to koopa
|
||||
ssh -o BatchMode=yes "$FIRECUDA" "cat > \"\$HOME/$REMOTE_BASE/site-gen/settings.conf\" <<'EOF'
|
||||
# auto-installed by install-firecuda-timer.sh — no secrets
|
||||
RUNNER_SSH=
|
||||
SKIP_SSH=1
|
||||
CONTINUE_ON_ERROR=1
|
||||
PHASES_ERR=urls versions
|
||||
PHASES_OK=urls versions
|
||||
DEPLOY_SSH=koopa-external
|
||||
DEPLOY_STAGING=/home/hernani/monitoring-sites-staging
|
||||
WORK_ROOT=
|
||||
MONITORING_ROOT=..
|
||||
SOURCE_REPO_WEB=https://git.hacktivism.ch/hernani/taler-monitoring
|
||||
SOURCE_COMMIT_URL_TMPL={repo}/src/commit/{commit}
|
||||
SOURCE_SUITE_PATH=.
|
||||
EOF
|
||||
# WORK_ROOT expanded on remote
|
||||
sed -i.bak \"s|^WORK_ROOT=.*|WORK_ROOT=\$HOME/var/taler-monitoring-sites-work|\" \"\$HOME/$REMOTE_BASE/site-gen/settings.conf\" 2>/dev/null || \
|
||||
sed -i '' \"s|^WORK_ROOT=.*|WORK_ROOT=\$HOME/var/taler-monitoring-sites-work|\" \"\$HOME/$REMOTE_BASE/site-gen/settings.conf\"
|
||||
"
|
||||
|
||||
# launchd plist with absolute paths
|
||||
ssh -o BatchMode=yes "$FIRECUDA" bash -s <<REMOTE
|
||||
set -euo pipefail
|
||||
HOME_DIR=\$HOME
|
||||
SITE_GEN="\$HOME_DIR/$REMOTE_BASE/site-gen"
|
||||
PLIST_SRC="\$SITE_GEN/com.hacktivism.taler-monitoring-sites.plist"
|
||||
PLIST_DST="\$HOME_DIR/Library/LaunchAgents/com.hacktivism.taler-monitoring-sites.plist"
|
||||
sed -e "s|__SITE_GEN__|\${SITE_GEN}|g" -e "s|__HOME__|\${HOME_DIR}|g" "\$PLIST_SRC" > "\$PLIST_DST"
|
||||
chmod +x "\$SITE_GEN"/*.sh "\$SITE_GEN"/console_to_html.py 2>/dev/null || true
|
||||
chmod +x "\$SITE_GEN/run-on-firecuda.sh"
|
||||
launchctl bootout "gui/\$(id -u)/com.hacktivism.taler-monitoring-sites" 2>/dev/null || true
|
||||
launchctl unload "\$PLIST_DST" 2>/dev/null || true
|
||||
launchctl bootstrap "gui/\$(id -u)" "\$PLIST_DST" 2>/dev/null || launchctl load "\$PLIST_DST"
|
||||
launchctl enable "gui/\$(id -u)/com.hacktivism.taler-monitoring-sites" 2>/dev/null || true
|
||||
launchctl print "gui/\$(id -u)/com.hacktivism.taler-monitoring-sites" 2>/dev/null | head -20 || launchctl list | grep taler-monitoring || true
|
||||
echo "OK: launchd com.hacktivism.taler-monitoring-sites (StartInterval=14400)"
|
||||
REMOTE
|
||||
|
||||
if [ "$RUN_NOW" = "1" ]; then
|
||||
echo "run once now…"
|
||||
ssh -o BatchMode=yes "$FIRECUDA" "\"\$HOME/$REMOTE_BASE/site-gen/run-on-firecuda.sh\"" || true
|
||||
fi
|
||||
|
||||
echo "done. logs: $FIRECUDA:~/Library/Logs/taler-monitoring-sites/"
|
||||
echo "manual: ssh $FIRECUDA '~/taler-monitoring-site-gen/site-gen/run-on-firecuda.sh'"
|
||||
15
site-gen/pull-firecuda-to-koopa.sh
Executable file
15
site-gen/pull-firecuda-to-koopa.sh
Executable file
|
|
@ -0,0 +1,15 @@
|
|||
#!/usr/bin/env bash
|
||||
# pull-firecuda-to-koopa.sh — copy HTML from firecuda work dir → koopa staging
|
||||
# Run on laptop (hernani) where both SSH aliases work.
|
||||
set -euo pipefail
|
||||
FIRECUDA="${FIRECUDA_SSH:-firecuda-external}"
|
||||
KOOPA="${DEPLOY_SSH:-koopa-external}"
|
||||
REMOTE_HTML="${FIRECUDA_HTML:-var/taler-monitoring-sites-work/html/}"
|
||||
STAGE="${DEPLOY_STAGING:-/home/hernani/monitoring-sites-staging}"
|
||||
|
||||
echo "firecuda:$REMOTE_HTML → $KOOPA:$STAGE"
|
||||
ssh -o BatchMode=yes -o ConnectTimeout=20 "$KOOPA" "mkdir -p '$STAGE'"
|
||||
rsync -az --delete \
|
||||
"${FIRECUDA}:${REMOTE_HTML}" \
|
||||
"${KOOPA}:${STAGE}/"
|
||||
echo "OK staged. Root: rsync to /var/www/monitoring-sites (see ROOT-ON-KOOPA.md)"
|
||||
59
site-gen/run-on-firecuda.sh
Executable file
59
site-gen/run-on-firecuda.sh
Executable file
|
|
@ -0,0 +1,59 @@
|
|||
#!/usr/bin/env bash
|
||||
# run-on-firecuda.sh — entrypoint for launchd/cron on firecuda-external.
|
||||
# Outside-only monitoring (SKIP_SSH=1), all 9 sites, then stage HTML to koopa.
|
||||
#
|
||||
# Installed path on firecuda (default):
|
||||
# ~/taler-monitoring-site-gen/site-gen/run-on-firecuda.sh
|
||||
#
|
||||
set -uo pipefail
|
||||
|
||||
export PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:${PATH:-}"
|
||||
|
||||
ROOT=$(cd "$(dirname "$0")" && pwd)
|
||||
LOG_DIR="${FIRECUDA_LOG_DIR:-$HOME/Library/Logs/taler-monitoring-sites}"
|
||||
mkdir -p "$LOG_DIR"
|
||||
STAMP=$(date +%Y%m%d-%H%M%S)
|
||||
LOG="$LOG_DIR/run-${STAMP}.log"
|
||||
exec >>"$LOG" 2>&1
|
||||
|
||||
echo "======== $(date -Iseconds 2>/dev/null || date) firecuda monitoring sites ========"
|
||||
|
||||
# Always: public DNS only, no SSH checks into stacks
|
||||
export SKIP_SSH=1
|
||||
export CONTINUE_ON_ERROR=1
|
||||
export AUTH401_CONTINUE=1
|
||||
export RUNNER_SSH=
|
||||
export PHASES_ERR="${PHASES_ERR:-urls versions}"
|
||||
export PHASES_OK="${PHASES_OK:-urls versions}"
|
||||
export WORK_ROOT="${WORK_ROOT:-$HOME/var/taler-monitoring-sites-work}"
|
||||
export MONITORING_ROOT="${MONITORING_ROOT:-$ROOT/..}"
|
||||
export DEPLOY_SSH="${DEPLOY_SSH:-koopa-external}"
|
||||
export DEPLOY_STAGING="${DEPLOY_STAGING:-/home/hernani/monitoring-sites-staging}"
|
||||
|
||||
# Prefer full 9-site list from settings or defaults inside generate script
|
||||
unset ONLY_HOSTS
|
||||
|
||||
cd "$ROOT" || exit 1
|
||||
chmod +x generate-monitoring-sites.sh deploy-monitoring-sites.sh console_to_html.py 2>/dev/null || true
|
||||
|
||||
./generate-monitoring-sites.sh
|
||||
ec=$?
|
||||
|
||||
# HTML stays on firecuda under WORK_ROOT/html.
|
||||
# Optional push to koopa if this host can SSH there (often only from laptop):
|
||||
if [ -n "${DEPLOY_SSH:-}" ] && [ "${DEPLOY_FROM_FIRECUDA:-0}" = "1" ]; then
|
||||
if [ -x ./deploy-monitoring-sites.sh ]; then
|
||||
./deploy-monitoring-sites.sh || echo "WARN: deploy staging failed"
|
||||
fi
|
||||
else
|
||||
echo "note: HTML at ${WORK_ROOT}/html — pull from laptop:"
|
||||
echo " rsync -az firecuda-external:var/taler-monitoring-sites-work/html/ \\"
|
||||
echo " koopa-external:monitoring-sites-staging/"
|
||||
echo " # or: ./pull-firecuda-to-koopa.sh"
|
||||
fi
|
||||
|
||||
# keep last 20 logs
|
||||
ls -1t "$LOG_DIR"/run-*.log 2>/dev/null | tail -n +21 | xargs rm -f 2>/dev/null || true
|
||||
|
||||
echo "======== done ec=$ec ========"
|
||||
exit "$ec"
|
||||
55
site-gen/settings.conf.example
Normal file
55
site-gen/settings.conf.example
Normal file
|
|
@ -0,0 +1,55 @@
|
|||
# monitoring site-gen — non-secret settings (safe as example)
|
||||
# Copy: cp settings.conf.example settings.conf
|
||||
# Never put passwords/tokens here.
|
||||
|
||||
# On firecuda the suite runs *locally* (RUNNER_SSH empty). From a laptop you
|
||||
# can set RUNNER_SSH=firecuda-external to drive a one-off remote run instead.
|
||||
RUNNER_SSH=
|
||||
RUNNER_SSH_FALLBACKS=
|
||||
RUNNER_REMOTE_WORKDIR=/tmp/taler-monitoring-site-gen
|
||||
MONITORING_ROOT=..
|
||||
|
||||
# Outside-only: never SSH into bank/exchange/merchant containers
|
||||
SKIP_SSH=1
|
||||
|
||||
# Deploy staging on koopa (host Caddy later — root copies to /var/www)
|
||||
DEPLOY_SSH=koopa-external
|
||||
DEPLOY_SSH_FALLBACKS=koopa
|
||||
# Static files for Caddy file_server (created per hostname)
|
||||
DEPLOY_WWW_ROOT=/var/www/monitoring-sites
|
||||
DEPLOY_SUDO=sudo
|
||||
CADDY_CONFIG=/etc/caddy/Caddyfile
|
||||
CADDY_MIRROR_REL=configs/caddy/Caddyfile
|
||||
# Working tree on deploy host for apply (optional)
|
||||
KOOPA_CADDY_DIR=/home/hernani/koopa-caddy
|
||||
|
||||
# Source link for HTML footer (Forgejo)
|
||||
SOURCE_REPO_WEB=https://git.hacktivism.ch/hernani/taler-monitoring
|
||||
SOURCE_COMMIT_URL_TMPL={repo}/src/commit/{commit}
|
||||
SOURCE_SUITE_PATH=.
|
||||
|
||||
CONTINUE_ON_ERROR=1
|
||||
PHASES_ERR=urls versions
|
||||
PHASES_OK=urls versions
|
||||
SKIP_E2E=1
|
||||
|
||||
# Local work dir for logs + HTML
|
||||
WORK_ROOT=/tmp/taler-monitoring-sites-work
|
||||
|
||||
# Sites: hostname|domain_profile (9 fronts)
|
||||
# Start with hacktivism; stage + prod LFP follow
|
||||
SITES="
|
||||
bank.hacktivism.ch|hacktivism.ch
|
||||
exchange.hacktivism.ch|hacktivism.ch
|
||||
taler.hacktivism.ch|hacktivism.ch
|
||||
stage.bank.lefrancpaysan.ch|stage.lefrancpaysan.ch
|
||||
stage.exchange.lefrancpaysan.ch|stage.lefrancpaysan.ch
|
||||
stage.monnaie.lefrancpaysan.ch|stage.lefrancpaysan.ch
|
||||
bank.lefrancpaysan.ch|lefrancpaysan.ch
|
||||
exchange.lefrancpaysan.ch|lefrancpaysan.ch
|
||||
monnaie.lefrancpaysan.ch|lefrancpaysan.ch
|
||||
"
|
||||
|
||||
# Only process this subset (space-separated hostnames). Empty = all SITES.
|
||||
# Example for first deploy: ONLY_HOSTS=bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch
|
||||
ONLY_HOSTS=
|
||||
Loading…
Add table
Add a link
Reference in a new issue