monitoring: site-gen console HTML and sticky status bar
This commit is contained in:
Hernâni Marques
parent
6436ab0dd9
commit
46efd8a51c
11 changed files with 1724 additions and 0 deletions
81
scripts/taler-monitoring/site-gen/README.md
Normal file
81
scripts/taler-monitoring/site-gen/README.md
Normal file
|
|
@ -0,0 +1,81 @@
|
||||||
|
# monitoring site-gen
|
||||||
|
|
||||||
|
Build **console-style HTML** from `taler-monitoring` runs and stage them for
|
||||||
|
**host Caddy** (`/monitoring`, `/monitoring_err`).
|
||||||
|
|
||||||
|
Same **global reporting defaults** as host-agents (`run-host-report.sh`):
|
||||||
|
|
||||||
|
- **`RUN_TIMEOUT=600`** on each suite run
|
||||||
|
- **line-buffered** logs (continuous write)
|
||||||
|
- **always** write HTML (first run / empty logs too)
|
||||||
|
- Forgejo **commit link** in the page header
|
||||||
|
|
||||||
|
Applies to all nine fronts (GOA + FP stage + FP prod) in the default `SITES` list.
|
||||||
|
|
||||||
|
| Host | Role |
|
||||||
|
|------|------|
|
||||||
|
| **firecuda-external** | optional runner (public DNS) |
|
||||||
|
| **koopa-external** | deploy staging + Caddy (root service) |
|
||||||
|
| **host-agents** | primary: GOA + FP on-box (`../host-agent/`) |
|
||||||
|
|
||||||
|
No secrets in `settings.conf` — only hosts, paths, site list.
|
||||||
|
|
||||||
|
## firecuda-external (outside-only) — **disabled for now**
|
||||||
|
|
||||||
|
**firecuda-external could** run public-only monitoring (`SKIP_SSH=1`, phases
|
||||||
|
`urls versions`) on a schedule (launchd every 4h) and write HTML under
|
||||||
|
`~/var/taler-monitoring-sites-work/html/`. Scripts for that still live here
|
||||||
|
(`install-firecuda-timer.sh`, `run-on-firecuda.sh`, plist template).
|
||||||
|
|
||||||
|
**Current policy: do not run a real launchd job on firecuda.**
|
||||||
|
Primary GOA monitoring is **koopa host-agent** (`host-agent/`, user `hernani`,
|
||||||
|
local `podman exec`). Re-enable firecuda only if you explicitly want a second,
|
||||||
|
outside-only view.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# optional one-shot (no timer):
|
||||||
|
# ssh firecuda-external '~/taler-monitoring-site-gen/site-gen/run-on-firecuda.sh'
|
||||||
|
|
||||||
|
# if a timer was installed earlier, ensure it is off:
|
||||||
|
# ssh firecuda-external 'launchctl list | grep taler-monitoring || echo off'
|
||||||
|
```
|
||||||
|
|
||||||
|
Disabled plists may remain as `*.plist.disabled-*` under `~/Library/LaunchAgents/`.
|
||||||
|
|
||||||
|
## Quick start (laptop)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd scripts/taler-monitoring/site-gen
|
||||||
|
cp settings.conf.example settings.conf # optional
|
||||||
|
chmod +x generate-monitoring-sites.sh deploy-monitoring-sites.sh console_to_html.py
|
||||||
|
|
||||||
|
ONLY_HOSTS="bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch" \
|
||||||
|
SKIP_SSH=1 RUNNER_SSH= \
|
||||||
|
./generate-monitoring-sites.sh
|
||||||
|
|
||||||
|
./deploy-monitoring-sites.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Then on koopa **as root**: see **[ROOT-ON-KOOPA.md](./ROOT-ON-KOOPA.md)**.
|
||||||
|
|
||||||
|
## URL rules
|
||||||
|
|
||||||
|
| Path | When |
|
||||||
|
|------|------|
|
||||||
|
| `/monitoring_err/` | always: full console log + error list on top (CONTINUE_ON_ERROR run) |
|
||||||
|
| `/monitoring/` | **clean** full log if last strict run exit 0; else **stub** linking to `/monitoring_err/` |
|
||||||
|
|
||||||
|
Footer links **Forgejo commit** of the admin-log tree used for that run.
|
||||||
|
|
||||||
|
## Layout
|
||||||
|
|
||||||
|
```text
|
||||||
|
site-gen/
|
||||||
|
settings.conf.example
|
||||||
|
generate-monitoring-sites.sh
|
||||||
|
deploy-monitoring-sites.sh
|
||||||
|
console_to_html.py
|
||||||
|
caddy-monitoring-handles.snippet
|
||||||
|
ROOT-ON-KOOPA.md
|
||||||
|
README.md
|
||||||
|
```
|
||||||
92
scripts/taler-monitoring/site-gen/ROOT-ON-KOOPA.md
Normal file
92
scripts/taler-monitoring/site-gen/ROOT-ON-KOOPA.md
Normal file
|
|
@ -0,0 +1,92 @@
|
||||||
|
# Root on koopa (hacktivism monitoring sites)
|
||||||
|
|
||||||
|
Host Caddy runs as **root/systemd** (`caddy` user). Static monitoring HTML is
|
||||||
|
**not** put into Taler containers — only host paths + Caddy `handle`.
|
||||||
|
|
||||||
|
After `generate-monitoring-sites.sh` + `deploy-monitoring-sites.sh` (as hernani),
|
||||||
|
files live in:
|
||||||
|
|
||||||
|
```text
|
||||||
|
/home/hernani/monitoring-sites-staging/<hostname>/monitoring/index.html
|
||||||
|
/home/hernani/monitoring-sites-staging/<hostname>/monitoring_err/index.html
|
||||||
|
```
|
||||||
|
|
||||||
|
## One-shot as root (copy + Caddy)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# on koopa (koopa-external), as root:
|
||||||
|
|
||||||
|
# 1) web root
|
||||||
|
install -d -o caddy -g caddy -m 755 /var/www/monitoring-sites
|
||||||
|
rsync -a --delete /home/hernani/monitoring-sites-staging/ /var/www/monitoring-sites/
|
||||||
|
chown -R caddy:caddy /var/www/monitoring-sites
|
||||||
|
|
||||||
|
# 2) Caddyfile — merge handles from admin-log mirror, then:
|
||||||
|
# (either edit /etc/caddy/Caddyfile by hand using snippet below,
|
||||||
|
# or copy full mirror after review)
|
||||||
|
#
|
||||||
|
# install -m 644 /home/hernani/koopa-admin-log/configs/caddy/Caddyfile /etc/caddy/Caddyfile
|
||||||
|
# # or: ~/koopa-caddy/apply.sh after syncing Caddyfile into ~/koopa-caddy/
|
||||||
|
|
||||||
|
caddy validate --config /etc/caddy/Caddyfile
|
||||||
|
systemctl reload caddy
|
||||||
|
systemctl is-active caddy
|
||||||
|
```
|
||||||
|
|
||||||
|
## Snippet per hacktivism site (inside each `*.hacktivism.ch { }` block)
|
||||||
|
|
||||||
|
Place **before** the catch-all `reverse_proxy` (same idea as `/intro*`):
|
||||||
|
|
||||||
|
```caddy
|
||||||
|
# Public taler-monitoring console HTML (static; host only)
|
||||||
|
handle /monitoring_err* {
|
||||||
|
root * /var/www/monitoring-sites/{host}
|
||||||
|
rewrite * /monitoring_err{uri}
|
||||||
|
# uri is /monitoring_err or /monitoring_err/ → serve directory index
|
||||||
|
file_server
|
||||||
|
}
|
||||||
|
handle /monitoring* {
|
||||||
|
root * /var/www/monitoring-sites/{host}
|
||||||
|
file_server
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Simpler layout (recommended): path on disk matches URL under host root:
|
||||||
|
|
||||||
|
```text
|
||||||
|
/var/www/monitoring-sites/bank.hacktivism.ch/monitoring/index.html
|
||||||
|
/var/www/monitoring-sites/bank.hacktivism.ch/monitoring_err/index.html
|
||||||
|
```
|
||||||
|
|
||||||
|
```caddy
|
||||||
|
handle_path /monitoring_err/* {
|
||||||
|
root * /var/www/monitoring-sites/{host}/monitoring_err
|
||||||
|
file_server
|
||||||
|
}
|
||||||
|
handle /monitoring_err {
|
||||||
|
redir /monitoring_err/ 302
|
||||||
|
}
|
||||||
|
handle_path /monitoring/* {
|
||||||
|
root * /var/www/monitoring-sites/{host}/monitoring
|
||||||
|
file_server
|
||||||
|
}
|
||||||
|
handle /monitoring {
|
||||||
|
redir /monitoring/ 302
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Canonical snippet is also in `caddy-monitoring-handles.snippet` and applied
|
||||||
|
in `configs/caddy/Caddyfile` for the three GOA hosts.
|
||||||
|
|
||||||
|
## Check
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -sS -o /dev/null -w '%{http_code}\n' https://bank.hacktivism.ch/monitoring/
|
||||||
|
curl -sS -o /dev/null -w '%{http_code}\n' https://bank.hacktivism.ch/monitoring_err/
|
||||||
|
curl -sS https://bank.hacktivism.ch/monitoring_err/ | head
|
||||||
|
```
|
||||||
|
|
||||||
|
## Not needed as root
|
||||||
|
|
||||||
|
- Running `taler-monitoring` (use **firecuda-external** / laptop)
|
||||||
|
- Writing into podman Taler containers
|
||||||
|
|
@ -0,0 +1,57 @@
|
||||||
|
# Host Caddy on koopa (root service). Paste BEFORE catch-all reverse_proxy.
|
||||||
|
#
|
||||||
|
# Disk layout:
|
||||||
|
# /var/www/monitoring-sites/{host}/monitoring/index.html
|
||||||
|
# /var/www/monitoring-sites/{host}/monitoring_err/index.html
|
||||||
|
# /var/www/monitoring-sites/taler.hacktivism.ch/taler-monitoring-surface*/index.html
|
||||||
|
# /var/www/monitoring-sites/taler.hacktivism.ch/taler-monitoring-aptdeploy*/index.html
|
||||||
|
#
|
||||||
|
# /monitoring* → bank + exchange + taler.hacktivism.ch
|
||||||
|
# /taler-monitoring-surface* → taler.hacktivism.ch only
|
||||||
|
# /taler-monitoring-aptdeploy* → taler.hacktivism.ch only
|
||||||
|
|
||||||
|
# --- only inside taler.hacktivism.ch { ... } ---
|
||||||
|
handle /taler-monitoring-surface_err {
|
||||||
|
redir /taler-monitoring-surface_err/ 302
|
||||||
|
}
|
||||||
|
handle /taler-monitoring-surface_err/ {
|
||||||
|
root * /var/www/monitoring-sites/taler.hacktivism.ch/taler-monitoring-surface_err
|
||||||
|
file_server
|
||||||
|
}
|
||||||
|
handle /taler-monitoring-surface {
|
||||||
|
redir /taler-monitoring-surface/ 302
|
||||||
|
}
|
||||||
|
handle /taler-monitoring-surface/ {
|
||||||
|
root * /var/www/monitoring-sites/taler.hacktivism.ch/taler-monitoring-surface
|
||||||
|
file_server
|
||||||
|
}
|
||||||
|
handle /taler-monitoring-aptdeploy_err {
|
||||||
|
redir /taler-monitoring-aptdeploy_err/ 302
|
||||||
|
}
|
||||||
|
handle /taler-monitoring-aptdeploy_err/ {
|
||||||
|
root * /var/www/monitoring-sites/taler.hacktivism.ch/taler-monitoring-aptdeploy_err
|
||||||
|
file_server
|
||||||
|
}
|
||||||
|
handle /taler-monitoring-aptdeploy {
|
||||||
|
redir /taler-monitoring-aptdeploy/ 302
|
||||||
|
}
|
||||||
|
handle /taler-monitoring-aptdeploy/ {
|
||||||
|
root * /var/www/monitoring-sites/taler.hacktivism.ch/taler-monitoring-aptdeploy
|
||||||
|
file_server
|
||||||
|
}
|
||||||
|
|
||||||
|
# --- bank + exchange + taler (each site block) ---
|
||||||
|
handle /monitoring_err {
|
||||||
|
redir /monitoring_err/ 302
|
||||||
|
}
|
||||||
|
handle /monitoring_err/ {
|
||||||
|
root * /var/www/monitoring-sites/{host}/monitoring_err
|
||||||
|
file_server
|
||||||
|
}
|
||||||
|
handle /monitoring {
|
||||||
|
redir /monitoring/ 302
|
||||||
|
}
|
||||||
|
handle /monitoring/ {
|
||||||
|
root * /var/www/monitoring-sites/{host}/monitoring
|
||||||
|
file_server
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,34 @@
|
||||||
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||||
|
<!--
|
||||||
|
launchd user agent — firecuda (max): every 4 hours
|
||||||
|
Install: ./install-firecuda-timer.sh
|
||||||
|
-->
|
||||||
|
<plist version="1.0">
|
||||||
|
<dict>
|
||||||
|
<key>Label</key>
|
||||||
|
<string>com.hacktivism.taler-monitoring-sites</string>
|
||||||
|
<key>ProgramArguments</key>
|
||||||
|
<array>
|
||||||
|
<string>/bin/bash</string>
|
||||||
|
<string>__SITE_GEN__/run-on-firecuda.sh</string>
|
||||||
|
</array>
|
||||||
|
<key>StartInterval</key>
|
||||||
|
<integer>14400</integer>
|
||||||
|
<key>RunAtLoad</key>
|
||||||
|
<true/>
|
||||||
|
<key>StandardOutPath</key>
|
||||||
|
<string>__HOME__/Library/Logs/taler-monitoring-sites/launchd.out.log</string>
|
||||||
|
<key>StandardErrorPath</key>
|
||||||
|
<string>__HOME__/Library/Logs/taler-monitoring-sites/launchd.err.log</string>
|
||||||
|
<key>EnvironmentVariables</key>
|
||||||
|
<dict>
|
||||||
|
<key>PATH</key>
|
||||||
|
<string>/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin</string>
|
||||||
|
<key>SKIP_SSH</key>
|
||||||
|
<string>1</string>
|
||||||
|
<key>CONTINUE_ON_ERROR</key>
|
||||||
|
<string>1</string>
|
||||||
|
</dict>
|
||||||
|
</dict>
|
||||||
|
</plist>
|
||||||
935
scripts/taler-monitoring/site-gen/console_to_html.py
Executable file
935
scripts/taler-monitoring/site-gen/console_to_html.py
Executable file
|
|
@ -0,0 +1,935 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Convert taler-monitoring console log to a console-feel HTML page."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import html
|
||||||
|
import re
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
# ANSI strip
|
||||||
|
ANSI_RE = re.compile(r"\x1b\[[0-9;]*m")
|
||||||
|
ERR_BULLET_RE = re.compile(r"^\s*[•·*-]\s+(?P<body>.+)$")
|
||||||
|
TID_RE = re.compile(r"\b(?P<tid>[a-z0-9_.-]+-\d+)\b", re.I)
|
||||||
|
|
||||||
|
|
||||||
|
def strip_ansi(s: str) -> str:
|
||||||
|
return ANSI_RE.sub("", s)
|
||||||
|
|
||||||
|
|
||||||
|
def classify(line: str) -> str:
|
||||||
|
u = line.upper()
|
||||||
|
# section / summary headers before badge matching (they contain the word ERROR)
|
||||||
|
if "SUMMARY" in u or "--- ERRORS" in u or "ERRORS ·" in u or "ERRORS (" in u:
|
||||||
|
return "meta"
|
||||||
|
if "numbered checks" in line.lower() or "totals:" in line.lower():
|
||||||
|
return "meta"
|
||||||
|
if "BLOCKER" in u or "┌ BLOCKER" in u:
|
||||||
|
return "blocker"
|
||||||
|
if "ERROR" in u or "┌ ERROR" in u:
|
||||||
|
return "error"
|
||||||
|
if "WARN" in u or "┌ WARN" in u:
|
||||||
|
return "warn"
|
||||||
|
if "INFO" in u or "┌ INFO" in u:
|
||||||
|
return "info"
|
||||||
|
if " OK" in u or "[OK" in u or "┌ OK" in u or u.strip().startswith("OK"):
|
||||||
|
return "ok"
|
||||||
|
if u.strip().startswith("-- ") or u.strip().startswith("=="):
|
||||||
|
return "section"
|
||||||
|
return "plain"
|
||||||
|
|
||||||
|
|
||||||
|
def is_run_timeout_text(text: str) -> bool:
|
||||||
|
u = text.upper()
|
||||||
|
return "RUN.TIMEOUT" in u or "RUN_TIMEOUT" in u or "RUN TIMEOUT" in u
|
||||||
|
|
||||||
|
|
||||||
|
def is_summary_error_line(ln: str) -> bool:
|
||||||
|
"""Skip summary/header lines that contain ERROR but are not checks."""
|
||||||
|
low = ln.lower()
|
||||||
|
if "failed — see" in low or "failed - see" in low:
|
||||||
|
return True
|
||||||
|
if "errors · failed" in low or "errors (failed" in low:
|
||||||
|
return True
|
||||||
|
if "--- errors" in low:
|
||||||
|
return True
|
||||||
|
if "┌ summary" in low or "[ summary" in low:
|
||||||
|
return True
|
||||||
|
if "totals:" in low:
|
||||||
|
return True
|
||||||
|
# header-ish RUN_TIMEOUT= without error badge
|
||||||
|
if is_run_timeout_text(ln) and "ERROR" not in ln.upper() and "run.timeout" not in low:
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def extract_errors(lines: list[str]) -> list[str]:
|
||||||
|
errs: list[str] = []
|
||||||
|
timeout_detail = ""
|
||||||
|
in_block = False
|
||||||
|
for ln in lines:
|
||||||
|
if "--- ERRORS" in ln or "ERRORS (failed" in ln or "RUN TIMEOUT · extraordinary" in ln:
|
||||||
|
in_block = True
|
||||||
|
continue
|
||||||
|
if in_block:
|
||||||
|
if ln.strip().startswith("---") or "[ SUMMARY" in ln or "totals:" in ln:
|
||||||
|
in_block = False
|
||||||
|
continue
|
||||||
|
m = ERR_BULLET_RE.match(ln)
|
||||||
|
if m:
|
||||||
|
body = m.group("body").strip()
|
||||||
|
if is_run_timeout_text(body):
|
||||||
|
timeout_detail = body
|
||||||
|
else:
|
||||||
|
errs.append(body)
|
||||||
|
elif "ERROR" in ln.upper() and ln.strip() and not is_run_timeout_text(ln):
|
||||||
|
errs.append(ln.strip())
|
||||||
|
if re.search(r"\bERROR\b", ln, re.I) and "--- ERRORS" not in ln:
|
||||||
|
if is_run_timeout_text(ln):
|
||||||
|
if "run.timeout-01" in ln.lower() or "RUN_TIMEOUT exceeded" in ln:
|
||||||
|
timeout_detail = re.sub(r"\s+", " ", strip_ansi(ln)).strip()
|
||||||
|
continue
|
||||||
|
if is_summary_error_line(ln):
|
||||||
|
continue
|
||||||
|
body = re.sub(r"^.*\bERROR\b\s*\]?\s*", "", ln, flags=re.I).strip(" ·")
|
||||||
|
if body and body not in errs and "failed — see" not in body.lower():
|
||||||
|
if re.search(
|
||||||
|
r"#\d+|www\.|e2e\.|auth401\.|versions\.|inside\.|surface\.|aptdeploy\.",
|
||||||
|
body,
|
||||||
|
):
|
||||||
|
errs.append(body)
|
||||||
|
seen: set[str] = set()
|
||||||
|
out: list[str] = []
|
||||||
|
for e in errs:
|
||||||
|
if e in seen or is_run_timeout_text(e):
|
||||||
|
continue
|
||||||
|
seen.add(e)
|
||||||
|
out.append(e)
|
||||||
|
if timeout_detail or any(
|
||||||
|
is_run_timeout_text(l) and ("ERROR" in l.upper() or "run.timeout" in l.lower())
|
||||||
|
for l in lines
|
||||||
|
):
|
||||||
|
# only if real timeout ERROR badge, not mere RUN_TIMEOUT= header
|
||||||
|
if timeout_detail or any(
|
||||||
|
"run.timeout-01" in l.lower() or "RUN_TIMEOUT exceeded" in l for l in lines
|
||||||
|
):
|
||||||
|
canon = timeout_detail or "run.timeout-01 [run] RUN_TIMEOUT exceeded"
|
||||||
|
out = [canon] + out
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def count_status(lines: list[str]) -> tuple[int, int, int | None, int | None]:
|
||||||
|
"""Return (error_count, warn_count, first_error_idx, first_warn_idx)."""
|
||||||
|
n_err = 0
|
||||||
|
n_warn = 0
|
||||||
|
first_err: int | None = None
|
||||||
|
first_warn: int | None = None
|
||||||
|
for i, ln in enumerate(lines):
|
||||||
|
kind = classify(ln)
|
||||||
|
if kind in ("error", "blocker"):
|
||||||
|
if is_summary_error_line(ln):
|
||||||
|
continue
|
||||||
|
n_err += 1
|
||||||
|
if first_err is None:
|
||||||
|
first_err = i
|
||||||
|
elif kind == "warn":
|
||||||
|
n_warn += 1
|
||||||
|
if first_warn is None:
|
||||||
|
first_warn = i
|
||||||
|
return n_err, n_warn, first_err, first_warn
|
||||||
|
|
||||||
|
|
||||||
|
def slug_error(i: int, text: str) -> str:
|
||||||
|
if is_run_timeout_text(text):
|
||||||
|
return "err-run-timeout"
|
||||||
|
m = TID_RE.search(text)
|
||||||
|
if m:
|
||||||
|
return "err-" + re.sub(r"[^a-z0-9_-]+", "-", m.group("tid").lower())
|
||||||
|
return f"err-{i:03d}"
|
||||||
|
|
||||||
|
|
||||||
|
def render_line(ln: str, err_slugs: dict[str, str], extra_ids: list[str] | None = None) -> str:
|
||||||
|
kind = classify(ln)
|
||||||
|
esc = html.escape(ln)
|
||||||
|
for tid, slug in err_slugs.items():
|
||||||
|
if tid in ln:
|
||||||
|
esc = esc.replace(
|
||||||
|
html.escape(tid),
|
||||||
|
f'<a class="jump" href="#{html.escape(slug)}">{html.escape(tid)}</a>',
|
||||||
|
)
|
||||||
|
id_attr = ""
|
||||||
|
if extra_ids:
|
||||||
|
id_attr = f' id="{" ".join(extra_ids)}"' # invalid multi-id; use first only
|
||||||
|
# HTML allows one id — join carefully
|
||||||
|
id_attr = f' id="{html.escape(extra_ids[0])}"'
|
||||||
|
if len(extra_ids) > 1:
|
||||||
|
# secondary anchors as empty spans prepended
|
||||||
|
spans = "".join(f'<span id="{html.escape(x)}"></span>' for x in extra_ids[1:])
|
||||||
|
return f'{spans}<div{id_attr} class="line {kind}">{esc}</div>\n'
|
||||||
|
return f'<div{id_attr} class="line {kind}">{esc}</div>\n'
|
||||||
|
|
||||||
|
|
||||||
|
def status_level(n_err: int, n_warn: int) -> str:
|
||||||
|
if n_err > 0:
|
||||||
|
return "red"
|
||||||
|
if n_warn > 0:
|
||||||
|
return "yellow"
|
||||||
|
return "green"
|
||||||
|
|
||||||
|
|
||||||
|
def extract_phases(log_text: str) -> str:
|
||||||
|
"""Best-effort phases string from host-agent / taler-monitoring log header."""
|
||||||
|
for pat in (
|
||||||
|
r"phases=([a-z0-9 _.-]+)",
|
||||||
|
r"· phases=([a-z0-9 _.-]+)",
|
||||||
|
r"phases\s{2,}([a-z0-9 _.-]+)",
|
||||||
|
r"^\s*phases\s+([a-z0-9 _.-]+)\s*$",
|
||||||
|
):
|
||||||
|
m = re.search(pat, log_text, re.I | re.M)
|
||||||
|
if m:
|
||||||
|
return re.sub(r"\s+", " ", m.group(1)).strip(" ·")
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
def monitoring_scope(
|
||||||
|
page_label: str,
|
||||||
|
hostname: str,
|
||||||
|
log_text: str = "",
|
||||||
|
) -> dict[str, object]:
|
||||||
|
"""
|
||||||
|
Human-readable description of what this monitoring page covers.
|
||||||
|
Returned keys: title, summary, items (list[str]), kind
|
||||||
|
"""
|
||||||
|
label = (page_label or "monitoring").lower().replace("_", "-")
|
||||||
|
phases = extract_phases(log_text)
|
||||||
|
host = hostname or "?"
|
||||||
|
|
||||||
|
if "surface" in label:
|
||||||
|
return {
|
||||||
|
"kind": "surface",
|
||||||
|
"title": "Remote ecosystem surface inventory",
|
||||||
|
"summary": "Public catalog hosts · DNS / TCP / HTTPS / TLS · Server version · CVE",
|
||||||
|
"items": [
|
||||||
|
"Outside-in only: no SSH and no podman exec on remote targets",
|
||||||
|
"Catalog (surface-catalog.conf): taler.net, demo/test, gnunet.org, "
|
||||||
|
"taler-ops.ch, taler-systems.com, deb.taler.net, ftp.gnu.org, …",
|
||||||
|
"Per host: DNS resolve, ICMP (info), open ports, protocol probes, "
|
||||||
|
"TLS cert expiry, Server-header software version, optional CVE check",
|
||||||
|
f"HTML published only on taler.hacktivism.ch "
|
||||||
|
f"(this page host: {host})",
|
||||||
|
"Timer: taler-monitoring-surface.timer (hourly)",
|
||||||
|
],
|
||||||
|
}
|
||||||
|
|
||||||
|
if "aptdeploy" in label or "apt-deploy" in label or "apt_src" in label:
|
||||||
|
return {
|
||||||
|
"kind": "aptdeploy",
|
||||||
|
"title": "apt-src merchant deploy tests (podman on koopa)",
|
||||||
|
"summary": "4 containers · fresh + upgrade tracks · trixie & trixie-testing",
|
||||||
|
"items": [
|
||||||
|
"koopa-taler-deploy-test-apt-src-trixie (fresh, suite trixie)",
|
||||||
|
"koopa-taler-deploy-test-apt-src-trixie-testing (fresh, trixie-testing)",
|
||||||
|
"koopa-taler-deploy-test-apt-src-trixie-upgrade (upgrade track)",
|
||||||
|
"koopa-taler-deploy-test-apt-src-trixie-testing-upgrade (upgrade track)",
|
||||||
|
"Checks: packages, taler-merchant-httpd --version / ldd, systemd unit, "
|
||||||
|
"optional local /config probe — not public nginx/HTTPS",
|
||||||
|
f"HTML published only on taler.hacktivism.ch (this page host: {host})",
|
||||||
|
"Timer: taler-monitoring-aptdeploy.timer (4h)",
|
||||||
|
],
|
||||||
|
}
|
||||||
|
|
||||||
|
# Default: stack /monitoring pages (bank, exchange, merchant)
|
||||||
|
phase_txt = phases or "urls · inside · versions"
|
||||||
|
role = "stack"
|
||||||
|
if host.startswith("bank."):
|
||||||
|
role = "bank"
|
||||||
|
focus = "Libeufin bank public HTTPS + container inside checks"
|
||||||
|
elif host.startswith("exchange.") or host.startswith("stage.exchange."):
|
||||||
|
role = "exchange"
|
||||||
|
focus = "Exchange public HTTPS + container inside checks"
|
||||||
|
elif host.startswith("taler.") or "merchant" in host or host.startswith("monnaie."):
|
||||||
|
role = "merchant"
|
||||||
|
focus = "Merchant backend / SPA public HTTPS + container inside checks"
|
||||||
|
else:
|
||||||
|
focus = "Taler stack public + inside checks"
|
||||||
|
|
||||||
|
items = [
|
||||||
|
f"Focus: {focus}",
|
||||||
|
f"Phases this run: {phase_txt}",
|
||||||
|
"Typical: public URLs (HTTPS, QR, perf), inside (podman/ssh), package versions",
|
||||||
|
f"HTML host: {host}",
|
||||||
|
"Timer: taler-monitoring-hacktivism (path + 4h) · /monitoring on bank, exchange, taler",
|
||||||
|
]
|
||||||
|
if phases:
|
||||||
|
items.insert(2, f"Log header phases= {phases}")
|
||||||
|
|
||||||
|
return {
|
||||||
|
"kind": role,
|
||||||
|
"title": f"Stack monitoring · {host}",
|
||||||
|
"summary": f"{phase_txt} · public + inside",
|
||||||
|
"items": items,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def sticky_bar_html(
|
||||||
|
*,
|
||||||
|
generated_at: str,
|
||||||
|
generated_iso: str,
|
||||||
|
n_err: int,
|
||||||
|
n_warn: int,
|
||||||
|
hostname: str,
|
||||||
|
page_label: str,
|
||||||
|
mode: str,
|
||||||
|
commit_html: str,
|
||||||
|
suite_path: str,
|
||||||
|
link_other: str | None,
|
||||||
|
first_err_href: str | None,
|
||||||
|
first_warn_href: str | None,
|
||||||
|
scope: dict[str, object] | None = None,
|
||||||
|
) -> str:
|
||||||
|
level = status_level(n_err, n_warn)
|
||||||
|
if level == "red":
|
||||||
|
status_txt = "ERRORS"
|
||||||
|
elif level == "yellow":
|
||||||
|
status_txt = "WARNINGS"
|
||||||
|
else:
|
||||||
|
status_txt = "OK"
|
||||||
|
|
||||||
|
if first_err_href and n_err:
|
||||||
|
err_stat = (
|
||||||
|
f'<a class="stat err" href="{html.escape(first_err_href)}" '
|
||||||
|
f'title="Jump to first error">{n_err} error{"s" if n_err != 1 else ""}</a>'
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
err_stat = f'<span class="stat err muted">{n_err} errors</span>'
|
||||||
|
|
||||||
|
if first_warn_href and n_warn:
|
||||||
|
warn_stat = (
|
||||||
|
f'<a class="stat warn" href="{html.escape(first_warn_href)}" '
|
||||||
|
f'title="Jump to first warning">{n_warn} warning{"s" if n_warn != 1 else ""}</a>'
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
warn_stat = f'<span class="stat warn muted">{n_warn} warnings</span>'
|
||||||
|
|
||||||
|
other = ""
|
||||||
|
if link_other:
|
||||||
|
other = (
|
||||||
|
f'<a class="nav-link" href="{html.escape(link_other)}">'
|
||||||
|
f"{html.escape(link_other)}</a>"
|
||||||
|
)
|
||||||
|
|
||||||
|
if scope is None:
|
||||||
|
scope = monitoring_scope(page_label, hostname, "")
|
||||||
|
scope_title = html.escape(str(scope.get("title") or "Monitoring"))
|
||||||
|
scope_summary = html.escape(str(scope.get("summary") or ""))
|
||||||
|
scope_kind = html.escape(str(scope.get("kind") or "monitoring"))
|
||||||
|
lis = "\n".join(
|
||||||
|
f"<li>{html.escape(str(it))}</li>" for it in (scope.get("items") or [])
|
||||||
|
)
|
||||||
|
suite_bit = (
|
||||||
|
f" · <code>{html.escape(suite_path)}</code>" if suite_path else ""
|
||||||
|
)
|
||||||
|
|
||||||
|
# Compact sticky row always visible; "Was geprüft wird" expands inside sticky-bar
|
||||||
|
return f"""
|
||||||
|
<div class="sticky-bar sticky-{level}" id="status-bar" role="status"
|
||||||
|
data-errors="{n_err}" data-warnings="{n_warn}" data-level="{level}"
|
||||||
|
data-scope-kind="{scope_kind}">
|
||||||
|
<div class="sticky-bar-row primary">
|
||||||
|
<span class="status-pill sticky-{level}">{html.escape(status_txt)}</span>
|
||||||
|
<span class="host">{html.escape(page_label)} · {html.escape(hostname)}</span>
|
||||||
|
<span class="sep">·</span>
|
||||||
|
{err_stat}
|
||||||
|
<span class="sep">·</span>
|
||||||
|
{warn_stat}
|
||||||
|
<span class="sep">·</span>
|
||||||
|
<span class="generated"
|
||||||
|
data-generated-iso="{html.escape(generated_iso)}"
|
||||||
|
title="{html.escape(generated_at)}">
|
||||||
|
generated <time datetime="{html.escape(generated_iso)}">{html.escape(generated_at)}</time>
|
||||||
|
<span class="age" id="generated-age"></span>
|
||||||
|
</span>
|
||||||
|
<button type="button" class="scope-toggle" id="scope-toggle"
|
||||||
|
aria-expanded="false" aria-controls="sticky-scope-panel"
|
||||||
|
title="Aufklappen: was diese Seite prüft">
|
||||||
|
<span class="scope-chevron" aria-hidden="true"></span>
|
||||||
|
<span class="scope-toggle-label">Was geprüft wird</span>
|
||||||
|
<span class="scope-toggle-hint">{scope_summary}</span>
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
<div class="sticky-scope-panel" id="sticky-scope-panel" hidden>
|
||||||
|
<div class="sticky-bar-row secondary">
|
||||||
|
<span class="badge mode-{html.escape(mode)}">{html.escape(mode.upper())}</span>
|
||||||
|
source {commit_html}{suite_bit}
|
||||||
|
{(" · " + other) if other else ""}
|
||||||
|
</div>
|
||||||
|
<div class="scope-body">
|
||||||
|
<h3 class="scope-title">{scope_title}</h3>
|
||||||
|
<p class="scope-lead">{scope_summary}</p>
|
||||||
|
<ul class="scope-list">
|
||||||
|
{lis}
|
||||||
|
</ul>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
STICKY_CSS = """
|
||||||
|
.sticky-bar {
|
||||||
|
position: sticky; top: 0; z-index: 100;
|
||||||
|
border-bottom: 2px solid var(--border);
|
||||||
|
padding: 10px 14px 8px;
|
||||||
|
backdrop-filter: blur(8px);
|
||||||
|
-webkit-backdrop-filter: blur(8px);
|
||||||
|
box-shadow: 0 2px 12px #0008;
|
||||||
|
}
|
||||||
|
.sticky-bar.sticky-green {
|
||||||
|
background: linear-gradient(180deg, #0d1f14f2, #0a1410ee);
|
||||||
|
border-bottom-color: #1f6b3a;
|
||||||
|
}
|
||||||
|
.sticky-bar.sticky-yellow {
|
||||||
|
background: linear-gradient(180deg, #2a2210f2, #1a160cee);
|
||||||
|
border-bottom-color: #a68b2d;
|
||||||
|
}
|
||||||
|
.sticky-bar.sticky-red {
|
||||||
|
background: linear-gradient(180deg, #2a1010f2, #1a0808ee);
|
||||||
|
border-bottom-color: #a33;
|
||||||
|
}
|
||||||
|
.sticky-bar-row {
|
||||||
|
display: flex; flex-wrap: wrap; align-items: center; gap: 6px 10px;
|
||||||
|
max-width: 1200px;
|
||||||
|
}
|
||||||
|
.sticky-bar-row.secondary {
|
||||||
|
margin-top: 8px; font-size: 12px; color: var(--dim);
|
||||||
|
}
|
||||||
|
.sticky-bar-row.secondary a { color: var(--info); text-decoration: none; }
|
||||||
|
.sticky-bar-row.secondary a:hover { text-decoration: underline; }
|
||||||
|
.status-pill {
|
||||||
|
display: inline-block; font-size: 11px; font-weight: 800;
|
||||||
|
letter-spacing: 0.06em; padding: 2px 10px; border-radius: 3px;
|
||||||
|
border: 1px solid;
|
||||||
|
}
|
||||||
|
.status-pill.sticky-green { color: var(--ok); border-color: #1f6b3a; background: #0a1a10; }
|
||||||
|
.status-pill.sticky-yellow { color: var(--warn); border-color: #a68b2d; background: #1a1608; }
|
||||||
|
.status-pill.sticky-red { color: var(--err); border-color: #a33; background: #1a0a0a; }
|
||||||
|
.host { font-weight: 600; color: #eee; font-size: 13px; }
|
||||||
|
.sep { color: var(--dim); }
|
||||||
|
.stat {
|
||||||
|
font-weight: 700; font-size: 12px; text-decoration: none;
|
||||||
|
padding: 1px 8px; border-radius: 3px; border: 1px solid transparent;
|
||||||
|
}
|
||||||
|
.stat.err { color: var(--err); border-color: #522; background: #1a0a0a; }
|
||||||
|
.stat.warn { color: var(--warn); border-color: #664; background: #1a1608; }
|
||||||
|
.stat.muted { opacity: 0.55; font-weight: 600; }
|
||||||
|
a.stat:hover { text-decoration: underline; filter: brightness(1.15); }
|
||||||
|
.generated { color: var(--dim); font-size: 12px; }
|
||||||
|
.generated time { color: #bbb; }
|
||||||
|
.generated .age { color: #888; margin-left: 4px; }
|
||||||
|
.generated .age:not(:empty)::before { content: "("; }
|
||||||
|
.generated .age:not(:empty)::after { content: ")"; }
|
||||||
|
.badge {
|
||||||
|
display: inline-block; padding: 1px 8px; border-radius: 3px;
|
||||||
|
font-size: 11px; font-weight: 700; margin-right: 4px;
|
||||||
|
border: 1px solid var(--border);
|
||||||
|
}
|
||||||
|
.badge.mode-err { color: var(--err); border-color: #522; background: #1a0a0a; }
|
||||||
|
.badge.mode-ok { color: var(--ok); border-color: #143; background: #0a1a10; }
|
||||||
|
.badge.mode-redirect { color: var(--warn); border-color: #664; background: #1a1608; }
|
||||||
|
/* collapsible block inside sticky-bar */
|
||||||
|
.scope-toggle {
|
||||||
|
margin-left: auto;
|
||||||
|
display: inline-flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 8px;
|
||||||
|
max-width: min(42rem, 100%);
|
||||||
|
cursor: pointer;
|
||||||
|
font: inherit;
|
||||||
|
font-size: 12px;
|
||||||
|
color: #ddd;
|
||||||
|
background: #0006;
|
||||||
|
border: 1px solid #444;
|
||||||
|
border-radius: 4px;
|
||||||
|
padding: 4px 10px;
|
||||||
|
text-align: left;
|
||||||
|
}
|
||||||
|
.scope-toggle:hover { border-color: #888; background: #111a; color: #fff; }
|
||||||
|
.scope-toggle:focus-visible { outline: 2px solid var(--info); outline-offset: 2px; }
|
||||||
|
.scope-toggle[aria-expanded="true"] {
|
||||||
|
border-color: #777;
|
||||||
|
background: #141414;
|
||||||
|
}
|
||||||
|
.scope-chevron {
|
||||||
|
display: inline-block;
|
||||||
|
width: 0; height: 0;
|
||||||
|
border-left: 5px solid #aaa;
|
||||||
|
border-top: 4px solid transparent;
|
||||||
|
border-bottom: 4px solid transparent;
|
||||||
|
transition: transform 0.12s ease;
|
||||||
|
flex-shrink: 0;
|
||||||
|
}
|
||||||
|
.scope-toggle[aria-expanded="true"] .scope-chevron {
|
||||||
|
transform: rotate(90deg);
|
||||||
|
border-left-color: #eee;
|
||||||
|
}
|
||||||
|
.scope-toggle-label {
|
||||||
|
font-weight: 700;
|
||||||
|
letter-spacing: 0.03em;
|
||||||
|
text-transform: uppercase;
|
||||||
|
font-size: 11px;
|
||||||
|
white-space: nowrap;
|
||||||
|
}
|
||||||
|
.scope-toggle-hint {
|
||||||
|
color: #888;
|
||||||
|
overflow: hidden;
|
||||||
|
text-overflow: ellipsis;
|
||||||
|
white-space: nowrap;
|
||||||
|
min-width: 0;
|
||||||
|
}
|
||||||
|
.sticky-scope-panel {
|
||||||
|
margin-top: 8px;
|
||||||
|
max-width: 1200px;
|
||||||
|
border: 1px solid #333;
|
||||||
|
border-radius: 4px;
|
||||||
|
background: #080808e6;
|
||||||
|
padding: 8px 10px 10px;
|
||||||
|
}
|
||||||
|
.sticky-scope-panel[hidden] { display: none !important; }
|
||||||
|
.scope-body { padding: 4px 4px 2px 6px; }
|
||||||
|
.scope-title {
|
||||||
|
margin: 4px 0 4px;
|
||||||
|
font-size: 13px;
|
||||||
|
font-weight: 600;
|
||||||
|
color: #eee;
|
||||||
|
}
|
||||||
|
.scope-lead {
|
||||||
|
margin: 0 0 6px;
|
||||||
|
font-size: 12px;
|
||||||
|
color: #999;
|
||||||
|
}
|
||||||
|
.scope-list {
|
||||||
|
margin: 0;
|
||||||
|
padding-left: 1.15em;
|
||||||
|
color: #b0b0b0;
|
||||||
|
font-size: 12px;
|
||||||
|
line-height: 1.5;
|
||||||
|
}
|
||||||
|
.scope-list li { margin: 3px 0; }
|
||||||
|
#first-error, #first-warn,
|
||||||
|
[id^="err-"][id$="-line"] {
|
||||||
|
scroll-margin-top: 120px;
|
||||||
|
}
|
||||||
|
@media (max-width: 720px) {
|
||||||
|
.scope-toggle { margin-left: 0; width: 100%; }
|
||||||
|
.scope-toggle-hint { display: none; }
|
||||||
|
}
|
||||||
|
"""
|
||||||
|
|
||||||
|
STICKY_JS = """
|
||||||
|
<script>
|
||||||
|
(function () {
|
||||||
|
function fmtAge(sec) {
|
||||||
|
if (sec < 0) sec = 0;
|
||||||
|
if (sec < 60) return sec + "s ago";
|
||||||
|
if (sec < 3600) return Math.floor(sec / 60) + "m ago";
|
||||||
|
if (sec < 86400) return Math.floor(sec / 3600) + "h ago";
|
||||||
|
return Math.floor(sec / 86400) + "d ago";
|
||||||
|
}
|
||||||
|
function tick() {
|
||||||
|
var el = document.querySelector("[data-generated-iso]");
|
||||||
|
var age = document.getElementById("generated-age");
|
||||||
|
if (!el || !age) return;
|
||||||
|
var iso = el.getAttribute("data-generated-iso");
|
||||||
|
if (!iso) return;
|
||||||
|
var t = Date.parse(iso);
|
||||||
|
if (isNaN(t)) return;
|
||||||
|
var sec = Math.floor((Date.now() - t) / 1000);
|
||||||
|
age.textContent = fmtAge(sec);
|
||||||
|
age.title = "page age · updates every second";
|
||||||
|
}
|
||||||
|
tick();
|
||||||
|
setInterval(tick, 1000);
|
||||||
|
|
||||||
|
// Sticky-bar collapsible: what this page monitors
|
||||||
|
var btn = document.getElementById("scope-toggle");
|
||||||
|
var panel = document.getElementById("sticky-scope-panel");
|
||||||
|
if (btn && panel) {
|
||||||
|
var key = "taler-mon-scope-open:" + (location.pathname || "");
|
||||||
|
function setOpen(open) {
|
||||||
|
btn.setAttribute("aria-expanded", open ? "true" : "false");
|
||||||
|
if (open) panel.removeAttribute("hidden");
|
||||||
|
else panel.setAttribute("hidden", "");
|
||||||
|
try { localStorage.setItem(key, open ? "1" : "0"); } catch (e) {}
|
||||||
|
}
|
||||||
|
btn.addEventListener("click", function () {
|
||||||
|
setOpen(btn.getAttribute("aria-expanded") !== "true");
|
||||||
|
});
|
||||||
|
try {
|
||||||
|
if (localStorage.getItem(key) === "1") setOpen(true);
|
||||||
|
} catch (e) {}
|
||||||
|
}
|
||||||
|
})();
|
||||||
|
</script>
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def build_html(
|
||||||
|
*,
|
||||||
|
title: str,
|
||||||
|
hostname: str,
|
||||||
|
mode: str,
|
||||||
|
log_text: str,
|
||||||
|
commit: str,
|
||||||
|
commit_url: str,
|
||||||
|
suite_path: str,
|
||||||
|
generated_at: str,
|
||||||
|
generated_iso: str,
|
||||||
|
link_other: str | None,
|
||||||
|
page_label: str = "monitoring",
|
||||||
|
) -> str:
|
||||||
|
raw_lines = [strip_ansi(l).rstrip("\n") for l in log_text.splitlines()]
|
||||||
|
while raw_lines and not raw_lines[-1].strip():
|
||||||
|
raw_lines.pop()
|
||||||
|
|
||||||
|
n_err, n_warn, first_err_i, first_warn_i = count_status(raw_lines)
|
||||||
|
errors = extract_errors(raw_lines)
|
||||||
|
|
||||||
|
err_slugs: dict[str, str] = {}
|
||||||
|
for i, e in enumerate(errors, 1):
|
||||||
|
slug = slug_error(i, e)
|
||||||
|
err_slugs[e] = slug
|
||||||
|
m = TID_RE.search(e)
|
||||||
|
if m:
|
||||||
|
err_slugs[m.group("tid")] = slug
|
||||||
|
|
||||||
|
tid_to_slug = {
|
||||||
|
k: v for k, v in err_slugs.items() if re.match(r"^[a-z0-9_.-]+-\d+$", k, re.I)
|
||||||
|
}
|
||||||
|
|
||||||
|
# Build body with anchors for first error/warn + per-error slugs
|
||||||
|
body_lines: list[str] = []
|
||||||
|
claimed_err: set[str] = set()
|
||||||
|
for idx, ln in enumerate(raw_lines):
|
||||||
|
extra: list[str] = []
|
||||||
|
if first_err_i is not None and idx == first_err_i:
|
||||||
|
extra.append("first-error")
|
||||||
|
if first_warn_i is not None and idx == first_warn_i:
|
||||||
|
extra.append("first-warn")
|
||||||
|
# per-error jump targets (all modes)
|
||||||
|
if classify(ln) in ("error", "blocker") and not is_summary_error_line(ln):
|
||||||
|
for i, e in enumerate(errors, 1):
|
||||||
|
slug = slug_error(i, e)
|
||||||
|
if slug in claimed_err:
|
||||||
|
continue
|
||||||
|
tid_m = TID_RE.search(e)
|
||||||
|
if is_run_timeout_text(e):
|
||||||
|
if "run.timeout-01" in ln and "ERROR" in ln.upper():
|
||||||
|
extra.append(f"{slug}-line")
|
||||||
|
claimed_err.add(slug)
|
||||||
|
break
|
||||||
|
continue
|
||||||
|
key = tid_m.group(1) if tid_m else e[:40]
|
||||||
|
if key and key in ln:
|
||||||
|
extra.append(f"{slug}-line")
|
||||||
|
claimed_err.add(slug)
|
||||||
|
break
|
||||||
|
body_lines.append(render_line(ln, tid_to_slug, extra or None))
|
||||||
|
|
||||||
|
err_nav = ""
|
||||||
|
if errors and mode == "err":
|
||||||
|
items = []
|
||||||
|
timeout_items = []
|
||||||
|
for i, e in enumerate(errors, 1):
|
||||||
|
slug = slug_error(i, e)
|
||||||
|
li = (
|
||||||
|
f'<li id="{html.escape(slug)}">'
|
||||||
|
f'<a href="#{html.escape(slug)}-line">{html.escape(e)}</a></li>'
|
||||||
|
)
|
||||||
|
if is_run_timeout_text(e):
|
||||||
|
timeout_items.append(li)
|
||||||
|
else:
|
||||||
|
items.append(li)
|
||||||
|
if timeout_items:
|
||||||
|
err_nav += (
|
||||||
|
'<section class="err-top extraordinary" id="run-timeout-top">\n'
|
||||||
|
"<h2>EXTRAORDINARY · RUN TIMEOUT</h2>\n"
|
||||||
|
"<p>Whole-run wall clock exceeded. "
|
||||||
|
'<a href="#err-run-timeout-line">Jump to detail →</a></p>\n'
|
||||||
|
'<ol class="err-list">\n'
|
||||||
|
+ "\n".join(timeout_items[:1])
|
||||||
|
+ "\n</ol>\n</section>\n"
|
||||||
|
)
|
||||||
|
rest = items if timeout_items else (timeout_items + items)
|
||||||
|
if rest or not timeout_items:
|
||||||
|
err_nav += (
|
||||||
|
'<section class="err-top">\n'
|
||||||
|
f"<h2>ERRORS ({len(errors)}) — jump to detail</h2>\n"
|
||||||
|
'<ol class="err-list">\n'
|
||||||
|
+ "\n".join(rest if rest else items)
|
||||||
|
+ "\n</ol>\n</section>\n"
|
||||||
|
)
|
||||||
|
|
||||||
|
commit_short = commit[:12] if commit else "unknown"
|
||||||
|
commit_html = (
|
||||||
|
f'<a href="{html.escape(commit_url)}">{html.escape(commit_short)}</a>'
|
||||||
|
if commit_url
|
||||||
|
else html.escape(commit_short)
|
||||||
|
)
|
||||||
|
|
||||||
|
first_err_href = "#first-error" if first_err_i is not None else None
|
||||||
|
first_warn_href = "#first-warn" if first_warn_i is not None else None
|
||||||
|
scope = monitoring_scope(page_label, hostname, log_text)
|
||||||
|
|
||||||
|
bar = sticky_bar_html(
|
||||||
|
generated_at=generated_at,
|
||||||
|
generated_iso=generated_iso,
|
||||||
|
n_err=n_err,
|
||||||
|
n_warn=n_warn,
|
||||||
|
hostname=hostname,
|
||||||
|
page_label=page_label,
|
||||||
|
mode=mode,
|
||||||
|
commit_html=commit_html,
|
||||||
|
suite_path=suite_path,
|
||||||
|
link_other=link_other,
|
||||||
|
first_err_href=first_err_href,
|
||||||
|
first_warn_href=first_warn_href,
|
||||||
|
scope=scope,
|
||||||
|
)
|
||||||
|
|
||||||
|
return f"""<!DOCTYPE html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8"/>
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1"/>
|
||||||
|
<meta name="generated" content="{html.escape(generated_iso)}"/>
|
||||||
|
<title>{html.escape(title)}</title>
|
||||||
|
<style>
|
||||||
|
:root {{
|
||||||
|
--bg: #0c0c0c;
|
||||||
|
--fg: #c8c8c8;
|
||||||
|
--dim: #666;
|
||||||
|
--ok: #3ddc84;
|
||||||
|
--err: #ff5c5c;
|
||||||
|
--warn: #e6c07b;
|
||||||
|
--info: #61afef;
|
||||||
|
--blocker: #c678dd;
|
||||||
|
--sec: #56b6c2;
|
||||||
|
--border: #222;
|
||||||
|
--panel: #121212;
|
||||||
|
font-family: "JetBrains Mono", "Fira Code", "Cascadia Code", ui-monospace, Menlo, Consolas, monospace;
|
||||||
|
}}
|
||||||
|
* {{ box-sizing: border-box; }}
|
||||||
|
body {{
|
||||||
|
margin: 0; background: var(--bg); color: var(--fg);
|
||||||
|
font-size: 13px; line-height: 1.45;
|
||||||
|
}}
|
||||||
|
{STICKY_CSS}
|
||||||
|
main {{ padding: 12px 14px 48px; max-width: 1200px; }}
|
||||||
|
.err-top {{
|
||||||
|
background: var(--panel); border: 1px solid #422;
|
||||||
|
padding: 10px 12px; margin-bottom: 14px; border-radius: 4px;
|
||||||
|
}}
|
||||||
|
.err-top.extraordinary {{
|
||||||
|
border-color: #a44; background: #1a0808;
|
||||||
|
box-shadow: 0 0 0 1px #622 inset;
|
||||||
|
}}
|
||||||
|
.err-top.extraordinary h2 {{ color: #ff8a8a; }}
|
||||||
|
.err-top.extraordinary p {{ margin: 0 0 8px; color: var(--fg); font-size: 12px; }}
|
||||||
|
.err-top.extraordinary p a {{ color: #ff8a8a; font-weight: 600; }}
|
||||||
|
.err-top h2 {{ margin: 0 0 8px; font-size: 13px; color: var(--err); }}
|
||||||
|
.err-list {{ margin: 0; padding-left: 1.2em; }}
|
||||||
|
.err-list li {{ margin: 4px 0; }}
|
||||||
|
.err-list a {{ color: var(--err); text-decoration: none; }}
|
||||||
|
.err-list a:hover {{ text-decoration: underline; }}
|
||||||
|
#err-run-timeout-line {{
|
||||||
|
outline: 1px solid #a44; background: #1a0808;
|
||||||
|
padding: 4px 6px; margin: 4px 0; border-radius: 3px;
|
||||||
|
}}
|
||||||
|
.console {{
|
||||||
|
background: #0a0a0a; border: 1px solid var(--border);
|
||||||
|
border-radius: 4px; padding: 8px 10px;
|
||||||
|
white-space: pre-wrap; word-break: break-word;
|
||||||
|
}}
|
||||||
|
.line {{ padding: 1px 0; }}
|
||||||
|
.line.ok {{ color: var(--ok); }}
|
||||||
|
.line.error {{ color: var(--err); }}
|
||||||
|
.line.warn {{ color: var(--warn); }}
|
||||||
|
.line.info {{ color: var(--info); }}
|
||||||
|
.line.blocker {{ color: var(--blocker); }}
|
||||||
|
.line.section {{ color: var(--sec); margin-top: 8px; font-weight: 600; }}
|
||||||
|
.line.meta {{ color: var(--dim); }}
|
||||||
|
.line.plain {{ color: var(--fg); }}
|
||||||
|
a.jump {{ color: inherit; text-decoration: underline dotted; }}
|
||||||
|
footer {{
|
||||||
|
margin-top: 18px; color: var(--dim); font-size: 12px;
|
||||||
|
border-top: 1px solid var(--border); padding-top: 10px;
|
||||||
|
}}
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
{bar}
|
||||||
|
<main>
|
||||||
|
{err_nav}
|
||||||
|
<div class="console">
|
||||||
|
{"".join(body_lines)}
|
||||||
|
</div>
|
||||||
|
<footer>
|
||||||
|
Console-style render of taler-monitoring output.
|
||||||
|
Sticky bar: green = clean · yellow = warnings · red = errors.
|
||||||
|
Commit pins the exact tree used for this run.
|
||||||
|
</footer>
|
||||||
|
</main>
|
||||||
|
{STICKY_JS}
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def build_redirect_html(
|
||||||
|
*,
|
||||||
|
hostname: str,
|
||||||
|
commit: str,
|
||||||
|
commit_url: str,
|
||||||
|
path_err: str = "/monitoring_err/",
|
||||||
|
page_label: str = "monitoring",
|
||||||
|
generated_at: str = "",
|
||||||
|
generated_iso: str = "",
|
||||||
|
n_err: int = 0,
|
||||||
|
n_warn: int = 0,
|
||||||
|
log_text: str = "",
|
||||||
|
) -> str:
|
||||||
|
commit_short = commit[:12] if commit else "unknown"
|
||||||
|
link = path_err if path_err.endswith("/") else path_err + "/"
|
||||||
|
c = (
|
||||||
|
f'<a href="{html.escape(commit_url)}">{html.escape(commit_short)}</a>'
|
||||||
|
if commit_url
|
||||||
|
else html.escape(commit_short)
|
||||||
|
)
|
||||||
|
if not generated_at:
|
||||||
|
now = datetime.now(timezone.utc)
|
||||||
|
generated_iso = now.strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||||
|
generated_at = now.strftime("%Y-%m-%d %H:%M:%SZ")
|
||||||
|
# Redirect pages exist only when the run failed → treat as red if n_err unknown
|
||||||
|
if n_err <= 0:
|
||||||
|
n_err = max(n_err, 1)
|
||||||
|
scope = monitoring_scope(page_label, hostname, log_text)
|
||||||
|
bar = sticky_bar_html(
|
||||||
|
generated_at=generated_at,
|
||||||
|
generated_iso=generated_iso,
|
||||||
|
n_err=n_err,
|
||||||
|
n_warn=n_warn,
|
||||||
|
hostname=hostname,
|
||||||
|
page_label=page_label,
|
||||||
|
mode="redirect",
|
||||||
|
commit_html=c,
|
||||||
|
suite_path="",
|
||||||
|
link_other=link,
|
||||||
|
first_err_href=link,
|
||||||
|
first_warn_href=None,
|
||||||
|
scope=scope,
|
||||||
|
)
|
||||||
|
return f"""<!DOCTYPE html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8"/>
|
||||||
|
<meta http-equiv="refresh" content="2; url={html.escape(link)}"/>
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1"/>
|
||||||
|
<meta name="generated" content="{html.escape(generated_iso)}"/>
|
||||||
|
<title>{html.escape(page_label)} · errors · {html.escape(hostname)}</title>
|
||||||
|
<style>
|
||||||
|
:root {{
|
||||||
|
--bg: #0c0c0c; --fg: #c8c8c8; --dim: #666;
|
||||||
|
--ok: #3ddc84; --err: #ff5c5c; --warn: #e6c07b; --info: #61afef; --border: #222;
|
||||||
|
font-family: ui-monospace, Menlo, Consolas, monospace;
|
||||||
|
}}
|
||||||
|
body {{ margin: 0; background: var(--bg); color: var(--fg); font-size: 13px; }}
|
||||||
|
{STICKY_CSS}
|
||||||
|
.box {{
|
||||||
|
max-width: 520px; margin: 12vh auto; padding: 20px;
|
||||||
|
border: 1px solid #422; background: #121212; border-radius: 4px;
|
||||||
|
}}
|
||||||
|
.box a {{ color: var(--err); }}
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
{bar}
|
||||||
|
<div class="box">
|
||||||
|
<p><strong>{html.escape(page_label)}</strong> has failures.</p>
|
||||||
|
<p>See <a href="{html.escape(link)}">{html.escape(link)}</a> for the full console log,
|
||||||
|
error index, and sticky status bar.</p>
|
||||||
|
<p style="color:#666;font-size:12px">generated {html.escape(generated_at)} · source {c}</p>
|
||||||
|
</div>
|
||||||
|
{STICKY_JS}
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--log", required=True, type=Path)
|
||||||
|
ap.add_argument("--out", required=True, type=Path)
|
||||||
|
ap.add_argument("--hostname", required=True)
|
||||||
|
ap.add_argument("--mode", choices=("ok", "err", "redirect"), required=True)
|
||||||
|
ap.add_argument("--commit", default="")
|
||||||
|
ap.add_argument("--commit-url", default="")
|
||||||
|
ap.add_argument("--suite-path", default="scripts/taler-monitoring")
|
||||||
|
ap.add_argument("--title", default="")
|
||||||
|
ap.add_argument("--link-other", default="")
|
||||||
|
ap.add_argument(
|
||||||
|
"--path-err",
|
||||||
|
default="/monitoring_err/",
|
||||||
|
help="URL path for error page (redirect target)",
|
||||||
|
)
|
||||||
|
ap.add_argument(
|
||||||
|
"--page-label",
|
||||||
|
default="monitoring",
|
||||||
|
help="Short name shown in titles/redirect box",
|
||||||
|
)
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
log_text = args.log.read_text(errors="replace") if args.log.is_file() else ""
|
||||||
|
now = datetime.now(timezone.utc)
|
||||||
|
generated_iso = now.strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||||
|
generated_at = now.strftime("%Y-%m-%d %H:%M:%SZ")
|
||||||
|
title = args.title or f"{args.page_label} {args.mode} · {args.hostname}"
|
||||||
|
|
||||||
|
raw_for_counts = [strip_ansi(l).rstrip("\n") for l in log_text.splitlines()]
|
||||||
|
n_err, n_warn, _, _ = count_status(raw_for_counts)
|
||||||
|
|
||||||
|
if args.mode == "redirect":
|
||||||
|
html_out = build_redirect_html(
|
||||||
|
hostname=args.hostname,
|
||||||
|
commit=args.commit,
|
||||||
|
commit_url=args.commit_url,
|
||||||
|
path_err=args.path_err,
|
||||||
|
page_label=args.page_label,
|
||||||
|
generated_at=generated_at,
|
||||||
|
generated_iso=generated_iso,
|
||||||
|
n_err=n_err,
|
||||||
|
n_warn=n_warn,
|
||||||
|
log_text=log_text,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
html_out = build_html(
|
||||||
|
title=title,
|
||||||
|
hostname=args.hostname,
|
||||||
|
mode=args.mode,
|
||||||
|
log_text=log_text,
|
||||||
|
commit=args.commit,
|
||||||
|
commit_url=args.commit_url,
|
||||||
|
suite_path=args.suite_path,
|
||||||
|
generated_at=generated_at,
|
||||||
|
generated_iso=generated_iso,
|
||||||
|
link_other=args.link_other or None,
|
||||||
|
page_label=args.page_label,
|
||||||
|
)
|
||||||
|
|
||||||
|
args.out.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
args.out.write_text(html_out, encoding="utf-8")
|
||||||
|
print(f"wrote {args.out} (errors={n_err} warnings={n_warn} level={status_level(n_err, n_warn)})")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
44
scripts/taler-monitoring/site-gen/deploy-monitoring-sites.sh
Executable file
44
scripts/taler-monitoring/site-gen/deploy-monitoring-sites.sh
Executable file
|
|
@ -0,0 +1,44 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# deploy-monitoring-sites.sh — rsync generated HTML to DEPLOY_WWW_ROOT on koopa.
|
||||||
|
# Does NOT reload Caddy (needs root — see ROOT-ON-KOOPA.md).
|
||||||
|
set -uo pipefail
|
||||||
|
|
||||||
|
ROOT=$(cd "$(dirname "$0")" && pwd)
|
||||||
|
SETTINGS="${SITE_GEN_SETTINGS:-$ROOT/settings.conf}"
|
||||||
|
[ -f "$SETTINGS" ] || SETTINGS="$ROOT/settings.conf.example"
|
||||||
|
# shellcheck disable=SC1090
|
||||||
|
set -a
|
||||||
|
source <(grep -v '^\s*#' "$SETTINGS" | grep -v '^\s*$' | sed 's/\r$//')
|
||||||
|
set +a
|
||||||
|
|
||||||
|
WORK_ROOT="${WORK_ROOT:-/tmp/taler-monitoring-sites-work}"
|
||||||
|
HTML_DIR="$WORK_ROOT/html"
|
||||||
|
DEPLOY_SSH="${DEPLOY_SSH:-koopa-external}"
|
||||||
|
DEPLOY_WWW_ROOT="${DEPLOY_WWW_ROOT:-/var/www/monitoring-sites}"
|
||||||
|
|
||||||
|
if [ ! -d "$HTML_DIR" ]; then
|
||||||
|
echo "no HTML_DIR $HTML_DIR — run generate-monitoring-sites.sh first" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "deploy $HTML_DIR → ${DEPLOY_SSH}:${DEPLOY_WWW_ROOT}/"
|
||||||
|
# Prefer writing to a hernani-owned staging dir if /var/www not writable
|
||||||
|
STAGE_REMOTE="${DEPLOY_STAGING:-/home/hernani/monitoring-sites-staging}"
|
||||||
|
|
||||||
|
ssh -o BatchMode=yes -o ConnectTimeout=20 "$DEPLOY_SSH" \
|
||||||
|
"mkdir -p '$STAGE_REMOTE' '$DEPLOY_WWW_ROOT' 2>/dev/null || mkdir -p '$STAGE_REMOTE'"
|
||||||
|
|
||||||
|
rsync -az --delete "$HTML_DIR/" "${DEPLOY_SSH}:${STAGE_REMOTE}/"
|
||||||
|
echo "staged on $DEPLOY_SSH:$STAGE_REMOTE"
|
||||||
|
echo
|
||||||
|
echo "If $DEPLOY_WWW_ROOT is root-owned, as root on koopa:"
|
||||||
|
echo " mkdir -p $DEPLOY_WWW_ROOT"
|
||||||
|
echo " rsync -a --delete $STAGE_REMOTE/ $DEPLOY_WWW_ROOT/"
|
||||||
|
echo " chown -R caddy:caddy $DEPLOY_WWW_ROOT # or root:caddy — match other www"
|
||||||
|
echo " # then Caddyfile handles + reload — see ROOT-ON-KOOPA.md"
|
||||||
|
echo
|
||||||
|
# try direct rsync to DEPLOY_WWW_ROOT if writable
|
||||||
|
if ssh -o BatchMode=yes "$DEPLOY_SSH" "test -w '$DEPLOY_WWW_ROOT' 2>/dev/null || test -w \"\$(dirname '$DEPLOY_WWW_ROOT')\""; then
|
||||||
|
rsync -az --delete "$HTML_DIR/" "${DEPLOY_SSH}:${DEPLOY_WWW_ROOT}/" && \
|
||||||
|
echo "also synced directly to $DEPLOY_WWW_ROOT"
|
||||||
|
fi
|
||||||
273
scripts/taler-monitoring/site-gen/generate-monitoring-sites.sh
Executable file
273
scripts/taler-monitoring/site-gen/generate-monitoring-sites.sh
Executable file
|
|
@ -0,0 +1,273 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# generate-monitoring-sites.sh — run taler-monitoring, build console HTML for
|
||||||
|
# /monitoring and /monitoring_err (static files for host Caddy later).
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# ./generate-monitoring-sites.sh # all SITES (or ONLY_HOSTS)
|
||||||
|
# ONLY_HOSTS="bank.hacktivism.ch taler.hacktivism.ch" ./generate-monitoring-sites.sh
|
||||||
|
# ./generate-monitoring-sites.sh --dry-run # HTML from empty/missing logs only
|
||||||
|
# ./generate-monitoring-sites.sh --skip-run # only convert existing logs
|
||||||
|
#
|
||||||
|
# Settings: settings.conf (from settings.conf.example). No secrets in settings.
|
||||||
|
# Secrets for e2e: ../secrets.env (optional; default phases are urls versions).
|
||||||
|
#
|
||||||
|
set -uo pipefail
|
||||||
|
|
||||||
|
export PYTHONUNBUFFERED=1
|
||||||
|
# Global reporting defaults (same as host-agent run-host-report.sh)
|
||||||
|
: "${RUN_TIMEOUT:=600}"
|
||||||
|
export RUN_TIMEOUT
|
||||||
|
|
||||||
|
ROOT=$(cd "$(dirname "$0")" && pwd)
|
||||||
|
SETTINGS="${SITE_GEN_SETTINGS:-$ROOT/settings.conf}"
|
||||||
|
EXAMPLE="$ROOT/settings.conf.example"
|
||||||
|
|
||||||
|
if [ ! -f "$SETTINGS" ]; then
|
||||||
|
if [ -f "$EXAMPLE" ]; then
|
||||||
|
echo "note: no $SETTINGS — using example defaults (copy to settings.conf to customize)"
|
||||||
|
SETTINGS="$EXAMPLE"
|
||||||
|
else
|
||||||
|
echo "missing settings: $EXAMPLE" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Load KEY=value lines; allow SITES already set in environment to win.
|
||||||
|
# Multiline SITES in the conf file: use SITES_FILE= or env SITES= instead.
|
||||||
|
while IFS= read -r _line || [ -n "$_line" ]; do
|
||||||
|
_line=${_line%%#*}
|
||||||
|
_line=$(echo "$_line" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')
|
||||||
|
[ -z "$_line" ] && continue
|
||||||
|
case "$_line" in
|
||||||
|
SITES=*|SITES_FILE=*) continue ;; # handled below / env
|
||||||
|
*=*)
|
||||||
|
_k=${_line%%=*}
|
||||||
|
_v=${_line#*=}
|
||||||
|
# do not override pre-set env
|
||||||
|
if [ -z "${!_k+x}" ] 2>/dev/null || [ -z "${!_k}" ]; then
|
||||||
|
export "$_k=$_v"
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
done < "$SETTINGS"
|
||||||
|
|
||||||
|
# SITES from dedicated file or keep env; else parse heredoc-style from example list
|
||||||
|
if [ -n "${SITES_FILE:-}" ] && [ -f "$SITES_FILE" ]; then
|
||||||
|
SITES=$(grep -v '^\s*#' "$SITES_FILE" | grep -v '^\s*$')
|
||||||
|
export SITES
|
||||||
|
elif [ -z "${SITES:-}" ]; then
|
||||||
|
# default 9 fronts
|
||||||
|
SITES="bank.hacktivism.ch|hacktivism.ch
|
||||||
|
exchange.hacktivism.ch|hacktivism.ch
|
||||||
|
taler.hacktivism.ch|hacktivism.ch
|
||||||
|
stage.bank.lefrancpaysan.ch|stage.lefrancpaysan.ch
|
||||||
|
stage.exchange.lefrancpaysan.ch|stage.lefrancpaysan.ch
|
||||||
|
stage.monnaie.lefrancpaysan.ch|stage.lefrancpaysan.ch
|
||||||
|
bank.lefrancpaysan.ch|lefrancpaysan.ch
|
||||||
|
exchange.lefrancpaysan.ch|lefrancpaysan.ch
|
||||||
|
monnaie.lefrancpaysan.ch|lefrancpaysan.ch"
|
||||||
|
export SITES
|
||||||
|
fi
|
||||||
|
|
||||||
|
SKIP_RUN=0
|
||||||
|
DRY=0
|
||||||
|
while [ $# -gt 0 ]; do
|
||||||
|
case "$1" in
|
||||||
|
--skip-run) SKIP_RUN=1; shift ;;
|
||||||
|
--dry-run) DRY=1; SKIP_RUN=1; shift ;;
|
||||||
|
-h|--help) sed -n '1,20p' "$0"; exit 0 ;;
|
||||||
|
*) echo "unknown arg: $1" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
MONITORING_ROOT=$(cd "$ROOT/${MONITORING_ROOT:-..}" && pwd)
|
||||||
|
WORK_ROOT="${WORK_ROOT:-/tmp/taler-monitoring-sites-work}"
|
||||||
|
LOG_DIR="$WORK_ROOT/logs"
|
||||||
|
HTML_DIR="$WORK_ROOT/html"
|
||||||
|
mkdir -p "$LOG_DIR" "$HTML_DIR"
|
||||||
|
|
||||||
|
# commit of monitoring suite used for this run
|
||||||
|
if git -C "$MONITORING_ROOT/../.." rev-parse HEAD >/dev/null 2>&1; then
|
||||||
|
REPO_ROOT=$(cd "$MONITORING_ROOT/../.." && pwd)
|
||||||
|
elif git -C "$MONITORING_ROOT" rev-parse HEAD >/dev/null 2>&1; then
|
||||||
|
REPO_ROOT=$(cd "$MONITORING_ROOT" && pwd)
|
||||||
|
else
|
||||||
|
REPO_ROOT=$(cd "$ROOT/../../.." && pwd)
|
||||||
|
fi
|
||||||
|
COMMIT=$(git -C "$REPO_ROOT" rev-parse HEAD 2>/dev/null || echo unknown)
|
||||||
|
COMMIT_SHORT=$(git -C "$REPO_ROOT" rev-parse --short=12 HEAD 2>/dev/null || echo unknown)
|
||||||
|
SOURCE_REPO_WEB="${SOURCE_REPO_WEB:-https://git.hacktivism.ch/hernani/koopa-admin-log}"
|
||||||
|
TMPL="${SOURCE_COMMIT_URL_TMPL:-\{repo\}/src/commit/\{commit\}}"
|
||||||
|
COMMIT_URL=$(printf '%s' "$TMPL" | sed "s|{repo}|$SOURCE_REPO_WEB|g; s|{commit}|$COMMIT|g")
|
||||||
|
SUITE_PATH="${SOURCE_SUITE_PATH:-scripts/taler-monitoring}"
|
||||||
|
|
||||||
|
# Outside-only public checks (no SSH into koopa/stage containers)
|
||||||
|
PHASES_ERR="${PHASES_ERR:-urls versions}"
|
||||||
|
PHASES_OK="${PHASES_OK:-urls versions}"
|
||||||
|
CONTINUE_ON_ERROR="${CONTINUE_ON_ERROR:-1}"
|
||||||
|
SKIP_SSH="${SKIP_SSH:-1}"
|
||||||
|
# Drop SSH phases if someone passes "all" / inside / server / e2e by mistake
|
||||||
|
filter_phases_outside() {
|
||||||
|
local out=() p
|
||||||
|
for p in "$@"; do
|
||||||
|
case "$p" in
|
||||||
|
inside|server|e2e|ladder|goa-ladder|auth401) continue ;;
|
||||||
|
all) out+=(urls versions);;
|
||||||
|
*) out+=("$p") ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
printf '%s\n' "${out[@]}"
|
||||||
|
}
|
||||||
|
|
||||||
|
run_mon() {
|
||||||
|
local domain="$1" phases="$2" logfile="$3" cont="$4"
|
||||||
|
# shellcheck disable=SC2206
|
||||||
|
local ph=( $phases )
|
||||||
|
mapfile -t ph < <(filter_phases_outside "${ph[@]}")
|
||||||
|
[ "${#ph[@]}" -gt 0 ] || ph=(urls versions)
|
||||||
|
|
||||||
|
: >"$logfile"
|
||||||
|
echo "+ -d $domain · phases=${ph[*]} · CONTINUE=$cont · SKIP_SSH=$SKIP_SSH · RUN_TIMEOUT=$RUN_TIMEOUT"
|
||||||
|
|
||||||
|
if [ -n "${RUNNER_SSH:-}" ]; then
|
||||||
|
echo "+ via ssh $RUNNER_SSH"
|
||||||
|
rsync -az --delete \
|
||||||
|
--exclude secrets.env \
|
||||||
|
--exclude 'android-test/artifacts' \
|
||||||
|
--exclude '.git' \
|
||||||
|
"$MONITORING_ROOT/" \
|
||||||
|
"${RUNNER_SSH}:${RUNNER_REMOTE_WORKDIR}/" \
|
||||||
|
|| return 1
|
||||||
|
# shellcheck disable=SC2029
|
||||||
|
if command -v stdbuf >/dev/null 2>&1; then
|
||||||
|
ssh -o BatchMode=yes -o ConnectTimeout=30 "$RUNNER_SSH" \
|
||||||
|
"cd '${RUNNER_REMOTE_WORKDIR}' && chmod +x taler-monitoring.sh check_*.sh 2>/dev/null; \
|
||||||
|
export CONTINUE_ON_ERROR='$cont' AUTH401_CONTINUE='$cont' SKIP_SSH='$SKIP_SSH' RUN_TIMEOUT='$RUN_TIMEOUT'; \
|
||||||
|
stdbuf -oL -eL ./taler-monitoring.sh -d '$domain' ${ph[*]}" \
|
||||||
|
2>&1 | stdbuf -oL -eL tee -a "$logfile"
|
||||||
|
else
|
||||||
|
ssh -o BatchMode=yes -o ConnectTimeout=30 "$RUNNER_SSH" \
|
||||||
|
"cd '${RUNNER_REMOTE_WORKDIR}' && chmod +x taler-monitoring.sh check_*.sh 2>/dev/null; \
|
||||||
|
export CONTINUE_ON_ERROR='$cont' AUTH401_CONTINUE='$cont' SKIP_SSH='$SKIP_SSH' RUN_TIMEOUT='$RUN_TIMEOUT'; \
|
||||||
|
./taler-monitoring.sh -d '$domain' ${ph[*]}" \
|
||||||
|
2>&1 | tee -a "$logfile"
|
||||||
|
fi
|
||||||
|
return "${PIPESTATUS[0]}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if command -v stdbuf >/dev/null 2>&1; then
|
||||||
|
( cd "$MONITORING_ROOT" && \
|
||||||
|
CONTINUE_ON_ERROR="$cont" AUTH401_CONTINUE="$cont" SKIP_SSH="$SKIP_SSH" \
|
||||||
|
RUN_TIMEOUT="$RUN_TIMEOUT" \
|
||||||
|
stdbuf -oL -eL ./taler-monitoring.sh -d "$domain" "${ph[@]}" ) \
|
||||||
|
2>&1 | stdbuf -oL -eL tee -a "$logfile"
|
||||||
|
else
|
||||||
|
( cd "$MONITORING_ROOT" && \
|
||||||
|
CONTINUE_ON_ERROR="$cont" AUTH401_CONTINUE="$cont" SKIP_SSH="$SKIP_SSH" \
|
||||||
|
RUN_TIMEOUT="$RUN_TIMEOUT" \
|
||||||
|
./taler-monitoring.sh -d "$domain" "${ph[@]}" ) \
|
||||||
|
2>&1 | tee -a "$logfile"
|
||||||
|
fi
|
||||||
|
return "${PIPESTATUS[0]}"
|
||||||
|
}
|
||||||
|
|
||||||
|
htmlify() {
|
||||||
|
local host="$1" mode="$2" log="$3" out="$4" other="$5"
|
||||||
|
mkdir -p "$(dirname "$out")"
|
||||||
|
if [ ! -f "$ROOT/console_to_html.py" ]; then
|
||||||
|
echo "WARN: console_to_html.py missing — bootstrap $out"
|
||||||
|
{
|
||||||
|
echo "<!DOCTYPE html><html><head><meta charset=utf-8><title>$host $mode</title></head>"
|
||||||
|
echo "<body><h1>$host · $mode</h1><p>commit <a href=\"$COMMIT_URL\">$COMMIT_SHORT</a></p>"
|
||||||
|
echo "<pre>"
|
||||||
|
tail -n 100 "$log" 2>/dev/null | sed 's/&/\&/g;s/</\</g' || true
|
||||||
|
echo "</pre></body></html>"
|
||||||
|
} >"$out"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
python3 "$ROOT/console_to_html.py" \
|
||||||
|
--log "$log" \
|
||||||
|
--out "$out" \
|
||||||
|
--hostname "$host" \
|
||||||
|
--mode "$mode" \
|
||||||
|
--commit "$COMMIT" \
|
||||||
|
--commit-url "$COMMIT_URL" \
|
||||||
|
--suite-path "$SUITE_PATH" \
|
||||||
|
--link-other "$other"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Parse SITES
|
||||||
|
mapfile -t SITE_LINES < <(printf '%s\n' "$SITES" | sed '/^\s*$/d' | sed 's/#.*//')
|
||||||
|
ONLY="${ONLY_HOSTS:-}"
|
||||||
|
|
||||||
|
for line in "${SITE_LINES[@]}"; do
|
||||||
|
line=$(echo "$line" | tr -d ' \t')
|
||||||
|
[ -z "$line" ] && continue
|
||||||
|
host=${line%%|*}
|
||||||
|
domain=${line#*|}
|
||||||
|
if [ -n "$ONLY" ]; then
|
||||||
|
echo " $ONLY " | grep -q " $host " || continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "======== $host (domain=$domain) commit=$COMMIT_SHORT ========"
|
||||||
|
base_html="$HTML_DIR/$host"
|
||||||
|
mkdir -p "$base_html/monitoring" "$base_html/monitoring_err"
|
||||||
|
log_err="$LOG_DIR/${host}.err.log"
|
||||||
|
log_ok="$LOG_DIR/${host}.ok.log"
|
||||||
|
ec_err=1
|
||||||
|
ec_ok=1
|
||||||
|
|
||||||
|
if [ "$SKIP_RUN" != "1" ]; then
|
||||||
|
run_mon "$domain" "$PHASES_ERR" "$log_err" "$CONTINUE_ON_ERROR"
|
||||||
|
ec_err=$?
|
||||||
|
# ok-run without continue (strict)
|
||||||
|
run_mon "$domain" "$PHASES_OK" "$log_ok" "0"
|
||||||
|
ec_ok=$?
|
||||||
|
else
|
||||||
|
[ -f "$log_err" ] || : >"$log_err"
|
||||||
|
[ -f "$log_ok" ] || : >"$log_ok"
|
||||||
|
# infer exit from log if present
|
||||||
|
if grep -q '\[ ERROR \]' "$log_err" 2>/dev/null || grep -q 'ERROR' "$log_err" 2>/dev/null; then
|
||||||
|
ec_err=1
|
||||||
|
else
|
||||||
|
ec_err=0
|
||||||
|
fi
|
||||||
|
ec_ok=$ec_err
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Always write pages (first run / empty logs too)
|
||||||
|
[ -f "$log_err" ] || : >"$log_err"
|
||||||
|
[ -f "$log_ok" ] || : >"$log_ok"
|
||||||
|
|
||||||
|
htmlify "$host" "err" "$log_err" \
|
||||||
|
"$base_html/monitoring_err/index.html" "/monitoring/"
|
||||||
|
|
||||||
|
if [ "$ec_ok" -eq 0 ] && [ "$ec_err" -eq 0 ]; then
|
||||||
|
htmlify "$host" "ok" "$log_ok" \
|
||||||
|
"$base_html/monitoring/index.html" "/monitoring_err/"
|
||||||
|
echo " → /monitoring (clean) + /monitoring_err"
|
||||||
|
else
|
||||||
|
htmlify "$host" "redirect" "$log_err" \
|
||||||
|
"$base_html/monitoring/index.html" "/monitoring_err/"
|
||||||
|
echo " → /monitoring (redirect stub) + /monitoring_err (errors) ec_err=$ec_err ec_ok=$ec_ok"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# first-run safety
|
||||||
|
if [ ! -f "$base_html/monitoring/index.html" ]; then
|
||||||
|
htmlify "$host" "err" "$log_err" \
|
||||||
|
"$base_html/monitoring/index.html" "/monitoring_err/"
|
||||||
|
fi
|
||||||
|
|
||||||
|
printf '%s\n' "$COMMIT" >"$base_html/COMMIT"
|
||||||
|
printf '%s\n' "$COMMIT_URL" >"$base_html/COMMIT_URL"
|
||||||
|
printf 'host=%s domain=%s ec_err=%s ec_ok=%s commit=%s run_timeout=%s\n' \
|
||||||
|
"$host" "$domain" "$ec_err" "$ec_ok" "$COMMIT" "$RUN_TIMEOUT" | tee "$base_html/STATUS.txt"
|
||||||
|
done
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "HTML under $HTML_DIR"
|
||||||
|
echo "commit $COMMIT_SHORT $COMMIT_URL"
|
||||||
|
echo "RUN_TIMEOUT=${RUN_TIMEOUT}s (global; 0=unlimited)"
|
||||||
|
echo "Deploy (as hernani, may need sudo for /var/www):"
|
||||||
|
echo " ./deploy-monitoring-sites.sh"
|
||||||
|
echo "Root on koopa: see ROOT-ON-KOOPA.md"
|
||||||
79
scripts/taler-monitoring/site-gen/install-firecuda-timer.sh
Executable file
79
scripts/taler-monitoring/site-gen/install-firecuda-timer.sh
Executable file
|
|
@ -0,0 +1,79 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# install-firecuda-timer.sh — OPTIONAL: rsync suite to firecuda + launchd every 4h
|
||||||
|
#
|
||||||
|
# POLICY (2026-07): firecuda launchd is **disabled**. Prefer koopa host-agent.
|
||||||
|
# This script remains for documentation / future re-enable only.
|
||||||
|
# Do not run unless you intentionally want a second outside-only scheduler.
|
||||||
|
#
|
||||||
|
# ./install-firecuda-timer.sh
|
||||||
|
# ./install-firecuda-timer.sh --run-now
|
||||||
|
#
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
ROOT=$(cd "$(dirname "$0")" && pwd)
|
||||||
|
MON_ROOT=$(cd "$ROOT/.." && pwd)
|
||||||
|
FIRECUDA="${FIRECUDA_SSH:-firecuda-external}"
|
||||||
|
REMOTE_BASE="${FIRECUDA_INSTALL_DIR:-taler-monitoring-site-gen}"
|
||||||
|
RUN_NOW=0
|
||||||
|
[ "${1:-}" = "--run-now" ] && RUN_NOW=1
|
||||||
|
|
||||||
|
echo "install → $FIRECUDA:~/$REMOTE_BASE (outside-only, every 4h)"
|
||||||
|
|
||||||
|
ssh -o BatchMode=yes -o ConnectTimeout=20 "$FIRECUDA" \
|
||||||
|
"mkdir -p \"\$HOME/$REMOTE_BASE\" \"\$HOME/Library/Logs/taler-monitoring-sites\" \"\$HOME/var/taler-monitoring-sites-work\" \"\$HOME/Library/LaunchAgents\""
|
||||||
|
|
||||||
|
# Sync full taler-monitoring suite (no secrets.env)
|
||||||
|
rsync -az --delete \
|
||||||
|
--exclude secrets.env \
|
||||||
|
--exclude 'android-test/artifacts' \
|
||||||
|
--exclude '.git' \
|
||||||
|
--exclude 'site-gen/settings.conf' \
|
||||||
|
"$MON_ROOT/" \
|
||||||
|
"${FIRECUDA}:$REMOTE_BASE/"
|
||||||
|
|
||||||
|
# settings on firecuda: outside only, run locally on firecuda, deploy to koopa
|
||||||
|
ssh -o BatchMode=yes "$FIRECUDA" "cat > \"\$HOME/$REMOTE_BASE/site-gen/settings.conf\" <<'EOF'
|
||||||
|
# auto-installed by install-firecuda-timer.sh — no secrets
|
||||||
|
RUNNER_SSH=
|
||||||
|
SKIP_SSH=1
|
||||||
|
CONTINUE_ON_ERROR=1
|
||||||
|
PHASES_ERR=urls versions
|
||||||
|
PHASES_OK=urls versions
|
||||||
|
DEPLOY_SSH=koopa-external
|
||||||
|
DEPLOY_STAGING=/home/hernani/monitoring-sites-staging
|
||||||
|
WORK_ROOT=
|
||||||
|
MONITORING_ROOT=..
|
||||||
|
SOURCE_REPO_WEB=https://git.hacktivism.ch/hernani/koopa-admin-log
|
||||||
|
SOURCE_COMMIT_URL_TMPL={repo}/src/commit/{commit}
|
||||||
|
SOURCE_SUITE_PATH=scripts/taler-monitoring
|
||||||
|
EOF
|
||||||
|
# WORK_ROOT expanded on remote
|
||||||
|
sed -i.bak \"s|^WORK_ROOT=.*|WORK_ROOT=\$HOME/var/taler-monitoring-sites-work|\" \"\$HOME/$REMOTE_BASE/site-gen/settings.conf\" 2>/dev/null || \
|
||||||
|
sed -i '' \"s|^WORK_ROOT=.*|WORK_ROOT=\$HOME/var/taler-monitoring-sites-work|\" \"\$HOME/$REMOTE_BASE/site-gen/settings.conf\"
|
||||||
|
"
|
||||||
|
|
||||||
|
# launchd plist with absolute paths
|
||||||
|
ssh -o BatchMode=yes "$FIRECUDA" bash -s <<REMOTE
|
||||||
|
set -euo pipefail
|
||||||
|
HOME_DIR=\$HOME
|
||||||
|
SITE_GEN="\$HOME_DIR/$REMOTE_BASE/site-gen"
|
||||||
|
PLIST_SRC="\$SITE_GEN/com.hacktivism.taler-monitoring-sites.plist"
|
||||||
|
PLIST_DST="\$HOME_DIR/Library/LaunchAgents/com.hacktivism.taler-monitoring-sites.plist"
|
||||||
|
sed -e "s|__SITE_GEN__|\${SITE_GEN}|g" -e "s|__HOME__|\${HOME_DIR}|g" "\$PLIST_SRC" > "\$PLIST_DST"
|
||||||
|
chmod +x "\$SITE_GEN"/*.sh "\$SITE_GEN"/console_to_html.py 2>/dev/null || true
|
||||||
|
chmod +x "\$SITE_GEN/run-on-firecuda.sh"
|
||||||
|
launchctl bootout "gui/\$(id -u)/com.hacktivism.taler-monitoring-sites" 2>/dev/null || true
|
||||||
|
launchctl unload "\$PLIST_DST" 2>/dev/null || true
|
||||||
|
launchctl bootstrap "gui/\$(id -u)" "\$PLIST_DST" 2>/dev/null || launchctl load "\$PLIST_DST"
|
||||||
|
launchctl enable "gui/\$(id -u)/com.hacktivism.taler-monitoring-sites" 2>/dev/null || true
|
||||||
|
launchctl print "gui/\$(id -u)/com.hacktivism.taler-monitoring-sites" 2>/dev/null | head -20 || launchctl list | grep taler-monitoring || true
|
||||||
|
echo "OK: launchd com.hacktivism.taler-monitoring-sites (StartInterval=14400)"
|
||||||
|
REMOTE
|
||||||
|
|
||||||
|
if [ "$RUN_NOW" = "1" ]; then
|
||||||
|
echo "run once now…"
|
||||||
|
ssh -o BatchMode=yes "$FIRECUDA" "\"\$HOME/$REMOTE_BASE/site-gen/run-on-firecuda.sh\"" || true
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "done. logs: $FIRECUDA:~/Library/Logs/taler-monitoring-sites/"
|
||||||
|
echo "manual: ssh $FIRECUDA '~/taler-monitoring-site-gen/site-gen/run-on-firecuda.sh'"
|
||||||
15
scripts/taler-monitoring/site-gen/pull-firecuda-to-koopa.sh
Executable file
15
scripts/taler-monitoring/site-gen/pull-firecuda-to-koopa.sh
Executable file
|
|
@ -0,0 +1,15 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# pull-firecuda-to-koopa.sh — copy HTML from firecuda work dir → koopa staging
|
||||||
|
# Run on laptop (hernani) where both SSH aliases work.
|
||||||
|
set -euo pipefail
|
||||||
|
FIRECUDA="${FIRECUDA_SSH:-firecuda-external}"
|
||||||
|
KOOPA="${DEPLOY_SSH:-koopa-external}"
|
||||||
|
REMOTE_HTML="${FIRECUDA_HTML:-var/taler-monitoring-sites-work/html/}"
|
||||||
|
STAGE="${DEPLOY_STAGING:-/home/hernani/monitoring-sites-staging}"
|
||||||
|
|
||||||
|
echo "firecuda:$REMOTE_HTML → $KOOPA:$STAGE"
|
||||||
|
ssh -o BatchMode=yes -o ConnectTimeout=20 "$KOOPA" "mkdir -p '$STAGE'"
|
||||||
|
rsync -az --delete \
|
||||||
|
"${FIRECUDA}:${REMOTE_HTML}" \
|
||||||
|
"${KOOPA}:${STAGE}/"
|
||||||
|
echo "OK staged. Root: rsync to /var/www/monitoring-sites (see ROOT-ON-KOOPA.md)"
|
||||||
59
scripts/taler-monitoring/site-gen/run-on-firecuda.sh
Executable file
59
scripts/taler-monitoring/site-gen/run-on-firecuda.sh
Executable file
|
|
@ -0,0 +1,59 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# run-on-firecuda.sh — entrypoint for launchd/cron on firecuda-external.
|
||||||
|
# Outside-only monitoring (SKIP_SSH=1), all 9 sites, then stage HTML to koopa.
|
||||||
|
#
|
||||||
|
# Installed path on firecuda (default):
|
||||||
|
# ~/taler-monitoring-site-gen/site-gen/run-on-firecuda.sh
|
||||||
|
#
|
||||||
|
set -uo pipefail
|
||||||
|
|
||||||
|
export PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:${PATH:-}"
|
||||||
|
|
||||||
|
ROOT=$(cd "$(dirname "$0")" && pwd)
|
||||||
|
LOG_DIR="${FIRECUDA_LOG_DIR:-$HOME/Library/Logs/taler-monitoring-sites}"
|
||||||
|
mkdir -p "$LOG_DIR"
|
||||||
|
STAMP=$(date +%Y%m%d-%H%M%S)
|
||||||
|
LOG="$LOG_DIR/run-${STAMP}.log"
|
||||||
|
exec >>"$LOG" 2>&1
|
||||||
|
|
||||||
|
echo "======== $(date -Iseconds 2>/dev/null || date) firecuda monitoring sites ========"
|
||||||
|
|
||||||
|
# Always: public DNS only, no SSH checks into stacks
|
||||||
|
export SKIP_SSH=1
|
||||||
|
export CONTINUE_ON_ERROR=1
|
||||||
|
export AUTH401_CONTINUE=1
|
||||||
|
export RUNNER_SSH=
|
||||||
|
export PHASES_ERR="${PHASES_ERR:-urls versions}"
|
||||||
|
export PHASES_OK="${PHASES_OK:-urls versions}"
|
||||||
|
export WORK_ROOT="${WORK_ROOT:-$HOME/var/taler-monitoring-sites-work}"
|
||||||
|
export MONITORING_ROOT="${MONITORING_ROOT:-$ROOT/..}"
|
||||||
|
export DEPLOY_SSH="${DEPLOY_SSH:-koopa-external}"
|
||||||
|
export DEPLOY_STAGING="${DEPLOY_STAGING:-/home/hernani/monitoring-sites-staging}"
|
||||||
|
|
||||||
|
# Prefer full 9-site list from settings or defaults inside generate script
|
||||||
|
unset ONLY_HOSTS
|
||||||
|
|
||||||
|
cd "$ROOT" || exit 1
|
||||||
|
chmod +x generate-monitoring-sites.sh deploy-monitoring-sites.sh console_to_html.py 2>/dev/null || true
|
||||||
|
|
||||||
|
./generate-monitoring-sites.sh
|
||||||
|
ec=$?
|
||||||
|
|
||||||
|
# HTML stays on firecuda under WORK_ROOT/html.
|
||||||
|
# Optional push to koopa if this host can SSH there (often only from laptop):
|
||||||
|
if [ -n "${DEPLOY_SSH:-}" ] && [ "${DEPLOY_FROM_FIRECUDA:-0}" = "1" ]; then
|
||||||
|
if [ -x ./deploy-monitoring-sites.sh ]; then
|
||||||
|
./deploy-monitoring-sites.sh || echo "WARN: deploy staging failed"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
echo "note: HTML at ${WORK_ROOT}/html — pull from laptop:"
|
||||||
|
echo " rsync -az firecuda-external:var/taler-monitoring-sites-work/html/ \\"
|
||||||
|
echo " koopa-external:monitoring-sites-staging/"
|
||||||
|
echo " # or: ./pull-firecuda-to-koopa.sh"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# keep last 20 logs
|
||||||
|
ls -1t "$LOG_DIR"/run-*.log 2>/dev/null | tail -n +21 | xargs rm -f 2>/dev/null || true
|
||||||
|
|
||||||
|
echo "======== done ec=$ec ========"
|
||||||
|
exit "$ec"
|
||||||
55
scripts/taler-monitoring/site-gen/settings.conf.example
Normal file
55
scripts/taler-monitoring/site-gen/settings.conf.example
Normal file
|
|
@ -0,0 +1,55 @@
|
||||||
|
# monitoring site-gen — non-secret settings (safe as example)
|
||||||
|
# Copy: cp settings.conf.example settings.conf
|
||||||
|
# Never put passwords/tokens here.
|
||||||
|
|
||||||
|
# On firecuda the suite runs *locally* (RUNNER_SSH empty). From a laptop you
|
||||||
|
# can set RUNNER_SSH=firecuda-external to drive a one-off remote run instead.
|
||||||
|
RUNNER_SSH=
|
||||||
|
RUNNER_SSH_FALLBACKS=
|
||||||
|
RUNNER_REMOTE_WORKDIR=/tmp/taler-monitoring-site-gen
|
||||||
|
MONITORING_ROOT=..
|
||||||
|
|
||||||
|
# Outside-only: never SSH into bank/exchange/merchant containers
|
||||||
|
SKIP_SSH=1
|
||||||
|
|
||||||
|
# Deploy staging on koopa (host Caddy later — root copies to /var/www)
|
||||||
|
DEPLOY_SSH=koopa-external
|
||||||
|
DEPLOY_SSH_FALLBACKS=koopa
|
||||||
|
# Static files for Caddy file_server (created per hostname)
|
||||||
|
DEPLOY_WWW_ROOT=/var/www/monitoring-sites
|
||||||
|
DEPLOY_SUDO=sudo
|
||||||
|
CADDY_CONFIG=/etc/caddy/Caddyfile
|
||||||
|
CADDY_MIRROR_REL=configs/caddy/Caddyfile
|
||||||
|
# Working tree on deploy host for apply (optional)
|
||||||
|
KOOPA_CADDY_DIR=/home/hernani/koopa-caddy
|
||||||
|
|
||||||
|
# Source link for HTML footer (Forgejo)
|
||||||
|
SOURCE_REPO_WEB=https://git.hacktivism.ch/hernani/koopa-admin-log
|
||||||
|
SOURCE_COMMIT_URL_TMPL={repo}/src/commit/{commit}
|
||||||
|
SOURCE_SUITE_PATH=scripts/taler-monitoring
|
||||||
|
|
||||||
|
CONTINUE_ON_ERROR=1
|
||||||
|
PHASES_ERR=urls versions
|
||||||
|
PHASES_OK=urls versions
|
||||||
|
SKIP_E2E=1
|
||||||
|
|
||||||
|
# Local work dir for logs + HTML
|
||||||
|
WORK_ROOT=/tmp/taler-monitoring-sites-work
|
||||||
|
|
||||||
|
# Sites: hostname|domain_profile (9 fronts)
|
||||||
|
# Start with hacktivism; stage + prod LFP follow
|
||||||
|
SITES="
|
||||||
|
bank.hacktivism.ch|hacktivism.ch
|
||||||
|
exchange.hacktivism.ch|hacktivism.ch
|
||||||
|
taler.hacktivism.ch|hacktivism.ch
|
||||||
|
stage.bank.lefrancpaysan.ch|stage.lefrancpaysan.ch
|
||||||
|
stage.exchange.lefrancpaysan.ch|stage.lefrancpaysan.ch
|
||||||
|
stage.monnaie.lefrancpaysan.ch|stage.lefrancpaysan.ch
|
||||||
|
bank.lefrancpaysan.ch|lefrancpaysan.ch
|
||||||
|
exchange.lefrancpaysan.ch|lefrancpaysan.ch
|
||||||
|
monnaie.lefrancpaysan.ch|lefrancpaysan.ch
|
||||||
|
"
|
||||||
|
|
||||||
|
# Only process this subset (space-separated hostnames). Empty = all SITES.
|
||||||
|
# Example for first deploy: ONLY_HOSTS=bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch
|
||||||
|
ONLY_HOSTS=
|
||||||
Loading…
Add table
Add a link
Reference in a new issue