feat: LLM stack status monitord catalogs and HTML
This commit is contained in:
Hernâni Marques
parent
6200a4d031
commit
666dca1f0d
7 changed files with 6943 additions and 0 deletions
|
|
@ -1,14 +1,97 @@
|
|||
# site-gen — HTML converter + Caddy notes
|
||||
|
||||
**Product:** **LLM** = **Lightweight Monitoring** (Taler-stack watcher). CLI/tree still `taler-monitoring` / `taler-monitord`.
|
||||
|
||||
**Required for host-agent:** `console_to_html.py` (log → sticky HTML).
|
||||
|
||||
| File | Role |
|
||||
|------|------|
|
||||
| `console_to_html.py` | Convert suite console log → public mon HTML |
|
||||
| `stack_status_lib.py` | Domain-stack traffic lights + aggregation |
|
||||
| `taler-monitord` | Daemon: probe stacks → `stack-status.json` (UI detached) |
|
||||
| `stack_status_html.py` | Render status cards from JSON only |
|
||||
| `caddy-monitoring-handles.snippet` | Caddy `handle` / bare redir notes |
|
||||
| `ROOT-ON-KOOPA.md` | Publish staging → `/var/www` (paths from env) |
|
||||
| `settings.conf.example` | Optional settings for **legacy** multi-host generator |
|
||||
|
||||
## Stack status (taler-monitord)
|
||||
|
||||
Like `llm-controld`: the daemon collects; the page only reads JSON.
|
||||
|
||||
```bash
|
||||
./site-gen/taler-monitord --once
|
||||
./site-gen/stack_status_html.py --out /tmp/taler-stack-status.html
|
||||
# optional: ./site-gen/taler-monitord --interval 20
|
||||
# local GOA (hacktivism.ch / taler.hacktivism.ch) is included by default;
|
||||
# skip with --no-include-local or TALER_MONITORD_INCLUDE_LOCAL=0
|
||||
```
|
||||
|
||||
Lights: green ok · blue present · yellow soft · orange hard/recent · red acute · black long · white gone.
|
||||
Box colour = weakest member; **muted** if partial; **full** if all members red/black.
|
||||
|
||||
**Total score** (header): `N/M green` across stack+service+mail cards. Ratio thresholds (env override):
|
||||
1.0 green · ≥0.85 blue · ≥0.60 yellow · ≥0.35 orange · ≥0.01 red · 0 white (gone) / all-black → black.
|
||||
|
||||
Stacks with currency `-` in `domains.conf` are **`static_pending`** (e.g. lefrancpaysan.ch: static site, no Taler prod yet). monitord probes the apex as `site` and treats missing bank/exchange/merchant as **blue `no_prod_yet`**, not orange/red. Stage TESTPAYSAN remains a normal Taler stack.
|
||||
|
||||
**Bank backend:** exchange `/keys` `payto://` methods decide whether `bank.*` is expected.
|
||||
`payto://x-taler-bank/…` → probe bank (libeufin).
|
||||
`payto://iban/…` only (EBICS / TOPS CHF) → bank member **blue**, host **`IBAN`**, reason `ebics_iban` (not red).
|
||||
Mixed (e.g. regio) still probes bank. Override in `domains.conf`: bank column `-` / `ebics` / `none`.
|
||||
|
||||
**Network / SSH:** each stack gets a coarse **zone** (GOA/local, FP stage, TOPS, pixel, firefly, public demo).
|
||||
SSH = **TCP:22 banner** (service present), not Magikoopa jump login.
|
||||
Zone-mapped SSH targets come **only** from `~/.config/taler-monitoring/env` (`TALER_MONITORD_SSH_TOPS` / `_FP` / `_GOA` / `_TOPS_PROD` via **taler-monitoring-env**); hostnames are **hidden** in the UI when set. Without env, probes use public stack FQDNs. Disable: `TALER_MONITORD_SSH=0`.
|
||||
|
||||
**Outside CVEs:** `Server` headers (nginx/apache/caddy + version) → OSV.
|
||||
**CRITICAL → red**, **HIGH → orange**. UI: CVE tags **link to osv.dev**; card click opens detail (severity / version / summary). Logos for nginx/apache/caddy. Disable: `TALER_MONITORD_CVE=0`.
|
||||
|
||||
**Why Degraded:** each card shows a one-line aggregation (`4× OK, 3× Soft → Degraded`); click opens a dialog with per-member reasons.
|
||||
|
||||
**Company / services:** `services-catalog.conf` (ERPnext, MantisBT, company apexes, …). Disable: `TALER_MONITORD_SERVICES=0`.
|
||||
|
||||
**Mail:** `mail-catalog.conf` → MX / SPF / DMARC / A+AAAA / rDNS + SMTP/IMAP TCP ports (587 soft).
|
||||
**Deliverability** (regular, not every cycle): SMTP `MAIL FROM`/`RCPT TO` smoke to `postmaster@domain` (no DATA).
|
||||
Cached under monitord state (`mail-deliverability.json`); default interval **6h**, stale after **30h** → yellow.
|
||||
Send/server failures → orange/red (degraded). Env: `TALER_MONITORD_DELIVER=0`, `_INTERVAL_S`, `_STALE_S`, `_FROM`, `_RCPT`.
|
||||
Disable plane: `TALER_MONITORD_MAIL=0`; ports only: `TALER_MONITORD_MAIL_PORTS=0`.
|
||||
|
||||
**TLS certs + orphans:** cert days-left + SANs. Orphans = **DNS-missing** SANs not covered by monitored hosts; wildcards/siblings skipped.
|
||||
HTTP orphans opt-in: `TALER_MONITORD_ORPHAN_HTTP=1`. Disable: `TALER_MONITORD_CERT=0` / `TALER_MONITORD_ORPHAN=0`.
|
||||
|
||||
**Port scan (outside-in only):** TCP connect to notable ports on publicly named hosts. No internal firewall maps required.
|
||||
Baseline allows 80/443 (+ mail ports on mail-catalog hosts). Anything else open → **suspicious** until listed in `ports-whitelist.conf`.
|
||||
Many odd open ports ⇒ worth investigating. Whitelist only silences *known intentional public* opens (e.g. hacktivism **8080** Tor OR, **9200** git-SSH) so they do not turn cards orange.
|
||||
Only **hard** findings (DB/RDP/SMB/…) raise card colour; soft stays informational. Disable: `TALER_MONITORD_PORTSCAN=0`.
|
||||
|
||||
**Software versions:** per card from Taler `/config` (`build_version`), `Server:` (nginx/apache/caddy), SMTP banner (Postfix).
|
||||
Compared to **published** packages on `deb.taler.net` (`TALER_MONITORD_APT_SUITE`, default trixie): current / behind / ahead; chips show `installed → published` when behind.
|
||||
**Apps / SPAs:** page section with published WebUI + wallet-cli (+ GNU FTP wallet tarball when listed). Disable: `TALER_MONITORD_PUBLISHED=0`.
|
||||
|
||||
**Publish:** Cyon **`hacktivism`**. Defaults (both):
|
||||
`…/public_html/stacks/index.html` → **https://hacktivism.ch/stacks/**
|
||||
`…/public_html/monitoring.hacktivism.ch/index.html` (addon when DNS ready)
|
||||
Override: `TALER_MONITORD_PUBLISH_PATH` or `TALER_MONITORD_PUBLISH_PATHS=a,b`. Off: `TALER_MONITORD_PUBLISH=0`.
|
||||
|
||||
**Result index (optional):** bottom of the stacks page lists every probe grouped by traffic-light colour.
|
||||
`TALER_MONITORD_RESULT_INDEX=1` (or `colors`) · with concrete command: `=cmd` / `full`, or `=1` plus `TALER_MONITORD_RESULT_INDEX_CMD=1`. Off by default (`=0`).
|
||||
|
||||
**Git soft probe:** Git-labelled services run infrequent `git ls-remote` over HTTPS + SSH (cached ~6h). Soft yellow on fail. Disable: `TALER_MONITORD_GIT=0`.
|
||||
|
||||
**VeciGate:** `services-catalog.conf` zone `vecigate` (WAN IP) + intentional DNAT ports in `ports-whitelist.conf` (incl. tortuna companion).
|
||||
|
||||
**Payability:** `TALER_MONITORD_PAY=1` shows CHF:0 / zero-pay status on matching stacks. Live wallet run only with `TALER_MONITORD_PAY_RUN=1` (interval `TALER_MONITORD_PAY_INTERVAL_S`, default 4h). Domains: `TALER_MONITORD_PAY_DOMAINS`.
|
||||
|
||||
**Ansible smoke (opt-in phase):** `./taler-monitoring.sh ansible-smoke` → `check_ansible_smoke.sh` against `TALER_DEPLOYMENT` (`regional-currency-ng/site.yml`). Podman tracks: `ANSIBLE_SMOKE_PODMAN=1` on koopa (else reuse `aptdeploy`).
|
||||
|
||||
**Local VCS mirror:** `~/git/www-hacktivism-wiki` rsyncs Cyon → git every **4h**
|
||||
(`ch.hacktivism.www-wiki-sync`). Tracks `stacks/` (+ optional addon). No push by default.
|
||||
|
||||
**Custom /intro:** only **GOA** (hacktivism) and **FrancPaysan / TESTPAYSAN stage**.
|
||||
**Landing stats:** on intro stacks, `…/intro/stats.json` → compact tag.
|
||||
**Currency / coins:** from exchange `/config` `currency_specification`.
|
||||
**Endpoints:** suite-like paths; soft ≤ yellow. **`/terms` + `/privacy`:** medium → orange **only TOPS prod**.
|
||||
|
||||
## Optional / moved
|
||||
|
||||
Legacy multi-host generate/deploy and firecuda launchd tooling live under
|
||||
|
|
|
|||
1050
site-gen/stack_status_html.py
Executable file
1050
site-gen/stack_status_html.py
Executable file
File diff suppressed because it is too large
Load diff
4410
site-gen/stack_status_lib.py
Normal file
4410
site-gen/stack_status_lib.py
Normal file
File diff suppressed because it is too large
Load diff
224
site-gen/taler-monitord
Executable file
224
site-gen/taler-monitord
Executable file
|
|
@ -0,0 +1,224 @@
|
|||
#!/usr/bin/env python3
|
||||
"""taler-monitord — probe domain stacks on an interval; UI reads JSON only."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
sys.path.insert(0, str(HERE))
|
||||
import stack_status_html as H # noqa: E402
|
||||
import stack_status_lib as L # noqa: E402
|
||||
|
||||
# Match lib.sh: load ~/.config/taler-monitoring/env (or TALER_MONITORING_ENV), then
|
||||
# refresh module flags that were read at import time.
|
||||
_ENV_LOADED = L.load_monitoring_env()
|
||||
if _ENV_LOADED:
|
||||
L.refresh_runtime_flags_from_env()
|
||||
|
||||
|
||||
def _pid_path() -> Path:
|
||||
return L.monitord_state_dir() / "taler-monitord.pid"
|
||||
|
||||
|
||||
def _write_pid() -> None:
|
||||
_pid_path().write_text(str(os.getpid()) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def _clear_pid() -> None:
|
||||
p = _pid_path()
|
||||
try:
|
||||
p.unlink()
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def _publish_one(local: Path, ssh_host: str, remote: str) -> str | None:
|
||||
remote_dir = str(Path(remote).parent)
|
||||
mk = subprocess.run(
|
||||
[
|
||||
"ssh",
|
||||
"-o",
|
||||
"BatchMode=yes",
|
||||
"-o",
|
||||
"ConnectTimeout=8",
|
||||
ssh_host,
|
||||
f"mkdir -p -- {remote_dir!s}",
|
||||
],
|
||||
capture_output=True,
|
||||
timeout=15,
|
||||
check=False,
|
||||
)
|
||||
if mk.returncode != 0:
|
||||
err = (mk.stderr or b"").decode("utf-8", "replace").strip()[:200]
|
||||
print(f"publish mkdir fail ({remote}): {err}", file=sys.stderr, flush=True)
|
||||
return None
|
||||
r = subprocess.run(
|
||||
[
|
||||
"scp",
|
||||
"-o",
|
||||
"BatchMode=yes",
|
||||
"-o",
|
||||
"ConnectTimeout=8",
|
||||
str(local),
|
||||
f"{ssh_host}:{remote}",
|
||||
],
|
||||
capture_output=True,
|
||||
timeout=30,
|
||||
check=False,
|
||||
)
|
||||
if r.returncode != 0:
|
||||
err = (r.stderr or b"").decode("utf-8", "replace").strip()[:200]
|
||||
print(f"publish fail ({remote}): {err}", file=sys.stderr, flush=True)
|
||||
return None
|
||||
if "/var/www/monitoring-sites/" in remote:
|
||||
subprocess.run(
|
||||
[
|
||||
"ssh",
|
||||
"-o",
|
||||
"BatchMode=yes",
|
||||
"-o",
|
||||
"ConnectTimeout=8",
|
||||
ssh_host,
|
||||
(
|
||||
f"chmod 755 -- {remote_dir!s} && chmod 644 -- {remote!s} && "
|
||||
f"setfacl -m u:caddy:rx,m::r-x -- {remote_dir!s} && "
|
||||
f"setfacl -m u:caddy:r,m::r-- -- {remote!s} || true"
|
||||
),
|
||||
],
|
||||
capture_output=True,
|
||||
timeout=15,
|
||||
check=False,
|
||||
)
|
||||
else:
|
||||
subprocess.run(
|
||||
[
|
||||
"ssh",
|
||||
"-o",
|
||||
"BatchMode=yes",
|
||||
"-o",
|
||||
"ConnectTimeout=8",
|
||||
ssh_host,
|
||||
f"chmod 755 -- {remote_dir!s} && chmod 644 -- {remote!s} || true",
|
||||
],
|
||||
capture_output=True,
|
||||
timeout=15,
|
||||
check=False,
|
||||
)
|
||||
return f"{ssh_host}:{remote}"
|
||||
|
||||
|
||||
def publish_html(doc: dict) -> str | None:
|
||||
"""Render HTML and scp to Cyon (hacktivism) paths.
|
||||
|
||||
Env:
|
||||
TALER_MONITORD_PUBLISH=0 to disable
|
||||
TALER_MONITORD_PUBLISH_SSH=hacktivism
|
||||
TALER_MONITORD_PUBLISH_PATH=… single path (overrides multi default)
|
||||
TALER_MONITORD_PUBLISH_PATHS=comma-separated paths
|
||||
Default paths: public_html/stacks + public_html/monitoring.hacktivism.ch
|
||||
"""
|
||||
if os.environ.get("TALER_MONITORD_PUBLISH", "1") in ("0", "false", "no", ""):
|
||||
return None
|
||||
ssh_host = os.environ.get("TALER_MONITORD_PUBLISH_SSH", "hacktivism").strip() or "hacktivism"
|
||||
single = (os.environ.get("TALER_MONITORD_PUBLISH_PATH") or "").strip()
|
||||
multi = (os.environ.get("TALER_MONITORD_PUBLISH_PATHS") or "").strip()
|
||||
if single:
|
||||
remotes = [single]
|
||||
elif multi:
|
||||
remotes = [p.strip() for p in multi.split(",") if p.strip()]
|
||||
else:
|
||||
remotes = [
|
||||
"/home/hacktivi/public_html/stacks/index.html",
|
||||
"/home/hacktivi/public_html/monitoring.hacktivism.ch/index.html",
|
||||
]
|
||||
html_text = H.render_html(doc)
|
||||
with tempfile.NamedTemporaryFile("w", encoding="utf-8", suffix=".html", delete=False) as f:
|
||||
f.write(html_text)
|
||||
local = Path(f.name)
|
||||
published: list[str] = []
|
||||
try:
|
||||
for remote in remotes:
|
||||
got = _publish_one(local, ssh_host, remote)
|
||||
if got:
|
||||
published.append(got)
|
||||
return "; ".join(published) if published else None
|
||||
except (OSError, subprocess.TimeoutExpired) as e:
|
||||
print(f"publish error: {e}", file=sys.stderr, flush=True)
|
||||
return None
|
||||
finally:
|
||||
try:
|
||||
local.unlink()
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def run_once(*, domains: Path, out_paths: list[Path] | None, public_only: bool) -> Path:
|
||||
stacks = L.parse_domains_conf(domains)
|
||||
prev = L.read_status(out_paths[0] if out_paths else None)
|
||||
if not prev and out_paths:
|
||||
for p in out_paths:
|
||||
prev = L.read_status(p)
|
||||
if prev:
|
||||
break
|
||||
doc = L.build_stack_status(stacks, prev_doc=prev, public_only=public_only)
|
||||
paths = out_paths or L.status_json_paths()
|
||||
path = L.write_stack_status(doc, paths)
|
||||
# always keep a local HTML copy too
|
||||
local_html = L.monitord_state_dir() / "index.html"
|
||||
local_html.write_text(H.render_html(doc), encoding="utf-8")
|
||||
published = publish_html(doc)
|
||||
if published:
|
||||
print(f"published {published}", flush=True)
|
||||
return path
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description="Collect Taler stack status for the detached UI")
|
||||
ap.add_argument("--interval", type=float, default=float(os.environ.get("TALER_MONITORD_INTERVAL", "20")))
|
||||
ap.add_argument("--once", action="store_true", help="Single probe cycle then exit")
|
||||
ap.add_argument("--domains-conf", type=Path, default=None)
|
||||
ap.add_argument("--out", type=Path, action="append", default=None, help="Output JSON path (repeatable)")
|
||||
# Default on: include GOA / hacktivism.ch (taler.hacktivism.ch merchant). Opt out: --no-include-local
|
||||
_incl = os.environ.get("TALER_MONITORD_INCLUDE_LOCAL", "1") not in ("0", "false", "no", "")
|
||||
ap.add_argument(
|
||||
"--include-local",
|
||||
action=argparse.BooleanOptionalAction,
|
||||
default=_incl,
|
||||
help="Probe local=1 stacks (hacktivism GOA). Default on; --no-include-local to skip",
|
||||
)
|
||||
args = ap.parse_args()
|
||||
if _ENV_LOADED:
|
||||
print(f"env {_ENV_LOADED}", flush=True)
|
||||
domains = args.domains_conf or L.default_domains_conf()
|
||||
if not domains.is_file():
|
||||
print(f"domains.conf not found: {domains}", file=sys.stderr)
|
||||
return 2
|
||||
public_only = not bool(args.include_local)
|
||||
out_paths = list(args.out) if args.out else None
|
||||
|
||||
if args.once:
|
||||
path = run_once(domains=domains, out_paths=out_paths, public_only=public_only)
|
||||
print(path)
|
||||
return 0
|
||||
|
||||
_write_pid()
|
||||
try:
|
||||
while True:
|
||||
try:
|
||||
path = run_once(domains=domains, out_paths=out_paths, public_only=public_only)
|
||||
print(f"ok {path}", flush=True)
|
||||
except Exception as e: # keep loop alive
|
||||
print(f"error {type(e).__name__}: {e}", file=sys.stderr, flush=True)
|
||||
time.sleep(max(1.0, float(args.interval)))
|
||||
finally:
|
||||
_clear_pid()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Reference in a new issue