monitoring: koopa login-health timer snapshot on SSH login
This commit is contained in:
Hernâni Marques
parent
dc94b30fed
commit
e4dc9abac5
6 changed files with 739 additions and 0 deletions
18
configs/systemd/user/koopa-login-health.service
Normal file
18
configs/systemd/user/koopa-login-health.service
Normal file
|
|
@ -0,0 +1,18 @@
|
||||||
|
[Unit]
|
||||||
|
Description=Collect koopa login-health snapshot (read-only probes)
|
||||||
|
Documentation=file:%h/src/koopa/koopa-admin-log/scripts/login-health/README.md
|
||||||
|
# Best-effort: failures must not cascade
|
||||||
|
After=default.target
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=oneshot
|
||||||
|
# Snapshot only — never block login; login only cats ~/.local/state/…
|
||||||
|
ExecStart=%h/.local/bin/koopa-login-health collect
|
||||||
|
# Treat overall FAIL as success for the timer (we still wrote a snapshot)
|
||||||
|
SuccessExitStatus=0 1
|
||||||
|
Nice=10
|
||||||
|
IOSchedulingClass=best-effort
|
||||||
|
IOSchedulingPriority=6
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=default.target
|
||||||
13
configs/systemd/user/koopa-login-health.timer
Normal file
13
configs/systemd/user/koopa-login-health.timer
Normal file
|
|
@ -0,0 +1,13 @@
|
||||||
|
[Unit]
|
||||||
|
Description=Timer · koopa login-health snapshot (every 5 min)
|
||||||
|
Documentation=file:%h/src/koopa/koopa-admin-log/scripts/login-health/README.md
|
||||||
|
|
||||||
|
[Timer]
|
||||||
|
OnBootSec=2min
|
||||||
|
OnUnitActiveSec=5min
|
||||||
|
AccuracySec=30s
|
||||||
|
Persistent=true
|
||||||
|
Unit=koopa-login-health.service
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=timers.target
|
||||||
|
|
@ -12,6 +12,7 @@
|
||||||
| `taler-shared/` | shared helpers (`upgrade-goa-debs.sh` = apt upgrade **inside** bank/exchange/merchant containers) |
|
| `taler-shared/` | shared helpers (`upgrade-goa-debs.sh` = apt upgrade **inside** bank/exchange/merchant containers) |
|
||||||
| `taler-landing/` | **hernani user systemd**: central landing `stats.json` (full bank scan + alt units) |
|
| `taler-landing/` | **hernani user systemd**: central landing `stats.json` (full bank scan + alt units) |
|
||||||
| `monitoring/` | host `/home/hernani/scripts` (tor relay stats) |
|
| `monitoring/` | host `/home/hernani/scripts` (tor relay stats) |
|
||||||
|
| `login-health/` | **SSH login banner**: timer **collect** every 5m; login only **show** snapshot (no probes) |
|
||||||
| `nym/` | **koopa-nym** build/up/status (nym.com nym-node) |
|
| `nym/` | **koopa-nym** build/up/status (nym.com nym-node) |
|
||||||
| `taler-wallet-cli/` | thin wrappers; **benchmarks live in** `../benchmarks/` |
|
| `taler-wallet-cli/` | thin wrappers; **benchmarks live in** `../benchmarks/` |
|
||||||
| `castopod/` | host `hernani` podman-compose `~/koopa-castopod` — see `castopod/README.md` |
|
| `castopod/` | host `hernani` podman-compose `~/koopa-castopod` — see `castopod/README.md` |
|
||||||
|
|
|
||||||
102
scripts/login-health/README.md
Normal file
102
scripts/login-health/README.md
Normal file
|
|
@ -0,0 +1,102 @@
|
||||||
|
# login-health — status on SSH login (koopa)
|
||||||
|
|
||||||
|
When you **SSH into koopa**, show a short **pre-collected** health snapshot so you see whether **git.hacktivism.ch** (Forgejo), Taler GOA, Caddy edge, and the rest of the LIVE inventory are up.
|
||||||
|
|
||||||
|
Inventory source: `host/overview/LIVE.md`, `configs/ports.md`.
|
||||||
|
|
||||||
|
## Design: no probes at login
|
||||||
|
|
||||||
|
| Path | What runs |
|
||||||
|
|------|-----------|
|
||||||
|
| **systemd user timer** (every **5 min** + soon after boot) | `koopa-login-health collect` — podman, Caddy, loopback HTTP, write snapshot |
|
||||||
|
| **SSH / bashrc** | `koopa-login-health show` only — **cat** last snapshot (milliseconds, no curl/podman) |
|
||||||
|
|
||||||
|
Snapshot files (`XDG_STATE_HOME`, default `~/.local/state/koopa-login-health/`):
|
||||||
|
|
||||||
|
| File | Content |
|
||||||
|
|------|---------|
|
||||||
|
| `status.compact` | login view (WARN/FAIL + overall) |
|
||||||
|
| `status.verbose` | full line list |
|
||||||
|
| `status.meta` | `overall=`, `collected_at=` (unix), RFC date |
|
||||||
|
|
||||||
|
If the snapshot is older than **`KOOPA_HEALTH_STALE_SEC`** (default **900** = 15 min), `show` prints a **stale** warning (timer likely down).
|
||||||
|
|
||||||
|
## Login must never break
|
||||||
|
|
||||||
|
| Guard | What it does |
|
||||||
|
|-------|----------------|
|
||||||
|
| **show only** | bashrc never runs `collect` |
|
||||||
|
| **`KOOPA_HEALTH_FROM_LOGIN=1`** | always exit 0 |
|
||||||
|
| **`timeout 2s`** | wall cap to cat a few KB |
|
||||||
|
| **`\|\| true` / `set +e`** | hook cannot abort the shell |
|
||||||
|
| **`bash -n`** on install | refuse a broken bashrc hook |
|
||||||
|
|
||||||
|
Skip: `KOOPA_SKIP_HEALTH=1 ssh koopa`
|
||||||
|
|
||||||
|
## What you see
|
||||||
|
|
||||||
|
```text
|
||||||
|
koopa health koopa Wed, …
|
||||||
|
git.hacktivism.ch = Forgejo :9024/:9200 · edge Caddy :9000/:9001
|
||||||
|
OK all critical checks passed (42 ok)
|
||||||
|
overall: OK ok=42 warn=0 fail=0
|
||||||
|
checked: Wed, … (3m ago) [timer snapshot; not live at login]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Checks (on collect only)
|
||||||
|
|
||||||
|
| Area | What |
|
||||||
|
|------|------|
|
||||||
|
| Edge | `caddy.service`, proxy sockets |
|
||||||
|
| Listen | critical + optional ports |
|
||||||
|
| Podman | LIVE container names |
|
||||||
|
| HTTP | loopback backends (2s curl, parallel) |
|
||||||
|
| Git SSH | TCP `:9200` |
|
||||||
|
|
||||||
|
**Critical** (→ overall FAIL): merchant, exchange, bank, forgejo, edge/critical ports.
|
||||||
|
**Optional** (→ WARN): castopod, bonfire, prime, bt, paivana, tops, tor, nym.
|
||||||
|
|
||||||
|
Read-only. No restarts, no secrets.
|
||||||
|
|
||||||
|
## Install (on koopa as `hernani`)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd ~/src/koopa/koopa-admin-log
|
||||||
|
./scripts/login-health/install-login-health.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Installs binary, user units, bashrc hook, enables timer, runs one collect.
|
||||||
|
|
||||||
|
Linger (so the timer runs after reboot without an open SSH session):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo loginctl enable-linger hernani
|
||||||
|
```
|
||||||
|
|
||||||
|
## Manual
|
||||||
|
|
||||||
|
```bash
|
||||||
|
koopa-login-health # show compact snapshot
|
||||||
|
koopa-login-health show -v # verbose snapshot
|
||||||
|
koopa-login-health show -q # overall (+ stale note)
|
||||||
|
koopa-login-health collect # refresh now (same as the timer unit)
|
||||||
|
systemctl --user start koopa-login-health.service
|
||||||
|
systemctl --user status koopa-login-health.timer
|
||||||
|
```
|
||||||
|
|
||||||
|
## Uninstall
|
||||||
|
|
||||||
|
```bash
|
||||||
|
systemctl --user disable --now koopa-login-health.timer
|
||||||
|
rm -f ~/.config/systemd/user/koopa-login-health.{service,timer}
|
||||||
|
systemctl --user daemon-reload
|
||||||
|
rm -f ~/.local/bin/koopa-login-health
|
||||||
|
# remove marked block in ~/.bashrc
|
||||||
|
rm -rf ~/.local/state/koopa-login-health
|
||||||
|
```
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- Grok workflow `koopa-services-health` (laptop public HTTPS + generated mon pages)
|
||||||
|
- `scripts/taler-sanity/check_stack-health.sh` (deeper Taler, often root)
|
||||||
|
- units: `configs/systemd/user/koopa-login-health.{service,timer}`
|
||||||
137
scripts/login-health/install-login-health.sh
Executable file
137
scripts/login-health/install-login-health.sh
Executable file
|
|
@ -0,0 +1,137 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# Install koopa-login-health:
|
||||||
|
# - ~/.local/bin/koopa-login-health (show + collect)
|
||||||
|
# - user systemd timer → collect every 5 min (fresh snapshot)
|
||||||
|
# - bashrc hook → show only (no probes on login; never blocks shell)
|
||||||
|
#
|
||||||
|
# On koopa as hernani:
|
||||||
|
# cd ~/src/koopa/koopa-admin-log
|
||||||
|
# ./scripts/login-health/install-login-health.sh
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
ROOT=$(cd "$(dirname "$0")" && pwd)
|
||||||
|
REPO_ROOT=$(cd "$ROOT/../.." && pwd)
|
||||||
|
SRC="$ROOT/koopa-login-health.sh"
|
||||||
|
UNIT_SRC_DIR="$REPO_ROOT/configs/systemd/user"
|
||||||
|
BIN_DIR="${HOME}/.local/bin"
|
||||||
|
DEST="$BIN_DIR/koopa-login-health"
|
||||||
|
BASHRC="${HOME}/.bashrc"
|
||||||
|
UNIT_DIR="${XDG_CONFIG_HOME:-$HOME/.config}/systemd/user"
|
||||||
|
MARKER_BEGIN="# >>> koopa-login-health (koopa-admin-log) >>>"
|
||||||
|
MARKER_END="# <<< koopa-login-health (koopa-admin-log) <<<"
|
||||||
|
|
||||||
|
if [ ! -f "$SRC" ]; then
|
||||||
|
echo "missing $SRC" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Login: ONLY show snapshot. Never collect. Instant cat; always continue.
|
||||||
|
HOOK_BODY=$(cat <<'HOOK'
|
||||||
|
# Best-effort: print last health snapshot. MUST NOT break login. MUST NOT probe.
|
||||||
|
# Skip: KOOPA_SKIP_HEALTH=1
|
||||||
|
# Refresh is the user timer koopa-login-health.timer (not this hook).
|
||||||
|
__koopa_lh() {
|
||||||
|
[ -n "${KOOPA_SKIP_HEALTH:-}" ] && return 0
|
||||||
|
case $- in *i*) ;; *) return 0 ;; esac
|
||||||
|
_b="${HOME}/.local/bin/koopa-login-health"
|
||||||
|
[ -x "$_b" ] || return 0
|
||||||
|
if [ -z "${SSH_CONNECTION:-}" ] && [ -z "${SSH_TTY:-}" ]; then
|
||||||
|
[ "${SHLVL:-1}" -le 2 ] 2>/dev/null || return 0
|
||||||
|
fi
|
||||||
|
# show only — no collect; 2s wall cap is enough to cat a few KB
|
||||||
|
if command -v timeout >/dev/null 2>&1; then
|
||||||
|
timeout -k 1s 2s env KOOPA_HEALTH_FROM_LOGIN=1 "$_b" show </dev/null >/dev/tty 2>/dev/null \
|
||||||
|
|| timeout -k 1s 2s env KOOPA_HEALTH_FROM_LOGIN=1 "$_b" show </dev/null 2>/dev/null \
|
||||||
|
|| true
|
||||||
|
else
|
||||||
|
env KOOPA_HEALTH_FROM_LOGIN=1 "$_b" show </dev/null 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
set +e
|
||||||
|
__koopa_lh || true
|
||||||
|
unset -f __koopa_lh 2>/dev/null || true
|
||||||
|
HOOK
|
||||||
|
)
|
||||||
|
|
||||||
|
mkdir -p "$BIN_DIR" "$UNIT_DIR"
|
||||||
|
install -m 0755 "$SRC" "$DEST"
|
||||||
|
echo "installed $DEST"
|
||||||
|
|
||||||
|
for u in koopa-login-health.service koopa-login-health.timer; do
|
||||||
|
if [ ! -f "$UNIT_SRC_DIR/$u" ]; then
|
||||||
|
echo "missing unit $UNIT_SRC_DIR/$u" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
install -m 0644 "$UNIT_SRC_DIR/$u" "$UNIT_DIR/$u"
|
||||||
|
echo "installed $UNIT_DIR/$u"
|
||||||
|
done
|
||||||
|
|
||||||
|
touch "$BASHRC"
|
||||||
|
if grep -qF "$MARKER_BEGIN" "$BASHRC" 2>/dev/null; then
|
||||||
|
tmp=$(mktemp)
|
||||||
|
awk -v b="$MARKER_BEGIN" -v e="$MARKER_END" '
|
||||||
|
$0 == b {skip=1; next}
|
||||||
|
$0 == e {skip=0; next}
|
||||||
|
!skip {print}
|
||||||
|
' "$BASHRC" >"$tmp"
|
||||||
|
mv "$tmp" "$BASHRC"
|
||||||
|
echo "removed previous bashrc hook"
|
||||||
|
fi
|
||||||
|
{
|
||||||
|
echo ""
|
||||||
|
echo "$MARKER_BEGIN"
|
||||||
|
printf '%s\n' "$HOOK_BODY"
|
||||||
|
echo "$MARKER_END"
|
||||||
|
} >>"$BASHRC"
|
||||||
|
echo "installed bashrc hook (show-only; no probes on login)"
|
||||||
|
|
||||||
|
hook_tmp=$(mktemp)
|
||||||
|
{
|
||||||
|
echo "$MARKER_BEGIN"
|
||||||
|
printf '%s\n' "$HOOK_BODY"
|
||||||
|
echo "$MARKER_END"
|
||||||
|
} >"$hook_tmp"
|
||||||
|
if ! bash -n "$hook_tmp" || ! bash -n "$BASHRC"; then
|
||||||
|
echo "ERROR: bashrc syntax check failed — removing hook" >&2
|
||||||
|
tmp=$(mktemp)
|
||||||
|
awk -v b="$MARKER_BEGIN" -v e="$MARKER_END" '
|
||||||
|
$0 == b {skip=1; next}
|
||||||
|
$0 == e {skip=0; next}
|
||||||
|
!skip {print}
|
||||||
|
' "$BASHRC" >"$tmp"
|
||||||
|
mv "$tmp" "$BASHRC"
|
||||||
|
rm -f "$hook_tmp"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
rm -f "$hook_tmp"
|
||||||
|
echo "bash -n OK"
|
||||||
|
|
||||||
|
if ! echo ":$PATH:" | grep -q ":$BIN_DIR:"; then
|
||||||
|
if ! grep -q '\.local/bin' "$BASHRC" 2>/dev/null; then
|
||||||
|
echo "export PATH=\"\$HOME/.local/bin:\$PATH\"" >>"$BASHRC"
|
||||||
|
echo "appended PATH += ~/.local/bin"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Enable timer + first collect (needs lingering for boot without login)
|
||||||
|
systemctl --user daemon-reload
|
||||||
|
systemctl --user enable --now koopa-login-health.timer
|
||||||
|
systemctl --user start koopa-login-health.service || true
|
||||||
|
echo "timer: $(systemctl --user is-enabled koopa-login-health.timer 2>/dev/null || echo unknown)"
|
||||||
|
systemctl --user list-timers koopa-login-health.timer --no-pager 2>/dev/null || true
|
||||||
|
|
||||||
|
# linger so timer runs without interactive session
|
||||||
|
if command -v loginctl >/dev/null 2>&1; then
|
||||||
|
if [ "$(loginctl show-user "$USER" -p Linger --value 2>/dev/null || echo no)" != "yes" ]; then
|
||||||
|
echo "NOTE: enable linger so the timer runs after reboot without login:"
|
||||||
|
echo " sudo loginctl enable-linger $USER"
|
||||||
|
else
|
||||||
|
echo "linger already enabled for $USER"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "smoke show (FROM_LOGIN):"
|
||||||
|
KOOPA_HEALTH_FROM_LOGIN=1 "$DEST" show -q || true
|
||||||
|
echo "done. Login cats snapshot only; collect runs every 5 min via systemd --user."
|
||||||
468
scripts/login-health/koopa-login-health.sh
Executable file
468
scripts/login-health/koopa-login-health.sh
Executable file
|
|
@ -0,0 +1,468 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# koopa-login-health — read-only GOA/koopa service health.
|
||||||
|
# Inventory: koopa-admin-log host/overview/LIVE.md (+ ports.md).
|
||||||
|
#
|
||||||
|
# Architecture:
|
||||||
|
# collect — run probes, write snapshot (systemd user timer; NOT on login)
|
||||||
|
# show — print last snapshot only (default; used by bashrc on SSH login)
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# koopa-login-health # show compact snapshot (login path)
|
||||||
|
# koopa-login-health show -v # show verbose snapshot
|
||||||
|
# koopa-login-health show -q # overall line only
|
||||||
|
# koopa-login-health collect # refresh snapshot now
|
||||||
|
# koopa-login-health collect -v # same (verbose always stored)
|
||||||
|
# KOOPA_SKIP_HEALTH=1 … # bashrc skip
|
||||||
|
# KOOPA_HEALTH_FROM_LOGIN=1 # always exit 0 (login hook)
|
||||||
|
#
|
||||||
|
# Never mutates host services. Login never runs probes.
|
||||||
|
|
||||||
|
set +e
|
||||||
|
set +u
|
||||||
|
set +o pipefail 2>/dev/null || true
|
||||||
|
|
||||||
|
FROM_LOGIN=0
|
||||||
|
[ -n "${KOOPA_HEALTH_FROM_LOGIN:-}" ] && FROM_LOGIN=1
|
||||||
|
|
||||||
|
_exit_health() {
|
||||||
|
local c="${1:-0}"
|
||||||
|
[ "${FROM_LOGIN}" = "1" ] && exit 0
|
||||||
|
exit "${c}"
|
||||||
|
}
|
||||||
|
|
||||||
|
_bail_login_safe() {
|
||||||
|
echo "koopa health: ${1:-error} (ignored; login continues)" >&2
|
||||||
|
_exit_health 0
|
||||||
|
}
|
||||||
|
|
||||||
|
# --- paths (XDG) ---
|
||||||
|
STATE_DIR="${KOOPA_HEALTH_STATE_DIR:-${XDG_STATE_HOME:-${HOME:-/tmp}/.local/state}/koopa-login-health}"
|
||||||
|
COMPACT_FILE="${STATE_DIR}/status.compact"
|
||||||
|
VERBOSE_FILE="${STATE_DIR}/status.verbose"
|
||||||
|
META_FILE="${STATE_DIR}/status.meta"
|
||||||
|
# Max age for "current" snapshot (seconds). Older → WARN stale on show.
|
||||||
|
STALE_SEC="${KOOPA_HEALTH_STALE_SEC:-900}" # 15 min (timer default 5 min)
|
||||||
|
CURL_M="${KOOPA_HEALTH_CURL_M:-2}"
|
||||||
|
|
||||||
|
CMD="show"
|
||||||
|
VERBOSE=0
|
||||||
|
QUIET=0
|
||||||
|
|
||||||
|
while [ "${1:-}" != "" ]; do
|
||||||
|
case "$1" in
|
||||||
|
show|collect|status)
|
||||||
|
CMD="$1"
|
||||||
|
[ "$CMD" = "status" ] && CMD="show"
|
||||||
|
;;
|
||||||
|
-v|--verbose) VERBOSE=1 ;;
|
||||||
|
-q|--quiet) QUIET=1 ;;
|
||||||
|
-h|--help)
|
||||||
|
sed -n '2,22p' "$0" 2>/dev/null | sed 's/^# \?//' || true
|
||||||
|
_exit_health 0
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo "unknown arg: $1 (try: show | collect | -v | -q | -h)" >&2
|
||||||
|
_exit_health 2
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
shift
|
||||||
|
done
|
||||||
|
|
||||||
|
# ---------- SHOW: never probe ----------
|
||||||
|
do_show() {
|
||||||
|
if [ ! -f "${COMPACT_FILE}" ] && [ ! -f "${META_FILE}" ]; then
|
||||||
|
if [ "${QUIET}" -eq 1 ]; then
|
||||||
|
echo "unknown (no snapshot — run: systemctl --user start koopa-login-health.service)"
|
||||||
|
else
|
||||||
|
echo "koopa health: no snapshot yet"
|
||||||
|
echo " Collect runs on a user timer (not at login). Refresh once:"
|
||||||
|
echo " systemctl --user start koopa-login-health.service"
|
||||||
|
echo " or: koopa-login-health collect"
|
||||||
|
fi
|
||||||
|
_exit_health 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
local overall="unknown" collected_at="" age="?" age_sec=-1 now
|
||||||
|
if [ -f "${META_FILE}" ]; then
|
||||||
|
# shellcheck disable=SC1090
|
||||||
|
overall=$(grep '^overall=' "${META_FILE}" 2>/dev/null | tail -1 | cut -d= -f2-)
|
||||||
|
collected_at=$(grep '^collected_at_rfc=' "${META_FILE}" 2>/dev/null | tail -1 | cut -d= -f2-)
|
||||||
|
local ts
|
||||||
|
ts=$(grep '^collected_at=' "${META_FILE}" 2>/dev/null | tail -1 | cut -d= -f2-)
|
||||||
|
now=$(date +%s 2>/dev/null || echo 0)
|
||||||
|
if [ -n "${ts}" ] && [ "${ts}" -gt 0 ] 2>/dev/null; then
|
||||||
|
age_sec=$((now - ts))
|
||||||
|
if [ "${age_sec}" -lt 0 ] 2>/dev/null; then age_sec=0; fi
|
||||||
|
if [ "${age_sec}" -lt 120 ] 2>/dev/null; then
|
||||||
|
age="${age_sec}s ago"
|
||||||
|
elif [ "${age_sec}" -lt 7200 ] 2>/dev/null; then
|
||||||
|
age="$((age_sec / 60))m ago"
|
||||||
|
else
|
||||||
|
age="$((age_sec / 3600))h ago"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
local stale=0
|
||||||
|
if [ "${age_sec}" -ge 0 ] 2>/dev/null && [ "${age_sec}" -gt "${STALE_SEC}" ] 2>/dev/null; then
|
||||||
|
stale=1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "${QUIET}" -eq 1 ]; then
|
||||||
|
if [ "${stale}" -eq 1 ]; then
|
||||||
|
echo "${overall} (stale ${age})"
|
||||||
|
else
|
||||||
|
echo "${overall}"
|
||||||
|
fi
|
||||||
|
[ "${stale}" -eq 1 ] && _exit_health 1
|
||||||
|
case "${overall}" in ok|warn) _exit_health 0 ;; *) _exit_health 1 ;; esac
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "${VERBOSE}" -eq 1 ] && [ -f "${VERBOSE_FILE}" ]; then
|
||||||
|
cat "${VERBOSE_FILE}" 2>/dev/null || true
|
||||||
|
elif [ -f "${COMPACT_FILE}" ]; then
|
||||||
|
cat "${COMPACT_FILE}" 2>/dev/null || true
|
||||||
|
else
|
||||||
|
echo "koopa health: snapshot incomplete (meta only) overall=${overall}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo " checked: ${collected_at:-?} (${age}) [timer snapshot; not live at login]"
|
||||||
|
if [ "${stale}" -eq 1 ]; then
|
||||||
|
echo " WARN: snapshot older than ${STALE_SEC}s — is koopa-login-health.timer active?"
|
||||||
|
echo " systemctl --user status koopa-login-health.timer"
|
||||||
|
fi
|
||||||
|
|
||||||
|
[ "${stale}" -eq 1 ] && _exit_health 1
|
||||||
|
case "${overall}" in ok|warn) _exit_health 0 ;; *) _exit_health 1 ;; esac
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------- COLLECT: probes → write snapshot ----------
|
||||||
|
CRITICAL_CONTAINERS=(
|
||||||
|
taler-hacktivism
|
||||||
|
taler-hacktivism-exchange-ansible
|
||||||
|
taler-hacktivism-bank
|
||||||
|
koopa-forgejo
|
||||||
|
)
|
||||||
|
|
||||||
|
EXPECT_CONTAINERS=(
|
||||||
|
taler-hacktivism
|
||||||
|
taler-hacktivism-exchange-ansible
|
||||||
|
taler-hacktivism-bank
|
||||||
|
koopa-castopod
|
||||||
|
koopa-bonfire
|
||||||
|
koopa-prime-jellyfin
|
||||||
|
koopa-prime-qbittorrent
|
||||||
|
koopa-forgejo
|
||||||
|
koopa-paivana
|
||||||
|
koopa-tor-relay
|
||||||
|
koopa-nym
|
||||||
|
koopa-tops-ng1
|
||||||
|
koopa-tops-ng2
|
||||||
|
koopa-tops-ng3
|
||||||
|
)
|
||||||
|
|
||||||
|
OPTIONAL_CONTAINERS=(
|
||||||
|
koopa-prime-jellyfin
|
||||||
|
koopa-prime-qbittorrent
|
||||||
|
koopa-nym
|
||||||
|
koopa-tor-relay
|
||||||
|
koopa-tops-ng1
|
||||||
|
koopa-tops-ng2
|
||||||
|
koopa-tops-ng3
|
||||||
|
koopa-castopod
|
||||||
|
koopa-bonfire
|
||||||
|
koopa-paivana
|
||||||
|
)
|
||||||
|
|
||||||
|
PROBES=(
|
||||||
|
"merchant|http://127.0.0.1:9010/config|200"
|
||||||
|
"exchange|http://127.0.0.1:9011/keys|200"
|
||||||
|
"bank|http://127.0.0.1:9012/config|200"
|
||||||
|
"forgejo|http://127.0.0.1:9024/|200,302,303"
|
||||||
|
"castopod|http://127.0.0.1:9020/|200,302,303"
|
||||||
|
"bonfire|http://127.0.0.1:9021/|200,302,303"
|
||||||
|
"prime|http://127.0.0.1:9022/|200,302,303"
|
||||||
|
"bt|http://127.0.0.1:9023/|200,302,303,401"
|
||||||
|
"paivana|http://127.0.0.1:9025/|200,302,303"
|
||||||
|
"tops-ng1|http://127.0.0.1:9090/|200,302,303"
|
||||||
|
"tops-ng2|http://127.0.0.1:9091/|200,302,303"
|
||||||
|
"tops-ng3|http://127.0.0.1:9092/|200,302,303"
|
||||||
|
)
|
||||||
|
|
||||||
|
OPTIONAL_PROBES=(castopod bonfire prime bt paivana tops-ng1 tops-ng2 tops-ng3)
|
||||||
|
|
||||||
|
C_OK=; C_WARN=; C_FAIL=; C_DIM=; C_RST=; C_BOLD=
|
||||||
|
# colors only when writing for a tty later; snapshots are plain + optional ANSI if COLLECT_COLOR=1
|
||||||
|
if [ "${KOOPA_HEALTH_COLLECT_COLOR:-0}" = "1" ] && [ -t 1 ]; then
|
||||||
|
C_OK=$'\033[32m'; C_WARN=$'\033[33m'; C_FAIL=$'\033[31m'
|
||||||
|
C_DIM=$'\033[2m'; C_RST=$'\033[0m'; C_BOLD=$'\033[1m'
|
||||||
|
fi
|
||||||
|
|
||||||
|
OK_N=0
|
||||||
|
WARN_N=0
|
||||||
|
FAIL_N=0
|
||||||
|
LINES=()
|
||||||
|
|
||||||
|
is_optional_container() {
|
||||||
|
local n="$1" x
|
||||||
|
for x in "${OPTIONAL_CONTAINERS[@]}"; do [ "$x" = "$n" ] && return 0; done
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
is_optional_probe() {
|
||||||
|
local n="$1" x
|
||||||
|
for x in "${OPTIONAL_PROBES[@]}"; do [ "$x" = "$n" ] && return 0; done
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
record() {
|
||||||
|
local level="$1" msg="$2"
|
||||||
|
case "$level" in
|
||||||
|
OK) OK_N=$((OK_N + 1)); LINES+=(" ${C_OK}OK ${C_RST}${msg}") ;;
|
||||||
|
WARN) WARN_N=$((WARN_N + 1)); LINES+=(" ${C_WARN}WARN${C_RST} ${msg}") ;;
|
||||||
|
FAIL) FAIL_N=$((FAIL_N + 1)); LINES+=(" ${C_FAIL}FAIL${C_RST} ${msg}") ;;
|
||||||
|
INFO) LINES+=(" ${C_DIM}··· ${C_RST}${msg}") ;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
code_in_list() {
|
||||||
|
local code="$1" list="$2" part
|
||||||
|
local IFS=','
|
||||||
|
# shellcheck disable=SC2086
|
||||||
|
for part in $list; do
|
||||||
|
[ "$code" = "$part" ] && return 0
|
||||||
|
done
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
do_collect() {
|
||||||
|
mkdir -p "${STATE_DIR}" 2>/dev/null || {
|
||||||
|
echo "koopa health collect: cannot mkdir ${STATE_DIR}" >&2
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
local HOST NOW caddy_st st sock listen_blob port
|
||||||
|
local PODMAN running_names name dep tmpdir pids pid
|
||||||
|
local pname purl pok outf code overall line
|
||||||
|
|
||||||
|
HOST=$(hostname -s 2>/dev/null || hostname 2>/dev/null || echo koopa)
|
||||||
|
NOW=$(date -R 2>/dev/null || date 2>/dev/null || echo "")
|
||||||
|
local NOW_UNIX
|
||||||
|
NOW_UNIX=$(date +%s 2>/dev/null || echo 0)
|
||||||
|
|
||||||
|
if [ "$(id -u 2>/dev/null)" = "0" ]; then
|
||||||
|
record INFO "running as root (podman rootless via hernani if available)"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if command -v systemctl >/dev/null 2>&1; then
|
||||||
|
caddy_st=$(systemctl is-active caddy 2>/dev/null || echo unknown)
|
||||||
|
if [ "${caddy_st}" = "active" ]; then
|
||||||
|
record OK "caddy.service active"
|
||||||
|
else
|
||||||
|
record FAIL "caddy.service is ${caddy_st} (edge HTTPS)"
|
||||||
|
fi
|
||||||
|
for sock in http-proxy.socket https-proxy.socket; do
|
||||||
|
st=$(systemctl is-active "${sock}" 2>/dev/null || echo unknown)
|
||||||
|
if [ "${st}" = "active" ] || [ "${st}" = "listening" ]; then
|
||||||
|
record OK "${sock} ${st}"
|
||||||
|
else
|
||||||
|
record WARN "${sock} is ${st}"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
else
|
||||||
|
record WARN "systemctl not available"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if command -v ss >/dev/null 2>&1; then
|
||||||
|
listen_blob=$(ss -lnt 2>/dev/null | awk '{print $4}' || true)
|
||||||
|
for port in 9000 9001 9010 9011 9012 9024 9200; do
|
||||||
|
if printf '%s\n' "${listen_blob}" | grep -qE "[:.]${port}\$"; then
|
||||||
|
record OK "listen :${port}"
|
||||||
|
else
|
||||||
|
record FAIL "nothing listening on :${port}"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
for port in 9020 9021 9022 9023 9025 9090 9091 9092 8080 9080; do
|
||||||
|
if printf '%s\n' "${listen_blob}" | grep -qE "[:.]${port}\$"; then
|
||||||
|
record OK "listen :${port} (optional)"
|
||||||
|
else
|
||||||
|
record WARN "nothing listening on :${port}"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
|
||||||
|
PODMAN=(podman)
|
||||||
|
if [ "$(id -u 2>/dev/null)" = "0" ] && command -v runuser >/dev/null 2>&1; then
|
||||||
|
if id hernani >/dev/null 2>&1; then
|
||||||
|
PODMAN=(runuser -u hernani -- podman)
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
if ! command -v "${PODMAN[0]}" >/dev/null 2>&1 && [ "${PODMAN[0]}" = "podman" ]; then
|
||||||
|
record FAIL "podman not found"
|
||||||
|
running_names=""
|
||||||
|
else
|
||||||
|
running_names=$("${PODMAN[@]}" ps --format '{{.Names}}' 2>/dev/null || true)
|
||||||
|
if [ -z "${running_names}" ]; then
|
||||||
|
record FAIL "podman ps empty or failed"
|
||||||
|
fi
|
||||||
|
for name in "${EXPECT_CONTAINERS[@]}"; do
|
||||||
|
if printf '%s\n' "${running_names}" | grep -qxF "${name}"; then
|
||||||
|
st=$("${PODMAN[@]}" ps --format '{{.Names}}\t{{.Status}}' 2>/dev/null \
|
||||||
|
| awk -F'\t' -v n="${name}" '$1==n {print $2; exit}')
|
||||||
|
record OK "container ${name} (${st:-up})"
|
||||||
|
else
|
||||||
|
if is_optional_container "${name}"; then
|
||||||
|
record WARN "container ${name} not running"
|
||||||
|
else
|
||||||
|
record FAIL "container ${name} not running"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
for dep in koopa-forgejo-db koopa-castopod-mariadb koopa-castopod-redis koopa-bonfire-db; do
|
||||||
|
if printf '%s\n' "${running_names}" | grep -qxF "${dep}"; then
|
||||||
|
record INFO "dep ${dep} running"
|
||||||
|
else
|
||||||
|
record INFO "dep ${dep} not in running set (name may differ)"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
|
||||||
|
tmpdir=$(mktemp -d 2>/dev/null || true)
|
||||||
|
if [ -n "${tmpdir}" ] && [ -d "${tmpdir}" ]; then
|
||||||
|
probe_one() {
|
||||||
|
local _url="$1" _out="$2" _code
|
||||||
|
# curl still prints http_code on connect fail; do not append another "000" via ||
|
||||||
|
_code=$(curl -sk -m "${CURL_M}" -o /dev/null -w '%{http_code}' "${_url}" 2>/dev/null)
|
||||||
|
[ -n "${_code}" ] || _code="000"
|
||||||
|
printf '%s\n' "${_code}" >"${_out}"
|
||||||
|
}
|
||||||
|
pids=()
|
||||||
|
for spec in "${PROBES[@]}"; do
|
||||||
|
IFS='|' read -r pname purl pok <<EOF
|
||||||
|
${spec}
|
||||||
|
EOF
|
||||||
|
probe_one "${purl}" "${tmpdir}/${pname}" &
|
||||||
|
pids+=($!)
|
||||||
|
done
|
||||||
|
for pid in "${pids[@]:-}"; do
|
||||||
|
wait "${pid}" 2>/dev/null || true
|
||||||
|
done
|
||||||
|
for spec in "${PROBES[@]}"; do
|
||||||
|
IFS='|' read -r pname purl pok <<EOF
|
||||||
|
${spec}
|
||||||
|
EOF
|
||||||
|
code="000"
|
||||||
|
[ -f "${tmpdir}/${pname}" ] && code=$(awk '{print $1; exit}' "${tmpdir}/${pname}" 2>/dev/null || echo "000")
|
||||||
|
if code_in_list "${code}" "${pok}"; then
|
||||||
|
record OK "http ${pname} → ${code} (${purl})"
|
||||||
|
else
|
||||||
|
if is_optional_probe "${pname}"; then
|
||||||
|
record WARN "http ${pname} → ${code} (${purl})"
|
||||||
|
else
|
||||||
|
record FAIL "http ${pname} → ${code} (${purl})"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
rm -rf "${tmpdir}" 2>/dev/null || true
|
||||||
|
else
|
||||||
|
record WARN "mktemp failed; skipped HTTP probes"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if command -v timeout >/dev/null 2>&1; then
|
||||||
|
if timeout 1 bash -c 'echo >/dev/tcp/127.0.0.1/9200' 2>/dev/null; then
|
||||||
|
record OK "forgejo git-SSH TCP :9200 (git.hacktivism.ch)"
|
||||||
|
else
|
||||||
|
record FAIL "forgejo git-SSH TCP :9200 closed"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
overall="ok"
|
||||||
|
[ "${WARN_N}" -gt 0 ] 2>/dev/null && overall="warn"
|
||||||
|
[ "${FAIL_N}" -gt 0 ] 2>/dev/null && overall="fail"
|
||||||
|
|
||||||
|
local header verbose_body compact_body
|
||||||
|
header="koopa health ${HOST} ${NOW}
|
||||||
|
git.hacktivism.ch = Forgejo :9024/:9200 · edge Caddy :9000/:9001"
|
||||||
|
|
||||||
|
verbose_body="${header}
|
||||||
|
"
|
||||||
|
if [ "${#LINES[@]}" -gt 0 ]; then
|
||||||
|
printf -v _lines '%s\n' "${LINES[@]}"
|
||||||
|
verbose_body+="${_lines}"
|
||||||
|
fi
|
||||||
|
case "${overall}" in
|
||||||
|
ok) verbose_body+="overall: OK ok=${OK_N} warn=${WARN_N} fail=${FAIL_N}"$'\n' ;;
|
||||||
|
warn) verbose_body+="overall: WARN ok=${OK_N} warn=${WARN_N} fail=${FAIL_N}"$'\n' ;;
|
||||||
|
*) verbose_body+="overall: FAIL ok=${OK_N} warn=${WARN_N} fail=${FAIL_N}"$'\n' ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
compact_body="${header}
|
||||||
|
"
|
||||||
|
local has_issue=0
|
||||||
|
for line in "${LINES[@]:-}"; do
|
||||||
|
case "${line}" in
|
||||||
|
*FAIL*|*WARN*) compact_body+="${line}"$'\n'; has_issue=1 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
if [ "${has_issue}" -eq 0 ]; then
|
||||||
|
compact_body+=" OK all critical checks passed (${OK_N} ok)"$'\n'
|
||||||
|
else
|
||||||
|
compact_body+=" (${OK_N} ok — details: koopa-login-health show -v)"$'\n'
|
||||||
|
fi
|
||||||
|
case "${overall}" in
|
||||||
|
ok) compact_body+="overall: OK ok=${OK_N} warn=${WARN_N} fail=${FAIL_N}"$'\n' ;;
|
||||||
|
warn) compact_body+="overall: WARN ok=${OK_N} warn=${WARN_N} fail=${FAIL_N}"$'\n' ;;
|
||||||
|
*) compact_body+="overall: FAIL ok=${OK_N} warn=${WARN_N} fail=${FAIL_N}"$'\n' ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
# Atomic writes
|
||||||
|
local tmpc tmpv tmpm
|
||||||
|
tmpc="${STATE_DIR}/.status.compact.$$"
|
||||||
|
tmpv="${STATE_DIR}/.status.verbose.$$"
|
||||||
|
tmpm="${STATE_DIR}/.status.meta.$$"
|
||||||
|
printf '%s' "${compact_body}" >"${tmpc}" 2>/dev/null || return 1
|
||||||
|
printf '%s' "${verbose_body}" >"${tmpv}" 2>/dev/null || return 1
|
||||||
|
{
|
||||||
|
echo "overall=${overall}"
|
||||||
|
echo "collected_at=${NOW_UNIX}"
|
||||||
|
echo "collected_at_rfc=${NOW}"
|
||||||
|
echo "ok=${OK_N}"
|
||||||
|
echo "warn=${WARN_N}"
|
||||||
|
echo "fail=${FAIL_N}"
|
||||||
|
echo "host=${HOST}"
|
||||||
|
} >"${tmpm}" 2>/dev/null || return 1
|
||||||
|
mv -f "${tmpc}" "${COMPACT_FILE}" 2>/dev/null || return 1
|
||||||
|
mv -f "${tmpv}" "${VERBOSE_FILE}" 2>/dev/null || return 1
|
||||||
|
mv -f "${tmpm}" "${META_FILE}" 2>/dev/null || return 1
|
||||||
|
|
||||||
|
# Optional: print when run interactively
|
||||||
|
if [ -t 1 ] || [ "${VERBOSE}" -eq 1 ]; then
|
||||||
|
printf '%s' "${compact_body}"
|
||||||
|
echo " wrote snapshot → ${STATE_DIR}/"
|
||||||
|
fi
|
||||||
|
case "${overall}" in ok|warn) return 0 ;; *) return 1 ;; esac
|
||||||
|
}
|
||||||
|
|
||||||
|
case "${CMD}" in
|
||||||
|
show)
|
||||||
|
if [ "${FROM_LOGIN}" = "1" ]; then
|
||||||
|
( do_show ) 2>&1 || true
|
||||||
|
_exit_health 0
|
||||||
|
else
|
||||||
|
do_show
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
collect)
|
||||||
|
# Never invoke collect from login hook
|
||||||
|
if [ "${FROM_LOGIN}" = "1" ]; then
|
||||||
|
_bail_login_safe "collect refused under FROM_LOGIN"
|
||||||
|
fi
|
||||||
|
do_collect
|
||||||
|
_exit_health $?
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo "unknown command ${CMD}" >&2
|
||||||
|
_exit_health 2
|
||||||
|
;;
|
||||||
|
esac
|
||||||
Loading…
Add table
Add a link
Reference in a new issue