hernani/koopa-admin-log
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

monitoring: koopa login-health timer snapshot on SSH login

Browse source
This commit is contained in:
Hernâni Marques 2026-08-06 17:12:00 +02:00
parent dc94b30fed
commit e4dc9abac5
No known key found for this signature in database
GPG key ID: CB5738652768F7E9
6 changed files with 739 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

18
configs/systemd/user/koopa-login-health.service Normal file
View file

@ -0,0 +1,18 @@
[Unit]
Description=Collect koopa login-health snapshot (read-only probes)
Documentation=file:%h/src/koopa/koopa-admin-log/scripts/login-health/README.md
# Best-effort: failures must not cascade
After=default.target
[Service]
Type=oneshot
# Snapshot only — never block login; login only cats ~/.local/state/…
ExecStart=%h/.local/bin/koopa-login-health collect
# Treat overall FAIL as success for the timer (we still wrote a snapshot)
SuccessExitStatus=0 1
Nice=10
IOSchedulingClass=best-effort
IOSchedulingPriority=6
[Install]
WantedBy=default.target

13
configs/systemd/user/koopa-login-health.timer Normal file
View file

@ -0,0 +1,13 @@
[Unit]
Description=Timer · koopa login-health snapshot (every 5 min)
Documentation=file:%h/src/koopa/koopa-admin-log/scripts/login-health/README.md
[Timer]
OnBootSec=2min
OnUnitActiveSec=5min
AccuracySec=30s
Persistent=true
Unit=koopa-login-health.service
[Install]
WantedBy=timers.target

1
scripts/README.md
View file

@ -12,6 +12,7 @@
| `taler-shared/` | shared helpers (`upgrade-goa-debs.sh` = apt upgrade **inside** bank/exchange/merchant containers) |
| `taler-landing/` | **hernani user systemd**: central landing `stats.json` (full bank scan + alt units) |
| `monitoring/` | host `/home/hernani/scripts` (tor relay stats) |
| `login-health/` | **SSH login banner**: timer **collect** every 5m; login only **show** snapshot (no probes) |
| `nym/` | **koopa-nym** build/up/status (nym.com nym-node) |
| `taler-wallet-cli/` | thin wrappers; **benchmarks live in** `../benchmarks/` |
| `castopod/` | host `hernani` podman-compose `~/koopa-castopod` — see `castopod/README.md` |

102
scripts/login-health/README.md Normal file
View file

@ -0,0 +1,102 @@
# login-health — status on SSH login (koopa)
When you **SSH into koopa**, show a short **pre-collected** health snapshot so you see whether **git.hacktivism.ch** (Forgejo), Taler GOA, Caddy edge, and the rest of the LIVE inventory are up.
Inventory source: `host/overview/LIVE.md`, `configs/ports.md`.
## Design: no probes at login
| Path | What runs |
|------|-----------|
| **systemd user timer** (every **5 min** + soon after boot) | `koopa-login-health collect` — podman, Caddy, loopback HTTP, write snapshot |
| **SSH / bashrc** | `koopa-login-health show` only — **cat** last snapshot (milliseconds, no curl/podman) |
Snapshot files (`XDG_STATE_HOME`, default `~/.local/state/koopa-login-health/`):
| File | Content |
|------|---------|
| `status.compact` | login view (WARN/FAIL + overall) |
| `status.verbose` | full line list |
| `status.meta` | `overall=`, `collected_at=` (unix), RFC date |
If the snapshot is older than **`KOOPA_HEALTH_STALE_SEC`** (default **900** = 15 min), `show` prints a **stale** warning (timer likely down).
## Login must never break
| Guard | What it does |
|-------|----------------|
| **show only** | bashrc never runs `collect` |
| **`KOOPA_HEALTH_FROM_LOGIN=1`** | always exit 0 |
| **`timeout 2s`** | wall cap to cat a few KB |
| **`\|\| true` / `set +e`** | hook cannot abort the shell |
| **`bash -n`** on install | refuse a broken bashrc hook |
Skip: `KOOPA_SKIP_HEALTH=1 ssh koopa`
## What you see
```text
koopa health koopa Wed, …
git.hacktivism.ch = Forgejo :9024/:9200 · edge Caddy :9000/:9001
OK all critical checks passed (42 ok)
overall: OK ok=42 warn=0 fail=0
checked: Wed, … (3m ago) [timer snapshot; not live at login]
```
## Checks (on collect only)
| Area | What |
|------|------|
| Edge | `caddy.service`, proxy sockets |
| Listen | critical + optional ports |
| Podman | LIVE container names |
| HTTP | loopback backends (2s curl, parallel) |
| Git SSH | TCP `:9200` |
**Critical** (→ overall FAIL): merchant, exchange, bank, forgejo, edge/critical ports.
**Optional** (→ WARN): castopod, bonfire, prime, bt, paivana, tops, tor, nym.
Read-only. No restarts, no secrets.
## Install (on koopa as `hernani`)
```bash
cd ~/src/koopa/koopa-admin-log
./scripts/login-health/install-login-health.sh
```
Installs binary, user units, bashrc hook, enables timer, runs one collect.
Linger (so the timer runs after reboot without an open SSH session):
```bash
sudo loginctl enable-linger hernani
```
## Manual
```bash
koopa-login-health # show compact snapshot
koopa-login-health show -v # verbose snapshot
koopa-login-health show -q # overall (+ stale note)
koopa-login-health collect # refresh now (same as the timer unit)
systemctl --user start koopa-login-health.service
systemctl --user status koopa-login-health.timer
```
## Uninstall
```bash
systemctl --user disable --now koopa-login-health.timer
rm -f ~/.config/systemd/user/koopa-login-health.{service,timer}
systemctl --user daemon-reload
rm -f ~/.local/bin/koopa-login-health
# remove marked block in ~/.bashrc
rm -rf ~/.local/state/koopa-login-health
```
## Related
- Grok workflow `koopa-services-health` (laptop public HTTPS + generated mon pages)
- `scripts/taler-sanity/check_stack-health.sh` (deeper Taler, often root)
- units: `configs/systemd/user/koopa-login-health.{service,timer}`

137
scripts/login-health/install-login-health.sh Executable file
View file

@ -0,0 +1,137 @@
#!/usr/bin/env bash
# Install koopa-login-health:
# - ~/.local/bin/koopa-login-health (show + collect)
# - user systemd timer → collect every 5 min (fresh snapshot)
# - bashrc hook → show only (no probes on login; never blocks shell)
#
# On koopa as hernani:
# cd ~/src/koopa/koopa-admin-log
# ./scripts/login-health/install-login-health.sh
set -euo pipefail
ROOT=$(cd "$(dirname "$0")" && pwd)
REPO_ROOT=$(cd "$ROOT/../.." && pwd)
SRC="$ROOT/koopa-login-health.sh"
UNIT_SRC_DIR="$REPO_ROOT/configs/systemd/user"
BIN_DIR="${HOME}/.local/bin"
DEST="$BIN_DIR/koopa-login-health"
BASHRC="${HOME}/.bashrc"
UNIT_DIR="${XDG_CONFIG_HOME:-$HOME/.config}/systemd/user"
MARKER_BEGIN="# >>> koopa-login-health (koopa-admin-log) >>>"
MARKER_END="# <<< koopa-login-health (koopa-admin-log) <<<"
if [ ! -f "$SRC" ]; then
echo "missing $SRC" >&2
exit 1
fi
# Login: ONLY show snapshot. Never collect. Instant cat; always continue.
HOOK_BODY=$(cat <<'HOOK'
# Best-effort: print last health snapshot. MUST NOT break login. MUST NOT probe.
# Skip: KOOPA_SKIP_HEALTH=1
# Refresh is the user timer koopa-login-health.timer (not this hook).
__koopa_lh() {
[ -n "${KOOPA_SKIP_HEALTH:-}" ] && return 0
case $- in *i*) ;; *) return 0 ;; esac
_b="${HOME}/.local/bin/koopa-login-health"
[ -x "$_b" ] || return 0
if [ -z "${SSH_CONNECTION:-}" ] && [ -z "${SSH_TTY:-}" ]; then
[ "${SHLVL:-1}" -le 2 ] 2>/dev/null || return 0
fi
# show only — no collect; 2s wall cap is enough to cat a few KB
if command -v timeout >/dev/null 2>&1; then
timeout -k 1s 2s env KOOPA_HEALTH_FROM_LOGIN=1 "$_b" show </dev/null >/dev/tty 2>/dev/null \
|| timeout -k 1s 2s env KOOPA_HEALTH_FROM_LOGIN=1 "$_b" show </dev/null 2>/dev/null \
|| true
else
env KOOPA_HEALTH_FROM_LOGIN=1 "$_b" show </dev/null 2>/dev/null || true
fi
return 0
}
set +e
__koopa_lh || true
unset -f __koopa_lh 2>/dev/null || true
HOOK
)
mkdir -p "$BIN_DIR" "$UNIT_DIR"
install -m 0755 "$SRC" "$DEST"
echo "installed $DEST"
for u in koopa-login-health.service koopa-login-health.timer; do
if [ ! -f "$UNIT_SRC_DIR/$u" ]; then
echo "missing unit $UNIT_SRC_DIR/$u" >&2
exit 1
fi
install -m 0644 "$UNIT_SRC_DIR/$u" "$UNIT_DIR/$u"
echo "installed $UNIT_DIR/$u"
done
touch "$BASHRC"
if grep -qF "$MARKER_BEGIN" "$BASHRC" 2>/dev/null; then
tmp=$(mktemp)
awk -v b="$MARKER_BEGIN" -v e="$MARKER_END" '
$0 == b {skip=1; next}
$0 == e {skip=0; next}
!skip {print}
' "$BASHRC" >"$tmp"
mv "$tmp" "$BASHRC"
echo "removed previous bashrc hook"
fi
{
echo ""
echo "$MARKER_BEGIN"
printf '%s\n' "$HOOK_BODY"
echo "$MARKER_END"
} >>"$BASHRC"
echo "installed bashrc hook (show-only; no probes on login)"
hook_tmp=$(mktemp)
{
echo "$MARKER_BEGIN"
printf '%s\n' "$HOOK_BODY"
echo "$MARKER_END"
} >"$hook_tmp"
if ! bash -n "$hook_tmp" || ! bash -n "$BASHRC"; then
echo "ERROR: bashrc syntax check failed — removing hook" >&2
tmp=$(mktemp)
awk -v b="$MARKER_BEGIN" -v e="$MARKER_END" '
$0 == b {skip=1; next}
$0 == e {skip=0; next}
!skip {print}
' "$BASHRC" >"$tmp"
mv "$tmp" "$BASHRC"
rm -f "$hook_tmp"
exit 1
fi
rm -f "$hook_tmp"
echo "bash -n OK"
if ! echo ":$PATH:" | grep -q ":$BIN_DIR:"; then
if ! grep -q '\.local/bin' "$BASHRC" 2>/dev/null; then
echo "export PATH=\"\$HOME/.local/bin:\$PATH\"" >>"$BASHRC"
echo "appended PATH += ~/.local/bin"
fi
fi
# Enable timer + first collect (needs lingering for boot without login)
systemctl --user daemon-reload
systemctl --user enable --now koopa-login-health.timer
systemctl --user start koopa-login-health.service || true
echo "timer: $(systemctl --user is-enabled koopa-login-health.timer 2>/dev/null || echo unknown)"
systemctl --user list-timers koopa-login-health.timer --no-pager 2>/dev/null || true
# linger so timer runs without interactive session
if command -v loginctl >/dev/null 2>&1; then
if [ "$(loginctl show-user "$USER" -p Linger --value 2>/dev/null || echo no)" != "yes" ]; then
echo "NOTE: enable linger so the timer runs after reboot without login:"
echo " sudo loginctl enable-linger $USER"
else
echo "linger already enabled for $USER"
fi
fi
echo "smoke show (FROM_LOGIN):"
KOOPA_HEALTH_FROM_LOGIN=1 "$DEST" show -q || true
echo "done. Login cats snapshot only; collect runs every 5 min via systemd --user."

468
scripts/login-health/koopa-login-health.sh Executable file
View file

@ -0,0 +1,468 @@
#!/usr/bin/env bash
# koopa-login-health — read-only GOA/koopa service health.
# Inventory: koopa-admin-log host/overview/LIVE.md (+ ports.md).
#
# Architecture:
# collect — run probes, write snapshot (systemd user timer; NOT on login)
# show — print last snapshot only (default; used by bashrc on SSH login)
#
# Usage:
# koopa-login-health # show compact snapshot (login path)
# koopa-login-health show -v # show verbose snapshot
# koopa-login-health show -q # overall line only
# koopa-login-health collect # refresh snapshot now
# koopa-login-health collect -v # same (verbose always stored)
# KOOPA_SKIP_HEALTH=1 … # bashrc skip
# KOOPA_HEALTH_FROM_LOGIN=1 # always exit 0 (login hook)
#
# Never mutates host services. Login never runs probes.
set +e
set +u
set +o pipefail 2>/dev/null || true
FROM_LOGIN=0
[ -n "${KOOPA_HEALTH_FROM_LOGIN:-}" ] && FROM_LOGIN=1
_exit_health() {
local c="${1:-0}"
[ "${FROM_LOGIN}" = "1" ] && exit 0
exit "${c}"
}
_bail_login_safe() {
echo "koopa health: ${1:-error} (ignored; login continues)" >&2
_exit_health 0
}
# --- paths (XDG) ---
STATE_DIR="${KOOPA_HEALTH_STATE_DIR:-${XDG_STATE_HOME:-${HOME:-/tmp}/.local/state}/koopa-login-health}"
COMPACT_FILE="${STATE_DIR}/status.compact"
VERBOSE_FILE="${STATE_DIR}/status.verbose"
META_FILE="${STATE_DIR}/status.meta"
# Max age for "current" snapshot (seconds). Older → WARN stale on show.
STALE_SEC="${KOOPA_HEALTH_STALE_SEC:-900}" # 15 min (timer default 5 min)
CURL_M="${KOOPA_HEALTH_CURL_M:-2}"
CMD="show"
VERBOSE=0
QUIET=0
while [ "${1:-}" != "" ]; do
case "$1" in
show|collect|status)
CMD="$1"
[ "$CMD" = "status" ] && CMD="show"
;;
-v|--verbose) VERBOSE=1 ;;
-q|--quiet) QUIET=1 ;;
-h|--help)
sed -n '2,22p' "$0" 2>/dev/null | sed 's/^# \?//' || true
_exit_health 0
;;
*)
echo "unknown arg: $1 (try: show | collect | -v | -q | -h)" >&2
_exit_health 2
;;
esac
shift
done
# ---------- SHOW: never probe ----------
do_show() {
if [ ! -f "${COMPACT_FILE}" ] && [ ! -f "${META_FILE}" ]; then
if [ "${QUIET}" -eq 1 ]; then
echo "unknown (no snapshot — run: systemctl --user start koopa-login-health.service)"
else
echo "koopa health: no snapshot yet"
echo " Collect runs on a user timer (not at login). Refresh once:"
echo " systemctl --user start koopa-login-health.service"
echo " or: koopa-login-health collect"
fi
_exit_health 1
fi
local overall="unknown" collected_at="" age="?" age_sec=-1 now
if [ -f "${META_FILE}" ]; then
# shellcheck disable=SC1090
overall=$(grep '^overall=' "${META_FILE}" 2>/dev/null | tail -1 | cut -d= -f2-)
collected_at=$(grep '^collected_at_rfc=' "${META_FILE}" 2>/dev/null | tail -1 | cut -d= -f2-)
local ts
ts=$(grep '^collected_at=' "${META_FILE}" 2>/dev/null | tail -1 | cut -d= -f2-)
now=$(date +%s 2>/dev/null || echo 0)
if [ -n "${ts}" ] && [ "${ts}" -gt 0 ] 2>/dev/null; then
age_sec=$((now - ts))
if [ "${age_sec}" -lt 0 ] 2>/dev/null; then age_sec=0; fi
if [ "${age_sec}" -lt 120 ] 2>/dev/null; then
age="${age_sec}s ago"
elif [ "${age_sec}" -lt 7200 ] 2>/dev/null; then
age="$((age_sec / 60))m ago"
else
age="$((age_sec / 3600))h ago"
fi
fi
fi
local stale=0
if [ "${age_sec}" -ge 0 ] 2>/dev/null && [ "${age_sec}" -gt "${STALE_SEC}" ] 2>/dev/null; then
stale=1
fi
if [ "${QUIET}" -eq 1 ]; then
if [ "${stale}" -eq 1 ]; then
echo "${overall} (stale ${age})"
else
echo "${overall}"
fi
[ "${stale}" -eq 1 ] && _exit_health 1
case "${overall}" in ok|warn) _exit_health 0 ;; *) _exit_health 1 ;; esac
fi
if [ "${VERBOSE}" -eq 1 ] && [ -f "${VERBOSE_FILE}" ]; then
cat "${VERBOSE_FILE}" 2>/dev/null || true
elif [ -f "${COMPACT_FILE}" ]; then
cat "${COMPACT_FILE}" 2>/dev/null || true
else
echo "koopa health: snapshot incomplete (meta only) overall=${overall}"
fi
echo " checked: ${collected_at:-?} (${age}) [timer snapshot; not live at login]"
if [ "${stale}" -eq 1 ]; then
echo " WARN: snapshot older than ${STALE_SEC}s — is koopa-login-health.timer active?"
echo " systemctl --user status koopa-login-health.timer"
fi
[ "${stale}" -eq 1 ] && _exit_health 1
case "${overall}" in ok|warn) _exit_health 0 ;; *) _exit_health 1 ;; esac
}
# ---------- COLLECT: probes → write snapshot ----------
CRITICAL_CONTAINERS=(
taler-hacktivism
taler-hacktivism-exchange-ansible
taler-hacktivism-bank
koopa-forgejo
)
EXPECT_CONTAINERS=(
taler-hacktivism
taler-hacktivism-exchange-ansible
taler-hacktivism-bank
koopa-castopod
koopa-bonfire
koopa-prime-jellyfin
koopa-prime-qbittorrent
koopa-forgejo
koopa-paivana
koopa-tor-relay
koopa-nym
koopa-tops-ng1
koopa-tops-ng2
koopa-tops-ng3
)
OPTIONAL_CONTAINERS=(
koopa-prime-jellyfin
koopa-prime-qbittorrent
koopa-nym
koopa-tor-relay
koopa-tops-ng1
koopa-tops-ng2
koopa-tops-ng3
koopa-castopod
koopa-bonfire
koopa-paivana
)
PROBES=(
"merchant|http://127.0.0.1:9010/config|200"
"exchange|http://127.0.0.1:9011/keys|200"
"bank|http://127.0.0.1:9012/config|200"
"forgejo|http://127.0.0.1:9024/|200,302,303"
"castopod|http://127.0.0.1:9020/|200,302,303"
"bonfire|http://127.0.0.1:9021/|200,302,303"
"prime|http://127.0.0.1:9022/|200,302,303"
"bt|http://127.0.0.1:9023/|200,302,303,401"
"paivana|http://127.0.0.1:9025/|200,302,303"
"tops-ng1|http://127.0.0.1:9090/|200,302,303"
"tops-ng2|http://127.0.0.1:9091/|200,302,303"
"tops-ng3|http://127.0.0.1:9092/|200,302,303"
)
OPTIONAL_PROBES=(castopod bonfire prime bt paivana tops-ng1 tops-ng2 tops-ng3)
C_OK=; C_WARN=; C_FAIL=; C_DIM=; C_RST=; C_BOLD=
# colors only when writing for a tty later; snapshots are plain + optional ANSI if COLLECT_COLOR=1
if [ "${KOOPA_HEALTH_COLLECT_COLOR:-0}" = "1" ] && [ -t 1 ]; then
C_OK=$'\033[32m'; C_WARN=$'\033[33m'; C_FAIL=$'\033[31m'
C_DIM=$'\033[2m'; C_RST=$'\033[0m'; C_BOLD=$'\033[1m'
fi
OK_N=0
WARN_N=0
FAIL_N=0
LINES=()
is_optional_container() {
local n="$1" x
for x in "${OPTIONAL_CONTAINERS[@]}"; do [ "$x" = "$n" ] && return 0; done
return 1
}
is_optional_probe() {
local n="$1" x
for x in "${OPTIONAL_PROBES[@]}"; do [ "$x" = "$n" ] && return 0; done
return 1
}
record() {
local level="$1" msg="$2"
case "$level" in
OK) OK_N=$((OK_N + 1)); LINES+=(" ${C_OK}OK ${C_RST}${msg}") ;;
WARN) WARN_N=$((WARN_N + 1)); LINES+=(" ${C_WARN}WARN${C_RST} ${msg}") ;;
FAIL) FAIL_N=$((FAIL_N + 1)); LINES+=(" ${C_FAIL}FAIL${C_RST} ${msg}") ;;
INFO) LINES+=(" ${C_DIM}··· ${C_RST}${msg}") ;;
esac
}
code_in_list() {
local code="$1" list="$2" part
local IFS=','
# shellcheck disable=SC2086
for part in $list; do
[ "$code" = "$part" ] && return 0
done
return 1
}
do_collect() {
mkdir -p "${STATE_DIR}" 2>/dev/null || {
echo "koopa health collect: cannot mkdir ${STATE_DIR}" >&2
return 1
}
local HOST NOW caddy_st st sock listen_blob port
local PODMAN running_names name dep tmpdir pids pid
local pname purl pok outf code overall line
HOST=$(hostname -s 2>/dev/null || hostname 2>/dev/null || echo koopa)
NOW=$(date -R 2>/dev/null || date 2>/dev/null || echo "")
local NOW_UNIX
NOW_UNIX=$(date +%s 2>/dev/null || echo 0)
if [ "$(id -u 2>/dev/null)" = "0" ]; then
record INFO "running as root (podman rootless via hernani if available)"
fi
if command -v systemctl >/dev/null 2>&1; then
caddy_st=$(systemctl is-active caddy 2>/dev/null || echo unknown)
if [ "${caddy_st}" = "active" ]; then
record OK "caddy.service active"
else
record FAIL "caddy.service is ${caddy_st} (edge HTTPS)"
fi
for sock in http-proxy.socket https-proxy.socket; do
st=$(systemctl is-active "${sock}" 2>/dev/null || echo unknown)
if [ "${st}" = "active" ] || [ "${st}" = "listening" ]; then
record OK "${sock} ${st}"
else
record WARN "${sock} is ${st}"
fi
done
else
record WARN "systemctl not available"
fi
if command -v ss >/dev/null 2>&1; then
listen_blob=$(ss -lnt 2>/dev/null | awk '{print $4}' || true)
for port in 9000 9001 9010 9011 9012 9024 9200; do
if printf '%s\n' "${listen_blob}" | grep -qE "[:.]${port}\$"; then
record OK "listen :${port}"
else
record FAIL "nothing listening on :${port}"
fi
done
for port in 9020 9021 9022 9023 9025 9090 9091 9092 8080 9080; do
if printf '%s\n' "${listen_blob}" | grep -qE "[:.]${port}\$"; then
record OK "listen :${port} (optional)"
else
record WARN "nothing listening on :${port}"
fi
done
fi
PODMAN=(podman)
if [ "$(id -u 2>/dev/null)" = "0" ] && command -v runuser >/dev/null 2>&1; then
if id hernani >/dev/null 2>&1; then
PODMAN=(runuser -u hernani -- podman)
fi
fi
if ! command -v "${PODMAN[0]}" >/dev/null 2>&1 && [ "${PODMAN[0]}" = "podman" ]; then
record FAIL "podman not found"
running_names=""
else
running_names=$("${PODMAN[@]}" ps --format '{{.Names}}' 2>/dev/null || true)
if [ -z "${running_names}" ]; then
record FAIL "podman ps empty or failed"
fi
for name in "${EXPECT_CONTAINERS[@]}"; do
if printf '%s\n' "${running_names}" | grep -qxF "${name}"; then
st=$("${PODMAN[@]}" ps --format '{{.Names}}\t{{.Status}}' 2>/dev/null \
| awk -F'\t' -v n="${name}" '$1==n {print $2; exit}')
record OK "container ${name} (${st:-up})"
else
if is_optional_container "${name}"; then
record WARN "container ${name} not running"
else
record FAIL "container ${name} not running"
fi
fi
done
for dep in koopa-forgejo-db koopa-castopod-mariadb koopa-castopod-redis koopa-bonfire-db; do
if printf '%s\n' "${running_names}" | grep -qxF "${dep}"; then
record INFO "dep ${dep} running"
else
record INFO "dep ${dep} not in running set (name may differ)"
fi
done
fi
tmpdir=$(mktemp -d 2>/dev/null || true)
if [ -n "${tmpdir}" ] && [ -d "${tmpdir}" ]; then
probe_one() {
local _url="$1" _out="$2" _code
# curl still prints http_code on connect fail; do not append another "000" via ||
_code=$(curl -sk -m "${CURL_M}" -o /dev/null -w '%{http_code}' "${_url}" 2>/dev/null)
[ -n "${_code}" ] || _code="000"
printf '%s\n' "${_code}" >"${_out}"
}
pids=()
for spec in "${PROBES[@]}"; do
IFS='|' read -r pname purl pok <<EOF
${spec}
EOF
probe_one "${purl}" "${tmpdir}/${pname}" &
pids+=($!)
done
for pid in "${pids[@]:-}"; do
wait "${pid}" 2>/dev/null || true
done
for spec in "${PROBES[@]}"; do
IFS='|' read -r pname purl pok <<EOF
${spec}
EOF
code="000"
[ -f "${tmpdir}/${pname}" ] && code=$(awk '{print $1; exit}' "${tmpdir}/${pname}" 2>/dev/null || echo "000")
if code_in_list "${code}" "${pok}"; then
record OK "http ${pname} → ${code} (${purl})"
else
if is_optional_probe "${pname}"; then
record WARN "http ${pname} → ${code} (${purl})"
else
record FAIL "http ${pname} → ${code} (${purl})"
fi
fi
done
rm -rf "${tmpdir}" 2>/dev/null || true
else
record WARN "mktemp failed; skipped HTTP probes"
fi
if command -v timeout >/dev/null 2>&1; then
if timeout 1 bash -c 'echo >/dev/tcp/127.0.0.1/9200' 2>/dev/null; then
record OK "forgejo git-SSH TCP :9200 (git.hacktivism.ch)"
else
record FAIL "forgejo git-SSH TCP :9200 closed"
fi
fi
overall="ok"
[ "${WARN_N}" -gt 0 ] 2>/dev/null && overall="warn"
[ "${FAIL_N}" -gt 0 ] 2>/dev/null && overall="fail"
local header verbose_body compact_body
header="koopa health ${HOST} ${NOW}
git.hacktivism.ch = Forgejo :9024/:9200 · edge Caddy :9000/:9001"
verbose_body="${header}
"
if [ "${#LINES[@]}" -gt 0 ]; then
printf -v _lines '%s\n' "${LINES[@]}"
verbose_body+="${_lines}"
fi
case "${overall}" in
ok) verbose_body+="overall: OK ok=${OK_N} warn=${WARN_N} fail=${FAIL_N}"$'\n' ;;
warn) verbose_body+="overall: WARN ok=${OK_N} warn=${WARN_N} fail=${FAIL_N}"$'\n' ;;
*) verbose_body+="overall: FAIL ok=${OK_N} warn=${WARN_N} fail=${FAIL_N}"$'\n' ;;
esac
compact_body="${header}
"
local has_issue=0
for line in "${LINES[@]:-}"; do
case "${line}" in
*FAIL*|*WARN*) compact_body+="${line}"$'\n'; has_issue=1 ;;
esac
done
if [ "${has_issue}" -eq 0 ]; then
compact_body+=" OK all critical checks passed (${OK_N} ok)"$'\n'
else
compact_body+=" (${OK_N} ok — details: koopa-login-health show -v)"$'\n'
fi
case "${overall}" in
ok) compact_body+="overall: OK ok=${OK_N} warn=${WARN_N} fail=${FAIL_N}"$'\n' ;;
warn) compact_body+="overall: WARN ok=${OK_N} warn=${WARN_N} fail=${FAIL_N}"$'\n' ;;
*) compact_body+="overall: FAIL ok=${OK_N} warn=${WARN_N} fail=${FAIL_N}"$'\n' ;;
esac
# Atomic writes
local tmpc tmpv tmpm
tmpc="${STATE_DIR}/.status.compact.$$"
tmpv="${STATE_DIR}/.status.verbose.$$"
tmpm="${STATE_DIR}/.status.meta.$$"
printf '%s' "${compact_body}" >"${tmpc}" 2>/dev/null || return 1
printf '%s' "${verbose_body}" >"${tmpv}" 2>/dev/null || return 1
{
echo "overall=${overall}"
echo "collected_at=${NOW_UNIX}"
echo "collected_at_rfc=${NOW}"
echo "ok=${OK_N}"
echo "warn=${WARN_N}"
echo "fail=${FAIL_N}"
echo "host=${HOST}"
} >"${tmpm}" 2>/dev/null || return 1
mv -f "${tmpc}" "${COMPACT_FILE}" 2>/dev/null || return 1
mv -f "${tmpv}" "${VERBOSE_FILE}" 2>/dev/null || return 1
mv -f "${tmpm}" "${META_FILE}" 2>/dev/null || return 1
# Optional: print when run interactively
if [ -t 1 ] || [ "${VERBOSE}" -eq 1 ]; then
printf '%s' "${compact_body}"
echo " wrote snapshot → ${STATE_DIR}/"
fi
case "${overall}" in ok|warn) return 0 ;; *) return 1 ;; esac
}
case "${CMD}" in
show)
if [ "${FROM_LOGIN}" = "1" ]; then
( do_show ) 2>&1 || true
_exit_health 0
else
do_show
fi
;;
collect)
# Never invoke collect from login hook
if [ "${FROM_LOGIN}" = "1" ]; then
_bail_login_safe "collect refused under FROM_LOGIN"
fi
do_collect
_exit_health $?
;;
*)
echo "unknown command ${CMD}" >&2
_exit_health 2
;;
esac