diff --git a/scripts/taler-monitoring/.gitignore b/scripts/taler-monitoring/.gitignore new file mode 100644 index 0000000..754bb8e --- /dev/null +++ b/scripts/taler-monitoring/.gitignore @@ -0,0 +1,6 @@ +android-test/apks/ +android-test/out*/ +android-test/artifacts/ +__pycache__/ +secrets.env +*.pyc diff --git a/scripts/taler-monitoring/DEPENDENCIES.md b/scripts/taler-monitoring/DEPENDENCIES.md new file mode 100644 index 0000000..3e7db4c --- /dev/null +++ b/scripts/taler-monitoring/DEPENDENCIES.md @@ -0,0 +1,70 @@ +# taler-monitoring — host package dependencies + +## koopa (openSUSE Tumbleweed, user `hernani`) + +### Required (host-agent / GOA primary: `urls inside versions`) + +| Binary | Package (zypper) | Role | +|--------|------------------|------| +| `bash` | (base) | runner | +| `curl` | `curl` | HTTPS/API probes | +| `python3` | `python313-base` (provides `/usr/bin/python3`) | JSON, HTML site-gen, metrics | +| `podman` | `podman` | **inside**: `podman exec` into taler-hacktivism* | +| `git` | `git` | pin commit SHA in monitoring HTML footer | +| `rsync` | `rsync` | optional: deploy/sync suite | +| `timeout` / `mktemp` / `getent` | coreutils / glibc | helpers | + +**Status on koopa (2026-07-18):** all of the above present; rootless podman owns GOA containers. + +### Recommended (full `urls` QR group) + +| Binary | Package | Role | +|--------|---------|------| +| `qrencode` | **`qrencode`** | encode taler:// / payto:// → PNG | +| `zbarimg` | **`zbar`** | decode PNG; exact payload match | + +Without these, QR checks **WARN** and skip encode/decode (`apt install` messages in code are Debian-oriented; on Tumbleweed use zypper). + +```bash +# as root on koopa +zypper refresh +zypper in qrencode zbar +# verify as hernani: +command -v qrencode zbarimg +``` + +### Optional (not needed for host-agent default) + +| Binary / tool | When | +|---------------|------| +| `jq` | nice-to-have; suite uses python3 for JSON | +| `dig` / `bind-utils` | debug DNS; not required for default phases | +| `ssh` | only for access mode **ssh** (laptop → `KOOPA_SSH` / stage `INSIDE_SSH`); not needed for **host-podman** | +| `node` / `pnpm` + wallet-cli | **e2e** / ladder only (+ secrets) | +| `secrets.env` / admin passwords | e2e only — never system packages | + +### systemd user (no extra packages) + +- `systemd` user session + **`loginctl enable-linger hernani`** (already **Linger=yes** on koopa) +- Units: `scripts/taler-monitoring/host-agent/*.service|path|timer` + +### Caddy HTML publish (separate from monitoring run) + +- Host Caddy already installed (root service). +- Writing `/var/www/monitoring-sites`: either root `rsync` or directory owned/writable by `hernani` (you set this up). + +--- + +## firecuda-external (macOS, outside-only timer) + +| Need | Notes | +|------|--------| +| `bash`, `curl`, `python3`, `rsync`, `git` (optional) | usually Homebrew / Xcode CLT | +| **No** `podman` | phases are `urls versions` only (`SKIP_SSH=1`) | +| `qrencode` / `zbarimg` | optional for QR; `brew install qrencode zbar` if desired | + +--- + +## Franc Paysan hosts (francpaysan / stagepaysan) + +Document package lists under **`francpaysan-admin-log`** when those host-agents are installed (same idea: curl, python3, podman if inside, qrencode/zbar for full urls). diff --git a/scripts/taler-monitoring/README.md b/scripts/taler-monitoring/README.md index 578b683..7d3ae2f 100644 --- a/scripts/taler-monitoring/README.md +++ b/scripts/taler-monitoring/README.md @@ -1,5 +1,10 @@ # taler-monitoring +Host packages (koopa / runners): **[DEPENDENCIES.md](./DEPENDENCIES.md)**. +Host-agents (GOA + FP; shared report pipeline): **[host-agent/README.md](./host-agent/README.md)**. +Public HTML site-gen (same timeout/log/HTML defaults): **[site-gen/README.md](./site-gen/README.md)**. +FP install notes: `francpaysan-admin-log/docs/taler-monitoring-host-agents.md`. + Report for the **GOA** stack with boxed severity badges and **grouped test IDs** `area.group-NN` (e.g. `www.exchange-01`, `e2e.pay-03`): diff --git a/scripts/taler-monitoring/TESTS.md b/scripts/taler-monitoring/TESTS.md index fb829d0..7b667f4 100644 --- a/scripts/taler-monitoring/TESTS.md +++ b/scripts/taler-monitoring/TESTS.md @@ -18,6 +18,8 @@ Every check line has a **global** run number and a **grouped** id: | **www** | `check_urls.sh` | `exchange` `perf` `stats` `bank` `merchant` `paivana` `landing` `qr` | | **inside** | `check_inside.sh` | `ssh` `bank` `exchange` `merchant` `caddy` `load` | | **versions** | `check_versions.sh` | `outside` `inside` `compare` | +| **aptdeploy** | `check_apt_deploy.sh` | `trixie` `trixie-testing` (koopa podman apt-src merchant smoke) | +| **surface** | `check_surface.sh` | **remote-only** ecosystem / `-d` domain inventory (not in default/all/full) | | **sanity** | `check_sanity.sh` | `bank` `exchange` `merchant` | | **server** | `check_server.sh` | (flat `server-NN` or host groups) | | **e2e** | `check_e2e.sh` | `prereq` `load` `bank` `wallet` `atm` `settle` `pay` `shop` `paivana` `dig` `report` | @@ -73,12 +75,14 @@ Numbering follows **executed** checks (early skip may shift later NN inside the | Group | Checks | |-------|--------| -| **inside.ssh-** | ssh reachability / remote collect | +| **inside.host-** | **host-podman**: reachability via `podman exec` on this machine (GOA host / `INSIDE_PODMAN=1`) — not SSH | +| **inside.ssh-** | **ssh**: from laptop/remote — SSH to `KOOPA_SSH` / `INSIDE_SSH`, then podman there | | **inside.bank-** | container, libeufin, postgres, local `/config`, nginx, DNS pins | | **inside.exchange-** | container, httpd, wirewatch, aggregator, transfer, local keys, DNS | | **inside.merchant-** | container, httpd, wirewatch, depositcheck, DNS | | **inside.caddy-** | host reverse-proxy process | | **inside.load-** | host loadavg + RAM; per-container RSS/CPU | +| **inside.disk-** / **aptdeploy.disk-** / **versions.disk-** / **server.disk-** | free space via `df`: **WARN** if use ≥ `DISK_WARN_USED_PCT` (85), **ERROR** if full or use ≥ `DISK_ERR_USED_PCT` (95); host + containers | Remote lines `E|comp|LEVEL|key|detail` each become one numbered result under that component group. @@ -112,8 +116,28 @@ When filing an issue, quote the full id + label, e.g. | Area.group | Meaning | |------------|---------| | **versions.outside-** | deb.taler.net suite index | -| **versions.inside-** | packages in containers | +| **versions.host-** | host-podman: packages via local `podman exec` (`INSIDE_PODMAN=1`) | +| **versions.ssh-** | ssh: packages via `KOOPA_SSH` / `INSIDE_SSH` from laptop/remote | | **versions.compare-** | installed vs suite | +| **aptdeploy.trixie-fresh-** | fresh rebuild container `…-apt-src-trixie` | +| **aptdeploy.trixie-testing-fresh-** | fresh rebuild `…-trixie-testing` | +| **aptdeploy.trixie-upgrade-** | upgrade-track `…-trixie-upgrade` (install once, then mytops-style upgrade) | +| **aptdeploy.trixie-testing-upgrade-** | upgrade-track `…-trixie-testing-upgrade` | + +Standalone: `host-agent/run-aptdeploy.sh` (on koopa local; else `ssh koopa-external`). + +### surface (remote-only public inventory) + +```bash +./taler-monitoring.sh surface # taler.net / gnunet / taler-systems / mattermost / … +./taler-monitoring.sh -d hacktivism.ch surface # all things under that domain +``` + +- Catalog: `surface-catalog.conf` (extend as hosts are found) +- DNS, optional ICMP, TCP ports, then **protocol** checks (HTTPS/HTTP/SSH) before ERROR +- TLS cert expiry; Server-header / `/config` version hints; else **WARN: version unknown** +- OSV CVE query when package+version known → **ERROR** on hits +- Never SSH / never podman on the *targets* | **sanity.bank-** / **.exchange-** / **.merchant-** | public + optional server-side per component | | **ladder.plan-** / **.load-** / **.withdraw-** / **.pay-** / **.report-** | amount ladder (GOA ceiling or stage TESTPAYSAN `max_wire`) | | **server-** | SSH host ports / processes (flat unless grouped later) | diff --git a/scripts/taler-monitoring/check_apt_deploy.sh b/scripts/taler-monitoring/check_apt_deploy.sh new file mode 100755 index 0000000..5240e82 --- /dev/null +++ b/scripts/taler-monitoring/check_apt_deploy.sh @@ -0,0 +1,279 @@ +#!/usr/bin/env bash +# check_apt_deploy.sh — podman apt-src merchant deploy tests on koopa. +# +# Containers (default): +# Fresh (rebuild when repo versions change): +# koopa-taler-deploy-test-apt-src-trixie +# koopa-taler-deploy-test-apt-src-trixie-testing +# Upgrade track (initial install, then mytops-style apt upgrade): +# koopa-taler-deploy-test-apt-src-trixie-upgrade +# koopa-taler-deploy-test-apt-src-trixie-testing-upgrade +# +# Env: +# APT_DEPLOY_CONTAINERS "name:suite:mode …" mode=fresh|upgrade (optional, default fresh) +# APT_DEPLOY_SKIP=1 +# APT_DEPLOY_PODMAN=podman +# +set -euo pipefail +ROOT=$(cd "$(dirname "$0")" && pwd) +# shellcheck source=lib.sh +source "$ROOT/lib.sh" + +if [ "${APT_DEPLOY_SKIP:-0}" = "1" ]; then + echo "[INFO] skip phase aptdeploy (APT_DEPLOY_SKIP=1)" + exit 0 +fi + +set_area aptdeploy +section "aptdeploy · podman apt-src merchant (fresh + upgrade tracks)" + +PODMAN_BIN="${APT_DEPLOY_PODMAN:-podman}" +DEFAULT_LIST="\ +koopa-taler-deploy-test-apt-src-trixie:trixie:fresh \ +koopa-taler-deploy-test-apt-src-trixie-testing:trixie-testing:fresh \ +koopa-taler-deploy-test-apt-src-trixie-upgrade:trixie:upgrade \ +koopa-taler-deploy-test-apt-src-trixie-testing-upgrade:trixie-testing:upgrade" +LIST="${APT_DEPLOY_CONTAINERS:-$DEFAULT_LIST}" + +if ! command -v "$PODMAN_BIN" >/dev/null 2>&1; then + err "podman" "podman binary missing ($PODMAN_BIN)" + summary + exit 1 +fi +ok "podman" "$("$PODMAN_BIN" --version 2>/dev/null | head -1)" +info "host" "$(hostname 2>/dev/null || echo unknown) · whoami=$(whoami)" + +fail_any=0 + +print_pkg_table() { + local name=$1 + local rows + rows=$("$PODMAN_BIN" exec "$name" bash -lc ' + echo "| package | version |" + echo "|---------|---------|" + for p in taler-merchant libtalermerchant libtalerexchange libdonau libgnunet \ + taler-merchant-webui taler-merchant-typst taler-terms-generator; do + v=$(dpkg-query -W -f="\${Version}" "$p" 2>/dev/null || echo missing) + echo "| $p | $v |" + done + ' 2>/dev/null || echo "| (query failed) | |") + info "pkg-table" "$name" + # emit as plain lines so HTML log keeps the table + while IFS= read -r line; do + [ -n "$line" ] || continue + info "pkg" "$line" + done <<<"$rows" +} + +check_ldd_deep() { + local name=$1 + local out + out=$("$PODMAN_BIN" exec "$name" bash -lc ' + set +e + bin=$(command -v taler-merchant-httpd) + echo "=== ldd taler-merchant-httpd ===" + ldd "$bin" 2>&1 | grep -E "libtalerutil|libdonau|libgnunet|not found" || true + echo "=== ldd libdonau (if present) ===" + for f in /usr/lib/*/libdonau.so* /usr/lib/*/libdonauutil.so*; do + [ -e "$f" ] || continue + echo "-- $f --" + ldd "$f" 2>&1 | grep -E "libtalerutil|not found" || true + done + echo "=== libtalerutil files ===" + ls -la /usr/lib/*/libtalerutil* 2>/dev/null || echo "(none)" + ' 2>/dev/null || echo "ldd probe failed") + while IFS= read -r line; do + [ -n "$line" ] || continue + info "ldd" "$line" + done <<<"$out" + if echo "$out" | grep -q 'not found'; then + err "ldd" "shared library missing in $name" "see ldd lines above" + return 1 + fi + return 0 +} + +check_systemd() { + local name=$1 + local out active_target active_httpd enabled + out=$("$PODMAN_BIN" exec "$name" bash -lc ' + set +e + echo "system: $(systemctl is-system-running 2>&1)" + echo "target_enabled: $(systemctl is-enabled taler-merchant.target 2>&1)" + echo "target_active: $(systemctl is-active taler-merchant.target 2>&1)" + echo "httpd_active: $(systemctl is-active taler-merchant-httpd.service 2>&1)" + echo "httpd_enabled: $(systemctl is-enabled taler-merchant-httpd.service 2>&1)" + systemctl start taler-merchant.target 2>&1 | tail -5 + sleep 2 + echo "after_start_target: $(systemctl is-active taler-merchant.target 2>&1)" + echo "after_start_httpd: $(systemctl is-active taler-merchant-httpd.service 2>&1)" + systemctl --failed --no-legend 2>/dev/null | grep -i taler || echo "failed_taler: none" + ' 2>/dev/null || echo "systemd probe failed") + while IFS= read -r line; do + [ -n "$line" ] || continue + info "systemd" "$line" + done <<<"$out" + + active_httpd=$(echo "$out" | sed -n 's/^after_start_httpd: //p' | tail -1) + if [ "$active_httpd" = "active" ]; then + ok "systemd httpd" "taler-merchant-httpd active after start taler-merchant.target" + return 0 + fi + # soft if target not shipped as enabled in container — still ERROR for deploy smoke + err "systemd httpd" "taler-merchant-httpd not active" "after_start_httpd=${active_httpd:-?} · see systemd lines" + return 1 +} + +check_merchant_basics() { + local name=$1 + local out code + # unix socket or curl localhost if httpd listens + out=$("$PODMAN_BIN" exec "$name" bash -lc ' + set +e + echo "=== binaries ===" + command -v taler-merchant-httpd taler-merchant-dbinit taler-config 2>/dev/null + echo "=== --version ===" + taler-merchant-httpd --version 2>&1 + echo "=== config probe ===" + # try common paths without TLS + for u in \ + http://127.0.0.1:9966/config \ + http://127.0.0.1:8081/config \ + http://127.0.0.1/config + do + code=$(curl -sS -m 2 -o /tmp/mcfg -w "%{http_code}" "$u" 2>/dev/null || echo 000) + echo "curl $u -> $code" + [ "$code" = "200" ] && head -c 120 /tmp/mcfg && echo + done + sock=$(ls /run/taler-merchant/httpd/*.sock 2>/dev/null | head -1) + if [ -n "$sock" ]; then + echo "socket: $sock" + code=$(curl -sS -m 2 --unix-socket "$sock" -o /tmp/mcfg -w "%{http_code}" http://localhost/config 2>/dev/null || echo 000) + echo "curl --unix-socket -> $code" + [ "$code" = "200" ] && head -c 160 /tmp/mcfg && echo + else + echo "socket: (none under /run/taler-merchant/httpd/)" + fi + ' 2>/dev/null || echo "basics probe failed") + + while IFS= read -r line; do + [ -n "$line" ] || continue + info "basics" "$line" + done <<<"$out" + + if echo "$out" | grep -qi 'error while loading shared libraries'; then + err "httpd" "shared library error on --version" "see basics/ldd" + return 1 + fi + if ! echo "$out" | grep -q 'taler-merchant-httpd --version\|v\.\|taler-merchant-httpd'; then + # version line varies; success if exit was ok — check via separate exec + : + fi + + set +e + "$PODMAN_BIN" exec "$name" taler-merchant-httpd --version >/dev/null 2>&1 + local ec=$? + set -e + if [ "$ec" -eq 0 ]; then + ok "httpd --version" "exit 0" + else + err "httpd --version" "exit $ec" + return 1 + fi + + # /config is nice-to-have if stack is fully configured + if echo "$out" | grep -qE 'curl .* -> 200'; then + ok "merchant /config" "HTTP 200 (local)" + else + warn "merchant /config" "no local HTTP 200 (unit may need BASE_URL/db — still report version/ldd)" + fi + return 0 +} + +check_one() { + local name="$1" expect_suite="$2" mode="${3:-fresh}" + local st suite_line pkgs + + # group ids: aptdeploy.trixie-01 / aptdeploy.trixie-testing-upgrade-01 + set_group "${expect_suite}${mode:+-$mode}" + section "aptdeploy · $name (suite=$expect_suite mode=$mode)" + + if ! "$PODMAN_BIN" container exists "$name" 2>/dev/null; then + err "container" "$name missing" "run ensure-apt-deploy-test-containers.sh" + fail_any=1 + return + fi + st=$("$PODMAN_BIN" inspect -f '{{.State.Status}}' "$name" 2>/dev/null || echo unknown) + if [ "$st" != "running" ]; then + err "container" "$name not running" "status=$st" + fail_any=1 + return + fi + ok "container" "$name running (mode=$mode)" + + suite_line=$("$PODMAN_BIN" exec "$name" bash -lc \ + 'grep -h "^Suites:" /etc/apt/sources.list.d/*taler* 2>/dev/null | head -1' 2>/dev/null || true) + if echo "$suite_line" | grep -qE "Suites:[[:space:]]*${expect_suite}([[:space:]]|$)"; then + ok "apt suite" "$expect_suite" + else + err "apt suite" "expected $expect_suite" "got: ${suite_line:-empty}" + fail_any=1 + fi + + print_pkg_table "$name" + + if ! check_merchant_basics "$name"; then + fail_any=1 + check_ldd_deep "$name" || fail_any=1 + else + # still show ldd summary as info when ok + ldd_out=$("$PODMAN_BIN" exec "$name" bash -lc \ + 'ldd "$(command -v taler-merchant-httpd)" 2>&1 | grep -E "libtalerutil|not found" || true' 2>/dev/null || true) + if echo "$ldd_out" | grep -q 'not found'; then + err "libtalerutil" "not found" "$ldd_out" + check_ldd_deep "$name" || true + fail_any=1 + else + info "libtalerutil" "$(echo "$ldd_out" | tr '\n' ' ')" + fi + fi + + if ! check_systemd "$name"; then + fail_any=1 + check_ldd_deep "$name" || true + fi + + # Disk free inside this deploy-test container (+ host once, first container only) + if [ "${_APT_DEPLOY_HOST_DISK_DONE:-0}" != "1" ]; then + set_group disk + mon_disk_check_host "host" || fail_any=1 + _APT_DEPLOY_HOST_DISK_DONE=1 + fi + set_group disk + if ! mon_disk_check_podman "$name" "$PODMAN_BIN"; then + fail_any=1 + fi +} + +for entry in $LIST; do + # name:suite or name:suite:mode + cname=${entry%%:*} + rest=${entry#*:} + case "$rest" in + *:*) + suite=${rest%%:*} + mode=${rest#*:} + ;; + *) + suite=$rest + mode=fresh + ;; + esac + check_one "$cname" "$suite" "$mode" +done + +summary +if [ "$fail_any" -ne 0 ]; then + exit 1 +fi +exit 0 diff --git a/scripts/taler-monitoring/check_inside.sh b/scripts/taler-monitoring/check_inside.sh index 0584c9b..7f62880 100755 --- a/scripts/taler-monitoring/check_inside.sh +++ b/scripts/taler-monitoring/check_inside.sh @@ -1,9 +1,14 @@ #!/usr/bin/env bash # Inside status for bank / exchange / merchant containers. +# +# Access modes (shown in the log as flags): +# host-podman — this host runs podman exec into containers (no SSH). +# INSIDE_PODMAN=1 or INSIDE_MODE=local-podman, or auto on GOA host. +# ssh — from a laptop/remote: SSH to KOOPA_SSH / INSIDE_SSH, then podman there. +# # Profiles: -# koopa (default LOCAL_STACK=1) — SSH KOOPA_SSH, containers taler-hacktivism* -# stage-lfp (TESTPAYSAN) — SSH INSIDE_SSH (francpaysan-stage-user / stagepaysan), -# low-priv podman only; ports 9030–9032; stats remain outside-in (urls). +# koopa (default LOCAL_STACK=1) — taler-hacktivism* containers +# stage-lfp (TESTPAYSAN) — SSH INSIDE_SSH (stagepaysan), low-priv podman set -euo pipefail ROOT=$(cd "$(dirname "$0")" && pwd) # shellcheck source=lib.sh @@ -12,7 +17,6 @@ source "$ROOT/lib.sh" source "$ROOT/metrics.sh" set_area inside -set_group ssh PROFILE="${INSIDE_PROFILE:-}" if [ -z "$PROFILE" ]; then @@ -25,7 +29,27 @@ if [ -z "$PROFILE" ]; then fi fi -section "inside · collect (${PROFILE})" +# Resolve host-podman vs ssh before first check ID +_use_local_podman=0 +if [ "$PROFILE" != "stage-lfp" ]; then + if [ "${INSIDE_PODMAN:-0}" = "1" ] || [ "${INSIDE_MODE:-}" = "local-podman" ]; then + _use_local_podman=1 + elif command -v podman >/dev/null 2>&1 \ + && podman ps --format '{{.Names}}' 2>/dev/null | grep -qE 'taler-hacktivism'; then + _use_local_podman=1 + fi +fi +if [ "$_use_local_podman" = "1" ]; then + INSIDE_ACCESS=host-podman + set_group host +else + INSIDE_ACCESS=ssh + set_group ssh +fi +export INSIDE_ACCESS + +section "inside · collect (${PROFILE} · access=${INSIDE_ACCESS})" +info "flags" "INSIDE_ACCESS=${INSIDE_ACCESS} INSIDE_PODMAN=${INSIDE_PODMAN:-0} INSIDE_MODE=${INSIDE_MODE:-} LOCAL_STACK=${LOCAL_STACK:-0} SKIP_SSH=${SKIP_SSH:-0} KOOPA_SSH=${KOOPA_SSH:-} INSIDE_SSH=${INSIDE_SSH:-}" # --------------------------------------------------------------------------- # stage-lfp: low-priv stagepaysan on francpaysan-host @@ -244,31 +268,70 @@ EOF info "stage host" "load probe empty" fi + set_group disk + section "inside · disk free space (stage host + containers)" + _disk_raw=$(mon_ssh_bash "$SSH_HOST" "${STAGE_SSH_T:-24}" <<'DISK' || true +set +e +echo "###HOST###" +df -Pk / /var /home /tmp /mnt/data 2>/dev/null || df -Pk +echo "###CTRS###" +for c in $(podman ps --format '{{.Names}}' 2>/dev/null); do + echo "###CTR $c###" + podman exec "$c" df -Pk / /var /tmp 2>/dev/null || podman exec "$c" df -Pk 2>/dev/null +done +DISK +) + _host_df=$(printf '%s\n' "$_disk_raw" | sed -n '/^###HOST###$/,/^###CTRS###$/p' | sed '1d;$d') + mon_disk_check_remote_text "ssh:${SSH_HOST}" "$_host_df" || true + _ctr=""; _buf="" + while IFS= read -r _line || [ -n "$_line" ]; do + case "$_line" in + '###CTR '*) + if [ -n "$_ctr" ]; then mon_disk_check_remote_text "ctr:${_ctr}" "$_buf" || true; fi + _ctr=${_line####CTR }; _ctr=${_ctr%###}; _buf="" + ;; + '###CTRS###'|'###HOST###') ;; + *) [ -n "$_ctr" ] && _buf="${_buf}${_line}"$'\n' ;; + esac + done <<<"$_disk_raw" + [ -n "$_ctr" ] && mon_disk_check_remote_text "ctr:${_ctr}" "$_buf" || true + summary exit 0 fi # --------------------------------------------------------------------------- -# koopa (default) — existing remote collect +# koopa (default) — host-podman exec (on GOA host) or SSH to KOOPA_SSH (laptop) # --------------------------------------------------------------------------- -section "inside · collect from koopa" +section "inside · collect from koopa (access=${INSIDE_ACCESS})" -if [ "${SKIP_SSH}" = "1" ] && [ "${LOCAL_STACK:-0}" != "1" ]; then - warn "ssh" "SKIP_SSH=1 and not local — skipped" +if [ "$_use_local_podman" = "1" ]; then + set_group host + if ! command -v podman >/dev/null 2>&1; then + err "host" "INSIDE_PODMAN/host-podman but podman missing" + summary + exit 1 + fi + ok "host→container" "podman exec on this host (INSIDE_ACCESS=host-podman · no SSH)" +elif [ "${SKIP_SSH}" = "1" ] && [ "${LOCAL_STACK:-0}" != "1" ]; then + set_group ssh + warn "ssh" "SKIP_SSH=1 and not host-podman — skipped" summary exit 0 -fi - -if ! koopa_ssh_ok; then - err "ssh" "cannot reach ${KOOPA_SSH} in ${SSH_CONNECT_TIMEOUT}s — set SKIP_SSH=1 to skip inside" +elif ! koopa_ssh_ok; then + set_group ssh + err "ssh" "cannot reach ${KOOPA_SSH} — from laptop use KOOPA_SSH=; on host set INSIDE_PODMAN=1" summary exit 1 +else + set_group ssh + ok "ssh ${KOOPA_SSH}" "remote host then podman (INSIDE_ACCESS=ssh · laptop/remote)" fi -ok "ssh ${KOOPA_SSH}" -# One short remote script (≤ SSH_CMD_TIMEOUT). Every slow step is local curl -m 3 or quick pgrep. -RAW=$( - koopa_ssh_bash "${SSH_CMD_TIMEOUT}" <<'REMOTE' || true +# Collect script (file → bash local or ssh bash -s) +_INSIDE_SCRIPT=$(mktemp) +trap 'rm -f "${_INSIDE_SCRIPT:-}"' RETURN +cat >"$_INSIDE_SCRIPT" <<'REMOTE' set +e emit() { printf 'E|%s|%s|%s|%s\n' "$1" "$2" "$3" "$(printf '%s' "${4:-}" | tr '\n\r' ' ' | head -c 200)"; } # quick curl @@ -376,23 +439,37 @@ else fi echo DONE REMOTE -) + +if [ "$_use_local_podman" = "1" ]; then + RAW=$(bash "$_INSIDE_SCRIPT" || true) +else + RAW=$(koopa_ssh_bash "${SSH_CMD_TIMEOUT}" <"$_INSIDE_SCRIPT" || true) +fi +rm -f "$_INSIDE_SCRIPT" if [ -z "$RAW" ] || ! echo "$RAW" | grep -q '^E|'; then - err "ssh" "remote timed out or empty (cap ${SSH_CMD_TIMEOUT}s)" + if [ "$_use_local_podman" = "1" ]; then + err "host" "collect empty (INSIDE_ACCESS=host-podman · podman exec failed)" + else + err "ssh" "collect empty (INSIDE_ACCESS=ssh · cap ${SSH_CMD_TIMEOUT}s · host=${KOOPA_SSH})" + fi summary exit 1 fi +# Fallback group for non-component rows: host (podman) vs ssh (remote) +_access_grp=ssh +[ "$_use_local_podman" = "1" ] && _access_grp=host + _last_inside_grp="" while IFS= read -r line; do case "$line" in E\|*) IFS='|' read -r _ comp level key detail <<<"$line" - # Group IDs by component so issues map cleanly: inside.bank-02, inside.exchange-04 + # Group IDs by component: inside.bank-02, inside.exchange-04; access = host|ssh case "$comp" in bank|exchange|merchant|caddy) _g="$comp" ;; - *) _g="ssh" ;; + *) _g="$_access_grp" ;; esac if [ "$_g" != "$_last_inside_grp" ]; then set_group "$_g" @@ -415,4 +492,55 @@ METRICS_DIR="${METRICS_DIR:-$(mktemp -d)}" export METRICS_DIR metrics_report_load "${METRICS_DIR}/load-inside.json" "inside" || true +# Disk free space: host + taler containers (WARN tight, ERROR full) +set_group disk +section "inside · disk free space" +_disk_ec=0 +if [ "$_use_local_podman" = "1" ]; then + mon_disk_check_host "host" || _disk_ec=1 + while read -r _cname; do + [ -n "$_cname" ] || continue + mon_disk_check_podman "$_cname" || _disk_ec=1 + done < <(podman ps --format '{{.Names}}' 2>/dev/null | grep -iE 'hacktivism|taler-|stage-lfp|lfp-' || true) +else + # remote host via SSH: df on host + each container + _disk_raw=$(koopa_ssh_bash "${SSH_CMD_TIMEOUT:-20}" <<'DISK' || true +set +e +echo "###HOST###" +df -Pk / /var /home /tmp 2>/dev/null || df -Pk +echo "###CTRS###" +for c in $(podman ps --format '{{.Names}}' 2>/dev/null); do + echo "###CTR $c###" + podman exec "$c" df -Pk / /var /tmp 2>/dev/null || podman exec "$c" df -Pk 2>/dev/null +done +DISK +) + _host_df=$(printf '%s\n' "$_disk_raw" | sed -n '/^###HOST###$/,/^###CTRS###$/p' | sed '1d;$d') + mon_disk_check_remote_text "ssh:${KOOPA_SSH:-remote}" "$_host_df" || _disk_ec=1 + _ctr="" + _buf="" + while IFS= read -r _line || [ -n "$_line" ]; do + case "$_line" in + '###CTR '*) + if [ -n "$_ctr" ]; then + mon_disk_check_remote_text "ctr:${_ctr}" "$_buf" || _disk_ec=1 + fi + _ctr=${_line####CTR } + _ctr=${_ctr%###} + _buf="" + ;; + '###CTRS###'|'###HOST###') ;; + *) + if [ -n "$_ctr" ]; then + _buf="${_buf}${_line}"$'\n' + fi + ;; + esac + done <<<"$_disk_raw" + if [ -n "$_ctr" ]; then + mon_disk_check_remote_text "ctr:${_ctr}" "$_buf" || _disk_ec=1 + fi +fi +unset _disk_raw _host_df _ctr _buf _line _cname _disk_ec + summary diff --git a/scripts/taler-monitoring/check_server.sh b/scripts/taler-monitoring/check_server.sh index 8412a12..3a9618a 100755 --- a/scripts/taler-monitoring/check_server.sh +++ b/scripts/taler-monitoring/check_server.sh @@ -110,6 +110,15 @@ if systemctl is-active caddy >/dev/null 2>&1 || pgrep -x caddy >/dev/null 2>&1; else report WARN "caddy" "not detected as active" fi + +# disk free (host + containers) — lines for mon_disk_check_remote_text on laptop side +echo "###DISK_HOST###" +df -Pk / /var /home /tmp 2>/dev/null || df -Pk +echo "###DISK_CTRS###" +for c in $(podman ps --format '{{.Names}}' 2>/dev/null); do + echo "###CTR $c###" + podman exec "$c" df -Pk / /var /tmp 2>/dev/null || podman exec "$c" df -Pk 2>/dev/null +done REMOTE ) @@ -126,4 +135,24 @@ while IFS= read -r line; do esac done <<<"$REMOTE" +set_group disk +section "server · disk free space" +_host_df=$(printf '%s\n' "$REMOTE" | sed -n '/^###DISK_HOST###$/,/^###DISK_CTRS###$/p' | sed '1d;$d') +mon_disk_check_remote_text "ssh:${KOOPA_SSH}" "$_host_df" || true +_ctr=""; _buf="" +while IFS= read -r _line || [ -n "$_line" ]; do + case "$_line" in + '###CTR '*) + if [ -n "$_ctr" ]; then mon_disk_check_remote_text "ctr:${_ctr}" "$_buf" || true; fi + _ctr=${_line####CTR }; _ctr=${_ctr%###}; _buf="" + ;; + '###DISK_CTRS###'|'###DISK_HOST###') ;; + *) + # only buffer after we entered CTR section + if [ -n "$_ctr" ]; then _buf="${_buf}${_line}"$'\n'; fi + ;; + esac +done <<<"$REMOTE" +[ -n "$_ctr" ] && mon_disk_check_remote_text "ctr:${_ctr}" "$_buf" || true + summary diff --git a/scripts/taler-monitoring/check_surface.sh b/scripts/taler-monitoring/check_surface.sh new file mode 100755 index 0000000..1452d3b --- /dev/null +++ b/scripts/taler-monitoring/check_surface.sh @@ -0,0 +1,692 @@ +#!/usr/bin/env bash +# check_surface.sh — REMOTE-ONLY public surface / ecosystem scan. +# +# NOT in default phases. Explicit: +# ./taler-monitoring.sh surface +# ./taler-monitoring.sh -d hacktivism.ch surface +# +# Without -d / with generic ecosystem: catalog of taler.net, gnunet.org, +# taler-systems.com, taler-ops, mattermost, … +# With -d DOMAIN: expand domains.conf stack hosts + catalog entries matching +# that domain + common subdomain guesses; port/protocol probes only remote. +# +# Rules: +# - no SSH into targets, no local podman on targets +# - ICMP/port alone never proves "down" — confirm via expected protocol +# - catalogued host unreachable via protocol → ERROR +# - version unknown after probes → WARN +# - CVE hit (OSV) for identified software+version → ERROR +# +set -euo pipefail +ROOT=$(cd "$(dirname "$0")" && pwd) +# shellcheck source=lib.sh +source "$ROOT/lib.sh" + +set_area surface +section "surface · remote ecosystem / domain inventory (outside-in only)" + +CATALOG="${SURFACE_CATALOG:-$ROOT/surface-catalog.conf}" +# Common ports when scanning a host discovered without explicit list +DEFAULT_SCAN_PORTS="${SURFACE_SCAN_PORTS:-22,80,443,993,8443,9418}" +PORT_TIMEOUT="${SURFACE_PORT_TIMEOUT:-2}" +HTTP_TIMEOUT="${SURFACE_HTTP_TIMEOUT:-12}" +# CVE via OSV (public, no key). Disable: SURFACE_CVE=0 +: "${SURFACE_CVE:=1}" +# Extra ports when -d domain mode +DOMAIN_EXTRA_PORTS="${SURFACE_DOMAIN_PORTS:-22,80,443,8443}" + +declare -A HOST_PORTS # host -> comma ports +declare -A HOST_PROTO # host -> expect proto +declare -A HOST_LABEL # host -> short label +declare -A HOST_EXPECT # host -> 1 if catalogued (must respond) +ORDERED_HOSTS=() + +add_host() { + local h="$1" ports="${2:-}" proto="${3:-https}" label="${4:-}" expect="${5:-1}" + h=$(printf '%s' "$h" | tr 'A-Z' 'a-z' | sed 's#^https\?://##;s#/.*##;s/\.$//') + [ -n "$h" ] || return 0 + if [ -z "${HOST_PORTS[$h]+x}" ]; then + ORDERED_HOSTS+=("$h") + HOST_PORTS[$h]="${ports:-443}" + HOST_PROTO[$h]="${proto:-https}" + HOST_LABEL[$h]="${label:-$h}" + HOST_EXPECT[$h]="$expect" + else + # merge ports + local p + for p in ${ports//,/ }; do + case ",${HOST_PORTS[$h]}," in + *",$p,"*) ;; + *) HOST_PORTS[$h]="${HOST_PORTS[$h]},$p" ;; + esac + done + [ "${HOST_EXPECT[$h]}" = "1" ] || HOST_EXPECT[$h]="$expect" + fi +} + +load_catalog() { + local line h ports proto label + [ -f "$CATALOG" ] || { + warn "catalog" "missing $CATALOG — using empty base list" + return 0 + } + while IFS= read -r line || [ -n "$line" ]; do + line=${line%%#*} + line=$(echo "$line" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//') + [ -z "$line" ] && continue + # host ports proto label + h=$(echo "$line" | awk '{print $1}') + ports=$(echo "$line" | awk '{print $2}') + proto=$(echo "$line" | awk '{print $3}') + label=$(echo "$line" | awk '{print $4}') + [ "$ports" = "-" ] && ports="" + add_host "$h" "$ports" "${proto:-https}" "${label:-}" 1 + done <"$CATALOG" + info "catalog" "loaded $CATALOG · ${#ORDERED_HOSTS[@]} hosts" +} + +# Clear inventory (used when switching to domain-only scope) +clear_hosts() { + ORDERED_HOSTS=() + unset HOST_PORTS HOST_PROTO HOST_LABEL HOST_EXPECT + declare -gA HOST_PORTS HOST_PROTO HOST_LABEL HOST_EXPECT +} + +# Expand for -d domain: ONLY that domain (remote). No whole-ecosystem catalog. +expand_domain_scope() { + local d="$1" h + d=${d#https://}; d=${d%/} + d=${d#http://} + info "scope" "domain mode −d $d (remote only · not full ecosystem catalog)" + + clear_hosts + + # stack endpoints from applied profile (always expected) + for h in \ + "${BANK_PUBLIC:-}" \ + "${EXCHANGE_PUBLIC:-}" \ + "${MERCHANT_PUBLIC:-}" \ + "${PAIVANA_PUBLIC:-}" + do + [ -n "$h" ] || continue + add_host "$h" "80,443,$DOMAIN_EXTRA_PORTS" https "stack" 1 + done + + # apex + common subdomains (expected if DNS exists — set expect after DNS in scan, + # but catalogued stack already expected; guesses start optional) + for sub in "" www bank exchange taler merchant backend shop shops stage \ + git docs www2 api static media landing mon401 + do + if [ -z "$sub" ]; then + h="$d" + else + h="${sub}.${d}" + fi + # stack hosts already added as expect=1; guesses optional until we promote + if [ -z "${HOST_PORTS[$h]+x}" ]; then + add_host "$h" "$DOMAIN_EXTRA_PORTS" https "guess" 0 + fi + done + + # also pick catalog lines that belong to this domain only + if [ -f "$CATALOG" ]; then + local line ports proto label + while IFS= read -r line || [ -n "$line" ]; do + line=${line%%#*} + line=$(echo "$line" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//') + [ -z "$line" ] && continue + h=$(echo "$line" | awk '{print $1}') + h=$(printf '%s' "$h" | tr 'A-Z' 'a-z' | sed 's#^https\?://##;s#/.*##') + if [ "$h" = "$d" ] || [[ "$h" == *".$d" ]]; then + ports=$(echo "$line" | awk '{print $2}') + proto=$(echo "$line" | awk '{print $3}') + label=$(echo "$line" | awk '{print $4}') + [ "$ports" = "-" ] && ports="" + add_host "$h" "$ports" "${proto:-https}" "${label:-dom}" 1 + fi + done <"$CATALOG" + fi +} + +# Ecosystem mode: full catalog; all entries expected +expand_ecosystem_scope() { + info "scope" "ecosystem mode (taler.net / gnunet.org / taler-systems.com / taler-ops / mattermost, … · remote only)" + local h + for h in "${ORDERED_HOSTS[@]}"; do + HOST_EXPECT[$h]=1 + done +} + +# --- remote probes (no SSH) --- +dns_ok() { + local h="$1" + getent ahosts "$h" >/dev/null 2>&1 || getent hosts "$h" >/dev/null 2>&1 +} + +# TCP connect only (not proof of service) +tcp_open() { + local h="$1" port="$2" + if command -v timeout >/dev/null 2>&1; then + timeout "$PORT_TIMEOUT" bash -c "echo >/dev/tcp/${h}/${port}" 2>/dev/null + else + bash -c "echo >/dev/tcp/${h}/${port}" 2>/dev/null + fi +} + +# ICMP optional — never sole grounds for ERROR +ping_host() { + local h="$1" + ping -c 1 -W 2 "$h" >/dev/null 2>&1 || ping -c 1 -w 2 "$h" >/dev/null 2>&1 +} + +# HTTPS probe: code, server header, version hints, cert subject/dates +probe_https() { + local h="$1" port="${2:-443}" + local url hdr body code server via cert_end subj san + url="https://${h}/" + [ "$port" != "443" ] && url="https://${h}:${port}/" + body=$(mktemp) + hdr=$(mktemp) + code=$(curl -skS -m "$HTTP_TIMEOUT" -D "$hdr" -o "$body" -w '%{http_code}' \ + --connect-timeout "$PORT_TIMEOUT" "$url" 2>/dev/null || echo 000) + server=$(awk 'BEGIN{IGNORECASE=1} /^server:/{sub(/\r$/,""); sub(/^server:[[:space:]]*/,""); print; exit}' "$hdr" 2>/dev/null || true) + via=$(awk 'BEGIN{IGNORECASE=1} /^x-powered-by:/{sub(/\r$/,""); sub(/^[^:]+:[[:space:]]*/,""); print; exit}' "$hdr" 2>/dev/null || true) + # cert + cert_end=$(echo | openssl s_client -servername "$h" -connect "${h}:${port}" 2>/dev/null \ + | openssl x509 -noout -enddate 2>/dev/null | sed 's/notAfter=//') + subj=$(echo | openssl s_client -servername "$h" -connect "${h}:${port}" 2>/dev/null \ + | openssl x509 -noout -subject 2>/dev/null | head -1) + # taler version hints in body + local taler_hint + taler_hint=$(grep -oE 'taler[^"[:space:]]{0,40}|GNU Taler|libeufin|gnunet' "$body" 2>/dev/null | head -3 | tr '\n' ' ' || true) + # config JSON version if /config works + local cfg_ver="" + local ccode + ccode=$(curl -skS -m "$HTTP_TIMEOUT" -o "$body" -w '%{http_code}' \ + "https://${h}/config" 2>/dev/null || echo 000) + if [ "$ccode" = "200" ]; then + cfg_ver=$(python3 -c 'import json,sys +try: + d=json.load(open(sys.argv[1])) + print(d.get("version") or d.get("name") or d.get("currency") or "config-json") +except Exception: + print("")' "$body" 2>/dev/null || true) + fi + rm -f "$hdr" "$body" + printf 'code=%s server=%s powered=%s cert_end=%s cfg=%s hint=%s' \ + "$code" "${server:-}" "${via:-}" "${cert_end:-}" "${cfg_ver:-}" "${taler_hint:-}" + # return 0 if HTTP answered (any 2xx/3xx/4xx — service is up) + case "$code" in + 2??|3??|4??) return 0 ;; + *) return 1 ;; + esac +} + +probe_http() { + local h="$1" port="${2:-80}" + local url code + url="http://${h}/" + [ "$port" != "80" ] && url="http://${h}:${port}/" + code=$(curl -sS -m "$HTTP_TIMEOUT" -o /dev/null -w '%{http_code}' \ + --connect-timeout "$PORT_TIMEOUT" "$url" 2>/dev/null || echo 000) + printf 'code=%s' "$code" + case "$code" in 2??|3??|4??) return 0 ;; *) return 1 ;; esac +} + +probe_ssh_banner() { + local h="$1" port="${2:-22}" + local ban + ban=$(timeout "$PORT_TIMEOUT" bash -c "exec 3<>/dev/tcp/${h}/${port}; dd bs=256 count=1 <&3 2>/dev/null" 2>/dev/null \ + | tr -d '\r' | head -1 || true) + printf 'banner=%s' "${ban:-}" + [ -n "$ban" ] +} + +# Map Server header → package name guess for OSV +guess_package() { + local server="$1" + local s + s=$(printf '%s' "$server" | tr 'A-Z' 'a-z') + case "$s" in + *nginx*) echo "nginx" ;; + *apache*|*httpd*) echo "apache" ;; + *caddy*) echo "caddy" ;; + *openbsd\ httpd*) echo "openbsd-httpd" ;; + *) echo "" ;; + esac +} + +extract_version() { + local server="$1" + # nginx/1.22.1 → 1.22.1 + printf '%s' "$server" | sed -n 's/.*\/\([0-9][0-9.]*\).*/\1/p' | head -1 +} + +# OSV / CVE check for software version taken from Server headers. +# +# Important: headers only expose upstream versions (nginx/1.26.3), never the +# Debian package revision (1.26.3-3+deb13u7). Querying OSV ecosystem=Debian +# with the bare version yields massive false positives (ancient DEBIAN-CVE-* +# with introduced:0 and no fixed event, plus every package revision that +# merely *starts with* 1.26.3). +# +# Policy: +# - bare upstream version → upstream SEMVER signal only (WARN by default) +# - full Debian package version (contains '-') → Debian OSV, filtered +# - SURFACE_CVE=0 disables; SURFACE_CVE_LEVEL=error|warn (default warn for bare) +check_cves() { + local pkg="$1" ver="$2" host="$3" + [ "$SURFACE_CVE" = "1" ] || return 0 + [ -n "$pkg" ] && [ -n "$ver" ] || return 0 + local level out rc + # bare header versions are approximate → WARN unless overridden + if [[ "$ver" == *-* ]] || [[ "$ver" == *+* ]]; then + level="${SURFACE_CVE_LEVEL:-error}" + else + level="${SURFACE_CVE_LEVEL:-warn}" + fi + out=$( + SURFACE_CVE_PKG="$pkg" SURFACE_CVE_VER="$ver" SURFACE_CVE_HOST="$host" python3 - <<'PY' 2>/dev/null || true +import json, os, re, urllib.request + +pkg = os.environ.get("SURFACE_CVE_PKG", "") +ver = os.environ.get("SURFACE_CVE_VER", "") +host = os.environ.get("SURFACE_CVE_HOST", "") + +def ver_tuple(s: str): + nums = [int(x) for x in re.findall(r"\d+", s or "")[:5]] + return tuple(nums) if nums else () + +def vt_cmp(a, b): + n = max(len(a), len(b)) + a = a + (0,) * (n - len(a)) + b = b + (0,) * (n - len(b)) + return (a > b) - (a < b) + +def in_semver_events(ver, events): + """True if ver is in [introduced, fixed) for SEMVER-like event list.""" + vt = ver_tuple(ver) + if not vt: + return False + # process sequential introduced/fixed pairs + intro = None + for e in events or []: + if "introduced" in e: + intro = e.get("introduced") + elif "fixed" in e or "last_affected" in e: + fixed = e.get("fixed") + last = e.get("last_affected") + lo = ver_tuple("0" if intro in (None, "0") else str(intro)) + if vt_cmp(vt, lo) < 0: + intro = None + continue + if fixed is not None: + if vt_cmp(vt, ver_tuple(str(fixed))) < 0: + return True + elif last is not None: + if vt_cmp(vt, ver_tuple(str(last))) <= 0: + return True + intro = None + # open-ended introduced without fixed → ignore (never clears; Debian noise) + return False + +def osv_post(body): + req = urllib.request.Request( + "https://api.osv.dev/v1/query", + data=json.dumps(body).encode(), + headers={"Content-Type": "application/json"}, + method="POST", + ) + try: + with urllib.request.urlopen(req, timeout=20) as r: + return json.load(r) + except Exception: + return {} + +def osv_get(vid): + try: + with urllib.request.urlopen(f"https://api.osv.dev/v1/vulns/{vid}", timeout=15) as r: + return json.load(r) + except Exception: + return {} + +bare = not (("-" in ver) or ("+" in ver)) +hits = [] + +if bare: + # Upstream signal via CVE records that carry extracted_events for this package + # (Server header has no Debian revision — do NOT use ecosystem=Debian). + seed = { + "nginx": [ + "CVE-2025-23419", + "CVE-2025-53859", + "CVE-2024-7347", + "CVE-2024-34161", + "CVE-2024-32760", + "CVE-2024-31079", + "CVE-2024-24989", + "CVE-2024-24990", + ], + "apache": [ + "CVE-2024-38474", + "CVE-2024-38476", + "CVE-2024-38477", + "CVE-2023-31122", + "CVE-2023-43622", + ], + "caddy": [ + "CVE-2022-29718", + "CVE-2023-50463", + ], + }.get(pkg, []) + for vid in seed: + doc = osv_get(vid) + if not doc: + continue + ok_hit = False + for a in doc.get("affected") or []: + # prefer extracted_events (human SEMVER) on GIT/nginx ranges + for rg in a.get("ranges") or []: + db = rg.get("database_specific") or {} + extracted = db.get("extracted_events") or [] + if extracted and in_semver_events(ver, extracted): + # package name filter when present + cpes = db.get("cpe") or [] + if isinstance(cpes, str): + cpes = [cpes] + blob = json.dumps(a).lower() + json.dumps(cpes).lower() + if pkg == "nginx" and "nginx" not in blob and "f5" not in blob: + continue + ok_hit = True + break + if rg.get("type") == "SEMVER" and in_semver_events(ver, rg.get("events") or []): + ok_hit = True + break + if ok_hit: + break + if ok_hit: + hits.append(vid) +else: + # Full package version — Debian ecosystem is meaningful + data = osv_post({"package": {"name": pkg, "ecosystem": "Debian"}, "version": ver}) + for v in data.get("vulns") or []: + vid = v.get("id") or "?" + actionable = False + for a in v.get("affected") or []: + for rg in a.get("ranges") or []: + events = rg.get("events") or [] + has_end = any(("fixed" in e) or ("last_affected" in e) for e in events) + if not has_end: + continue # open-ended Debian noise + if rg.get("type") in ("ECOSYSTEM", "SEMVER"): + # version string is full deb version; trust OSV query match + # but only if a fixed event exists (actionable) + actionable = True + break + if actionable: + break + # exact version listed + versions = a.get("versions") or [] + if ver in versions: + actionable = True + break + if actionable: + hits.append(vid) + +# de-dup preserve order +seen = set() +uniq = [] +for h in hits: + if h not in seen: + seen.add(h) + uniq.append(h) + +if uniq: + print(f"HIT {len(uniq)} " + ",".join(uniq[:12])) +else: + print("CLEAN") +PY + ) + rc=0 + case "$out" in + HIT\ *) + local n ids + n=$(printf '%s' "$out" | awk '{print $2}') + ids=$(printf '%s' "$out" | cut -d' ' -f3-) + if [ "$level" = "error" ]; then + err "cve" "$host $pkg $ver — $n actionable vuln(s)" "$ids" + rc=1 + else + warn "cve" "$host $pkg $ver — $n actionable vuln(s) (header version · not Debian pkg)" "$ids" + rc=0 + fi + ;; + CLEAN) + if [[ "$ver" == *-* ]] || [[ "$ver" == *+* ]]; then + info "cve" "$host $pkg $ver — OSV clean (Debian package version)" + else + info "cve" "$host $pkg $ver — no actionable upstream CVE for bare Server-header version" + fi + ;; + *) + info "cve" "$host $pkg $ver — CVE probe skipped/unavailable" + ;; + esac + return "$rc" +} + +cert_expiry_check() { + local h="$1" end="$2" + [ -n "$end" ] || return 0 + local end_epoch now left days + end_epoch=$(date -d "$end" +%s 2>/dev/null || date -j -f "%b %e %T %Y %Z" "$end" +%s 2>/dev/null || echo 0) + now=$(date +%s) + [ "$end_epoch" -gt 0 ] || return 0 + left=$((end_epoch - now)) + days=$((left / 86400)) + if [ "$left" -le 0 ]; then + err "tls" "$h certificate EXPIRED" "notAfter=$end" + return 1 + fi + if [ "$days" -le 14 ]; then + warn "tls" "$h certificate expires in ${days}d" "notAfter=$end" + else + info "tls" "$h cert ok · ${days}d left · notAfter=$end" + fi + return 0 +} + +scan_host() { + local h="$1" + local ports proto label expect + local p open_ports=() any_proto_ok=0 ping_ok=0 dns=0 + local probe_detail server_hdr pkg ver + + ports=${HOST_PORTS[$h]:-443} + proto=${HOST_PROTO[$h]:-https} + label=${HOST_LABEL[$h]:-$h} + expect=${HOST_EXPECT[$h]:-0} + + set_group "$label" + + if dns_ok "$h"; then + dns=1 + ok "dns" "$h resolves" + else + if [ "$expect" = "1" ]; then + err "dns" "$h does not resolve (catalogued)" + else + info "dns" "$h no resolve (optional guess) — skip" + fi + return 0 + fi + + if ping_host "$h"; then + ping_ok=1 + info "ping" "$h ICMP ok" + else + info "ping" "$h ICMP no reply (not decisive)" + fi + + # port scan (unique ports) + local _seen_ports=" " + for p in ${ports//,/ }; do + [ -n "$p" ] || continue + case "$_seen_ports" in *" $p "*) continue ;; esac + _seen_ports="$_seen_ports$p " + if tcp_open "$h" "$p"; then + open_ports+=("$p") + info "port" "$h:$p open (TCP)" + else + info "port" "$h:$p closed/filtered (TCP)" + fi + done + + # Protocol verification on open ports (and always try 443/80 for https/http expect) + local try_ports=("${open_ports[@]}") + if [ ${#try_ports[@]} -eq 0 ]; then + # still try expected protocol ports even if scan said closed (scan false negatives) + case "$proto" in + https) try_ports=(443) ;; + http) try_ports=(80) ;; + ssh) try_ports=(22) ;; + *) try_ports=(443 80) ;; + esac + fi + + server_hdr="" + ver="" + for p in "${try_ports[@]}"; do + case "$p" in + 443|8443) + if probe_detail=$(probe_https "$h" "$p"); then + any_proto_ok=1 + ok "https" "$h:$p up · $probe_detail" + server_hdr=$(printf '%s' "$probe_detail" | sed -n 's/.*server=\([^ ]*\).*/\1/p') + # cert + local cend + cend=$(printf '%s' "$probe_detail" | sed -n 's/.*cert_end=\([^ ]*\).*/\1/p') + cert_expiry_check "$h" "$cend" || true + local cfg + cfg=$(printf '%s' "$probe_detail" | sed -n 's/.*cfg=\([^ ]*\).*/\1/p') + if [ -n "$cfg" ]; then + info "version" "$h config/api hint: $cfg" + ver="$cfg" + fi + else + info "https" "$h:$p no HTTP response · ${probe_detail:-}" + fi + ;; + 80|8080) + if probe_detail=$(probe_http "$h" "$p"); then + any_proto_ok=1 + ok "http" "$h:$p up · $probe_detail" + else + info "http" "$h:$p no HTTP response" + fi + ;; + 22) + if probe_detail=$(probe_ssh_banner "$h" "$p"); then + any_proto_ok=1 + ok "ssh" "$h:$p banner · $probe_detail" + ver=$(printf '%s' "$probe_detail" | sed 's/banner=//') + else + info "ssh" "$h:$p no SSH banner" + fi + ;; + *) + if tcp_open "$h" "$p"; then + info "tcp" "$h:$p open · protocol unknown" + fi + ;; + esac + done + + # Version summary + if [ -n "$server_hdr" ]; then + info "server-header" "$h · $server_hdr" + pkg=$(guess_package "$server_hdr") + ver_soft=$(extract_version "$server_hdr") + if [ -n "$pkg" ] && [ -n "$ver_soft" ]; then + info "version" "$h software $pkg $ver_soft (from Server header)" + check_cves "$pkg" "$ver_soft" "$h" || true + elif [ -n "$server_hdr" ]; then + warn "version" "$h Server header present but version unknown · $server_hdr" + fi + elif [ -n "$ver" ]; then + info "version" "$h · $ver" + else + if [ "$any_proto_ok" = "1" ]; then + warn "version" "$h reachable but software version unknown" + fi + fi + + # Expected service must answer protocol (not just ping/port) + # Note: ${array[*]:-x} is NOT valid default syntax (bash treats : as slice). + local ports_txt="${open_ports[*]}" + ports_txt=${ports_txt:-none} + if [ "$expect" = "1" ]; then + if [ "$any_proto_ok" = "1" ]; then + ok "reachability" "$h catalogued service OK (protocol confirmed)" + else + err "reachability" "$h catalogued but not reachable via ${proto}/protocol" \ + "dns=$dns ping=$ping_ok open_ports=${ports_txt} (ICMP/TCP alone not enough; protocol failed)" + fi + else + if [ "$any_proto_ok" = "1" ]; then + info "reachability" "$h optional host responds" + else + info "reachability" "$h optional · no protocol response (ok)" + fi + fi +} + +# --- main --- +# Scope (remote only): +# ./taler-monitoring.sh surface → full ecosystem catalog +# ./taler-monitoring.sh -d hacktivism.ch surface → only that domain +if [ "${SURFACE_SCOPE:-}" = "ecosystem" ]; then + load_catalog + expand_ecosystem_scope +elif [ "${SURFACE_SCOPE:-}" = "domain" ] || [ "${TALER_DOMAIN_FROM_CLI:-0}" = "1" ]; then + # domain mode: do not load whole ecosystem first + CATALOG="${SURFACE_CATALOG:-$ROOT/surface-catalog.conf}" + expand_domain_scope "${TALER_DOMAIN:-hacktivism.ch}" +elif [ "${SURFACE_SCOPE:-auto}" = "auto" ]; then + # no -d → ecosystem; with -d → domain (TALER_DOMAIN_FROM_CLI) + if [ "${TALER_DOMAIN_FROM_CLI:-0}" = "1" ]; then + CATALOG="${SURFACE_CATALOG:-$ROOT/surface-catalog.conf}" + expand_domain_scope "$TALER_DOMAIN" + else + load_catalog + expand_ecosystem_scope + fi +else + load_catalog + expand_ecosystem_scope +fi + +info "inventory" "${#ORDERED_HOSTS[@]} hosts to probe (remote only · no SSH · no local podman on targets)" +info "flags" "SURFACE_CVE=${SURFACE_CVE:-1} PORT_TIMEOUT=${PORT_TIMEOUT:-2} HTTP_TIMEOUT=${HTTP_TIMEOUT:-12}" + +for h in "${ORDERED_HOSTS[@]}"; do + [ -n "$h" ] || continue + scan_host "$h" || true +done + +# mon_disk on the *runner* only (optional) — not remote servers' disks +if [ "${SURFACE_CHECK_RUNNER_DISK:-0}" = "1" ]; then + set_group disk + if declare -F mon_disk_check_host >/dev/null 2>&1; then + mon_disk_check_host "runner" || true + else + warn "disk" "mon_disk_check_host not available" + fi +fi + +# summary returns non-zero when FAIL_N>0; do not trip set -e mid-script +summary || true +if [ "${FAIL_N:-0}" -eq 0 ]; then + exit 0 +fi +exit 1 diff --git a/scripts/taler-monitoring/check_versions.sh b/scripts/taler-monitoring/check_versions.sh index 3580732..8fc7c66 100755 --- a/scripts/taler-monitoring/check_versions.sh +++ b/scripts/taler-monitoring/check_versions.sh @@ -166,16 +166,16 @@ fi # --------------------------------------------------------------------------- # 2) INSIDE — containers can reach deb.taler.net (pasta / install path) -# koopa: KOOPA_SSH · stage-lfp: INSIDE_SSH (stagepaysan, low-priv) +# host-podman: INSIDE_PODMAN=1 on GOA host · ssh: KOOPA_SSH / INSIDE_SSH from laptop # --------------------------------------------------------------------------- -set_group inside -section "versions · inside · containers → deb.taler.net" - +_VERS_LOCAL=0 _VERS_SSH_HOST="" _VERS_BANK="taler-hacktivism-bank" _VERS_EX="taler-hacktivism-exchange-ansible" _VERS_MER="taler-hacktivism" -if [ "${INSIDE_PROFILE:-}" = "stage-lfp" ] \ +if [ "${INSIDE_PODMAN:-0}" = "1" ] || [ "${INSIDE_MODE:-}" = "local-podman" ]; then + _VERS_LOCAL=1 +elif [ "${INSIDE_PROFILE:-}" = "stage-lfp" ] \ || [ "${EXPECT_CURRENCY:-}" = "TESTPAYSAN" ] \ || [ -n "${INSIDE_SSH:-}" ]; then _VERS_SSH_HOST="${INSIDE_SSH:-francpaysan-stage-user}" @@ -183,24 +183,44 @@ if [ "${INSIDE_PROFILE:-}" = "stage-lfp" ] \ _VERS_EX="${INSIDE_EXCHANGE_CTR:-stage-lfp-exchange-ansible}" _VERS_MER="${INSIDE_MERCHANT_CTR:-stage-lfp-merchant}" elif [ "${LOCAL_STACK:-0}" = "1" ] && [ "${SKIP_SSH:-0}" != "1" ]; then + # laptop → koopa via SSH (not host-podman) _VERS_SSH_HOST="${KOOPA_SSH}" fi -if [ -z "$_VERS_SSH_HOST" ]; then - info "ssh" "no container SSH for this domain — outside-only package index" +if [ "$_VERS_LOCAL" = "1" ]; then + VERS_ACCESS=host-podman + set_group host +else + VERS_ACCESS=ssh + set_group ssh +fi + +section "versions · inside · containers → deb.taler.net (access=${VERS_ACCESS})" +info "flags" "VERS_ACCESS=${VERS_ACCESS} INSIDE_PODMAN=${INSIDE_PODMAN:-0} INSIDE_MODE=${INSIDE_MODE:-} LOCAL_STACK=${LOCAL_STACK:-0} SKIP_SSH=${SKIP_SSH:-0} KOOPA_SSH=${KOOPA_SSH:-} INSIDE_SSH=${INSIDE_SSH:-}" + +if [ "$_VERS_LOCAL" != "1" ] && [ -z "$_VERS_SSH_HOST" ]; then + info "access" "no host-podman and no SSH target — outside-only package index" info "outside-only" "deb.taler.net public checks completed above" summary exit 0 fi -if [ "$_VERS_SSH_HOST" = "${KOOPA_SSH:-}" ] || [ "${INSIDE_PROFILE:-}" = "koopa" ]; then - if ! koopa_ssh_ok; then - err "ssh" "cannot reach ${KOOPA_SSH} — set SKIP_SSH=1 to skip container checks" +if [ "$_VERS_LOCAL" = "1" ]; then + if ! command -v podman >/dev/null 2>&1; then + err "host" "INSIDE_PODMAN/host-podman but podman missing" info "outside-only" "deb.taler.net public checks completed above" summary exit 1 fi - ok "ssh ${KOOPA_SSH}" + ok "host→container" "podman exec on this host (VERS_ACCESS=host-podman · no SSH)" +elif [ "$_VERS_SSH_HOST" = "${KOOPA_SSH:-}" ] || [ "${INSIDE_PROFILE:-}" = "koopa" ]; then + if ! koopa_ssh_ok; then + err "ssh" "cannot reach ${KOOPA_SSH} — from laptop set KOOPA_SSH=; on host set INSIDE_PODMAN=1" + info "outside-only" "deb.taler.net public checks completed above" + summary + exit 1 + fi + ok "ssh ${KOOPA_SSH}" "remote then podman (VERS_ACCESS=ssh)" else if ! mon_ssh_ok "$_VERS_SSH_HOST"; then err "ssh" "cannot reach ${_VERS_SSH_HOST} (stagepaysan) — skip container package compare" @@ -208,7 +228,7 @@ else summary exit 1 fi - ok "ssh ${_VERS_SSH_HOST}" "stagepaysan container package compare" + ok "ssh ${_VERS_SSH_HOST}" "stagepaysan (VERS_ACCESS=ssh)" fi INRELEASE_URL="${APT_BASE}/dists/${SUITE}/InRelease" @@ -257,7 +277,9 @@ done REMOTE } >"$tmp/remote-versions.sh" -if [ "$_VERS_SSH_HOST" = "${KOOPA_SSH:-}" ] || [ "${INSIDE_PROFILE:-}" = "koopa" ]; then +if [ "$_VERS_LOCAL" = "1" ]; then + REMOTE_OUT=$(bash "$tmp/remote-versions.sh" 2>/dev/null || true) +elif [ "$_VERS_SSH_HOST" = "${KOOPA_SSH:-}" ] || [ "${INSIDE_PROFILE:-}" = "koopa" ]; then REMOTE_OUT=$(koopa_ssh_bash 60 <"$tmp/remote-versions.sh" || true) else REMOTE_OUT=$(mon_ssh_bash "$_VERS_SSH_HOST" 60 <"$tmp/remote-versions.sh" || true) @@ -501,4 +523,41 @@ done info "suite" "deb.taler.net ${SUITE} (testing_index=${HAVE_TESTING})" info "tally" "match=${ok_n} ahead=${ahead_n} behind=${behind_n}" +# Disk free on the machine(s) we just probed (host-podman or SSH target) +set_group disk +section "versions · disk free space" +if [ "${_VERS_LOCAL:-0}" = "1" ]; then + mon_disk_check_host "host" || true + while read -r _vc; do + [ -n "$_vc" ] || continue + mon_disk_check_podman "$_vc" || true + done < <(podman ps --format '{{.Names}}' 2>/dev/null | grep -iE 'hacktivism|taler-|stage-lfp|lfp-' || true) +elif [ -n "${_VERS_SSH_HOST:-}" ]; then + _vdisk=$(mon_ssh_bash "$_VERS_SSH_HOST" 20 <<'DISK' || true +set +e +echo "###HOST###" +df -Pk / /var /home /tmp 2>/dev/null || df -Pk +echo "###CTRS###" +for c in $(podman ps --format '{{.Names}}' 2>/dev/null); do + echo "###CTR $c###" + podman exec "$c" df -Pk / /var /tmp 2>/dev/null || true +done +DISK +) + mon_disk_check_remote_text "ssh:${_VERS_SSH_HOST}" \ + "$(printf '%s\n' "$_vdisk" | sed -n '/^###HOST###$/,/^###CTRS###$/p' | sed '1d;$d')" || true + _ctr=""; _buf="" + while IFS= read -r _line || [ -n "$_line" ]; do + case "$_line" in + '###CTR '*) + [ -n "$_ctr" ] && mon_disk_check_remote_text "ctr:${_ctr}" "$_buf" || true + _ctr=${_line####CTR }; _ctr=${_ctr%###}; _buf="" + ;; + '###CTRS###'|'###HOST###') ;; + *) [ -n "$_ctr" ] && _buf="${_buf}${_line}"$'\n' ;; + esac + done <<<"$_vdisk" + [ -n "$_ctr" ] && mon_disk_check_remote_text "ctr:${_ctr}" "$_buf" || true +fi + summary diff --git a/scripts/taler-monitoring/lib.sh b/scripts/taler-monitoring/lib.sh index 841d6e8..8455c6b 100755 --- a/scripts/taler-monitoring/lib.sh +++ b/scripts/taler-monitoring/lib.sh @@ -23,6 +23,9 @@ MERCHANT_INSTANCE=$(printf '%s' "${MERCHANT_INSTANCE}" | tr '[:upper:]' '[:lower : "${TIMEOUT:=12}" : "${E2E_TIMEOUT:=55}" # whole e2e budget; skip rest when exceeded (e2e raises as needed) : "${E2E_PAY_SECS:=22}" # dedicated seconds for pay handle-uri (avoid Alarm clock) +# Whole-run wall clock for taler-monitoring.sh (seconds). 0 = unlimited. +# Host-agent default phases (urls/inside/versions) should finish under this. +: "${RUN_TIMEOUT:=600}" # Local GOA: public paivana paywall (https://paivana.hacktivism.ch · template GOA:4200) : "${PAIVANA_PUBLIC:=https://paivana.hacktivism.ch}" : "${E2E_PAIVANA:=1}" # 0 = skip paivana section in e2e @@ -855,6 +858,113 @@ summary() { [ "$FAIL_N" -eq 0 ] } +# --------------------------------------------------------------------------- +# Disk space on hosts / containers under test +# WARN when tight, ERROR when full / critically low +# DISK_WARN_USED_PCT=85 # free below 15% → WARN +# DISK_ERR_USED_PCT=95 # free below 5% → ERROR +# DISK_ERR_FREE_BYTES=0 # free == 0 always ERROR +# --------------------------------------------------------------------------- +: "${DISK_WARN_USED_PCT:=85}" +: "${DISK_ERR_USED_PCT:=95}" + +# Parse one POSIX `df -P` body line (no header): fs size used avail capacity mount +# size/used/avail in 1K-blocks when using df -Pk +_mon_disk_eval_line() { + local where="$1" fs="$2" size="$3" used="$4" avail="$5" cap="$6" mnt="$7" + local pct label detail + pct=${cap%%%} + # non-numeric capacity → skip + case "$pct" in + ''|*[!0-9]*) return 0 ;; + esac + label="disk ${where} ${mnt}" + detail="fs=$fs size_1k=$size used_1k=$used avail_1k=$avail use=${pct}%" + if [ "$avail" = "0" ] || [ "$pct" -ge 100 ]; then + err "disk" "${where} ${mnt} FULL / overflow" "$detail" + return 1 + fi + if [ "$pct" -ge "${DISK_ERR_USED_PCT}" ]; then + err "disk" "${where} ${mnt} critically low free space (≥${DISK_ERR_USED_PCT}% used)" "$detail" + return 1 + fi + if [ "$pct" -ge "${DISK_WARN_USED_PCT}" ]; then + warn "disk" "${where} ${mnt} free space tight (≥${DISK_WARN_USED_PCT}% used)" "$detail" + return 0 + fi + ok "$label" "$detail" + return 0 +} + +# Report all filesystems from `df -P` / `df -Pk` output (with header). +# $1 = where label (host, container:name, ssh:host) +# $2 = full df text +mon_disk_report_df() { + local where="$1" text="$2" line fs size used avail cap mnt rest ec=0 + [ -n "$text" ] || { + warn "disk" "${where}: no df output" + return 0 + } + while IFS= read -r line || [ -n "$line" ]; do + [ -n "$line" ] || continue + case "$line" in + Filesystem*|Filesystem*) continue ;; + esac + # df -P: Filesystem 1024-blocks Used Available Capacity Mounted on + # shellcheck disable=SC2086 + set -- $line + [ "$#" -ge 6 ] || continue + fs=$1 + size=$2 + used=$3 + avail=$4 + cap=$5 + shift 5 + mnt=$* + # skip special/pseudo if tiny and not root-ish + case "$fs" in + tmpfs|devtmpfs|overlay|shm|nsfs|proc|sysfs|cgroup*) + # still check root-like mounts that fill (overlay / in containers) + case "$mnt" in + /|/var|/var/*|/home|/home/*|/data|/mnt/*) ;; + *) continue ;; + esac + ;; + esac + if ! _mon_disk_eval_line "$where" "$fs" "$size" "$used" "$avail" "$cap" "$mnt"; then + ec=1 + fi + done <<<"$text" + return "$ec" +} + +# Local host: important mounts +mon_disk_check_host() { + local where text + where="${1:-host}" + text="" + text=$(df -Pk / /var /home /tmp 2>/dev/null | awk 'NR==1 || !seen[$1,$6]++' || true) + # fallback whole table + if [ -z "$text" ]; then + text=$(df -Pk 2>/dev/null || true) + fi + mon_disk_report_df "$where" "$text" +} + +# Run df inside podman container +mon_disk_check_podman() { + local cname="$1" text + local bin="${2:-podman}" + text=$("$bin" exec "$cname" df -Pk / /var /tmp 2>/dev/null || "$bin" exec "$cname" df -Pk 2>/dev/null || true) + mon_disk_report_df "ctr:${cname}" "$text" +} + +# df text already collected remotely +mon_disk_check_remote_text() { + local where="$1" text="$2" + mon_disk_report_df "$where" "$text" +} + http_code() { local url="$1"; shift curl -skS --max-redirs 0 -m "${TIMEOUT}" -o /dev/null -w '%{http_code}' "$@" "$url" 2>/dev/null || echo 000 diff --git a/scripts/taler-monitoring/surface-catalog.conf b/scripts/taler-monitoring/surface-catalog.conf new file mode 100644 index 0000000..16d5755 --- /dev/null +++ b/scripts/taler-monitoring/surface-catalog.conf @@ -0,0 +1,52 @@ +# surface-catalog.conf — known public Taler / GNUnet / Taler Systems hosts +# Format (whitespace-separated; # comments): +# host ports expect_proto label +# expect_proto: https | http | ssh | tcp | any +# ports: comma-separated (empty = defaults for proto) +# +# This list is expanded at runtime (DNS discovery, cert SANs, -d domain profile). +# Phase `surface` is NOT in default phases — run explicitly: +# ./taler-monitoring.sh surface +# ./taler-monitoring.sh -d hacktivism.ch surface + +# --- taler.net ecosystem --- +www.taler.net 443 https taler-www +docs.taler.net 443 https taler-docs +git.taler.net 443,22 https taler-git +bugs.taler.net 443 https taler-bugs +lists.taler.net 443 https taler-lists +tutorials.taler.net 443 https taler-tutorials +bank.demo.taler.net 443 https demo-bank +exchange.demo.taler.net 443 https demo-exchange +backend.demo.taler.net 443 https demo-merchant +shop.demo.taler.net 443 https demo-shop +bank.test.taler.net 443 https test-bank +exchange.test.taler.net 443 https test-exchange +backend.test.taler.net 443 https test-merchant + +# --- taler-ops / TOPS --- +www.taler-ops.ch 443 https tops-www +taler-ops.ch 443 https tops-apex +bank.taler-ops.ch 443 https tops-bank +exchange.taler-ops.ch 443 https tops-exchange +my.taler-ops.ch 443 https tops-merchant +stage.taler-ops.ch 443 https tops-stage-www +stage.my.taler-ops.ch 443 https tops-stage-merchant +exchange.stage.taler-ops.ch 443 https tops-stage-exchange +nexus.stage.taler-ops.ch 443 https tops-stage-nexus +shops.taler-ops.ch 443 https tops-shops +mattermost.taler.net 443 https mattermost + +# --- gnunet.org --- +www.gnunet.org 443,80 https gnunet-www +gnunet.org 443,80 https gnunet-apex +git.gnunet.org 443,22 https gnunet-git +bugs.gnunet.org 443 https gnunet-bugs + +# --- taler-systems.com --- +www.taler-systems.com 443 https tsa-www +taler-systems.com 443 https tsa-apex + +# --- common infra hostnames (remote only; may 404/redirect) --- +deb.taler.net 443 https deb-taler +ftp.gnu.org 443 https gnu-ftp diff --git a/scripts/taler-monitoring/taler-monitoring.sh b/scripts/taler-monitoring/taler-monitoring.sh index 8b20e72..557c7a0 100755 --- a/scripts/taler-monitoring/taler-monitoring.sh +++ b/scripts/taler-monitoring/taler-monitoring.sh @@ -12,6 +12,12 @@ set -euo pipefail ROOT=$(cd "$(dirname "$0")" && pwd) +# Line-buffered stdout/stderr so redirected logs (tee/host-agent) flush each line. +if [ "${_TALER_MON_STDBUF:-0}" != "1" ] && command -v stdbuf >/dev/null 2>&1; then + export _TALER_MON_STDBUF=1 + exec stdbuf -oL -eL bash "$0" "$@" +fi + usage() { cat <<'EOF' taler-monitoring — bank / exchange / merchant checks @@ -29,8 +35,14 @@ Phases: e2e withdraw + pay (small amounts; remote aborts on login/KYC) ladder withdraw/pay amount ladder (GOA ceiling or stage TESTPAYSAN max_wire) auth401 merchant Basic-auth / case matrix (HTTP 401 paths; may create throwaway instance) + aptdeploy koopa podman apt-src smoke: taler-merchant in + koopa-taler-deploy-test-apt-src-trixie{,-testing} + surface REMOTE-ONLY public inventory (NOT in default/all/full): + ecosystem hosts (taler.net, gnunet.org, taler-systems.com, mattermost, …) + or -d DOMAIN → that domain’s surface; port/protocol/TLS/CVE (OSV) all urls + inside + versions + sanity + e2e (SSH phases only on koopa) full all + server + ladder + auth401 (maximum; long-running, needs secrets) + NOTE: surface is never included in all/full — pass it explicitly Options: -d, --domain DOMAIN load profile from domains.conf (bank/exchange/merchant) @@ -61,6 +73,10 @@ Env (same meaning): TALER_DOMAINS_CONF SKIP_SSH=1 NO_COLOR=1 / CLICOLOR=0 disable green/yellow/red tags (default: coloured) SKIP_SSH=1 NO_COLOR=1 + RUN_TIMEOUT=600 whole-run wall clock seconds (default 600; 0=unlimited) + long phases (ladder/full/e2e) need a higher value or 0 + DISK_WARN_USED_PCT=85 disk free: WARN when used ≥ this % + DISK_ERR_USED_PCT=95 disk free: ERROR when used ≥ this % (or avail=0 / 100%) PERF_WARN_MS PERF_FAIL_MS (urls latency; default 8000 / 20000) QR_CHECK=0 skip QR form + qrencode/zbarimg (urls phase) QR_ECC=M qrencode ECC level (default M) @@ -73,11 +89,23 @@ Env (same meaning): AUTH401_* see check_auth401.sh / secrets.env.example AUTH401_CONTINUE=1 / CONTINUE_ON_ERROR=1 auth401: run all groups, collect every ERROR (no mid-run halt) + APT_DEPLOY_SKIP=1 skip aptdeploy phase + APT_DEPLOY_CONTAINERS="name:suite …" default trixie + trixie-testing deploy-test pods + SURFACE_CVE=0 disable CVE queries in surface phase + SURFACE_CVE_LEVEL=warn|error bare Server-header versions default warn + (Debian package versions default error) + SURFACE_CATALOG=path override surface-catalog.conf Full load (GOA / hacktivism): ./taler-monitoring.sh -d hacktivism.ch full # or explicit: ./taler-monitoring.sh -d hacktivism.ch urls inside versions sanity server e2e ladder auth401 + # hacktivism host-agent also runs aptdeploy (apt-src merchant containers on koopa) + +Remote surface / ecosystem (explicit only — never default): + ./taler-monitoring.sh surface + ./taler-monitoring.sh -d hacktivism.ch surface + ./taler-monitoring.sh -d lefrancpaysan.ch surface SPA pin after selfbuild: EXPECT_WEBUI_VERSION=1.6.11 EXPECT_WEBUI_OVERLAY=selfbuild-v1.6.11 \\ @@ -111,6 +139,7 @@ while [ $# -gt 0 ]; do [ $# -ge 2 ] || { echo "missing arg for $1" >&2; exit 2; } apply_taler_domain "$2" DOMAIN_SET=1 + TALER_DOMAIN_FROM_CLI=1 shift 2 ;; --bank) @@ -130,12 +159,13 @@ while [ $# -gt 0 ]; do CURRENCY_OVERRIDE="$2"; shift 2 ;; --no-probe) NO_PROBE=1; shift ;; - urls|inside|versions|sanity|server|e2e|ladder|goa-ladder|auth401|all|full) PHASES+=("$1"); shift ;; + urls|inside|versions|sanity|server|e2e|ladder|goa-ladder|auth401|aptdeploy|apt-deploy|apt_src|surface|ecosystem|all|full) PHASES+=("$1"); shift ;; *) # bare domain shorthand: ./taler-monitoring.sh taler.net if [[ "$1" == *.* && "$1" != *://* && "$1" != -* ]]; then apply_taler_domain "$1" DOMAIN_SET=1 + TALER_DOMAIN_FROM_CLI=1 shift else echo "Unknown: $1" >&2; usage >&2; exit 2 @@ -170,6 +200,8 @@ fi # Export so check_*.sh (re-source lib) see the same targets via env export TALER_DOMAIN BANK_PUBLIC EXCHANGE_PUBLIC MERCHANT_PUBLIC export EXPECT_CURRENCY SKIP_SSH LOCAL_STACK TALER_DOMAIN_PROBE CHECK_LANDING +export TALER_DOMAIN_FROM_CLI="${TALER_DOMAIN_FROM_CLI:-0}" +export DOMAIN_SET="${DOMAIN_SET:-0}" export WITHDRAW_AMT PAY_AMT CREDIT_AMT MERCHANT_INSTANCE export E2E_FAKE_INCOMING E2E_REMOTE E2E_VARIABLE E2E_ATM_MAX export E2E_WITHDRAW_VALUES E2E_PAY_VALUES E2E_USE_TEMPLATES E2E_TEMPLATE_MAP @@ -288,33 +320,158 @@ if [ "${PROGRESS_TOTAL:-0}" = "0" ] || [ -z "${PROGRESS_TOTAL:-}" ]; then e2e) _pt=$((_pt + 240)) ;; ladder|goa-ladder) _pt=$((_pt + 120)) ;; auth401) _pt=$((_pt + 70)) ;; + aptdeploy) _pt=$((_pt + 20)) ;; + surface|ecosystem) _pt=$((_pt + 80)) ;; esac done set_progress_total "$_pt" unset _pt fi +# Whole-run wall clock (default 10 min). Override: RUN_TIMEOUT=0 (unlimited). +: "${RUN_TIMEOUT:=600}" +export RUN_TIMEOUT +MON_T0=$(date +%s) +export MON_T0 +RUN_TIMED_OUT=0 +RUN_TIMEOUT_AT_PHASE="" +RUN_SKIPPED_PHASES=() + +mon_seconds_left() { + if [ "${RUN_TIMEOUT:-0}" -eq 0 ]; then + printf '%s' "999999" + return 0 + fi + local now left + now=$(date +%s) + left=$((RUN_TIMEOUT - (now - MON_T0))) + [ "$left" -lt 0 ] && left=0 + printf '%s' "$left" +} + +# Run one phase script under remaining RUN_TIMEOUT budget. +# Exit 124 (timeout utility) → mark RUN_TIMED_OUT. +run_phase() { + local phase="$1" script="$2" left rc + left=$(mon_seconds_left) + if [ "${RUN_TIMEOUT:-0}" -gt 0 ] && [ "$left" -le 0 ]; then + RUN_TIMED_OUT=1 + RUN_TIMEOUT_AT_PHASE="${RUN_TIMEOUT_AT_PHASE:-$phase}" + return 1 + fi + if [ "${RUN_TIMEOUT:-0}" -eq 0 ]; then + "$script" + return $? + fi + # Keep at least 2s so timeout(1) can start the child. + [ "$left" -lt 2 ] && left=2 + set +e + with_timeout "$left" "$script" + rc=$? + set -e + if [ "$rc" -eq 124 ]; then + RUN_TIMED_OUT=1 + RUN_TIMEOUT_AT_PHASE="$phase" + return 1 + fi + return "$rc" +} + +# How inside/versions will reach containers (host-podman vs ssh) +_INSIDE_ACCESS_HINT=ssh +if [ "${INSIDE_PODMAN:-0}" = "1" ] || [ "${INSIDE_MODE:-}" = "local-podman" ]; then + _INSIDE_ACCESS_HINT=host-podman +elif [ "${SKIP_SSH:-0}" = "1" ]; then + _INSIDE_ACCESS_HINT=skipped +elif [ "${LOCAL_STACK:-0}" = "1" ] && command -v podman >/dev/null 2>&1 \ + && podman ps --format '{{.Names}}' 2>/dev/null | grep -qE 'taler-hacktivism'; then + _INSIDE_ACCESS_HINT=host-podman +fi + printf 'target domain=%s\n' "${TALER_DOMAIN}" printf ' bank %s\n' "$BANK_PUBLIC" printf ' exchange %s\n' "$EXCHANGE_PUBLIC" printf ' merchant %s\n' "$MERCHANT_PUBLIC" -printf ' currency expect=%s skip_ssh=%s\n' "${EXPECT_CURRENCY:-any}" "$SKIP_SSH" +printf ' currency expect=%s\n' "${EXPECT_CURRENCY:-any}" printf ' phases %s\n' "${PHASES[*]}" +printf ' flags LOCAL_STACK=%s SKIP_SSH=%s INSIDE_PODMAN=%s INSIDE_MODE=%s\n' \ + "${LOCAL_STACK:-}" "${SKIP_SSH:-0}" "${INSIDE_PODMAN:-0}" "${INSIDE_MODE:-}" +printf ' flags KOOPA_SSH=%s INSIDE_SSH=%s INSIDE_PROFILE=%s\n' \ + "${KOOPA_SSH:-}" "${INSIDE_SSH:-}" "${INSIDE_PROFILE:-}" +printf ' access inside/versions → %s' "$_INSIDE_ACCESS_HINT" +case "$_INSIDE_ACCESS_HINT" in + host-podman) printf ' (podman exec on this host; IDs inside.host-*)\n' ;; + ssh) printf ' (SSH then podman; IDs inside.ssh-*)\n' ;; + skipped) printf ' (SKIP_SSH=1)\n' ;; + *) printf '\n' ;; +esac +if [ "${RUN_TIMEOUT:-0}" -eq 0 ]; then + printf ' run_timeout unlimited (RUN_TIMEOUT=0)\n' +else + printf ' run_timeout %ss wall clock (RUN_TIMEOUT=; 0=unlimited)\n' "$RUN_TIMEOUT" +fi printf ' progress total≈%s (set PROGRESS_TOTAL= to override; PROGRESS_OFF=1 to hide)\n' "${PROGRESS_TOTAL:-0}" +unset _INSIDE_ACCESS_HINT chmod +x "$ROOT"/check_*.sh 2>/dev/null || true ec=0 for p in "${PHASES[@]}"; do + if [ "$RUN_TIMED_OUT" = "1" ]; then + RUN_SKIPPED_PHASES+=("$p") + continue + fi + left=$(mon_seconds_left) + if [ "${RUN_TIMEOUT:-0}" -gt 0 ] && [ "$left" -le 0 ]; then + RUN_TIMED_OUT=1 + RUN_TIMEOUT_AT_PHASE="${RUN_TIMEOUT_AT_PHASE:-$p}" + RUN_SKIPPED_PHASES+=("$p") + continue + fi case "$p" in - urls) "$ROOT/check_urls.sh" || ec=1 ;; - inside) "$ROOT/check_inside.sh" || ec=1 ;; - versions) "$ROOT/check_versions.sh" || ec=1 ;; - sanity) "$ROOT/check_sanity.sh" || ec=1 ;; - server) "$ROOT/check_server.sh" || ec=1 ;; - e2e) "$ROOT/check_e2e.sh" || ec=1 ;; - ladder|goa-ladder) "$ROOT/check_goa_ladder.sh" || ec=1 ;; - auth401) "$ROOT/check_auth401.sh" || ec=1 ;; + urls) run_phase urls "$ROOT/check_urls.sh" || ec=1 ;; + inside) run_phase inside "$ROOT/check_inside.sh" || ec=1 ;; + versions) run_phase versions "$ROOT/check_versions.sh" || ec=1 ;; + sanity) run_phase sanity "$ROOT/check_sanity.sh" || ec=1 ;; + server) run_phase server "$ROOT/check_server.sh" || ec=1 ;; + e2e) run_phase e2e "$ROOT/check_e2e.sh" || ec=1 ;; + ladder|goa-ladder) run_phase ladder "$ROOT/check_goa_ladder.sh" || ec=1 ;; + auth401) run_phase auth401 "$ROOT/check_auth401.sh" || ec=1 ;; + aptdeploy|apt-deploy|apt_src) run_phase aptdeploy "$ROOT/check_apt_deploy.sh" || ec=1 ;; + surface|ecosystem) run_phase surface "$ROOT/check_surface.sh" || ec=1 ;; esac done + +# Extraordinary run-budget failure: always report at end; HTML links top → here. +if [ "$RUN_TIMED_OUT" = "1" ]; then + ec=1 + elapsed=$(( $(date +%s) - MON_T0 )) + skipped="${RUN_SKIPPED_PHASES[*]:-}" + # Stable id for jump links (console HTML + err-top banner) + printf '\n' + printf '╔══════════════════════════════════════════════════════════╗\n' + printf '║ RUN TIMEOUT · extraordinary (see jump target below) ║\n' + printf '╚══════════════════════════════════════════════════════════╝\n' + printf '┌ ERROR ┐ #run.timeout-01 RUN_TIMEOUT exceeded · budget %ss · elapsed %ss\n' \ + "$RUN_TIMEOUT" "$elapsed" + printf ' detail: wall-clock limit hit' + if [ -n "${RUN_TIMEOUT_AT_PHASE:-}" ]; then + printf ' during/after phase "%s"' "$RUN_TIMEOUT_AT_PHASE" + fi + if [ -n "$skipped" ]; then + printf ' · skipped: %s' "$skipped" + fi + printf '\n' + printf ' id=run.timeout-01\n' + printf ' hint: raise RUN_TIMEOUT= (seconds) or set RUN_TIMEOUT=0 for unlimited\n' + printf -- '┌ RUN TIMEOUT · extraordinary ┐\n' + printf -- ' • run.timeout-01 [run] RUN_TIMEOUT=%ss exceeded (elapsed %ss)%s%s\n' \ + "$RUN_TIMEOUT" "$elapsed" \ + "${RUN_TIMEOUT_AT_PHASE:+ · phase $RUN_TIMEOUT_AT_PHASE}" \ + "${skipped:+ · skipped $skipped}" + printf -- '--- ERRORS (failed checks) ---\n' + printf -- ' • run.timeout-01 [run] RUN_TIMEOUT=%ss exceeded (elapsed %ss)\n' \ + "$RUN_TIMEOUT" "$elapsed" +fi + exit "$ec"