monitoring: surface inventory and apt-deploy check phases
This commit is contained in:
Hernâni Marques
parent
f4b556c3c0
commit
0fef40fd63
12 changed files with 1657 additions and 46 deletions
6
scripts/taler-monitoring/.gitignore
vendored
Normal file
6
scripts/taler-monitoring/.gitignore
vendored
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
android-test/apks/
|
||||
android-test/out*/
|
||||
android-test/artifacts/
|
||||
__pycache__/
|
||||
secrets.env
|
||||
*.pyc
|
||||
70
scripts/taler-monitoring/DEPENDENCIES.md
Normal file
70
scripts/taler-monitoring/DEPENDENCIES.md
Normal file
|
|
@ -0,0 +1,70 @@
|
|||
# taler-monitoring — host package dependencies
|
||||
|
||||
## koopa (openSUSE Tumbleweed, user `hernani`)
|
||||
|
||||
### Required (host-agent / GOA primary: `urls inside versions`)
|
||||
|
||||
| Binary | Package (zypper) | Role |
|
||||
|--------|------------------|------|
|
||||
| `bash` | (base) | runner |
|
||||
| `curl` | `curl` | HTTPS/API probes |
|
||||
| `python3` | `python313-base` (provides `/usr/bin/python3`) | JSON, HTML site-gen, metrics |
|
||||
| `podman` | `podman` | **inside**: `podman exec` into taler-hacktivism* |
|
||||
| `git` | `git` | pin commit SHA in monitoring HTML footer |
|
||||
| `rsync` | `rsync` | optional: deploy/sync suite |
|
||||
| `timeout` / `mktemp` / `getent` | coreutils / glibc | helpers |
|
||||
|
||||
**Status on koopa (2026-07-18):** all of the above present; rootless podman owns GOA containers.
|
||||
|
||||
### Recommended (full `urls` QR group)
|
||||
|
||||
| Binary | Package | Role |
|
||||
|--------|---------|------|
|
||||
| `qrencode` | **`qrencode`** | encode taler:// / payto:// → PNG |
|
||||
| `zbarimg` | **`zbar`** | decode PNG; exact payload match |
|
||||
|
||||
Without these, QR checks **WARN** and skip encode/decode (`apt install` messages in code are Debian-oriented; on Tumbleweed use zypper).
|
||||
|
||||
```bash
|
||||
# as root on koopa
|
||||
zypper refresh
|
||||
zypper in qrencode zbar
|
||||
# verify as hernani:
|
||||
command -v qrencode zbarimg
|
||||
```
|
||||
|
||||
### Optional (not needed for host-agent default)
|
||||
|
||||
| Binary / tool | When |
|
||||
|---------------|------|
|
||||
| `jq` | nice-to-have; suite uses python3 for JSON |
|
||||
| `dig` / `bind-utils` | debug DNS; not required for default phases |
|
||||
| `ssh` | only for access mode **ssh** (laptop → `KOOPA_SSH` / stage `INSIDE_SSH`); not needed for **host-podman** |
|
||||
| `node` / `pnpm` + wallet-cli | **e2e** / ladder only (+ secrets) |
|
||||
| `secrets.env` / admin passwords | e2e only — never system packages |
|
||||
|
||||
### systemd user (no extra packages)
|
||||
|
||||
- `systemd` user session + **`loginctl enable-linger hernani`** (already **Linger=yes** on koopa)
|
||||
- Units: `scripts/taler-monitoring/host-agent/*.service|path|timer`
|
||||
|
||||
### Caddy HTML publish (separate from monitoring run)
|
||||
|
||||
- Host Caddy already installed (root service).
|
||||
- Writing `/var/www/monitoring-sites`: either root `rsync` or directory owned/writable by `hernani` (you set this up).
|
||||
|
||||
---
|
||||
|
||||
## firecuda-external (macOS, outside-only timer)
|
||||
|
||||
| Need | Notes |
|
||||
|------|--------|
|
||||
| `bash`, `curl`, `python3`, `rsync`, `git` (optional) | usually Homebrew / Xcode CLT |
|
||||
| **No** `podman` | phases are `urls versions` only (`SKIP_SSH=1`) |
|
||||
| `qrencode` / `zbarimg` | optional for QR; `brew install qrencode zbar` if desired |
|
||||
|
||||
---
|
||||
|
||||
## Franc Paysan hosts (francpaysan / stagepaysan)
|
||||
|
||||
Document package lists under **`francpaysan-admin-log`** when those host-agents are installed (same idea: curl, python3, podman if inside, qrencode/zbar for full urls).
|
||||
|
|
@ -1,5 +1,10 @@
|
|||
# taler-monitoring
|
||||
|
||||
Host packages (koopa / runners): **[DEPENDENCIES.md](./DEPENDENCIES.md)**.
|
||||
Host-agents (GOA + FP; shared report pipeline): **[host-agent/README.md](./host-agent/README.md)**.
|
||||
Public HTML site-gen (same timeout/log/HTML defaults): **[site-gen/README.md](./site-gen/README.md)**.
|
||||
FP install notes: `francpaysan-admin-log/docs/taler-monitoring-host-agents.md`.
|
||||
|
||||
Report for the **GOA** stack with boxed severity badges and **grouped test IDs**
|
||||
`area.group-NN` (e.g. `www.exchange-01`, `e2e.pay-03`):
|
||||
|
||||
|
|
|
|||
|
|
@ -18,6 +18,8 @@ Every check line has a **global** run number and a **grouped** id:
|
|||
| **www** | `check_urls.sh` | `exchange` `perf` `stats` `bank` `merchant` `paivana` `landing` `qr` |
|
||||
| **inside** | `check_inside.sh` | `ssh` `bank` `exchange` `merchant` `caddy` `load` |
|
||||
| **versions** | `check_versions.sh` | `outside` `inside` `compare` |
|
||||
| **aptdeploy** | `check_apt_deploy.sh` | `trixie` `trixie-testing` (koopa podman apt-src merchant smoke) |
|
||||
| **surface** | `check_surface.sh` | **remote-only** ecosystem / `-d` domain inventory (not in default/all/full) |
|
||||
| **sanity** | `check_sanity.sh` | `bank` `exchange` `merchant` |
|
||||
| **server** | `check_server.sh` | (flat `server-NN` or host groups) |
|
||||
| **e2e** | `check_e2e.sh` | `prereq` `load` `bank` `wallet` `atm` `settle` `pay` `shop` `paivana` `dig` `report` |
|
||||
|
|
@ -73,12 +75,14 @@ Numbering follows **executed** checks (early skip may shift later NN inside the
|
|||
|
||||
| Group | Checks |
|
||||
|-------|--------|
|
||||
| **inside.ssh-** | ssh reachability / remote collect |
|
||||
| **inside.host-** | **host-podman**: reachability via `podman exec` on this machine (GOA host / `INSIDE_PODMAN=1`) — not SSH |
|
||||
| **inside.ssh-** | **ssh**: from laptop/remote — SSH to `KOOPA_SSH` / `INSIDE_SSH`, then podman there |
|
||||
| **inside.bank-** | container, libeufin, postgres, local `/config`, nginx, DNS pins |
|
||||
| **inside.exchange-** | container, httpd, wirewatch, aggregator, transfer, local keys, DNS |
|
||||
| **inside.merchant-** | container, httpd, wirewatch, depositcheck, DNS |
|
||||
| **inside.caddy-** | host reverse-proxy process |
|
||||
| **inside.load-** | host loadavg + RAM; per-container RSS/CPU |
|
||||
| **inside.disk-** / **aptdeploy.disk-** / **versions.disk-** / **server.disk-** | free space via `df`: **WARN** if use ≥ `DISK_WARN_USED_PCT` (85), **ERROR** if full or use ≥ `DISK_ERR_USED_PCT` (95); host + containers |
|
||||
|
||||
Remote lines `E|comp|LEVEL|key|detail` each become one numbered result under that component group.
|
||||
|
||||
|
|
@ -112,8 +116,28 @@ When filing an issue, quote the full id + label, e.g.
|
|||
| Area.group | Meaning |
|
||||
|------------|---------|
|
||||
| **versions.outside-** | deb.taler.net suite index |
|
||||
| **versions.inside-** | packages in containers |
|
||||
| **versions.host-** | host-podman: packages via local `podman exec` (`INSIDE_PODMAN=1`) |
|
||||
| **versions.ssh-** | ssh: packages via `KOOPA_SSH` / `INSIDE_SSH` from laptop/remote |
|
||||
| **versions.compare-** | installed vs suite |
|
||||
| **aptdeploy.trixie-fresh-** | fresh rebuild container `…-apt-src-trixie` |
|
||||
| **aptdeploy.trixie-testing-fresh-** | fresh rebuild `…-trixie-testing` |
|
||||
| **aptdeploy.trixie-upgrade-** | upgrade-track `…-trixie-upgrade` (install once, then mytops-style upgrade) |
|
||||
| **aptdeploy.trixie-testing-upgrade-** | upgrade-track `…-trixie-testing-upgrade` |
|
||||
|
||||
Standalone: `host-agent/run-aptdeploy.sh` (on koopa local; else `ssh koopa-external`).
|
||||
|
||||
### surface (remote-only public inventory)
|
||||
|
||||
```bash
|
||||
./taler-monitoring.sh surface # taler.net / gnunet / taler-systems / mattermost / …
|
||||
./taler-monitoring.sh -d hacktivism.ch surface # all things under that domain
|
||||
```
|
||||
|
||||
- Catalog: `surface-catalog.conf` (extend as hosts are found)
|
||||
- DNS, optional ICMP, TCP ports, then **protocol** checks (HTTPS/HTTP/SSH) before ERROR
|
||||
- TLS cert expiry; Server-header / `/config` version hints; else **WARN: version unknown**
|
||||
- OSV CVE query when package+version known → **ERROR** on hits
|
||||
- Never SSH / never podman on the *targets*
|
||||
| **sanity.bank-** / **.exchange-** / **.merchant-** | public + optional server-side per component |
|
||||
| **ladder.plan-** / **.load-** / **.withdraw-** / **.pay-** / **.report-** | amount ladder (GOA ceiling or stage TESTPAYSAN `max_wire`) |
|
||||
| **server-** | SSH host ports / processes (flat unless grouped later) |
|
||||
|
|
|
|||
279
scripts/taler-monitoring/check_apt_deploy.sh
Executable file
279
scripts/taler-monitoring/check_apt_deploy.sh
Executable file
|
|
@ -0,0 +1,279 @@
|
|||
#!/usr/bin/env bash
|
||||
# check_apt_deploy.sh — podman apt-src merchant deploy tests on koopa.
|
||||
#
|
||||
# Containers (default):
|
||||
# Fresh (rebuild when repo versions change):
|
||||
# koopa-taler-deploy-test-apt-src-trixie
|
||||
# koopa-taler-deploy-test-apt-src-trixie-testing
|
||||
# Upgrade track (initial install, then mytops-style apt upgrade):
|
||||
# koopa-taler-deploy-test-apt-src-trixie-upgrade
|
||||
# koopa-taler-deploy-test-apt-src-trixie-testing-upgrade
|
||||
#
|
||||
# Env:
|
||||
# APT_DEPLOY_CONTAINERS "name:suite:mode …" mode=fresh|upgrade (optional, default fresh)
|
||||
# APT_DEPLOY_SKIP=1
|
||||
# APT_DEPLOY_PODMAN=podman
|
||||
#
|
||||
set -euo pipefail
|
||||
ROOT=$(cd "$(dirname "$0")" && pwd)
|
||||
# shellcheck source=lib.sh
|
||||
source "$ROOT/lib.sh"
|
||||
|
||||
if [ "${APT_DEPLOY_SKIP:-0}" = "1" ]; then
|
||||
echo "[INFO] skip phase aptdeploy (APT_DEPLOY_SKIP=1)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
set_area aptdeploy
|
||||
section "aptdeploy · podman apt-src merchant (fresh + upgrade tracks)"
|
||||
|
||||
PODMAN_BIN="${APT_DEPLOY_PODMAN:-podman}"
|
||||
DEFAULT_LIST="\
|
||||
koopa-taler-deploy-test-apt-src-trixie:trixie:fresh \
|
||||
koopa-taler-deploy-test-apt-src-trixie-testing:trixie-testing:fresh \
|
||||
koopa-taler-deploy-test-apt-src-trixie-upgrade:trixie:upgrade \
|
||||
koopa-taler-deploy-test-apt-src-trixie-testing-upgrade:trixie-testing:upgrade"
|
||||
LIST="${APT_DEPLOY_CONTAINERS:-$DEFAULT_LIST}"
|
||||
|
||||
if ! command -v "$PODMAN_BIN" >/dev/null 2>&1; then
|
||||
err "podman" "podman binary missing ($PODMAN_BIN)"
|
||||
summary
|
||||
exit 1
|
||||
fi
|
||||
ok "podman" "$("$PODMAN_BIN" --version 2>/dev/null | head -1)"
|
||||
info "host" "$(hostname 2>/dev/null || echo unknown) · whoami=$(whoami)"
|
||||
|
||||
fail_any=0
|
||||
|
||||
print_pkg_table() {
|
||||
local name=$1
|
||||
local rows
|
||||
rows=$("$PODMAN_BIN" exec "$name" bash -lc '
|
||||
echo "| package | version |"
|
||||
echo "|---------|---------|"
|
||||
for p in taler-merchant libtalermerchant libtalerexchange libdonau libgnunet \
|
||||
taler-merchant-webui taler-merchant-typst taler-terms-generator; do
|
||||
v=$(dpkg-query -W -f="\${Version}" "$p" 2>/dev/null || echo missing)
|
||||
echo "| $p | $v |"
|
||||
done
|
||||
' 2>/dev/null || echo "| (query failed) | |")
|
||||
info "pkg-table" "$name"
|
||||
# emit as plain lines so HTML log keeps the table
|
||||
while IFS= read -r line; do
|
||||
[ -n "$line" ] || continue
|
||||
info "pkg" "$line"
|
||||
done <<<"$rows"
|
||||
}
|
||||
|
||||
check_ldd_deep() {
|
||||
local name=$1
|
||||
local out
|
||||
out=$("$PODMAN_BIN" exec "$name" bash -lc '
|
||||
set +e
|
||||
bin=$(command -v taler-merchant-httpd)
|
||||
echo "=== ldd taler-merchant-httpd ==="
|
||||
ldd "$bin" 2>&1 | grep -E "libtalerutil|libdonau|libgnunet|not found" || true
|
||||
echo "=== ldd libdonau (if present) ==="
|
||||
for f in /usr/lib/*/libdonau.so* /usr/lib/*/libdonauutil.so*; do
|
||||
[ -e "$f" ] || continue
|
||||
echo "-- $f --"
|
||||
ldd "$f" 2>&1 | grep -E "libtalerutil|not found" || true
|
||||
done
|
||||
echo "=== libtalerutil files ==="
|
||||
ls -la /usr/lib/*/libtalerutil* 2>/dev/null || echo "(none)"
|
||||
' 2>/dev/null || echo "ldd probe failed")
|
||||
while IFS= read -r line; do
|
||||
[ -n "$line" ] || continue
|
||||
info "ldd" "$line"
|
||||
done <<<"$out"
|
||||
if echo "$out" | grep -q 'not found'; then
|
||||
err "ldd" "shared library missing in $name" "see ldd lines above"
|
||||
return 1
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
check_systemd() {
|
||||
local name=$1
|
||||
local out active_target active_httpd enabled
|
||||
out=$("$PODMAN_BIN" exec "$name" bash -lc '
|
||||
set +e
|
||||
echo "system: $(systemctl is-system-running 2>&1)"
|
||||
echo "target_enabled: $(systemctl is-enabled taler-merchant.target 2>&1)"
|
||||
echo "target_active: $(systemctl is-active taler-merchant.target 2>&1)"
|
||||
echo "httpd_active: $(systemctl is-active taler-merchant-httpd.service 2>&1)"
|
||||
echo "httpd_enabled: $(systemctl is-enabled taler-merchant-httpd.service 2>&1)"
|
||||
systemctl start taler-merchant.target 2>&1 | tail -5
|
||||
sleep 2
|
||||
echo "after_start_target: $(systemctl is-active taler-merchant.target 2>&1)"
|
||||
echo "after_start_httpd: $(systemctl is-active taler-merchant-httpd.service 2>&1)"
|
||||
systemctl --failed --no-legend 2>/dev/null | grep -i taler || echo "failed_taler: none"
|
||||
' 2>/dev/null || echo "systemd probe failed")
|
||||
while IFS= read -r line; do
|
||||
[ -n "$line" ] || continue
|
||||
info "systemd" "$line"
|
||||
done <<<"$out"
|
||||
|
||||
active_httpd=$(echo "$out" | sed -n 's/^after_start_httpd: //p' | tail -1)
|
||||
if [ "$active_httpd" = "active" ]; then
|
||||
ok "systemd httpd" "taler-merchant-httpd active after start taler-merchant.target"
|
||||
return 0
|
||||
fi
|
||||
# soft if target not shipped as enabled in container — still ERROR for deploy smoke
|
||||
err "systemd httpd" "taler-merchant-httpd not active" "after_start_httpd=${active_httpd:-?} · see systemd lines"
|
||||
return 1
|
||||
}
|
||||
|
||||
check_merchant_basics() {
|
||||
local name=$1
|
||||
local out code
|
||||
# unix socket or curl localhost if httpd listens
|
||||
out=$("$PODMAN_BIN" exec "$name" bash -lc '
|
||||
set +e
|
||||
echo "=== binaries ==="
|
||||
command -v taler-merchant-httpd taler-merchant-dbinit taler-config 2>/dev/null
|
||||
echo "=== --version ==="
|
||||
taler-merchant-httpd --version 2>&1
|
||||
echo "=== config probe ==="
|
||||
# try common paths without TLS
|
||||
for u in \
|
||||
http://127.0.0.1:9966/config \
|
||||
http://127.0.0.1:8081/config \
|
||||
http://127.0.0.1/config
|
||||
do
|
||||
code=$(curl -sS -m 2 -o /tmp/mcfg -w "%{http_code}" "$u" 2>/dev/null || echo 000)
|
||||
echo "curl $u -> $code"
|
||||
[ "$code" = "200" ] && head -c 120 /tmp/mcfg && echo
|
||||
done
|
||||
sock=$(ls /run/taler-merchant/httpd/*.sock 2>/dev/null | head -1)
|
||||
if [ -n "$sock" ]; then
|
||||
echo "socket: $sock"
|
||||
code=$(curl -sS -m 2 --unix-socket "$sock" -o /tmp/mcfg -w "%{http_code}" http://localhost/config 2>/dev/null || echo 000)
|
||||
echo "curl --unix-socket -> $code"
|
||||
[ "$code" = "200" ] && head -c 160 /tmp/mcfg && echo
|
||||
else
|
||||
echo "socket: (none under /run/taler-merchant/httpd/)"
|
||||
fi
|
||||
' 2>/dev/null || echo "basics probe failed")
|
||||
|
||||
while IFS= read -r line; do
|
||||
[ -n "$line" ] || continue
|
||||
info "basics" "$line"
|
||||
done <<<"$out"
|
||||
|
||||
if echo "$out" | grep -qi 'error while loading shared libraries'; then
|
||||
err "httpd" "shared library error on --version" "see basics/ldd"
|
||||
return 1
|
||||
fi
|
||||
if ! echo "$out" | grep -q 'taler-merchant-httpd --version\|v\.\|taler-merchant-httpd'; then
|
||||
# version line varies; success if exit was ok — check via separate exec
|
||||
:
|
||||
fi
|
||||
|
||||
set +e
|
||||
"$PODMAN_BIN" exec "$name" taler-merchant-httpd --version >/dev/null 2>&1
|
||||
local ec=$?
|
||||
set -e
|
||||
if [ "$ec" -eq 0 ]; then
|
||||
ok "httpd --version" "exit 0"
|
||||
else
|
||||
err "httpd --version" "exit $ec"
|
||||
return 1
|
||||
fi
|
||||
|
||||
# /config is nice-to-have if stack is fully configured
|
||||
if echo "$out" | grep -qE 'curl .* -> 200'; then
|
||||
ok "merchant /config" "HTTP 200 (local)"
|
||||
else
|
||||
warn "merchant /config" "no local HTTP 200 (unit may need BASE_URL/db — still report version/ldd)"
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
check_one() {
|
||||
local name="$1" expect_suite="$2" mode="${3:-fresh}"
|
||||
local st suite_line pkgs
|
||||
|
||||
# group ids: aptdeploy.trixie-01 / aptdeploy.trixie-testing-upgrade-01
|
||||
set_group "${expect_suite}${mode:+-$mode}"
|
||||
section "aptdeploy · $name (suite=$expect_suite mode=$mode)"
|
||||
|
||||
if ! "$PODMAN_BIN" container exists "$name" 2>/dev/null; then
|
||||
err "container" "$name missing" "run ensure-apt-deploy-test-containers.sh"
|
||||
fail_any=1
|
||||
return
|
||||
fi
|
||||
st=$("$PODMAN_BIN" inspect -f '{{.State.Status}}' "$name" 2>/dev/null || echo unknown)
|
||||
if [ "$st" != "running" ]; then
|
||||
err "container" "$name not running" "status=$st"
|
||||
fail_any=1
|
||||
return
|
||||
fi
|
||||
ok "container" "$name running (mode=$mode)"
|
||||
|
||||
suite_line=$("$PODMAN_BIN" exec "$name" bash -lc \
|
||||
'grep -h "^Suites:" /etc/apt/sources.list.d/*taler* 2>/dev/null | head -1' 2>/dev/null || true)
|
||||
if echo "$suite_line" | grep -qE "Suites:[[:space:]]*${expect_suite}([[:space:]]|$)"; then
|
||||
ok "apt suite" "$expect_suite"
|
||||
else
|
||||
err "apt suite" "expected $expect_suite" "got: ${suite_line:-empty}"
|
||||
fail_any=1
|
||||
fi
|
||||
|
||||
print_pkg_table "$name"
|
||||
|
||||
if ! check_merchant_basics "$name"; then
|
||||
fail_any=1
|
||||
check_ldd_deep "$name" || fail_any=1
|
||||
else
|
||||
# still show ldd summary as info when ok
|
||||
ldd_out=$("$PODMAN_BIN" exec "$name" bash -lc \
|
||||
'ldd "$(command -v taler-merchant-httpd)" 2>&1 | grep -E "libtalerutil|not found" || true' 2>/dev/null || true)
|
||||
if echo "$ldd_out" | grep -q 'not found'; then
|
||||
err "libtalerutil" "not found" "$ldd_out"
|
||||
check_ldd_deep "$name" || true
|
||||
fail_any=1
|
||||
else
|
||||
info "libtalerutil" "$(echo "$ldd_out" | tr '\n' ' ')"
|
||||
fi
|
||||
fi
|
||||
|
||||
if ! check_systemd "$name"; then
|
||||
fail_any=1
|
||||
check_ldd_deep "$name" || true
|
||||
fi
|
||||
|
||||
# Disk free inside this deploy-test container (+ host once, first container only)
|
||||
if [ "${_APT_DEPLOY_HOST_DISK_DONE:-0}" != "1" ]; then
|
||||
set_group disk
|
||||
mon_disk_check_host "host" || fail_any=1
|
||||
_APT_DEPLOY_HOST_DISK_DONE=1
|
||||
fi
|
||||
set_group disk
|
||||
if ! mon_disk_check_podman "$name" "$PODMAN_BIN"; then
|
||||
fail_any=1
|
||||
fi
|
||||
}
|
||||
|
||||
for entry in $LIST; do
|
||||
# name:suite or name:suite:mode
|
||||
cname=${entry%%:*}
|
||||
rest=${entry#*:}
|
||||
case "$rest" in
|
||||
*:*)
|
||||
suite=${rest%%:*}
|
||||
mode=${rest#*:}
|
||||
;;
|
||||
*)
|
||||
suite=$rest
|
||||
mode=fresh
|
||||
;;
|
||||
esac
|
||||
check_one "$cname" "$suite" "$mode"
|
||||
done
|
||||
|
||||
summary
|
||||
if [ "$fail_any" -ne 0 ]; then
|
||||
exit 1
|
||||
fi
|
||||
exit 0
|
||||
|
|
@ -1,9 +1,14 @@
|
|||
#!/usr/bin/env bash
|
||||
# Inside status for bank / exchange / merchant containers.
|
||||
#
|
||||
# Access modes (shown in the log as flags):
|
||||
# host-podman — this host runs podman exec into containers (no SSH).
|
||||
# INSIDE_PODMAN=1 or INSIDE_MODE=local-podman, or auto on GOA host.
|
||||
# ssh — from a laptop/remote: SSH to KOOPA_SSH / INSIDE_SSH, then podman there.
|
||||
#
|
||||
# Profiles:
|
||||
# koopa (default LOCAL_STACK=1) — SSH KOOPA_SSH, containers taler-hacktivism*
|
||||
# stage-lfp (TESTPAYSAN) — SSH INSIDE_SSH (francpaysan-stage-user / stagepaysan),
|
||||
# low-priv podman only; ports 9030–9032; stats remain outside-in (urls).
|
||||
# koopa (default LOCAL_STACK=1) — taler-hacktivism* containers
|
||||
# stage-lfp (TESTPAYSAN) — SSH INSIDE_SSH (stagepaysan), low-priv podman
|
||||
set -euo pipefail
|
||||
ROOT=$(cd "$(dirname "$0")" && pwd)
|
||||
# shellcheck source=lib.sh
|
||||
|
|
@ -12,7 +17,6 @@ source "$ROOT/lib.sh"
|
|||
source "$ROOT/metrics.sh"
|
||||
|
||||
set_area inside
|
||||
set_group ssh
|
||||
|
||||
PROFILE="${INSIDE_PROFILE:-}"
|
||||
if [ -z "$PROFILE" ]; then
|
||||
|
|
@ -25,7 +29,27 @@ if [ -z "$PROFILE" ]; then
|
|||
fi
|
||||
fi
|
||||
|
||||
section "inside · collect (${PROFILE})"
|
||||
# Resolve host-podman vs ssh before first check ID
|
||||
_use_local_podman=0
|
||||
if [ "$PROFILE" != "stage-lfp" ]; then
|
||||
if [ "${INSIDE_PODMAN:-0}" = "1" ] || [ "${INSIDE_MODE:-}" = "local-podman" ]; then
|
||||
_use_local_podman=1
|
||||
elif command -v podman >/dev/null 2>&1 \
|
||||
&& podman ps --format '{{.Names}}' 2>/dev/null | grep -qE 'taler-hacktivism'; then
|
||||
_use_local_podman=1
|
||||
fi
|
||||
fi
|
||||
if [ "$_use_local_podman" = "1" ]; then
|
||||
INSIDE_ACCESS=host-podman
|
||||
set_group host
|
||||
else
|
||||
INSIDE_ACCESS=ssh
|
||||
set_group ssh
|
||||
fi
|
||||
export INSIDE_ACCESS
|
||||
|
||||
section "inside · collect (${PROFILE} · access=${INSIDE_ACCESS})"
|
||||
info "flags" "INSIDE_ACCESS=${INSIDE_ACCESS} INSIDE_PODMAN=${INSIDE_PODMAN:-0} INSIDE_MODE=${INSIDE_MODE:-} LOCAL_STACK=${LOCAL_STACK:-0} SKIP_SSH=${SKIP_SSH:-0} KOOPA_SSH=${KOOPA_SSH:-} INSIDE_SSH=${INSIDE_SSH:-}"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# stage-lfp: low-priv stagepaysan on francpaysan-host
|
||||
|
|
@ -244,31 +268,70 @@ EOF
|
|||
info "stage host" "load probe empty"
|
||||
fi
|
||||
|
||||
set_group disk
|
||||
section "inside · disk free space (stage host + containers)"
|
||||
_disk_raw=$(mon_ssh_bash "$SSH_HOST" "${STAGE_SSH_T:-24}" <<'DISK' || true
|
||||
set +e
|
||||
echo "###HOST###"
|
||||
df -Pk / /var /home /tmp /mnt/data 2>/dev/null || df -Pk
|
||||
echo "###CTRS###"
|
||||
for c in $(podman ps --format '{{.Names}}' 2>/dev/null); do
|
||||
echo "###CTR $c###"
|
||||
podman exec "$c" df -Pk / /var /tmp 2>/dev/null || podman exec "$c" df -Pk 2>/dev/null
|
||||
done
|
||||
DISK
|
||||
)
|
||||
_host_df=$(printf '%s\n' "$_disk_raw" | sed -n '/^###HOST###$/,/^###CTRS###$/p' | sed '1d;$d')
|
||||
mon_disk_check_remote_text "ssh:${SSH_HOST}" "$_host_df" || true
|
||||
_ctr=""; _buf=""
|
||||
while IFS= read -r _line || [ -n "$_line" ]; do
|
||||
case "$_line" in
|
||||
'###CTR '*)
|
||||
if [ -n "$_ctr" ]; then mon_disk_check_remote_text "ctr:${_ctr}" "$_buf" || true; fi
|
||||
_ctr=${_line####CTR }; _ctr=${_ctr%###}; _buf=""
|
||||
;;
|
||||
'###CTRS###'|'###HOST###') ;;
|
||||
*) [ -n "$_ctr" ] && _buf="${_buf}${_line}"$'\n' ;;
|
||||
esac
|
||||
done <<<"$_disk_raw"
|
||||
[ -n "$_ctr" ] && mon_disk_check_remote_text "ctr:${_ctr}" "$_buf" || true
|
||||
|
||||
summary
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# koopa (default) — existing remote collect
|
||||
# koopa (default) — host-podman exec (on GOA host) or SSH to KOOPA_SSH (laptop)
|
||||
# ---------------------------------------------------------------------------
|
||||
section "inside · collect from koopa"
|
||||
section "inside · collect from koopa (access=${INSIDE_ACCESS})"
|
||||
|
||||
if [ "${SKIP_SSH}" = "1" ] && [ "${LOCAL_STACK:-0}" != "1" ]; then
|
||||
warn "ssh" "SKIP_SSH=1 and not local — skipped"
|
||||
if [ "$_use_local_podman" = "1" ]; then
|
||||
set_group host
|
||||
if ! command -v podman >/dev/null 2>&1; then
|
||||
err "host" "INSIDE_PODMAN/host-podman but podman missing"
|
||||
summary
|
||||
exit 1
|
||||
fi
|
||||
ok "host→container" "podman exec on this host (INSIDE_ACCESS=host-podman · no SSH)"
|
||||
elif [ "${SKIP_SSH}" = "1" ] && [ "${LOCAL_STACK:-0}" != "1" ]; then
|
||||
set_group ssh
|
||||
warn "ssh" "SKIP_SSH=1 and not host-podman — skipped"
|
||||
summary
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if ! koopa_ssh_ok; then
|
||||
err "ssh" "cannot reach ${KOOPA_SSH} in ${SSH_CONNECT_TIMEOUT}s — set SKIP_SSH=1 to skip inside"
|
||||
elif ! koopa_ssh_ok; then
|
||||
set_group ssh
|
||||
err "ssh" "cannot reach ${KOOPA_SSH} — from laptop use KOOPA_SSH=; on host set INSIDE_PODMAN=1"
|
||||
summary
|
||||
exit 1
|
||||
else
|
||||
set_group ssh
|
||||
ok "ssh ${KOOPA_SSH}" "remote host then podman (INSIDE_ACCESS=ssh · laptop/remote)"
|
||||
fi
|
||||
ok "ssh ${KOOPA_SSH}"
|
||||
|
||||
# One short remote script (≤ SSH_CMD_TIMEOUT). Every slow step is local curl -m 3 or quick pgrep.
|
||||
RAW=$(
|
||||
koopa_ssh_bash "${SSH_CMD_TIMEOUT}" <<'REMOTE' || true
|
||||
# Collect script (file → bash local or ssh bash -s)
|
||||
_INSIDE_SCRIPT=$(mktemp)
|
||||
trap 'rm -f "${_INSIDE_SCRIPT:-}"' RETURN
|
||||
cat >"$_INSIDE_SCRIPT" <<'REMOTE'
|
||||
set +e
|
||||
emit() { printf 'E|%s|%s|%s|%s\n' "$1" "$2" "$3" "$(printf '%s' "${4:-}" | tr '\n\r' ' ' | head -c 200)"; }
|
||||
# quick curl
|
||||
|
|
@ -376,23 +439,37 @@ else
|
|||
fi
|
||||
echo DONE
|
||||
REMOTE
|
||||
)
|
||||
|
||||
if [ "$_use_local_podman" = "1" ]; then
|
||||
RAW=$(bash "$_INSIDE_SCRIPT" || true)
|
||||
else
|
||||
RAW=$(koopa_ssh_bash "${SSH_CMD_TIMEOUT}" <"$_INSIDE_SCRIPT" || true)
|
||||
fi
|
||||
rm -f "$_INSIDE_SCRIPT"
|
||||
|
||||
if [ -z "$RAW" ] || ! echo "$RAW" | grep -q '^E|'; then
|
||||
err "ssh" "remote timed out or empty (cap ${SSH_CMD_TIMEOUT}s)"
|
||||
if [ "$_use_local_podman" = "1" ]; then
|
||||
err "host" "collect empty (INSIDE_ACCESS=host-podman · podman exec failed)"
|
||||
else
|
||||
err "ssh" "collect empty (INSIDE_ACCESS=ssh · cap ${SSH_CMD_TIMEOUT}s · host=${KOOPA_SSH})"
|
||||
fi
|
||||
summary
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Fallback group for non-component rows: host (podman) vs ssh (remote)
|
||||
_access_grp=ssh
|
||||
[ "$_use_local_podman" = "1" ] && _access_grp=host
|
||||
|
||||
_last_inside_grp=""
|
||||
while IFS= read -r line; do
|
||||
case "$line" in
|
||||
E\|*)
|
||||
IFS='|' read -r _ comp level key detail <<<"$line"
|
||||
# Group IDs by component so issues map cleanly: inside.bank-02, inside.exchange-04
|
||||
# Group IDs by component: inside.bank-02, inside.exchange-04; access = host|ssh
|
||||
case "$comp" in
|
||||
bank|exchange|merchant|caddy) _g="$comp" ;;
|
||||
*) _g="ssh" ;;
|
||||
*) _g="$_access_grp" ;;
|
||||
esac
|
||||
if [ "$_g" != "$_last_inside_grp" ]; then
|
||||
set_group "$_g"
|
||||
|
|
@ -415,4 +492,55 @@ METRICS_DIR="${METRICS_DIR:-$(mktemp -d)}"
|
|||
export METRICS_DIR
|
||||
metrics_report_load "${METRICS_DIR}/load-inside.json" "inside" || true
|
||||
|
||||
# Disk free space: host + taler containers (WARN tight, ERROR full)
|
||||
set_group disk
|
||||
section "inside · disk free space"
|
||||
_disk_ec=0
|
||||
if [ "$_use_local_podman" = "1" ]; then
|
||||
mon_disk_check_host "host" || _disk_ec=1
|
||||
while read -r _cname; do
|
||||
[ -n "$_cname" ] || continue
|
||||
mon_disk_check_podman "$_cname" || _disk_ec=1
|
||||
done < <(podman ps --format '{{.Names}}' 2>/dev/null | grep -iE 'hacktivism|taler-|stage-lfp|lfp-' || true)
|
||||
else
|
||||
# remote host via SSH: df on host + each container
|
||||
_disk_raw=$(koopa_ssh_bash "${SSH_CMD_TIMEOUT:-20}" <<'DISK' || true
|
||||
set +e
|
||||
echo "###HOST###"
|
||||
df -Pk / /var /home /tmp 2>/dev/null || df -Pk
|
||||
echo "###CTRS###"
|
||||
for c in $(podman ps --format '{{.Names}}' 2>/dev/null); do
|
||||
echo "###CTR $c###"
|
||||
podman exec "$c" df -Pk / /var /tmp 2>/dev/null || podman exec "$c" df -Pk 2>/dev/null
|
||||
done
|
||||
DISK
|
||||
)
|
||||
_host_df=$(printf '%s\n' "$_disk_raw" | sed -n '/^###HOST###$/,/^###CTRS###$/p' | sed '1d;$d')
|
||||
mon_disk_check_remote_text "ssh:${KOOPA_SSH:-remote}" "$_host_df" || _disk_ec=1
|
||||
_ctr=""
|
||||
_buf=""
|
||||
while IFS= read -r _line || [ -n "$_line" ]; do
|
||||
case "$_line" in
|
||||
'###CTR '*)
|
||||
if [ -n "$_ctr" ]; then
|
||||
mon_disk_check_remote_text "ctr:${_ctr}" "$_buf" || _disk_ec=1
|
||||
fi
|
||||
_ctr=${_line####CTR }
|
||||
_ctr=${_ctr%###}
|
||||
_buf=""
|
||||
;;
|
||||
'###CTRS###'|'###HOST###') ;;
|
||||
*)
|
||||
if [ -n "$_ctr" ]; then
|
||||
_buf="${_buf}${_line}"$'\n'
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
done <<<"$_disk_raw"
|
||||
if [ -n "$_ctr" ]; then
|
||||
mon_disk_check_remote_text "ctr:${_ctr}" "$_buf" || _disk_ec=1
|
||||
fi
|
||||
fi
|
||||
unset _disk_raw _host_df _ctr _buf _line _cname _disk_ec
|
||||
|
||||
summary
|
||||
|
|
|
|||
|
|
@ -110,6 +110,15 @@ if systemctl is-active caddy >/dev/null 2>&1 || pgrep -x caddy >/dev/null 2>&1;
|
|||
else
|
||||
report WARN "caddy" "not detected as active"
|
||||
fi
|
||||
|
||||
# disk free (host + containers) — lines for mon_disk_check_remote_text on laptop side
|
||||
echo "###DISK_HOST###"
|
||||
df -Pk / /var /home /tmp 2>/dev/null || df -Pk
|
||||
echo "###DISK_CTRS###"
|
||||
for c in $(podman ps --format '{{.Names}}' 2>/dev/null); do
|
||||
echo "###CTR $c###"
|
||||
podman exec "$c" df -Pk / /var /tmp 2>/dev/null || podman exec "$c" df -Pk 2>/dev/null
|
||||
done
|
||||
REMOTE
|
||||
)
|
||||
|
||||
|
|
@ -126,4 +135,24 @@ while IFS= read -r line; do
|
|||
esac
|
||||
done <<<"$REMOTE"
|
||||
|
||||
set_group disk
|
||||
section "server · disk free space"
|
||||
_host_df=$(printf '%s\n' "$REMOTE" | sed -n '/^###DISK_HOST###$/,/^###DISK_CTRS###$/p' | sed '1d;$d')
|
||||
mon_disk_check_remote_text "ssh:${KOOPA_SSH}" "$_host_df" || true
|
||||
_ctr=""; _buf=""
|
||||
while IFS= read -r _line || [ -n "$_line" ]; do
|
||||
case "$_line" in
|
||||
'###CTR '*)
|
||||
if [ -n "$_ctr" ]; then mon_disk_check_remote_text "ctr:${_ctr}" "$_buf" || true; fi
|
||||
_ctr=${_line####CTR }; _ctr=${_ctr%###}; _buf=""
|
||||
;;
|
||||
'###DISK_CTRS###'|'###DISK_HOST###') ;;
|
||||
*)
|
||||
# only buffer after we entered CTR section
|
||||
if [ -n "$_ctr" ]; then _buf="${_buf}${_line}"$'\n'; fi
|
||||
;;
|
||||
esac
|
||||
done <<<"$REMOTE"
|
||||
[ -n "$_ctr" ] && mon_disk_check_remote_text "ctr:${_ctr}" "$_buf" || true
|
||||
|
||||
summary
|
||||
|
|
|
|||
692
scripts/taler-monitoring/check_surface.sh
Executable file
692
scripts/taler-monitoring/check_surface.sh
Executable file
|
|
@ -0,0 +1,692 @@
|
|||
#!/usr/bin/env bash
|
||||
# check_surface.sh — REMOTE-ONLY public surface / ecosystem scan.
|
||||
#
|
||||
# NOT in default phases. Explicit:
|
||||
# ./taler-monitoring.sh surface
|
||||
# ./taler-monitoring.sh -d hacktivism.ch surface
|
||||
#
|
||||
# Without -d / with generic ecosystem: catalog of taler.net, gnunet.org,
|
||||
# taler-systems.com, taler-ops, mattermost, …
|
||||
# With -d DOMAIN: expand domains.conf stack hosts + catalog entries matching
|
||||
# that domain + common subdomain guesses; port/protocol probes only remote.
|
||||
#
|
||||
# Rules:
|
||||
# - no SSH into targets, no local podman on targets
|
||||
# - ICMP/port alone never proves "down" — confirm via expected protocol
|
||||
# - catalogued host unreachable via protocol → ERROR
|
||||
# - version unknown after probes → WARN
|
||||
# - CVE hit (OSV) for identified software+version → ERROR
|
||||
#
|
||||
set -euo pipefail
|
||||
ROOT=$(cd "$(dirname "$0")" && pwd)
|
||||
# shellcheck source=lib.sh
|
||||
source "$ROOT/lib.sh"
|
||||
|
||||
set_area surface
|
||||
section "surface · remote ecosystem / domain inventory (outside-in only)"
|
||||
|
||||
CATALOG="${SURFACE_CATALOG:-$ROOT/surface-catalog.conf}"
|
||||
# Common ports when scanning a host discovered without explicit list
|
||||
DEFAULT_SCAN_PORTS="${SURFACE_SCAN_PORTS:-22,80,443,993,8443,9418}"
|
||||
PORT_TIMEOUT="${SURFACE_PORT_TIMEOUT:-2}"
|
||||
HTTP_TIMEOUT="${SURFACE_HTTP_TIMEOUT:-12}"
|
||||
# CVE via OSV (public, no key). Disable: SURFACE_CVE=0
|
||||
: "${SURFACE_CVE:=1}"
|
||||
# Extra ports when -d domain mode
|
||||
DOMAIN_EXTRA_PORTS="${SURFACE_DOMAIN_PORTS:-22,80,443,8443}"
|
||||
|
||||
declare -A HOST_PORTS # host -> comma ports
|
||||
declare -A HOST_PROTO # host -> expect proto
|
||||
declare -A HOST_LABEL # host -> short label
|
||||
declare -A HOST_EXPECT # host -> 1 if catalogued (must respond)
|
||||
ORDERED_HOSTS=()
|
||||
|
||||
add_host() {
|
||||
local h="$1" ports="${2:-}" proto="${3:-https}" label="${4:-}" expect="${5:-1}"
|
||||
h=$(printf '%s' "$h" | tr 'A-Z' 'a-z' | sed 's#^https\?://##;s#/.*##;s/\.$//')
|
||||
[ -n "$h" ] || return 0
|
||||
if [ -z "${HOST_PORTS[$h]+x}" ]; then
|
||||
ORDERED_HOSTS+=("$h")
|
||||
HOST_PORTS[$h]="${ports:-443}"
|
||||
HOST_PROTO[$h]="${proto:-https}"
|
||||
HOST_LABEL[$h]="${label:-$h}"
|
||||
HOST_EXPECT[$h]="$expect"
|
||||
else
|
||||
# merge ports
|
||||
local p
|
||||
for p in ${ports//,/ }; do
|
||||
case ",${HOST_PORTS[$h]}," in
|
||||
*",$p,"*) ;;
|
||||
*) HOST_PORTS[$h]="${HOST_PORTS[$h]},$p" ;;
|
||||
esac
|
||||
done
|
||||
[ "${HOST_EXPECT[$h]}" = "1" ] || HOST_EXPECT[$h]="$expect"
|
||||
fi
|
||||
}
|
||||
|
||||
load_catalog() {
|
||||
local line h ports proto label
|
||||
[ -f "$CATALOG" ] || {
|
||||
warn "catalog" "missing $CATALOG — using empty base list"
|
||||
return 0
|
||||
}
|
||||
while IFS= read -r line || [ -n "$line" ]; do
|
||||
line=${line%%#*}
|
||||
line=$(echo "$line" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')
|
||||
[ -z "$line" ] && continue
|
||||
# host ports proto label
|
||||
h=$(echo "$line" | awk '{print $1}')
|
||||
ports=$(echo "$line" | awk '{print $2}')
|
||||
proto=$(echo "$line" | awk '{print $3}')
|
||||
label=$(echo "$line" | awk '{print $4}')
|
||||
[ "$ports" = "-" ] && ports=""
|
||||
add_host "$h" "$ports" "${proto:-https}" "${label:-}" 1
|
||||
done <"$CATALOG"
|
||||
info "catalog" "loaded $CATALOG · ${#ORDERED_HOSTS[@]} hosts"
|
||||
}
|
||||
|
||||
# Clear inventory (used when switching to domain-only scope)
|
||||
clear_hosts() {
|
||||
ORDERED_HOSTS=()
|
||||
unset HOST_PORTS HOST_PROTO HOST_LABEL HOST_EXPECT
|
||||
declare -gA HOST_PORTS HOST_PROTO HOST_LABEL HOST_EXPECT
|
||||
}
|
||||
|
||||
# Expand for -d domain: ONLY that domain (remote). No whole-ecosystem catalog.
|
||||
expand_domain_scope() {
|
||||
local d="$1" h
|
||||
d=${d#https://}; d=${d%/}
|
||||
d=${d#http://}
|
||||
info "scope" "domain mode −d $d (remote only · not full ecosystem catalog)"
|
||||
|
||||
clear_hosts
|
||||
|
||||
# stack endpoints from applied profile (always expected)
|
||||
for h in \
|
||||
"${BANK_PUBLIC:-}" \
|
||||
"${EXCHANGE_PUBLIC:-}" \
|
||||
"${MERCHANT_PUBLIC:-}" \
|
||||
"${PAIVANA_PUBLIC:-}"
|
||||
do
|
||||
[ -n "$h" ] || continue
|
||||
add_host "$h" "80,443,$DOMAIN_EXTRA_PORTS" https "stack" 1
|
||||
done
|
||||
|
||||
# apex + common subdomains (expected if DNS exists — set expect after DNS in scan,
|
||||
# but catalogued stack already expected; guesses start optional)
|
||||
for sub in "" www bank exchange taler merchant backend shop shops stage \
|
||||
git docs www2 api static media landing mon401
|
||||
do
|
||||
if [ -z "$sub" ]; then
|
||||
h="$d"
|
||||
else
|
||||
h="${sub}.${d}"
|
||||
fi
|
||||
# stack hosts already added as expect=1; guesses optional until we promote
|
||||
if [ -z "${HOST_PORTS[$h]+x}" ]; then
|
||||
add_host "$h" "$DOMAIN_EXTRA_PORTS" https "guess" 0
|
||||
fi
|
||||
done
|
||||
|
||||
# also pick catalog lines that belong to this domain only
|
||||
if [ -f "$CATALOG" ]; then
|
||||
local line ports proto label
|
||||
while IFS= read -r line || [ -n "$line" ]; do
|
||||
line=${line%%#*}
|
||||
line=$(echo "$line" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')
|
||||
[ -z "$line" ] && continue
|
||||
h=$(echo "$line" | awk '{print $1}')
|
||||
h=$(printf '%s' "$h" | tr 'A-Z' 'a-z' | sed 's#^https\?://##;s#/.*##')
|
||||
if [ "$h" = "$d" ] || [[ "$h" == *".$d" ]]; then
|
||||
ports=$(echo "$line" | awk '{print $2}')
|
||||
proto=$(echo "$line" | awk '{print $3}')
|
||||
label=$(echo "$line" | awk '{print $4}')
|
||||
[ "$ports" = "-" ] && ports=""
|
||||
add_host "$h" "$ports" "${proto:-https}" "${label:-dom}" 1
|
||||
fi
|
||||
done <"$CATALOG"
|
||||
fi
|
||||
}
|
||||
|
||||
# Ecosystem mode: full catalog; all entries expected
|
||||
expand_ecosystem_scope() {
|
||||
info "scope" "ecosystem mode (taler.net / gnunet.org / taler-systems.com / taler-ops / mattermost, … · remote only)"
|
||||
local h
|
||||
for h in "${ORDERED_HOSTS[@]}"; do
|
||||
HOST_EXPECT[$h]=1
|
||||
done
|
||||
}
|
||||
|
||||
# --- remote probes (no SSH) ---
|
||||
dns_ok() {
|
||||
local h="$1"
|
||||
getent ahosts "$h" >/dev/null 2>&1 || getent hosts "$h" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
# TCP connect only (not proof of service)
|
||||
tcp_open() {
|
||||
local h="$1" port="$2"
|
||||
if command -v timeout >/dev/null 2>&1; then
|
||||
timeout "$PORT_TIMEOUT" bash -c "echo >/dev/tcp/${h}/${port}" 2>/dev/null
|
||||
else
|
||||
bash -c "echo >/dev/tcp/${h}/${port}" 2>/dev/null
|
||||
fi
|
||||
}
|
||||
|
||||
# ICMP optional — never sole grounds for ERROR
|
||||
ping_host() {
|
||||
local h="$1"
|
||||
ping -c 1 -W 2 "$h" >/dev/null 2>&1 || ping -c 1 -w 2 "$h" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
# HTTPS probe: code, server header, version hints, cert subject/dates
|
||||
probe_https() {
|
||||
local h="$1" port="${2:-443}"
|
||||
local url hdr body code server via cert_end subj san
|
||||
url="https://${h}/"
|
||||
[ "$port" != "443" ] && url="https://${h}:${port}/"
|
||||
body=$(mktemp)
|
||||
hdr=$(mktemp)
|
||||
code=$(curl -skS -m "$HTTP_TIMEOUT" -D "$hdr" -o "$body" -w '%{http_code}' \
|
||||
--connect-timeout "$PORT_TIMEOUT" "$url" 2>/dev/null || echo 000)
|
||||
server=$(awk 'BEGIN{IGNORECASE=1} /^server:/{sub(/\r$/,""); sub(/^server:[[:space:]]*/,""); print; exit}' "$hdr" 2>/dev/null || true)
|
||||
via=$(awk 'BEGIN{IGNORECASE=1} /^x-powered-by:/{sub(/\r$/,""); sub(/^[^:]+:[[:space:]]*/,""); print; exit}' "$hdr" 2>/dev/null || true)
|
||||
# cert
|
||||
cert_end=$(echo | openssl s_client -servername "$h" -connect "${h}:${port}" 2>/dev/null \
|
||||
| openssl x509 -noout -enddate 2>/dev/null | sed 's/notAfter=//')
|
||||
subj=$(echo | openssl s_client -servername "$h" -connect "${h}:${port}" 2>/dev/null \
|
||||
| openssl x509 -noout -subject 2>/dev/null | head -1)
|
||||
# taler version hints in body
|
||||
local taler_hint
|
||||
taler_hint=$(grep -oE 'taler[^"[:space:]]{0,40}|GNU Taler|libeufin|gnunet' "$body" 2>/dev/null | head -3 | tr '\n' ' ' || true)
|
||||
# config JSON version if /config works
|
||||
local cfg_ver=""
|
||||
local ccode
|
||||
ccode=$(curl -skS -m "$HTTP_TIMEOUT" -o "$body" -w '%{http_code}' \
|
||||
"https://${h}/config" 2>/dev/null || echo 000)
|
||||
if [ "$ccode" = "200" ]; then
|
||||
cfg_ver=$(python3 -c 'import json,sys
|
||||
try:
|
||||
d=json.load(open(sys.argv[1]))
|
||||
print(d.get("version") or d.get("name") or d.get("currency") or "config-json")
|
||||
except Exception:
|
||||
print("")' "$body" 2>/dev/null || true)
|
||||
fi
|
||||
rm -f "$hdr" "$body"
|
||||
printf 'code=%s server=%s powered=%s cert_end=%s cfg=%s hint=%s' \
|
||||
"$code" "${server:-}" "${via:-}" "${cert_end:-}" "${cfg_ver:-}" "${taler_hint:-}"
|
||||
# return 0 if HTTP answered (any 2xx/3xx/4xx — service is up)
|
||||
case "$code" in
|
||||
2??|3??|4??) return 0 ;;
|
||||
*) return 1 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
probe_http() {
|
||||
local h="$1" port="${2:-80}"
|
||||
local url code
|
||||
url="http://${h}/"
|
||||
[ "$port" != "80" ] && url="http://${h}:${port}/"
|
||||
code=$(curl -sS -m "$HTTP_TIMEOUT" -o /dev/null -w '%{http_code}' \
|
||||
--connect-timeout "$PORT_TIMEOUT" "$url" 2>/dev/null || echo 000)
|
||||
printf 'code=%s' "$code"
|
||||
case "$code" in 2??|3??|4??) return 0 ;; *) return 1 ;; esac
|
||||
}
|
||||
|
||||
probe_ssh_banner() {
|
||||
local h="$1" port="${2:-22}"
|
||||
local ban
|
||||
ban=$(timeout "$PORT_TIMEOUT" bash -c "exec 3<>/dev/tcp/${h}/${port}; dd bs=256 count=1 <&3 2>/dev/null" 2>/dev/null \
|
||||
| tr -d '\r' | head -1 || true)
|
||||
printf 'banner=%s' "${ban:-}"
|
||||
[ -n "$ban" ]
|
||||
}
|
||||
|
||||
# Map Server header → package name guess for OSV
|
||||
guess_package() {
|
||||
local server="$1"
|
||||
local s
|
||||
s=$(printf '%s' "$server" | tr 'A-Z' 'a-z')
|
||||
case "$s" in
|
||||
*nginx*) echo "nginx" ;;
|
||||
*apache*|*httpd*) echo "apache" ;;
|
||||
*caddy*) echo "caddy" ;;
|
||||
*openbsd\ httpd*) echo "openbsd-httpd" ;;
|
||||
*) echo "" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
extract_version() {
|
||||
local server="$1"
|
||||
# nginx/1.22.1 → 1.22.1
|
||||
printf '%s' "$server" | sed -n 's/.*\/\([0-9][0-9.]*\).*/\1/p' | head -1
|
||||
}
|
||||
|
||||
# OSV / CVE check for software version taken from Server headers.
|
||||
#
|
||||
# Important: headers only expose upstream versions (nginx/1.26.3), never the
|
||||
# Debian package revision (1.26.3-3+deb13u7). Querying OSV ecosystem=Debian
|
||||
# with the bare version yields massive false positives (ancient DEBIAN-CVE-*
|
||||
# with introduced:0 and no fixed event, plus every package revision that
|
||||
# merely *starts with* 1.26.3).
|
||||
#
|
||||
# Policy:
|
||||
# - bare upstream version → upstream SEMVER signal only (WARN by default)
|
||||
# - full Debian package version (contains '-') → Debian OSV, filtered
|
||||
# - SURFACE_CVE=0 disables; SURFACE_CVE_LEVEL=error|warn (default warn for bare)
|
||||
check_cves() {
|
||||
local pkg="$1" ver="$2" host="$3"
|
||||
[ "$SURFACE_CVE" = "1" ] || return 0
|
||||
[ -n "$pkg" ] && [ -n "$ver" ] || return 0
|
||||
local level out rc
|
||||
# bare header versions are approximate → WARN unless overridden
|
||||
if [[ "$ver" == *-* ]] || [[ "$ver" == *+* ]]; then
|
||||
level="${SURFACE_CVE_LEVEL:-error}"
|
||||
else
|
||||
level="${SURFACE_CVE_LEVEL:-warn}"
|
||||
fi
|
||||
out=$(
|
||||
SURFACE_CVE_PKG="$pkg" SURFACE_CVE_VER="$ver" SURFACE_CVE_HOST="$host" python3 - <<'PY' 2>/dev/null || true
|
||||
import json, os, re, urllib.request
|
||||
|
||||
pkg = os.environ.get("SURFACE_CVE_PKG", "")
|
||||
ver = os.environ.get("SURFACE_CVE_VER", "")
|
||||
host = os.environ.get("SURFACE_CVE_HOST", "")
|
||||
|
||||
def ver_tuple(s: str):
|
||||
nums = [int(x) for x in re.findall(r"\d+", s or "")[:5]]
|
||||
return tuple(nums) if nums else ()
|
||||
|
||||
def vt_cmp(a, b):
|
||||
n = max(len(a), len(b))
|
||||
a = a + (0,) * (n - len(a))
|
||||
b = b + (0,) * (n - len(b))
|
||||
return (a > b) - (a < b)
|
||||
|
||||
def in_semver_events(ver, events):
|
||||
"""True if ver is in [introduced, fixed) for SEMVER-like event list."""
|
||||
vt = ver_tuple(ver)
|
||||
if not vt:
|
||||
return False
|
||||
# process sequential introduced/fixed pairs
|
||||
intro = None
|
||||
for e in events or []:
|
||||
if "introduced" in e:
|
||||
intro = e.get("introduced")
|
||||
elif "fixed" in e or "last_affected" in e:
|
||||
fixed = e.get("fixed")
|
||||
last = e.get("last_affected")
|
||||
lo = ver_tuple("0" if intro in (None, "0") else str(intro))
|
||||
if vt_cmp(vt, lo) < 0:
|
||||
intro = None
|
||||
continue
|
||||
if fixed is not None:
|
||||
if vt_cmp(vt, ver_tuple(str(fixed))) < 0:
|
||||
return True
|
||||
elif last is not None:
|
||||
if vt_cmp(vt, ver_tuple(str(last))) <= 0:
|
||||
return True
|
||||
intro = None
|
||||
# open-ended introduced without fixed → ignore (never clears; Debian noise)
|
||||
return False
|
||||
|
||||
def osv_post(body):
|
||||
req = urllib.request.Request(
|
||||
"https://api.osv.dev/v1/query",
|
||||
data=json.dumps(body).encode(),
|
||||
headers={"Content-Type": "application/json"},
|
||||
method="POST",
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=20) as r:
|
||||
return json.load(r)
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
def osv_get(vid):
|
||||
try:
|
||||
with urllib.request.urlopen(f"https://api.osv.dev/v1/vulns/{vid}", timeout=15) as r:
|
||||
return json.load(r)
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
bare = not (("-" in ver) or ("+" in ver))
|
||||
hits = []
|
||||
|
||||
if bare:
|
||||
# Upstream signal via CVE records that carry extracted_events for this package
|
||||
# (Server header has no Debian revision — do NOT use ecosystem=Debian).
|
||||
seed = {
|
||||
"nginx": [
|
||||
"CVE-2025-23419",
|
||||
"CVE-2025-53859",
|
||||
"CVE-2024-7347",
|
||||
"CVE-2024-34161",
|
||||
"CVE-2024-32760",
|
||||
"CVE-2024-31079",
|
||||
"CVE-2024-24989",
|
||||
"CVE-2024-24990",
|
||||
],
|
||||
"apache": [
|
||||
"CVE-2024-38474",
|
||||
"CVE-2024-38476",
|
||||
"CVE-2024-38477",
|
||||
"CVE-2023-31122",
|
||||
"CVE-2023-43622",
|
||||
],
|
||||
"caddy": [
|
||||
"CVE-2022-29718",
|
||||
"CVE-2023-50463",
|
||||
],
|
||||
}.get(pkg, [])
|
||||
for vid in seed:
|
||||
doc = osv_get(vid)
|
||||
if not doc:
|
||||
continue
|
||||
ok_hit = False
|
||||
for a in doc.get("affected") or []:
|
||||
# prefer extracted_events (human SEMVER) on GIT/nginx ranges
|
||||
for rg in a.get("ranges") or []:
|
||||
db = rg.get("database_specific") or {}
|
||||
extracted = db.get("extracted_events") or []
|
||||
if extracted and in_semver_events(ver, extracted):
|
||||
# package name filter when present
|
||||
cpes = db.get("cpe") or []
|
||||
if isinstance(cpes, str):
|
||||
cpes = [cpes]
|
||||
blob = json.dumps(a).lower() + json.dumps(cpes).lower()
|
||||
if pkg == "nginx" and "nginx" not in blob and "f5" not in blob:
|
||||
continue
|
||||
ok_hit = True
|
||||
break
|
||||
if rg.get("type") == "SEMVER" and in_semver_events(ver, rg.get("events") or []):
|
||||
ok_hit = True
|
||||
break
|
||||
if ok_hit:
|
||||
break
|
||||
if ok_hit:
|
||||
hits.append(vid)
|
||||
else:
|
||||
# Full package version — Debian ecosystem is meaningful
|
||||
data = osv_post({"package": {"name": pkg, "ecosystem": "Debian"}, "version": ver})
|
||||
for v in data.get("vulns") or []:
|
||||
vid = v.get("id") or "?"
|
||||
actionable = False
|
||||
for a in v.get("affected") or []:
|
||||
for rg in a.get("ranges") or []:
|
||||
events = rg.get("events") or []
|
||||
has_end = any(("fixed" in e) or ("last_affected" in e) for e in events)
|
||||
if not has_end:
|
||||
continue # open-ended Debian noise
|
||||
if rg.get("type") in ("ECOSYSTEM", "SEMVER"):
|
||||
# version string is full deb version; trust OSV query match
|
||||
# but only if a fixed event exists (actionable)
|
||||
actionable = True
|
||||
break
|
||||
if actionable:
|
||||
break
|
||||
# exact version listed
|
||||
versions = a.get("versions") or []
|
||||
if ver in versions:
|
||||
actionable = True
|
||||
break
|
||||
if actionable:
|
||||
hits.append(vid)
|
||||
|
||||
# de-dup preserve order
|
||||
seen = set()
|
||||
uniq = []
|
||||
for h in hits:
|
||||
if h not in seen:
|
||||
seen.add(h)
|
||||
uniq.append(h)
|
||||
|
||||
if uniq:
|
||||
print(f"HIT {len(uniq)} " + ",".join(uniq[:12]))
|
||||
else:
|
||||
print("CLEAN")
|
||||
PY
|
||||
)
|
||||
rc=0
|
||||
case "$out" in
|
||||
HIT\ *)
|
||||
local n ids
|
||||
n=$(printf '%s' "$out" | awk '{print $2}')
|
||||
ids=$(printf '%s' "$out" | cut -d' ' -f3-)
|
||||
if [ "$level" = "error" ]; then
|
||||
err "cve" "$host $pkg $ver — $n actionable vuln(s)" "$ids"
|
||||
rc=1
|
||||
else
|
||||
warn "cve" "$host $pkg $ver — $n actionable vuln(s) (header version · not Debian pkg)" "$ids"
|
||||
rc=0
|
||||
fi
|
||||
;;
|
||||
CLEAN)
|
||||
if [[ "$ver" == *-* ]] || [[ "$ver" == *+* ]]; then
|
||||
info "cve" "$host $pkg $ver — OSV clean (Debian package version)"
|
||||
else
|
||||
info "cve" "$host $pkg $ver — no actionable upstream CVE for bare Server-header version"
|
||||
fi
|
||||
;;
|
||||
*)
|
||||
info "cve" "$host $pkg $ver — CVE probe skipped/unavailable"
|
||||
;;
|
||||
esac
|
||||
return "$rc"
|
||||
}
|
||||
|
||||
cert_expiry_check() {
|
||||
local h="$1" end="$2"
|
||||
[ -n "$end" ] || return 0
|
||||
local end_epoch now left days
|
||||
end_epoch=$(date -d "$end" +%s 2>/dev/null || date -j -f "%b %e %T %Y %Z" "$end" +%s 2>/dev/null || echo 0)
|
||||
now=$(date +%s)
|
||||
[ "$end_epoch" -gt 0 ] || return 0
|
||||
left=$((end_epoch - now))
|
||||
days=$((left / 86400))
|
||||
if [ "$left" -le 0 ]; then
|
||||
err "tls" "$h certificate EXPIRED" "notAfter=$end"
|
||||
return 1
|
||||
fi
|
||||
if [ "$days" -le 14 ]; then
|
||||
warn "tls" "$h certificate expires in ${days}d" "notAfter=$end"
|
||||
else
|
||||
info "tls" "$h cert ok · ${days}d left · notAfter=$end"
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
scan_host() {
|
||||
local h="$1"
|
||||
local ports proto label expect
|
||||
local p open_ports=() any_proto_ok=0 ping_ok=0 dns=0
|
||||
local probe_detail server_hdr pkg ver
|
||||
|
||||
ports=${HOST_PORTS[$h]:-443}
|
||||
proto=${HOST_PROTO[$h]:-https}
|
||||
label=${HOST_LABEL[$h]:-$h}
|
||||
expect=${HOST_EXPECT[$h]:-0}
|
||||
|
||||
set_group "$label"
|
||||
|
||||
if dns_ok "$h"; then
|
||||
dns=1
|
||||
ok "dns" "$h resolves"
|
||||
else
|
||||
if [ "$expect" = "1" ]; then
|
||||
err "dns" "$h does not resolve (catalogued)"
|
||||
else
|
||||
info "dns" "$h no resolve (optional guess) — skip"
|
||||
fi
|
||||
return 0
|
||||
fi
|
||||
|
||||
if ping_host "$h"; then
|
||||
ping_ok=1
|
||||
info "ping" "$h ICMP ok"
|
||||
else
|
||||
info "ping" "$h ICMP no reply (not decisive)"
|
||||
fi
|
||||
|
||||
# port scan (unique ports)
|
||||
local _seen_ports=" "
|
||||
for p in ${ports//,/ }; do
|
||||
[ -n "$p" ] || continue
|
||||
case "$_seen_ports" in *" $p "*) continue ;; esac
|
||||
_seen_ports="$_seen_ports$p "
|
||||
if tcp_open "$h" "$p"; then
|
||||
open_ports+=("$p")
|
||||
info "port" "$h:$p open (TCP)"
|
||||
else
|
||||
info "port" "$h:$p closed/filtered (TCP)"
|
||||
fi
|
||||
done
|
||||
|
||||
# Protocol verification on open ports (and always try 443/80 for https/http expect)
|
||||
local try_ports=("${open_ports[@]}")
|
||||
if [ ${#try_ports[@]} -eq 0 ]; then
|
||||
# still try expected protocol ports even if scan said closed (scan false negatives)
|
||||
case "$proto" in
|
||||
https) try_ports=(443) ;;
|
||||
http) try_ports=(80) ;;
|
||||
ssh) try_ports=(22) ;;
|
||||
*) try_ports=(443 80) ;;
|
||||
esac
|
||||
fi
|
||||
|
||||
server_hdr=""
|
||||
ver=""
|
||||
for p in "${try_ports[@]}"; do
|
||||
case "$p" in
|
||||
443|8443)
|
||||
if probe_detail=$(probe_https "$h" "$p"); then
|
||||
any_proto_ok=1
|
||||
ok "https" "$h:$p up · $probe_detail"
|
||||
server_hdr=$(printf '%s' "$probe_detail" | sed -n 's/.*server=\([^ ]*\).*/\1/p')
|
||||
# cert
|
||||
local cend
|
||||
cend=$(printf '%s' "$probe_detail" | sed -n 's/.*cert_end=\([^ ]*\).*/\1/p')
|
||||
cert_expiry_check "$h" "$cend" || true
|
||||
local cfg
|
||||
cfg=$(printf '%s' "$probe_detail" | sed -n 's/.*cfg=\([^ ]*\).*/\1/p')
|
||||
if [ -n "$cfg" ]; then
|
||||
info "version" "$h config/api hint: $cfg"
|
||||
ver="$cfg"
|
||||
fi
|
||||
else
|
||||
info "https" "$h:$p no HTTP response · ${probe_detail:-}"
|
||||
fi
|
||||
;;
|
||||
80|8080)
|
||||
if probe_detail=$(probe_http "$h" "$p"); then
|
||||
any_proto_ok=1
|
||||
ok "http" "$h:$p up · $probe_detail"
|
||||
else
|
||||
info "http" "$h:$p no HTTP response"
|
||||
fi
|
||||
;;
|
||||
22)
|
||||
if probe_detail=$(probe_ssh_banner "$h" "$p"); then
|
||||
any_proto_ok=1
|
||||
ok "ssh" "$h:$p banner · $probe_detail"
|
||||
ver=$(printf '%s' "$probe_detail" | sed 's/banner=//')
|
||||
else
|
||||
info "ssh" "$h:$p no SSH banner"
|
||||
fi
|
||||
;;
|
||||
*)
|
||||
if tcp_open "$h" "$p"; then
|
||||
info "tcp" "$h:$p open · protocol unknown"
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Version summary
|
||||
if [ -n "$server_hdr" ]; then
|
||||
info "server-header" "$h · $server_hdr"
|
||||
pkg=$(guess_package "$server_hdr")
|
||||
ver_soft=$(extract_version "$server_hdr")
|
||||
if [ -n "$pkg" ] && [ -n "$ver_soft" ]; then
|
||||
info "version" "$h software $pkg $ver_soft (from Server header)"
|
||||
check_cves "$pkg" "$ver_soft" "$h" || true
|
||||
elif [ -n "$server_hdr" ]; then
|
||||
warn "version" "$h Server header present but version unknown · $server_hdr"
|
||||
fi
|
||||
elif [ -n "$ver" ]; then
|
||||
info "version" "$h · $ver"
|
||||
else
|
||||
if [ "$any_proto_ok" = "1" ]; then
|
||||
warn "version" "$h reachable but software version unknown"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Expected service must answer protocol (not just ping/port)
|
||||
# Note: ${array[*]:-x} is NOT valid default syntax (bash treats : as slice).
|
||||
local ports_txt="${open_ports[*]}"
|
||||
ports_txt=${ports_txt:-none}
|
||||
if [ "$expect" = "1" ]; then
|
||||
if [ "$any_proto_ok" = "1" ]; then
|
||||
ok "reachability" "$h catalogued service OK (protocol confirmed)"
|
||||
else
|
||||
err "reachability" "$h catalogued but not reachable via ${proto}/protocol" \
|
||||
"dns=$dns ping=$ping_ok open_ports=${ports_txt} (ICMP/TCP alone not enough; protocol failed)"
|
||||
fi
|
||||
else
|
||||
if [ "$any_proto_ok" = "1" ]; then
|
||||
info "reachability" "$h optional host responds"
|
||||
else
|
||||
info "reachability" "$h optional · no protocol response (ok)"
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
# --- main ---
|
||||
# Scope (remote only):
|
||||
# ./taler-monitoring.sh surface → full ecosystem catalog
|
||||
# ./taler-monitoring.sh -d hacktivism.ch surface → only that domain
|
||||
if [ "${SURFACE_SCOPE:-}" = "ecosystem" ]; then
|
||||
load_catalog
|
||||
expand_ecosystem_scope
|
||||
elif [ "${SURFACE_SCOPE:-}" = "domain" ] || [ "${TALER_DOMAIN_FROM_CLI:-0}" = "1" ]; then
|
||||
# domain mode: do not load whole ecosystem first
|
||||
CATALOG="${SURFACE_CATALOG:-$ROOT/surface-catalog.conf}"
|
||||
expand_domain_scope "${TALER_DOMAIN:-hacktivism.ch}"
|
||||
elif [ "${SURFACE_SCOPE:-auto}" = "auto" ]; then
|
||||
# no -d → ecosystem; with -d → domain (TALER_DOMAIN_FROM_CLI)
|
||||
if [ "${TALER_DOMAIN_FROM_CLI:-0}" = "1" ]; then
|
||||
CATALOG="${SURFACE_CATALOG:-$ROOT/surface-catalog.conf}"
|
||||
expand_domain_scope "$TALER_DOMAIN"
|
||||
else
|
||||
load_catalog
|
||||
expand_ecosystem_scope
|
||||
fi
|
||||
else
|
||||
load_catalog
|
||||
expand_ecosystem_scope
|
||||
fi
|
||||
|
||||
info "inventory" "${#ORDERED_HOSTS[@]} hosts to probe (remote only · no SSH · no local podman on targets)"
|
||||
info "flags" "SURFACE_CVE=${SURFACE_CVE:-1} PORT_TIMEOUT=${PORT_TIMEOUT:-2} HTTP_TIMEOUT=${HTTP_TIMEOUT:-12}"
|
||||
|
||||
for h in "${ORDERED_HOSTS[@]}"; do
|
||||
[ -n "$h" ] || continue
|
||||
scan_host "$h" || true
|
||||
done
|
||||
|
||||
# mon_disk on the *runner* only (optional) — not remote servers' disks
|
||||
if [ "${SURFACE_CHECK_RUNNER_DISK:-0}" = "1" ]; then
|
||||
set_group disk
|
||||
if declare -F mon_disk_check_host >/dev/null 2>&1; then
|
||||
mon_disk_check_host "runner" || true
|
||||
else
|
||||
warn "disk" "mon_disk_check_host not available"
|
||||
fi
|
||||
fi
|
||||
|
||||
# summary returns non-zero when FAIL_N>0; do not trip set -e mid-script
|
||||
summary || true
|
||||
if [ "${FAIL_N:-0}" -eq 0 ]; then
|
||||
exit 0
|
||||
fi
|
||||
exit 1
|
||||
|
|
@ -166,16 +166,16 @@ fi
|
|||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2) INSIDE — containers can reach deb.taler.net (pasta / install path)
|
||||
# koopa: KOOPA_SSH · stage-lfp: INSIDE_SSH (stagepaysan, low-priv)
|
||||
# host-podman: INSIDE_PODMAN=1 on GOA host · ssh: KOOPA_SSH / INSIDE_SSH from laptop
|
||||
# ---------------------------------------------------------------------------
|
||||
set_group inside
|
||||
section "versions · inside · containers → deb.taler.net"
|
||||
|
||||
_VERS_LOCAL=0
|
||||
_VERS_SSH_HOST=""
|
||||
_VERS_BANK="taler-hacktivism-bank"
|
||||
_VERS_EX="taler-hacktivism-exchange-ansible"
|
||||
_VERS_MER="taler-hacktivism"
|
||||
if [ "${INSIDE_PROFILE:-}" = "stage-lfp" ] \
|
||||
if [ "${INSIDE_PODMAN:-0}" = "1" ] || [ "${INSIDE_MODE:-}" = "local-podman" ]; then
|
||||
_VERS_LOCAL=1
|
||||
elif [ "${INSIDE_PROFILE:-}" = "stage-lfp" ] \
|
||||
|| [ "${EXPECT_CURRENCY:-}" = "TESTPAYSAN" ] \
|
||||
|| [ -n "${INSIDE_SSH:-}" ]; then
|
||||
_VERS_SSH_HOST="${INSIDE_SSH:-francpaysan-stage-user}"
|
||||
|
|
@ -183,24 +183,44 @@ if [ "${INSIDE_PROFILE:-}" = "stage-lfp" ] \
|
|||
_VERS_EX="${INSIDE_EXCHANGE_CTR:-stage-lfp-exchange-ansible}"
|
||||
_VERS_MER="${INSIDE_MERCHANT_CTR:-stage-lfp-merchant}"
|
||||
elif [ "${LOCAL_STACK:-0}" = "1" ] && [ "${SKIP_SSH:-0}" != "1" ]; then
|
||||
# laptop → koopa via SSH (not host-podman)
|
||||
_VERS_SSH_HOST="${KOOPA_SSH}"
|
||||
fi
|
||||
|
||||
if [ -z "$_VERS_SSH_HOST" ]; then
|
||||
info "ssh" "no container SSH for this domain — outside-only package index"
|
||||
if [ "$_VERS_LOCAL" = "1" ]; then
|
||||
VERS_ACCESS=host-podman
|
||||
set_group host
|
||||
else
|
||||
VERS_ACCESS=ssh
|
||||
set_group ssh
|
||||
fi
|
||||
|
||||
section "versions · inside · containers → deb.taler.net (access=${VERS_ACCESS})"
|
||||
info "flags" "VERS_ACCESS=${VERS_ACCESS} INSIDE_PODMAN=${INSIDE_PODMAN:-0} INSIDE_MODE=${INSIDE_MODE:-} LOCAL_STACK=${LOCAL_STACK:-0} SKIP_SSH=${SKIP_SSH:-0} KOOPA_SSH=${KOOPA_SSH:-} INSIDE_SSH=${INSIDE_SSH:-}"
|
||||
|
||||
if [ "$_VERS_LOCAL" != "1" ] && [ -z "$_VERS_SSH_HOST" ]; then
|
||||
info "access" "no host-podman and no SSH target — outside-only package index"
|
||||
info "outside-only" "deb.taler.net public checks completed above"
|
||||
summary
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [ "$_VERS_SSH_HOST" = "${KOOPA_SSH:-}" ] || [ "${INSIDE_PROFILE:-}" = "koopa" ]; then
|
||||
if ! koopa_ssh_ok; then
|
||||
err "ssh" "cannot reach ${KOOPA_SSH} — set SKIP_SSH=1 to skip container checks"
|
||||
if [ "$_VERS_LOCAL" = "1" ]; then
|
||||
if ! command -v podman >/dev/null 2>&1; then
|
||||
err "host" "INSIDE_PODMAN/host-podman but podman missing"
|
||||
info "outside-only" "deb.taler.net public checks completed above"
|
||||
summary
|
||||
exit 1
|
||||
fi
|
||||
ok "ssh ${KOOPA_SSH}"
|
||||
ok "host→container" "podman exec on this host (VERS_ACCESS=host-podman · no SSH)"
|
||||
elif [ "$_VERS_SSH_HOST" = "${KOOPA_SSH:-}" ] || [ "${INSIDE_PROFILE:-}" = "koopa" ]; then
|
||||
if ! koopa_ssh_ok; then
|
||||
err "ssh" "cannot reach ${KOOPA_SSH} — from laptop set KOOPA_SSH=; on host set INSIDE_PODMAN=1"
|
||||
info "outside-only" "deb.taler.net public checks completed above"
|
||||
summary
|
||||
exit 1
|
||||
fi
|
||||
ok "ssh ${KOOPA_SSH}" "remote then podman (VERS_ACCESS=ssh)"
|
||||
else
|
||||
if ! mon_ssh_ok "$_VERS_SSH_HOST"; then
|
||||
err "ssh" "cannot reach ${_VERS_SSH_HOST} (stagepaysan) — skip container package compare"
|
||||
|
|
@ -208,7 +228,7 @@ else
|
|||
summary
|
||||
exit 1
|
||||
fi
|
||||
ok "ssh ${_VERS_SSH_HOST}" "stagepaysan container package compare"
|
||||
ok "ssh ${_VERS_SSH_HOST}" "stagepaysan (VERS_ACCESS=ssh)"
|
||||
fi
|
||||
|
||||
INRELEASE_URL="${APT_BASE}/dists/${SUITE}/InRelease"
|
||||
|
|
@ -257,7 +277,9 @@ done
|
|||
REMOTE
|
||||
} >"$tmp/remote-versions.sh"
|
||||
|
||||
if [ "$_VERS_SSH_HOST" = "${KOOPA_SSH:-}" ] || [ "${INSIDE_PROFILE:-}" = "koopa" ]; then
|
||||
if [ "$_VERS_LOCAL" = "1" ]; then
|
||||
REMOTE_OUT=$(bash "$tmp/remote-versions.sh" 2>/dev/null || true)
|
||||
elif [ "$_VERS_SSH_HOST" = "${KOOPA_SSH:-}" ] || [ "${INSIDE_PROFILE:-}" = "koopa" ]; then
|
||||
REMOTE_OUT=$(koopa_ssh_bash 60 <"$tmp/remote-versions.sh" || true)
|
||||
else
|
||||
REMOTE_OUT=$(mon_ssh_bash "$_VERS_SSH_HOST" 60 <"$tmp/remote-versions.sh" || true)
|
||||
|
|
@ -501,4 +523,41 @@ done
|
|||
info "suite" "deb.taler.net ${SUITE} (testing_index=${HAVE_TESTING})"
|
||||
info "tally" "match=${ok_n} ahead=${ahead_n} behind=${behind_n}"
|
||||
|
||||
# Disk free on the machine(s) we just probed (host-podman or SSH target)
|
||||
set_group disk
|
||||
section "versions · disk free space"
|
||||
if [ "${_VERS_LOCAL:-0}" = "1" ]; then
|
||||
mon_disk_check_host "host" || true
|
||||
while read -r _vc; do
|
||||
[ -n "$_vc" ] || continue
|
||||
mon_disk_check_podman "$_vc" || true
|
||||
done < <(podman ps --format '{{.Names}}' 2>/dev/null | grep -iE 'hacktivism|taler-|stage-lfp|lfp-' || true)
|
||||
elif [ -n "${_VERS_SSH_HOST:-}" ]; then
|
||||
_vdisk=$(mon_ssh_bash "$_VERS_SSH_HOST" 20 <<'DISK' || true
|
||||
set +e
|
||||
echo "###HOST###"
|
||||
df -Pk / /var /home /tmp 2>/dev/null || df -Pk
|
||||
echo "###CTRS###"
|
||||
for c in $(podman ps --format '{{.Names}}' 2>/dev/null); do
|
||||
echo "###CTR $c###"
|
||||
podman exec "$c" df -Pk / /var /tmp 2>/dev/null || true
|
||||
done
|
||||
DISK
|
||||
)
|
||||
mon_disk_check_remote_text "ssh:${_VERS_SSH_HOST}" \
|
||||
"$(printf '%s\n' "$_vdisk" | sed -n '/^###HOST###$/,/^###CTRS###$/p' | sed '1d;$d')" || true
|
||||
_ctr=""; _buf=""
|
||||
while IFS= read -r _line || [ -n "$_line" ]; do
|
||||
case "$_line" in
|
||||
'###CTR '*)
|
||||
[ -n "$_ctr" ] && mon_disk_check_remote_text "ctr:${_ctr}" "$_buf" || true
|
||||
_ctr=${_line####CTR }; _ctr=${_ctr%###}; _buf=""
|
||||
;;
|
||||
'###CTRS###'|'###HOST###') ;;
|
||||
*) [ -n "$_ctr" ] && _buf="${_buf}${_line}"$'\n' ;;
|
||||
esac
|
||||
done <<<"$_vdisk"
|
||||
[ -n "$_ctr" ] && mon_disk_check_remote_text "ctr:${_ctr}" "$_buf" || true
|
||||
fi
|
||||
|
||||
summary
|
||||
|
|
|
|||
|
|
@ -23,6 +23,9 @@ MERCHANT_INSTANCE=$(printf '%s' "${MERCHANT_INSTANCE}" | tr '[:upper:]' '[:lower
|
|||
: "${TIMEOUT:=12}"
|
||||
: "${E2E_TIMEOUT:=55}" # whole e2e budget; skip rest when exceeded (e2e raises as needed)
|
||||
: "${E2E_PAY_SECS:=22}" # dedicated seconds for pay handle-uri (avoid Alarm clock)
|
||||
# Whole-run wall clock for taler-monitoring.sh (seconds). 0 = unlimited.
|
||||
# Host-agent default phases (urls/inside/versions) should finish under this.
|
||||
: "${RUN_TIMEOUT:=600}"
|
||||
# Local GOA: public paivana paywall (https://paivana.hacktivism.ch · template GOA:4200)
|
||||
: "${PAIVANA_PUBLIC:=https://paivana.hacktivism.ch}"
|
||||
: "${E2E_PAIVANA:=1}" # 0 = skip paivana section in e2e
|
||||
|
|
@ -855,6 +858,113 @@ summary() {
|
|||
[ "$FAIL_N" -eq 0 ]
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Disk space on hosts / containers under test
|
||||
# WARN when tight, ERROR when full / critically low
|
||||
# DISK_WARN_USED_PCT=85 # free below 15% → WARN
|
||||
# DISK_ERR_USED_PCT=95 # free below 5% → ERROR
|
||||
# DISK_ERR_FREE_BYTES=0 # free == 0 always ERROR
|
||||
# ---------------------------------------------------------------------------
|
||||
: "${DISK_WARN_USED_PCT:=85}"
|
||||
: "${DISK_ERR_USED_PCT:=95}"
|
||||
|
||||
# Parse one POSIX `df -P` body line (no header): fs size used avail capacity mount
|
||||
# size/used/avail in 1K-blocks when using df -Pk
|
||||
_mon_disk_eval_line() {
|
||||
local where="$1" fs="$2" size="$3" used="$4" avail="$5" cap="$6" mnt="$7"
|
||||
local pct label detail
|
||||
pct=${cap%%%}
|
||||
# non-numeric capacity → skip
|
||||
case "$pct" in
|
||||
''|*[!0-9]*) return 0 ;;
|
||||
esac
|
||||
label="disk ${where} ${mnt}"
|
||||
detail="fs=$fs size_1k=$size used_1k=$used avail_1k=$avail use=${pct}%"
|
||||
if [ "$avail" = "0" ] || [ "$pct" -ge 100 ]; then
|
||||
err "disk" "${where} ${mnt} FULL / overflow" "$detail"
|
||||
return 1
|
||||
fi
|
||||
if [ "$pct" -ge "${DISK_ERR_USED_PCT}" ]; then
|
||||
err "disk" "${where} ${mnt} critically low free space (≥${DISK_ERR_USED_PCT}% used)" "$detail"
|
||||
return 1
|
||||
fi
|
||||
if [ "$pct" -ge "${DISK_WARN_USED_PCT}" ]; then
|
||||
warn "disk" "${where} ${mnt} free space tight (≥${DISK_WARN_USED_PCT}% used)" "$detail"
|
||||
return 0
|
||||
fi
|
||||
ok "$label" "$detail"
|
||||
return 0
|
||||
}
|
||||
|
||||
# Report all filesystems from `df -P` / `df -Pk` output (with header).
|
||||
# $1 = where label (host, container:name, ssh:host)
|
||||
# $2 = full df text
|
||||
mon_disk_report_df() {
|
||||
local where="$1" text="$2" line fs size used avail cap mnt rest ec=0
|
||||
[ -n "$text" ] || {
|
||||
warn "disk" "${where}: no df output"
|
||||
return 0
|
||||
}
|
||||
while IFS= read -r line || [ -n "$line" ]; do
|
||||
[ -n "$line" ] || continue
|
||||
case "$line" in
|
||||
Filesystem*|Filesystem*) continue ;;
|
||||
esac
|
||||
# df -P: Filesystem 1024-blocks Used Available Capacity Mounted on
|
||||
# shellcheck disable=SC2086
|
||||
set -- $line
|
||||
[ "$#" -ge 6 ] || continue
|
||||
fs=$1
|
||||
size=$2
|
||||
used=$3
|
||||
avail=$4
|
||||
cap=$5
|
||||
shift 5
|
||||
mnt=$*
|
||||
# skip special/pseudo if tiny and not root-ish
|
||||
case "$fs" in
|
||||
tmpfs|devtmpfs|overlay|shm|nsfs|proc|sysfs|cgroup*)
|
||||
# still check root-like mounts that fill (overlay / in containers)
|
||||
case "$mnt" in
|
||||
/|/var|/var/*|/home|/home/*|/data|/mnt/*) ;;
|
||||
*) continue ;;
|
||||
esac
|
||||
;;
|
||||
esac
|
||||
if ! _mon_disk_eval_line "$where" "$fs" "$size" "$used" "$avail" "$cap" "$mnt"; then
|
||||
ec=1
|
||||
fi
|
||||
done <<<"$text"
|
||||
return "$ec"
|
||||
}
|
||||
|
||||
# Local host: important mounts
|
||||
mon_disk_check_host() {
|
||||
local where text
|
||||
where="${1:-host}"
|
||||
text=""
|
||||
text=$(df -Pk / /var /home /tmp 2>/dev/null | awk 'NR==1 || !seen[$1,$6]++' || true)
|
||||
# fallback whole table
|
||||
if [ -z "$text" ]; then
|
||||
text=$(df -Pk 2>/dev/null || true)
|
||||
fi
|
||||
mon_disk_report_df "$where" "$text"
|
||||
}
|
||||
|
||||
# Run df inside podman container
|
||||
mon_disk_check_podman() {
|
||||
local cname="$1" text
|
||||
local bin="${2:-podman}"
|
||||
text=$("$bin" exec "$cname" df -Pk / /var /tmp 2>/dev/null || "$bin" exec "$cname" df -Pk 2>/dev/null || true)
|
||||
mon_disk_report_df "ctr:${cname}" "$text"
|
||||
}
|
||||
|
||||
# df text already collected remotely
|
||||
mon_disk_check_remote_text() {
|
||||
local where="$1" text="$2"
|
||||
mon_disk_report_df "$where" "$text"
|
||||
}
|
||||
|
||||
http_code() {
|
||||
local url="$1"; shift
|
||||
curl -skS --max-redirs 0 -m "${TIMEOUT}" -o /dev/null -w '%{http_code}' "$@" "$url" 2>/dev/null || echo 000
|
||||
|
|
|
|||
52
scripts/taler-monitoring/surface-catalog.conf
Normal file
52
scripts/taler-monitoring/surface-catalog.conf
Normal file
|
|
@ -0,0 +1,52 @@
|
|||
# surface-catalog.conf — known public Taler / GNUnet / Taler Systems hosts
|
||||
# Format (whitespace-separated; # comments):
|
||||
# host ports expect_proto label
|
||||
# expect_proto: https | http | ssh | tcp | any
|
||||
# ports: comma-separated (empty = defaults for proto)
|
||||
#
|
||||
# This list is expanded at runtime (DNS discovery, cert SANs, -d domain profile).
|
||||
# Phase `surface` is NOT in default phases — run explicitly:
|
||||
# ./taler-monitoring.sh surface
|
||||
# ./taler-monitoring.sh -d hacktivism.ch surface
|
||||
|
||||
# --- taler.net ecosystem ---
|
||||
www.taler.net 443 https taler-www
|
||||
docs.taler.net 443 https taler-docs
|
||||
git.taler.net 443,22 https taler-git
|
||||
bugs.taler.net 443 https taler-bugs
|
||||
lists.taler.net 443 https taler-lists
|
||||
tutorials.taler.net 443 https taler-tutorials
|
||||
bank.demo.taler.net 443 https demo-bank
|
||||
exchange.demo.taler.net 443 https demo-exchange
|
||||
backend.demo.taler.net 443 https demo-merchant
|
||||
shop.demo.taler.net 443 https demo-shop
|
||||
bank.test.taler.net 443 https test-bank
|
||||
exchange.test.taler.net 443 https test-exchange
|
||||
backend.test.taler.net 443 https test-merchant
|
||||
|
||||
# --- taler-ops / TOPS ---
|
||||
www.taler-ops.ch 443 https tops-www
|
||||
taler-ops.ch 443 https tops-apex
|
||||
bank.taler-ops.ch 443 https tops-bank
|
||||
exchange.taler-ops.ch 443 https tops-exchange
|
||||
my.taler-ops.ch 443 https tops-merchant
|
||||
stage.taler-ops.ch 443 https tops-stage-www
|
||||
stage.my.taler-ops.ch 443 https tops-stage-merchant
|
||||
exchange.stage.taler-ops.ch 443 https tops-stage-exchange
|
||||
nexus.stage.taler-ops.ch 443 https tops-stage-nexus
|
||||
shops.taler-ops.ch 443 https tops-shops
|
||||
mattermost.taler.net 443 https mattermost
|
||||
|
||||
# --- gnunet.org ---
|
||||
www.gnunet.org 443,80 https gnunet-www
|
||||
gnunet.org 443,80 https gnunet-apex
|
||||
git.gnunet.org 443,22 https gnunet-git
|
||||
bugs.gnunet.org 443 https gnunet-bugs
|
||||
|
||||
# --- taler-systems.com ---
|
||||
www.taler-systems.com 443 https tsa-www
|
||||
taler-systems.com 443 https tsa-apex
|
||||
|
||||
# --- common infra hostnames (remote only; may 404/redirect) ---
|
||||
deb.taler.net 443 https deb-taler
|
||||
ftp.gnu.org 443 https gnu-ftp
|
||||
|
|
@ -12,6 +12,12 @@
|
|||
set -euo pipefail
|
||||
ROOT=$(cd "$(dirname "$0")" && pwd)
|
||||
|
||||
# Line-buffered stdout/stderr so redirected logs (tee/host-agent) flush each line.
|
||||
if [ "${_TALER_MON_STDBUF:-0}" != "1" ] && command -v stdbuf >/dev/null 2>&1; then
|
||||
export _TALER_MON_STDBUF=1
|
||||
exec stdbuf -oL -eL bash "$0" "$@"
|
||||
fi
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
taler-monitoring — bank / exchange / merchant checks
|
||||
|
|
@ -29,8 +35,14 @@ Phases:
|
|||
e2e withdraw + pay (small amounts; remote aborts on login/KYC)
|
||||
ladder withdraw/pay amount ladder (GOA ceiling or stage TESTPAYSAN max_wire)
|
||||
auth401 merchant Basic-auth / case matrix (HTTP 401 paths; may create throwaway instance)
|
||||
aptdeploy koopa podman apt-src smoke: taler-merchant in
|
||||
koopa-taler-deploy-test-apt-src-trixie{,-testing}
|
||||
surface REMOTE-ONLY public inventory (NOT in default/all/full):
|
||||
ecosystem hosts (taler.net, gnunet.org, taler-systems.com, mattermost, …)
|
||||
or -d DOMAIN → that domain’s surface; port/protocol/TLS/CVE (OSV)
|
||||
all urls + inside + versions + sanity + e2e (SSH phases only on koopa)
|
||||
full all + server + ladder + auth401 (maximum; long-running, needs secrets)
|
||||
NOTE: surface is never included in all/full — pass it explicitly
|
||||
|
||||
Options:
|
||||
-d, --domain DOMAIN load profile from domains.conf (bank/exchange/merchant)
|
||||
|
|
@ -61,6 +73,10 @@ Env (same meaning):
|
|||
TALER_DOMAINS_CONF SKIP_SSH=1
|
||||
NO_COLOR=1 / CLICOLOR=0 disable green/yellow/red tags (default: coloured)
|
||||
SKIP_SSH=1 NO_COLOR=1
|
||||
RUN_TIMEOUT=600 whole-run wall clock seconds (default 600; 0=unlimited)
|
||||
long phases (ladder/full/e2e) need a higher value or 0
|
||||
DISK_WARN_USED_PCT=85 disk free: WARN when used ≥ this %
|
||||
DISK_ERR_USED_PCT=95 disk free: ERROR when used ≥ this % (or avail=0 / 100%)
|
||||
PERF_WARN_MS PERF_FAIL_MS (urls latency; default 8000 / 20000)
|
||||
QR_CHECK=0 skip QR form + qrencode/zbarimg (urls phase)
|
||||
QR_ECC=M qrencode ECC level (default M)
|
||||
|
|
@ -73,11 +89,23 @@ Env (same meaning):
|
|||
AUTH401_* see check_auth401.sh / secrets.env.example
|
||||
AUTH401_CONTINUE=1 / CONTINUE_ON_ERROR=1
|
||||
auth401: run all groups, collect every ERROR (no mid-run halt)
|
||||
APT_DEPLOY_SKIP=1 skip aptdeploy phase
|
||||
APT_DEPLOY_CONTAINERS="name:suite …" default trixie + trixie-testing deploy-test pods
|
||||
SURFACE_CVE=0 disable CVE queries in surface phase
|
||||
SURFACE_CVE_LEVEL=warn|error bare Server-header versions default warn
|
||||
(Debian package versions default error)
|
||||
SURFACE_CATALOG=path override surface-catalog.conf
|
||||
|
||||
Full load (GOA / hacktivism):
|
||||
./taler-monitoring.sh -d hacktivism.ch full
|
||||
# or explicit:
|
||||
./taler-monitoring.sh -d hacktivism.ch urls inside versions sanity server e2e ladder auth401
|
||||
# hacktivism host-agent also runs aptdeploy (apt-src merchant containers on koopa)
|
||||
|
||||
Remote surface / ecosystem (explicit only — never default):
|
||||
./taler-monitoring.sh surface
|
||||
./taler-monitoring.sh -d hacktivism.ch surface
|
||||
./taler-monitoring.sh -d lefrancpaysan.ch surface
|
||||
|
||||
SPA pin after selfbuild:
|
||||
EXPECT_WEBUI_VERSION=1.6.11 EXPECT_WEBUI_OVERLAY=selfbuild-v1.6.11 \\
|
||||
|
|
@ -111,6 +139,7 @@ while [ $# -gt 0 ]; do
|
|||
[ $# -ge 2 ] || { echo "missing arg for $1" >&2; exit 2; }
|
||||
apply_taler_domain "$2"
|
||||
DOMAIN_SET=1
|
||||
TALER_DOMAIN_FROM_CLI=1
|
||||
shift 2
|
||||
;;
|
||||
--bank)
|
||||
|
|
@ -130,12 +159,13 @@ while [ $# -gt 0 ]; do
|
|||
CURRENCY_OVERRIDE="$2"; shift 2
|
||||
;;
|
||||
--no-probe) NO_PROBE=1; shift ;;
|
||||
urls|inside|versions|sanity|server|e2e|ladder|goa-ladder|auth401|all|full) PHASES+=("$1"); shift ;;
|
||||
urls|inside|versions|sanity|server|e2e|ladder|goa-ladder|auth401|aptdeploy|apt-deploy|apt_src|surface|ecosystem|all|full) PHASES+=("$1"); shift ;;
|
||||
*)
|
||||
# bare domain shorthand: ./taler-monitoring.sh taler.net
|
||||
if [[ "$1" == *.* && "$1" != *://* && "$1" != -* ]]; then
|
||||
apply_taler_domain "$1"
|
||||
DOMAIN_SET=1
|
||||
TALER_DOMAIN_FROM_CLI=1
|
||||
shift
|
||||
else
|
||||
echo "Unknown: $1" >&2; usage >&2; exit 2
|
||||
|
|
@ -170,6 +200,8 @@ fi
|
|||
# Export so check_*.sh (re-source lib) see the same targets via env
|
||||
export TALER_DOMAIN BANK_PUBLIC EXCHANGE_PUBLIC MERCHANT_PUBLIC
|
||||
export EXPECT_CURRENCY SKIP_SSH LOCAL_STACK TALER_DOMAIN_PROBE CHECK_LANDING
|
||||
export TALER_DOMAIN_FROM_CLI="${TALER_DOMAIN_FROM_CLI:-0}"
|
||||
export DOMAIN_SET="${DOMAIN_SET:-0}"
|
||||
export WITHDRAW_AMT PAY_AMT CREDIT_AMT MERCHANT_INSTANCE
|
||||
export E2E_FAKE_INCOMING E2E_REMOTE E2E_VARIABLE E2E_ATM_MAX
|
||||
export E2E_WITHDRAW_VALUES E2E_PAY_VALUES E2E_USE_TEMPLATES E2E_TEMPLATE_MAP
|
||||
|
|
@ -288,33 +320,158 @@ if [ "${PROGRESS_TOTAL:-0}" = "0" ] || [ -z "${PROGRESS_TOTAL:-}" ]; then
|
|||
e2e) _pt=$((_pt + 240)) ;;
|
||||
ladder|goa-ladder) _pt=$((_pt + 120)) ;;
|
||||
auth401) _pt=$((_pt + 70)) ;;
|
||||
aptdeploy) _pt=$((_pt + 20)) ;;
|
||||
surface|ecosystem) _pt=$((_pt + 80)) ;;
|
||||
esac
|
||||
done
|
||||
set_progress_total "$_pt"
|
||||
unset _pt
|
||||
fi
|
||||
|
||||
# Whole-run wall clock (default 10 min). Override: RUN_TIMEOUT=0 (unlimited).
|
||||
: "${RUN_TIMEOUT:=600}"
|
||||
export RUN_TIMEOUT
|
||||
MON_T0=$(date +%s)
|
||||
export MON_T0
|
||||
RUN_TIMED_OUT=0
|
||||
RUN_TIMEOUT_AT_PHASE=""
|
||||
RUN_SKIPPED_PHASES=()
|
||||
|
||||
mon_seconds_left() {
|
||||
if [ "${RUN_TIMEOUT:-0}" -eq 0 ]; then
|
||||
printf '%s' "999999"
|
||||
return 0
|
||||
fi
|
||||
local now left
|
||||
now=$(date +%s)
|
||||
left=$((RUN_TIMEOUT - (now - MON_T0)))
|
||||
[ "$left" -lt 0 ] && left=0
|
||||
printf '%s' "$left"
|
||||
}
|
||||
|
||||
# Run one phase script under remaining RUN_TIMEOUT budget.
|
||||
# Exit 124 (timeout utility) → mark RUN_TIMED_OUT.
|
||||
run_phase() {
|
||||
local phase="$1" script="$2" left rc
|
||||
left=$(mon_seconds_left)
|
||||
if [ "${RUN_TIMEOUT:-0}" -gt 0 ] && [ "$left" -le 0 ]; then
|
||||
RUN_TIMED_OUT=1
|
||||
RUN_TIMEOUT_AT_PHASE="${RUN_TIMEOUT_AT_PHASE:-$phase}"
|
||||
return 1
|
||||
fi
|
||||
if [ "${RUN_TIMEOUT:-0}" -eq 0 ]; then
|
||||
"$script"
|
||||
return $?
|
||||
fi
|
||||
# Keep at least 2s so timeout(1) can start the child.
|
||||
[ "$left" -lt 2 ] && left=2
|
||||
set +e
|
||||
with_timeout "$left" "$script"
|
||||
rc=$?
|
||||
set -e
|
||||
if [ "$rc" -eq 124 ]; then
|
||||
RUN_TIMED_OUT=1
|
||||
RUN_TIMEOUT_AT_PHASE="$phase"
|
||||
return 1
|
||||
fi
|
||||
return "$rc"
|
||||
}
|
||||
|
||||
# How inside/versions will reach containers (host-podman vs ssh)
|
||||
_INSIDE_ACCESS_HINT=ssh
|
||||
if [ "${INSIDE_PODMAN:-0}" = "1" ] || [ "${INSIDE_MODE:-}" = "local-podman" ]; then
|
||||
_INSIDE_ACCESS_HINT=host-podman
|
||||
elif [ "${SKIP_SSH:-0}" = "1" ]; then
|
||||
_INSIDE_ACCESS_HINT=skipped
|
||||
elif [ "${LOCAL_STACK:-0}" = "1" ] && command -v podman >/dev/null 2>&1 \
|
||||
&& podman ps --format '{{.Names}}' 2>/dev/null | grep -qE 'taler-hacktivism'; then
|
||||
_INSIDE_ACCESS_HINT=host-podman
|
||||
fi
|
||||
|
||||
printf 'target domain=%s\n' "${TALER_DOMAIN}"
|
||||
printf ' bank %s\n' "$BANK_PUBLIC"
|
||||
printf ' exchange %s\n' "$EXCHANGE_PUBLIC"
|
||||
printf ' merchant %s\n' "$MERCHANT_PUBLIC"
|
||||
printf ' currency expect=%s skip_ssh=%s\n' "${EXPECT_CURRENCY:-any}" "$SKIP_SSH"
|
||||
printf ' currency expect=%s\n' "${EXPECT_CURRENCY:-any}"
|
||||
printf ' phases %s\n' "${PHASES[*]}"
|
||||
printf ' flags LOCAL_STACK=%s SKIP_SSH=%s INSIDE_PODMAN=%s INSIDE_MODE=%s\n' \
|
||||
"${LOCAL_STACK:-}" "${SKIP_SSH:-0}" "${INSIDE_PODMAN:-0}" "${INSIDE_MODE:-}"
|
||||
printf ' flags KOOPA_SSH=%s INSIDE_SSH=%s INSIDE_PROFILE=%s\n' \
|
||||
"${KOOPA_SSH:-}" "${INSIDE_SSH:-}" "${INSIDE_PROFILE:-}"
|
||||
printf ' access inside/versions → %s' "$_INSIDE_ACCESS_HINT"
|
||||
case "$_INSIDE_ACCESS_HINT" in
|
||||
host-podman) printf ' (podman exec on this host; IDs inside.host-*)\n' ;;
|
||||
ssh) printf ' (SSH then podman; IDs inside.ssh-*)\n' ;;
|
||||
skipped) printf ' (SKIP_SSH=1)\n' ;;
|
||||
*) printf '\n' ;;
|
||||
esac
|
||||
if [ "${RUN_TIMEOUT:-0}" -eq 0 ]; then
|
||||
printf ' run_timeout unlimited (RUN_TIMEOUT=0)\n'
|
||||
else
|
||||
printf ' run_timeout %ss wall clock (RUN_TIMEOUT=; 0=unlimited)\n' "$RUN_TIMEOUT"
|
||||
fi
|
||||
printf ' progress total≈%s (set PROGRESS_TOTAL= to override; PROGRESS_OFF=1 to hide)\n' "${PROGRESS_TOTAL:-0}"
|
||||
unset _INSIDE_ACCESS_HINT
|
||||
|
||||
chmod +x "$ROOT"/check_*.sh 2>/dev/null || true
|
||||
|
||||
ec=0
|
||||
for p in "${PHASES[@]}"; do
|
||||
if [ "$RUN_TIMED_OUT" = "1" ]; then
|
||||
RUN_SKIPPED_PHASES+=("$p")
|
||||
continue
|
||||
fi
|
||||
left=$(mon_seconds_left)
|
||||
if [ "${RUN_TIMEOUT:-0}" -gt 0 ] && [ "$left" -le 0 ]; then
|
||||
RUN_TIMED_OUT=1
|
||||
RUN_TIMEOUT_AT_PHASE="${RUN_TIMEOUT_AT_PHASE:-$p}"
|
||||
RUN_SKIPPED_PHASES+=("$p")
|
||||
continue
|
||||
fi
|
||||
case "$p" in
|
||||
urls) "$ROOT/check_urls.sh" || ec=1 ;;
|
||||
inside) "$ROOT/check_inside.sh" || ec=1 ;;
|
||||
versions) "$ROOT/check_versions.sh" || ec=1 ;;
|
||||
sanity) "$ROOT/check_sanity.sh" || ec=1 ;;
|
||||
server) "$ROOT/check_server.sh" || ec=1 ;;
|
||||
e2e) "$ROOT/check_e2e.sh" || ec=1 ;;
|
||||
ladder|goa-ladder) "$ROOT/check_goa_ladder.sh" || ec=1 ;;
|
||||
auth401) "$ROOT/check_auth401.sh" || ec=1 ;;
|
||||
urls) run_phase urls "$ROOT/check_urls.sh" || ec=1 ;;
|
||||
inside) run_phase inside "$ROOT/check_inside.sh" || ec=1 ;;
|
||||
versions) run_phase versions "$ROOT/check_versions.sh" || ec=1 ;;
|
||||
sanity) run_phase sanity "$ROOT/check_sanity.sh" || ec=1 ;;
|
||||
server) run_phase server "$ROOT/check_server.sh" || ec=1 ;;
|
||||
e2e) run_phase e2e "$ROOT/check_e2e.sh" || ec=1 ;;
|
||||
ladder|goa-ladder) run_phase ladder "$ROOT/check_goa_ladder.sh" || ec=1 ;;
|
||||
auth401) run_phase auth401 "$ROOT/check_auth401.sh" || ec=1 ;;
|
||||
aptdeploy|apt-deploy|apt_src) run_phase aptdeploy "$ROOT/check_apt_deploy.sh" || ec=1 ;;
|
||||
surface|ecosystem) run_phase surface "$ROOT/check_surface.sh" || ec=1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Extraordinary run-budget failure: always report at end; HTML links top → here.
|
||||
if [ "$RUN_TIMED_OUT" = "1" ]; then
|
||||
ec=1
|
||||
elapsed=$(( $(date +%s) - MON_T0 ))
|
||||
skipped="${RUN_SKIPPED_PHASES[*]:-}"
|
||||
# Stable id for jump links (console HTML + err-top banner)
|
||||
printf '\n'
|
||||
printf '╔══════════════════════════════════════════════════════════╗\n'
|
||||
printf '║ RUN TIMEOUT · extraordinary (see jump target below) ║\n'
|
||||
printf '╚══════════════════════════════════════════════════════════╝\n'
|
||||
printf '┌ ERROR ┐ #run.timeout-01 RUN_TIMEOUT exceeded · budget %ss · elapsed %ss\n' \
|
||||
"$RUN_TIMEOUT" "$elapsed"
|
||||
printf ' detail: wall-clock limit hit'
|
||||
if [ -n "${RUN_TIMEOUT_AT_PHASE:-}" ]; then
|
||||
printf ' during/after phase "%s"' "$RUN_TIMEOUT_AT_PHASE"
|
||||
fi
|
||||
if [ -n "$skipped" ]; then
|
||||
printf ' · skipped: %s' "$skipped"
|
||||
fi
|
||||
printf '\n'
|
||||
printf ' id=run.timeout-01\n'
|
||||
printf ' hint: raise RUN_TIMEOUT= (seconds) or set RUN_TIMEOUT=0 for unlimited\n'
|
||||
printf -- '┌ RUN TIMEOUT · extraordinary ┐\n'
|
||||
printf -- ' • run.timeout-01 [run] RUN_TIMEOUT=%ss exceeded (elapsed %ss)%s%s\n' \
|
||||
"$RUN_TIMEOUT" "$elapsed" \
|
||||
"${RUN_TIMEOUT_AT_PHASE:+ · phase $RUN_TIMEOUT_AT_PHASE}" \
|
||||
"${skipped:+ · skipped $skipped}"
|
||||
printf -- '--- ERRORS (failed checks) ---\n'
|
||||
printf -- ' • run.timeout-01 [run] RUN_TIMEOUT=%ss exceeded (elapsed %ss)\n' \
|
||||
"$RUN_TIMEOUT" "$elapsed"
|
||||
fi
|
||||
|
||||
exit "$ec"
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue