taler-monitoring/host-agent/README.md

host-agent — reporting generation (GOA + FP + any stack)

Global pipeline lives in run-host-report.sh (all stacks share it):

Behaviour	Default
Suite refresh	update-suite.sh → origin/main
Wall clock	RUN_TIMEOUT=600
Log	line-buffered run-*.log (continuous write)
HTML	always (first run / empty tree too) + commit URL
Timeout notice	end of log + top jump on /monitoring_err/

Thin wrappers only set stack defaults:

Wrapper	Stack	Typical user
run-hacktivism-monitoring.sh	GOA hacktivism.ch	hernani @ koopa
run-fp-stage-monitoring.sh	FP stage	stagepaysan
run-fp-prod-monitoring.sh	FP prod	francpaysan
run-aptdeploy.sh	CLI: ensure + aptdeploy (no HTML)	local if hostname=koopa, else ssh koopa-external
run-aptdeploy-monitoring.sh	aptdeploy → /taler-monitoring-aptdeploy*	hernani @ koopa (4h timer, taler only)
run-surface-monitoring.sh	surface → /taler-monitoring-surface*	hernani @ koopa (hourly, taler only)

apt-src deploy tests (hacktivism /monitoring)

Container	Mode
koopa-taler-deploy-test-apt-src-trixie	fresh: recreate when repo package fingerprint changes
koopa-taler-deploy-test-apt-src-trixie-testing	fresh for trixie-testing
koopa-taler-deploy-test-apt-src-trixie-upgrade	upgrade: initial install, then mytops-style apt install + upgrade
koopa-taler-deploy-test-apt-src-trixie-testing-upgrade	upgrade for testing suite

# from ops workstation → triggers on koopa-external
./host-agent/run-aptdeploy.sh
./host-agent/run-aptdeploy.sh upgrade
./host-agent/run-aptdeploy.sh fresh
./host-agent/run-aptdeploy.sh check-only

# on koopa
./taler-monitoring.sh -d hacktivism.ch aptdeploy

Checks: package table, httpd --version, libtalerutil/ldd on failure, systemctl for taler-merchant.target, basic local /config probe. No nginx/HTTPS in containers.

surface + aptdeploy HTML (taler.hacktivism.ch only)

URL	Role	Timer
…/taler-monitoring-surface/	ecosystem surface scan	1h
…/taler-monitoring-surface_err/	errors + jump list	″
…/taler-monitoring-aptdeploy/	apt-src container deploy tests	4h
…/taler-monitoring-aptdeploy_err/	errors + jump list	″

Not on bank/exchange. Staging: ~/monitoring-sites-staging/taler.hacktivism.ch/taler-monitoring-{surface,aptdeploy}*.

systemctl --user enable --now taler-monitoring-surface.timer
systemctl --user enable --now taler-monitoring-aptdeploy.timer
systemctl --user start taler-monitoring-surface.service
systemctl --user start taler-monitoring-aptdeploy.service
# Caddy: handles only in taler.hacktivism.ch block (root)

Env examples: env/*.env.example. FP: run-fp-prod-monitoring.sh / run-fp-stage-monitoring.sh (same run-host-report.sh pipeline).

GOA (koopa) — host-podman

Runs as user hernani (systemd user, linger). Primary mode:

• INSIDE_PODMAN=1 / INSIDE_MODE=local-podman: access mode host-podman —
  podman exec on this host into
  taler-hacktivism-bank / taler-hacktivism-exchange-ansible / taler-hacktivism
  (IDs inside.host-*, not inside.ssh-*).
  From an ops workstation, leave INSIDE_PODMAN off → access mode ssh via KOOPA_SSH
  (inside.ssh-*).
• Public urls + versions for the stack (-d hacktivism.ch).
• Writes console HTML for all three sites under
  ~/monitoring-sites-staging/{bank,exchange,taler}.hacktivism.ch/.

Caddy (host root) can serve that tree at /monitoring — you configure Caddy.

Latest software + config isolation

Source of truth is always Forgejo (git.hacktivism.ch), not a laptop rsync copy.

Every host-agent / aptdeploy run:

1. update-suite.sh: git fetch + SUITE_UPDATE_MODE=reset (default) → origin/main
   into SUITE_DIR (prefer ~/src/taler-monitoring).
2. Re-exec host-agent scripts from that tree so the running code matches COMMIT_URL.
3. Local overrides only in ~/.config/taler-monitoring/env — never overwritten by git.
4. HTML footer = exact commit:
   https://git.hacktivism.ch/hernani/taler-monitoring/src/commit/<full-sha>
5. RUN_TIMEOUT=600, line-buffered logs, HTML always written.

If fetch fails → previous tree + WARN.
To ship features (aptdeploy, surface, …): push to origin/main. Local-only edits vanish on the next reset.

If pull fails (no network / auth), the previous tree is used and a WARN is logged.

Install (on koopa as hernani)

# once: local config (optional)
mkdir -p ~/.config/taler-monitoring
cp ~/src/taler-monitoring/host-agent/env.example \
   ~/.config/taler-monitoring/env

# ops workstation:
./host-agent/install-host-agent.sh --remote koopa-external

# or on koopa:
~/src/taler-monitoring/host-agent/install-host-agent.sh

Units:

Unit	Role
taler-monitoring-hacktivism.path	fires on software stamp (touch-software-stamp.sh)
taler-monitoring-hacktivism.timer	every 4h fallback
taler-monitoring-hacktivism.service	oneshot → /monitoring* (urls inside versions)
taler-monitoring-surface.timer	every 1h
taler-monitoring-surface.service	→ /taler-monitoring-surface* (taler only)
taler-monitoring-mattermost.timer	every 1h
taler-monitoring-mattermost.service	→ /taler-monitoring-mattermost* (mattermost.taler.net)
taler-monitoring-mail.timer	every 4h
taler-monitoring-mail.service	→ /taler-monitoring-mail* (firefly + pixel MX/SMTP/IMAP)
taler-monitoring-aptdeploy.timer	every 4h
taler-monitoring-aptdeploy.service	→ /taler-monitoring-aptdeploy* (taler only)

After package / image / landing upgrades

~/src/taler-monitoring/host-agent/touch-software-stamp.sh
# → path unit starts the service

Manual

systemctl --user start taler-monitoring-hacktivism.service
systemctl --user status taler-monitoring-hacktivism.service --no-pager
journalctl --user -u taler-monitoring-hacktivism.service -n 50 --no-pager
ls ~/monitoring-sites-staging/bank.hacktivism.ch/monitoring_err/

Relation to optional outside-only timer

	host-agent (koopa)	outside-runner site-gen
Access	podman on host	public HTTPS only
Phases	urls inside versions	urls versions
Trigger	software path + 4h	could launchd 4h
Primary for GOA	yes	off (optional only)

outside-runner is not scheduled right now (launchd disabled). Scripts under site-gen/ can still one-shot a public run if needed; see site-gen/README.md.

No secrets required for default phases. secrets.env optional for e2e later.

Git commit on HTML pages

Header of each generated page includes a link like:

https://git.hacktivism.ch/hernani/taler-monitoring/src/commit/<full-sha>

That SHA is the tree after update-suite.sh for that run.

Packages on koopa

See ../DEPENDENCIES.md.

Default host-agent needs no new packages if curl/python3/podman/git are present (they are).

Root once only for full QR checks in urls:

zypper in qrencode zbar

Without them, monitoring still runs; QR group WARNs and skips encode/decode.

Caddy /var/www/monitoring-sites is separate (you configured hernani write access).

Exit codes (v1.7.1+)

Host-agent jobs default to STRICT_EXIT=1: non-zero exit if suite checks fail, DEPLOY_WWW_ROOT is not writable, or public monpages (FQDN) fail.

Set STRICT_EXIT=0 only if you want timers to stay green while still logging errors (not recommended for production).