taler-monitoring/host-agent

host-agent — reporting generation (GOA + FP + any stack)

Global pipeline lives in run-host-report.sh (all stacks share it):

Behaviour	Default
Suite refresh	update-suite.sh every run → git fetch --tags + hard reset to origin/main
Update strict	SUITE_UPDATE_STRICT=1 (default): abort if fetch/reset fails — no silent old tree
Wall clock	RUN_TIMEOUT=600
Log	line-buffered run-*.log (continuous write)
HTML	always (first run / empty tree too) + commit URL; bootstrap install stubs are monpages ERROR
Timeout notice	end of log + top jump on /monitoring_err/

Rule: public mon pages must show a current suite tag/commit, not install bootstrap or months-old trees. Timers exist so this keeps working without manual SSH.

Thin wrappers only set stack defaults:

Wrapper	Stack	Typical user
run-hacktivism-monitoring.sh	GOA hacktivism.ch	hernani @ koopa
run-fp-stage-monitoring.sh	FP stage	stagepaysan
run-fp-prod-monitoring.sh	FP prod	francpaysan
run-host-report.sh + taler-monitoring-mytops-stage.{service,timer}	mytops stage / betel only	taler-monitoring @ betel (4h)
run-aptdeploy.sh	CLI: ensure + aptdeploy (no HTML)	local if hostname=koopa, else ssh koopa-external
run-aptdeploy-monitoring.sh	apt-src tests → /taler-monitoring-aptdeploy*	hernani @ koopa (4h)
run-surface-monitoring.sh	surface → only /taler-monitoring-surface* (ecosystem + mail + mattermost + versions)	hernani @ koopa (hourly)
run-mail-monitoring.sh / run-mattermost-monitoring.sh	deprecated v1.8.0 — aliases of surface	disabled timers

apt-src deploy tests (hacktivism /monitoring)

Container	Mode
koopa-taler-deploy-test-apt-src-trixie	fresh: recreate when repo package fingerprint changes
koopa-taler-deploy-test-apt-src-trixie-testing	fresh for trixie-testing
koopa-taler-deploy-test-apt-src-trixie-upgrade	upgrade: initial install, then mytops-style apt install + upgrade
koopa-taler-deploy-test-apt-src-trixie-testing-upgrade	upgrade for testing suite

# from ops workstation → triggers on koopa-external
./host-agent/run-aptdeploy.sh
./host-agent/run-aptdeploy.sh upgrade
./host-agent/run-aptdeploy.sh fresh
./host-agent/run-aptdeploy.sh check-only

# on koopa
./taler-monitoring.sh -d hacktivism.ch aptdeploy

Checks: package table, httpd --version, libtalerutil/ldd on failure, systemctl for taler-merchant.target, basic local /config probe. No nginx/HTTPS in containers.

Public HTML layout (v1.8.0 simplified)

URL	Role
https://{bank,exchange,merchant}/monitoring/ (+ _err)	Landing-stack report — each of the 9 landing fronts (GOA 3 + FP stage 3 + FP prod 3)
https://taler.hacktivism.ch/taler-monitoring-surface/ (+ _err)	Only ecosystem page: software/versions in the Taler universe (nmap OS, mail firefly+anastasis, mattermost, …)

No separate public pages for mail/mattermost. Staging under ~/monitoring-sites-staging/.

systemctl --user enable --now taler-monitoring-surface.timer
systemctl --user start taler-monitoring-surface.service
# mail/mattermost timers stay disabled (v1.8.0)

Env examples: env/*.env.example. FP: run-fp-prod-monitoring.sh / run-fp-stage-monitoring.sh (same run-host-report.sh pipeline).

GOA (koopa) — host-podman

Runs as user hernani (systemd user, linger). Primary mode:

• INSIDE_PODMAN=1 / INSIDE_MODE=local-podman: access mode host-podman —
  podman exec on this host into
  taler-hacktivism-bank / taler-hacktivism-exchange-ansible / taler-hacktivism
  (IDs inside.host-*, not inside.ssh-*).
  From an ops workstation, leave INSIDE_PODMAN off → access mode ssh via KOOPA_SSH
  (inside.ssh-*).
• Public urls + versions for the stack (-d hacktivism.ch).
• Writes console HTML for all three sites under
  ~/monitoring-sites-staging/{bank,exchange,taler}.hacktivism.ch/.

Caddy (host root) can serve that tree at /monitoring — you configure Caddy.

Latest software + config isolation

Source of truth is always Forgejo (git.hacktivism.ch), not a laptop rsync copy.

Every host-agent / aptdeploy run:

1. update-suite.sh: git fetch + SUITE_UPDATE_MODE=reset (default) → origin/main
   into SUITE_DIR (prefer ~/src/taler-monitoring).
2. Re-exec host-agent scripts from that tree so the running code matches COMMIT_URL.
3. Local overrides only in ~/.config/taler-monitoring/env — never overwritten by git.
4. HTML footer = exact commit:
   https://git.hacktivism.ch/hernani/taler-monitoring/src/commit/<full-sha>
5. RUN_TIMEOUT=600, line-buffered logs, HTML always written.

If fetch fails → previous tree + WARN.
To ship features (aptdeploy, surface, …): push to origin/main. Local-only edits vanish on the next reset.

If pull fails (no network / auth), the previous tree is used and a WARN is logged.

Install (on koopa as hernani)

# once: local config (optional)
mkdir -p ~/.config/taler-monitoring
cp ~/src/taler-monitoring/host-agent/env.example \
   ~/.config/taler-monitoring/env

# ops workstation:
./host-agent/install-host-agent.sh --remote koopa-external

# or on koopa:
~/src/taler-monitoring/host-agent/install-host-agent.sh

Units:

Unit	Role
taler-monitoring-hacktivism.path	fires on software stamp (touch-software-stamp.sh)
taler-monitoring-hacktivism.timer	every 4h fallback
taler-monitoring-hacktivism.service	oneshot → /monitoring* (urls inside versions)
taler-monitoring-surface.timer	every 1h
taler-monitoring-surface.service	→ /taler-monitoring-surface* (taler only)
taler-monitoring-mattermost.* / mail.*	disabled (v1.8.0+); content via surface job
taler-monitoring-aptdeploy.timer	every 4h
taler-monitoring-aptdeploy.service	→ /taler-monitoring-aptdeploy* (taler only)

After package / image / landing upgrades

~/src/taler-monitoring/host-agent/touch-software-stamp.sh
# → path unit starts the service

Manual

systemctl --user start taler-monitoring-hacktivism.service
systemctl --user status taler-monitoring-hacktivism.service --no-pager
journalctl --user -u taler-monitoring-hacktivism.service -n 50 --no-pager
ls ~/monitoring-sites-staging/bank.hacktivism.ch/monitoring_err/

Relation to optional outside-only timer

	host-agent (koopa)	outside-runner site-gen
Access	podman on host	public HTTPS only
Phases	urls inside versions	urls versions
Trigger	software path + 4h	could launchd 4h
Primary for GOA	yes	off (optional only)

outside-runner is not scheduled right now (launchd disabled). Scripts under site-gen/ can still one-shot a public run if needed; see site-gen/README.md.

No secrets required for default phases. secrets.env optional for e2e later.

Git commit on HTML pages

Header of each generated page includes a link like:

https://git.hacktivism.ch/hernani/taler-monitoring/src/commit/<full-sha>

That SHA is the tree after update-suite.sh for that run.

Packages on koopa

See ../DEPENDENCIES.md.

Default host-agent needs no new packages if curl/python3/podman/git are present (they are).

Root once only for full QR checks in urls:

zypper in qrencode zbar

Without them, monitoring still runs; QR group WARNs and skips encode/decode.

Caddy /var/www/monitoring-sites is separate (you configured hernani write access).

Exit codes (v1.7.1+)

Host-agent jobs default to STRICT_EXIT=1: non-zero exit if suite checks fail, DEPLOY_WWW_ROOT is not writable, or public monpages (FQDN) fail.

Set STRICT_EXIT=0 only if you want timers to stay green while still logging errors (not recommended for production).

Monitoring pages (suite)

Public sticky-bar HTML under /monitoring* and /taler-monitoring-* is part of taler-monitoring (not a separate product). See main README — Monitoring pages. Phase monpages is obligatory (ERROR if missing): GOA checks the full suite inventory; FP only FP mon hosts.