taler-monitoring/host-agent/README.md

# host-agent — reporting generation (GOA + FP + any stack)

**Global pipeline** lives in **`run-host-report.sh`** (all stacks share it):

| Behaviour | Default |
|-----------|---------|
| Suite refresh | **`update-suite.sh` every run** → `git fetch --tags` + hard reset to `origin/main` |
| Update strict | `SUITE_UPDATE_STRICT=1` (default): **abort** if fetch/reset fails — no silent old tree |
| Wall clock | `RUN_TIMEOUT=600` |
| Log | line-buffered `run-*.log` (continuous write) |
| HTML | always (first run / empty tree too) + commit URL; **bootstrap install stubs are monpages ERROR** |
| Timeout notice | end of log + top jump on `/monitoring_err/` |

**Rule:** public mon pages must show a **current suite tag/commit**, not install bootstrap or months-old trees. Timers exist so this keeps working without manual SSH.

Thin wrappers only set stack defaults:

| Wrapper | Stack | Typical user |
|---------|-------|----------------|
| `run-hacktivism-monitoring.sh` | GOA `hacktivism.ch` | `hernani` @ koopa |
| `run-fp-stage-monitoring.sh` | FP stage | `stagepaysan` |
| `run-fp-prod-monitoring.sh` | FP prod | `francpaysan` |
| `run-host-report.sh` + `taler-monitoring-mytops-stage.{service,timer}` | mytops **stage / betel** only | `taler-monitoring` @ betel (4h) |
| `run-aptdeploy.sh` | CLI: ensure + **aptdeploy** (no HTML) | local if hostname=koopa, else **ssh $DEPLOY_SSH** |
| `run-aptdeploy-monitoring.sh` | apt-src tests → `/taler-monitoring-aptdeploy*` | `hernani` @ koopa (4h) |
| `run-surface-monitoring.sh` | **surface** → only `/taler-monitoring-surface*` (ecosystem + mail + mattermost + versions) | `hernani` @ koopa (hourly) |

Deprecated mail/mattermost wrappers: **`../meta/deprecated/`** (not installed).

### apt-src deploy tests (hacktivism `/monitoring`)

| Container | Mode |
|-----------|------|
| `koopa-taler-deploy-test-apt-src-trixie` | **fresh**: recreate when repo package fingerprint changes |
| `koopa-taler-deploy-test-apt-src-trixie-testing` | **fresh** for `trixie-testing` |
| `koopa-taler-deploy-test-apt-src-trixie-upgrade` | **upgrade**: initial install, then mytops-style `apt install` + `upgrade` |
| `koopa-taler-deploy-test-apt-src-trixie-testing-upgrade` | **upgrade** for testing suite |

```bash
# from ops workstation → triggers on koopa-external
./host-agent/run-aptdeploy.sh
./host-agent/run-aptdeploy.sh upgrade
./host-agent/run-aptdeploy.sh fresh
./host-agent/run-aptdeploy.sh check-only

# on koopa
./taler-monitoring.sh -d hacktivism.ch aptdeploy
```

Checks: package table, `httpd --version`, libtalerutil/`ldd` on failure, `systemctl` for
`taler-merchant.target`, basic local `/config` probe. No nginx/HTTPS in containers.

### Public HTML layout (v1.8.0 simplified)

| URL | Role |
|-----|------|
| `https://{bank,exchange,merchant}/monitoring/` (+ `_err`) | **Landing-stack** report — each of the **9** landing fronts (GOA 3 + FP stage 3 + FP prod 3) |
| `https://taler.hacktivism.ch/taler-monitoring-surface/` (+ `_err`) | **Only** ecosystem page: software/versions in the Taler universe (nmap OS, mail firefly+anastasis, mattermost, …) |

No separate public pages for mail/mattermost. Staging under `~/monitoring-sites-staging/`.

```bash
systemctl --user enable --now taler-monitoring-surface.timer
systemctl --user start taler-monitoring-surface.service
```

Env examples: **`env/*.env.example`**. FP: `run-fp-prod-monitoring.sh` / `run-fp-stage-monitoring.sh` (same `run-host-report.sh` pipeline).

## GOA (koopa) — host-podman

Runs **as user `hernani`** (systemd user, linger). **Primary** mode:

- **`INSIDE_PODMAN=1`** / **`INSIDE_MODE=local-podman`**: access mode **`host-podman`** —  
  `podman exec` on this host into  
  `taler-hacktivism-bank` / `taler-hacktivism-exchange-ansible` / `taler-hacktivism`  
  (IDs `inside.host-*`, not `inside.ssh-*`).  
  From an **ops workstation**, leave `INSIDE_PODMAN` off → access mode **`ssh`** via `KOOPA_SSH`  
  (`inside.ssh-*`).
- Public **`urls` + `versions`** for the stack (`-d hacktivism.ch`).
- Writes console HTML for all three sites under  
  `~/monitoring-sites-staging/{bank,exchange,taler}.hacktivism.ch/`.

Caddy (host root) can serve that tree at `/monitoring` — you configure Caddy.

## Latest software + config isolation

**Source of truth is always Forgejo** (`git.hacktivism.ch`), not a laptop rsync copy.

Every host-agent / aptdeploy run:

1. **`update-suite.sh`**: `git fetch` + **`SUITE_UPDATE_MODE=reset`** (default) → `origin/main`  
   into `SUITE_DIR` (prefer `~/src/taler-monitoring`).
2. **Re-exec** host-agent scripts from that tree so the running code matches `COMMIT_URL`.
3. Local overrides only in **`~/.config/taler-monitoring/env`** — never overwritten by git.
4. HTML footer = exact commit:  
   `https://git.hacktivism.ch/hernani/taler-monitoring/src/commit/<full-sha>`
5. **`RUN_TIMEOUT=600`**, line-buffered logs, HTML always written.

If fetch fails → previous tree + WARN.  
**To ship features (aptdeploy, surface, …): push to `origin/main`.** Local-only edits vanish on the next reset.

If pull fails (no network / auth), the previous tree is used and a WARN is logged.

## Install (on koopa as hernani)

```bash
# once: local config (optional)
mkdir -p ~/.config/taler-monitoring
cp ~/src/taler-monitoring/host-agent/env.example \
   ~/.config/taler-monitoring/env

# ops workstation:
./host-agent/install-host-agent.sh --remote $DEPLOY_SSH

# or on koopa:
~/src/taler-monitoring/host-agent/install-host-agent.sh
```

Units:

| Unit | Role |
|------|------|
| `taler-monitoring-hacktivism.path` | fires on software stamp (`touch-software-stamp.sh`) |
| `taler-monitoring-hacktivism.timer` | every **4h** fallback |
| `taler-monitoring-hacktivism.service` | oneshot → `/monitoring*` (urls inside versions) |
| `taler-monitoring-surface.timer` | every **1h** |
| `taler-monitoring-surface.service` | → `/taler-monitoring-surface*` (taler only) |
| `taler-monitoring-aptdeploy.timer` | every **4h** |
| `taler-monitoring-aptdeploy.service` | → `/taler-monitoring-aptdeploy*` (taler only) |

## After package / image / landing upgrades

```bash
~/src/taler-monitoring/host-agent/touch-software-stamp.sh
# → path unit starts the service
```

## Manual

```bash
systemctl --user start taler-monitoring-hacktivism.service
systemctl --user status taler-monitoring-hacktivism.service --no-pager
journalctl --user -u taler-monitoring-hacktivism.service -n 50 --no-pager
ls ~/monitoring-sites-staging/bank.hacktivism.ch/monitoring_err/
```

## Relation to optional outside-only timer

| | host-agent (koopa) | outside-runner site-gen |
|--|--------------------|-------------------|
| Access | **podman on host** | public HTTPS only |
| Phases | urls **inside** versions | urls versions |
| Trigger | software path + 4h | *could* launchd 4h |
| Primary for GOA | **yes** | **off** (optional only) |

**outside-runner is not scheduled** right now (launchd disabled). Scripts under
`site-gen/` can still one-shot a public run if needed; see `site-gen/README.md`.

No secrets required for default phases. `secrets.env` optional for e2e later.

## Git commit on HTML pages

Header of each generated page includes a link like:

`https://git.hacktivism.ch/hernani/taler-monitoring/src/commit/<full-sha>`

That SHA is the tree **after** `update-suite.sh` for that run.

## Packages on koopa

See **[../DEPENDENCIES.md](../DEPENDENCIES.md)**.

**Default host-agent needs no new packages** if curl/python3/podman/git are present (they are).

**Root once** only for full QR checks in `urls`:

```bash
zypper in qrencode zbar
```

Without them, monitoring still runs; QR group WARNs and skips encode/decode.

Caddy `/var/www/monitoring-sites` is separate (you configured hernani write access).


## Exit codes (v1.7.1+)

Host-agent jobs default to **`STRICT_EXIT=1`**: non-zero exit if suite checks fail, `DEPLOY_WWW_ROOT` is not writable, or public monpages (FQDN) fail.

Set `STRICT_EXIT=0` only if you want timers to stay green while still logging errors (not recommended for production).

## Monitoring pages (suite)

Public sticky-bar HTML under `/monitoring*` and `/taler-monitoring-*` is part of **taler-monitoring** (not a separate product). See main [README — Monitoring pages](../README.md#monitoring-pages-part-of-the-suite). Phase **monpages** is obligatory (**ERROR** if missing): GOA checks the full suite inventory; FP only FP mon hosts.