release 1.13.7: optional and meta scripts under meta/

Move deprecated mail/mattermost host-agent units, firecuda outside-runner
helpers, and legacy multi-host site-gen into meta/. Production stays
host-agent + site-gen/console_to_html.py; install no longer ships mail/mm timers.
This commit is contained in:
Hernâni Marques 2026-07-19 03:36:12 +02:00
parent 0c94fb3d2e
commit 2a20def806
No known key found for this signature in database
GPG key ID: CB5738652768F7E9
20 changed files with 89 additions and 121 deletions

View file

@ -62,6 +62,8 @@ IDs reset per **group** inside an area — so “too many www tests” become
`www.exchange-*`, `www.bank-*`, `www.landing-*`, etc.
Catalog: **[TESTS.md](./TESTS.md)**.
Git / tags / suite tree: **[VERSIONS.md](./VERSIONS.md)**.
Optional / deprecated tooling: **[meta/](./meta/)** (firecuda, legacy site-gen, old mail/mm units).
GUI Android notes: **[android-test/GUI-AUTOMATION-NOTES.md](./android-test/GUI-AUTOMATION-NOTES.md)**.
CLI / wallet-cli recurring issues: **[CLI-AUTOMATION-NOTES.md](./CLI-AUTOMATION-NOTES.md)**.

View file

@ -1 +1 @@
1.13.6
1.13.7

View file

@ -17,6 +17,7 @@ Git tags: `vMAJOR.FEATURE.FIX` (e.g. `v1.8.0`). File `VERSION` omits the `v` pre
| Tag | Date (UTC) | Notes |
|-----|------------|--------|
| **v1.13.7** | 2026-07-19 | **Bugfix:** move optional/meta scripts under `meta/` (deprecated mail/mm units, firecuda outside-runner, legacy multi-host site-gen); production path stays host-agent + `site-gen/console_to_html.py`. |
| **v1.13.6** | 2026-07-19 | **Bugfix:** machine paths and SSH hosts only via env (`~/.config/taler-monitoring/env` / `taler-monitoring-env`); suite loads env before defaults; no `/home/hernani` or bogus host hardcodes. |
| **v1.13.5** | 2026-07-19 | **Bugfix:** progress bar — env-aware phase estimates (CHECK_BANK/LANDING, mon hosts); re-fit total after each phase (done+remaining); drop fake +12 for host-agent monpages post-check (fixes stage 42/139→46/46 snap). |
| **v1.13.4** | 2026-07-19 | **Bugfix:** monpages race-proof — atomic HTML write; rsync `--delay-updates` + settle; fetch retries; larger head windows; pre-publish soft content (`MONPAGES_PRE_PUBLISH`); bootstrap markers. |

View file

@ -24,7 +24,8 @@ Thin wrappers only set stack defaults:
| `run-aptdeploy.sh` | CLI: ensure + **aptdeploy** (no HTML) | local if hostname=koopa, else **ssh $DEPLOY_SSH** |
| `run-aptdeploy-monitoring.sh` | apt-src tests → `/taler-monitoring-aptdeploy*` | `hernani` @ koopa (4h) |
| `run-surface-monitoring.sh` | **surface** → only `/taler-monitoring-surface*` (ecosystem + mail + mattermost + versions) | `hernani` @ koopa (hourly) |
| `run-mail-monitoring.sh` / `run-mattermost-monitoring.sh` | **deprecated v1.8.0** — aliases of surface | disabled timers |
Deprecated mail/mattermost wrappers: **`../meta/deprecated/`** (not installed).
### apt-src deploy tests (hacktivism `/monitoring`)
@ -61,7 +62,6 @@ No separate public pages for mail/mattermost. Staging under `~/monitoring-sites-
```bash
systemctl --user enable --now taler-monitoring-surface.timer
systemctl --user start taler-monitoring-surface.service
# mail/mattermost timers stay disabled (v1.8.0)
```
Env examples: **`env/*.env.example`**. FP: `run-fp-prod-monitoring.sh` / `run-fp-stage-monitoring.sh` (same `run-host-report.sh` pipeline).
@ -125,7 +125,6 @@ Units:
| `taler-monitoring-hacktivism.service` | oneshot → `/monitoring*` (urls inside versions) |
| `taler-monitoring-surface.timer` | every **1h** |
| `taler-monitoring-surface.service` | → `/taler-monitoring-surface*` (taler only) |
| `taler-monitoring-mattermost.*` / `mail.*` | **disabled** (v1.8.0+); content via surface job |
| `taler-monitoring-aptdeploy.timer` | every **4h** |
| `taler-monitoring-aptdeploy.service` | → `/taler-monitoring-aptdeploy*` (taler only) |

View file

@ -63,20 +63,19 @@ install_local() {
# optional aptdeploy (not a public mon page layout — v1.8.0 simplified overview)
install -m 644 "$ROOT/taler-monitoring-aptdeploy.service" "$home_unit/"
install -m 644 "$ROOT/taler-monitoring-aptdeploy.timer" "$home_unit/"
# units still installed for migration; timers disabled (folded into surface)
install -m 644 "$ROOT/taler-monitoring-mattermost.service" "$home_unit/"
install -m 644 "$ROOT/taler-monitoring-mattermost.timer" "$home_unit/"
install -m 644 "$ROOT/taler-monitoring-mail.service" "$home_unit/"
install -m 644 "$ROOT/taler-monitoring-mail.timer" "$home_unit/"
systemctl --user daemon-reload
systemctl --user enable --now taler-monitoring-hacktivism.path
systemctl --user enable --now taler-monitoring-hacktivism.timer
systemctl --user enable --now taler-monitoring-surface.timer
systemctl --user enable --now taler-monitoring-aptdeploy.timer
# v1.8.0: mail + mattermost folded into surface — stop separate timers
# Remove leftover mail/mattermost units if present (moved to meta/deprecated/)
systemctl --user disable --now taler-monitoring-mattermost.timer 2>/dev/null || true
systemctl --user disable --now taler-monitoring-mail.timer 2>/dev/null || true
rm -f "$home_unit/taler-monitoring-mattermost.service" \
"$home_unit/taler-monitoring-mattermost.timer" \
"$home_unit/taler-monitoring-mail.service" \
"$home_unit/taler-monitoring-mail.timer" 2>/dev/null || true
systemctl --user daemon-reload 2>/dev/null || true
# initial stamp so path exists
"$ROOT/touch-software-stamp.sh"
systemctl --user start taler-monitoring-hacktivism.service || true
@ -93,7 +92,7 @@ install_local() {
echo " · hacktivism 4h → /monitoring on bank/exchange/taler (landing stacks)"
echo " · surface 1h → /taler-monitoring-surface* (ecosystem versions/software)"
echo " · aptdeploy 4h (optional tests; not a public mon-page type)"
echo " · mail/mattermost timers disabled (content in surface)"
echo " · mail/mattermost: not installed (meta/deprecated/; content in surface)"
echo "suite: $mon"
echo "manual surface: systemctl --user start taler-monitoring-surface.service"
echo "after upgrades: $ROOT/touch-software-stamp.sh"

21
meta/README.md Normal file
View file

@ -0,0 +1,21 @@
# meta/ — optional & deprecated tooling
Not required for the normal **host-agent** monitoring pipeline
(`host-agent/run-*.sh` + `site-gen/console_to_html.py`).
| Subdir | Contents |
|--------|----------|
| **`deprecated/`** | Pre-v1.8 mail/mattermost host-agent wrappers + user systemd units (folded into **surface**) |
| **`firecuda/`** | Outside-runner install (launchd) + pull helpers — optional; prefer koopa host-agent |
| **`site-gen-legacy/`** | Multi-host `generate-monitoring-sites` / `deploy-monitoring-sites` (older laptop/outside path) |
## Production path (keep using)
```text
host-agent/run-host-report.sh
host-agent/run-{hacktivism,surface,aptdeploy,fp-*}.sh
host-agent/update-suite.sh
site-gen/console_to_html.py
```
Machine-local paths/SSH: **`taler-monitoring-env`** → `~/.config/taler-monitoring/env`.

View file

@ -10,7 +10,7 @@ Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin
Environment=CONTINUE_ON_ERROR=1
Environment=RUN_TIMEOUT=600
WorkingDirectory=%h/src/taler-monitoring
ExecStart=%h/src/taler-monitoring/host-agent/run-mail-monitoring.sh
ExecStart=%h/src/taler-monitoring/meta/deprecated/run-mail-monitoring.sh
Nice=10
TimeoutStartSec=20min

View file

@ -10,7 +10,7 @@ Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin
Environment=CONTINUE_ON_ERROR=1
Environment=RUN_TIMEOUT=300
WorkingDirectory=%h/src/taler-monitoring
ExecStart=%h/src/taler-monitoring/host-agent/run-mattermost-monitoring.sh
ExecStart=%h/src/taler-monitoring/meta/deprecated/run-mattermost-monitoring.sh
Nice=10
TimeoutStartSec=15min

View file

@ -11,7 +11,7 @@
set -euo pipefail
ROOT=$(cd "$(dirname "$0")" && pwd)
MON_ROOT=$(cd "$ROOT/.." && pwd)
MON_ROOT=$(cd "$ROOT/../.." && pwd)
FIRECUDA="${FIRECUDA_SSH:-}"
if [ -z "$FIRECUDA" ]; then
echo "Set FIRECUDA_SSH to your outside-runner SSH host alias" >&2
@ -60,12 +60,14 @@ sed -i.bak \"s|^WORK_ROOT=.*|WORK_ROOT=\$HOME/var/taler-monitoring-sites-work|\"
ssh -o BatchMode=yes "$FIRECUDA" bash -s <<REMOTE
set -euo pipefail
HOME_DIR=\$HOME
FIRECUDA_META="\$HOME_DIR/$REMOTE_BASE/meta/firecuda"
SITE_GEN="\$HOME_DIR/$REMOTE_BASE/site-gen"
PLIST_SRC="\$SITE_GEN/com.hacktivism.taler-monitoring-sites.plist"
PLIST_SRC="\$FIRECUDA_META/com.hacktivism.taler-monitoring-sites.plist"
PLIST_DST="\$HOME_DIR/Library/LaunchAgents/com.hacktivism.taler-monitoring-sites.plist"
sed -e "s|__SITE_GEN__|\${SITE_GEN}|g" -e "s|__HOME__|\${HOME_DIR}|g" "\$PLIST_SRC" > "\$PLIST_DST"
chmod +x "\$SITE_GEN"/*.sh "\$SITE_GEN"/console_to_html.py 2>/dev/null || true
chmod +x "\$SITE_GEN/run-on-firecuda.sh"
sed -e "s|__SITE_GEN__|\${FIRECUDA_META}|g" -e "s|__HOME__|\${HOME_DIR}|g" "\$PLIST_SRC" > "\$PLIST_DST"
chmod +x "\$FIRECUDA_META"/*.sh "\$SITE_GEN"/console_to_html.py 2>/dev/null || true
chmod +x "\$HOME_DIR/$REMOTE_BASE/meta/site-gen-legacy"/*.sh 2>/dev/null || true
chmod +x "\$FIRECUDA_META/run-on-firecuda.sh"
launchctl bootout "gui/\$(id -u)/com.hacktivism.taler-monitoring-sites" 2>/dev/null || true
launchctl unload "\$PLIST_DST" 2>/dev/null || true
launchctl bootstrap "gui/\$(id -u)" "\$PLIST_DST" 2>/dev/null || launchctl load "\$PLIST_DST"
@ -76,8 +78,8 @@ REMOTE
if [ "$RUN_NOW" = "1" ]; then
echo "run once now…"
ssh -o BatchMode=yes "$FIRECUDA" "\"\$HOME/$REMOTE_BASE/site-gen/run-on-firecuda.sh\"" || true
ssh -o BatchMode=yes "$FIRECUDA" "\"\$HOME/$REMOTE_BASE/meta/firecuda/run-on-firecuda.sh\"" || true
fi
echo "done. logs: $FIRECUDA:~/Library/Logs/taler-monitoring-sites/"
echo "manual: ssh $FIRECUDA '~/taler-monitoring-site-gen/site-gen/run-on-firecuda.sh'"
echo "manual: ssh $FIRECUDA '~/taler-monitoring-site-gen/meta/firecuda/run-on-firecuda.sh'"

View file

@ -2,14 +2,17 @@
# run-on-firecuda.sh — entrypoint for launchd/cron on outside-runner.
# Outside-only monitoring (SKIP_SSH=1), all 9 sites, then stage HTML to koopa.
#
# Installed path on outside-runner (default):
# ~/taler-monitoring-site-gen/site-gen/run-on-firecuda.sh
# Lives in meta/firecuda/ (optional). Prefer koopa host-agent for production mon.
#
set -uo pipefail
export PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:${PATH:-}"
ROOT=$(cd "$(dirname "$0")" && pwd)
HERE=$(cd "$(dirname "$0")" && pwd)
SUITE_ROOT=$(cd "$HERE/../.." && pwd)
LEGACY="$SUITE_ROOT/meta/site-gen-legacy"
SITE_GEN="$SUITE_ROOT/site-gen"
LOG_DIR="${FIRECUDA_LOG_DIR:-$HOME/Library/Logs/taler-monitoring-sites}"
mkdir -p "$LOG_DIR"
STAMP=$(date +%Y%m%d-%H%M%S)
@ -18,7 +21,6 @@ exec >>"$LOG" 2>&1
echo "======== $(date -Iseconds 2>/dev/null || date) outside-runner monitoring sites ========"
# Always: public DNS only, no SSH checks into stacks
export SKIP_SSH=1
export CONTINUE_ON_ERROR=1
export AUTH401_CONTINUE=1
@ -26,21 +28,20 @@ export RUNNER_SSH=
export PHASES_ERR="${PHASES_ERR:-urls versions}"
export PHASES_OK="${PHASES_OK:-urls versions}"
export WORK_ROOT="${WORK_ROOT:-$HOME/var/taler-monitoring-sites-work}"
export MONITORING_ROOT="${MONITORING_ROOT:-$ROOT/..}"
export MONITORING_ROOT="${MONITORING_ROOT:-$SUITE_ROOT}"
export DEPLOY_SSH="${DEPLOY_SSH:-}"
export DEPLOY_STAGING="${DEPLOY_STAGING:-${DEPLOY_STAGING:-$HOME/monitoring-sites-staging}}"
export DEPLOY_STAGING="${DEPLOY_STAGING:-$HOME/monitoring-sites-staging}"
export SITE_GEN_DIR="$SITE_GEN"
# Prefer full 9-site list from settings or defaults inside generate script
unset ONLY_HOSTS
cd "$ROOT" || exit 1
chmod +x generate-monitoring-sites.sh deploy-monitoring-sites.sh console_to_html.py 2>/dev/null || true
cd "$LEGACY" || exit 1
chmod +x generate-monitoring-sites.sh deploy-monitoring-sites.sh 2>/dev/null || true
chmod +x "$SITE_GEN/console_to_html.py" 2>/dev/null || true
./generate-monitoring-sites.sh
ec=$?
# HTML stays on outside-runner under WORK_ROOT/html.
# Optional push to koopa if this host can SSH there (often only from laptop):
if [ -n "${DEPLOY_SSH:-}" ] && [ "${DEPLOY_FROM_FIRECUDA:-0}" = "1" ]; then
if [ -x ./deploy-monitoring-sites.sh ]; then
./deploy-monitoring-sites.sh || echo "WARN: deploy staging failed"
@ -48,11 +49,10 @@ if [ -n "${DEPLOY_SSH:-}" ] && [ "${DEPLOY_FROM_FIRECUDA:-0}" = "1" ]; then
else
echo "note: HTML at ${WORK_ROOT}/html — pull from laptop:"
echo " rsync -az outside-runner:var/taler-monitoring-sites-work/html/ \\"
echo " $DEPLOY_SSH:monitoring-sites-staging/"
echo " # or: ./pull-firecuda-to-koopa.sh"
echo " \${DEPLOY_SSH}:monitoring-sites-staging/"
echo " # or: $HERE/pull-firecuda-to-koopa.sh"
fi
# keep last 20 logs
ls -1t "$LOG_DIR"/run-*.log 2>/dev/null | tail -n +21 | xargs rm -f 2>/dev/null || true
echo "======== done ec=$ec ========"

View file

@ -3,9 +3,12 @@
# Does NOT reload Caddy (needs root — see ROOT-ON-KOOPA.md).
set -uo pipefail
ROOT=$(cd "$(dirname "$0")" && pwd)
SETTINGS="${SITE_GEN_SETTINGS:-$ROOT/settings.conf}"
[ -f "$SETTINGS" ] || SETTINGS="$ROOT/settings.conf.example"
HERE=$(cd "$(dirname "$0")" && pwd)
SUITE_ROOT=$(cd "$HERE/../.." && pwd)
SITE_GEN="${SITE_GEN_DIR:-$SUITE_ROOT/site-gen}"
ROOT="$HERE"
SETTINGS="${SITE_GEN_SETTINGS:-$SITE_GEN/settings.conf}"
[ -f "$SETTINGS" ] || SETTINGS="$SITE_GEN/settings.conf.example"
# shellcheck disable=SC1090
set -a
source <(grep -v '^\s*#' "$SETTINGS" | grep -v '^\s*$' | sed 's/\r$//')

View file

@ -18,9 +18,13 @@ export PYTHONUNBUFFERED=1
: "${RUN_TIMEOUT:=600}"
export RUN_TIMEOUT
ROOT=$(cd "$(dirname "$0")" && pwd)
SETTINGS="${SITE_GEN_SETTINGS:-$ROOT/settings.conf}"
EXAMPLE="$ROOT/settings.conf.example"
HERE=$(cd "$(dirname "$0")" && pwd)
SUITE_ROOT=$(cd "$HERE/../.." && pwd)
SITE_GEN="${SITE_GEN_DIR:-$SUITE_ROOT/site-gen}"
ROOT="$HERE"
SETTINGS="${SITE_GEN_SETTINGS:-$SITE_GEN/settings.conf}"
EXAMPLE="${SITE_GEN}/settings.conf.example"
[ -f "$EXAMPLE" ] || EXAMPLE="$HERE/settings.conf.example"
if [ ! -f "$SETTINGS" ]; then
if [ -f "$EXAMPLE" ]; then
@ -80,7 +84,7 @@ while [ $# -gt 0 ]; do
esac
done
MONITORING_ROOT=$(cd "$ROOT/${MONITORING_ROOT:-..}" && pwd)
MONITORING_ROOT=$(cd "${MONITORING_ROOT:-$SUITE_ROOT}" && pwd)
WORK_ROOT="${WORK_ROOT:-/tmp/taler-monitoring-sites-work}"
LOG_DIR="$WORK_ROOT/logs"
HTML_DIR="$WORK_ROOT/html"
@ -174,7 +178,7 @@ run_mon() {
htmlify() {
local host="$1" mode="$2" log="$3" out="$4" other="$5"
mkdir -p "$(dirname "$out")"
if [ ! -f "$ROOT/console_to_html.py" ]; then
if [ ! -f "$SITE_GEN/console_to_html.py" ]; then
echo "WARN: console_to_html.py missing — bootstrap $out"
{
echo "<!DOCTYPE html><html><head><meta charset=utf-8><title>$host $mode</title></head>"
@ -185,7 +189,7 @@ htmlify() {
} >"$out"
return 0
fi
python3 "$ROOT/console_to_html.py" \
python3 "$SITE_GEN/console_to_html.py" \
--log "$log" \
--out "$out" \
--hostname "$host" \
@ -269,5 +273,5 @@ echo "HTML under $HTML_DIR"
echo "commit $COMMIT_SHORT $COMMIT_URL"
echo "RUN_TIMEOUT=${RUN_TIMEOUT}s (global; 0=unlimited)"
echo "Deploy (as hernani, may need sudo for /var/www):"
echo " ./deploy-monitoring-sites.sh"
echo " $HERE/deploy-monitoring-sites.sh"
echo "Root on koopa: see ROOT-ON-KOOPA.md"

View file

@ -1,83 +1,20 @@
# monitoring site-gen
# site-gen — HTML converter + Caddy notes
Build **console-style HTML** from `taler-monitoring` runs and stage them for
**host Caddy** (`/monitoring`, `/monitoring_err`).
**Required for host-agent:** `console_to_html.py` (log → sticky HTML).
Same **global reporting defaults** as host-agents (`run-host-report.sh`):
- **`RUN_TIMEOUT=600`** on each suite run
- **line-buffered** logs (continuous write)
- **always** write HTML (first run / empty logs too)
- Forgejo **commit link** in the page header
Applies to all nine fronts (GOA + FP stage + FP prod) in the default `SITES` list.
| Host | Role |
| File | Role |
|------|------|
| **optional outside runner** (SSH alias via `FIRECUDA_SSH`) | optional runner (public DNS) |
| **deploy host (`DEPLOY_SSH`)** | deploy staging + Caddy (root service) |
| **host-agents** | primary: GOA + FP on-box (`../host-agent/`) |
| `console_to_html.py` | Convert suite console log → public mon HTML |
| `caddy-monitoring-handles.snippet` | Caddy `handle` / bare redir notes |
| `ROOT-ON-KOOPA.md` | Publish staging → `/var/www` (paths from env) |
| `settings.conf.example` | Optional settings for **legacy** multi-host generator |
No secrets in `settings.conf` — only hosts, paths, site list.
## Optional / moved
## Optional outside runner (outside-only) — **disabled for now**
An **optional outside runner** could run public-only monitoring (`SKIP_SSH=1`, phases
`urls versions`) on a schedule (launchd every 4h) and write HTML under
`~/var/taler-monitoring-sites-work/html/`. Scripts for that still live here
(`install-firecuda-timer.sh`, `run-on-firecuda.sh`, plist template).
**Current policy: do not run a real launchd job on the outside runner.**
Primary GOA monitoring is **koopa host-agent** (`host-agent/`, user `hernani`,
local `podman exec`). Re-enable the outside runner only if you explicitly want a second,
outside-only view.
Legacy multi-host generate/deploy and firecuda launchd tooling live under
**`../meta/`** (see `meta/README.md`).
```bash
# optional one-shot (no timer):
# ssh "$FIRECUDA_SSH" # e.g. your outside-runner alias
# ssh "$FIRECUDA_SSH" '~/taler-monitoring-site-gen/site-gen/run-on-firecuda.sh'
# if a timer was installed earlier, ensure it is off:
# ssh "$FIRECUDA_SSH" # e.g. your outside-runner alias
# ssh 'launchctl list | grep taler-monitoring || echo off # macOS outside runner'
```
Disabled plists may remain as `*.plist.disabled-*` under `~/Library/LaunchAgents/`.
## Quick start (ops workstation)
```bash
cd site-gen
cp settings.conf.example settings.conf # optional
chmod +x generate-monitoring-sites.sh deploy-monitoring-sites.sh console_to_html.py
ONLY_HOSTS="bank.hacktivism.ch exchange.hacktivism.ch taler.hacktivism.ch" \
SKIP_SSH=1 RUNNER_SSH= \
./generate-monitoring-sites.sh
./deploy-monitoring-sites.sh
```
Then on koopa **as root**: see **[ROOT-ON-KOOPA.md](./ROOT-ON-KOOPA.md)**.
## URL rules
| Path | When |
|------|------|
| `/monitoring_err/` | always: full console log + error list on top (CONTINUE_ON_ERROR run) |
| `/monitoring/` | **clean** full log if last strict run exit 0; else **stub** linking to `/monitoring_err/` |
Footer links **Forgejo commit** of the **taler-monitoring** suite used for that run.
## Layout
```text
site-gen/
settings.conf.example
generate-monitoring-sites.sh
deploy-monitoring-sites.sh
console_to_html.py
caddy-monitoring-handles.snippet
ROOT-ON-KOOPA.md
README.md
# normal publish is host-agent on the mon host, not these scripts
../host-agent/run-hacktivism-monitoring.sh # etc.
```

View file

@ -3,7 +3,7 @@
Host Caddy runs as **root/systemd** (`caddy` user). Static monitoring HTML is
**not** put into Taler containers — only host paths + Caddy `handle`.
After host-agent / `generate-monitoring-sites.sh` + deploy (as hernani), files live in:
After host-agent (or optional `meta/site-gen-legacy/generate-monitoring-sites.sh`) + deploy, files live in:
```text
$DEPLOY_STAGING (or $HOME/monitoring-sites-staging)/<hostname>/monitoring/index.html