Compare commits

...

5 commits

Author SHA1 Message Date
Hernâni Marques
1f1277b36a
fix: reload mon state before progress_finish (cross-process done/total) 2026-07-19 02:35:37 +02:00
Hernâni Marques
74e76aff2c
release 1.13.1: global progress done/total stable across phases
Phase summary no longer resets PROGRESS_TOTAL to local done (which made
bars like 34/34 then 38/47). Total is fixed at run start; only grows if
short; progress_finish snaps once at the end of the whole run.
2026-07-19 02:35:09 +02:00
Hernâni Marques
2931006e3c
release 1.13.0: devtesting fake-franken CHF via rusty
New phase devtesting (aliases franken/fake-franken) probes
taler-devtesting on rusty.taler-ops.ch: geniban + fake-incoming CHF
to the exchange IBAN. Nexus bounce without reserve pub is OK for the
plumbing probe. Stage host-agent can include the phase.
2026-07-19 02:28:35 +02:00
Hernâni Marques
e67ede881c
release 1.12.0: collapsible monitoring env context
Agent run header (env_file, suite pin, target domain, hosts, flags) is
shown in a <details> block above the console, collapsed by default.
2026-07-19 02:23:17 +02:00
Hernâni Marques
db6722e40f
release 1.11.0: sticky error/warn log filter
Click error or warning counts in the sticky bar to show only matching
console lines; sticky stats and top overviews remain. Hashes
#filter-error / #filter-warn (and #first-*) activate the filter.
2026-07-19 02:16:39 +02:00
12 changed files with 760 additions and 75 deletions

View file

@ -421,7 +421,24 @@ Soft mode only with `MONPAGES_REQUIRE_PUBLIC=0` (escape hatch).
If pages are missing publicly: `sudo /home/hernani/koopa-caddy/apply-monitoring-live.sh` on koopa If pages are missing publicly: `sudo /home/hernani/koopa-caddy/apply-monitoring-live.sh` on koopa
(FP: Infomaniak vhost for `/monitoring*`). (FP: Infomaniak vhost for `/monitoring*`).
## Devtesting · fake-franken CHF (v1.13+)
Phase **`devtesting`** (aliases: `franken`, `fake-franken`) exercises the
Taler Operations **rusty** host (`taler-devtesting` forced command):
```bash
./taler-monitoring.sh -d stage.taler-ops.ch --currency CHF devtesting
```
Needs SSH to `devtesting@rusty.taler-ops.ch` (Host `taler-rusty-devtesting`).
Probes: CLI help, `geniban` (CH IBAN), `fake-incoming` CHF to exchange credit
IBAN. A nexus bounce *missing reserve public key* is **OK** for the plumbing
probe (no real wallet reserve required).
## Version on monitoring pages (v1.3+ / sticky bar) ## Version on monitoring pages (v1.3+ / sticky bar)
The sticky bar shows the suite **tag** (e.g. `v1.4.1`) and links to that tag on Forgejo. The sticky bar shows the suite **tag** (e.g. `v1.11.0`) and links to that tag on Forgejo.
Planned **v1.4.0**: fuller L10n (fr-CH for FrancPaysan, de-CH for hacktivism/Taler CH).
**v1.12.0:** Run header (**monitoring env context**: env_file, suite pin, domain/hosts/flags) is a collapsed `<details>` block above the console log — expand to inspect agent context.
**v1.11.0:** Click the sticky **error** or **warning** count to filter the console log to only those lines (other log lines hide; sticky bar, stats, and top overviews stay). Same via `#filter-error` / `#filter-warn` (or `#first-error` / `#first-warn`). Click again or **Show all** to clear.

View file

@ -193,3 +193,13 @@ totals: 40 OK, 1 ERROR, 2 WARN, 5 INFO
`NO_COLOR=1` or `CLICOLOR=0` disables boxes/colours (ASCII `[ OK ]` + `-- group --` headers). `NO_COLOR=1` or `CLICOLOR=0` disables boxes/colours (ASCII `[ OK ]` + `-- group --` headers).
Always use `printf --` friendly plain headers when colour is off (leading `---` is not a printf option). Always use `printf --` friendly plain headers when colour is off (leading `---` is not a printf option).
## devtesting · fake-franken (v1.13+)
| ID prefix | What |
|-----------|------|
| **devtesting.cli-** | SSH to rusty · `taler-devtesting` CLI |
| **devtesting.geniban-** | synthetic CH IBAN |
| **devtesting.credit-payto-** | exchange credit IBAN for nexus |
| **devtesting.fake-incoming-** | CHF IN via nexus; bounce missing reserve pub = plumbing OK |

View file

@ -1 +1 @@
1.10.11 1.13.1

View file

@ -17,6 +17,10 @@ Git tags: `vMAJOR.FEATURE.FIX` (e.g. `v1.8.0`). File `VERSION` omits the `v` pre
| Tag | Date (UTC) | Notes | | Tag | Date (UTC) | Notes |
|-----|------------|--------| |-----|------------|--------|
| **v1.13.1** | 2026-07-19 | **Bugfix:** progress bar **global** done/total — phase `summary()` no longer snaps total to phase-local N (fixes 34/34 then 38/47); final `progress_finish` snaps once; header shows 0/N. |
| **v1.13.0** | 2026-07-19 | **Feature:** phase **devtesting** / **franken** — fake-franken CHF via `rusty.taler-ops.ch` (`taler-devtesting` geniban + fake-incoming); stage host-agent can include it. |
| **v1.12.0** | 2026-07-19 | **Feature:** collapsible **monitoring env context** (agent header: env_file, suite pin, domain/hosts/flags) — `<details>` default closed; summary shows agent · domain · suite. |
| **v1.11.0** | 2026-07-19 | **Feature:** sticky **error/warn filter** — click counts to show only error or only warn log lines; sticky bar + overviews stay; `#filter-error` / `#filter-warn` (+ `#first-*` still works). |
| **v1.10.11** | 2026-07-19 | **Bugfix:** surface/aptdeploy wrappers **force** MON_HOSTS+PHASES+MONPAGES_INVENTORY (env cannot expand job to bank/exchange); wrap more mon knobs. | | **v1.10.11** | 2026-07-19 | **Bugfix:** surface/aptdeploy wrappers **force** MON_HOSTS+PHASES+MONPAGES_INVENTORY (env cannot expand job to bank/exchange); wrap more mon knobs. |
| **v1.10.10** | 2026-07-19 | **Bugfix:** monpages bare code 21 is WARN again by default (slash form remains ERROR); bare redir still needs live Caddy `redir *` — set MONPAGES_BARE_STRICT=1 to enforce. | | **v1.10.10** | 2026-07-19 | **Bugfix:** monpages bare code 21 is WARN again by default (slash form remains ERROR); bare redir still needs live Caddy `redir *` — set MONPAGES_BARE_STRICT=1 to enforce. |
| **v1.10.9** | 2026-07-19 | **Bugfix:** stats policy INFO no longer says bare “ERROR” (false #first-error on www.stats-01); extract_errors only from `┌ ERROR` badges. | | **v1.10.9** | 2026-07-19 | **Bugfix:** stats policy INFO no longer says bare “ERROR” (false #first-error on www.stats-01); extract_errors only from `┌ ERROR` badges. |

147
check_devtesting.sh Executable file
View file

@ -0,0 +1,147 @@
#!/usr/bin/env bash
# check_devtesting.sh — fake-franken (CHF) via rusty.taler-ops.ch taler-devtesting
#
# Requires SSH to Host DEVTESTING_SSH (default: taler-rusty-devtesting →
# rusty.taler-ops.ch user devtesting, forced command taler-devtesting).
#
# Probes:
# 1) CLI reachable (fake-incoming --help)
# 2) geniban → Swiss IBAN
# 3) fake-incoming CHF amount to exchange credit IBAN (nexus IN)
# Bounce "missing reserve public key" = plumbing OK (no real wallet reserve)
# Full credit needs a real reserve pub as --subject (optional later)
#
# Env:
# DEVTESTING_SSH default taler-rusty-devtesting (or user@host)
# DEVTESTING_AMOUNT default CHF:1.00
# DEVTESTING_CREDIT_PAYTO exchange IBAN payto (auto from EXCHANGE_PUBLIC/keys if empty)
# DEVTESTING_EXCHANGE_IBAN fallback CH6808573105529100001 (TOPS CHF exchange)
# DEVTESTING_SKIP=1 skip phase
# SKIP_SSH=1 skip (same as other SSH phases)
#
set -euo pipefail
ROOT=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=lib.sh
source "$ROOT/lib.sh"
set_area devtesting
section "devtesting · fake-franken CHF (rusty taler-devtesting)"
# Do NOT honor SKIP_SSH here: stage/tops profiles set SKIP_SSH=1 for container
# inside checks, but rusty.taler-ops.ch is a separate SSH hop (devtesting key).
if [ "${DEVTESTING_SKIP:-0}" = "1" ]; then
info "skip" "DEVTESTING_SKIP=1"
exit 0
fi
: "${DEVTESTING_SSH:=taler-rusty-devtesting}"
: "${DEVTESTING_AMOUNT:=CHF:1.00}"
: "${DEVTESTING_EXCHANGE_IBAN:=CH6808573105529100001}"
: "${SSH_CONNECT_TIMEOUT:=8}"
: "${SSH_CMD_TIMEOUT:=45}"
ssh_dt() {
# Prefer Host alias; fall back to explicit user@host if config missing
local target="$DEVTESTING_SSH"
if ! ssh -G "$target" >/dev/null 2>&1; then
target="devtesting@rusty.taler-ops.ch"
fi
if command -v timeout >/dev/null 2>&1; then
timeout "${SSH_CMD_TIMEOUT}" ssh \
-o BatchMode=yes \
-o ConnectTimeout="${SSH_CONNECT_TIMEOUT}" \
-o StrictHostKeyChecking=accept-new \
"$target" "$@"
else
ssh \
-o BatchMode=yes \
-o ConnectTimeout="${SSH_CONNECT_TIMEOUT}" \
-o StrictHostKeyChecking=accept-new \
"$target" "$@"
fi
}
set_group cli
info "ssh" "DEVTESTING_SSH=${DEVTESTING_SSH} amount=${DEVTESTING_AMOUNT}"
help_out=$(ssh_dt fake-incoming --help 2>&1) || true
if printf '%s' "$help_out" | grep -q 'fake-incoming'; then
ok "cli" "taler-devtesting fake-incoming reachable via ${DEVTESTING_SSH}"
else
fail "cli" "cannot run taler-devtesting on ${DEVTESTING_SSH}" \
"$(printf '%s' "$help_out" | tr '\n' ' ' | head -c 200)"
exit 1
fi
set_group geniban
iban_out=$(ssh_dt geniban 2>&1) || true
iban=$(printf '%s\n' "$iban_out" | grep -E '^CH[0-9A-Z]+$' | tail -1 || true)
if [ -n "$iban" ]; then
ok "geniban" "synthetic IBAN $iban"
else
fail "geniban" "expected CH… IBAN" "$(printf '%s' "$iban_out" | tr '\n' ' ' | head -c 160)"
exit 1
fi
# Credit payto: rusty libeufin-nexus only accepts the *production* exchange IBAN
# configured there (DEVTESTING_EXCHANGE_IBAN). Do not use stage /keys first-account.
set_group credit-payto
credit_payto="${DEVTESTING_CREDIT_PAYTO:-}"
if [ -z "$credit_payto" ]; then
credit_payto="payto://iban/${DEVTESTING_EXCHANGE_IBAN}?receiver-name=Taler+Operations+AG"
fi
ok "credit-payto" "$credit_payto"
# Subject: not a real reserve — nexus should bounce missing reserve public key
# (proves CHF fake-incoming path). Real reserve pub would credit wirewatch.
subject="mon-fake-franken-$(date -u +%Y%m%d%H%M%S)-$$"
debit_payto="payto://iban/${iban}?receiver-name=MonDevTest"
set_group fake-incoming
info "fake-incoming" "amount=${DEVTESTING_AMOUNT} subject=${subject}"
fi_rc=0
fi_out=$(ssh_dt fake-incoming \
--amount "${DEVTESTING_AMOUNT}" \
--subject "${subject}" \
--credit-payto "${credit_payto}" \
--debit-payto "${debit_payto}" 2>&1) || fi_rc=$?
# Normalize multi-line for matching
fi_flat=$(printf '%s' "$fi_out" | tr '\n' ' ')
if printf '%s' "$fi_flat" | grep -qiE 'Permission denied|Connection refused|Could not resolve|No such command'; then
fail "fake-incoming" "SSH/CLI failure" "$(printf '%s' "$fi_flat" | head -c 200)"
exit 1
fi
if printf '%s' "$fi_flat" | grep -qiE 'Creditor must be the exchange'; then
fail "fake-incoming" "wrong credit IBAN (not exchange)" "$(printf '%s' "$fi_flat" | head -c 220)"
exit 1
fi
if printf '%s' "$fi_flat" | grep -qiE 'Missing amount|unable to read secrets|CalledProcessError'; then
# secrets warn alone is soft if we still see IN line
if ! printf '%s' "$fi_flat" | grep -qE 'libeufin-nexus - IN .* CHF'; then
fail "fake-incoming" "nexus rejected request" "$(printf '%s' "$fi_flat" | head -c 220)"
exit 1
fi
fi
if printf '%s' "$fi_flat" | grep -qE 'libeufin-nexus - IN .* CHF:[0-9]'; then
if printf '%s' "$fi_flat" | grep -qiE 'bounced.*missing reserve public key'; then
ok "fake-incoming" "CHF IN accepted by nexus · bounced (no reserve pub) — fake-franken plumbing OK"
info "note" "use real reserve pub as --subject for wirewatch credit (not required for mon probe)"
elif printf '%s' "$fi_flat" | grep -qiE 'bounced'; then
warn "fake-incoming" "CHF IN bounced (other reason)" "$(printf '%s' "$fi_flat" | grep -oiE 'bounced[^ ]* [^ ]*' | head -c 160)"
else
ok "fake-incoming" "CHF IN recorded by nexus (no bounce line)"
fi
elif [ "$fi_rc" -eq 0 ] && printf '%s' "$fi_flat" | grep -qi 'Faking incoming CHF'; then
ok "fake-incoming" "CLI completed CHF fake-incoming"
else
fail "fake-incoming" "no CHF IN line from nexus" "rc=${fi_rc} $(printf '%s' "$fi_flat" | head -c 200)"
exit 1
fi
info "done" "devtesting fake-franken probe finished"
exit 0

35
host-agent/env/mytops-stage.env.example vendored Normal file
View file

@ -0,0 +1,35 @@
# Copy to: ~/.config/taler-monitoring/env (user taler-monitoring @ betel)
# mytops CHF stage — NOT lifeline / production
#
SUITE_GIT_URL=https://git.hacktivism.ch/hernani/taler-monitoring.git
SUITE_GIT_REF=main
SUITE_DIR=$HOME/src/taler-monitoring
TALER_DOMAIN=stage.taler-ops.ch
MON_HOSTS="stage.my.taler-ops.ch stage.taler-ops.ch"
HTML_OUT=$HOME/monitoring-sites-staging
DEPLOY_WWW_ROOT=/var/www/monitoring-sites
HTML_URL_OK=/monitoring/
HTML_URL_ERR=/monitoring_err/
HTML_OK_DIR=monitoring
HTML_ERR_DIR=monitoring_err
# urls + monpages + fake-franken CHF (rusty.taler-ops.ch)
PHASES="urls monpages devtesting"
SKIP_SSH=1
LOCAL_STACK=0
INSIDE_PODMAN=0
CONTINUE_ON_ERROR=1
RUN_TIMEOUT=600
STRICT_EXIT=1
MONPAGES_REQUIRE_PUBLIC=1
MONPAGES_INVENTORY=job
TALER_MON_LANG=en
AGENT_LABEL=mytops-stage-host-agent
STATE_NAME=taler-monitoring-mytops-stage
SUITE_UPDATE_STRICT=1
CHECK_BANK=0
MERCHANT_REQUIRED=1
CHECK_LANDING=0
# rusty devtesting (SSH Host taler-rusty-devtesting → devtesting@rusty.taler-ops.ch)
DEVTESTING_SSH=taler-rusty-devtesting
DEVTESTING_AMOUNT=CHF:1.00
# DEVTESTING_SKIP=1

View file

@ -55,6 +55,9 @@ i18n_text() {
fi fi
case "$s" in case "$s" in
"numbered checks this run:") printf 'controles numerotes de cette execution :' ;; "numbered checks this run:") printf 'controles numerotes de cette execution :' ;;
"numbered checks:") printf 'controles numerotes :' ;;
"(global done/total)") printf '(fait/total global)' ;;
"global checks:") printf 'controles globaux :' ;;
"failed — see list above"|"failed - see list above") printf 'echec — voir la liste ci-dessus' ;; "failed — see list above"|"failed - see list above") printf 'echec — voir la liste ci-dessus' ;;
"BLOCKERS · pay/withdraw cannot finish") printf 'BLOCAGES · paiement/retrait impossible' ;; "BLOCKERS · pay/withdraw cannot finish") printf 'BLOCAGES · paiement/retrait impossible' ;;
"BLOCKERS (pay/withdraw cannot finish)") printf 'BLOCAGES (paiement/retrait impossible)' ;; "BLOCKERS (pay/withdraw cannot finish)") printf 'BLOCAGES (paiement/retrait impossible)' ;;

52
lib.sh
View file

@ -590,20 +590,38 @@ add_progress_total() {
PROGRESS_TOTAL=$((PROGRESS_TOTAL + n)) PROGRESS_TOTAL=$((PROGRESS_TOTAL + n))
_mon_state_save _mon_state_save
} }
# When estimate was short, grow total so we never print done>total (e.g. 224/205). # When the global estimate was short, grow total just enough so done≤total.
# Keeps ~10% headroom so the bar is not stuck at 100% while checks continue. # Never shrink. Mid-run headroom is small and monotonic (stable N in done/N).
_progress_rebalance() { _progress_rebalance() {
[ "${PROGRESS_TOTAL:-0}" -gt 0 ] || return 0 [ "${PROGRESS_TOTAL:-0}" -gt 0 ] || return 0
[ "${PROGRESS_DONE:-0}" -gt 0 ] || return 0 [ "${PROGRESS_DONE:-0}" -gt 0 ] || return 0
if [ "$PROGRESS_DONE" -gt "$PROGRESS_TOTAL" ]; then if [ "$PROGRESS_DONE" -gt "$PROGRESS_TOTAL" ]; then
local head=$((PROGRESS_DONE / 10)) # + max(8, 5% of done) so we do not snap to 100% then jump again next check
[ "$head" -lt 12 ] && head=12 local head=$((PROGRESS_DONE / 20))
[ "$head" -lt 8 ] && head=8
PROGRESS_TOTAL=$((PROGRESS_DONE + head)) PROGRESS_TOTAL=$((PROGRESS_DONE + head))
_mon_state_save
fi
}
# Snap total to actual done — only at whole-run end (not after each phase).
progress_finish() {
[ "${PROGRESS_OFF:-0}" = "1" ] && return 0
[ "${PROGRESS_DONE:-0}" -gt 0 ] || return 0
if [ "${PROGRESS_TOTAL:-0}" -ne "$PROGRESS_DONE" ]; then
PROGRESS_TOTAL=$PROGRESS_DONE
_mon_state_save
fi
_progress_bar_line "$PROGRESS_DONE" "$PROGRESS_TOTAL"
if [ "${GLOBAL_N:-0}" -gt 0 ]; then
printf -- '%s %s %d/%d (#001…#%03d)%s\n' \
"$D" "$(i18n_text 'global checks:')" \
"$PROGRESS_DONE" "$PROGRESS_TOTAL" "$GLOBAL_N" "$N"
fi fi
} }
_progress_bar_line() { _progress_bar_line() {
# Format (grows left → right; badge = percent, not "PROG"): # Format (grows left → right; badge = percent):
# ┌ 42% ┐ ████████░░░░░░░░░░░░░░░░ 31/74 # ┌ 42% ┐ ████████░░░░░░░░░░░░░░░░ 31/74
# done/total = global numbered checks so far / global expected total
local done="$1" total="$2" width=28 pct=0 filled empty i bar pct_lab local done="$1" total="$2" width=28 pct=0 filled empty i bar pct_lab
# Defensive: never display done > total # Defensive: never display done > total
if [ "$total" -gt 0 ] && [ "$done" -gt "$total" ]; then if [ "$total" -gt 0 ] && [ "$done" -gt "$total" ]; then
@ -636,7 +654,7 @@ _progress_bar_line() {
while [ "$i" -lt "$empty" ]; do bar="${bar}"; i=$((i + 1)); done while [ "$i" -lt "$empty" ]; do bar="${bar}"; i=$((i + 1)); done
pct_lab=$(printf '%3d%%' "$pct") pct_lab=$(printf '%3d%%' "$pct")
if [ "${BOX:-0}" = "1" ]; then if [ "${BOX:-0}" = "1" ]; then
# badge = percent (clearer than "PROG"); bar grows ░→█ left to right # badge = percent; bar grows ░→█ left to right; N/M = global check count
printf '%s┌%s┐%s %s%s%s %s%d/%d%s\n' \ printf '%s┌%s┐%s %s%s%s %s%d/%d%s\n' \
"$BG_PROG" "$pct_lab" "$N" \ "$BG_PROG" "$pct_lab" "$N" \
"$C" "$bar" "$N" \ "$C" "$bar" "$N" \
@ -647,10 +665,10 @@ _progress_bar_line() {
fi fi
else else
if [ "${BOX:-0}" = "1" ]; then if [ "${BOX:-0}" = "1" ]; then
printf '%s┌ … ┐%s %sdone=%d%s (set PROGRESS_TOTAL= for %%)\n' \ printf '%s┌ … ┐%s %s%d/?%s (set PROGRESS_TOTAL= for global %%)\n' \
"$BG_PROG" "$N" "$D" "$done" "$N" "$BG_PROG" "$N" "$D" "$done" "$N"
else else
printf -- '%sdone=%d%s (set PROGRESS_TOTAL= for %%)\n' "$D" "$done" "$N" printf -- '%s%d/?%s (set PROGRESS_TOTAL= for global %%)\n' "$D" "$done" "$N"
fi fi
fi fi
} }
@ -811,18 +829,30 @@ section() {
summary() { summary() {
local blk_n=${#BLOCKERS[@]} local blk_n=${#BLOCKERS[@]}
echo "" echo ""
# final progress line (no _take_tid — plain printf) # Phase progress snapshot — keep GLOBAL total (do not snap N to phase-local done).
if [ "${PROGRESS_OFF:-0}" != "1" ] && [ "$PROGRESS_DONE" -gt 0 ]; then # Snapping here made mid-run bars like 34/34 then 38/47. Whole-run snap: progress_finish.
# Snap total to actual so last line is exact N/N 100% if [ "${PROGRESS_OFF:-0}" != "1" ] && [ "${PROGRESS_DONE:-0}" -gt 0 ]; then
if [ "${PROGRESS_FINAL:-0}" = "1" ]; then
# last phase / explicit final: done/total exact
if [ "$PROGRESS_TOTAL" -gt 0 ] && [ "$PROGRESS_DONE" -ne "$PROGRESS_TOTAL" ]; then if [ "$PROGRESS_TOTAL" -gt 0 ] && [ "$PROGRESS_DONE" -ne "$PROGRESS_TOTAL" ]; then
PROGRESS_TOTAL=$PROGRESS_DONE PROGRESS_TOTAL=$PROGRESS_DONE
_mon_state_save _mon_state_save
fi fi
fi
_progress_rebalance
_progress_bar_line "$PROGRESS_DONE" "$PROGRESS_TOTAL" _progress_bar_line "$PROGRESS_DONE" "$PROGRESS_TOTAL"
fi fi
if [ "$GLOBAL_N" -gt 0 ]; then if [ "$GLOBAL_N" -gt 0 ]; then
# Always show global position: current # / expected total (if known)
if [ "${PROGRESS_TOTAL:-0}" -gt 0 ]; then
printf -- '%s %s #001…#%03d · %d/%d %s%s\n' \
"$D" "$(i18n_text 'numbered checks:')" "$GLOBAL_N" \
"$PROGRESS_DONE" "$PROGRESS_TOTAL" \
"$(i18n_text '(global done/total)')" "$N"
else
printf -- '%s %s #001…#%03d%s\n' "$D" "$(i18n_text 'numbered checks this run:')" "$GLOBAL_N" "$N" printf -- '%s %s #001…#%03d%s\n' "$D" "$(i18n_text 'numbered checks this run:')" "$GLOBAL_N" "$N"
fi fi
fi
if [ "$blk_n" -gt 0 ]; then if [ "$blk_n" -gt 0 ]; then
if [ "${BOX:-0}" = "1" ]; then if [ "${BOX:-0}" = "1" ]; then
printf -- '%s┌ %s ┐%s\n' "$BG_BLK" "$(i18n_text 'BLOCKERS · pay/withdraw cannot finish')" "$N" printf -- '%s┌ %s ┐%s\n' "$BG_BLK" "$(i18n_text 'BLOCKERS · pay/withdraw cannot finish')" "$N"

View file

@ -69,6 +69,15 @@
# AUTH401_ADMIN_USER=admin # AUTH401_ADMIN_USER=admin
# FRANCPAYSAN_SECRETS=$HOME/francpaysan-secrets # FRANCPAYSAN_SECRETS=$HOME/francpaysan-secrets
# --- Devtesting / fake-franken CHF (rusty.taler-ops.ch) ---
# Phase: devtesting | franken | fake-franken
# SSH Host taler-rusty-devtesting (user devtesting, forced taler-devtesting CLI).
# DEVTESTING_SSH=taler-rusty-devtesting
# DEVTESTING_AMOUNT=CHF:1.00
# DEVTESTING_CREDIT_PAYTO= # optional; auto from EXCHANGE_PUBLIC/keys or TOPS IBAN
# DEVTESTING_EXCHANGE_IBAN=CH6808573105529100001
# DEVTESTING_SKIP=1 # skip phase
# --- Ladder (GOA withdraw ladder · explorer pool) --- # --- Ladder (GOA withdraw ladder · explorer pool) ---
# Prefer SECRETS_ROOT / KOOPA_ADMIN_SECRETS (bank-explorer-password.txt). # Prefer SECRETS_ROOT / KOOPA_ADMIN_SECRETS (bank-explorer-password.txt).
# Overrides only if needed: # Overrides only if needed:

View file

@ -1,23 +1,27 @@
# Host Caddy (root service). Place BEFORE catch-all reverse_proxy in each site block. # Host Caddy (root service). Place BEFORE catch-all reverse_proxy in each site block.
# #
# CRITICAL (v1.7.2): root must be the *host directory* (…/monitoring-sites/{host}), # CRITICAL: bare paths (no trailing slash) must redirect to slash, or the request
# and the matcher must be a path *prefix* (`/monitoring*`), so that URI # falls through to merchant/bank/exchange API → Taler JSON code 21.
# /monitoring/index.html maps to {root}/monitoring/index.html.
# #
# WRONG (causes empty HTTP 404 from Caddy file_server): # CORRECT (site-level named matcher — preferred):
# handle /monitoring/ { # @mon_bare path /monitoring /monitoring_err \
# root * …/monitoring-sites/{host}/monitoring # /taler-monitoring-surface /taler-monitoring-surface_err \
# file_server # /taler-monitoring-aptdeploy /taler-monitoring-aptdeploy_err
# } # redir @mon_bare {path}/ 302
# → looks for …/monitoring/monitoring/index.html
# #
# CRITICAL (v1.9.2): bare-path redir inside handle MUST use matcher `*`: # ALSO OK inside handle (matcher must be `*`):
# handle /taler-monitoring-surface { # handle /taler-monitoring-surface {
# redir * /taler-monitoring-surface/ 302 # redir * /taler-monitoring-surface/ 302
# } # }
# WRONG: `redir /taler-monitoring-surface/ 302` is parsed as #
# matcher=/taler-monitoring-surface/ to="302" → Location: 302, merchant code 21 # WRONG (Caddy footgun — causes merchant code 21 on bare URLs):
# (same footgun for /monitoring, aptdeploy, mail, mattermost bare paths). # redir /taler-monitoring-surface/ 302
# → parsed as matcher=/taler-monitoring-surface/ to="302"
# → Location: 302, or no-op then fallthrough to reverse_proxy
#
# root must be the *host directory* (…/monitoring-sites/{host}), and the
# file_server matcher a path *prefix* (`/monitoring*`), so
# /monitoring/index.html maps to {root}/monitoring/index.html.
# #
# Disk layout: # Disk layout:
# /var/www/monitoring-sites/{host}/monitoring/index.html # /var/www/monitoring-sites/{host}/monitoring/index.html
@ -25,49 +29,44 @@
# #
# Live apply on koopa (absolute path — ~ as root is /root): # Live apply on koopa (absolute path — ~ as root is /root):
# sudo /home/hernani/koopa-caddy/apply-monitoring-live.sh # sudo /home/hernani/koopa-caddy/apply-monitoring-live.sh
# Runtime-only (no sudo; hernani admin API):
# curl -g -X POST http://[::1]:2019/load -H 'Content-Type: text/caddyfile' \
# --data-binary @/home/hernani/koopa-caddy/Caddyfile
# --- only inside taler.hacktivism.ch { ... } --- # --- only inside taler.hacktivism.ch { ... } ---
handle /taler-monitoring-surface { @mon_bare path /monitoring /monitoring_err \
redir * /taler-monitoring-surface/ 302 /taler-monitoring-surface /taler-monitoring-surface_err \
/taler-monitoring-aptdeploy /taler-monitoring-aptdeploy_err \
/taler-monitoring-mail /taler-monitoring-mail_err \
/taler-monitoring-mattermost /taler-monitoring-mattermost_err
redir @mon_bare {path}/ 302
handle /taler-monitoring-surface_err* {
root * /var/www/monitoring-sites/taler.hacktivism.ch
file_server
} }
handle /taler-monitoring-surface* { handle /taler-monitoring-surface* {
root * /var/www/monitoring-sites/taler.hacktivism.ch root * /var/www/monitoring-sites/taler.hacktivism.ch
file_server file_server
} }
handle /taler-monitoring-surface_err { handle /taler-monitoring-aptdeploy_err* {
redir * /taler-monitoring-surface_err/ 302
}
handle /taler-monitoring-surface_err* {
root * /var/www/monitoring-sites/taler.hacktivism.ch root * /var/www/monitoring-sites/taler.hacktivism.ch
file_server file_server
} }
handle /taler-monitoring-aptdeploy {
redir * /taler-monitoring-aptdeploy/ 302
}
handle /taler-monitoring-aptdeploy* { handle /taler-monitoring-aptdeploy* {
root * /var/www/monitoring-sites/taler.hacktivism.ch root * /var/www/monitoring-sites/taler.hacktivism.ch
file_server file_server
} }
handle /taler-monitoring-aptdeploy_err {
redir * /taler-monitoring-aptdeploy_err/ 302
}
handle /taler-monitoring-aptdeploy_err* {
root * /var/www/monitoring-sites/taler.hacktivism.ch
file_server
}
# --- bank + exchange + taler (each site block; set root host dir) --- # --- bank + exchange + taler (each site block; set root host dir) ---
handle /monitoring { @mon_bare path /monitoring /monitoring_err
redir * /monitoring/ 302 redir @mon_bare {path}/ 302
handle /monitoring_err* {
root * /var/www/monitoring-sites/{host}
file_server
} }
handle /monitoring* { handle /monitoring* {
root * /var/www/monitoring-sites/{host} root * /var/www/monitoring-sites/{host}
file_server file_server
} }
handle /monitoring_err {
redir * /monitoring_err/ 302
}
handle /monitoring_err* {
root * /var/www/monitoring-sites/{host}
file_server
}

View file

@ -31,6 +31,15 @@ def ui(lang: str, key: str, **kwargs) -> str:
"warn_many": "{n} warnings", "warn_many": "{n} warnings",
"jump_first_error": "Jump to first error", "jump_first_error": "Jump to first error",
"jump_first_warn": "Jump to first warning", "jump_first_warn": "Jump to first warning",
"filter_errors": "Filter: show all errors (hide other log lines)",
"filter_warns": "Filter: show all warnings (hide other log lines)",
"filter_clear": "Clear filter · show all lines",
"filter_active_err": "Filter: errors only",
"filter_active_warn": "Filter: warnings only",
"filter_show_all": "Show all",
"env_context": "Monitoring env context",
"env_context_title": "Expand: agent env, suite pin, domain hosts & flags (from run header)",
"env_context_empty": "No run header in log",
"generated": "generated", "generated": "generated",
"version": "version", "version": "version",
@ -83,6 +92,7 @@ def ui(lang: str, key: str, **kwargs) -> str:
"stage_i3": "Phases: {phases} · suite auto-upgrade each timer run", "stage_i3": "Phases: {phases} · suite auto-upgrade each timer run",
"stage_i4": "Not GOA: no surface/aptdeploy/mail pages · no koopa-admin-log", "stage_i4": "Not GOA: no surface/aptdeploy/mail pages · no koopa-admin-log",
"stage_i5": "Timer: taler-monitoring-mytops-stage.timer (4h) · user taler-monitoring", "stage_i5": "Timer: taler-monitoring-mytops-stage.timer (4h) · user taler-monitoring",
"stage_i6": "devtesting: fake-franken CHF via rusty.taler-ops.ch (geniban + fake-incoming)",
"pages_section_lead": "Monitoring pages themselves (suite):", "pages_section_lead": "Monitoring pages themselves (suite):",
"mm_title": "Mattermost chat health", "mm_title": "Mattermost chat health",
"mm_summary": "mattermost.taler.net · SPA + /api/v4/system/ping · TLS", "mm_summary": "mattermost.taler.net · SPA + /api/v4/system/ping · TLS",
@ -94,7 +104,7 @@ def ui(lang: str, key: str, **kwargs) -> str:
"mail_i1": "Outside-in phase mail: MX, SMTP/IMAP ports and handshakes, SPF/DMARC", "mail_i1": "Outside-in phase mail: MX, SMTP/IMAP ports and handshakes, SPF/DMARC",
"mail_i2": "Mail hosts: firefly.gnunet.org (taler.net/gnunet.org) and anastasis.taler-systems.com", "mail_i2": "Mail hosts: firefly.gnunet.org (taler.net/gnunet.org) and anastasis.taler-systems.com",
"mail_i3": "Reported on taler-monitoring-surface HTML (no separate /taler-monitoring-mail* page)", "mail_i3": "Reported on taler-monitoring-surface HTML (no separate /taler-monitoring-mail* page)",
"footer": "Console-style render of taler-monitoring output. Sticky bar: green = clean · yellow = warnings · red = errors. Commit pins the exact tree used for this run.", "footer": "Console-style render of taler-monitoring output. Sticky bar: green = clean · yellow = warnings · red = errors. Click error/warning counts to filter the log. Monitoring env context (agent header) is collapsed by default. Commit pins the exact tree used for this run.",
"redirect_fail": "has failures.", "redirect_fail": "has failures.",
"redirect_see": "See {link} for the full console log, error index, and sticky status bar.", "redirect_see": "See {link} for the full console log, error index, and sticky status bar.",
"ago_s": "{n}s ago", "ago_s": "{n}s ago",
@ -113,6 +123,15 @@ def ui(lang: str, key: str, **kwargs) -> str:
"warn_many": "{n} avertissements", "warn_many": "{n} avertissements",
"jump_first_error": "Aller à la première erreur", "jump_first_error": "Aller à la première erreur",
"jump_first_warn": "Aller au premier avertissement", "jump_first_warn": "Aller au premier avertissement",
"filter_errors": "Filtrer : toutes les erreurs (masquer le reste du journal)",
"filter_warns": "Filtrer : tous les avertissements (masquer le reste du journal)",
"filter_clear": "Effacer le filtre · tout afficher",
"filter_active_err": "Filtre : erreurs seulement",
"filter_active_warn": "Filtre : avertissements seulement",
"filter_show_all": "Tout afficher",
"env_context": "Contexte d'env. de monitoring",
"env_context_title": "Déplier : env agent, pin de suite, hôtes domaine et flags (en-tête d'exécution)",
"env_context_empty": "Pas d'en-tête d'exécution dans le journal",
"generated": "généré", "generated": "généré",
"version": "version", "version": "version",
@ -164,6 +183,7 @@ def ui(lang: str, key: str, **kwargs) -> str:
"stage_i3": "Phases : {phases} · auto-upgrade suite à chaque minuterie", "stage_i3": "Phases : {phases} · auto-upgrade suite à chaque minuterie",
"stage_i4": "Pas GOA : pas de pages surface/aptdeploy/mail · pas de koopa-admin-log", "stage_i4": "Pas GOA : pas de pages surface/aptdeploy/mail · pas de koopa-admin-log",
"stage_i5": "Minuterie : taler-monitoring-mytops-stage.timer (4 h) · user taler-monitoring", "stage_i5": "Minuterie : taler-monitoring-mytops-stage.timer (4 h) · user taler-monitoring",
"stage_i6": "devtesting : fake-franken CHF via rusty.taler-ops.ch (geniban + fake-incoming)",
"pages_i6": "Hôte de cette page : {host} · libellé : {label}", "pages_i6": "Hôte de cette page : {host} · libellé : {label}",
"pages_section_lead": "Les pages de monitoring elles-mêmes (suite) :", "pages_section_lead": "Les pages de monitoring elles-mêmes (suite) :",
"mm_title": "Santé du chat Mattermost", "mm_title": "Santé du chat Mattermost",
@ -176,7 +196,7 @@ def ui(lang: str, key: str, **kwargs) -> str:
"mail_i1": "Phase outside-in mail : MX, ports SMTP/IMAP et handshakes, SPF/DMARC", "mail_i1": "Phase outside-in mail : MX, ports SMTP/IMAP et handshakes, SPF/DMARC",
"mail_i2": "Hôtes mail : firefly.gnunet.org (taler.net/gnunet.org) et anastasis.taler-systems.com", "mail_i2": "Hôtes mail : firefly.gnunet.org (taler.net/gnunet.org) et anastasis.taler-systems.com",
"mail_i3": "Rapporté dans le HTML taler-monitoring-surface (pas de page /taler-monitoring-mail*)", "mail_i3": "Rapporté dans le HTML taler-monitoring-surface (pas de page /taler-monitoring-mail*)",
"footer": "Rendu console de taler-monitoring. Barre collante : vert = OK · jaune = avertissements · rouge = erreurs. Le commit fixe l'arbre exact de cette exécution.", "footer": "Rendu console de taler-monitoring. Barre collante : vert = OK · jaune = avertissements · rouge = erreurs. Compteurs = filtre. Contexte d'env. (en-tête agent) replié par défaut. Le commit fixe l'arbre exact de cette exécution.",
"redirect_fail": "a des échecs.", "redirect_fail": "a des échecs.",
"redirect_see": "Voir {link} pour le journal console complet, l'index d'erreurs et la barre de statut.", "redirect_see": "Voir {link} pour le journal console complet, l'index d'erreurs et la barre de statut.",
"ago_s": "il y a {n}s", "ago_s": "il y a {n}s",
@ -594,6 +614,7 @@ def monitoring_scope(
ui(lang, "stage_i3", phases=phase_txt), ui(lang, "stage_i3", phases=phase_txt),
ui(lang, "stage_i4"), ui(lang, "stage_i4"),
ui(lang, "stage_i5"), ui(lang, "stage_i5"),
ui(lang, "stage_i6"),
ui(lang, "pages_i6", host=host, label=label), ui(lang, "pages_i6", host=host, label=label),
] ]
return { return {
@ -637,6 +658,172 @@ def monitoring_scope(
} }
def is_env_context_start(ln: str) -> bool:
"""Host-agent / suite banner that opens the monitoring env header."""
s = ln.strip()
if re.match(r"^={4,}\s*.+\s*={4,}$", s):
return True
if s.startswith("env_file="):
return True
return False
def is_env_context_line(ln: str) -> bool:
"""
Lines belonging to the pre-check monitoring env header
(agent banner, suite pin, target domain / flags / timeout).
"""
s = ln.rstrip("\n")
if not s.strip():
return True # blank lines inside header block
t = s.strip()
if is_env_context_start(s):
return True
# agent header keys
if t.startswith(
(
"env_file=",
"domain=",
"hosts=",
"flags ",
"suite ",
"using suite",
"commit=",
"target domain=",
"note:",
"WARN: Forgejo",
"WARN suite-update",
)
):
return True
if t.startswith(" dir=") or t.startswith("dir="):
return True
# indented target-domain block from taler-monitoring.sh
if re.match(
r"^\s+(bank|exchange|merchant|currency|phases|flags|access|run_timeout|progress)\b",
s,
):
return True
# re-applying overlay chatter still part of startup
if "suite-overlay" in t.lower() and (
t.startswith("WARN") or t.startswith(" ") or "re-applying" in t.lower()
):
return True
return False
def is_env_context_end(ln: str) -> bool:
"""First line that is clearly past the env header (phases / checks start)."""
s = ln.strip()
if not s:
return False
# box headers / section titles for real check phases
if s.startswith("") or s.startswith("") or s.startswith(""):
return True
if s.startswith("") or s.startswith("") or s.startswith("["):
return True
if re.match(r"^==\s+", s) or re.match(r"^--\s+", s):
return True
if "numbered checks" in s.lower():
return True
if re.match(r"^\s*\d+%\s", s): # bare progress
return True
return False
def extract_env_context(lines: list[str]) -> tuple[list[str], list[str], str]:
"""
Split leading monitoring env header from the rest of the log.
Returns (env_lines, body_lines, hint_summary).
"""
if not lines:
return [], [], ""
# Find start: first env-context start within the first ~40 lines
start = None
for i, ln in enumerate(lines[:40]):
if is_env_context_start(ln):
start = i
break
if start is None:
# no banner — maybe only "target domain=" block at top (CLI without agent)
for i, ln in enumerate(lines[:15]):
if ln.strip().startswith("target domain=") or ln.strip().startswith(
"env_file="
):
start = i
break
if start is None:
return [], lines, ""
end = start
for j in range(start, len(lines)):
ln = lines[j]
if j > start and is_env_context_end(ln):
end = j
break
if j > start and not is_env_context_line(ln) and ln.strip():
# non-empty line that doesn't look like env → end before it
end = j
break
end = j + 1
else:
end = len(lines)
env = lines[start:end]
# trim trailing blanks from env
while env and not env[-1].strip():
env.pop()
body = lines[:start] + lines[end:]
# hint: suite + domain + phases
hint_parts: list[str] = []
for ln in env:
t = ln.strip()
if t.startswith("suite "):
hint_parts.append(t.split(" ")[0][:48])
elif t.startswith("domain="):
hint_parts.append(t.replace("domain=", "").split("·")[0].strip()[:40])
elif t.startswith("target domain="):
if not any("domain" in h for h in hint_parts):
hint_parts.append(t.split("=", 1)[1].strip()[:40])
elif re.match(r"^={4,}", t):
# agent label between equals
m = re.match(r"^={4,}\s*(.+?)\s*={4,}$", t)
if m:
lab = m.group(1)
# drop timestamp prefix if present
lab2 = re.sub(r"^\d{4}-\d{2}-\d{2}T\S+\s+", "", lab)
hint_parts.insert(0, lab2[:40])
hint = " · ".join(hint_parts[:3]) if hint_parts else f"{len(env)} lines"
return env, body, hint
def env_context_html(
env_lines: list[str],
*,
tid_to_slug: dict[str, str],
lang: str,
hint: str,
) -> str:
"""Collapsible monitoring env context (default closed)."""
if not env_lines:
return ""
body = "".join(render_line(ln, tid_to_slug, None) for ln in env_lines)
label = html.escape(ui(lang, "env_context"))
title = html.escape(ui(lang, "env_context_title"))
hint_esc = html.escape(hint or "")
# native <details>: collapsed by default (no open attr)
return f"""<details class="env-context" id="mon-env-context">
<summary class="env-context-summary" title="{title}">
<span class="env-context-chevron" aria-hidden="true"></span>
<span class="env-context-label">{label}</span>
<span class="env-context-hint">{hint_esc}</span>
</summary>
<div class="env-context-body" role="region" aria-label="{label}">
{body} </div>
</details>
"""
def is_log_noise_line(ln: str) -> bool: def is_log_noise_line(ln: str) -> bool:
"""Drop git/update chatter that must not appear in public mon HTML.""" """Drop git/update chatter that must not appear in public mon HTML."""
s = ln.strip() s = ln.strip()
@ -701,26 +888,50 @@ def sticky_bar_html(
else: else:
status_txt = ui(lang, "ok") status_txt = ui(lang, "ok")
if first_err_href and n_err: # Click error/warn stats → filter console to matching lines only (v1.11.0).
# Hash #filter-error / #filter-warn; #first-error / #first-warn also activate filter.
if n_err:
err_lbl = ui(lang, "error_one", n=n_err) if n_err == 1 else ui(lang, "error_many", n=n_err) err_lbl = ui(lang, "error_one", n=n_err) if n_err == 1 else ui(lang, "error_many", n=n_err)
err_href = first_err_href or "#filter-error"
# Prefer filter hash; keep first-error as secondary scroll target via JS
err_stat = ( err_stat = (
f'<a class="stat err" href="{html.escape(first_err_href)}" ' f'<a class="stat err" href="#filter-error" data-filter="error" '
f'title="{html.escape(ui(lang, "jump_first_error"))}">{html.escape(err_lbl)}</a>' f'role="button" aria-pressed="false" '
f'data-first="{html.escape(err_href)}" '
f'title="{html.escape(ui(lang, "filter_errors"))}">{html.escape(err_lbl)}</a>'
) )
else: else:
err_lbl = ui(lang, "error_many", n=n_err) err_lbl = ui(lang, "error_many", n=n_err)
err_stat = f'<span class="stat err muted">{html.escape(err_lbl)}</span>' err_stat = f'<span class="stat err muted">{html.escape(err_lbl)}</span>'
if first_warn_href and n_warn: if n_warn:
warn_lbl = ui(lang, "warn_one", n=n_warn) if n_warn == 1 else ui(lang, "warn_many", n=n_warn) warn_lbl = ui(lang, "warn_one", n=n_warn) if n_warn == 1 else ui(lang, "warn_many", n=n_warn)
warn_href = first_warn_href or "#filter-warn"
warn_stat = ( warn_stat = (
f'<a class="stat warn" href="{html.escape(first_warn_href)}" ' f'<a class="stat warn" href="#filter-warn" data-filter="warn" '
f'title="{html.escape(ui(lang, "jump_first_warn"))}">{html.escape(warn_lbl)}</a>' f'role="button" aria-pressed="false" '
f'data-first="{html.escape(warn_href)}" '
f'title="{html.escape(ui(lang, "filter_warns"))}">{html.escape(warn_lbl)}</a>'
) )
else: else:
warn_lbl = ui(lang, "warn_many", n=n_warn) warn_lbl = ui(lang, "warn_many", n=n_warn)
warn_stat = f'<span class="stat warn muted">{html.escape(warn_lbl)}</span>' warn_stat = f'<span class="stat warn muted">{html.escape(warn_lbl)}</span>'
filter_chip = (
f'<span class="filter-chip" id="filter-chip" hidden>'
f'<span class="filter-chip-label" id="filter-chip-label"></span>'
f'<button type="button" class="filter-clear" id="filter-clear" '
f'title="{html.escape(ui(lang, "filter_clear"))}">'
f'{html.escape(ui(lang, "filter_show_all"))}</button>'
f"</span>"
)
# data attributes for JS i18n
filter_i18n = (
f'data-i18n-filter-err="{html.escape(ui(lang, "filter_active_err"))}" '
f'data-i18n-filter-warn="{html.escape(ui(lang, "filter_active_warn"))}" '
f'data-i18n-filter-clear="{html.escape(ui(lang, "filter_clear"))}"'
)
other = "" other = ""
if link_other: if link_other:
other = ( other = (
@ -758,7 +969,7 @@ def sticky_bar_html(
return f""" return f"""
<div class="sticky-bar sticky-{level}" id="status-bar" role="status" <div class="sticky-bar sticky-{level}" id="status-bar" role="status"
data-errors="{n_err}" data-warnings="{n_warn}" data-level="{level}" data-errors="{n_err}" data-warnings="{n_warn}" data-level="{level}"
data-scope-kind="{scope_kind}"> data-scope-kind="{scope_kind}" {filter_i18n}>
<div class="sticky-bar-row primary"> <div class="sticky-bar-row primary">
<span class="status-pill sticky-{level}">{html.escape(status_txt)}</span> <span class="status-pill sticky-{level}">{html.escape(status_txt)}</span>
<span class="host">{html.escape(page_label)} · {html.escape(hostname)}</span> <span class="host">{html.escape(page_label)} · {html.escape(hostname)}</span>
@ -766,6 +977,7 @@ def sticky_bar_html(
{err_stat} {err_stat}
<span class="sep">·</span> <span class="sep">·</span>
{warn_stat} {warn_stat}
{filter_chip}
<span class="sep">·</span> <span class="sep">·</span>
{version_html} {version_html}
<span class="sep">·</span> <span class="sep">·</span>
@ -849,6 +1061,93 @@ STICKY_CSS = """
.stat.warn { color: var(--warn); border-color: #664; background: #1a1608; } .stat.warn { color: var(--warn); border-color: #664; background: #1a1608; }
.stat.muted { opacity: 0.55; font-weight: 600; } .stat.muted { opacity: 0.55; font-weight: 600; }
a.stat:hover { text-decoration: underline; filter: brightness(1.15); } a.stat:hover { text-decoration: underline; filter: brightness(1.15); }
a.stat[aria-pressed="true"] {
box-shadow: 0 0 0 1px currentColor inset;
filter: brightness(1.2);
}
.filter-chip {
display: inline-flex; align-items: center; gap: 6px;
font-size: 11px; font-weight: 600;
padding: 1px 4px 1px 8px; border-radius: 3px;
border: 1px solid #456; background: #0a1520; color: #9cf;
}
.filter-chip[hidden] { display: none !important; }
body.filter-error .filter-chip { border-color: #a33; background: #1a0a0a; color: var(--err); }
body.filter-warn .filter-chip { border-color: #a68b2d; background: #1a1608; color: var(--warn); }
.filter-clear {
font: inherit; font-size: 11px; font-weight: 700;
cursor: pointer; color: inherit;
background: #0006; border: 1px solid #555; border-radius: 3px;
padding: 1px 8px;
}
.filter-clear:hover { border-color: #aaa; background: #222; }
/* Console filter: sticky + overviews stay; hide non-matching log lines */
body.filter-error #mon-console .line:not(.error):not(.blocker) { display: none !important; }
body.filter-warn #mon-console .line:not(.warn) { display: none !important; }
body.filter-error .env-context,
body.filter-warn .env-context { display: none !important; }
body.filter-warn .err-top { display: none; }
/* v1.12.0: collapsible monitoring env context (default closed) */
.env-context {
margin: 0 0 12px;
max-width: 1200px;
border: 1px solid #333;
border-radius: 4px;
background: #0a0a0a;
}
.env-context-summary {
display: flex;
align-items: center;
gap: 8px;
cursor: pointer;
list-style: none;
padding: 8px 12px;
font-size: 12px;
color: #ccc;
user-select: none;
}
.env-context-summary::-webkit-details-marker { display: none; }
.env-context-summary:hover { background: #121212; color: #fff; }
.env-context[open] .env-context-summary {
border-bottom: 1px solid #2a2a2a;
background: #101010;
}
.env-context-chevron {
display: inline-block;
width: 0; height: 0;
border-left: 5px solid #888;
border-top: 4px solid transparent;
border-bottom: 4px solid transparent;
transition: transform 0.12s ease;
flex-shrink: 0;
}
.env-context[open] .env-context-chevron {
transform: rotate(90deg);
border-left-color: #ccc;
}
.env-context-label {
font-weight: 700;
letter-spacing: 0.03em;
text-transform: uppercase;
font-size: 11px;
color: #9ab;
white-space: nowrap;
}
.env-context-hint {
color: #777;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
min-width: 0;
}
.env-context-body {
padding: 8px 12px 10px;
white-space: pre-wrap;
word-break: break-word;
font-size: 12px;
max-height: min(50vh, 28rem);
overflow: auto;
}
.version-link { .version-link {
font-weight: 700; font-size: 12px; color: var(--info); font-weight: 700; font-size: 12px; color: var(--info);
text-decoration: none; padding: 1px 8px; border-radius: 3px; text-decoration: none; padding: 1px 8px; border-radius: 3px;
@ -1015,6 +1314,107 @@ STICKY_JS = """
if (localStorage.getItem(key) === "1") setOpen(true); if (localStorage.getItem(key) === "1") setOpen(true);
} catch (e) {} } catch (e) {}
} }
// v1.11.0: click error/warn stats show only those log lines; sticky + overviews stay
var bar = document.getElementById("status-bar");
var chip = document.getElementById("filter-chip");
var chipLabel = document.getElementById("filter-chip-label");
var clearBtn = document.getElementById("filter-clear");
var consoleEl = document.getElementById("mon-console");
function currentFilter() {
if (document.body.classList.contains("filter-error")) return "error";
if (document.body.classList.contains("filter-warn")) return "warn";
return "";
}
function setFilter(mode, opts) {
opts = opts || {};
document.body.classList.remove("filter-error", "filter-warn");
var errA = document.querySelector('a.stat.err[data-filter="error"]');
var warnA = document.querySelector('a.stat.warn[data-filter="warn"]');
if (errA) errA.setAttribute("aria-pressed", "false");
if (warnA) warnA.setAttribute("aria-pressed", "false");
if (mode === "error") {
document.body.classList.add("filter-error");
if (errA) errA.setAttribute("aria-pressed", "true");
} else if (mode === "warn") {
document.body.classList.add("filter-warn");
if (warnA) warnA.setAttribute("aria-pressed", "true");
}
if (chip && chipLabel && bar) {
if (mode === "error") {
chip.hidden = false;
chipLabel.textContent = bar.getAttribute("data-i18n-filter-err") || "Filter: errors only";
} else if (mode === "warn") {
chip.hidden = false;
chipLabel.textContent = bar.getAttribute("data-i18n-filter-warn") || "Filter: warnings only";
} else {
chip.hidden = true;
chipLabel.textContent = "";
}
}
if (opts.updateHash !== false) {
var want = mode === "error" ? "#filter-error"
: mode === "warn" ? "#filter-warn" : "";
if (want) {
if (location.hash !== want) {
try { history.replaceState(null, "", want); } catch (e) { location.hash = want; }
}
} else if (location.hash === "#filter-error" || location.hash === "#filter-warn"
|| location.hash === "#first-error" || location.hash === "#first-warn") {
try { history.replaceState(null, "", location.pathname + location.search); } catch (e) {}
}
}
if (opts.scrollFirst && mode) {
var id = mode === "error" ? "first-error" : "first-warn";
var el = document.getElementById(id);
if (el && el.scrollIntoView) {
setTimeout(function () { el.scrollIntoView({ block: "start", behavior: "smooth" }); }, 30);
} else if (consoleEl) {
var sel = mode === "error" ? ".line.error, .line.blocker" : ".line.warn";
var first = consoleEl.querySelector(sel);
if (first && first.scrollIntoView) {
setTimeout(function () { first.scrollIntoView({ block: "start", behavior: "smooth" }); }, 30);
}
}
}
}
function toggleFilter(mode) {
if (currentFilter() === mode) setFilter("", { updateHash: true });
else setFilter(mode, { updateHash: true, scrollFirst: true });
}
document.querySelectorAll("a.stat[data-filter]").forEach(function (a) {
a.addEventListener("click", function (ev) {
ev.preventDefault();
var mode = a.getAttribute("data-filter");
if (mode === "error" || mode === "warn") toggleFilter(mode);
});
});
if (clearBtn) {
clearBtn.addEventListener("click", function () {
setFilter("", { updateHash: true });
});
}
function applyHash() {
var h = (location.hash || "").replace(/^#/, "");
if (h === "filter-error" || h === "first-error") {
setFilter("error", { updateHash: false, scrollFirst: h === "first-error" });
} else if (h === "filter-warn" || h === "first-warn") {
setFilter("warn", { updateHash: false, scrollFirst: h === "first-warn" });
} else if (currentFilter()) {
// keep filter if hash is something else (e.g. err-slug)
}
}
applyHash();
window.addEventListener("hashchange", applyHash);
})(); })();
</script> </script>
""" """
@ -1044,7 +1444,11 @@ def build_html(
while raw_lines and not raw_lines[-1].strip(): while raw_lines and not raw_lines[-1].strip():
raw_lines.pop() raw_lines.pop()
n_err, n_warn, first_err_i, first_warn_i = count_status(raw_lines) # Split monitoring env context (collapsible, default closed) from check log
env_lines, body_raw, env_hint = extract_env_context(raw_lines)
# Sticky counts from full log; first-error anchors index into body only
n_err, n_warn, _, _ = count_status(raw_lines)
_, _, first_err_i, first_warn_i = count_status(body_raw)
errors = extract_errors(raw_lines) errors = extract_errors(raw_lines)
err_slugs: dict[str, str] = {} err_slugs: dict[str, str] = {}
@ -1062,7 +1466,7 @@ def build_html(
# Build body with anchors for first error/warn + per-error slugs # Build body with anchors for first error/warn + per-error slugs
body_lines: list[str] = [] body_lines: list[str] = []
claimed_err: set[str] = set() claimed_err: set[str] = set()
for idx, ln in enumerate(raw_lines): for idx, ln in enumerate(body_raw):
extra: list[str] = [] extra: list[str] = []
if first_err_i is not None and idx == first_err_i: if first_err_i is not None and idx == first_err_i:
extra.append("first-error") extra.append("first-error")
@ -1153,6 +1557,9 @@ def build_html(
page_lang = lang page_lang = lang
footer_txt = ui(lang, "footer") footer_txt = ui(lang, "footer")
env_html = env_context_html(
env_lines, tid_to_slug=tid_to_slug, lang=lang, hint=env_hint
)
bar = sticky_bar_html( bar = sticky_bar_html(
generated_at=generated_at, generated_at=generated_at,
generated_iso=generated_iso, generated_iso=generated_iso,
@ -1285,6 +1692,7 @@ footer {{
{bar} {bar}
<main> <main>
{err_nav} {err_nav}
{env_html}
<div class="console" id="mon-console"> <div class="console" id="mon-console">
{"".join(body_lines)} {"".join(body_lines)}
</div> </div>

View file

@ -104,6 +104,9 @@ Phases:
mail MX/SMTP/IMAP for Taler mail (firefly, pixel/TSA, catalogued domains) mail MX/SMTP/IMAP for Taler mail (firefly, pixel/TSA, catalogued domains)
monpages public monitoring HTML via FQDN (obligatory ERROR; GOA full inventory, FP only FP) monpages public monitoring HTML via FQDN (obligatory ERROR; GOA full inventory, FP only FP)
uses MON_HOSTS + HTML_URL_OK (same as host-agent); v1.3.1+ uses MON_HOSTS + HTML_URL_OK (same as host-agent); v1.3.1+
devtesting fake-franken CHF via rusty.taler-ops.ch (taler-devtesting)
geniban + fake-incoming; needs SSH Host DEVTESTING_SSH (key to devtesting@rusty)
franken|fake-franken aliases for devtesting
all urls + inside + versions + sanity + e2e (SSH phases only on koopa) all urls + inside + versions + sanity + e2e (SSH phases only on koopa)
full all + server + ladder + auth401 (maximum; long-running, needs secrets) full all + server + ladder + auth401 (maximum; long-running, needs secrets)
NOTE: surface is never included in all/full — pass it explicitly NOTE: surface is never included in all/full — pass it explicitly
@ -245,7 +248,7 @@ while [ $# -gt 0 ]; do
CURRENCY_OVERRIDE="$2"; shift 2 CURRENCY_OVERRIDE="$2"; shift 2
;; ;;
--no-probe) NO_PROBE=1; shift ;; --no-probe) NO_PROBE=1; shift ;;
urls|inside|versions|sanity|server|e2e|ladder|goa-ladder|auth401|aptdeploy|apt-deploy|apt_src|surface|ecosystem|mattermost|mail|monpages|pages|all|full) PHASES+=("$1"); shift ;; urls|inside|versions|sanity|server|e2e|ladder|goa-ladder|auth401|aptdeploy|apt-deploy|apt_src|surface|ecosystem|mattermost|mail|monpages|pages|devtesting|franken|fake-franken|fake_franken|all|full) PHASES+=("$1"); shift ;;
*) *)
# bare domain shorthand: ./taler-monitoring.sh taler.net # bare domain shorthand: ./taler-monitoring.sh taler.net
if [[ "$1" == *.* && "$1" != *://* && "$1" != -* ]]; then if [[ "$1" == *.* && "$1" != *://* && "$1" != -* ]]; then
@ -392,28 +395,34 @@ if [ "${#PHASES[@]}" -eq 0 ]; then
PHASES=(urls) PHASES=(urls)
fi fi
# Rough expected check counts for progress % (override: PROGRESS_TOTAL=N). # Global expected check count for progress bar done/total (override: PROGRESS_TOTAL=N).
# Prefer slightly high — lib.sh rebalances if still short (never shows done>total). # One estimate for the whole run — phase summary() must NOT reset this mid-run.
# Prefer slightly high; lib.sh only grows if short (never shrinks until progress_finish).
if [ "${PROGRESS_TOTAL:-0}" = "0" ] || [ -z "${PROGRESS_TOTAL:-}" ]; then if [ "${PROGRESS_TOTAL:-0}" = "0" ] || [ -z "${PROGRESS_TOTAL:-}" ]; then
_pt=0 _pt=0
for p in "${PHASES[@]}"; do for p in "${PHASES[@]}"; do
case "$p" in case "$p" in
urls) _pt=$((_pt + 95)) ;; # +qr + webui SPA fingerprints urls) _pt=$((_pt + 95)) ;; # +qr + webui SPA fingerprints
inside) _pt=$((_pt + 30)) ;; inside) _pt=$((_pt + 30)) ;;
versions) _pt=$((_pt + 25)) ;; versions) _pt=$((_pt + 35)) ;; # + disk probes can be many
sanity) _pt=$((_pt + 35)) ;; sanity) _pt=$((_pt + 35)) ;;
server) _pt=$((_pt + 20)) ;; server) _pt=$((_pt + 20)) ;;
# e2e: ATM ladder emits many INFO lines (coins before/after each note) # e2e: ATM ladder emits many INFO lines (coins before/after each note)
e2e) _pt=$((_pt + 240)) ;; e2e) _pt=$((_pt + 240)) ;;
ladder|goa-ladder) _pt=$((_pt + 120)) ;; ladder|goa-ladder) _pt=$((_pt + 120)) ;;
auth401) _pt=$((_pt + 70)) ;; auth401) _pt=$((_pt + 70)) ;;
aptdeploy) _pt=$((_pt + 20)) ;; aptdeploy) _pt=$((_pt + 40)) ;;
surface|ecosystem) _pt=$((_pt + 80)) ;; surface|ecosystem) _pt=$((_pt + 120)) ;;
monpages|pages) _pt=$((_pt + 30)) ;; monpages|pages) _pt=$((_pt + 20)) ;;
mattermost) _pt=$((_pt + 25)) ;; mattermost) _pt=$((_pt + 25)) ;;
mail) _pt=$((_pt + 40)) ;; mail) _pt=$((_pt + 40)) ;;
devtesting|franken|fake-franken|fake_franken) _pt=$((_pt + 12)) ;;
esac esac
done done
# monpages post-check after htmlify can add another inventory pass
case " ${PHASES[*]} " in
*" monpages "*|*" pages "*) _pt=$((_pt + 12)) ;;
esac
set_progress_total "$_pt" set_progress_total "$_pt"
unset _pt unset _pt
fi fi
@ -500,7 +509,8 @@ if [ "${RUN_TIMEOUT:-0}" -eq 0 ]; then
else else
printf ' run_timeout %ss wall clock (RUN_TIMEOUT=; 0=unlimited)\n' "$RUN_TIMEOUT" printf ' run_timeout %ss wall clock (RUN_TIMEOUT=; 0=unlimited)\n' "$RUN_TIMEOUT"
fi fi
printf ' progress total≈%s (set PROGRESS_TOTAL= to override; PROGRESS_OFF=1 to hide)\n' "${PROGRESS_TOTAL:-0}" printf ' progress %s/%s (global done/total · set PROGRESS_TOTAL= to override; PROGRESS_OFF=1 to hide)\n' \
"${PROGRESS_DONE:-0}" "${PROGRESS_TOTAL:-0}"
unset _INSIDE_ACCESS_HINT unset _INSIDE_ACCESS_HINT
chmod +x "$ROOT"/check_*.sh 2>/dev/null || true chmod +x "$ROOT"/check_*.sh 2>/dev/null || true
@ -532,6 +542,9 @@ for p in "${PHASES[@]}"; do
mattermost) run_phase mattermost "$ROOT/check_mattermost.sh" || ec=1 ;; mattermost) run_phase mattermost "$ROOT/check_mattermost.sh" || ec=1 ;;
mail) run_phase mail "$ROOT/check_mail.sh" || ec=1 ;; mail) run_phase mail "$ROOT/check_mail.sh" || ec=1 ;;
monpages|pages) run_phase monpages "$ROOT/check_monitoring_pages.sh" || ec=1 ;; monpages|pages) run_phase monpages "$ROOT/check_monitoring_pages.sh" || ec=1 ;;
devtesting|franken|fake-franken|fake_franken)
run_phase devtesting "$ROOT/check_devtesting.sh" || ec=1
;;
esac esac
done done
@ -567,4 +580,14 @@ if [ "$RUN_TIMED_OUT" = "1" ]; then
"$RUN_TIMEOUT" "$elapsed" "$RUN_TIMEOUT" "$elapsed"
fi fi
# Snap progress bar to real global count (done/total) once for the whole run.
# Reload TALER_MON_STATE — check_*.sh update done/total in a separate process.
if type _mon_state_load >/dev/null 2>&1; then
_mon_state_load
fi
if type progress_finish >/dev/null 2>&1; then
printf '\n'
progress_finish
fi
exit "$ec" exit "$ec"