hernani/taler-monitoring
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
taler-monitoring/README.md
Hernâni Marques efb9047a67
release 1.18.3: Bank withdraw wording (was ATM for CHF-style note amounts) 
Rename user-facing e2e "ATM withdraw" to "Bank withdraw". The ladder
amounts (20/50/100/200/…) were labeled ATM because they match typical
CHF cash-note sizes (Bankomat), not because anything hits a physical ATM.
Keep E2E_ATM_MAX as alias of E2E_BANK_WITHDRAW_MAX; keep e2e.atm-* group ids.
2026-07-19 09:06:06 +02:00

449 lines
21 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# taler-monitoring
Checks for **GNU Taler** stacks (bank, exchange, merchant; optional surface inventory).
| Mode | What |
|------|------|
| **Outside-only** | Public HTTPS from anywhere — no SSH, no host-agent. Phases such as `urls`, `monpages`, `surface`, … |
| **Inside / local** | Where you operate the stack: podman/SSH (`inside`, `versions`, `e2e`, `ladder`, …) |
| **Host-agent + HTML** | Optional: timers run the suite and publish sticky-bar HTML (see simplified page layout below) |
Outside-only example: `./taler-monitoring.sh -d demo.taler.net urls`
Surface (ecosystem + mail + mattermost + versions): `./taler-monitoring.sh surface mattermost mail versions`
Full local GOA example: `./taler-monitoring.sh -d hacktivism.ch urls inside versions`
Repo: https://git.hacktivism.ch/hernani/taler-monitoring · tree: `~/src/taler-monitoring`
Version: `./taler-monitoring.sh --ver` (or `--version` / `-V`)
## Monitoring pages — public URL families (only these three)
**`taler-monitoring` is the suite name (git / CLI), not a public URL path.**
There is **no** mon page at `/taler-monitoring` or `/taler-monitoring/`.
Exactly **three** first-class public path families (each must work **bare and with trailing slash**; reverse-proxy should `redir` bare → slash). Error companions use `…_err` when the last run failed:
| Family | Paths (bare + slash) | Where | Purpose |
|--------|----------------------|--------|---------|
| **monitoring** | `/monitoring` · `/monitoring/` · optional `/monitoring_err` · `/monitoring_err/` | Per **landing** host (GOA bank/exchange/taler · FP · mytops stage, …) | Stack report for that host |
| **taler-monitoring-surface** | `/taler-monitoring-surface` · `/…/` · optional `…_err` | **only** `taler.hacktivism.ch` | Ecosystem surface (ports, nmap, mail, mattermost, versions) |
| **taler-monitoring-aptdeploy** | `/taler-monitoring-aptdeploy` · `/…/` · optional `…_err` | **only** `taler.hacktivism.ch` | apt-src merchant deploy tests (podman on koopa) |
**Not** public mon pages: `/taler-monitoring-mail*`, `/taler-monitoring-mattermost*`, bare `/taler-monitoring` — mail/mm are folded into **surface**.
Related:
- **Phase `monpages`:** obligatory **ERROR** if a listed page is missing, bootstrap stub, not suite HTML, or **content markers fail**. Bare paths that fall through to merchant **code 21** are ERROR (not “OK because slash works”).
- **Publish:** staging `~/monitoring-sites-staging` → `DEPLOY_WWW_ROOT` + reverse-proxy handles for **only** the three families above.
```bash
./taler-monitoring.sh monpages
./taler-monitoring.sh -d hacktivism.ch monpages
```
Host packages (koopa / runners): **[DEPENDENCIES.md](./DEPENDENCIES.md)**.
Host-agents (GOA + FP; shared report pipeline): **[host-agent/README.md](./host-agent/README.md)**.
Public HTML site-gen (same timeout/log/HTML defaults): **[site-gen/README.md](./site-gen/README.md)**.
FP stage host-agent: `host-agent/run-fp-stage-monitoring.sh` + `host-agent/env/stagepaysan.env.example` (see `host-agent/README.md`).
Report for the **GOA** stack with boxed severity badges and **grouped test IDs**
`area.group-NN` (e.g. `www.exchange-01`, `e2e.pay-03`):
| Tag | Meaning |
|-----|---------|
| `┌ OK ┐` | check passed |
| `┌ INFO ┐` | status / note |
| `┌ WARN ┐` | degraded but maybe not fatal |
| `┌ ERROR ┐` | component problem |
| `┌ BLOCKER ┐` | **withdraw/pay path cannot complete** because of this |
IDs reset per **group** inside an area — so “too many www tests” become
`www.exchange-*`, `www.bank-*`, `www.landing-*`, etc.
Catalog: **[TESTS.md](./TESTS.md)**.
Git / tags / suite tree: **[VERSIONS.md](./VERSIONS.md)**.
Optional / deprecated tooling: **[meta/](./meta/)** (firecuda, legacy site-gen, old mail/mm units).
GUI Android notes: **[android-test/GUI-AUTOMATION-NOTES.md](./android-test/GUI-AUTOMATION-NOTES.md)**.
CLI / wallet-cli recurring issues: **[CLI-AUTOMATION-NOTES.md](./CLI-AUTOMATION-NOTES.md)**.
```text
┌ OK ┐ www.exchange-01 exchange /config · HTTP 200
┌ BLOCKER┐ e2e.prereq-02 merchant HTTP 502
```
End of each phase: totals + list of **BLOCKERS** and **ERRORS** (ids included).
## Secrets (local GOA e2e)
**Do not commit passwords.** Suite tree: `~/src/taler-monitoring` · host ops (optional):
sibling `~/src/koopa/koopa-admin-secrets`.
### Local file (recommended on laptop)
```bash
cd ~/src/taler-monitoring
cp secrets.env.example secrets.env # gitignored
$EDITOR secrets.env
./taler-monitoring.sh e2e
```
See **[secrets.env.example](./secrets.env.example)** for every variable that
may appear (admin pass, merchant token, SECRETS_ROOT, progress knobs).
### Search order
1. Already-exported env (`E2E_BANK_ADMIN_PASS`, …)
2. `secrets.env` next to this suite (`MONITORING_SECRETS_ENV=` override)
3. `SECRETS_ROOT` / `KOOPA_ADMIN_SECRETS` file tree
4. Sibling `../koopa-admin-secrets/koopa/host-root`
5. `$HOME/src/koopa/koopa-admin-secrets/koopa/host-root`
6. `$HOME/.config/taler-landing/bank-admin-password.txt`
7. SSH koopa `/root/…` or bank container
### Line IDs + progress
Each check prints **global** and **grouped** ids:
```text
┌ OK ┐ #014 www.exchange-03 exchange /config · HTTP 200
┌ WARN ┐ #015 www.stats-02 stats age soft · yellow badge (not red)
┌ 42% ┐ ████████░░░░░░░░░░░░░░░░░░░░ 14/55
```
- `#014` — run-wide counter (whole `./taler-monitoring.sh` invocation)
- `www.exchange-03` — area.group-NN
- **Colours (left badge + label):**
- **OK** green · **INFO** blue/cyan · **WARN** yellow (never red)
- **ERROR** red · **BLOCKER** magenta
- Progress: badge shows **percent** (not “PROG”); bar fills **left → right**;
`done/total` on the right. Estimate auto-grows if short. Override:
`PROGRESS_TOTAL=N` / `PROGRESS_OFF=1`
- **QR (urls):** harvest `taler://` / `payto://` (+ app store `data-qr-url`) from landings and
`demo-withdraw.json` / `auto-account.json` → validate form → `qrencode` PNG → `zbarimg`
decode must match. Requires packages **`qrencode`** + **`zbar-tools`**.
## Commands
```bash
# Local GOA stack (may use SSH for inside/e2e)
./taler-monitoring.sh # urls + inside + versions + e2e
./taler-monitoring.sh inside # containers on koopa
./taler-monitoring.sh versions # deb.taler.net + package versions vs trixie
./taler-monitoring.sh sanity
./taler-monitoring.sh e2e
# FrancPaysan STAGE (TESTPAYSAN) — like hacktivism breadth:
# default phases: urls + inside + versions + e2e
./taler-monitoring.sh -d stage.lefrancpaysan.ch
./taler-monitoring.sh -d stage.lefrancpaysan.ch inside # stagepaysan only
# inside: low-priv SSH francpaysan-stage-user (User stagepaysan) → podman
# containers stage-lfp-{bank,exchange-ansible,merchant} · ports 9032/9031/9030
# stats: outside-in public …/intro/stats.json (age via generated_at_unix)
# Admin credit for e2e: E2E_BANK_ADMIN_PASS=… or stage host secrets
# (bank-admin-password.txt under /mnt/data/stagepaysan/bank/secrets/)
# Override: INSIDE_SSH=… INSIDE_PROFILE=stage-lfp
# Other domains — public HTTPS only (pass e2e/inside explicitly)
./taler-monitoring.sh -d taler.net
./taler-monitoring.sh --domain taler-ops.ch
./taler-monitoring.sh -d my.taler-ops.ch urls
./taler-monitoring.sh -d demo.taler.net urls
./taler-monitoring.sh taler.net # bare domain = same as -d
# Explicit endpoints (any mix; override profile or skip -d)
./taler-monitoring.sh --bank https://bank.demo.taler.net \
--exchange https://exchange.demo.taler.net \
--merchant https://backend.demo.taler.net --currency KUDOS urls
# Landing stats.json freshness (outside-in; age via generated_at_unix)
STATS_STALE_SECS=900 STATS_FAIL_SECS=3600 ./taler-monitoring.sh -d stage.lefrancpaysan.ch urls
# --- Full load (GOA / hacktivism) ---
# all = urls + inside + versions + sanity + e2e
# full = all + server + ladder + auth401 (long; needs secrets.env)
./taler-monitoring.sh -d hacktivism.ch full
./taler-monitoring.sh -d hacktivism.ch urls inside versions sanity server e2e ladder auth401
# SPA pin after selfbuild-webui.sh (container):
EXPECT_WEBUI_VERSION=1.6.11 EXPECT_WEBUI_OVERLAY=selfbuild-v1.6.11 \
WEBUI_OVERLAY_DENY='c778af04|master-11340' \
./taler-monitoring.sh -d hacktivism.ch urls
# Only 401 matrix (may create throwaway instance):
./taler-monitoring.sh -d hacktivism.ch auth401
# Stage FrancPaysan — durable mon401 account (password in francpaysan-secrets):
./taler-monitoring.sh -d stage.monnaie.lefrancpaysan.ch auth401
# secrets: ~/francpaysan-secrets/stage/instances/mon401.password
# ~/francpaysan-secrets/stage/default-instance-token.txt
# force throwaway create instead: AUTH401_PREFER_DURABLE=0 AUTH401_CREATE=1 …
```
### Landing `stats.json` (www.stats)
Public `GET …/intro/stats.json` must expose timestamps so monitoring can tell
whether the collector is still updating:
| Field | Role |
|-------|------|
| `generated_at_unix` | epoch seconds — **preferred** for age = now − unix |
| `generated_at` | ISO-8601 (fallback if unix missing) |
| `generated_at_human` | UI “updated” / footer string |
Also checks display payloads (bank-shape `withdraws` / accounts, or exchange-db /
merchant-db counters). Soft WARN if age > `STATS_STALE_SECS` (default 900);
ERROR if age > `STATS_FAIL_SECS` (default 3600; set `0` to disable).
Stage shared feed: same `gen_unix` on bank + exchange + monnaies.
## Domain profiles (`domains.conf`)
Each monitored stack must say **what is bank, exchange, and merchant-backend**.
Edit **[domains.conf](./domains.conf)** (or set `TALER_DOMAINS_CONF=`):
```text
# name bank exchange merchant-backend currency local [canonical]
taler-ops.ch bank.taler-ops.ch exchange.taler-ops.ch my.taler-ops.ch CHF 0
```
| Field | Meaning |
|-------|---------|
| **name** | `-d NAME` / `TALER_DOMAIN` match (add alias lines as needed) |
| **bank** | libeufin/bank public host or `https://…` |
| **exchange** | exchange public host or URL |
| **merchant** | merchant **backend** (SPA/API), e.g. `my.taler-ops.ch` |
| **currency** | `GOA` / `KUDOS` / `CHF` / … |
| **local** | `1` = koopa SSH stack; `0` = public only |
| **landing** | `1` = check `/intro` landings + assets; `0` = skip (TOPS / mytops) |
| **canonical** | optional label after alias (e.g. `my.taler-ops.ch` → `taler-ops.ch`) |
Then:
```bash
./taler-monitoring.sh -d taler-ops.ch urls
```
Overrides (always win after profile load):
```bash
./taler-monitoring.sh -d taler-ops.ch \
--merchant https://my.taler-ops.ch \
--exchange https://exchange.taler-ops.ch \
--bank https://bank.taler-ops.ch urls
```
Helpers in `lib.sh`: `set_taler_stack`, `load_domain_profile`, `apply_taler_domain`.
### Built-in profiles (see `domains.conf`)
| Domain / alias | Bank | Exchange | Merchant | Currency |
|----------------|------|----------|----------|----------|
| `hacktivism.ch` / `koopa` | bank.hacktivism.ch | exchange.hacktivism.ch | taler.hacktivism.ch | GOA |
| `taler.net` / `demo.taler.net` | bank.demo.taler.net | exchange.demo.taler.net | backend.demo.taler.net | KUDOS |
| `test.taler.net` | bank.test… | exchange.test… | backend.test… | **TESTKUDOS** · no landings |
| `lefrancpaysan.ch` | bank.… | exchange.… | **monnaie.…** (merchant) | report-only · no landings |
| **`stage.lefrancpaysan.ch`** | stage.bank.… | stage.exchange.… | **stage.monnaie.…** | **TESTPAYSAN** · landings · **inside via stagepaysan** |
| `taler-ops.ch` / `my.taler-ops.ch` | bank.taler-ops.ch | exchange.taler-ops.ch | **my.taler-ops.ch** | CHF · **no landings** |
| `stage.taler-ops.ch` | bank.stage… | exchange.stage… | **my.stage…** | CHF · **no landings** |
| unknown | bank.DOMAIN | exchange.DOMAIN | my.DOMAIN then probe | any · no landings |
**Landings:** GOA, stage LFP (`TESTPAYSAN`), and optionally demo use public `/intro` pages. **taler-ops.ch does not** — `CHECK_LANDING=0` skips intro crawl.
**SSH:**
| Stack | Host alias | User | Scope |
|-------|------------|------|--------|
| koopa / hacktivism | `koopa` | hernani | full host + containers |
| stage LFP | `francpaysan-stage-user` | **stagepaysan** | own podman only (no sudo / no prod data) |
Other domains: never SSH. Optional **e2e** aborts cleanly on login/KYC.
### E2E amounts (variable)
| | Local (koopa) | Remote |
|--|---------------|--------|
| **Bank withdraw** | 20 · 50 · 100 · 200 · **4200** (paivana) | 10 · 20 · 50 |
| **Pay ladder** | 0.01 … 10 | 0.01 … 1 |
| **Paivana** | HTTP paywall + template pay **GOA:4200** (`goa-shop` / `paivana`) | skipped |
```bash
./taler-monitoring.sh e2e
./taler-monitoring.sh -d taler.net urls e2e
# Amount ladder (withdraw 0 → … → max, then pay): same script for GOA + stage
./taler-monitoring.sh ladder # GOA / current domain
./taler-monitoring.sh -d stage.lefrancpaysan.ch ladder # TESTPAYSAN maxima
# Stage auto-reads bank max_wire_transfer_amount (e.g. 2000) and min denom (0.01).
# Explorer password: francpaysan-stage-user …/bank-explorer-password.txt (or EXP_PW=).
# Override: LADDER_MAX_AMOUNT=100 LADDER_STEPS=7 LADDER_PAY=0 …
# customize e2e:
E2E_WITHDRAW_VALUES="20 50 100" E2E_PAY_VALUES="0.05 1 5" ./taler-monitoring.sh e2e
E2E_VARIABLE=0 WITHDRAW_AMT=GOA:50 PAY_AMT=GOA:1 ./taler-monitoring.sh e2e # single fixed
# GOA shop catalog (local hacktivism): full list in E2E_SHOP_PRODUCTS; each run
# shuffles and pays E2E_SHOP_PICK_N products (default 2).
# E2E_SHOP_PRODUCTS lines: id|Product name|GOA:amount
# E2E_SHOP_PICK_N=2
# (landing QR = taler://pay-template/…/{id}; popup = live taler://pay after POST templates/{id})
# Paivana paywall (local GOA only):
# E2E_PAIVANA=1 (default) PAIVANA_PUBLIC=https://paivana.hacktivism.ch
# E2E_PAIVANA_TEMPLATE=paivana E2E_PAIVANA_AMOUNT=GOA:4200 E2E_PAIVANA_INSTANCE=goa-shop
# E2E_PAIVANA=0 # skip
# remote secrets:
# E2E_BANK_ADMIN_PASS=… E2E_MERCHANT_TOKEN=…
```
## Phases
| Phase | What |
|-------|------|
| **inside** | SSH koopa: processes, postgres, local /config,/keys, wirewatch, recent log ERRORs |
| **versions** | `deb.taler.net` reachable (InRelease/Packages/pool `.deb`); containers can reach it + have apt source; installed Taler packages vs **trixie** |
| **sanity** | bank · exchange · merchant sections (public + server) |
| **e2e** | account → credit → withdraw → wallet → confirm → order → pay |
| **ladder** | GOA withdraw: fixed **0** + low bands + **random high ranges** + fixed **max** |
```bash
# package suite (default trixie on deb.taler.net)
TALER_APT_SUITE=trixie ./taler-monitoring.sh versions
TALER_PKG_BEHIND=error ./taler-monitoring.sh versions # any behind = ERROR
```
E2E maps failures to blockers, e.g.:
- `bank-confirm HTTP 409` → wallet did not select exchange
- `no GOA balance` → wirewatch / transfer
- `create order failed` → merchant auth/instance
- `Alarm clock` during pay → usually missing `handle-uri --yes` or wrong pay URI (must be `…/instances/{inst}/{oid}/?c={token}` from merchant `taler_pay_uri`)
- `insufficient balance` → withdraw incomplete
## Performance (urls phase)
Outside-in HTTPS RTT for bank / exchange / merchant critical paths. Each probe prints
`HTTP … · N ms` on the `[OK]`/`[WARN]`/`[ERROR]` line; end of the block prints
`perf summary` (n / min / p50 / avg / max).
| Env | Default | Meaning |
|-----|---------|---------|
| `PERF_WARN_MS` | `8000` | WARN if latency ≥ this |
| `PERF_FAIL_MS` | `20000` | ERROR if latency ≥ this |
| `PERF_CURL_TIMEOUT` | `25` | curl max seconds per probe |
```bash
PERF_WARN_MS=3000 PERF_FAIL_MS=10000 ./taler-monitoring.sh urls
```
## Wallet coins (e2e · ladder)
At every relevant withdraw/pay step the wallet is snapped via `advanced dump-coins`.
Amounts use exchange **`alt_unit_names`** (Kilo-GOA, Mega-GOA, …) with the base
**GOA value in parentheses**:
```text
[INFO] coins after-bank-wd — amount_circ=2 Kilo-GOA (GOA:2000)
[INFO] coins after-bank-wd — denoms_in_circ 1 Kilo-GOA (GOA:1000)×2 (=2 Kilo-GOA (GOA:2000))
== ladder · rung 12 GOA:1000000 · 1 Mega-GOA (GOA:1000000) ==
```
Also: coin counts (in circulation / spent), status histogram, Δ vs previous snap.
History TSV: `$METRICS_DIR/coins-history.tsv`. Map loaded from
`${EXCHANGE_PUBLIC}/config` → `currency_specification.alt_unit_names`.
### Final statistics dashboard (e2e · ladder)
At the end of **e2e** and **ladder**, a boxed **FINAL STATISTICS** block prints:
| Section | Content |
|---------|---------|
| **Coins** | in circulation / spent counts + amounts + denoms (alt names) |
| **Money flow** | withdrawn events/amount, spent (paid), net (wd − spent) |
| **Tendency** | coin/amount trend over snaps (↑↓→), slope, last history rows |
| **Performance** | min / p50 / avg / max per phase; first-half vs second-half latency |
| **Load** | host loadavg + mem before/after; per-container RSS/CPU; tendencies |
Successful withdraw/pay steps append to `flow-withdrawn.txt` / `flow-spent.txt` (ladder also uses its TSVs).
## Load / memory (inside · e2e · ladder)
Host **loadavg**, **RAM used/avail**, and per-container **RSS / CPU / block I/O**
(plus postgres DB sizes and role RSS: libeufin, exchange, merchant, …) are
snapshotted via SSH whenever the stack is under load:
| Phase | When |
|-------|------|
| **inside** | end of container status |
| **e2e** | before work · after bank withdraws · after payments · after shop · e2e-end (+ delta) |
| **ladder** | before first rung · after last rung (+ delta + timing overall) |
```bash
# LAN down? force WAN DNAT host
KOOPA_SSH=koopa-external ./taler-monitoring.sh inside
# or rely on automatic fallback (KOOPA_SSH_FALLBACKS from env)
METRICS_LOAD=0 ./taler-monitoring.sh e2e # skip host probes
LADDER_LOAD=0 ./taler-monitoring.sh ladder
```
| Env | Default | Meaning |
|-----|---------|---------|
| `KOOPA_SSH` | `koopa` | primary SSH host |
| `KOOPA_SSH_FALLBACKS` | *(from env)* | tried if primary fails |
| `METRICS_LOAD` | `1` | `0` = skip all host/container load probes |
| `LADDER_LOAD` | `1` | `0` = skip load in ladder only |
| `LADDER_PAY` | `1` | `0` = skip payment ladder after withdraws |
| `LADDER_WITHDRAW_SCALE` | `1.5` | withdraw mids / pay mids ratio (funding headroom) |
| `LADDER_STEPS` | `23` | rungs for both withdraw and pay (0 + mids + max−1 + max) |
| `METRICS_LOAD_SSH_TIMEOUT` | `90` | seconds for remote load python |
## Needs
- SSH via `KOOPA_SSH` / `KOOPA_SSH_FALLBACKS` (env) (inside / load / sanity server bits)
- secrets under `koopa-admin-secrets/...` for e2e
- `taler-wallet-cli` for e2e
## Languages (i18n) — **v1.2+**
Default **English**. Select language with `--lang en|fr` or `TALER_MON_LANG`.
FrancPaysan stacks auto-select **French** (`*lefrancpaysan*`). See `VERSIONS.md` and `i18n.sh`.
## Public monitoring pages via FQDN (v1.3.1+ / obligatory v1.7.6+ / layout v1.8.0+)
Phase **`monpages`** is **obligatory**: missing or non-suite HTML is **ERROR**.
Soft mode only with `MONPAGES_REQUIRE_PUBLIC=0` (escape hatch).
| Stack | What monpages expects (v1.9.0) |
|-------|--------------------------------|
| **GOA** | 3× `/monitoring/` + **surface** (existence **and** top/bottom/content markers) |
| **FrancPaysan** | Stack mon hosts `/monitoring/` with same content rules |
| **Nine landings (model)** | Each landing front: `/monitoring` + `/monitoring_err` (err when a failed run exists) |
```bash
./taler-monitoring.sh -d hacktivism.ch monpages
./taler-monitoring.sh -d lefrancpaysan.ch monpages
```
If pages are missing publicly: `sudo $APPLY_MONITORING_LIVE` on koopa
(FP: Infomaniak vhost for `/monitoring*`).
## Devtesting · fake-franken CHF (v1.13+)
Phase **`devtesting`** (aliases: `franken`, `fake-franken`) exercises the
Taler Operations **rusty** host (`taler-devtesting` forced command):
```bash
./taler-monitoring.sh -d stage.taler-ops.ch --currency CHF devtesting
```
Needs SSH to `devtesting@rusty.taler-ops.ch` (Host `taler-rusty-devtesting`).
Probes: CLI help, `geniban` (CH IBAN / synthetic debit funding), baseline `fake-incoming`,
optional **amount ladder** (`DEVTESTING_LADDER=1`, default **23** rungs up to **CHF:50000**).
Each rung reuses geniban debit payto so volume covers the max rung without a real bank withdraw.
`fake-incoming` CHF to exchange credit
IBAN. A nexus bounce *missing reserve public key* is **OK** for the plumbing
probe (no real wallet reserve required).
## Version on monitoring pages (v1.3+ / sticky bar)
The sticky bar shows the suite **tag** (e.g. `v1.11.0`) and links to that tag on Forgejo.
**v1.12.0:** Run header (**monitoring env context**: env_file, suite pin, domain/hosts/flags) is a collapsed `<details>` block above the console log — expand to inspect agent context.
**v1.11.0:** Click the sticky **error** or **warning** count to filter the console log to only those lines (other log lines hide; sticky bar, stats, and top overviews stay). Same via `#filter-error` / `#filter-warn` (or `#first-error` / `#first-warn`). Click again or **Show all** to clear.
Reference in a new issue View git blame Copy permalink