# firecuda-probe — external last-alive from koopa

**Why:** firecuda panics (e.g. 1GbE PCIe completion timeout 2026-09-20) and hard hangs leave the internal HB dead with the box. External ICMP+SSH from **koopa** distinguishes:

| Pattern | Likely meaning |
|---------|----------------|
| ICMP fail + SSH fail, long gap | hang, power cut, or NIC/kernel down |
| ICMP ok + SSH fail | userspace/sshd stuck, or max lock |
| both ok, HB stamp stale vs wall | rare clock skew / forced-cmd path oddity / HB LaunchDaemon stopped |
| both ok, HB fresh | alive |

Log is **hernani-readable** on koopa (`~/.local/state/firecuda-probe/`).
SSH key is **max-only** on firecuda, restricted to koopa LAN IP + forced `cat` of HB mirror.

Requires firecuda crash-debug HB live (`/usr/local/var/lib/firecuda/heartbeat.last`) — see `firecuda-admin-log/configs/crash-debug/`.

## Pieces

| Piece | Path |
|-------|------|
| Script | `scripts/firecuda-probe/koopa-firecuda-probe.sh` → `~/.local/bin/koopa-firecuda-probe` |
| Timer | `configs/systemd/user/koopa-firecuda-probe.{service,timer}` (~15 s) |
| Key | `~/.ssh/id_ed25519_firecuda_probe` (hernani on koopa) |
| Log | `~/.local/state/firecuda-probe/firecuda-probe.log` (+ `.last` / `.meta`) |
| SSH Host | `firecuda-probe` → `max@192.168.100.84` with that IdentityFile |

## Install (koopa as hernani) — Freigabe

Laptop SoT: `$HOME/git/admin-logs/local/koopa-admin-log`. Live mirror: `~/src/koopa/koopa-admin-log` (no git commit on koopa).

```bash
# on koopa after syncing the tree:
cd ~/src/koopa/koopa-admin-log
./scripts/firecuda-probe/install-firecuda-probe.sh --keygen
# copy the printed authorized_keys line → firecuda max (see below)
./scripts/firecuda-probe/install-firecuda-probe.sh   # units + first probe
```

Linger already yes for hernani → timer survives logout.

## firecuda max authorized_keys (one line)

After `--keygen`, install prints the exact line. Template:

```text
from="192.168.100.95",no-agent-forwarding,no-port-forwarding,no-X11-forwarding,no-pty,command="/bin/cat /usr/local/var/lib/firecuda/heartbeat.last" ssh-ed25519 <PUBKEY> koopa-firecuda-probe@hernani-koopa
```

- **User:** `max` only (not root).
- **from=:** only koopa LAN `192.168.100.95`.
- **forced command:** read-only HB mirror (world-readable after crash-debug install).
- Daylog: `firecuda-admin-log/2026/2026-09-20--pcie-lan-panic+crash-debug.md`.

## Verify

```bash
# koopa
systemctl --user list-timers koopa-firecuda-probe.timer
koopa-firecuda-probe status
koopa-firecuda-probe show
ssh firecuda-probe   # should print one HB line, no shell

# gaps after a crash: look for icmp=fail/ssh=fail with large gap_s=
```

## Uninstall

```bash
systemctl --user disable --now koopa-firecuda-probe.timer
rm -f ~/.local/bin/koopa-firecuda-probe \
  ~/.config/systemd/user/koopa-firecuda-probe.{service,timer}
systemctl --user daemon-reload
# optional: remove key + Host block + max authorized_keys line
```

## Do not

- Put this key on **root** authorized_keys.
- Enable timer / write authorized_keys without Freigabe.
- Expect this to fix PCIe/Ethernet panics — it only timestamps last-alive.