koopa-admin-log/scripts/firecuda-probe

firecuda-probe — external last-alive from koopa

Why: firecuda panics (e.g. 1GbE PCIe completion timeout 2026-09-20) and hard hangs leave the internal HB dead with the box. External ICMP+SSH from koopa distinguishes:

Pattern	Likely meaning
ICMP fail + SSH fail, long gap	hang, power cut, or NIC/kernel down
ICMP ok + SSH fail	userspace/sshd stuck, or max lock
both ok, HB stamp stale vs wall	rare clock skew / forced-cmd path oddity / HB LaunchDaemon stopped
both ok, HB fresh	alive

Log is hernani-readable on koopa (~/.local/state/firecuda-probe/). SSH key is max-only on firecuda, restricted to koopa LAN IP + forced cat of HB mirror.

Requires firecuda crash-debug HB live (/usr/local/var/lib/firecuda/heartbeat.last) — see firecuda-admin-log/configs/crash-debug/.

Pieces

Piece	Path
Script	scripts/firecuda-probe/koopa-firecuda-probe.sh → ~/.local/bin/koopa-firecuda-probe
Timer	configs/systemd/user/koopa-firecuda-probe.{service,timer} (~15 s)
Key	~/.ssh/id_ed25519_firecuda_probe (hernani on koopa)
Log	~/.local/state/firecuda-probe/firecuda-probe.log (+ .last / .meta)
SSH Host	firecuda-probe → max@192.168.100.84 with that IdentityFile

Install (koopa as hernani) — Freigabe

Laptop SoT: $HOME/git/admin-logs/local/koopa-admin-log. Live mirror: ~/src/koopa/koopa-admin-log (no git commit on koopa).

# on koopa after syncing the tree:
cd ~/src/koopa/koopa-admin-log
./scripts/firecuda-probe/install-firecuda-probe.sh --keygen
# copy the printed authorized_keys line → firecuda max (see below)
./scripts/firecuda-probe/install-firecuda-probe.sh   # units + first probe

Linger already yes for hernani → timer survives logout.

firecuda max authorized_keys (one line)

After --keygen, install prints the exact line. Template:

from="192.168.100.95",no-agent-forwarding,no-port-forwarding,no-X11-forwarding,no-pty,command="/bin/cat /usr/local/var/lib/firecuda/heartbeat.last" ssh-ed25519 <PUBKEY> koopa-firecuda-probe@hernani-koopa

• User: max only (not root).
• from=: only koopa LAN 192.168.100.95.
• forced command: read-only HB mirror (world-readable after crash-debug install).
• Daylog: firecuda-admin-log/2026/2026-09-20--pcie-lan-panic+crash-debug.md.

Verify

# koopa
systemctl --user list-timers koopa-firecuda-probe.timer
koopa-firecuda-probe status
koopa-firecuda-probe show
ssh firecuda-probe   # should print one HB line, no shell

# gaps after a crash: look for icmp=fail/ssh=fail with large gap_s=

Uninstall

systemctl --user disable --now koopa-firecuda-probe.timer
rm -f ~/.local/bin/koopa-firecuda-probe \
  ~/.config/systemd/user/koopa-firecuda-probe.{service,timer}
systemctl --user daemon-reload
# optional: remove key + Host block + max authorized_keys line

Do not

• Put this key on root authorized_keys.
• Enable timer / write authorized_keys without Freigabe.
• Expect this to fix PCIe/Ethernet panics — it only timestamps last-alive.