80 lines
3.1 KiB
Markdown
80 lines
3.1 KiB
Markdown
# firecuda-probe — external last-alive from koopa
|
||
|
||
**Why:** firecuda panics (e.g. 1GbE PCIe completion timeout 2026-09-20) and hard hangs leave the internal HB dead with the box. External ICMP+SSH from **koopa** distinguishes:
|
||
|
||
| Pattern | Likely meaning |
|
||
|---------|----------------|
|
||
| ICMP fail + SSH fail, long gap | hang, power cut, or NIC/kernel down |
|
||
| ICMP ok + SSH fail | userspace/sshd stuck, or max lock |
|
||
| both ok, HB stamp stale vs wall | rare clock skew / forced-cmd path oddity / HB LaunchDaemon stopped |
|
||
| both ok, HB fresh | alive |
|
||
|
||
Log is **hernani-readable** on koopa (`~/.local/state/firecuda-probe/`).
|
||
SSH key is **max-only** on firecuda, restricted to koopa LAN IP + forced `cat` of HB mirror.
|
||
|
||
Requires firecuda crash-debug HB live (`/usr/local/var/lib/firecuda/heartbeat.last`) — see `firecuda-admin-log/configs/crash-debug/`.
|
||
|
||
## Pieces
|
||
|
||
| Piece | Path |
|
||
|-------|------|
|
||
| Script | `scripts/firecuda-probe/koopa-firecuda-probe.sh` → `~/.local/bin/koopa-firecuda-probe` |
|
||
| Timer | `configs/systemd/user/koopa-firecuda-probe.{service,timer}` (~15 s) |
|
||
| Key | `~/.ssh/id_ed25519_firecuda_probe` (hernani on koopa) |
|
||
| Log | `~/.local/state/firecuda-probe/firecuda-probe.log` (+ `.last` / `.meta`) |
|
||
| SSH Host | `firecuda-probe` → `max@192.168.100.84` with that IdentityFile |
|
||
|
||
## Install (koopa as hernani) — Freigabe
|
||
|
||
Laptop SoT: `$HOME/git/admin-logs/local/koopa-admin-log`. Live mirror: `~/src/koopa/koopa-admin-log` (no git commit on koopa).
|
||
|
||
```bash
|
||
# on koopa after syncing the tree:
|
||
cd ~/src/koopa/koopa-admin-log
|
||
./scripts/firecuda-probe/install-firecuda-probe.sh --keygen
|
||
# copy the printed authorized_keys line → firecuda max (see below)
|
||
./scripts/firecuda-probe/install-firecuda-probe.sh # units + first probe
|
||
```
|
||
|
||
Linger already yes for hernani → timer survives logout.
|
||
|
||
## firecuda max authorized_keys (one line)
|
||
|
||
After `--keygen`, install prints the exact line. Template:
|
||
|
||
```text
|
||
from="192.168.100.95",no-agent-forwarding,no-port-forwarding,no-X11-forwarding,no-pty,command="/bin/cat /usr/local/var/lib/firecuda/heartbeat.last" ssh-ed25519 <PUBKEY> koopa-firecuda-probe@hernani-koopa
|
||
```
|
||
|
||
- **User:** `max` only (not root).
|
||
- **from=:** only koopa LAN `192.168.100.95`.
|
||
- **forced command:** read-only HB mirror (world-readable after crash-debug install).
|
||
- Daylog: `firecuda-admin-log/2026/2026-09-20--pcie-lan-panic+crash-debug.md`.
|
||
|
||
## Verify
|
||
|
||
```bash
|
||
# koopa
|
||
systemctl --user list-timers koopa-firecuda-probe.timer
|
||
koopa-firecuda-probe status
|
||
koopa-firecuda-probe show
|
||
ssh firecuda-probe # should print one HB line, no shell
|
||
|
||
# gaps after a crash: look for icmp=fail/ssh=fail with large gap_s=
|
||
```
|
||
|
||
## Uninstall
|
||
|
||
```bash
|
||
systemctl --user disable --now koopa-firecuda-probe.timer
|
||
rm -f ~/.local/bin/koopa-firecuda-probe \
|
||
~/.config/systemd/user/koopa-firecuda-probe.{service,timer}
|
||
systemctl --user daemon-reload
|
||
# optional: remove key + Host block + max authorized_keys line
|
||
```
|
||
|
||
## Do not
|
||
|
||
- Put this key on **root** authorized_keys.
|
||
- Enable timer / write authorized_keys without Freigabe.
|
||
- Expect this to fix PCIe/Ethernet panics — it only timestamps last-alive.
|