Source: This page and docs/case_studies/truenas-scale-corgicave-homelab-networking-ssh-case-study-2026-04.md are kept in sync; edit both when the narrative changes.

Case study: TrueNAS Scale (corgicave) — homelab networking, Mac client errno 49, vault memory, and SSH hardening (April 2026)

A full reconstruction of a multi-session troubleshooting arc: TrueNAS Scale Web UI and LAN access on a Minisforum N5 Pro-class NAS, an isolated office switch segment, Apple Silicon Mac mini (10G + monitor 2.5G), Thunderbolt console access, ClawQL memory_ingest / memory_recall usage, and final SSH enablement to prevent lock-out when the UI or Thunderbolt path fails.

Audience: Homelab operators running TrueNAS SCALE on compact hardware, macOS clients with Docker Desktop / Kubernetes, and anyone who wants a repeatable triage ladder (L1 physical → L3 IP → L4 TCP → HTTP/TLS → services) plus durable documentation via vault memory.

Related vault note (ingest target): Memory/truenas-scale-ui-and-lan-connectivity-corgicave.md (many dated sections; this case study is the canonical narrative).

1. Why this case study exists

Homelab NAS outages are expensive in time because symptoms stack. "UI down" might mean no route, wrong IP, firewall, TLS only, browser policy, mDNS, service crash, or client-side socket failure — all of which present differently. Single admin paths (only Thunderbolt console, only one browser, only Wi-Fi) amplify risk: one broken path becomes total lock-out. Cross-machine debugging (Cursor agent host vs Mac mini vs NAS) changes subnets and evidence unless each hop is labeled.

This session produced durable notes via memory_ingest under a stable title, so memory_recall can later answer "What did we learn about corgicave networking?" without re-deriving from chat scrollback.

2. System under test (inventory)

3. Initial symptoms and user actions (TrueNAS side)

Reported:

• Web UI not loading at http://192.168.0.123/ or https://corgicave.local/ui/signin.
• User restarted nginx, middlewared, and rebooted the NAS — UI still broken.

Inference at this layer: Blindly restarting app units when HTTP/S never completes can be necessary — but first, separate reachability from application. When TCP to 80/443 times out from some clients while TCP 22 refuses quickly, the picture is often a firewall/DROP on web ports or listeners not bound, which is a different problem from the NAS being powered off.

4. Phase A: Cursor agent host probes

Action: From the Cursor workspace host (not necessarily the Mac mini on the office switch), ping 192.168.0.123 and curl/nc probes.

Results:

• Ping: 100% loss from that environment.
• Follow-up probes: Inconclusive (background shell / incomplete capture).

Decision: ICMP loss is weak evidence here. Run TCP probes from a host known to be on the same segment as the NAS. Agent location matters — a laptop on 192.168.1.x default-routed elsewhere can show no path to 192.168.0.123 even when the homelab island is fine.

5. Phase B: memory_recall then memory_ingest

Action: Per ClawQL workspace rules, memory_recall with a TrueNAS-oriented query, then memory_ingest with structured insights and verbatim toolOutputs.

memory_recall outcome (one MCP session): Failed: Cannot scan vault: ENOENT ... '/vault/Memory'.

Decision: Triage continued without blocking on recall. The recall path must match the ingest vault root (CLAWQL_OBSIDIAN_VAULT_PATH or MCP config) — path mismatch was the root cause.

memory_ingest outcome: Succeeded, writing Memory/truenas-scale-ui-and-lan-connectivity-corgicave.md, then many appends (session IDs dated 2026-04-20).

Long homelab incidents lose detail across days. Appending under one stable title preserves a timeline without fragmenting Obsidian [[wikilinks]].

6. Phase C: Another Mac on a richer path

Context: A different Mac (multi-homed: 192.168.0.x on en0, 192.168.1.x on en1, etc.) probed 192.168.0.123.

Observations:

• netstat: Host route to .123, ARP 38:5:25:32:5d:8f on en0 — L2 neighbor present.
• curl to :80 / :443: Timeouts (~8s), not immediate refusal.
• nc to :22: Connection refused quickly — TCP stack alive, SSH not listening (or RST), not a silent black hole for all ports.

Inference: Web port behavior differed from :22 — consistent with DROP/listener patterns. The multi-homed Mac's en0 path had L2 reachability to the NAS, but validation still required the actual admin Mac mini on the office switch.

Decision: When multiple clients exist, label which host produced which capture.

7. Phase D: Thunderbolt console path

User correction: SSH was never enabled during the worst period. Thunderbolt to a monitor / dock provided local terminal on the NAS for systemctl, reboots, and service checks.

Thunderbolt regression: After switching the monitor's TB source away and back, TB display/console stopped. HDMI also showed no signal at times.

Inference: TB link training / monitor mux issues are common. Ethernet + IP remain the objective path for UI when video is flaky. Once any shell exists, enable SSH early so TB/HDMI aren't single points of failure.

8. Phase E: Homelab topology — the office island

User description:

• Mac mini + NAS on MS510TXUP; no cable to router → no internet on that segment.
• Planned TP-Link EAP770 backhaul later.

Inference: No WAN doesn't imply no LAN UI — RFC1918 + static IPs + mDNS/hosts are sufficient for browser access. Without a router DHCP table on that island, "check router lease" won't help for discovery; static addressing and a known IP are what matters.

Decision: Document the static IP plan on paper: NAS .123, Mac .100, mask /24, no gateway unless intentionally routed.

9. Phase F: Mac mini curl openssl errno 49 EADDRNOTAVAIL

Symptoms (Mac mini → 192.168.0.123):

• curl: Can't assign requested address, immediate failure in some runs.
• openssl s_client: BIO_connect: Can't assign requested address, errno=49.
• Chrome: ERR_ADDRESS_INVALID for https://192.168.0.123/....
• Intermittent success: nc to :443 succeeded once, while nc to :80 and curl still failed in other runs — inconsistent surface.

Parallel checks:

• route -n get 192.168.0.123 often showed en0 (correct interface for the 10G island).
• ifconfig en0: 192.168.0.100, 10G, active — yet connectx still failed in bad states.

Root cause family (client macOS):

1. Dual IPv4 on the same /24 (192.168.0.100 on en0 + 192.168.0.222 on USB LAN) → ambiguous source selection / connectx failures.
2. Tunnel / Network Extension bloat: netstat -rn showed many default routes via fe80::…%utun0–utun15. Docker Desktop + Kubernetes were prime suspects. Any filter/VPN-like extension fits the pattern.
3. External "Gemini narrative" in networknotes.txt claimed ping always succeeded and used kernel deadlock language — overstated vs measured evidence; reboot remained a valid hygiene step regardless.

Decisions:

• Collapse to one IPv4 on 192.168.0.0/24 on the NIC cabled to the switch (en0 / 10G).
• Quit Docker Desktop (and keep Kubernetes off) while validating NAS reachability.
• Reboot Mac mini (and later MacBook + NAS) when stuck in errno 49 loops.

Outcome that confirmed the ladder:

• utun count dropped (e.g. 9 after reboot + no Docker in one capture).
• ping: 0% loss, ~1 ms RTT.
• /usr/bin/nc -4 -zv -G 8 192.168.0.123 443: Connection … succeeded!
• TrueNAS Web UI reachable again; dashboard showed eno1 UP, 192.168.0.123, pools ONLINE.

How each output informed the next:

10. Phase G: SSH enablement and first-login friction

Goal: Avoid future TB-only or UI-only lock-out.

UI steps (TrueNAS SCALE 25.10):

1. System → Services → SSH — configure Allow Password Authentication / Password Login Groups as needed; Save; Start automatically ON; service Running.
2. Credentials → Users → truenas_admin → Edit — SSH password login enabled; ensure shell + home directory per docs.

First SSH attempt:

Permission denied (publickey).

Interpretation: sshd is up but password authentication was not effectively enabled at service level and/or user level, so the client offered publickey only.

Successful login:

ssh truenas_admin@corgicave.local

Observation: Client resolved corgicave.local to IPv6 link-local fe80::…%en0 — valid on-LAN path; 192.168.0.123 remains a good IPv4 target.

Decision: Prefer SSH keys long-term on trusted admin machines; keep password SSH policy aligned with 2FA settings (TrueNAS warns about global 2FA + password SSH).

11. Preventive runbook (condensed)

12. memory_recall on this topic

Example query:

TrueNAS Scale corgicave Minisforum N5 SSH nginx middlewared Mac mini errno49 utun MS510TXUP

Top hit: Memory/truenas-scale-ui-and-lan-connectivity-corgicave.md — keyword match on ingested sections (timeline, tool outputs, hypotheses).

For graph recall, add wikilinks in vault notes to this case study file title once mirrored in vault (optional).

13. memory_ingest of this case study

After publishing this markdown under docs/case_studies/ and the website route /case-studies/truenas-scale-corgicave-homelab, run memory_ingest with:

• title: stable, e.g. Case study: TrueNAS Scale corgicave homelab networking SSH 2026-04 (or append to existing TrueNAS note with append: true).
• insights: short abstract + pointer to this file path in repo / website URL.
• wikilinks: e.g. ["TrueNAS Scale UI and LAN connectivity (corgicave)"] to connect graph edges.

14. References

• Case study index: docs/case_studies/README.md
• Vault skill: .cursor/skills/clawql-vault-memory/SKILL.md
• TrueNAS SSH tutorial (SCALE 25.10): SSH service — iXsystems docs

15. Detailed chronological timeline (cross-thread)

This table merges Cursor + Gemini + user terminal work. "Gemini-only" means steps recorded in networknotes.txt but not verified in the Cursor transcript.

16. Appendix A: Command snippets (reference)

Mac mini — reachability block (after fixing dual-homing / Docker off):

ifconfig | grep -c '^utun' || true
ifconfig en0
route -n get 192.168.0.123
arp -n 192.168.0.123
ping -c 4 192.168.0.123
/usr/bin/nc -4 -zv -G 8 192.168.0.123 443

Bracketed paste pitfall (zsh): pasting from some terminals injects [200~ / ~] — causes zsh: bad pattern; retype or fix terminal paste mode.

SSH tests:

ssh -o StrictHostKeyChecking=accept-new truenas_admin@192.168.0.123
ssh truenas_admin@corgicave.local

17. Appendix B: Vault memory_ingest session IDs (2026-04-20)

Representative sessionId values appended to TrueNAS Scale UI and LAN connectivity (corgicave) (non-exhaustive):

• 2026-04-20-truenas-ui — initial triage + tool outputs.
• 2026-04-20-full-session-trace — full step trace + Gemini correction notes.
• 2026-04-20-enable-ssh-from-console — midclt / cli paths when UI down.
• 2026-04-20-minisforum-n5-pro-no-video-console — TB/HDMI local display.
• 2026-04-20-console-path-clarification-tb-not-ssh — TB vs SSH correction.
• 2026-04-20-homelab-topology-office-island — MS510TXUP island, no WAN.
• 2026-04-20-mac-mini-dual-nic-same-subnet — .100 + .222 same /24.
• 2026-04-20-mac-route-en0-eaddrnotavail — route vs curl mismatch phase.
• 2026-04-20-mac-mini-nc443-ok-curl80-eaddrnotavail — nc :443 success vs curl failures.
• 2026-04-20-mac-mini-utun-default-spam-eaddrnotavail — netstat utun spam hypothesis.
• 2026-04-20-post-reboot-still-unreachable — state after reboot wave.
• 2026-04-20-mac-mini-connectivity-restored-ping-nc443-ok — proof of path.
• 2026-04-20-resolution-ui-restored-dashboard — UI/dashboard confirmation.
• 2026-04-20-ssh-publickey-only-truenas_admin — publickey denial.
• 2026-04-20-ssh-working-truenas_admin — successful SSH session.

18. Appendix C: Evidence hygiene (Gemini vs measured)

19. Changelog

© Copyright 2026. All rights reserved. · ClawQL on GitHub