bigbox1 (Ryzen 5 4600G, B550 GAMING X V2, BIOS FC) hard-froze 5 times
in 16 days (Nov 4 19:07, Nov 16 01:22, Nov 17 02:03, Nov 17 22:59,
Nov 19 18:20 ET), always while idle, each time with no log line and an
"uncleanly shut down" journal on the next boot. That matches the known
Ryzen deep-idle (C6) freeze.
acpi_idle exposes C1 (MWAIT), C2 and C3 (lat 350us, the deep state);
processor.max_cstate=1 keeps cores out of C2/C3. Cost: a few watts
more at idle.
The runtime watchdog (common.dns, SP5100 TCO, 30s) was armed during the
Nov 19 boot and did not reset the box, so it is not a safety net here
and is left as is.
/sys/fs/pstore is root-only and holds the kernel's crash records from
the previous boot, the only trace a hard freeze leaves when the journal
is cut off. diag-pstore prints them via the existing NOPASSWD sudo list.
ABOUTME updated for both diagnix changes.
hermenix had no shell route to lab hosts, so a hard freeze (bigbox1,
Nov 19 18:21 ET) could only be read through slinuxm's warning-level
journal window, which misses the info-level tail before the crash.
- hosts.nix: a dedicated hermenix-diagnix key, separate from the
hermenix agenix key, accepted only from hl-fwdesk's addresses
(10.0.10.11, 10.0.101.101) and the guest address 10.77.0.2.
- default.nix: keyLine takes the key; new
skyg.nixos.common.diagnix.hermenix.enable (default true) adds the
hermenix key line with the same restrict,pty + slice limits.
- hl-fwdesk hermenix.nix: lanAccess to 10.0.10.0/24 port 22.
qbittorrent shares gluetun's network. When it starts before the VPN
tunnel is up it never reaches the swarm, and torrents sit on
"downloading metadata" until it is restarted by hand (seen 2026-11-19:
qbit up 12:37:40, tun0 only up 12:38:57 after a failed first server).
- qbittorrent depends_on gluetun with condition: service_healthy and
restart: true, so it waits for the tunnel and restarts with gluetun
- drop HEALTH_VPN_DURATION_INITIAL, which gluetun v3.41 reports as obsolete
Re-encrypted to the same recipients as secrets.nix: ari, minipc2, hermenix.
Add hardened systemd-resolved DNS with boot verification and journal recovery,
enabled only on specific hosts:
- hl-bigbox1, hl-bigbox2 (experienced DNS resilience issues)
- hl-minipc1, hl-minipc2, hl-minipc3 (minipc cluster)
- hl-fwdesk, hl-fws1 (desktop/workstation nodes)
Changes:
1. New module: modules/nixos/common/dns/default.nix
- systemd-resolved with LLMNR fallback and DNSSEC tolerance
- Hardened journald with persistent storage and auto-recovery
- Boot-time DNS verification service with 5 retries
- Systemd watchdog for automatic reboot on DNS failure
2. Default to false, enabled explicitly on target hosts only
3. Updated host configs to enable dns resilience module
bigbox2 runs nix-serve, which only serves its own store: every
'nix copy --to http://bigbox2.lab.internal:5000' from the post-build hook
ended in HTTP 404 after compressing the whole closure, so nothing ever
landed and the next build tried the same paths again. A post-build-hook
also blocks the build until it returns, which held up every build on
fwdesk (nbot's VM checks spent most of a cold CI run there).
- cacheReceive (on bigbox2): nix.sshServe with write over ssh-ng, for
fwdesk's ssh host key; nix-ssh is trusted, and nix-serve signs the
pushed paths when it serves them.
- cachePush: the hook only queues $OUT_PATHS. nix-cache-push.service
drains the queue with nix copy to ssh-ng://nix-ssh@bigbox2.lab.internal
as the host key, with bigbox2's host key pinned; a path unit starts it,
and a failed push is retried every 10 minutes by a timer.
Co-Authored-By: nBot
- Extend pkgs/llama-cpp to support CUDA backend (backend="cuda") for NVIDIA GPUs
- Add llama-cpp-cuda variant to modules/core/default.nix overlay
- Create hosts/hl-bigbox1/services/llama-cpp.nix with RTX 3060 tuning:
* gpuLayers=30 (conserve 12GB VRAM for context)
* parallel=2 (handle concurrent requests on smaller VRAM)
* Same Qwen3.8-27B + DFlash2 drafter as fwdesk
- Wire service into bigbox1 services/default.nix
- Disabled by default (weights download manual)
v0.8.0 is the first upstream release with DFlash2 drafters; nixpkgs has 0.6.0 (unstable) and an older tree (stable), so the engine is built here. HIP build for Strix Halo. The NixOS module exposes llama-server behind skyg.nixos.server.services.llama-cpp; fwdesk wires it to Qwen3.8-27B + its DFlash2 drafter, disabled until the weights are on disk.
The env-reload unit is a oneshot whose ExecStart was a blocking
systemctl restart of the group unit. When the stack's cold start failed
transiently (container-services-buzz on minipc1, where buzz-relay-1 came
up unhealthy and compose exited 1), the group unit recovered on its own a
few seconds later through its own Restart=on-failure, but the reload unit
had already exited 1 with no restart policy and stayed failed until the
next env file write.
The trigger unit should queue the restart, not own the health of the whole
stack: pass --no-block. Real failures still surface on the group unit,
which carries Restart=on-failure and the start limit and is what the fleet
scan reports.