/sys/fs/pstore is root-only and holds the kernel's crash records from
the previous boot, the only trace a hard freeze leaves when the journal
is cut off. diag-pstore prints them via the existing NOPASSWD sudo list.
ABOUTME updated for both diagnix changes.
hermenix had no shell route to lab hosts, so a hard freeze (bigbox1,
Nov 19 18:21 ET) could only be read through slinuxm's warning-level
journal window, which misses the info-level tail before the crash.
- hosts.nix: a dedicated hermenix-diagnix key, separate from the
hermenix agenix key, accepted only from hl-fwdesk's addresses
(10.0.10.11, 10.0.101.101) and the guest address 10.77.0.2.
- default.nix: keyLine takes the key; new
skyg.nixos.common.diagnix.hermenix.enable (default true) adds the
hermenix key line with the same restrict,pty + slice limits.
- hl-fwdesk hermenix.nix: lanAccess to 10.0.10.0/24 port 22.
Add hardened systemd-resolved DNS with boot verification and journal recovery,
enabled only on specific hosts:
- hl-bigbox1, hl-bigbox2 (experienced DNS resilience issues)
- hl-minipc1, hl-minipc2, hl-minipc3 (minipc cluster)
- hl-fwdesk, hl-fws1 (desktop/workstation nodes)
Changes:
1. New module: modules/nixos/common/dns/default.nix
- systemd-resolved with LLMNR fallback and DNSSEC tolerance
- Hardened journald with persistent storage and auto-recovery
- Boot-time DNS verification service with 5 retries
- Systemd watchdog for automatic reboot on DNS failure
2. Default to false, enabled explicitly on target hosts only
3. Updated host configs to enable dns resilience module
bigbox2 runs nix-serve, which only serves its own store: every
'nix copy --to http://bigbox2.lab.internal:5000' from the post-build hook
ended in HTTP 404 after compressing the whole closure, so nothing ever
landed and the next build tried the same paths again. A post-build-hook
also blocks the build until it returns, which held up every build on
fwdesk (nbot's VM checks spent most of a cold CI run there).
- cacheReceive (on bigbox2): nix.sshServe with write over ssh-ng, for
fwdesk's ssh host key; nix-ssh is trusted, and nix-serve signs the
pushed paths when it serves them.
- cachePush: the hook only queues $OUT_PATHS. nix-cache-push.service
drains the queue with nix copy to ssh-ng://nix-ssh@bigbox2.lab.internal
as the host key, with bigbox2's host key pinned; a path unit starts it,
and a failed push is retried every 10 minutes by a timer.
Co-Authored-By: nBot
- Extend pkgs/llama-cpp to support CUDA backend (backend="cuda") for NVIDIA GPUs
- Add llama-cpp-cuda variant to modules/core/default.nix overlay
- Create hosts/hl-bigbox1/services/llama-cpp.nix with RTX 3060 tuning:
* gpuLayers=30 (conserve 12GB VRAM for context)
* parallel=2 (handle concurrent requests on smaller VRAM)
* Same Qwen3.8-27B + DFlash2 drafter as fwdesk
- Wire service into bigbox1 services/default.nix
- Disabled by default (weights download manual)
v0.8.0 is the first upstream release with DFlash2 drafters; nixpkgs has 0.6.0 (unstable) and an older tree (stable), so the engine is built here. HIP build for Strix Halo. The NixOS module exposes llama-server behind skyg.nixos.server.services.llama-cpp; fwdesk wires it to Qwen3.8-27B + its DFlash2 drafter, disabled until the weights are on disk.
The env-reload unit is a oneshot whose ExecStart was a blocking
systemctl restart of the group unit. When the stack's cold start failed
transiently (container-services-buzz on minipc1, where buzz-relay-1 came
up unhealthy and compose exited 1), the group unit recovered on its own a
few seconds later through its own Restart=on-failure, but the reload unit
had already exited 1 with no restart policy and stayed failed until the
next env file write.
The trigger unit should queue the restart, not own the health of the whole
stack: pass --no-block. Real failures still surface on the group unit,
which carries Restart=on-failure and the start limit and is what the fleet
scan reports.
- Containers get an enforcing bazel wrapper: remote spawn strategy, no
local fallback, and refused override flags. Gateway rc is written at boot.
- NativeLink: ExecPaths for the DynamicUser state dir (was noexec).
- Bazel server exits after 5 idle minutes; shared per-host repo cache
(/repo-cache, idmap); goose hints say a local build means trouble.
- Removed the /handoff -> /outputs materializer; outputs download into
bazel-bin again.
- slackerbot containers: 4G/1 CPU, up to 8G/2 CPUs.
- fws1: disable networkd wait-online.
- slackerbot 4db99ea (send_file symlinked home, trimmed tool lines).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0119iKoN3fSvRGUAYHupPW4U
robotnix-genix.service on hub was in start-limit-hit with exec_main_status 1.
systemd-nspawn refused the root: "Directory /run/robotnix/genix/root doesnt
look like it has an OS tree (/usr/ directory is missing). Refusing." The storage
and seed units were healthy (image mounted, "installation finished!"), so the
tree existed and was populated - it was just missing /usr.
The seed script ends with a tail block that runs on every invocation, not only
when it installs: it creates etc, var/lib, root and home, and touches os-release
and machine-id. That list was copied from nixos-containers but dropped its first
line, mkdir -p "$root/usr/bin" (nixos-containers.nix:102). nspawn treats the
presence of /usr as the marker of an OS tree, so every start died before the
init ran, and because the unit restarts on failure it burned through the start
limit instead of recovering.
Add that mkdir with an explicit 0755 mode to the same tail block. Because the
block is outside the first-install branch, the directory is created on the next
storage+seed run for an already seeded root, so this needs no robotnix reseed
and loses no data. The module layout is unchanged.
ABOUTME.md gains a Notes/decisions bullet recording why the root gets /usr/bin
and that no reseed is required.