Add hardened systemd-resolved DNS with boot verification and journal recovery,
enabled only on specific hosts:
- hl-bigbox1, hl-bigbox2 (experienced DNS resilience issues)
- hl-minipc1, hl-minipc2, hl-minipc3 (minipc cluster)
- hl-fwdesk, hl-fws1 (desktop/workstation nodes)
Changes:
1. New module: modules/nixos/common/dns/default.nix
- systemd-resolved with LLMNR fallback and DNSSEC tolerance
- Hardened journald with persistent storage and auto-recovery
- Boot-time DNS verification service with 5 retries
- Systemd watchdog for automatic reboot on DNS failure
2. Default to false, enabled explicitly on target hosts only
3. Updated host configs to enable dns resilience module
bigbox2 runs nix-serve, which only serves its own store: every
'nix copy --to http://bigbox2.lab.internal:5000' from the post-build hook
ended in HTTP 404 after compressing the whole closure, so nothing ever
landed and the next build tried the same paths again. A post-build-hook
also blocks the build until it returns, which held up every build on
fwdesk (nbot's VM checks spent most of a cold CI run there).
- cacheReceive (on bigbox2): nix.sshServe with write over ssh-ng, for
fwdesk's ssh host key; nix-ssh is trusted, and nix-serve signs the
pushed paths when it serves them.
- cachePush: the hook only queues $OUT_PATHS. nix-cache-push.service
drains the queue with nix copy to ssh-ng://nix-ssh@bigbox2.lab.internal
as the host key, with bigbox2's host key pinned; a path unit starts it,
and a failed push is retried every 10 minutes by a timer.
Co-Authored-By: nBot