/sys/fs/pstore is root-only and holds the kernel's crash records from
the previous boot, the only trace a hard freeze leaves when the journal
is cut off. diag-pstore prints them via the existing NOPASSWD sudo list.
ABOUTME updated for both diagnix changes.
hermenix had no shell route to lab hosts, so a hard freeze (bigbox1,
Nov 19 18:21 ET) could only be read through slinuxm's warning-level
journal window, which misses the info-level tail before the crash.
- hosts.nix: a dedicated hermenix-diagnix key, separate from the
hermenix agenix key, accepted only from hl-fwdesk's addresses
(10.0.10.11, 10.0.101.101) and the guest address 10.77.0.2.
- default.nix: keyLine takes the key; new
skyg.nixos.common.diagnix.hermenix.enable (default true) adds the
hermenix key line with the same restrict,pty + slice limits.
- hl-fwdesk hermenix.nix: lanAccess to 10.0.10.0/24 port 22.
Add hardened systemd-resolved DNS with boot verification and journal recovery,
enabled only on specific hosts:
- hl-bigbox1, hl-bigbox2 (experienced DNS resilience issues)
- hl-minipc1, hl-minipc2, hl-minipc3 (minipc cluster)
- hl-fwdesk, hl-fws1 (desktop/workstation nodes)
Changes:
1. New module: modules/nixos/common/dns/default.nix
- systemd-resolved with LLMNR fallback and DNSSEC tolerance
- Hardened journald with persistent storage and auto-recovery
- Boot-time DNS verification service with 5 retries
- Systemd watchdog for automatic reboot on DNS failure
2. Default to false, enabled explicitly on target hosts only
3. Updated host configs to enable dns resilience module
The env-reload unit is a oneshot whose ExecStart was a blocking
systemctl restart of the group unit. When the stack's cold start failed
transiently (container-services-buzz on minipc1, where buzz-relay-1 came
up unhealthy and compose exited 1), the group unit recovered on its own a
few seconds later through its own Restart=on-failure, but the reload unit
had already exited 1 with no restart policy and stayed failed until the
next env file write.
The trigger unit should queue the restart, not own the health of the whole
stack: pass --no-block. Real failures still surface on the group unit,
which carries Restart=on-failure and the start limit and is what the fleet
scan reports.
nbot 04ade8b -> 45c6a48 (revCount 654 -> 857). The host module gained a new
kvm option to pass -kvm to the nbot binary, which binds /dev/kvm into every
container on the node for workloads that need to run VMs.
KVM is enabled on both fwdesk (master) and fws1 (slave).
The relay image is ghcr.io/block/buzz:main with autoUpdate and
BUZZ_AUTO_MIGRATE=true. On 2026-11-02 an auto-update pulled a newer build
whose migrations include 0078_artifact_sync_indexes.sql. That migration
refuses to create its four indexes inline once the target table is
populated -- it raises "prebuild <index> concurrently" and points at
docs/artifact-sync-deployment.md. channel_members is populated on a live
relay, so migration 78 aborts, the relay exits 1 and has been
crash-looping ever since. The schema sits at 0077, so every earlier
migration (including 0077_artifact_sync_timestamps) is applied; rolling
the image back is not an option because sqlx rejects a schema that is
ahead of the binary's known migrations.
Fix the gate instead of the image: a one-shot compose service,
relay-index-prebuild, runs on the postgres image after postgres reports
healthy and builds the four indexes CONCURRENTLY. Each index is checked
first -- missing, valid, or invalid (present but not
indisvalid/indisready/indislive); an invalid leftover is dropped
concurrently before the rebuild, and an already-valid index is left
alone, so re-runs are cheap and safe. The relay now waits on
relay-index-prebuild completing successfully, next to its existing
postgres/redis/minio/minio-init conditions.
Authored by hermenix.
systemd-networkd-wait-online fails on every boot of hl-fwdesk with
"Timeout occurred while waiting for network connectivity" and
status=1/FAILURE. The hermes VM runs on this host via services.hermenix
(ari/hermenix flake input), and its fabric interfaces -- the bridge
hermes-br and the tap vm-hermes -- are managed by systemd-networkd
(10-hermes-br.network, 11-vm-hermes.network). networkd-managed links
default to RequiredForOnline=yes, but those two links only gain carrier
when the VM starts, which on the sampled boot was 0.2s after the 120s
wait-online timeout fired. The only networkd link that never blocks is
robotnix's own bridge, and the physical uplink enp191s0 is owned by
NetworkManager, so it is invisible to networkd. The result is a unit
that always burns the full timeout and then fails.
hermenix now passes --ignore=hermes-br --ignore=vm-hermes to the unit,
so it no longer treats the VM's fabric links as a condition for
"online": with every candidate ignored the wait returns immediately
instead of blocking and then failing. The unit itself stays enabled
and anyInterface is untouched.
The durable fix belongs in ari/hermenix: set RequiredForOnline = "no" on
the hermes-br and vm-hermes .network files, the same way
modules/nixos/server/robotnix/network.nix already does for the robotnix
fabric. That repository is out of scope for this change; once it lands
this ignoredInterfaces entry can be dropped.
Found by hermenix from the fleet monitor's persistent failure record
for unit:hub:systemd-networkd-wait-online.service.
robotnix-genix.service on hub was in start-limit-hit with exec_main_status 1.
systemd-nspawn refused the root: "Directory /run/robotnix/genix/root doesnt
look like it has an OS tree (/usr/ directory is missing). Refusing." The storage
and seed units were healthy (image mounted, "installation finished!"), so the
tree existed and was populated - it was just missing /usr.
The seed script ends with a tail block that runs on every invocation, not only
when it installs: it creates etc, var/lib, root and home, and touches os-release
and machine-id. That list was copied from nixos-containers but dropped its first
line, mkdir -p "$root/usr/bin" (nixos-containers.nix:102). nspawn treats the
presence of /usr as the marker of an OS tree, so every start died before the
init ran, and because the unit restarts on failure it burned through the start
limit instead of recovering.
Add that mkdir with an explicit 0755 mode to the same tail block. Because the
block is outside the first-install branch, the directory is created on the next
storage+seed run for an already seeded root, so this needs no robotnix reseed
and loses no data. The module layout is unchanged.
ABOUTME.md gains a Notes/decisions bullet recording why the root gets /usr/bin
and that no reseed is required.