systemd-networkd-wait-online fails on every boot of hl-fwdesk with
"Timeout occurred while waiting for network connectivity" and
status=1/FAILURE. The hermes VM runs on this host via services.hermenix
(ari/hermenix flake input), and its fabric interfaces -- the bridge
hermes-br and the tap vm-hermes -- are managed by systemd-networkd
(10-hermes-br.network, 11-vm-hermes.network). networkd-managed links
default to RequiredForOnline=yes, but those two links only gain carrier
when the VM starts, which on the sampled boot was 0.2s after the 120s
wait-online timeout fired. The only networkd link that never blocks is
robotnix's own bridge, and the physical uplink enp191s0 is owned by
NetworkManager, so it is invisible to networkd. The result is a unit
that always burns the full timeout and then fails.
hermenix now passes --ignore=hermes-br --ignore=vm-hermes to the unit,
so it no longer treats the VM's fabric links as a condition for
"online": with every candidate ignored the wait returns immediately
instead of blocking and then failing. The unit itself stays enabled
and anyInterface is untouched.
The durable fix belongs in ari/hermenix: set RequiredForOnline = "no" on
the hermes-br and vm-hermes .network files, the same way
modules/nixos/server/robotnix/network.nix already does for the robotnix
fabric. That repository is out of scope for this change; once it lands
this ignoredInterfaces entry can be dropped.
Found by hermenix from the fleet monitor's persistent failure record
for unit:hub:systemd-networkd-wait-online.service.
robotnix-genix.service on hub was in start-limit-hit with exec_main_status 1.
systemd-nspawn refused the root: "Directory /run/robotnix/genix/root doesnt
look like it has an OS tree (/usr/ directory is missing). Refusing." The storage
and seed units were healthy (image mounted, "installation finished!"), so the
tree existed and was populated - it was just missing /usr.
The seed script ends with a tail block that runs on every invocation, not only
when it installs: it creates etc, var/lib, root and home, and touches os-release
and machine-id. That list was copied from nixos-containers but dropped its first
line, mkdir -p "$root/usr/bin" (nixos-containers.nix:102). nspawn treats the
presence of /usr as the marker of an OS tree, so every start died before the
init ran, and because the unit restarts on failure it burned through the start
limit instead of recovering.
Add that mkdir with an explicit 0755 mode to the same tail block. Because the
block is outside the first-install branch, the directory is created on the next
storage+seed run for an already seeded root, so this needs no robotnix reseed
and loses no data. The module layout is unchanged.
ABOUTME.md gains a Notes/decisions bullet recording why the root gets /usr/bin
and that no reseed is required.