The env-reload unit is a oneshot whose ExecStart was a blocking
systemctl restart of the group unit. When the stack's cold start failed
transiently (container-services-buzz on minipc1, where buzz-relay-1 came
up unhealthy and compose exited 1), the group unit recovered on its own a
few seconds later through its own Restart=on-failure, but the reload unit
had already exited 1 with no restart policy and stayed failed until the
next env file write.
The trigger unit should queue the restart, not own the health of the whole
stack: pass --no-block. Real failures still surface on the group unit,
which carries Restart=on-failure and the start limit and is what the fleet
scan reports.
nbot 04ade8b -> 45c6a48 (revCount 654 -> 857). The host module gained a new
kvm option to pass -kvm to the nbot binary, which binds /dev/kvm into every
container on the node for workloads that need to run VMs.
KVM is enabled on both fwdesk (master) and fws1 (slave).
nbot 8daeca5 -> a6451cb (revCount 492 -> 652). The two config shape
changes in that range are allow_subdir on a drive and master_fingerprint
as a list; summaries.enabled is now ignored because summaries are always
on. The host module gained drives and stateDir, which feed
RequiresMountsFor= so the unit waits for its disks.
The nbot input stays on git+ssh: the repo is private, so git+https
cannot fetch it without credentials.
The relay image is ghcr.io/block/buzz:main with autoUpdate and
BUZZ_AUTO_MIGRATE=true. On 2026-11-02 an auto-update pulled a newer build
whose migrations include 0078_artifact_sync_indexes.sql. That migration
refuses to create its four indexes inline once the target table is
populated -- it raises "prebuild <index> concurrently" and points at
docs/artifact-sync-deployment.md. channel_members is populated on a live
relay, so migration 78 aborts, the relay exits 1 and has been
crash-looping ever since. The schema sits at 0077, so every earlier
migration (including 0077_artifact_sync_timestamps) is applied; rolling
the image back is not an option because sqlx rejects a schema that is
ahead of the binary's known migrations.
Fix the gate instead of the image: a one-shot compose service,
relay-index-prebuild, runs on the postgres image after postgres reports
healthy and builds the four indexes CONCURRENTLY. Each index is checked
first -- missing, valid, or invalid (present but not
indisvalid/indisready/indislive); an invalid leftover is dropped
concurrently before the rebuild, and an already-valid index is left
alone, so re-runs are cheap and safe. The relay now waits on
relay-index-prebuild completing successfully, next to its existing
postgres/redis/minio/minio-init conditions.
Authored by hermenix.
- Containers get an enforcing bazel wrapper: remote spawn strategy, no
local fallback, and refused override flags. Gateway rc is written at boot.
- NativeLink: ExecPaths for the DynamicUser state dir (was noexec).
- Bazel server exits after 5 idle minutes; shared per-host repo cache
(/repo-cache, idmap); goose hints say a local build means trouble.
- Removed the /handoff -> /outputs materializer; outputs download into
bazel-bin again.
- slackerbot containers: 4G/1 CPU, up to 8G/2 CPUs.
- fws1: disable networkd wait-online.
- slackerbot 4db99ea (send_file symlinked home, trimmed tool lines).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0119iKoN3fSvRGUAYHupPW4U
systemd-networkd-wait-online fails on every boot of hl-fwdesk with
"Timeout occurred while waiting for network connectivity" and
status=1/FAILURE. The hermes VM runs on this host via services.hermenix
(ari/hermenix flake input), and its fabric interfaces -- the bridge
hermes-br and the tap vm-hermes -- are managed by systemd-networkd
(10-hermes-br.network, 11-vm-hermes.network). networkd-managed links
default to RequiredForOnline=yes, but those two links only gain carrier
when the VM starts, which on the sampled boot was 0.2s after the 120s
wait-online timeout fired. The only networkd link that never blocks is
robotnix's own bridge, and the physical uplink enp191s0 is owned by
NetworkManager, so it is invisible to networkd. The result is a unit
that always burns the full timeout and then fails.
hermenix now passes --ignore=hermes-br --ignore=vm-hermes to the unit,
so it no longer treats the VM's fabric links as a condition for
"online": with every candidate ignored the wait returns immediately
instead of blocking and then failing. The unit itself stays enabled
and anyInterface is untouched.
The durable fix belongs in ari/hermenix: set RequiredForOnline = "no" on
the hermes-br and vm-hermes .network files, the same way
modules/nixos/server/robotnix/network.nix already does for the robotnix
fabric. That repository is out of scope for this change; once it lands
this ignoredInterfaces entry can be dropped.
Found by hermenix from the fleet monitor's persistent failure record
for unit:hub:systemd-networkd-wait-online.service.
robotnix-genix.service on hub was in start-limit-hit with exec_main_status 1.
systemd-nspawn refused the root: "Directory /run/robotnix/genix/root doesnt
look like it has an OS tree (/usr/ directory is missing). Refusing." The storage
and seed units were healthy (image mounted, "installation finished!"), so the
tree existed and was populated - it was just missing /usr.
The seed script ends with a tail block that runs on every invocation, not only
when it installs: it creates etc, var/lib, root and home, and touches os-release
and machine-id. That list was copied from nixos-containers but dropped its first
line, mkdir -p "$root/usr/bin" (nixos-containers.nix:102). nspawn treats the
presence of /usr as the marker of an OS tree, so every start died before the
init ran, and because the unit restarts on failure it burned through the start
limit instead of recovering.
Add that mkdir with an explicit 0755 mode to the same tail block. Because the
block is outside the first-install branch, the directory is created on the next
storage+seed run for an already seeded root, so this needs no robotnix reseed
and loses no data. The module layout is unchanged.
ABOUTME.md gains a Notes/decisions bullet recording why the root gets /usr/bin
and that no reseed is required.