robotnix-genix.service on hub was in start-limit-hit with exec_main_status 1. systemd-nspawn refused the root: "Directory /run/robotnix/genix/root doesnt look like it has an OS tree (/usr/ directory is missing). Refusing." The storage and seed units were healthy (image mounted, "installation finished!"), so the tree existed and was populated - it was just missing /usr. The seed script ends with a tail block that runs on every invocation, not only when it installs: it creates etc, var/lib, root and home, and touches os-release and machine-id. That list was copied from nixos-containers but dropped its first line, mkdir -p "$root/usr/bin" (nixos-containers.nix:102). nspawn treats the presence of /usr as the marker of an OS tree, so every start died before the init ran, and because the unit restarts on failure it burned through the start limit instead of recovering. Add that mkdir with an explicit 0755 mode to the same tail block. Because the block is outside the first-install branch, the directory is created on the next storage+seed run for an already seeded root, so this needs no robotnix reseed and loses no data. The module layout is unchanged. ABOUTME.md gains a Notes/decisions bullet recording why the root gets /usr/bin and that no reseed is required.
6.0 KiB
robotnix — self-managed NixOS nspawn robots
skyg.nixos.server.robotnix runs persistent NixOS systemd-nspawn containers
("robots") that fully own their own system. Each robot has a real LAN IP, hard
disk limits, btrfs snapshots, and network access the host controls completely.
Model
- The robot owns its system. On first install (
robotnix-<r>-seed.service) the host builds a starting system with the same entry point the robot's own flake uses (lib.nixosSystem= eval-config withsystem = null), installs it intoroot.imgwithnixos-install --no-bootloader, and writes/etc/nixos/{flake.nix,robotnix.nix,configuration.nix}. After that the host never touches the robot's root: it rebuilds withnixos-rebuild switch --flake /etc/nixosagainst its own pinned nixpkgs and its own store. Changes survive reboots. robotnix.nixis host-owned (network, user, ssh keys, git-sync).configuration.nixis the robot's own config (seed.configuration, default./seed/configuration.nix). Changing a seed config only matters at the nextrobotnix reseed.- No host root. Robots run with
--private-users=pickunlessprivateUsers = false.
Storage
Per robot under <baseDir>/<robot>/ (btrfs subvolumes, nodatacow images):
root/root.img 100G (default) sparse ext4, whole OS incl. /nix
home/home.img 60G (default) sparse ext4, /home
snapshots/<id>/{root,home} read-only subvolume snapshots
The host mounts the images at /run/robotnix/<robot>/ and passes them in, so a
snapshot freezes both filesystems (fsfreeze) for a consistent point in time
while the robot runs. Sizes are hard limits; raise them and the images grow on
the next restart (shrinking is not supported).
Snapshots: per-robot snapshots.enable/interval/keep via skyg.server.timers
(robotnix-<r>-snapshot), robotnix snapshot|restore|forget|snapshots.
restore saves a pre-restore snapshot first. Because images are nodatacow,
keep the automatic snapshot count modest — every live snapshot reintroduces
copy-on-write for the referenced blocks. Backups are out of scope for now.
Network
One routed virtual switch:
robotnix-brbridge (networkd) on 10.78.0.1/24, unmanaged by NetworkManager; each robot'svb-<robot>port isIsolated(no robot-to- robot L2).- Robots hold
<lanIP>/32with a default route via the switch. The host routes to them and answers ARP for that IP onexternalInterface(proxy ARP per IP; global proxy ARP stays off), so each packet passes the host firewall. - nftables: a bridge table drops spoofed addresses; an
inet robotnixtable does the per-robot filtering beforenixos-fw(priority filter - 10). IPv6 from robots is dropped (host0is IPv4-only). Inbound from the LAN to a robot is open on every port — make your own decisions inside the robot. - Per-robot dnsmasq on the host (port 5300+id) is the robot's only resolver: all port 53 is DNAT'd to it, DoT is dropped. The robot cannot change its resolv.conf meaningfully; it cannot touch the host's nftables/networkd.
Access tiers
access.shared + robots.<r>.access merge per robot (internet from the
robot tier wins, all lists are unions). Rule order in the out-chain:
- established/related
- port 53 served by the host / 853 dropped (DoT)
blockedCidrsand blocked-domain IPs (nftsets fed by dnsmasqnftset=)lanAccess/hostAccessendpoints (host input is matched per-robot by port-source rules)- RFC1918/CGNAT/link-local drop unless
allowLan - internet:
all(default) or, withallowlist, onlyallowedCidrs+ allowed-domain IPs
Domain rules are IP-based: acceptable for host-controlled filtering, but a
CDN shared with an allowed domain also lets that IP's other sites through (and
dnsmasq learns current IPs). blockedDomains resolve to NXDOMAIN; in
allowlist mode everything not allowed is NXDOMAIN + dropped.
Git-sync of the robot's flake (default)
On first install / reseed, the seed creates a git repo at /etc/nixos
(git.enable, default on) and sets git.remoteUrl as origin when given. A
timer inside the robot (robotnix-flake-gitsync = git add -A, commit, push
if origin exists) runs on git.interval (daily default). The robot needs the
remote reachable (e.g. through lanAccess) and its own credentials if the
remote is private.
The robotnix CLI
Nushell script (pkgs.writers.writeNuBin), part of this module:
list, status, start/stop/restart, shell, snapshot,
snapshots, forget, restore, reseed [--wipe-home], access.
Runtime state comes from /etc/robotnix.json, written by the module.
Notes / decisions
- nspawn, not
oci-containers: the server/GUIDELINES rule aboutoci-containerstargets Docker/Podman. nspawn containers needsystemd-nspawnand the NixOS container init; there is no OCI equivalent. - This is why
systemd-nspawnis launched by a hand-written unit rather than thecontainers.*module: that module shares the host store, which would break the robot's self-rebuilds. The unit mirrorsnixos-containers.nix(Type=notify, SIGRTMIN+3, exit status 133,machine.slice). - The robot's root gets
/usr/binbefore its first start:systemd-nspawnrefuses a tree without/usr("doesn't look like it has an OS tree"), the same reasonnixos-containers.nixcreates it. The seed's tail block runs on every start, so no reseed is needed after this landed. systemd-nspawnruns inmachine.slice(systemd's nspawn home), notsystem.slice.- Timers use
skyg.server.timers; eval asserts unique robot ids/IPs, name length ≤ 12 (interface-name limit), no clash withskyg.internalNetworkingMap.apps, and no domain/CIDR both allowed and blocked. fsfreezefails on unfrozen ro/sparse quirks only when the images are not mounted (robot stopped); the CLI snapshots only mounted filesystems' mounts, otherwise it snapshots the subvolumes directly.- flake inputs: only the host nixpkgs is pinned into the robot's flake
(
github:NixOS/nixpkgs/<host-rev>). Extra inputs belong in the robot's ownconfiguration.nixedits.