11 KiB
robotnix — self-managed NixOS nspawn robots
skyg.nixos.server.robotnix runs persistent NixOS systemd-nspawn containers
("robots") that fully own their own system. Each robot has a real LAN IP, hard
disk limits, btrfs snapshots, and network access the host controls completely.
Model
- The robot owns its system. On first install (
robotnix-<r>-seed.service) the host builds a starting system with the same entry point the robot's own flake uses (lib.nixosSystem= eval-config withsystem = null), installs it intoroot.imgwithnixos-install --no-bootloader, and writes/etc/nixos/{flake.nix,robotnix.nix,configuration.nix}. After that the host never touches the robot's root: it rebuilds withnixos-rebuild switch --flake /etc/nixosagainst its own pinned nixpkgs and its own store. Changes survive reboots. robotnix.nixis host-owned (network, user, ssh keys, git-sync).configuration.nixis the robot's own config (seed.configuration, default./seed/configuration.nix). Changing a seed config only matters at the nextrobotnix reseed.- No host root. Robots run with
--private-users=pickunlessprivateUsers = false.
Storage
Per robot under <baseDir>/<robot>/ (btrfs subvolumes, nodatacow images):
root/root.img 100G (default) sparse ext4, whole OS incl. /nix
home/home.img 60G (default) sparse ext4, /home
snapshots/<id>/{root,home} read-only subvolume snapshots
The host mounts the images at /run/robotnix/<robot>/ and passes them in, so a
snapshot freezes both filesystems (fsfreeze) for a consistent point in time
while the robot runs. Sizes are hard limits; raise them and the images grow on
the next restart (shrinking is not supported).
Snapshots: per-robot snapshots.enable/interval/keep via skyg.server.timers
(robotnix-<r>-snapshot), robotnix snapshot|restore|forget|snapshots.
restore saves a pre-restore snapshot first. Because images are nodatacow,
keep the automatic snapshot count modest — every live snapshot reintroduces
copy-on-write for the referenced blocks. Backups are out of scope for now.
Network
One routed virtual switch:
robotnix-brbridge (networkd) on 10.78.0.1/24, unmanaged by NetworkManager; each robot'svb-<robot>port isIsolated(no robot-to- robot L2).- Robots hold
<lanIP>/32with a default route via the switch. The host routes to them and answers ARP for that IP onexternalInterface(proxy ARP per IP; global proxy ARP stays off), so each packet passes the host firewall. - nftables: a bridge table drops spoofed addresses; an
inet robotnixtable does the per-robot filtering beforenixos-fw(priority filter - 10). IPv6 from robots is dropped (host0is IPv4-only). Inbound from the LAN to a robot is open on every port — make your own decisions inside the robot. - Guards in the same table, so other modules' NAT can't reroute robot
traffic (hermenix once DNAT'd any-destination traffic on some ports into
its VM):
- a connection meant for a robot IP that leaves anywhere but the switch,
or lands on the host itself, is dropped (
ct original ip daddr; replies are untouched viact direction original); - a robot's own new connection that was DNAT'd is dropped. The robot DNS redirect still works: it goes to the host (input), not forward.
- a connection meant for a robot IP that leaves anywhere but the switch,
or lands on the host itself, is dropped (
- Per-robot dnsmasq on the host (port 5300+id) is the robot's only resolver: all port 53 is DNAT'd to it, DoT is dropped. The robot cannot change its resolv.conf meaningfully; it cannot touch the host's nftables/networkd.
Access tiers
access.shared + robots.<r>.access merge per robot (internet from the
robot tier wins, all lists are unions). Rule order in the out-chain:
- established/related
- port 53 served by the host / 853 dropped (DoT)
blockedCidrsand blocked-domain IPs (nftsets fed by dnsmasqnftset=)lanAccess/hostAccessendpoints (host input is matched per-robot by port-source rules)- RFC1918/CGNAT/link-local drop unless
allowLan - internet:
all(default) or, withallowlist, onlyallowedCidrs+ allowed-domain IPs
Domain rules are IP-based: acceptable for host-controlled filtering, but a
CDN shared with an allowed domain also lets that IP's other sites through (and
dnsmasq learns current IPs). blockedDomains resolve to NXDOMAIN; in
allowlist mode everything not allowed is NXDOMAIN + dropped.
Git-sync of the robot's flake (default)
On first install / reseed, the seed creates a git repo at /etc/nixos
(git.enable, default on) and sets git.remoteUrl as origin when given. A
timer inside the robot (robotnix-flake-gitsync = git add -A, commit, push
if origin exists) runs on git.interval (daily default). The robot needs the
remote reachable (e.g. through lanAccess) and its own credentials if the
remote is private.
Ephemeral robots (opt-in)
ephemeral = true resets root to a clean copy of the seed on every start
(host boot, robotnix restart, or a reboot inside). Only /home survives.
- The seed step saves the fresh root as a read-only subvolume
<base>/pristinebefore the robot first boots, pluspristine.id(hash of the seed system and the files it installs). - Before each start,
robotnix-<r>-reset.serviceunmounts root, swaps<base>/rootfor a snapshot ofpristine, and remounts (about a second). It is its own oneshot (noRemainAfterExit, so every robot start re-runs it) because the robot unit'sDevicePolicy=closedblocks the loop devices the mount needs. - When the seed changes, the storage step sees a different id on the next host switch, wipes root, and the seed step reinstalls (a few minutes).
persist = [ "/var/lib/..." ]keeps folders: stored in/home/.persistinside the home image and bind-mounted at their usual path. SSH host keys always go to/home/.persist/ssh, so they don't change each boot.- The robot still has its own writable store, so
nix/devenvwork until the next restart. Changes go throughseed.configurationon the host. - Snapshots and
restorecover home only;reseeddropspristinetoo. - git-sync can't work (fresh repo each start): eval asserts it is off.
Hermes in every robot (opt-in)
hermes.enable (shared) runs one Hermes agent per robot with the upstream
services.hermes-agent module; robots.<r>.hermes.enable can opt out.
- The robot's flake gets a
hermes-agentinput pinned to the host's locked rev, and the host's seed build includes the same module. - Host-managed:
hermes.settings(shared + robot, deep-merged) goes to config.yaml;hermes.namewrites SOUL.md ("You are , a Hermes-based agent ...") on every boot; null keeps the stock identity. - Secrets:
hermes.environmentFiles(shared + robot, KEY=value, e.g. agenix) are passed with--load-credentialand loaded by thehermes-agentunit from/run/credentials/@system/robotnix-hermes-<n>(later files win). No file ownership games with the user namespace. Restart the robot after a secret changes. - The
hermesuser/group have a fixed id (870) so kept files stay owned right across reinstalls; the login user is in thehermesgroup and has thehermesCLI. - Ephemeral robots keep only
cron/,memories/andskills/of/var/lib/hermes/.hermes(added topersistautomatically). Sessions, logs, OAuth logins and pairings reset each start: use API keys and*_ALLOWED_USERSenv vars instead. - Gateway API (
hermes.gateway, on by default, port 8642): the host setsservices.hermes-agent.environment(API_SERVER_ENABLED,_HOST=0.0.0.0,_PORT,_MODEL_NAME=hermes.nameor the robot name). These go to$HERMES_HOME/.env, which beats the same names in the secrets. Each robot's secrets needAPI_SERVER_KEY, or the API server won't start. - Dashboard (
hermes.dashboard, on by default, port 80):backend.mode = "dashboard"on 0.0.0.0, run ashermes-backend.servicewith the same secret files, plusCAP_NET_BIND_SERVICEwhen the port is below 1024. A non-loopback dashboard needs a login in the secrets:HERMES_DASHBOARD_BASIC_AUTH_USERNAME,..._PASSWORD_HASH(or..._PASSWORD) and..._SECRET(keeps sessions across restarts). Without one it exits and systemd keeps restarting it; the gateway keeps working. Port 80 may clash with the robot's own compose stacks: change the port. - Both: top-level
enable/port/openFirewall, robots override any of them (null = follow).openFirewallopens the TCP port in the robot's own firewall. Eval asserts the two don't share a port. - Existing robots only get Hermes (and gateway/dashboard changes) after
robotnix reseed(flake.nix and robotnix.nix are written at install). Ephemeral robots pick changes up on the next host switch.
Docker and Tailscale (opt-in, host side only)
These options only change what the host allows. Turn the service on in the
robot's own config (e.g. its seed.configuration).
allowDocker: nspawn gets--system-call-filter=add_key keyctl bpf(blocked by default, Docker needs them). Docker runs inside the robot's user namespace, so images using UIDs above 65535 won't run.allowTun:/dev/net/tunis bound in and allowed on the unit (DevicePolicy=closedblocks it otherwise). Used for tailscale: the robot is its own tailnet machine, log in once inside withsudo tailscale up. Direct LAN paths to other tailnet peers hit the private-range drop, so it uses public endpoints or DERP.- Both apply on the next robot restart; no reseed needed.
The robotnix CLI
Nushell script (pkgs.writers.writeNuBin), part of this module:
list, status, start/stop/restart, shell, snapshot,
snapshots, forget, restore, reseed [--wipe-home], access.
Runtime state comes from /etc/robotnix.json, written by the module.
Notes / decisions
- nspawn, not
oci-containers: the server/GUIDELINES rule aboutoci-containerstargets Docker/Podman. nspawn containers needsystemd-nspawnand the NixOS container init; there is no OCI equivalent. - This is why
systemd-nspawnis launched by a hand-written unit rather than thecontainers.*module: that module shares the host store, which would break the robot's self-rebuilds. The unit mirrorsnixos-containers.nix(Type=notify, SIGRTMIN+3, exit status 133,machine.slice). - The robot's root gets
/usr/binbefore its first start:systemd-nspawnrefuses a tree without/usr("doesn't look like it has an OS tree"), the same reasonnixos-containers.nixcreates it. The seed's tail block runs on every start, so no reseed is needed after this landed. systemd-nspawnruns inmachine.slice(systemd's nspawn home), notsystem.slice.- Timers use
skyg.server.timers; eval asserts unique robot ids/IPs, name length ≤ 12 (interface-name limit), no clash withskyg.internalNetworkingMap.apps, and no domain/CIDR both allowed and blocked. fsfreezefails on unfrozen ro/sparse quirks only when the images are not mounted (robot stopped); the CLI snapshots only mounted filesystems' mounts, otherwise it snapshots the subvolumes directly.- flake inputs: only the host nixpkgs is pinned into the robot's flake
(
github:NixOS/nixpkgs/<host-rev>). Extra inputs belong in the robot's ownconfiguration.nixedits.