robotnix-genix.service on hub was in start-limit-hit with exec_main_status 1. systemd-nspawn refused the root: "Directory /run/robotnix/genix/root doesnt look like it has an OS tree (/usr/ directory is missing). Refusing." The storage and seed units were healthy (image mounted, "installation finished!"), so the tree existed and was populated - it was just missing /usr. The seed script ends with a tail block that runs on every invocation, not only when it installs: it creates etc, var/lib, root and home, and touches os-release and machine-id. That list was copied from nixos-containers but dropped its first line, mkdir -p "$root/usr/bin" (nixos-containers.nix:102). nspawn treats the presence of /usr as the marker of an OS tree, so every start died before the init ran, and because the unit restarts on failure it burned through the start limit instead of recovering. Add that mkdir with an explicit 0755 mode to the same tail block. Because the block is outside the first-install branch, the directory is created on the next storage+seed run for an already seeded root, so this needs no robotnix reseed and loses no data. The module layout is unchanged. ABOUTME.md gains a Notes/decisions bullet recording why the root gets /usr/bin and that no reseed is required.
123 lines
6.0 KiB
Markdown
123 lines
6.0 KiB
Markdown
# robotnix — self-managed NixOS nspawn robots
|
|
|
|
`skyg.nixos.server.robotnix` runs persistent NixOS `systemd-nspawn` containers
|
|
("robots") that fully own their own system. Each robot has a real LAN IP, hard
|
|
disk limits, btrfs snapshots, and network access the host controls completely.
|
|
|
|
## Model
|
|
|
|
- **The robot owns its system.** On first install (`robotnix-<r>-seed.service`)
|
|
the host builds a starting system with the same entry point the robot's own
|
|
flake uses (`lib.nixosSystem` = eval-config with `system = null`), installs it
|
|
into `root.img` with `nixos-install --no-bootloader`, and writes
|
|
`/etc/nixos/{flake.nix,robotnix.nix,configuration.nix}`. After that the host
|
|
never touches the robot's root: it rebuilds with
|
|
`nixos-rebuild switch --flake /etc/nixos` against its own pinned nixpkgs and
|
|
its own store. Changes survive reboots.
|
|
- **`robotnix.nix` is host-owned** (network, user, ssh keys, git-sync).
|
|
`configuration.nix` is the robot's own config (`seed.configuration`, default
|
|
`./seed/configuration.nix`). Changing a seed config only matters at the next
|
|
`robotnix reseed`.
|
|
- **No host root.** Robots run with `--private-users=pick` unless
|
|
`privateUsers = false`.
|
|
|
|
## Storage
|
|
|
|
Per robot under `<baseDir>/<robot>/` (btrfs subvolumes, `nodatacow` images):
|
|
|
|
```
|
|
root/root.img 100G (default) sparse ext4, whole OS incl. /nix
|
|
home/home.img 60G (default) sparse ext4, /home
|
|
snapshots/<id>/{root,home} read-only subvolume snapshots
|
|
```
|
|
|
|
The host mounts the images at `/run/robotnix/<robot>/` and passes them in, so a
|
|
snapshot freezes both filesystems (`fsfreeze`) for a consistent point in time
|
|
while the robot runs. Sizes are hard limits; raise them and the images grow on
|
|
the next restart (shrinking is not supported).
|
|
|
|
Snapshots: per-robot `snapshots.enable/interval/keep` via `skyg.server.timers`
|
|
(`robotnix-<r>-snapshot`), `robotnix snapshot|restore|forget|snapshots`.
|
|
`restore` saves a `pre-restore` snapshot first. Because images are `nodatacow`,
|
|
keep the automatic snapshot count modest — every live snapshot reintroduces
|
|
copy-on-write for the referenced blocks. Backups are out of scope for now.
|
|
|
|
## Network
|
|
|
|
One routed virtual switch:
|
|
|
|
- `robotnix-br` bridge (networkd) on 10.78.0.1/24, unmanaged by
|
|
NetworkManager; each robot's `vb-<robot>` port is `Isolated` (no robot-to-
|
|
robot L2).
|
|
- Robots hold `<lanIP>/32` with a default route via the switch. The host routes
|
|
to them and answers ARP for that IP on `externalInterface` (proxy ARP per IP;
|
|
global proxy ARP stays off), so each packet passes the host firewall.
|
|
- nftables: a bridge table drops spoofed addresses; an `inet robotnix` table
|
|
does the per-robot filtering before `nixos-fw` (priority filter - 10). IPv6
|
|
from robots is dropped (`host0` is IPv4-only). Inbound from the LAN to a
|
|
robot is open on every port — make your own decisions inside the robot.
|
|
- Per-robot dnsmasq on the host (port 5300+id) is the robot's only resolver:
|
|
all port 53 is DNAT'd to it, DoT is dropped. The robot cannot change its
|
|
resolv.conf meaningfully; it cannot touch the host's nftables/networkd.
|
|
|
|
## Access tiers
|
|
|
|
`access.shared` + `robots.<r>.access` merge per robot (`internet` from the
|
|
robot tier wins, all lists are unions). Rule order in the out-chain:
|
|
|
|
1. established/related
|
|
2. port 53 served by the host / 853 dropped (DoT)
|
|
3. `blockedCidrs` and blocked-domain IPs (nftsets fed by dnsmasq `nftset=`)
|
|
4. `lanAccess` / `hostAccess` endpoints (host input is matched per-robot by
|
|
port-source rules)
|
|
5. RFC1918/CGNAT/link-local drop unless `allowLan`
|
|
6. internet: `all` (default) or, with `allowlist`, only `allowedCidrs` +
|
|
allowed-domain IPs
|
|
|
|
Domain rules are IP-based: acceptable for host-controlled filtering, but a
|
|
CDN shared with an allowed domain also lets that IP's other sites through (and
|
|
dnsmasq learns current IPs). `blockedDomains` resolve to NXDOMAIN; in
|
|
allowlist mode everything not allowed is NXDOMAIN + dropped.
|
|
|
|
## Git-sync of the robot's flake (default)
|
|
|
|
On first install / reseed, the seed creates a git repo at `/etc/nixos`
|
|
(`git.enable`, default on) and sets `git.remoteUrl` as `origin` when given. A
|
|
timer inside the robot (`robotnix-flake-gitsync` = `git add -A`, commit, push
|
|
if origin exists) runs on `git.interval` (`daily` default). The robot needs the
|
|
remote reachable (e.g. through `lanAccess`) and its own credentials if the
|
|
remote is private.
|
|
|
|
## The `robotnix` CLI
|
|
|
|
Nushell script (`pkgs.writers.writeNuBin`), part of this module:
|
|
`list`, `status`, `start/stop/restart`, `shell`, `snapshot`,
|
|
`snapshots`, `forget`, `restore`, `reseed [--wipe-home]`, `access`.
|
|
Runtime state comes from `/etc/robotnix.json`, written by the module.
|
|
|
|
## Notes / decisions
|
|
|
|
- **nspawn, not `oci-containers`**: the server/GUIDELINES rule about
|
|
`oci-containers` targets Docker/Podman. nspawn containers need
|
|
`systemd-nspawn` and the NixOS container init; there is no OCI equivalent.
|
|
- This is why `systemd-nspawn` is launched by a hand-written unit rather than
|
|
the `containers.*` module: that module shares the host store, which would
|
|
break the robot's self-rebuilds. The unit mirrors `nixos-containers.nix`
|
|
(`Type=notify`, SIGRTMIN+3, exit status 133, `machine.slice`).
|
|
- The robot's root gets `/usr/bin` before its first start: `systemd-nspawn`
|
|
refuses a tree without `/usr` ("doesn't look like it has an OS tree"), the
|
|
same reason `nixos-containers.nix` creates it. The seed's tail block runs on
|
|
every start, so no reseed is needed after this landed.
|
|
- `systemd-nspawn` runs in `machine.slice` (systemd's nspawn home), not
|
|
`system.slice`.
|
|
- Timers use `skyg.server.timers`; eval asserts unique robot ids/IPs, name
|
|
length ≤ 12 (interface-name limit), no clash with
|
|
`skyg.internalNetworkingMap.apps`, and no domain/CIDR both allowed and
|
|
blocked.
|
|
- `fsfreeze` fails on unfrozen ro/sparse quirks only when the images are not
|
|
mounted (robot stopped); the CLI snapshots only mounted filesystems' mounts,
|
|
otherwise it snapshots the subvolumes directly.
|
|
- flake inputs: only the host nixpkgs is pinned into the robot's flake
|
|
(`github:NixOS/nixpkgs/<host-rev>`). Extra inputs belong in the robot's own
|
|
`configuration.nix` edits.
|