Files
nixos-setup/modules/nixos/server/robotnix/ABOUTME.md
T
2026-10-26 16:18:48 -04:00

231 lines
12 KiB
Markdown

# robotnix — self-managed NixOS nspawn robots
`skyg.nixos.server.robotnix` runs persistent NixOS `systemd-nspawn` containers
("robots") that fully own their own system. Each robot has a real LAN IP, hard
disk limits, btrfs snapshots, and network access the host controls completely.
## Model
- **The robot owns its system.** On first install (`robotnix-<r>-seed.service`)
the host builds a starting system with the same entry point the robot's own
flake uses (`lib.nixosSystem` = eval-config with `system = null`), installs it
into `root.img` with `nixos-install --no-bootloader`, and writes
`/etc/nixos/{flake.nix,robotnix.nix,configuration.nix}`. After that the host
never touches the robot's root: it rebuilds with
`nixos-rebuild switch --flake /etc/nixos` against its own pinned nixpkgs and
its own store. Changes survive reboots.
- **`robotnix.nix` is host-owned** (network, user, ssh keys, git-sync).
`configuration.nix` is the robot's own config (`seed.configuration`, default
`./seed/configuration.nix`). Changing a seed config only matters at the next
`robotnix reseed`.
- **No host root.** Robots run with `--private-users=pick` unless
`privateUsers = false`.
## Storage
Per robot under `<baseDir>/<robot>/` (btrfs subvolumes, `nodatacow` images):
```
root/root.img 100G (default) sparse ext4, whole OS incl. /nix
home/home.img 60G (default) sparse ext4, /home
snapshots/<id>/{root,home} read-only subvolume snapshots
```
The host mounts the images at `/run/robotnix/<robot>/` and passes them in, so a
snapshot freezes both filesystems (`fsfreeze`) for a consistent point in time
while the robot runs. Sizes are hard limits; raise them and the images grow on
the next restart (shrinking is not supported).
Snapshots: per-robot `snapshots.enable/interval/keep` via `skyg.server.timers`
(`robotnix-<r>-snapshot`), `robotnix snapshot|restore|forget|snapshots`.
`restore` saves a `pre-restore` snapshot first. Because images are `nodatacow`,
keep the automatic snapshot count modest — every live snapshot reintroduces
copy-on-write for the referenced blocks. Backups are out of scope for now.
## Network
One routed virtual switch:
- `robotnix-br` bridge (networkd) on 10.78.0.1/24, unmanaged by
NetworkManager; each robot's `vb-<robot>` port is `Isolated` (no robot-to-
robot L2).
- Robots hold `<lanIP>/32` with a default route via the switch. The host routes
to them and answers ARP for that IP on `externalInterface` (proxy ARP per IP;
global proxy ARP stays off), so each packet passes the host firewall.
- nftables: a bridge table drops spoofed addresses; an `inet robotnix` table
does the per-robot filtering before `nixos-fw` (priority filter - 10). IPv6
from robots is dropped (`host0` is IPv4-only). Inbound from the LAN to a
robot is open on every port — make your own decisions inside the robot.
- Guards in the same table, so other modules' NAT can't reroute robot
traffic (hermenix once DNAT'd any-destination traffic on some ports into
its VM):
- a connection meant for a robot IP that leaves anywhere but the switch,
or lands on the host itself, is dropped (`ct original ip daddr`; replies
are untouched via `ct direction original`);
- a robot's own new connection that was DNAT'd is dropped. The robot DNS
redirect still works: it goes to the host (input), not forward.
- Per-robot dnsmasq on the host (port 5300+id) is the robot's only resolver:
all port 53 is DNAT'd to it, DoT is dropped. The robot cannot change its
resolv.conf meaningfully; it cannot touch the host's nftables/networkd.
## Access tiers
`access.shared` + `robots.<r>.access` merge per robot (`internet` from the
robot tier wins, all lists are unions). Rule order in the out-chain:
1. established/related
2. port 53 served by the host / 853 dropped (DoT)
3. `blockedCidrs` and blocked-domain IPs (nftsets fed by dnsmasq `nftset=`)
4. `lanAccess` / `hostAccess` endpoints (host input is matched per-robot by
port-source rules)
5. RFC1918/CGNAT/link-local drop unless `allowLan`
6. internet: `all` (default) or, with `allowlist`, only `allowedCidrs` +
allowed-domain IPs
Domain rules are IP-based: acceptable for host-controlled filtering, but a
CDN shared with an allowed domain also lets that IP's other sites through (and
dnsmasq learns current IPs). `blockedDomains` resolve to NXDOMAIN; in
allowlist mode everything not allowed is NXDOMAIN + dropped.
## Git-sync of the robot's flake (default)
On first install / reseed, the seed creates a git repo at `/etc/nixos`
(`git.enable`, default on) and sets `git.remoteUrl` as `origin` when given. A
timer inside the robot (`robotnix-flake-gitsync` = `git add -A`, commit, push
if origin exists) runs on `git.interval` (`daily` default). The robot needs the
remote reachable (e.g. through `lanAccess`) and its own credentials if the
remote is private.
## Ephemeral robots (opt-in)
`ephemeral = true` resets root to a clean copy of the seed on every start
(host boot, `robotnix restart`, or a reboot inside). Only `/home` survives.
- The seed step saves the fresh root as a read-only subvolume
`<base>/pristine` before the robot first boots, plus `pristine.id` (hash of
the seed system and the files it installs).
- Before each start, `robotnix-<r>-reset.service` unmounts root, swaps
`<base>/root` for a snapshot of `pristine`, and remounts (about a second).
It is its own oneshot (no `RemainAfterExit`, so every robot start re-runs
it) because the robot unit's `DevicePolicy=closed` blocks the loop devices
the mount needs.
- When the seed changes, the storage step sees a different id on the next
host switch, wipes root, and the seed step reinstalls (a few minutes).
- `persist = [ "/var/lib/..." ]` keeps folders: stored in `/home/.persist`
inside the home image and bind-mounted at their usual path. SSH host keys
always go to `/home/.persist/ssh`, so they don't change each boot.
- The robot still has its own writable store, so `nix`/`devenv` work until
the next restart. Changes go through `seed.configuration` on the host.
- Snapshots and `restore` cover home only; `reseed` drops `pristine` too.
- git-sync can't work (fresh repo each start): eval asserts it is off.
## Hermes in every robot (opt-in)
`hermes.enable` (shared) runs one Hermes agent per robot with the upstream
`services.hermes-agent` module; `robots.<r>.hermes.enable` can opt out.
- The robot's flake gets a `hermes-agent` input pinned to the host's locked
rev, and the host's seed build includes the same module.
- Host-managed: `hermes.settings` (shared + robot, deep-merged) goes to
config.yaml; `hermes.name` writes SOUL.md ("You are <name>, a Hermes-based
agent ...") on every boot; null keeps the stock identity.
- Secrets: `hermes.environmentFiles` (shared + robot, KEY=value, e.g. agenix)
are passed with `--load-credential` and loaded by the `hermes-agent` unit
from `/run/credentials/@system/robotnix-hermes-<n>` (later files win). No
file ownership games with the user namespace. Restart the robot after a
secret changes.
- The `hermes` user/group have a fixed id (870) so kept files stay owned
right across reinstalls; the login user is in the `hermes` group and has
the `hermes` CLI.
- Ephemeral robots keep only `cron/`, `memories/` and `skills/` of
`/var/lib/hermes/.hermes` (added to `persist` automatically). Sessions,
logs, OAuth logins and pairings reset each start: use API keys and
`*_ALLOWED_USERS` env vars instead.
- Gateway API (`hermes.gateway`, on by default, port 8642): the host sets
`services.hermes-agent.environment` (`API_SERVER_ENABLED`, `_HOST=0.0.0.0`,
`_PORT`, `_MODEL_NAME` = `hermes.name` or the robot name). These go to
`$HERMES_HOME/.env`, which beats the same names in the secrets. Each
robot's secrets need `API_SERVER_KEY`, or the API server won't start.
- Dashboard (`hermes.dashboard`, on by default, port 80):
`backend.mode = "dashboard"` on 0.0.0.0, run as `hermes-backend.service`
with the same secret files, plus `CAP_NET_BIND_SERVICE` when the port is
below 1024. A non-loopback dashboard needs a login in the secrets:
`HERMES_DASHBOARD_BASIC_AUTH_USERNAME`, `..._PASSWORD_HASH` (or
`..._PASSWORD`) and `..._SECRET` (keeps sessions across restarts). Without
one it exits and systemd keeps restarting it; the gateway keeps working.
Port 80 may clash with the robot's own compose stacks: change the port.
- Both: top-level `enable/port/openFirewall`, robots override any of them
(null = follow). `openFirewall` opens the TCP port in the robot's own
firewall. Eval asserts the two don't share a port.
- Existing robots only get Hermes (and gateway/dashboard changes) after
`robotnix reseed` (flake.nix and robotnix.nix are written at install).
Ephemeral robots pick changes up on the next host switch.
### Buzz identity for a robot's Hermes
Each robot gets its own Buzz (Nostr) identity in its own `robots-<name>` secret:
1. `skyg buzz identity <robot>` makes a key and
merges `BUZZ_RELAY_URL` / `BUZZ_PRIVATE_KEY` (and the optional `BUZZ_*`
vars) into `secrets/robots-<robot>.age`. Other lines are kept. It needs the
secret to exist and refuses to replace a key without `--rotate`.
The bot answers anyone on the relay by default; `--allowed-users <npubs>`
limits it, `--restricted` allows no one.
2. Add the printed npub to the relay: `sudo buzz-manage add-member <npub>` on
hl-minipc1.
3. The robot needs `lanAccess` to the relay and the `buzz` CLI. The robot's
nixpkgs has no host overlay, so its config builds it with
`pkgs.callPackage <repo>/pkgs/buzz-cli { }` and points `BUZZ_CLI_PATH` at it
(`services.hermes-agent.environment`), because the service PATH has no
system packages. That relative path only resolves for the host's seed
build, not for a rebuild inside the robot. Restart the robot after the
secret changes.
## Docker and Tailscale (opt-in, host side only)
These options only change what the host allows. Turn the service on in the
robot's own config (e.g. its `seed.configuration`).
- `allowDocker`: nspawn gets `--system-call-filter=add_key keyctl bpf`
(blocked by default, Docker needs them). Docker runs inside the robot's user
namespace, so images using UIDs above 65535 won't run.
- `allowTun`: `/dev/net/tun` is bound in and allowed on the unit
(`DevicePolicy=closed` blocks it otherwise). Used for tailscale: the robot
is its own tailnet machine, log in once inside with `sudo tailscale up`.
Direct LAN paths to other tailnet peers hit the private-range drop, so it
uses public endpoints or DERP.
- Both apply on the next robot restart; no reseed needed.
## The `robotnix` CLI
Nushell script (`pkgs.writers.writeNuBin`), part of this module:
`list`, `status`, `start/stop/restart`, `shell`, `snapshot`,
`snapshots`, `forget`, `restore`, `reseed [--wipe-home]`, `access`.
Runtime state comes from `/etc/robotnix.json`, written by the module.
## Notes / decisions
- **nspawn, not `oci-containers`**: the server/GUIDELINES rule about
`oci-containers` targets Docker/Podman. nspawn containers need
`systemd-nspawn` and the NixOS container init; there is no OCI equivalent.
- This is why `systemd-nspawn` is launched by a hand-written unit rather than
the `containers.*` module: that module shares the host store, which would
break the robot's self-rebuilds. The unit mirrors `nixos-containers.nix`
(`Type=notify`, SIGRTMIN+3, exit status 133, `machine.slice`).
- The robot's root gets `/usr/bin` before its first start: `systemd-nspawn`
refuses a tree without `/usr` ("doesn't look like it has an OS tree"), the
same reason `nixos-containers.nix` creates it. The seed's tail block runs on
every start, so no reseed is needed after this landed.
- `systemd-nspawn` runs in `machine.slice` (systemd's nspawn home), not
`system.slice`.
- Timers use `skyg.server.timers`; eval asserts unique robot ids/IPs, name
length ≤ 12 (interface-name limit), no clash with
`skyg.internalNetworkingMap.apps`, and no domain/CIDR both allowed and
blocked.
- `fsfreeze` fails on unfrozen ro/sparse quirks only when the images are not
mounted (robot stopped); the CLI snapshots only mounted filesystems' mounts,
otherwise it snapshots the subvolumes directly.
- flake inputs: only the host nixpkgs is pinned into the robot's flake
(`github:NixOS/nixpkgs/<host-rev>`). Extra inputs belong in the robot's own
`configuration.nix` edits.