231 lines
12 KiB
Markdown
231 lines
12 KiB
Markdown
# robotnix — self-managed NixOS nspawn robots
|
|
|
|
`skyg.nixos.server.robotnix` runs persistent NixOS `systemd-nspawn` containers
|
|
("robots") that fully own their own system. Each robot has a real LAN IP, hard
|
|
disk limits, btrfs snapshots, and network access the host controls completely.
|
|
|
|
## Model
|
|
|
|
- **The robot owns its system.** On first install (`robotnix-<r>-seed.service`)
|
|
the host builds a starting system with the same entry point the robot's own
|
|
flake uses (`lib.nixosSystem` = eval-config with `system = null`), installs it
|
|
into `root.img` with `nixos-install --no-bootloader`, and writes
|
|
`/etc/nixos/{flake.nix,robotnix.nix,configuration.nix}`. After that the host
|
|
never touches the robot's root: it rebuilds with
|
|
`nixos-rebuild switch --flake /etc/nixos` against its own pinned nixpkgs and
|
|
its own store. Changes survive reboots.
|
|
- **`robotnix.nix` is host-owned** (network, user, ssh keys, git-sync).
|
|
`configuration.nix` is the robot's own config (`seed.configuration`, default
|
|
`./seed/configuration.nix`). Changing a seed config only matters at the next
|
|
`robotnix reseed`.
|
|
- **No host root.** Robots run with `--private-users=pick` unless
|
|
`privateUsers = false`.
|
|
|
|
## Storage
|
|
|
|
Per robot under `<baseDir>/<robot>/` (btrfs subvolumes, `nodatacow` images):
|
|
|
|
```
|
|
root/root.img 100G (default) sparse ext4, whole OS incl. /nix
|
|
home/home.img 60G (default) sparse ext4, /home
|
|
snapshots/<id>/{root,home} read-only subvolume snapshots
|
|
```
|
|
|
|
The host mounts the images at `/run/robotnix/<robot>/` and passes them in, so a
|
|
snapshot freezes both filesystems (`fsfreeze`) for a consistent point in time
|
|
while the robot runs. Sizes are hard limits; raise them and the images grow on
|
|
the next restart (shrinking is not supported).
|
|
|
|
Snapshots: per-robot `snapshots.enable/interval/keep` via `skyg.server.timers`
|
|
(`robotnix-<r>-snapshot`), `robotnix snapshot|restore|forget|snapshots`.
|
|
`restore` saves a `pre-restore` snapshot first. Because images are `nodatacow`,
|
|
keep the automatic snapshot count modest — every live snapshot reintroduces
|
|
copy-on-write for the referenced blocks. Backups are out of scope for now.
|
|
|
|
## Network
|
|
|
|
One routed virtual switch:
|
|
|
|
- `robotnix-br` bridge (networkd) on 10.78.0.1/24, unmanaged by
|
|
NetworkManager; each robot's `vb-<robot>` port is `Isolated` (no robot-to-
|
|
robot L2).
|
|
- Robots hold `<lanIP>/32` with a default route via the switch. The host routes
|
|
to them and answers ARP for that IP on `externalInterface` (proxy ARP per IP;
|
|
global proxy ARP stays off), so each packet passes the host firewall.
|
|
- nftables: a bridge table drops spoofed addresses; an `inet robotnix` table
|
|
does the per-robot filtering before `nixos-fw` (priority filter - 10). IPv6
|
|
from robots is dropped (`host0` is IPv4-only). Inbound from the LAN to a
|
|
robot is open on every port — make your own decisions inside the robot.
|
|
- Guards in the same table, so other modules' NAT can't reroute robot
|
|
traffic (hermenix once DNAT'd any-destination traffic on some ports into
|
|
its VM):
|
|
- a connection meant for a robot IP that leaves anywhere but the switch,
|
|
or lands on the host itself, is dropped (`ct original ip daddr`; replies
|
|
are untouched via `ct direction original`);
|
|
- a robot's own new connection that was DNAT'd is dropped. The robot DNS
|
|
redirect still works: it goes to the host (input), not forward.
|
|
- Per-robot dnsmasq on the host (port 5300+id) is the robot's only resolver:
|
|
all port 53 is DNAT'd to it, DoT is dropped. The robot cannot change its
|
|
resolv.conf meaningfully; it cannot touch the host's nftables/networkd.
|
|
|
|
## Access tiers
|
|
|
|
`access.shared` + `robots.<r>.access` merge per robot (`internet` from the
|
|
robot tier wins, all lists are unions). Rule order in the out-chain:
|
|
|
|
1. established/related
|
|
2. port 53 served by the host / 853 dropped (DoT)
|
|
3. `blockedCidrs` and blocked-domain IPs (nftsets fed by dnsmasq `nftset=`)
|
|
4. `lanAccess` / `hostAccess` endpoints (host input is matched per-robot by
|
|
port-source rules)
|
|
5. RFC1918/CGNAT/link-local drop unless `allowLan`
|
|
6. internet: `all` (default) or, with `allowlist`, only `allowedCidrs` +
|
|
allowed-domain IPs
|
|
|
|
Domain rules are IP-based: acceptable for host-controlled filtering, but a
|
|
CDN shared with an allowed domain also lets that IP's other sites through (and
|
|
dnsmasq learns current IPs). `blockedDomains` resolve to NXDOMAIN; in
|
|
allowlist mode everything not allowed is NXDOMAIN + dropped.
|
|
|
|
## Git-sync of the robot's flake (default)
|
|
|
|
On first install / reseed, the seed creates a git repo at `/etc/nixos`
|
|
(`git.enable`, default on) and sets `git.remoteUrl` as `origin` when given. A
|
|
timer inside the robot (`robotnix-flake-gitsync` = `git add -A`, commit, push
|
|
if origin exists) runs on `git.interval` (`daily` default). The robot needs the
|
|
remote reachable (e.g. through `lanAccess`) and its own credentials if the
|
|
remote is private.
|
|
|
|
## Ephemeral robots (opt-in)
|
|
|
|
`ephemeral = true` resets root to a clean copy of the seed on every start
|
|
(host boot, `robotnix restart`, or a reboot inside). Only `/home` survives.
|
|
|
|
- The seed step saves the fresh root as a read-only subvolume
|
|
`<base>/pristine` before the robot first boots, plus `pristine.id` (hash of
|
|
the seed system and the files it installs).
|
|
- Before each start, `robotnix-<r>-reset.service` unmounts root, swaps
|
|
`<base>/root` for a snapshot of `pristine`, and remounts (about a second).
|
|
It is its own oneshot (no `RemainAfterExit`, so every robot start re-runs
|
|
it) because the robot unit's `DevicePolicy=closed` blocks the loop devices
|
|
the mount needs.
|
|
- When the seed changes, the storage step sees a different id on the next
|
|
host switch, wipes root, and the seed step reinstalls (a few minutes).
|
|
- `persist = [ "/var/lib/..." ]` keeps folders: stored in `/home/.persist`
|
|
inside the home image and bind-mounted at their usual path. SSH host keys
|
|
always go to `/home/.persist/ssh`, so they don't change each boot.
|
|
- The robot still has its own writable store, so `nix`/`devenv` work until
|
|
the next restart. Changes go through `seed.configuration` on the host.
|
|
- Snapshots and `restore` cover home only; `reseed` drops `pristine` too.
|
|
- git-sync can't work (fresh repo each start): eval asserts it is off.
|
|
|
|
## Hermes in every robot (opt-in)
|
|
|
|
`hermes.enable` (shared) runs one Hermes agent per robot with the upstream
|
|
`services.hermes-agent` module; `robots.<r>.hermes.enable` can opt out.
|
|
|
|
- The robot's flake gets a `hermes-agent` input pinned to the host's locked
|
|
rev, and the host's seed build includes the same module.
|
|
- Host-managed: `hermes.settings` (shared + robot, deep-merged) goes to
|
|
config.yaml; `hermes.name` writes SOUL.md ("You are <name>, a Hermes-based
|
|
agent ...") on every boot; null keeps the stock identity.
|
|
- Secrets: `hermes.environmentFiles` (shared + robot, KEY=value, e.g. agenix)
|
|
are passed with `--load-credential` and loaded by the `hermes-agent` unit
|
|
from `/run/credentials/@system/robotnix-hermes-<n>` (later files win). No
|
|
file ownership games with the user namespace. Restart the robot after a
|
|
secret changes.
|
|
- The `hermes` user/group have a fixed id (870) so kept files stay owned
|
|
right across reinstalls; the login user is in the `hermes` group and has
|
|
the `hermes` CLI.
|
|
- Ephemeral robots keep only `cron/`, `memories/` and `skills/` of
|
|
`/var/lib/hermes/.hermes` (added to `persist` automatically). Sessions,
|
|
logs, OAuth logins and pairings reset each start: use API keys and
|
|
`*_ALLOWED_USERS` env vars instead.
|
|
- Gateway API (`hermes.gateway`, on by default, port 8642): the host sets
|
|
`services.hermes-agent.environment` (`API_SERVER_ENABLED`, `_HOST=0.0.0.0`,
|
|
`_PORT`, `_MODEL_NAME` = `hermes.name` or the robot name). These go to
|
|
`$HERMES_HOME/.env`, which beats the same names in the secrets. Each
|
|
robot's secrets need `API_SERVER_KEY`, or the API server won't start.
|
|
- Dashboard (`hermes.dashboard`, on by default, port 80):
|
|
`backend.mode = "dashboard"` on 0.0.0.0, run as `hermes-backend.service`
|
|
with the same secret files, plus `CAP_NET_BIND_SERVICE` when the port is
|
|
below 1024. A non-loopback dashboard needs a login in the secrets:
|
|
`HERMES_DASHBOARD_BASIC_AUTH_USERNAME`, `..._PASSWORD_HASH` (or
|
|
`..._PASSWORD`) and `..._SECRET` (keeps sessions across restarts). Without
|
|
one it exits and systemd keeps restarting it; the gateway keeps working.
|
|
Port 80 may clash with the robot's own compose stacks: change the port.
|
|
- Both: top-level `enable/port/openFirewall`, robots override any of them
|
|
(null = follow). `openFirewall` opens the TCP port in the robot's own
|
|
firewall. Eval asserts the two don't share a port.
|
|
- Existing robots only get Hermes (and gateway/dashboard changes) after
|
|
`robotnix reseed` (flake.nix and robotnix.nix are written at install).
|
|
Ephemeral robots pick changes up on the next host switch.
|
|
|
|
### Buzz identity for a robot's Hermes
|
|
|
|
Each robot gets its own Buzz (Nostr) identity in its own `robots-<name>` secret:
|
|
|
|
1. `skyg buzz identity <robot>` makes a key and
|
|
merges `BUZZ_RELAY_URL` / `BUZZ_PRIVATE_KEY` (and the optional `BUZZ_*`
|
|
vars) into `secrets/robots-<robot>.age`. Other lines are kept. It needs the
|
|
secret to exist and refuses to replace a key without `--rotate`.
|
|
The bot answers anyone on the relay by default; `--allowed-users <npubs>`
|
|
limits it, `--restricted` allows no one.
|
|
2. Add the printed npub to the relay: `sudo buzz-manage add-member <npub>` on
|
|
hl-minipc1.
|
|
3. The robot needs `lanAccess` to the relay and the `buzz` CLI. The robot's
|
|
nixpkgs has no host overlay, so its config builds it with
|
|
`pkgs.callPackage <repo>/pkgs/buzz-cli { }` and points `BUZZ_CLI_PATH` at it
|
|
(`services.hermes-agent.environment`), because the service PATH has no
|
|
system packages. That relative path only resolves for the host's seed
|
|
build, not for a rebuild inside the robot. Restart the robot after the
|
|
secret changes.
|
|
|
|
## Docker and Tailscale (opt-in, host side only)
|
|
|
|
These options only change what the host allows. Turn the service on in the
|
|
robot's own config (e.g. its `seed.configuration`).
|
|
|
|
- `allowDocker`: nspawn gets `--system-call-filter=add_key keyctl bpf`
|
|
(blocked by default, Docker needs them). Docker runs inside the robot's user
|
|
namespace, so images using UIDs above 65535 won't run.
|
|
- `allowTun`: `/dev/net/tun` is bound in and allowed on the unit
|
|
(`DevicePolicy=closed` blocks it otherwise). Used for tailscale: the robot
|
|
is its own tailnet machine, log in once inside with `sudo tailscale up`.
|
|
Direct LAN paths to other tailnet peers hit the private-range drop, so it
|
|
uses public endpoints or DERP.
|
|
- Both apply on the next robot restart; no reseed needed.
|
|
|
|
## The `robotnix` CLI
|
|
|
|
Nushell script (`pkgs.writers.writeNuBin`), part of this module:
|
|
`list`, `status`, `start/stop/restart`, `shell`, `snapshot`,
|
|
`snapshots`, `forget`, `restore`, `reseed [--wipe-home]`, `access`.
|
|
Runtime state comes from `/etc/robotnix.json`, written by the module.
|
|
|
|
## Notes / decisions
|
|
|
|
- **nspawn, not `oci-containers`**: the server/GUIDELINES rule about
|
|
`oci-containers` targets Docker/Podman. nspawn containers need
|
|
`systemd-nspawn` and the NixOS container init; there is no OCI equivalent.
|
|
- This is why `systemd-nspawn` is launched by a hand-written unit rather than
|
|
the `containers.*` module: that module shares the host store, which would
|
|
break the robot's self-rebuilds. The unit mirrors `nixos-containers.nix`
|
|
(`Type=notify`, SIGRTMIN+3, exit status 133, `machine.slice`).
|
|
- The robot's root gets `/usr/bin` before its first start: `systemd-nspawn`
|
|
refuses a tree without `/usr` ("doesn't look like it has an OS tree"), the
|
|
same reason `nixos-containers.nix` creates it. The seed's tail block runs on
|
|
every start, so no reseed is needed after this landed.
|
|
- `systemd-nspawn` runs in `machine.slice` (systemd's nspawn home), not
|
|
`system.slice`.
|
|
- Timers use `skyg.server.timers`; eval asserts unique robot ids/IPs, name
|
|
length ≤ 12 (interface-name limit), no clash with
|
|
`skyg.internalNetworkingMap.apps`, and no domain/CIDR both allowed and
|
|
blocked.
|
|
- `fsfreeze` fails on unfrozen ro/sparse quirks only when the images are not
|
|
mounted (robot stopped); the CLI snapshots only mounted filesystems' mounts,
|
|
otherwise it snapshots the subvolumes directly.
|
|
- flake inputs: only the host nixpkgs is pinned into the robot's flake
|
|
(`github:NixOS/nixpkgs/<host-rev>`). Extra inputs belong in the robot's own
|
|
`configuration.nix` edits.
|