using seperate robotnix
This commit is contained in:
1 parent
0efbc5817b
commit
62e73eb4d2
17 files changed
+158
-2086
No files matched your search
@@ -1,234 +1,31 @@
|
||||
# robotnix — self-managed NixOS nspawn robots
|
||||
# robotnix — adapter for the robotnix flake
|
||||
|
||||
`skyg.nixos.server.robotnix` runs persistent NixOS `systemd-nspawn` containers
|
||||
("robots") that fully own their own system. Each robot has a real LAN IP, hard
|
||||
disk limits, btrfs snapshots, and network access the host controls completely.
|
||||
robotnix (self-managed NixOS `systemd-nspawn` robots) lives in its own repo:
|
||||
`git+ssh://gitea@gitea.app.internal:22/ari/robotnix.git`, a flake input of
|
||||
this repo. Its README and `docs/` describe the design, options, CLI and
|
||||
tests. Hosts configure it under `services.robotnix` (like `services.hermenix`).
|
||||
|
||||
## Model
|
||||
This folder only holds `default.nix`, which:
|
||||
|
||||
- **The robot owns its system.** On first install (`robotnix-<r>-seed.service`)
|
||||
the host builds a starting system with the same entry point the robot's own
|
||||
flake uses (`lib.nixosSystem` = eval-config with `system = null`), installs it
|
||||
into `root.img` with `nixos-install --no-bootloader`, and writes
|
||||
`/etc/nixos/{flake.nix,robotnix.nix,configuration.nix}`. After that the host
|
||||
never touches the robot's root: it rebuilds with
|
||||
`nixos-rebuild switch --flake /etc/nixos` against its own pinned nixpkgs and
|
||||
its own store. Changes survive reboots.
|
||||
- **`robotnix.nix` is host-owned** (network, user, ssh keys, git-sync).
|
||||
`configuration.nix` is the robot's own config (`seed.configuration`, default
|
||||
`./seed/configuration.nix`). Changing a seed config only matters at the next
|
||||
`robotnix reseed`.
|
||||
- **No host root.** Robots run with `--private-users=pick` unless
|
||||
`privateUsers = false`.
|
||||
- imports `robotnix.nixosModules.default` (passed in via `specialArgs`);
|
||||
- defaults `user`, `authorizedKeys`, `dnsUpstreams` and `localDomains` from
|
||||
the `skyg` user, the SSH master keys and the internal networking map;
|
||||
- asserts no robot address clashes with an app in
|
||||
`skyg.internalNetworkingMap.apps`.
|
||||
|
||||
## Storage
|
||||
## Decisions
|
||||
|
||||
Per robot under `<baseDir>/<robot>/` (btrfs subvolumes, `nodatacow` images):
|
||||
|
||||
```
|
||||
root/root.img 100G (default) sparse ext4, whole OS incl. /nix
|
||||
home/home.img 60G (default) sparse ext4, /home
|
||||
snapshots/<id>/{root,home} read-only subvolume snapshots
|
||||
```
|
||||
|
||||
The host mounts the images at `/run/robotnix/<robot>/` and passes them in, so a
|
||||
snapshot freezes both filesystems (`fsfreeze`) for a consistent point in time
|
||||
while the robot runs. Sizes are hard limits; raise them and the images grow on
|
||||
the next restart (shrinking is not supported).
|
||||
|
||||
Snapshots: per-robot `snapshots.enable/interval/keep` via `skyg.server.timers`
|
||||
(`robotnix-<r>-snapshot`), `robotnix snapshot|restore|forget|snapshots`.
|
||||
`restore` saves a `pre-restore` snapshot first. Because images are `nodatacow`,
|
||||
keep the automatic snapshot count modest — every live snapshot reintroduces
|
||||
copy-on-write for the referenced blocks. Backups are out of scope for now.
|
||||
|
||||
## Network
|
||||
|
||||
One routed virtual switch:
|
||||
|
||||
- `robotnix-br` bridge (networkd) on 10.78.0.1/24, unmanaged by
|
||||
NetworkManager; each robot's `vb-<robot>` port is `Isolated` (no robot-to-
|
||||
robot L2).
|
||||
- Robots hold `<lanIP>/32` with a default route via the switch. The host routes
|
||||
to them and answers ARP for that IP on `externalInterface` (proxy ARP per IP;
|
||||
global proxy ARP stays off), so each packet passes the host firewall.
|
||||
- nftables: a bridge table drops spoofed addresses; an `inet robotnix` table
|
||||
does the per-robot filtering before `nixos-fw` (priority filter - 10). IPv6
|
||||
from robots is dropped (`host0` is IPv4-only). Inbound from the LAN to a
|
||||
robot is open on every port — make your own decisions inside the robot.
|
||||
- Guards in the same table, so other modules' NAT can't reroute robot
|
||||
traffic (hermenix once DNAT'd any-destination traffic on some ports into
|
||||
its VM):
|
||||
- a connection meant for a robot IP that leaves anywhere but the switch,
|
||||
or lands on the host itself, is dropped (`ct original ip daddr`; replies
|
||||
are untouched via `ct direction original`);
|
||||
- a robot's own new connection that was DNAT'd is dropped. The robot DNS
|
||||
redirect still works: it goes to the host (input), not forward.
|
||||
- Per-robot dnsmasq on the host (port 5300+id) is the robot's only resolver:
|
||||
all port 53 is DNAT'd to it, DoT is dropped. The robot cannot change its
|
||||
resolv.conf meaningfully; it cannot touch the host's nftables/networkd.
|
||||
|
||||
## Access tiers
|
||||
|
||||
`access.shared` + `robots.<r>.access` merge per robot (`internet` from the
|
||||
robot tier wins, all lists are unions). Rule order in the out-chain:
|
||||
|
||||
1. established/related
|
||||
2. port 53 served by the host / 853 dropped (DoT)
|
||||
3. `blockedCidrs` and blocked-domain IPs (nftsets fed by dnsmasq `nftset=`)
|
||||
4. `lanAccess` / `hostAccess` endpoints (host input is matched per-robot by
|
||||
port-source rules)
|
||||
5. RFC1918/CGNAT/link-local drop unless `allowLan`
|
||||
6. internet: `all` (default) or, with `allowlist`, only `allowedCidrs` +
|
||||
allowed-domain IPs
|
||||
|
||||
Domain rules are IP-based: acceptable for host-controlled filtering, but a
|
||||
CDN shared with an allowed domain also lets that IP's other sites through (and
|
||||
dnsmasq learns current IPs). `blockedDomains` resolve to NXDOMAIN; in
|
||||
allowlist mode everything not allowed is NXDOMAIN + dropped.
|
||||
|
||||
## Git-sync of the robot's flake (default)
|
||||
|
||||
On first install / reseed, the seed creates a git repo at `/etc/nixos`
|
||||
(`git.enable`, default on) and sets `git.remoteUrl` as `origin` when given. A
|
||||
timer inside the robot (`robotnix-flake-gitsync` = `git add -A`, commit, push
|
||||
if origin exists) runs on `git.interval` (`daily` default). The robot needs the
|
||||
remote reachable (e.g. through `lanAccess`) and its own credentials if the
|
||||
remote is private.
|
||||
|
||||
## Ephemeral robots (opt-in)
|
||||
|
||||
`ephemeral = true` resets root to a clean copy of the seed on every start
|
||||
(host boot, `robotnix restart`, or a reboot inside). Only `/home` survives.
|
||||
|
||||
- The seed step saves the fresh root as a read-only subvolume
|
||||
`<base>/pristine` before the robot first boots, plus `pristine.id` (hash of
|
||||
the seed system and the files it installs).
|
||||
- Before each start, `robotnix-<r>-reset.service` unmounts root, swaps
|
||||
`<base>/root` for a snapshot of `pristine`, and remounts (about a second).
|
||||
It is its own oneshot (no `RemainAfterExit`, so every robot start re-runs
|
||||
it) because the robot unit's `DevicePolicy=closed` blocks the loop devices
|
||||
the mount needs.
|
||||
- When the seed changes, the storage step sees a different id on the next
|
||||
host switch, wipes root, and the seed step reinstalls (a few minutes).
|
||||
- `persist = [ "/var/lib/..." ]` keeps folders: stored in `/home/.persist`
|
||||
inside the home image and bind-mounted at their usual path. SSH host keys
|
||||
always go to `/home/.persist/ssh`, so they don't change each boot.
|
||||
- The robot still has its own writable store, so `nix`/`devenv` work until
|
||||
the next restart. Changes go through `seed.configuration` on the host.
|
||||
- Snapshots and `restore` cover home only; `reseed` drops `pristine` too.
|
||||
- git-sync can't work (fresh repo each start): eval asserts it is off.
|
||||
|
||||
## Hermes in every robot (opt-in)
|
||||
|
||||
`hermes.enable` (shared) runs one Hermes agent per robot with the upstream
|
||||
`services.hermes-agent` module; `robots.<r>.hermes.enable` can opt out.
|
||||
|
||||
- The robot's flake gets a `hermes-agent` input pinned to the host's locked
|
||||
rev, and the host's seed build includes the same module.
|
||||
- Host-managed: `hermes.settings` (shared + robot, deep-merged) goes to
|
||||
config.yaml; `hermes.name` writes SOUL.md ("You are <name>, a Hermes-based
|
||||
agent ...") on every boot; null keeps the stock identity.
|
||||
- Secrets: `hermes.environmentFiles` (shared + robot, KEY=value, e.g. agenix)
|
||||
are passed with `--load-credential` and loaded by the `hermes-agent` unit
|
||||
from `/run/credentials/@system/robotnix-hermes-<n>` (later files win). No
|
||||
file ownership games with the user namespace. Restart the robot after a
|
||||
secret changes.
|
||||
- The `hermes` user/group have a fixed id (870) so kept files stay owned
|
||||
right across reinstalls; the login user is in the `hermes` group and has
|
||||
the `hermes` CLI.
|
||||
- Ephemeral robots keep only `cron/`, `memories/` and `skills/` of
|
||||
`/var/lib/hermes/.hermes` (added to `persist` automatically). Sessions,
|
||||
logs, OAuth logins and pairings reset each start: use API keys and
|
||||
`*_ALLOWED_USERS` env vars instead.
|
||||
- Gateway API (`hermes.gateway`, on by default, port 8642): the host sets
|
||||
`services.hermes-agent.environment` (`API_SERVER_ENABLED`, `_HOST=0.0.0.0`,
|
||||
`_PORT`, `_MODEL_NAME` = `hermes.name` or the robot name). These go to
|
||||
`$HERMES_HOME/.env`, which beats the same names in the secrets. Each
|
||||
robot's secrets need `API_SERVER_KEY`, or the API server won't start.
|
||||
- Dashboard (`hermes.dashboard`, on by default, port 80):
|
||||
`backend.mode = "dashboard"` on 0.0.0.0, run as `hermes-backend.service`
|
||||
with the same secret files, plus `CAP_NET_BIND_SERVICE` when the port is
|
||||
below 1024. A non-loopback dashboard needs a login in the secrets:
|
||||
`HERMES_DASHBOARD_BASIC_AUTH_USERNAME`, `..._PASSWORD_HASH` (or
|
||||
`..._PASSWORD`) and `..._SECRET` (keeps sessions across restarts). Without
|
||||
one it exits and systemd keeps restarting it; the gateway keeps working.
|
||||
Port 80 may clash with the robot's own compose stacks: change the port.
|
||||
- Both: top-level `enable/port/openFirewall`, robots override any of them
|
||||
(null = follow). `openFirewall` opens the TCP port in the robot's own
|
||||
firewall. Eval asserts the two don't share a port.
|
||||
- Existing robots only get Hermes (and gateway/dashboard changes) after
|
||||
`robotnix reseed` (flake.nix and robotnix.nix are written at install).
|
||||
Ephemeral robots pick changes up on the next host switch.
|
||||
|
||||
### Buzz identity for a robot's Hermes
|
||||
|
||||
Each robot gets its own Buzz (Nostr) identity in its own `robots-<name>` secret:
|
||||
|
||||
1. `skyg buzz identity <robot>` makes a key and
|
||||
merges `BUZZ_RELAY_URL` / `BUZZ_PRIVATE_KEY` (and the optional `BUZZ_*`
|
||||
vars) into `secrets/robots-<robot>.age`. Other lines are kept. It needs the
|
||||
secret to exist and refuses to replace a key without `--rotate`.
|
||||
The bot answers anyone on the relay by default; `--allowed-users <npubs>`
|
||||
limits it, `--restricted` allows no one.
|
||||
2. Add the printed npub to the relay: `sudo buzz-manage add-member <npub>` on
|
||||
hl-minipc1.
|
||||
3. The robot needs `lanAccess` to the relay (`10.0.101.3` ports 443, 3000,
|
||||
8080, 9102; the default relay URL uses 443) and the `buzz` CLI. The robot's
|
||||
nixpkgs has no host overlay, so its config builds it with
|
||||
`pkgs.callPackage <repo>/pkgs/buzz-cli { }` and points `BUZZ_CLI_PATH` at it
|
||||
(`services.hermes-agent.environment`), because the service PATH has no
|
||||
system packages. That relative path only resolves for the host's seed
|
||||
build, not for a rebuild inside the robot. Restart the robot after the
|
||||
secret changes.
|
||||
4. A new key has no profile, and Hermes won't connect without one
|
||||
(`buzz users get returned no profile`). sibli runs a `buzz-profile`
|
||||
oneshot at start that calls `buzz users set-profile` when the key has none.
|
||||
|
||||
## Docker and Tailscale (opt-in, host side only)
|
||||
|
||||
These options only change what the host allows. Turn the service on in the
|
||||
robot's own config (e.g. its `seed.configuration`).
|
||||
|
||||
- `allowDocker`: nspawn gets `--system-call-filter=add_key keyctl bpf`
|
||||
(blocked by default, Docker needs them). Docker runs inside the robot's user
|
||||
namespace, so images using UIDs above 65535 won't run.
|
||||
- `allowTun`: `/dev/net/tun` is bound in and allowed on the unit
|
||||
(`DevicePolicy=closed` blocks it otherwise). Used for tailscale: the robot
|
||||
is its own tailnet machine, log in once inside with `sudo tailscale up`.
|
||||
Direct LAN paths to other tailnet peers hit the private-range drop, so it
|
||||
uses public endpoints or DERP.
|
||||
- Both apply on the next robot restart; no reseed needed.
|
||||
|
||||
## The `robotnix` CLI
|
||||
|
||||
Nushell script (`pkgs.writers.writeNuBin`), part of this module:
|
||||
`list`, `status`, `start/stop/restart`, `shell`, `snapshot`,
|
||||
`snapshots`, `forget`, `restore`, `reseed [--wipe-home]`, `access`.
|
||||
Runtime state comes from `/etc/robotnix.json`, written by the module.
|
||||
|
||||
## Notes / decisions
|
||||
|
||||
- **nspawn, not `oci-containers`**: the server/GUIDELINES rule about
|
||||
`oci-containers` targets Docker/Podman. nspawn containers need
|
||||
`systemd-nspawn` and the NixOS container init; there is no OCI equivalent.
|
||||
- This is why `systemd-nspawn` is launched by a hand-written unit rather than
|
||||
the `containers.*` module: that module shares the host store, which would
|
||||
break the robot's self-rebuilds. The unit mirrors `nixos-containers.nix`
|
||||
(`Type=notify`, SIGRTMIN+3, exit status 133, `machine.slice`).
|
||||
- The robot's root gets `/usr/bin` before its first start: `systemd-nspawn`
|
||||
refuses a tree without `/usr` ("doesn't look like it has an OS tree"), the
|
||||
same reason `nixos-containers.nix` creates it. The seed's tail block runs on
|
||||
every start, so no reseed is needed after this landed.
|
||||
- `systemd-nspawn` runs in `machine.slice` (systemd's nspawn home), not
|
||||
`system.slice`.
|
||||
- Timers use `skyg.server.timers`; eval asserts unique robot ids/IPs, name
|
||||
length ≤ 12 (interface-name limit), no clash with
|
||||
`skyg.internalNetworkingMap.apps`, and no domain/CIDR both allowed and
|
||||
blocked.
|
||||
- `fsfreeze` fails on unfrozen ro/sparse quirks only when the images are not
|
||||
mounted (robot stopped); the CLI snapshots only mounted filesystems' mounts,
|
||||
otherwise it snapshots the subvolumes directly.
|
||||
- flake inputs: only the host nixpkgs is pinned into the robot's flake
|
||||
(`github:NixOS/nixpkgs/<host-rev>`). Extra inputs belong in the robot's own
|
||||
`configuration.nix` edits.
|
||||
- **Own repo** (2026-10): robotnix moved out of `modules/nixos/server` so it
|
||||
can grow (storage backends, Genix, child robots) with its own tests.
|
||||
- **Namespace**: `services.robotnix`, not `skyg.*`, same exception as
|
||||
hermenix: the options belong to an external flake.
|
||||
- **hermes-agent pin**: the input follows this repo's `hermes-agent`, so
|
||||
robots pin the same revision the host locked.
|
||||
- **Buzz stays here**: robotnix knows nothing about Buzz. A robot's seed
|
||||
config (e.g. `hosts/hl-fwdesk/services/robotnix/sibli.nix`) builds
|
||||
`pkgs/buzz-cli` from this repo and sets `BUZZ_CLI_PATH` itself. That
|
||||
relative path works for the seed build only, which is fine for ephemeral
|
||||
robots; a persistent robot that rebuilds itself would need its own copy.
|
||||
- **nspawn, not `oci-containers`**: the oci-containers rule targets
|
||||
Docker/Podman. nspawn robots need their own store for self-rebuilds, which
|
||||
the `containers.*` module can't give.
|
||||
Reference in new issue
Block a user