diff --git a/AGENTS.md b/AGENTS.md index c0166a1..0d14eab 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -8,10 +8,12 @@ Key non-negotiable rules from that file: - **Always present a plan first** and wait for user approval before making any changes — no matter how small the task. -- **Never run builds or disk-writing steps** (`nix build`, `hermenix setup`'s - disk creation/image-write steps, `dd` to any disk image, `nixos-rebuild - switch` on any consuming host) — zero exceptions, even if asked. -- Never run `hermes-vm rollback` or `hermes-ctl reseed` without explicit, +- **Never run builds or disk-writing steps directly** (`nix build`, + `nix flake check`, `hermenix setup`'s disk/image steps, `dd` to any disk + image, `nixos-rebuild` on any consuming host). The **only** allowed build + path is the test wrappers: `bin/runtest`, `bin/runvmtest`, and their + `-debug` variants. +- Never run `hermenix vm rollback` or `hermenix ctl reseed` without explicit, freshly-given confirmation — both destroy live guest state. - Never edit `guest/hardware.nix`'s disk labels without the operator having confirmed `lsblk -f` output first. diff --git a/GUIDELINES.md b/GUIDELINES.md index 4658280..64dacc2 100644 --- a/GUIDELINES.md +++ b/GUIDELINES.md @@ -24,20 +24,37 @@ Only after the user approves the plan may you proceed with edits. ## Hard Rules — Never Break These -### ❌ NEVER run builds or disk-writing steps — ZERO EXCEPTIONS +### ❌ NEVER run builds or disk-writing steps directly — ZERO EXCEPTIONS -**DO NOT RUN ANY OF THESE COMMANDS AT ANY TIME, FOR ANY REASON:** +**DO NOT RUN ANY OF THESE COMMANDS DIRECTLY AT ANY TIME, FOR ANY REASON:** ``` -nix build # unless it's to test the vm via 'bin/runvmtest' -nix flake check +nix build # never directly — see the test-proxy exception below +nix flake check # never directly hermenix setup # unless the user is actively running it themselves with you watching hermenix migrate # same — it writes into a live guest dd ... of=/var/lib/hermes-vm/** # writing any disk image nixos-rebuild switch / test / build (on any host consuming this flake) ``` -**Why:** +**Test-proxy exception (the only allowed build path for agents):** + +Agents **may** run the repo's test wrapper scripts. Those scripts are the +deliberate proxy through which builds happen — they scope what gets +evaluated/built and never touch live disks under `/var/lib/hermes-vm/**`: + +``` +bin/runtest # CLI unit tests (nu/tests/) — no nix build +bin/runtest-debug +bin/runvmtest # guest VM checks via nix build of flake checks only +bin/runvmtest-debug +``` + +Do **not** invent your own `nix build .#...` / `nix flake check` invocations. +If a check is not reachable through those wrappers, describe it and let the +operator run it. + +**Why the direct ban exists:** - `hermenix setup` creates/writes disk images and generates host secrets (a binary cache signing key) — destructive if repeated carelessly, and it's @@ -47,14 +64,14 @@ nixos-rebuild switch / test / build (on any host consuming this flake) - A `nixos-rebuild switch` on a consuming host is a live, production-affecting action — for `hl-fwdesk` specifically, it's the machine running the operator's actual daily-use Hermes deployment. -- These commands can take a long time and consume real resources on a - machine you don't control from here. +- Bare `nix build` / `nix flake check` can take a long time and consume real + resources on a machine you don't control from here. **Your job:** Write good Nix, good Nu, and good docs. The operator runs -`hermenix setup`, `hermenix migrate`, the image build, and any -`nixos-rebuild` themselves, on the host, watching each step. +`hermenix setup`, `hermenix migrate`, image builds outside the test wrappers, +and any `nixos-rebuild` themselves, on the host, watching each step. -### ❌ NEVER run `hermes-vm rollback` or `hermes-ctl reseed` without fresh confirmation +### ❌ NEVER run `hermenix vm rollback` or `hermenix ctl reseed` without fresh confirmation Both are destructive to live guest state — `rollback` replaces the current state disk with an old snapshot; `reseed` overwrites `SOUL.md`, memories, and @@ -73,12 +90,12 @@ and told you the real labels. That directory is a **template seed** for fresh deployments — it gets copied into the guest on first boot and is world-readable in git history forever after. Real credentials belong in the guest's own `~/.hermes/.env` -(`hermes-ctl secrets edit`), never in this repo, never in the Nix store. +(`hermenix ctl secrets edit`), never in this repo, never in the Nix store. ### ❌ NEVER move or delete source data during a migration -When carrying over an existing Hermes deployment (see `nu/hermenix.nu`'s -`migrate` subcommand), only ever read from and copy the source. Leave it in +When carrying over an existing Hermes deployment (see `hermenix migrate` / +`nu/host/migrate.nu`), only ever read from and copy the source. Leave it in place and untouched until the operator has verified the new deployment end-to-end. This is the rollback safety net — don't remove it. @@ -96,13 +113,16 @@ These are always safe to perform without asking: - Reading any file in this repo - Editing `modules/host.nix` to add/adjust options (gated behind `mkIf cfg.enable`, following the existing style) -- Editing `nu/hermenix.nu` — check any change by parsing it (`nu -nu/hermenix.nu --help` and `nu nu/hermenix.nu --help` parse - the whole file without running any subcommand body) before handing it back +- Editing anything under `nu/` — check changes by parsing + (`nu nu/hermenix.nu --help` and `nu nu/hermenix.nu --help` + parse the whole tree without running any subcommand body) before handing + it back - Editing `guest/hermes/home` / `guest/hermes/workspace` template seed content (as long as it stays free of real secrets — see above) -- Editing `README.md` / `GUIDELINES.md` / `AGENTS.md` +- Editing `README.md` / `GUIDELINES.md` / `AGENTS.md` / anything under `docs/` - `nix-instantiate --parse .nix` — syntax check only, no eval, no build +- Running `bin/runtest`, `bin/runtest-debug`, `bin/runvmtest`, + `bin/runvmtest-debug` (the test-proxy path above) --- @@ -117,15 +137,33 @@ don't run it.** Describe what you would do and ask. ``` hermenix/ -├── flake.nix # bootstrap flake: builds the guest image + the hermenix CLI package, exposes nixosModules.default -├── modules/host.nix # services.hermenix — the host-side NixOS module -├── nu/hermenix.nu # the hermenix CLI: `setup` (guided host-side setup) and `migrate` (carry over existing data) +├── flake.nix # guest image + hermenix CLI package + nixosModules.default + checks +├── modules/host.nix # services.hermenix — host-side NixOS module +├── nu/ +│ ├── hermenix.nu # top-level CLI dispatch +│ ├── lib/ # shared helpers (common.nu, guest.nu) +│ ├── host/ # setup, migrate, remote +│ ├── vm/ # hermenix vm * +│ ├── ctl/ # hermenix ctl * +│ ├── tests/ # CLI unit tests (nutest) +│ ├── completions.nu # generated tab completions +│ └── gen-completions.nu # regenerates completions.nu from usage blocks +├── bin/ # runtest / runvmtest wrappers (agent-safe test proxy) ├── guest/ -│ ├── flake.nix # the GUEST's own /etc/nixos flake — belongs to the agent, not the operator -│ ├── common.nix # guest NixOS config baked into the bootstrap image -│ ├── hardware.nix # guest disk layout — verify labels before ever editing +│ ├── flake.nix # the GUEST's own /etc/nixos flake — belongs to the agent +│ ├── common.nix # guest NixOS config baked into the bootstrap image +│ ├── hardware.nix # guest disk layout — verify labels before ever editing +│ ├── local.nix.example # shape of /etc/hermenix/local.nix on the host +│ ├── vm-tests/ # NixOS VM checks (bridged / unbridged / migrate) │ └── hermes/ -│ ├── home/ # template seed -> ~/.hermes on first boot (SOUL.md, config.yaml, memories/, skills/) -│ └── workspace/ # template seed -> ~/workspace on first boot -└── README.md # full usage docs +│ ├── home/ # template seed → ~/.hermes on first boot +│ └── workspace/ # template seed → ~/workspace on first boot +├── tests/ # static module evaluation checks (no VM boot) +├── docs/ # operator documentation (see README index) +├── AGENTS.md # pointer at this file for coding agents +├── GUIDELINES.md # this file +└── README.md # quickstart + doc index ``` + +Operator-facing docs live under [`docs/`](docs/). Start at the +[README](README.md). diff --git a/README.md b/README.md index d50b31b..de6329c 100644 --- a/README.md +++ b/README.md @@ -2,1096 +2,85 @@ Logo -Hermes Agent in a NixOS VM that owns its own bootloader and its own -`/nix/store`, on two btrfs-backed disks, under an unprivileged hardened QEMU. +[Hermes Agent](https://github.com/NousResearch/hermes-agent) in a NixOS VM that +owns its own bootloader and `/nix/store`, on two btrfs-backed disks, under an +unprivileged hardened QEMU. **THE ULTIMATE WAY TO RUN HERMES**! The agent can rebuild its own system and update itself, durably. It shares no -filesystem with the host, and reaches **nothing** on your network unless **you say -so**. +filesystem with the host, and reaches **nothing** on your network unless **you +say so**. -One CLI, `hermenix`, with five subcommands: +Hermes product behavior (CLI, providers, gateway, skills, memory) lives in +**upstream Hermes docs** — this repo documents the VM, host module, and +`hermenix` CLI around it. -| | For | -| -------------------------------- | --------------------------------------------------------------- | -| `hermenix setup` | one-off host provisioning — disks, cache key, image build | -| `hermenix migrate ` | carry an existing flat `HERMES_HOME` into the guest | -| `hermenix remote ""` | run `setup`/`migrate` on a remote host over SSH | -| `hermenix vm ` | control the VM itself — start/stop/console/snapshots/backups | -| `hermenix ctl ` | drive the agent inside the VM — identity/config/secrets/gateway | - ---- - -## Installation - -### Guided - -The `hermenix` CLI (`nu/hermenix.nu`) is a flake package — no checkout needed -anywhere. Run it straight from its Gitea URL: +## Quickstart ```bash -nix run git+ssh://gitea@minipc1.lab.internal:22/ari/hermenix.git#hermenix -- setup +nix run git+ssh://gitea@gitea.skyg.ca/ari/hermenix.git#hermenix -- setup ``` -(Or `nix profile install git+ssh://gitea@minipc1.lab.internal:22/ari/hermenix.git#hermenix` -once, then just `hermenix setup` from here on — or get it declaratively via -the nixos-setup overlay, same as any other package. Any of these work from -any directory, on any host with SSH access to Gitea.) - -Walks the host-side steps: checks prerequisites, creates the disks, generates -the cache signing key, writes `/etc/hermenix/local.nix` from your SSH keys, -builds the bootstrap image and writes it to the disk. It prints the module -snippet to paste into your own configuration at the end, with your uplink -interface already filled in. - -Safe to re-run — every step detects what's already done and offers to skip, -and nothing destructive happens without an explicit confirmation. - -It deliberately stops short of `nixos-rebuild`, since that's your config. - -The rest of this section is what the command does, for when you'd rather do -it by hand or something goes sideways. - -### 1. Create the disks - -Two of them, and the split is the whole design: - -| | Holds | Size | Snapshotted | Backed up | -| ---------- | ------------------------------------------- | ---- | ----------- | --------- | -| **system** | NixOS, `/nix/store` | 80G | No | No | -| **state** | identity, memories, workspace, `/etc/nixos` | 20G | Yes | Yes | - -The system disk is large, takes all the write churn, and is entirely -rebuildable from this flake — so there's no point protecting it. The state -disk holds the only things that can't be reproduced, which makes snapshots -cheap and backups quick. +Paste the printed module snippet into your host flake, then: ```bash -sudo btrfs subvolume create /var/lib/hermes-vm/system -sudo btrfs subvolume create /var/lib/hermes-vm/state -sudo mkdir -p /var/lib/hermes-vm/snapshots - -# BEFORE creating any file inside them -sudo chattr +C /var/lib/hermes-vm/system /var/lib/hermes-vm/state -sudo btrfs property set /var/lib/hermes-vm/system compression none -sudo btrfs property set /var/lib/hermes-vm/state compression none - -sudo truncate -s 80G /var/lib/hermes-vm/system/system.raw -sudo truncate -s 20G /var/lib/hermes-vm/state/state.raw -sudo mkfs.ext4 -L hermes-state /var/lib/hermes-vm/state/state.raw +sudo nixos-rebuild switch +hermenix vm console +hermenix vm ssh # labels, determinate-nixd login, flake lock — see install doc ``` -**The `chattr +C` ordering matters.** Btrfs is copy-on-write and so is a VM -writing to a disk image; stacked, every guest write forks an extent and the -image fragments badly. `+C` turns that off — but only for files created -afterwards. Set it on empty subvolumes and confirm with `lsattr` before -writing anything. If you get it backwards, recreate the subvolume and copy -the image in fresh. - -Separate subvolumes because a btrfs snapshot covers a whole subvolume. Keeping -the system image out of the state subvolume is what makes snapshots small. - -### 2. Generate the cache signing key - -The host serves its store to the guest as a read-only binary cache, so the -guest doesn't re-download from the internet what you already have. - -```bash -sudo mkdir -p /var/lib/secrets -sudo nix-store --generate-binary-cache-key hermes-cache-1 \ - /var/lib/secrets/hermes-cache.secret /var/lib/secrets/hermes-cache.pub -sudo chown nix-serve /var/lib/secrets/hermes-cache.secret -sudo chmod 0400 /var/lib/secrets/hermes-cache.secret -cat /var/lib/secrets/hermes-cache.pub -``` - -### 3. Write `/etc/hermenix/local.nix` - -Host-specific settings live here, not in this repo — see -`guest/local.nix.example` for the reference shape: - -```nix -{ - cachePublicKey = "hermes-cache-1:AbCd..."; # the key you just printed; leave - # empty and the guest just skips - # the host cache - authorizedKeys = [ "ssh-ed25519 AAAA... you@laptop" ]; -} -``` - -`sudo mkdir -p /etc/hermenix` first if it doesn't exist. - -### 4. Build the image and write it to the system disk - -The two extra substituter options stop you compiling Determinate Nix from -source. `--impure` is needed because `guest/common.nix` reads -`/etc/hermenix/local.nix` from outside this flake's own source tree — -deliberately, so the flake itself stays generic and shareable while your -settings stay on your host. - -```bash -nix build --impure git+ssh://gitea@minipc1.lab.internal:22/ari/hermenix.git#hermes-guest-image \ - --option extra-substituters https://install.determinate.systems \ - --option extra-trusted-public-keys cache.flakehub.com-3:hJuILl5sVK4iKm86JzgdXW12Y2Hwd5G07qKtHTOcDCM= - -# notrunc keeps the disk at its full size; the image is smaller -sudo dd if=result/nixos.img of=/var/lib/hermes-vm/system/system.raw \ - bs=4M conv=notrunc,fsync status=progress -``` - -(Building from a local checkout instead — e.g. while developing this flake — -works the same way with `.` in place of the git URL.) - -### 5. Add the module to your system flake - -```nix -{ - inputs.hermenix.url = "git+ssh://gitea@minipc1.lab.internal:22/ari/hermenix.git"; - - outputs = { nixpkgs, hermenix, ... }: { - nixosConfigurations.myserver = nixpkgs.lib.nixosSystem { - system = "x86_64-linux"; - modules = [ hermenix.nixosModules.default ./configuration.nix ]; - }; - }; -} -``` - -```nix -services.hermenix = { - enable = true; - disk.baseDir = "/var/lib/hermes-vm"; - network.externalInterface = "eno1"; # your uplink - # network.bridged = true; # guest takes its own LAN IP via DHCP; - # # every port it opens is then LAN-reachable - vcpu = 4; - mem = 8192; -}; -``` - -`sudo nixos-rebuild switch`, then `hermenix vm console` to watch it boot. - -### 6. First boot, inside the guest - -```bash -hermenix vm ssh - -# Confirm the disk labels match guest/hardware.nix BEFORE any rebuild -lsblk -f - -# FlakeHub Cache login. Device-code flow: it prints a URL you approve in a -# browser anywhere, so the headless guest never needs one. -determinate-nixd login - -# Pin the seeded flake -sudo nix flake lock /etc/nixos -``` - -The Hermes CLI installs itself automatically on first boot (`systemd.services.hermes-install` -in `common.nix`) — it runs unattended, so there's nothing to run for that -part. Give it a few minutes (it's compiling native Node modules); watch it -with `journalctl --user -u hermes-install -f` if you want to see it happen. -Once it's done: - -```bash -hermes setup -hermes gateway install -``` - -These two need real input (model provider, API keys, messaging platform -tokens), so they stay manual. - -The identity seed is already in place — `SOUL.md`, a starter `config.yaml`, a -`USER.md` and a skill covering safe self-rebuilds. Back on the host, check -the gateway came up and that the unit name matches the default: - -```bash -hermenix ctl units -hermenix ctl status -hermenix ctl soul -``` - -> **Check the labels in step 6 before the first rebuild.** -> `guest/hardware.nix` assumes the root partition is labelled `nixos` and the -> ESP `ESP`. If the image generator used different ones, fix that file first. -> Otherwise the agent's first `nixos-rebuild switch` produces a system that -> won't boot and you're restoring a snapshot on day one. - -From here `hermes update` self-updates in place and -`sudo nixos-rebuild switch --flake /etc/nixos#hermes` is a durable change to -the machine. Neither needs anything from you. - ---- - -## Running it on a remote host - -`setup` and `migrate` both act on whatever machine they run on — local -disks, local sudo, vsock into the local guest. There's no way around SSHing -into the target host first. `hermenix remote` does that SSH step for you: - -```bash -hermenix remote fwdesk.lab.internal setup -hermenix remote fwdesk.lab.internal "setup --clean" -hermenix remote fwdesk.lab.internal "migrate /var/lib/hermes" -``` - -Quote the subcommand and its own flags/args as **one string** whenever -there's more than just the bare subcommand name — nu's own argument parser -treats any bare `--flag`-shaped word as an attempt to match a flag on -`remote` itself, so `hermenix remote host setup --clean` (unquoted) fails -before it ever reaches the SSH step. - -It runs `nix run #hermenix -- ` directly on -the remote host — no checkout involved, on either end. The remote host does -need its own SSH access to Gitea (the same access a `git clone` would need). -Override the flake URL with `--flake` if you're pointing at a fork or a -branch. - -It connects with `ssh -t`, which allocates a real pty — required for the -interactive prompts in `setup`/`migrate` to work at all over the connection. - ---- - -## SSH into the guest - -```bash -hermenix vm ssh # interactive shell -hermenix vm ssh "systemctl status" # run a command and exit -hermenix vm console # follow the VM's boot/journal output -``` - -`ssh` takes its remote command as a single string, not separate words — quote -anything past a single bare word (nu's own parser otherwise tries to match -`--flag`-shaped tokens against `vm ssh`'s own flags, and errors on extra -positional words either way). - -This goes over **vsock**, not the network. There is no listening SSH port -exposed anywhere, and the guest needs no inbound network path for you to -reach it. `console` is just `journalctl -u hermes-vm -f` on the host — a -read-only follow of the VM unit's journal, useful for watching it boot or -diagnosing why SSH won't come up, but it can't take input. There's currently -no interactive fallback for when the guest itself is unresponsive. - -Under the hood `ssh` is `ssh -o ProxyCommand='socat - VSOCK-CONNECT::22'` -(cid defaults to 77, `services.hermenix.vsockCid`), if you'd rather drive it -by hand. - -The VM itself is a systemd unit: `hermenix vm start`/`stop`/`restart` control -it directly, same as `systemctl {start,stop,restart} hermes-vm` on the host. - ---- - -## Bootstrapping Hermes's identity - -The seed lives in `guest/hermes/`: - -``` -guest/hermes/ - home/ - SOUL.md -> ~/.hermes/SOUL.md - config.yaml -> ~/.hermes/config.yaml - memories/USER.md -> ~/.hermes/memories/USER.md - skills/nixos-self-config/SKILL.md -> ~/.hermes/skills/... - workspace/ - AGENTS.md -> ~/workspace/AGENTS.md -``` - -Edit these before building the image and the agent wakes up already knowing -who it is, how you like to be spoken to, and how to rebuild its own system -without bricking itself. - -**Seeding happens on first boot only, and this is deliberate.** Hermes -rewrites `SOUL.md` and its memories as it learns — that's the product. If the -seed were re-applied on every rebuild it would silently delete everything the -agent had worked out about you. So the copy is guarded by a marker file and -never repeats. - -Two consequences: - -- Editing `guest/hermes/` after first boot changes nothing by itself. Use - `hermenix ctl reseed` when you actually want to overwrite. -- The seed is the _starting_ identity, not the current one. To see what - Hermes has made of itself, `hermenix ctl soul`. - -### Machine facts vs identity - -`/etc/hermes/ENVIRONMENT.md` is generated from `guest/common.nix` on every -rebuild and describes the machine: what's reachable, where the store is, what -persists. It's always accurate and there's no point editing it. - -`SOUL.md` is identity and belongs to the agent. The seeded version points at -`ENVIRONMENT.md` rather than repeating it, so the two can't drift. - -### Folding what it learned back into the seed - -```bash -hermenix ctl pull ./hermes-home # excludes .env, sessions, state.db -diff -ru guest/hermes/home ./hermes-home -``` - -Worth doing occasionally. If the agent has written something genuinely good -into its own SOUL.md, you want that in the seed before you ever rebuild from -scratch. - ---- - -## Managing Hermes from the host - -`hermenix vm` drives the virtual machine. `hermenix ctl` drives the agent -inside it. - -```bash -hermenix ctl soul # current identity -hermenix ctl soul edit # pull, $EDITOR, push, offer to restart -hermenix ctl memory edit # memories/USER.md -hermenix ctl conf edit # config.yaml - -hermenix ctl secrets # keys, values masked -hermenix ctl secrets edit # edit hermesHome's .env, written 0600 -hermenix ctl secrets show # values, when you really mean it - -hermenix ctl status # gateway service -hermenix ctl logs # follow it -hermenix ctl restart -hermenix ctl units # find the unit name if the default is wrong - -hermenix ctl reseed # overwrite identity from the seed -hermenix ctl pull [dir] # copy live identity to the host -``` - -Anything unrecognised is passed straight through to the `hermes` CLI in the -guest, with a TTY attached — quote it as one string if it's more than a -single word (same rule as `vm ssh` above): - -```bash -hermenix ctl chat -hermenix ctl update -hermenix ctl "sessions list" -hermenix ctl doctor -``` - -The `edit` variants pull the file out over vsock, open your `$EDITOR` -locally, write it back, and offer to restart the gateway. Secrets go in at -mode 0600 and never pass through the Nix store — which is the whole reason -there's no declarative option for them. - -If `status` and `logs` come back empty, the gateway unit is probably named -something other than `hermes-agent`. Run `hermenix ctl units` and set -`services.hermenix.gatewayUnit` to match; upstream has used more than one -name for it. - -## Seeing what the guest is running - -```bash -hermenix vm config -``` - -Prints the current system store path, the last ten generations, the git log -of `/etc/nixos`, and anything uncommitted. That last one matters: the agent -editing files without committing is the usual reason the config on disk and -the running system disagree. - -To read or diff the config locally: - -```bash -hermenix vm config-pull ./guest-live -diff -ru guest/ guest-live/ -``` - -Expect that diff to grow. The `guest/` directory in this repo is the seed -that was written on first boot; it is deliberately not kept in sync. Once the -agent edits its config, this repo and the guest diverge permanently, and a -host rebuild will not fix a guest config problem. - ---- - -## Letting the guest reach a service on your network - -By default the guest can reach the public internet and nothing private. To -open a single endpoint — say Ollama on another machine: - -```nix -services.hermenix.lanAccess = [ - { address = "192.168.1.50"; port = 11434; } # ollama - { address = "192.168.1.50"; port = 9000; proto = "udp"; } -]; -``` - -Each entry punches one destination and port through the RFC1918 drop. The -guest reaches it as `192.168.1.50:11434` directly. Traffic is masqueraded, so -the other machine sees it arriving from your host's address, not the guest's. - -There's also `allowLan = true`, which opens all private address space at -once. It exists, but reach for `lanAccess` instead — the whole point of this -design is that an agent which goes wrong can't go wandering. - -### A service on the host itself - -Different chain, different option. ComfyUI on the host, port 8188: - -```nix -services.hermenix.hostAccess = [ { port = 8188; } ]; -``` - -**The service must be bound somewhere the guest can see.** If it listens -on `127.0.0.1` the rule won't help, because the guest arrives on the bridge -address. Bind it to `10.77.0.1` (bridge only, invisible to your LAN) or -`0.0.0.0`: - -```nix -services.comfyui.listenAddress = "10.77.0.1"; # or whatever your service calls it -``` - -From the guest that's then `http://10.77.0.1:8188`. - ---- - -## Exposing a service the guest is running - -Say the agent has built a web app on guest port 8080. Two halves. - -**Guest side.** The agent opens its own firewall, in `/etc/nixos`: - -```nix -networking.firewall.allowedTCPPorts = [ 22 8080 ]; -``` - -**Host side.** Nothing at all, if you only want to reach it from the host — -host-to-guest is outbound traffic and already permitted: - -```bash -curl http://10.77.0.2:8080 -``` - -To publish it to the wider network: - -```nix -services.hermenix.publish = [ - { hostPort = 8080; guestPort = 8080; } -]; -``` - -That DNATs the port on your uplink to the guest and opens the matching -forward rule. `hostPort` and `guestPort` don't have to match, so you can park -an awkward guest port somewhere sensible. - -A note on this one: publishing a port is the one change here that lets the -outside world start conversations with the agent's machine. Everything else -in this design is the guest reaching out. Put a reverse proxy with TLS and -auth in front of anything that isn't a toy, and keep the list short. - ---- - -## Putting the guest directly on your LAN - -The default setup keeps the guest on an isolated bridge behind the host's -NAT, reachable from outside only through `publish`. This section is for -when you want the VM to be a first-class device on your LAN instead: its -own fixed address on the network, reachable on any port from any device, -while keeping everything it can see outbound locked down. - -Enable it and tell it your LAN's numbers: - -```nix -services.hermenix.network = { - bridged = true; - externalInterface = "enp191s0"; # your uplink (required) - - # Static LAN addressing: bridged mode does not use your LAN's DHCP. - lanAddress = "10.0.12.50"; # the guest's fixed LAN address - hostLanAddress = "10.0.12.10"; # this host's own LAN address - lanGateway = "10.0.12.1"; # your router - lanPrefixLength = 24; # default 24 - # lanDns = [ "1.1.1.1" ]; # optional; defaults to lanGateway -}; -``` - -What that gives you: the bridge absorbs the uplink, the host and the guest -share the LAN, and each side has a fixed address. The guest never needs to -know its own address - the module runs a small DHCP server on the bridge -that reserves `lanAddress` for the guest's MAC, so the VM stays a plain -DHCP client and just receives its address. This host keeps its LAN identity -too: the bridge clones your uplink's MAC, so the LAN sees this host under -its usual MAC (no duplicate-address surprises), and the guest's DHCP -broadcast never leaves the bridge, so nothing on your LAN can race the -reservation. - -### What is and isn't reachable - -- **Inbound to the guest, from anywhere on the LAN: every port** is open. - The guest's own firewall decides what answers - by default it only opens - port 22, so the agent opens whatever it wants LAN-reachable. -- **Guest egress is still gated**, exactly as in NAT mode: `lanAccess` - entries and `allowLan` are unchanged. The VM sees the LAN, but only the - parts you have allowed. -- **Guest to host still works**: `hostAccess` and the host Nix cache are - reachable over the service net at `10.77.0.1` in both modes. -- **`publish` is ignored** in this mode - every inbound port is already - reachable. Drop the entries if you come back to NAT mode. - -### Switching over - -1. Pick three free addresses: the guest (`lanAddress`), this host - (`hostLanAddress` - use the address it already has on that interface, - e.g. a router reservation, so nothing else on the LAN breaks), and the - router itself (`lanGateway`). Keep `hostLanAddress` and `lanAddress` - outside any DHCP pool, or reserve them in the router first. -2. Set `network.bridged = true` with the options above, then - `sudo nixos-rebuild switch`. The network drops briefly while the host - reconfigures - plan for it. -3. Verify from the host: `hermenix vm ssh`; inside the guest, - `ip -4 addr` shows the service net (`10.77.0.2`) **and** its fixed LAN - address. From any other device on the LAN: - - ```bash - ssh hermes@ # and any other port the guest opens - ``` - -### Switching back - -Set `bridged = false` (the `lan*` options are ignored) and rebuild the -host. The guest's DHCP client finds no server on the isolated bridge and -settles onto its static route via the host again; no guest-side change -needed. - -### Gotchas - -- `hostLanAddress` is a static claim on the LAN - pick one your router - won't hand out, or reserve it. -- Any network config you had for the uplink interface is replaced by its - bridge-port role while bridged (this module's networkd unit for it takes - priority over most others). -- An older guest image that predates the DHCP-on-NIC setup in the current - `guest/common.nix` won't take an address in bridged mode; upgrade the - image or apply that config inside the VM first. - ---- - -## Passing hardware to the guest - -### Before reaching for this - -If what you want is "the agent can use the GPU", passthrough is often the -wrong answer. It is exclusive — the host loses the card completely while the -VM runs — and it is the single largest reduction in isolation available in -this setup. - -The alternative: keep Ollama or ComfyUI on the host where the GPU already -works, and open the port with `hostAccess`. The agent gets the capability -over HTTP and the device stays on your side of the boundary. That is usually -both easier and better. - -Pass the hardware through when the guest genuinely needs the device itself — -CUDA in-process, a capture card, a radio, a display output. - -### What passthrough costs you - -A PCI device does DMA. The IOMMU is the only thing standing between a -passed-through device and host memory, which is why `passthrough.iommu` is a -hard requirement and the module refuses to build without it. - -Two further constraints follow from how IOMMUs work: - -- **Whole groups only.** Every device in an IOMMU group goes together. Check - yours before planning anything: - - ```bash - for d in /sys/kernel/iommu_groups/*/devices/*; do - echo "group $(basename "$(dirname "$(dirname "$d")")"): $(lspci -nns "$(basename "$d")")" - done | sort -V - ``` - - If your GPU shares a group with a controller you need, you cannot pass it - cleanly. ACS override patches exist; they work by lying about isolation, - and they have no place in a setup built to contain an autonomous agent. - -- **Every function.** A GPU is at least two entries — graphics and its HDMI - audio function. - -### GPU - -```bash -lspci -Dnn | grep -i vga # addresses and vendor:device IDs -``` - -```nix -services.hermenix.passthrough = { - iommu = "amd"; # or "intel" - pci = [ - { address = "0000:01:00.0"; id = "10de:2204"; } # graphics - { address = "0000:01:00.1"; id = "10de:1aef"; } # HDMI audio - ]; -}; -``` - -Setting `id` makes vfio-pci claim the card in the initrd, before `nvidia` or -`amdgpu` can. Do this for GPUs — unbinding a live graphics driver afterwards -is unreliable, and the failure mode is a host with no display. Devices given -only an `address` are bound at boot by a oneshot service instead, which is -fine for things you swap around. - -Rebuild and **reboot** — early binding happens in the initrd. Then check: - -```bash -lspci -nnk -d 10de:2204 | grep "Kernel driver" # want: vfio-pci -``` - -On the guest side, the agent installs the drivers itself. It owns -`/etc/nixos`, so `hardware.nvidia` and `nvidia-container-toolkit` are its -problem, not yours — which is rather the point of the arrangement. - -### HDMI output - -A display hanging off a passed-through GPU works once the guest drives the -card; nothing extra is needed here. A capture card or HDMI grabber is just -another PCI or USB device — find it with `lspci -Dnn` or `lsusb` and use the -matching section. - -### USB devices and Bluetooth - -Matched by vendor and product ID: - -```bash -lsusb # e.g. "Bus 001 Device 004: ID 8087:0026 Intel ..." -``` - -```nix -services.hermenix.passthrough.usb = [ - { vendorId = "8087"; productId = "0026"; } # bluetooth radio - { vendorId = "0403"; productId = "6001"; } # serial adapter -]; -``` - -This adds an xHCI controller to the guest and forwards the matching devices. -No IOMMU requirement, no VFIO, and much milder than PCI passthrough — QEMU -mediates rather than handing over raw hardware. - -Two things that catch people: - -- **The host must not be using it.** For an integrated Bluetooth radio, keep - `btusb` off it: `boot.blacklistedKernelModules = [ "btusb" ];`, and don't - enable `hardware.bluetooth` on the host. -- **No hotplug.** QEMU resolves these at start, so a device that's unplugged - when the VM boots won't appear when you plug it in. `hermenix vm restart` - after plugging. Passing a whole USB _controller_ through as a PCI device - gets you real hotplug for anything in its ports, at the cost of that - controller being exclusive to the guest. - -### When it doesn't start - -`hermenix vm console` first. Then, in rough order of likelihood: the device -didn't bind to vfio-pci (`lspci -nnk`), the IOMMU isn't on (`dmesg | grep -i -iommu`), or something in the group is still held by the host. The module -lifts `LimitMEMLOCK` for you when PCI passthrough is configured, since VFIO -pins the whole guest RAM allocation and the default limit otherwise produces -a memorably unhelpful error. - -## Virtual devices - -The VM boots a fixed set of virtual hardware, and `services.hermenix.devices` -controls which optional pieces are attached. Everything here is a QEMU device: -none of it touches host hardware, and none of it needs anything the guest -doesn't already ship (the mainline `virtio_*` drivers). - -| Option | Default | Attaches | -| ------------------------- | ---------- | -------------------------------------------------------- | -| `devices.rng.enable` | `true` | virtio entropy source | -| `devices.vsock.enable` | `true` | the host<->guest vsock (`hermenix ctl`, backups) | -| `devices.video.enable` | `false` | a virtual display adapter | -| `devices.video.model` | `"virtio"` | `virtio` (has a render node) or a legacy VGA model | -| `devices.balloon.enable` | `false` | virtio memory balloon | -| `devices.watchdog.enable` | `false` | hardware watchdog (`i6300esb`/`ib700`, action `reset`/…) | -| `devices.input.enable` | `false` | virtio keyboard/tablet | -| `devices.audio.enable` | `false` | audio device (`intel-hda`/`virtio-sound`) | -| `devices.tpm.enable` | `false` | TPM 2.0 via swtpm | -| `devices.cdroms` | `[]` | read-only CD images | -| `devices.extraDisks` | `[]` | extra raw disks | -| `devices.usbController` | `"auto"` | `qemu-xhci` (`auto` = on iff `passthrough.usb` is set) | - -`extraQemuArgs` appends raw arguments to the QEMU command line for anything -not covered above. It is unvalidated - prefer a typed option where one exists. - -### A render node (Wayland, GPU compute, anything wanting /dev/dri) - -```nix -services.hermenix.devices.video.enable = true; # model defaults to "virtio" -``` - -Gives the guest `/dev/dri/card0` and `/dev/dri/renderD128`. The console stays -serial and nothing is shown on the host. Only the `virtio` model exposes the -render node; the legacy models give `card0` alone. - -### TPM - -`devices.tpm.enable = true` starts an `swtpm` unit first and points QEMU at -it. State lives in the unit's `StateDirectory` (`/var/lib/hermes-vm-tpm` by -default), so the guest gets a TPM 2.0 it can seal secrets to. - -### Audio, with a caveat - -`devices.audio.enable` is wired up, but `hermes-vm.service` is deliberately -hardened (`ProtectHome`, `PrivateTmp`, a dedicated system user), so it cannot -reach your PipeWire socket without loosening that hardening. Leave it off -unless you mean to relax the unit for it. - -## Recovery - -Four layers, cheapest first. The first two live inside the guest and cover -almost every real mistake. - -```bash -# The agent's own config history -hermenix ctl soul # what it thinks it is now -hermenix vm ssh "git -C /etc/nixos log --oneline" - -# Previous NixOS generations, also in the guest's boot menu -hermenix vm ssh "sudo nixos-rebuild switch --rollback" - -# Snapshot of the state disk -hermenix vm snapshot # trigger one now, e.g. before a risky change -hermenix vm snapshots -hermenix vm rollback state-20260914-1400 # asks you to type 'rollback' to confirm - -# Restore from backup -hermenix vm backup # trigger one now -hermenix vm backups -``` - -### Snapshots - -Hourly by default, keeping 48. Only the state disk — the system disk isn't -snapshotted because `hermenix setup` rebuilds it from the flake in minutes. - -The host fsfreezes the guest through qemu-guest-agent around each one, so the -snapshot captures a consistent filesystem rather than the equivalent of -pulling the power. Takes milliseconds; the VM keeps running. - -`hermenix vm rollback` stops the VM, moves the current state subvolume aside as -`state.old-`, and restores the snapshot in its place. Nothing is -deleted, so a mistaken rollback is itself reversible. Clean the `state.old-*` -directories up yourself once you're happy. - -**Keep retention tighter than you would on ZFS.** The state image is -nodatacow, and every live snapshot reintroduces copy-on-write for the blocks -it still references. Performance sags while snapshots exist and recovers as -they're pruned. - -### Backups - -```nix -services.hermenix.backup = { - enable = true; - repository = "sftp:backup@nas:/hermes"; - passwordFile = "/var/lib/secrets/hermes-restic"; -}; -``` - -This backs up the guest's _state_, not its disk. Files are pulled over the -SSH channel that already exists and streamed into restic. - -Two reasons it works that way. The backup stays portable — it restores onto -any filesystem, or a different machine entirely, and doesn't care that you're -on btrfs today. And it avoids loop-mounting a guest-written filesystem on the -host, which would mean the host kernel parsing data the agent controls. That -would undo a fair bit of what the rest of this design is for. - -If the repository is on your LAN rather than the internet, note this runs on -the _host_, so it needs no `lanAccess` entry — the guest never touches it. - -### When the guest won't boot at all - -The system disk is disposable by design. Re-run `hermenix setup`, write a -fresh image to the system disk, and leave the state disk alone — Hermes comes -back with its identity, memories, workspace and NixOS config intact. That is -the intended repair, not a last resort. - -### Starting completely fresh - -`hermenix setup --clean` stops the VM, deletes the existing -`system.raw`/`state.raw`/UEFI vars, and drops the stale `known_hosts` entry -for the guest (a fresh state disk means a fresh SSH host key on next boot, -so the old entry would otherwise fail with a host-key-mismatch on your next -connect) — then walks through setup again for a genuinely clean first boot, -rather than the disk-exists, skip-ahead path. It leaves the btrfs -subvolumes and any snapshots alone (only the files inside them), and asks -you to type `clean` to confirm before touching anything. - ---- - -## Nix flavour - -The guest runs **Determinate Nix**, nixpkgs pinned to FlakeHub -(`f/NixOS/nixpkgs/0.1`, rolling; use `0` for current stable). - -- `/etc/nix/nix.conf` belongs to Determinate and must not be edited. Custom - settings go in `/etc/nix/nix.custom.conf`, written from `guest/common.nix`. - There is deliberately no `nix.settings` block. -- No `nix.gc` either — `determinate-nixd` runs the collector. -- Lazy trees are on by default, which makes the agent's flake evaluations - much cheaper. That matters more than it sounds when something re-evaluates - nixpkgs on a schedule. -- Neither flake uses `inputs.nixpkgs.follows` for `determinate`. Determinate - warn it causes FlakeHub Cache misses. - -Substituter order in the guest: FlakeHub Cache, your host cache, -`cache.nixos.org`. If FlakeHub Cache goes quiet, the login has expired — run -`determinate-nixd login` again. - -The host stays on stock Nix. Nothing here requires changing that. - ---- - -## What the guest can and cannot reach - -| | Default | -| ------------------------- | --------------------------------------------------- | -| Public internet | Yes — it needs this for model APIs and messaging | -| Host Nix cache | Yes, one port, `hostCache.enable = false` closes it | -| Anything else on the host | No, unless in `hostAccess` | -| Your LAN and other VMs | No, unless in `lanAccess` | -| Inbound from the network | No, unless in `publish` | -| Host hardware | None, unless in `passthrough` | - -With `network.bridged = true` the "Inbound from the network" row changes: -with the guest on the LAN, every port it opens is reachable from the LAN; -`publish` stops doing anything. The outbound rows are unchanged - the -`lanAccess` / `allowLan` gates still apply. - -The guest owns its kernel and its config, so nothing inside it is a security -boundary. The nftables rules in `modules/host.nix` are. QEMU runs as an -unprivileged `hermes-vm` user with its own seccomp sandbox and systemd -hardening, with a thin device model: virtio-blk, virtio-net, vsock, RNG, -serial. No virtiofs. `vhost-net` is off deliberately — it would move packet -handling into the host kernel in exchange for throughput one agent doesn't -need. - -The host cache does let the guest read every store path on the host. Store -paths are world-readable anyway, so this isn't new exposure, but don't bake -secrets into your host's store. - ---- - -## Known rough edges - -- The Determinate NixOS module's option names for custom Nix settings aren't - published — only the nix-darwin `determinateNix.customSettings` equivalent - is. So `guest/common.nix` writes `/etc/nix/nix.custom.conf` via - `environment.etc`. If a future module version claims that file, you'll get - a conflict; switch to its option then. -- `determinate-nixd login` is manual. There's no way to bake the credential - into the image without parking a token somewhere it shouldn't be. -- `MemoryDenyWriteExecute = true` is correct under KVM but is the first thing - to flip if QEMU refuses to start. -- `pkgs.OVMF.firmware` / `pkgs.OVMF.variables` are current attribute names; - older nixpkgs put these under `pkgs.OVMF.fd` in an `FV/` subdirectory. -- PCI passthrough needs a reboot to take effect, because early vfio binding - happens in the initrd. -- Snapshots need btrfs. Everything else works on any filesystem; without it - you rely on backups alone for recovery. -- `state.old-*` directories left by `hermenix vm rollback` are not cleaned up - automatically, on purpose. Delete them yourself once you're sure. -- First boot builds a lot. Give it time. -- `services.hermenix.hermesHome` currently defaults to a path that doesn't - exist in the guest (it's built from the host's disk-image directory, not - the guest's mounted state disk) — set it explicitly on every deployment, - migrated or not. See "Migrating an existing deployment" below for the - value to use. - ---- - -## Migrating an existing (e.g. Docker-based) Hermes deployment - -If you already have a running Hermes with real data (`HERMES_HOME` pointing -at some directory full of `SOUL.md`, `memories/`, `sessions/`, `state.db`, -etc.): - -```bash -nix run git+ssh://gitea@minipc1.lab.internal:22/ari/hermenix.git#hermenix -- migrate /path/to/existing/HERMES_HOME -``` - -Run `... -- migrate --help` for the full set of options -(vsock CID, guest user, destination path, hostname rewrites). It copies — -never moves or deletes — the source data, so the original stays a safe -fallback until you've verified the new VM. To run it from a different -machine than the one hosting the VM, see `hermenix remote` above — the same -`migrate /path/...` argument goes in the quoted command string. - -The module's `services.hermenix.hermesHome` option lets `hermenix ctl` (and -your own `HERMES_HOME` export inside the guest) point at wherever that -existing flat layout lands, instead of the default seeded `.hermes` under the -state disk. - -**Set `hermesHome` explicitly, even if you're not migrating.** The option's -shipped default doesn't currently resolve to a real path inside the guest — -it's built from the host's own disk-image directory rather than the guest's -mounted state disk, so `hermenix ctl soul`/`conf`/`secrets`/etc. will fail -against it unhelpfully until you override it. For the default seeded layout: - -```nix -services.hermenix.hermesHome = "/var/lib/hermes/.hermes"; -``` - -For a migrated flat layout, use whatever `--dest` you passed to `migrate` -(default `/var/lib/hermes`). - -### Don't run it with `sudo` - -`migrate` (like `hermenix vm ssh` and `hermenix ctl`) reaches the guest with -a plain `ssh` call over vsock, authenticating with whatever key or agent the -invoking user already has — the same identity that's in the guest's -`authorizedKeys`. Prefixing the command with `sudo` runs that `ssh` as -**root** instead, which has no matching key, and you'll hit: - -``` -hermes@hermes: Permission denied (publickey,keyboard-interactive). -``` - -If `hermenix vm ssh` works for you as your normal user, `migrate` doesn't -need anything more privileged — run it the same way. - -### If some of the source can't be read - -Old flat `HERMES_HOME` directories, especially ones left behind by a -rootless-Docker deployment, sometimes contain a stray entry owned by a -container-internal uid with no mapping on the host — unreadable by anyone -outside the container. `migrate` tars the source with `--ignore-failed-read`, -so a warning like this is expected and non-fatal: - -``` -tar: ./.hermes: Cannot open: Permission denied -``` - -Everything else still gets copied and the run still completes, including the -hostname rewrite step. If you actually need what's in the unreadable entry, -grab it separately, as root, before or after the main migration — e.g.: - -```bash -sudo tar -C /path/to/HERMES_HOME/.hermes -cf - . \ - | hermenix vm ssh 'mkdir -p /var/lib/hermes/.hermes-old && tar -C /var/lib/hermes/.hermes-old -xf -' -``` - -### `hermes setup` fails with "docker not found on PATH. Cannot route to container" - -`hermenix migrate` automatically excludes `.local/bin` and `.hermes-installed` -from the migration tar, so this should not happen with current versions. -If you see it after a migration with an older build, the source's -`.local/bin/hermes` (a Docker-routing launcher from the previous environment) -was copied into the guest and is shadowing the native install. - -Fix inside the guest: - -```bash -# Remove the stale launcher -rm -f ~/.local/bin/hermes - -# Reinstall the native hermes binary -curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh \ - | bash -s -- --non-interactive --skip-setup - -# Verify it's a real binary, not a shell script -file ~/.local/bin/hermes - -# Now proceed with setup -hermes setup -``` - -If `hermes-install.service` never ran because `.hermes-installed` was also -migrated from the source, you may instead let systemd do it: - -```bash -rm -f ~/.local/bin/hermes ~/.hermes-installed -sudo systemctl start hermes-install -journalctl -fu hermes-install # watch until complete -hermes setup -``` - -## Hacking on this repo - -The devshell provides nushell, nufmt, and pre-commit hooks. Enter it once: +Full path: **[docs/INSTALL.md](docs/INSTALL.md)**. + +## CLI + +| | For | +| -------------------------------- | ---------------------------------------------- | +| `hermenix setup` | host provisioning | +| `hermenix migrate ` | existing `HERMES_HOME` → guest | +| `hermenix migrate connections` | connection files only | +| `hermenix remote ""` | `setup`/`migrate` over SSH | +| `hermenix vm ` | VM lifecycle, snapshots, backups | +| `hermenix ctl ` | host bridge into guest files / unit / `hermes` | +| `hermenix version` | package version | + +**[docs/CLI.md](docs/CLI.md)**. + +## Documentation + +| Doc | Contents | +| ------------------------------------------------------------ | -------------------------------------------------- | +| [docs/CONCEPTS.md](docs/CONCEPTS.md) | disks, boundary, seed-once, vsock, Determinate Nix | +| [docs/INSTALL.md](docs/INSTALL.md) | guided setup, module, first boot, remote | +| [docs/MANUAL_SETUP.md](docs/MANUAL_SETUP.md) | hand-run `setup` | +| [docs/CLI.md](docs/CLI.md) | `vm` / `ctl` | +| [docs/NETWORK.md](docs/NETWORK.md) | NAT vs bridged, allowlists | +| [docs/PASSTHROUGH.md](docs/PASSTHROUGH.md) | PCI/USB, virtual devices | +| [docs/RECOVERY.md](docs/RECOVERY.md) | generations, snapshots, backups, `--clean` | +| [docs/MIGRATE.md](docs/MIGRATE.md) | carry over an existing home | +| [docs/DEVELOP.md](docs/DEVELOP.md) | devenv, tests | +| [docs/ROUGH_EDGES.md](docs/ROUGH_EDGES.md) | known issues | +| [GUIDELINES.md](GUIDELINES.md) | coding-agent rules for this repo | +| [Hermes Agent](https://github.com/NousResearch/hermes-agent) | upstream product docs | + +## Boundary (short) + +| | Default | +| -------------------- | ------------------------ | +| Public internet | yes | +| Host Nix cache | yes (one port) | +| Host / LAN / inbound | deny, unless allowlisted | +| Host hardware | none, unless passthrough | + +[docs/NETWORK.md](docs/NETWORK.md) · [docs/CONCEPTS.md](docs/CONCEPTS.md) + +## Hacking ```bash direnv allow +runtest +runvmtest ``` -On first entry devenv installs the pre-commit hooks automatically. Hooks run -on every commit, including: - -- **nufmt** — formats all `*.nu` files in place -- **gen-completions** — reruns `nu/gen-completions.nu` and stages - `nu/completions.nu` whenever a usage block in the source files changes -- **nu-tests** — runs the CLI test suite (see below) whenever anything - under `nu/` changes - -### Testing - -Three test suites. None is ever run for you by an agent working in this -repo — they're for you (or CI) to run. - -**CLI tests** (`nu/tests/`), via [nutest](https://github.com/vyadh/nutest): - -```bash -runtest # everything (bin/ is on PATH inside the devShell) -runtest vm_test # just nu/tests/vm_test.nu -runtest-debug vm_test # load vm_test.nu into an interactive nu session instead -``` - -Covers the CLI's pure helper functions (path/env defaults, `shquote`, -`normalize-size`, the `gen-completions.nu` parser) and a smoke test that the -whole `hermenix`/`vm`/`ctl` subcommand tree parses and dispatches correctly -— without ever invoking a subcommand's real body (no ssh, no disk writes, -no `nix build`). - -**Guest VM tests** (`guest/vm-tests/test_unbridge.nix` / `guest/vm-tests/test_bridge.nix`): -boot `guest/common.nix` — the same module set `hermes-guest-image` is built -from — in ephemeral, throwaway NixOS test VMs. No real disk under -`/var/lib/hermes-vm/**` is ever touched: - -```bash -runvmtest # NAT/isolated-bridge mode (plain pass/fail) -runvmtest guest-vm-bridged # bridged mode with a simulated LAN -runvmtest-debug # interactive driver REPL (ptpython) for the default suite -``` - -Both tests boot the guest the same way and share every check that must hold -in both modes — first-boot seeding (`~/.hermes`, `~/workspace`, the -`/etc/nixos` symlink), that seeding is idempotent across a reboot (the whole -point of the `.seeded` marker), the `hermes` user/sudo setup, and SSH. That -shared node config and assertion script are defined once in -`guest/vm-tests/common.nix` and imported by both, so nothing can drift -between the two modes' common checks. - -Each test then adds the checks that only make sense for its mode: - -- `guest-vm-unbridged` — the isolated bridge has no DHCP server, so it - asserts the guest stands on its static config alone: exactly the one - `10.77.0.2/24` address and exactly one default route, via the host. -- `guest-vm-bridged` — a second node simulates the LAN (a dnsmasq DHCP - server, a throwaway HTTP service, and a plain device). It asserts the - guest takes a DHCP lease beside the static service-net address, keeps - both default routes (the LAN gateway at metric 100 and the host fallback - at metric 200), can reach a service on the LAN, and that a LAN device can - connect to a port the guest exposes (the guest's own firewall still - decides what answers). The host-side half of bridged mode — the bridge - itself, the mark-based rules — is covered statically by the check below. - -Both deliberately do not cover the real `hermes-install` script (no internet -route inside the test VM) or `modules/host.nix`'s own `hermes-vm` QEMU -service — those stay manual, operator-run checks. - -`runvmtest-debug` gives you `start_all()`, `machine.wait_for_unit(...)`, and -`machine.shell_interact()` (a live shell inside the guest) to step through a -failure by hand instead of just getting pass/fail. - -**Module evaluation checks** (`tests/network-modes-check.nix`): a static -check — no build, no VM boot — that evaluates `modules/host.nix` in both -networking modes (NAT and `network.bridged = true`) plus `guest/common.nix`, -and asserts on the generated nftables rulesets, the firewall input/forward -rules, the networkd units, and the guest's DHCP/route-metric config: - -```bash -nix build .#checks.x86_64-linux.network-modes -``` - -Catches regressions in the bridged-mode flag without booting anything. If -it fails, the assertion that broke is named in the error. +Agents: builds **only** via `bin/runtest` / `bin/runvmtest` (`-debug` ok). +[GUIDELINES.md](GUIDELINES.md) · [docs/DEVELOP.md](docs/DEVELOP.md) diff --git a/bin/runtest b/bin/runtest index c492d82..e9ed56e 100755 --- a/bin/runtest +++ b/bin/runtest @@ -1,21 +1,10 @@ #!/usr/bin/env nu -# -# runtest — run the hermenix CLI test suite (nu/tests/). -# -# Requires the nutest module on NU_LIB_DIRS, which `nix develop`'s devShell -# sets up (that devShell also puts bin/ on PATH, so `runtest` works bare). -# Runs everything by default; pass a suite name to run just one file -# instead: -# runtest # everything -# runtest vm_test # only nu/tests/vm_test.nu -# -# To step through a single suite interactively instead of via nutest's -# runner, use runtest-debug. +# CLI tests (nu/tests/). Needs nutest on NU_LIB_DIRS (devShell). use nutest def main [ - suite?: string # only run nu/tests/.nu, e.g. vm_test + suite?: string # e.g. vm_test → nu/tests/vm_test.nu ] { let repo_root = ($env.FILE_PWD | path dirname) let tests_path = ($repo_root | path join "nu" "tests") diff --git a/bin/runtest-debug b/bin/runtest-debug index 67d55b8..a58b6e3 100755 --- a/bin/runtest-debug +++ b/bin/runtest-debug @@ -1,16 +1,5 @@ #!/usr/bin/env nu -# -# runtest-debug — interactively debug one CLI test file. -# -# Sources the given suite (e.g. `runtest-debug vm_test`) and drops you into -# an interactive nushell session with it loaded, instead of running it -# through nutest. Call any @test-tagged command by its full name to run -# just that test on its own and inspect what happens — nutest's runner -# captures output and runs tests concurrently, which gets in the way of -# stepping through a failure by hand. -# -# Test files don't need NU_LIB_DIRS/nutest themselves (only runtest's own -# runner does), so this works without the devShell's nutest wiring too. +# Load one suite into an interactive nu session (no nutest runner). def main [suite: string] { let repo_root = ($env.FILE_PWD | path dirname) diff --git a/bin/runvmtest b/bin/runvmtest index 75e662b..28b76cd 100755 --- a/bin/runvmtest +++ b/bin/runvmtest @@ -1,23 +1,8 @@ #!/usr/bin/env nu -# -# runvmtest — build and run the guest-config VM tests. The mode-independent -# assertions (seeding, idempotency, account, SSH) are shared between the two -# tests via guest/vm-tests/common.nix; each test adds its own mode's -# networking checks. -# -# Plain pass/fail: the build fails if any testScript assertion fails. For -# stepping through it interactively instead, use runvmtest-debug. -# -# --impure is required because guest/common.nix reads -# /etc/hermenix/local.nix if present, same as the real hermes-guest-image -# build — a pre-existing property of common.nix, not something the tests -# introduce. -# -# Not run for you automatically: this needs a real build, which agents in -# this repo don't run (see GUIDELINES.md). You (or CI) run this by hand. +# Guest VM checks. --impure: guest/common.nix may read /etc/hermenix/local.nix. def main [ - suite?: string # which check to run: guest-vm-unbridged (default), guest-vm-bridged, or migrate-connections-vm + suite?: string # guest-vm-unbridged (default) | guest-vm-bridged | migrate-connections-vm ] { let check = $suite | default "guest-vm-unbridged" let attr = $".#checks.x86_64-linux.($check)" diff --git a/bin/runvmtest-debug b/bin/runvmtest-debug index b15ba5a..c954711 100755 --- a/bin/runvmtest-debug +++ b/bin/runvmtest-debug @@ -1,21 +1,8 @@ #!/usr/bin/env nu -# -# runvmtest-debug — build a guest-config VM test's interactive driver and -# drop into it, instead of just running it for pass/fail (see runvmtest). -# -# Opens a Python (ptpython) REPL with the test's node defined but not yet -# started. Useful from there: -# start_all() -# machine.wait_for_unit("multi-user.target") -# machine.shell_interact() # live shell inside the guest, over the console -# machine.succeed("...") # run any assertion from the guest-vm tests by hand -# -# Same --impure requirement as runvmtest, same reason (guest/common.nix -# reads /etc/hermenix/local.nix if present). Not run for you automatically -# — needs a real build, which agents in this repo don't run. +# Interactive driver REPL for a guest VM check. Same --impure as runvmtest. def main [ - suite?: string # which check to debug: guest-vm-unbridged (default), guest-vm-bridged, or migrate-connections-vm + suite?: string # guest-vm-unbridged (default) | guest-vm-bridged | migrate-connections-vm ] { let check = $suite | default "guest-vm-unbridged" let attr = $".#checks.x86_64-linux.($check).driverInteractive" diff --git a/devenv.nix b/devenv.nix index fa290a4..e282783 100644 --- a/devenv.nix +++ b/devenv.nix @@ -2,7 +2,6 @@ let nu = "${pkgs.nushell}/bin/nu"; - # nu-check based linter: fails on any file that does not parse. nuCheck = pkgs.writeShellScript "nu-check-all" '' status=0 for f in "$@"; do @@ -32,7 +31,6 @@ in echo "Welcome to the Hermenix devshell!" ''; - # ── Formatters / linters ──────────────────────────────────────────────── treefmt = { enable = true; config = { @@ -41,7 +39,6 @@ in deadnix.enable = true; # nix: dead code removal statix.enable = true; # nix: anti-pattern lints }; - # No upstream treefmt-nix module for nushell, so wire nufmt manually. settings.formatter.nufmt = { command = "${pkgs.nufmt}/bin/nufmt"; includes = [ "*.nu" ]; @@ -49,9 +46,7 @@ in }; }; - # ── Git hooks ─────────────────────────────────────────────────────────── git-hooks.hooks = { - # Formats nix + nushell; package is wired up by the treefmt integration. treefmt.enable = true; nu-check = { diff --git a/docs/CLI.md b/docs/CLI.md new file mode 100644 index 0000000..5fc5b16 --- /dev/null +++ b/docs/CLI.md @@ -0,0 +1,101 @@ +# CLI reference + +One binary, `hermenix`: + +| | For | +| -------------------------------- | ---------------------------------------------------------- | +| `hermenix setup` | host provisioning — disks, cache key, image | +| `hermenix migrate ` | existing flat `HERMES_HOME` → guest | +| `hermenix migrate connections` | connection files only | +| `hermenix remote ""` | `setup`/`migrate` over SSH | +| `hermenix vm ` | VM lifecycle, snapshots, backups | +| `hermenix ctl ` | host-side bridge into the guest (files, unit, passthrough) | +| `hermenix version` | packaged version string | + +Quote multi-word remote/passthrough args as one string — +[CONCEPTS.md](CONCEPTS.md#quote-multi-word-subcommands). + +Setup: [INSTALL.md](INSTALL.md). Migrate: [MIGRATE.md](MIGRATE.md). + +--- + +## `hermenix vm` + +```bash +hermenix vm ssh # interactive shell (vsock) +hermenix vm ssh "systemctl status" # run a command and exit +hermenix vm console # journalctl -u hermes-vm -f (read-only) +hermenix vm start | stop | restart + +hermenix vm config # guest generation, history, git status +hermenix vm config-pull [dir] # copy guest /etc/nixos here + +hermenix vm snapshot | snapshots +hermenix vm rollback # type 'rollback' to confirm + +hermenix vm backup | backups +``` + +`console` is follow-only (no input). No interactive fallback if the guest is +dead. + +Repo `guest/` is the **seed**; live guest `/etc/nixos` diverges after the +agent edits it. `config-pull` is for reading/diffing — a host rebuild will not +fix a guest config problem. + +--- + +## `hermenix ctl` + +Host-side helpers over vsock. They know where Hermes's files live in _this_ +VM layout and how to reach the gateway unit — they are not a second Hermes +manual. For Hermes features (chat, sessions, providers, skills, …) use the +upstream CLI or [Hermes Agent docs](https://github.com/NousResearch/hermes-agent); unknown `ctl` words are +passed through to `hermes` in the guest. + +```bash +# Files under hermesHome (default /var/lib/hermes/.hermes) +hermenix ctl soul [edit] +hermenix ctl memory [edit] # memories/USER.md if present +hermenix ctl conf [edit] # config.yaml +hermenix ctl secrets [edit|show] # .env, mode 0600; masked by default + +# Gateway user unit (name: services.hermenix.gatewayUnit) +hermenix ctl status | logs | restart +hermenix ctl units + +# Seed / fold-back (destructive where noted) +hermenix ctl reseed # overwrite from /etc/nixos/hermes seed +hermenix ctl pull [dir] # live home → host (no .env/sessions/state.db) +``` + +Passthrough examples (Hermes's own commands): + +```bash +hermenix ctl chat +hermenix ctl update +hermenix ctl "sessions list" +hermenix ctl doctor +``` + +`edit` pulls over vsock, opens local `$EDITOR`, writes back, offers restart. +Secrets never enter the Nix store — no declarative option for them. + +If `status`/`logs` are empty, check `hermenix ctl units` and set +`services.hermenix.gatewayUnit` to the real unit name. + +`services.hermenix.hermesHome` is where `ctl` looks. Override for a flat +migrated layout; export `HERMES_HOME` inside the guest for the gateway +process if it differs from the default. + +--- + +## Env baked in by the host module + +| Env | Option | Default | +| ---------------------- | -------------------------------- | ------------------------- | +| `HERMES_VSOCK_CID` | `services.hermenix.vsockCid` | `77` | +| `HERMES_GUEST_NAME` | `services.hermenix.name` | `hermes` | +| `HERMES_DISK_BASE_DIR` | `services.hermenix.disk.baseDir` | `/var/lib/hermes-vm` | +| `HERMES_HOME_DIR` | `services.hermenix.hermesHome` | `/var/lib/hermes/.hermes` | +| `HERMES_GATEWAY_UNIT` | `services.hermenix.gatewayUnit` | `hermes-agent` | diff --git a/docs/CONCEPTS.md b/docs/CONCEPTS.md new file mode 100644 index 0000000..207d0c0 --- /dev/null +++ b/docs/CONCEPTS.md @@ -0,0 +1,98 @@ +# Concepts + +Shared model for everything else in these docs. Read once; other pages assume it. + +## Two disks + +| | Holds | Size (default) | Snapshotted | Backed up | +| ---------- | ---------------------------------------------- | -------------- | ----------- | --------- | +| **system** | NixOS, `/nix/store` | 80G | No | No | +| **state** | identity, workspace, `/etc/nixos`, Hermes home | 20G | Yes | Yes | + +System is large, churny, rebuildable from this flake. State holds what cannot +be reproduced — snapshots stay cheap. + +Host images under `services.hermenix.disk.baseDir` (default `/var/lib/hermes-vm`): + +``` +$baseDir/system/system.raw +$baseDir/state/state.raw +$baseDir/snapshots/ +``` + +Guest: state disk at **`/var/lib/hermes`**. Hermes home defaults to +`/var/lib/hermes/.hermes` (`~/.hermes` for user `hermes`). `/etc/nixos` → +`/var/lib/hermes/nixos`. + +What belongs _inside_ Hermes home (SOUL, memories, skills, sessions, …) is +defined by Hermes itself — see [upstream docs](https://github.com/NousResearch/hermes-agent). hermenix only +decides which disk it lives on and that the seed is applied once. + +## The boundary is the host + +The guest owns its kernel and config. Nothing _inside_ the guest is a security +boundary; `modules/host.nix` nftables are. QEMU runs as unprivileged +`hermes-vm` with seccomp + systemd hardening. Thin device model: virtio-blk, +virtio-net, vsock, RNG, serial. No virtiofs. `vhost-net` off on purpose. + +| | Default | +| ------------------------- | --------------------------------------------------- | +| Public internet | Yes | +| Host Nix cache | Yes, one port; `hostCache.enable = false` closes it | +| Anything else on the host | No, unless `hostAccess` | +| LAN / other VMs | No, unless `lanAccess` | +| Inbound from the network | No, unless `publish` (NAT mode) | +| Host hardware | None, unless `passthrough` | + +`network.bridged = true`: LAN inbound = every port the guest opens; `publish` +ignored. Outbound gates unchanged. [NETWORK.md](NETWORK.md). + +## Seed once + +`guest/hermes/` is a **template** copied into the guest on first boot only +(`/var/lib/hermes/.seeded`): + +``` +guest/hermes/ + home/ → ~/.hermes (SOUL.md, config.yaml, skills/, …) + workspace/ → ~/workspace +``` + +Hermes owns and rewrites its home after that — re-seeding on every rebuild +would wipe learned state. hermenix consequences: + +- Editing `guest/hermes/` post-first-boot does nothing alone → + `hermenix ctl reseed` (destructive; needs fresh confirmation). +- Live files: `hermenix ctl soul` / `conf` / … or `hermenix ctl pull`. +- Machine facts (not identity): `/etc/hermes/ENVIRONMENT.md`, regenerated + every guest rebuild from `guest/common.nix`. + +## vsock, not network SSH + +Host → guest (`vm ssh`, `ctl`, backups) uses **vsock** (default CID 77). No +management SSH port on the network path. + +``` +ssh -o ProxyCommand='socat - VSOCK-CONNECT::22' hermes@hermes +``` + +## Determinate Nix in the guest + +Guest: **Determinate Nix**, nixpkgs on FlakeHub. Host: stock Nix. + +- Do not edit `/etc/nix/nix.conf` — custom settings in + `/etc/nix/nix.custom.conf` (from `guest/common.nix`). +- No guest `nix.settings` / `nix.gc`; `determinate-nixd` collects. +- Substituters: FlakeHub Cache → host cache → `cache.nixos.org`. +- Do not set `inputs.nixpkgs.follows` on `determinate` (cache misses). + +## Quote multi-word subcommands + +nu treats bare `--flag` tokens as flags on the _outer_ command. Quote anything +past a single bare word: + +```bash +hermenix remote fwdesk.lab.internal "setup --clean" +hermenix vm ssh "systemctl status" +hermenix ctl "sessions list" +``` diff --git a/docs/DEVELOP.md b/docs/DEVELOP.md new file mode 100644 index 0000000..4183f69 --- /dev/null +++ b/docs/DEVELOP.md @@ -0,0 +1,82 @@ +# Hacking on this repo + +## Devshell + +```bash +direnv allow +``` + +Provides nushell, nufmt, nixfmt, and pre-commit hooks (installed on first +entry): + +- **treefmt** — nixfmt / deadnix / statix / nufmt +- **nu-check** — parse-check `*.nu` +- **gen-completions** — regenerates `nu/completions.nu` from usage blocks +- **nu-tests** — `bin/runtest` when anything under `nu/` changes + +## Layout + +See [GUIDELINES.md](../GUIDELINES.md#repository-layout). The CLI is several +nushell files concatenated by `flake.nix` in dependency order +(`lib/` → `host/` → `vm/` → `ctl/` → `hermenix.nu`). + +## Testing + +Three suites. **Agents may run them only through the wrappers below** — never +via bare `nix build` / `nix flake check`. See [GUIDELINES.md](../GUIDELINES.md). + +### CLI tests (`nu/tests/`, nutest) + +```bash +runtest # everything (bin/ on PATH in the devShell) +runtest vm_test +runtest-debug vm_test +``` + +Pure helpers + smoke that the subcommand tree parses — no ssh, no disk writes, +no `nix build`. + +### Guest VM tests (`guest/vm-tests/`) + +Boot `guest/common.nix` in ephemeral NixOS test VMs. Never touches +`/var/lib/hermes-vm/**`: + +```bash +runvmtest # NAT / isolated-bridge (default) +runvmtest guest-vm-bridged # bridged mode with simulated LAN +runvmtest-debug # interactive driver REPL +``` + +Shared seeding/SSH/user assertions live in `guest/vm-tests/common.nix`. +Mode-specific checks: + +- **unbridged** — static `10.77.0.2/24` only, one default route via host +- **bridged** — DHCP lease beside service net, dual default routes, LAN + reachability both ways + +Does not cover real `hermes-install` (no internet in the test VM) or the host +`hermes-vm` QEMU unit. + +### Module evaluation checks + +Static — no VM boot. Evaluates `modules/host.nix` in both network modes plus +`guest/common.nix`, asserts on nftables / networkd / guest DHCP metrics. + +Operator/CI (not the agent-safe wrapper path today): + +```bash +nix build .#checks.x86_64-linux.network-modes +``` + +Agents: describe the check and let the operator run it, unless/until a +`bin/` wrapper exists for it. + +## Parsing the CLI without running it + +```bash +nu nu/hermenix.nu --help +nu nu/hermenix.nu setup --help +# etc. +``` + +Nix syntax only: `nix-instantiate --parse path/to/file.nix`. diff --git a/docs/INSTALL.md b/docs/INSTALL.md new file mode 100644 index 0000000..76936b0 --- /dev/null +++ b/docs/INSTALL.md @@ -0,0 +1,117 @@ +# Install + +## Guided (recommended) + +The `hermenix` CLI is a flake package — no checkout needed: + +```bash +nix run git+ssh://gitea@gitea.skyg.ca/ari/hermenix.git#hermenix -- setup +``` + +Or install once: + +```bash +nix profile install git+ssh://gitea@gitea.skyg.ca/ari/hermenix.git#hermenix +hermenix setup +``` + +`setup` walks host-side steps: prerequisites, disks, cache signing key, +`/etc/hermenix/local.nix` from your SSH keys, bootstrap image build, write to +the system disk. It prints the module snippet to paste into your configuration +(uplink filled in). + +Safe to re-run — steps skip what's done; nothing destructive without +confirmation. It never runs `nixos-rebuild` (that's your config). + +Hand-run equivalent: [MANUAL_SETUP.md](MANUAL_SETUP.md). + +## Add the module + +```nix +# flake.nix +{ + inputs.hermenix.url = "git+ssh://gitea@gitea.skyg.ca/ari/hermenix.git"; + + outputs = { nixpkgs, hermenix, ... }: { + nixosConfigurations.myserver = nixpkgs.lib.nixosSystem { + system = "x86_64-linux"; + modules = [ hermenix.nixosModules.default ./configuration.nix ]; + }; + }; +} +``` + +```nix +# configuration.nix +services.hermenix = { + enable = true; + disk.baseDir = "/var/lib/hermes-vm"; + network.externalInterface = "eno1"; # your uplink + # network.bridged = true; # see docs/NETWORK.md + vcpu = 4; + mem = 8192; +}; +``` + +```bash +sudo nixos-rebuild switch +hermenix vm console # watch it boot +``` + +## First boot (hermenix side) + +```bash +hermenix vm ssh + +# Labels must match guest/hardware.nix BEFORE any guest rebuild +lsblk -f + +# FlakeHub Cache (device-code flow — approve URL from any browser) +determinate-nixd login + +sudo nix flake lock /etc/nixos +``` + +The Hermes CLI installs unattended on first boot (`hermes-install.service`; +a few minutes for native modules). After that, configure Hermes itself with +its own CLI — provider, keys, gateway, etc. That is **upstream Hermes**, not +hermenix; see the [Hermes Agent docs](https://github.com/NousResearch/hermes-agent). + +From the host you can drive the guest install/gateway unit once it exists: + +```bash +hermenix ctl units +hermenix ctl status +``` + +> **Check labels before the first guest `nixos-rebuild`.** +> `guest/hardware.nix` assumes root `nixos` and ESP `ESP`. Wrong labels → +> unbootable system after the agent's first self-rebuild. + +Guest NixOS changes are durable via +`sudo nixos-rebuild switch --flake /etc/nixos#hermes` inside the VM. Hermes +self-updates are upstream (`hermes update`); use `hermenix ctl` only as a +vsock passthrough if you prefer running them from the host. + +## Remote host + +`setup` / `migrate` act on the machine they run on. `hermenix remote` SSHs: + +```bash +hermenix remote fwdesk.lab.internal setup +hermenix remote fwdesk.lab.internal "setup --clean" +hermenix remote fwdesk.lab.internal "migrate /var/lib/hermes" +``` + +Quote subcommand + flags as **one string** when more than a bare name — +[CONCEPTS.md](CONCEPTS.md#quote-multi-word-subcommands). + +Runs `nix run #hermenix -- …` on the remote (no checkout). Remote needs +SSH access to Gitea. Override URL with `--flake`. Uses `ssh -t` for prompts. + +## Next + +- [CLI.md](CLI.md) — day-to-day `vm` / `ctl` +- [NETWORK.md](NETWORK.md) — allowlists / bridged +- [MIGRATE.md](MIGRATE.md) — existing `HERMES_HOME` +- [RECOVERY.md](RECOVERY.md) — snapshots / unbootable guest diff --git a/docs/MANUAL_SETUP.md b/docs/MANUAL_SETUP.md new file mode 100644 index 0000000..fd17a5f --- /dev/null +++ b/docs/MANUAL_SETUP.md @@ -0,0 +1,84 @@ +# Manual setup + +What `hermenix setup` does, step by step, for hand-running or when something +goes sideways. Prefer the guided command in [INSTALL.md](INSTALL.md) when you +can. + +## 1. Create the disks + +Two disks — design in [CONCEPTS.md](CONCEPTS.md#two-disks). + +```bash +sudo btrfs subvolume create /var/lib/hermes-vm/system +sudo btrfs subvolume create /var/lib/hermes-vm/state +sudo mkdir -p /var/lib/hermes-vm/snapshots + +# BEFORE creating any file inside them +sudo chattr +C /var/lib/hermes-vm/system /var/lib/hermes-vm/state +sudo btrfs property set /var/lib/hermes-vm/system compression none +sudo btrfs property set /var/lib/hermes-vm/state compression none + +sudo truncate -s 80G /var/lib/hermes-vm/system/system.raw +sudo truncate -s 20G /var/lib/hermes-vm/state/state.raw +sudo mkfs.ext4 -L hermes-state /var/lib/hermes-vm/state/state.raw +``` + +**`chattr +C` ordering matters.** Btrfs CoW stacked under a VM disk image +fragments badly. `+C` disables that — but only for files created _after_. +Set it on empty subvolumes and confirm with `lsattr` before writing anything. +Wrong order → recreate the subvolume and copy the image in fresh. + +Separate subvolumes so a btrfs snapshot of state does not include the system +image. + +## 2. Generate the cache signing key + +The host serves its store to the guest as a read-only binary cache. + +```bash +sudo mkdir -p /var/lib/secrets +sudo nix-store --generate-binary-cache-key hermes-cache-1 \ + /var/lib/secrets/hermes-cache.secret /var/lib/secrets/hermes-cache.pub +sudo chown nix-serve /var/lib/secrets/hermes-cache.secret +sudo chmod 0400 /var/lib/secrets/hermes-cache.secret +cat /var/lib/secrets/hermes-cache.pub +``` + +## 3. Write `/etc/hermenix/local.nix` + +Host-specific settings live here, not in this repo — see +`guest/local.nix.example`: + +```nix +{ + cachePublicKey = "hermes-cache-1:AbCd..."; # key you just printed; empty = skip host cache + authorizedKeys = [ "ssh-ed25519 AAAA... you@laptop" ]; +} +``` + +`sudo mkdir -p /etc/hermenix` first if needed. + +## 4. Build the image and write it to the system disk + +`--impure` is required because `guest/common.nix` reads +`/etc/hermenix/local.nix` outside the flake source tree. The extra substituter +flags avoid compiling Determinate Nix from source. `setup` also passes +`--no-write-lock-file --refresh` when building from the git URL — mirror that +if you're doing the same by hand. + +```bash +nix build --impure --no-write-lock-file --refresh \ + git+ssh://gitea@gitea.skyg.ca/ari/hermenix.git#hermes-guest-image \ + --option extra-substituters https://install.determinate.systems \ + --option extra-trusted-public-keys cache.flakehub.com-3:hJuILl5sVK4iKm86JzgdXW12Y2Hwd5G07qKtHTOcDCM= + +# notrunc keeps the disk at its full size; the image is smaller +sudo dd if=result/nixos.img of=/var/lib/hermes-vm/system/system.raw \ + bs=4M conv=notrunc,fsync status=progress +``` + +From a local checkout, use `.` instead of the git URL (still `--impure`). + +## 5. Module + first boot + +Continue from [INSTALL.md](INSTALL.md#add-the-module). diff --git a/docs/MIGRATE.md b/docs/MIGRATE.md new file mode 100644 index 0000000..0748c21 --- /dev/null +++ b/docs/MIGRATE.md @@ -0,0 +1,74 @@ +# Migrate an existing deployment + +Copy an existing flat `HERMES_HOME` into the guest after hermenix setup + host +`nixos-rebuild` and `hermenix vm ssh true` work: + +```bash +nix run git+ssh://gitea@gitea.skyg.ca/ari/hermenix.git#hermenix -- \ + migrate /path/to/existing/HERMES_HOME +``` + +`migrate --help` for vsock CID, guest user, `--dest`, hostname rewrites. + +**Copy only — never move or delete the source** until the new side is verified. + +Remote: [INSTALL.md](INSTALL.md#remote-host) — put `migrate /path/...` in the +quoted command string. + +What a `HERMES_HOME` contains and how Hermes uses it is upstream territory — +[Hermes Agent](https://github.com/NousResearch/hermes-agent). This page is only the hermenix carry path. + +## `hermesHome` + +`services.hermenix.hermesHome` is where `hermenix ctl` looks in the guest. + +| Layout | Value | +| -------------------------- | -------------------------------------------- | +| Seeded `.hermes` (default) | `/var/lib/hermes/.hermes` | +| Flat at state root | `migrate --dest` (default `/var/lib/hermes`) | + +Export `HERMES_HOME` inside the guest to the same path for the gateway process. + +## Don't use `sudo` + +`migrate` uses your user's ssh-over-vsock key (same as `authorizedKeys`). +`sudo` → root has no key → `Permission denied (publickey)`. If +`hermenix vm ssh` works as you, run `migrate` the same way. + +## `hermenix migrate connections` + +Connection/integration files only (no identity wipe), local or `--to-guest`: + +```bash +hermenix migrate connections /old/HERMES_HOME ./staging +hermenix migrate connections /old/HERMES_HOME --to-guest +``` + +Defaults: files `config.yaml`, `auth.json`, `.env`; dir `connections/`. +Rewrites apply in the destination only. + +## Troubleshooting + +### Unreadable source entries + +Rootless-Docker leftovers may be unreadable; `tar --ignore-failed-read` makes +that non-fatal. Pull those paths as root separately if needed: + +```bash +sudo tar -C /path/to/HERMES_HOME/.hermes -cf - . \ + | hermenix vm ssh 'mkdir -p /var/lib/hermes/.hermes-old && tar -C /var/lib/hermes/.hermes-old -xf -' +``` + +### Stale Docker launcher after an old migrate + +Current `migrate` excludes `.local/bin` and `.hermes-installed`. If an older +run left a Docker-routing `~/.local/bin/hermes` ("docker not found on PATH"): + +```bash +rm -f ~/.local/bin/hermes ~/.hermes-installed +sudo systemctl start hermes-install # or reinstall per upstream docs +journalctl -fu hermes-install +``` + +Then finish Hermes configuration with the upstream CLI +([docs](https://github.com/NousResearch/hermes-agent)), not with hermenix-specific steps. diff --git a/docs/NETWORK.md b/docs/NETWORK.md new file mode 100644 index 0000000..8f0d8c4 --- /dev/null +++ b/docs/NETWORK.md @@ -0,0 +1,120 @@ +# Network + +Default model and reachability table: [CONCEPTS.md](CONCEPTS.md#the-boundary-is-the-host). + +## Mode matrix + +| | NAT (default) | Bridged (`network.bridged = true`) | +| ------------------------------ | ------------------------------------- | ---------------------------------------- | +| Guest address | `10.77.0.2` on isolated bridge | service net **plus** fixed LAN address | +| Inbound from LAN / internet | only via `publish` | every port the guest opens | +| Guest → public internet | yes (masqueraded) | yes (L2 / LAN gateway) | +| Guest → private LAN | `lanAccess` / `allowLan` only | same gates | +| Guest → host services | `hostAccess` + host cache on `10.77.0.1` | same over service net | +| `publish` | DNAT on uplink | **ignored** | + +## Guest → a service on your LAN + +```nix +services.hermenix.lanAccess = [ + { address = "192.168.1.50"; port = 11434; } # ollama + { address = "192.168.1.50"; port = 9000; proto = "udp"; } +]; +``` + +One destination:port per entry, punched through the RFC1918 drop. Traffic is +masqueraded; the peer sees the host's address. + +`allowLan = true` opens all private space at once. Prefer `lanAccess` — an +agent that goes wrong should not wander. + +## Guest → a service on the host + +```nix +services.hermenix.hostAccess = [ { port = 8188; } ]; +``` + +The service must bind somewhere the guest can see. `127.0.0.1` will not work +(guest arrives on the bridge). Bind to `10.77.0.1` (bridge only) or +`0.0.0.0`: + +```nix +services.comfyui.listenAddress = "10.77.0.1"; +``` + +From the guest: `http://10.77.0.1:8188`. + +Often better than GPU passthrough when you only need the capability over HTTP +— see [PASSTHROUGH.md](PASSTHROUGH.md). + +## Expose a guest service + +**Guest side** — agent opens its firewall in `/etc/nixos`: + +```nix +networking.firewall.allowedTCPPorts = [ 22 8080 ]; +``` + +**Host only** — already allowed (outbound host→guest): + +```bash +curl http://10.77.0.2:8080 +``` + +**Wider network** (NAT mode): + +```nix +services.hermenix.publish = [ + { hostPort = 8080; guestPort = 8080; } +]; +``` + +DNATs the uplink port and opens the forward rule. Ports need not match. +Publishing is the one change that lets the outside world start conversations +with the agent's machine — put TLS/auth in front of anything that isn't a toy. + +## Bridged mode (guest on the LAN) + +Guest becomes a first-class LAN device: fixed address, reachable on any port it +opens, outbound still gated. + +```nix +services.hermenix.network = { + bridged = true; + externalInterface = "enp191s0"; + + lanAddress = "10.0.12.50"; # guest's fixed LAN address + hostLanAddress = "10.0.12.10"; # this host's LAN address + lanGateway = "10.0.12.1"; + lanPrefixLength = 24; + # lanDns = [ "1.1.1.1" ]; # optional; defaults to lanGateway +}; +``` + +The bridge absorbs the uplink. The host-side DHCP on the bridge reserves +`lanAddress` for the guest MAC; the guest stays a plain DHCP client. The +bridge clones the uplink MAC so the host keeps its LAN identity. Guest DHCP +broadcast never leaves the bridge. + +### Switching over + +1. Pick free addresses for guest, host (use the host's existing LAN address if + reserved), and gateway. Keep them outside the DHCP pool or reserve them. +2. Set options above, `sudo nixos-rebuild switch` (brief network drop). +3. Verify: `hermenix vm ssh`; inside, `ip -4 addr` shows `10.77.0.2` **and** + the LAN address. From another LAN device: + `ssh hermes@`. + +### Switching back + +`bridged = false` and rebuild. Guest DHCP finds no server on the isolated +bridge and falls back to the static route via the host. No guest-side change. + +### Gotchas + +- `hostLanAddress` is a static LAN claim — reserve it. +- Uplink network config is replaced by the bridge-port role while bridged. +- Older guest images without DHCP-on-NIC in `guest/common.nix` won't take a + bridged address; upgrade the image or apply that config inside the VM first. +- Drop `publish` entries if you return to NAT mode; they do nothing while + bridged. diff --git a/docs/PASSTHROUGH.md b/docs/PASSTHROUGH.md new file mode 100644 index 0000000..b95f40a --- /dev/null +++ b/docs/PASSTHROUGH.md @@ -0,0 +1,124 @@ +# Hardware passthrough & virtual devices + +## Before reaching for passthrough + +If the goal is "the agent can use the GPU", passthrough is often wrong. It is +exclusive (host loses the card while the VM runs) and is the largest isolation +reduction in this setup. + +Prefer: keep Ollama/ComfyUI on the host, open the port with `hostAccess` +([NETWORK.md](NETWORK.md#guest--a-service-on-the-host)). Pass hardware through +when the guest genuinely needs the device — CUDA in-process, capture card, +radio, display output. + +## What PCI passthrough costs + +A PCI device does DMA. The IOMMU is the only thing between it and host memory, +so `passthrough.iommu` is required; the module refuses to build without it. + +- **Whole IOMMU groups only.** Check before planning: + + ```bash + for d in /sys/kernel/iommu_groups/*/devices/*; do + echo "group $(basename "$(dirname "$(dirname "$d")")"): $(lspci -nns "$(basename "$d")")" + done | sort -V + ``` + + ACS override patches lie about isolation — not appropriate here. + +- **Every function.** A GPU is at least graphics + HDMI audio. + +## GPU + +```bash +lspci -Dnn | grep -i vga +``` + +```nix +services.hermenix.passthrough = { + iommu = "amd"; # or "intel" + pci = [ + { address = "0000:01:00.0"; id = "10de:2204"; } # graphics + { address = "0000:01:00.1"; id = "10de:1aef"; } # HDMI audio + ]; +}; +``` + +Setting `id` makes vfio-pci claim the card in the initrd before `nvidia` / +`amdgpu`. Do this for GPUs. Address-only devices bind at boot via a oneshot +instead. + +Rebuild and **reboot**, then: + +```bash +lspci -nnk -d 10de:2204 | grep "Kernel driver" # want: vfio-pci +``` + +Guest-side drivers are the agent's problem (`/etc/nixos`). + +## USB / Bluetooth + +```bash +lsusb +``` + +```nix +services.hermenix.passthrough.usb = [ + { vendorId = "8087"; productId = "0026"; } + { vendorId = "0403"; productId = "6001"; } +]; +``` + +QEMU mediates (no IOMMU/VFIO). Host must not be using the device (`btusb` +blacklist, no `hardware.bluetooth` for an integrated radio). No hotplug — +`hermenix vm restart` after plugging. Whole USB controller as PCI gives real +hotplug at the cost of exclusivity. + +## When the VM won't start + +`hermenix vm console` first. Then: device not on vfio-pci (`lspci -nnk`), +IOMMU off (`dmesg | grep -i iommu`), or something in the group still held by +the host. The module raises `LimitMEMLOCK` when PCI passthrough is configured. + +--- + +## Virtual devices + +`services.hermenix.devices` attaches optional QEMU devices (no host hardware): + +| Option | Default | Attaches | +| ------------------------- | ---------- | ----------------------------------------------------- | +| `devices.rng.enable` | `true` | virtio entropy | +| `devices.vsock.enable` | `true` | host↔guest vsock (`ctl`, backups) | +| `devices.video.enable` | `false` | virtual display / render node | +| `devices.video.model` | `"virtio"` | `virtio` (render node) or legacy VGA | +| `devices.balloon.enable` | `false` | virtio memory balloon | +| `devices.watchdog.enable` | `false` | `i6300esb`/`ib700` | +| `devices.input.enable` | `false` | virtio keyboard/tablet | +| `devices.audio.enable` | `false` | `intel-hda`/`virtio-sound` (see caveat) | +| `devices.tpm.enable` | `false` | TPM 2.0 via swtpm | +| `devices.cdroms` | `[]` | read-only CD images | +| `devices.extraDisks` | `[]` | extra raw disks | +| `devices.usbController` | `"auto"` | `qemu-xhci` iff `passthrough.usb` set | + +`extraQemuArgs` appends raw QEMU args (unvalidated). Prefer typed options. + +### Render node + +```nix +services.hermenix.devices.video.enable = true; +``` + +Gives `/dev/dri/card0` and `/dev/dri/renderD128` (`virtio` model only). +Console stays serial. + +### TPM + +`devices.tpm.enable = true` starts swtpm; state under +`/var/lib/hermes-vm-tpm` by default. + +### Audio caveat + +`hermes-vm.service` is hardened (`ProtectHome`, `PrivateTmp`, dedicated user) +and cannot reach PipeWire without loosening that. Leave audio off unless you +mean to relax the unit. diff --git a/docs/RECOVERY.md b/docs/RECOVERY.md new file mode 100644 index 0000000..fbbc090 --- /dev/null +++ b/docs/RECOVERY.md @@ -0,0 +1,63 @@ +# Recovery + +Cheapest first. Layers 1–2 are inside the guest; 3–4 are host-side hermenix. + +```bash +# Guest NixOS history +hermenix vm ssh "git -C /etc/nixos log --oneline" +hermenix vm ssh "sudo nixos-rebuild switch --rollback" # also in boot menu + +# State-disk snapshot +hermenix vm snapshot +hermenix vm snapshots +hermenix vm rollback state-20260914-1400 # type 'rollback' to confirm + +# Restic backup of state files +hermenix vm backup +hermenix vm backups +``` + +Identity files under Hermes home can be inspected with `hermenix ctl soul` +etc.; restoring _Hermes_ app state is mostly "roll the state disk" or restore +from backup — not a separate product flow. + +## Snapshots + +Hourly, keep 48, **state disk only**. System disk: re-image with +`hermenix setup`. + +fsfreeze via qemu-guest-agent around each shot. `rollback` stops the VM, moves +current state to `state.old-`, restores snapshot — reversible until +you delete `state.old-*`. + +Keep retention tighter than on ZFS: state image is nodatacow; live snapshots +reintroduce CoW on referenced blocks. + +## Backups + +```nix +services.hermenix.backup = { + enable = true; + repository = "sftp:backup@nas:/hermes"; + passwordFile = "/var/lib/secrets/hermes-restic"; +}; +``` + +State **files** over vsock/SSH into restic — not the raw disk. Portable; +avoids loop-mounting agent-controlled FS on the host. Runs on the host (no +`lanAccess` needed for a LAN repo). + +## Guest won't boot + +Re-image **system** only (`hermenix setup` write step); leave state disk. +Identity + `/etc/nixos` + workspace return with it. + +## Starting completely fresh + +```bash +hermenix setup --clean +``` + +Stops VM; deletes `system.raw` / `state.raw` / UEFI vars; drops stale +`known_hosts` for the guest host key; walks setup again. Subvolumes/snapshots +kept. Type `clean` to confirm. diff --git a/docs/ROUGH_EDGES.md b/docs/ROUGH_EDGES.md new file mode 100644 index 0000000..94a7cd7 --- /dev/null +++ b/docs/ROUGH_EDGES.md @@ -0,0 +1,21 @@ +# Known rough edges + +- The Determinate NixOS module's option names for custom Nix settings aren't + published — only the nix-darwin `determinateNix.customSettings` equivalent + is. So `guest/common.nix` writes `/etc/nix/nix.custom.conf` via + `environment.etc`. If a future module version claims that file, switch to + its option. +- `determinate-nixd login` is manual. No safe way to bake the credential into + the image. +- `MemoryDenyWriteExecute = true` is correct under KVM but is the first thing + to flip if QEMU refuses to start. +- `pkgs.OVMF.firmware` / `pkgs.OVMF.variables` are current attribute names; + older nixpkgs put these under `pkgs.OVMF.fd` in an `FV/` subdirectory. +- PCI passthrough needs a reboot (early vfio binding in initrd). +- Snapshots need btrfs. Everything else works on any filesystem; without btrfs + you rely on backups alone. +- `state.old-*` left by `hermenix vm rollback` are not cleaned automatically. + Delete them once you're sure. +- First boot builds a lot. Give it time. +- `devices.audio.enable` fights unit hardening — see + [PASSTHROUGH.md](PASSTHROUGH.md#audio-caveat). diff --git a/flake.nix b/flake.nix index bfc3263..2422fd4 100644 --- a/flake.nix +++ b/flake.nix @@ -20,8 +20,7 @@ url = "github:numtide/treefmt-nix"; inputs.nixpkgs.follows = "nixpkgs"; }; - # A nushell test framework, used by nu/tests/. No nixpkgs package exists - # for it, so it's pulled in as a plain source input instead. + # No nixpkgs package; plain source for nu/tests. nutest = { url = "github:vyadh/nutest/v1.2.0"; flake = false; @@ -117,23 +116,12 @@ modules = [ ./devenv.nix ]; }; - # Never run by an agent — operator/CI only: - # nix build .#checks.x86_64-linux.guest-vm-unbridged --impure -L - # nix build .#checks.x86_64-linux.guest-vm-bridged --impure -L - # nix build .#checks.x86_64-linux.migrate-connections-vm --impure -L - # - # The guest VM tests (guest/vm-tests/) share their node config and - # mode-independent assertions via guest/vm-tests/common.nix; each test - # adds the checks that only apply to its mode. default.nix aggregates - # them into this attrset, which the static network-modes check joins. checks.${system} = (import ./guest/vm-tests { inherit pkgs determinate; hermenix = self.packages.${system}.hermenix; }) // { - # No build, no VM boot: evaluates host.nix in both networking modes - # and guest/common.nix, and asserts on the generated rulesets/units. network-modes = import ./tests/network-modes-check.nix { inherit pkgs; }; }; }; diff --git a/guest/common.nix b/guest/common.nix index 9c7e3a2..8db4e47 100644 --- a/guest/common.nix +++ b/guest/common.nix @@ -6,22 +6,13 @@ }: let - # These must match services.hermenix.* on the host. If you changed the - # defaults there, change them here too before building the image. + # Must match services.hermenix.* defaults on the host. hostAddress = "10.77.0.1"; guestAddress = "10.77.0.2"; prefixLength = 24; cachePort = 5000; - # Host-specific settings (cache signing key, your SSH key) live at a fixed - # path on the HOST, not in this repo - see local.nix.example for the shape. - # `hermenix setup` writes it there. Deliberately outside the flake's own - # source tree: this flake is meant to be built straight from its git+ssh - # URL with no local checkout at all (hence --impure, to allow reading a - # path Nix can't otherwise prove is pure), and a repo-relative file would - # either need a checkout to hold it or get baked into the immutable store - # copy Nix fetches. Falls back to safe defaults (no host cache, no SSH - # access) if it hasn't been created yet. + # Host-only settings outside the flake (--impure). See local.nix.example. local = if builtins.pathExists /etc/hermenix/local.nix then import /etc/hermenix/local.nix else { }; cachePublicKey = local.cachePublicKey or ""; authorizedKeys = local.authorizedKeys or [ ]; @@ -29,19 +20,7 @@ let stateDir = "/var/lib/hermes"; useHostCache = cachePublicKey != ""; - # Shared between environment.systemPackages (the interactive PATH) and - # hermes-install's own `path` (below). environment.systemPackages does NOT - # propagate into systemd services' PATH on its own - a bare `curl` inside - # a systemd script only resolves if the unit is explicitly given a `path`, - # which is what actually failed here (confirmed live: "curl: command not - # found", then separately "sh: command not found" from the upstream - # installer's uv-install step, both from inside hermes-install despite the - # tools existing in systemPackages). `bash` is included explicitly because - # its output provides `bin/sh` (the same symlink NixOS's own `/bin/sh` - # compat path points at) - nothing else in this list provides a bare `sh`. - # The rest of the coreutils-adjacent tools are a baseline so the next - # third-party install script that shells out to some common Unix tool by - # bare name doesn't trigger another one of these round-trips. + # systemPackages does not populate systemd unit PATH; list deps once for both. hermesInstallDeps = with pkgs; [ git curl @@ -74,11 +53,7 @@ let ]; in { - # Both disks are virtio-blk, including the one this very image boots from - # (see modules/host.nix's qemu invocation) - so this has to be here, not - # only in hardware.nix. hardware.nix is excluded from this image's build - # (nixos-generators supplies its own partitioning/bootloader instead), but - # driver support for the VM environment itself is needed either way. + # virtio drivers needed in bootstrap image too (hardware.nix is not imported there). imports = [ (modulesPath + "/profiles/qemu-guest.nix") ]; boot.initrd.availableKernelModules = [ "virtio_pci" @@ -91,17 +66,7 @@ in networking.hostName = "hermes"; - # ===================================================================== - # The state disk. - # - # Second virtio disk, holding everything that cannot be rebuilt: Hermes's - # identity and memories, its workspace, and the NixOS config it maintains - # for itself. The system disk holds NixOS and the Nix store and is - # deliberately disposable. - # - # neededForBoot so it is mounted in the initrd, before activation runs and - # tries to seed into it. - # ===================================================================== + # State disk (identity, workspace, /etc/nixos). neededForBoot: seed runs before activation. fileSystems."${stateDir}" = { device = "/dev/disk/by-label/hermes-state"; fsType = "ext4"; @@ -112,14 +77,7 @@ in ]; }; - # ===================================================================== - # Seed identity and configuration on first boot, then get out of the way. - # - # FIRST BOOT ONLY, deliberately. Hermes rewrites SOUL.md, its memories and - # its own NixOS config as it goes - that is the point of the thing. Copying - # the seed over it on every rebuild would quietly delete everything it had - # worked out. Use `hermes-ctl reseed` when you actually mean to overwrite. - # ===================================================================== + # First boot only — re-applying would wipe agent-owned SOUL/memories. Use hermenix ctl reseed. system.activationScripts.seedHermesState = { deps = [ "users" @@ -139,15 +97,7 @@ in chmod -R u+w ${stateDir}/nixos ${stateDir}/.hermes ${stateDir}/workspace - # Git history here is the agent's cheapest undo, and it stops - # nixos-rebuild complaining about an untracked flake. - # - # Must run before the chown below, while ${stateDir}/nixos is still - # root-owned: activation runs as root, and git refuses to operate on - # a repository whose working tree is owned by a different user - # ("detected dubious ownership") - it would otherwise fail here on - # every boot, silently (activation script failures are non-fatal), - # and the seed would never actually get committed. + # Commit while still root-owned (git refuses foreign ownership; activation failures are silent). if [ ! -d ${stateDir}/nixos/.git ]; then ${pkgs.git}/bin/git -C ${stateDir}/nixos init -q -b main ${pkgs.git}/bin/git -C ${stateDir}/nixos add -A @@ -162,33 +112,13 @@ in chown hermes:hermes ${stateDir}/.seeded fi - # /etc/nixos points at the state disk, so the config is snapshotted and - # backed up along with everything else the agent owns. - # - # /etc/nixos is a real directory by default on NixOS, not a symlink or - # missing path - `ln -sfn` only replaces an existing LINKNAME in place - # if it's already a symlink; against a plain directory it instead - # creates the link *inside* it (`/etc/nixos/nixos -> ...`), leaving - # /etc/nixos itself untouched. rm -rf first so this is idempotent - # either way. + # /etc/nixos is a real dir by default; ln -sfn would nest inside it — rm first. rm -rf /etc/nixos ln -sfn ${stateDir}/nixos /etc/nixos ''; }; - # ===================================================================== - # Networking: a static service-net address for talking to the host (nix - # cache, hostAccess), plus DHCP so an operator can later switch this VM - # onto the LAN (services.hermenix.network.bridged = true on the host). - # - NAT mode: no DHCP server on the host bridge; the static default - # route via the host is the only one and carries all traffic. - # - bridged mode: the host runs a DHCP server on its bridge that hands - # this NIC its fixed LAN address (services.hermenix.network.lanAddress, - # keyed to this NIC's MAC); that lease's lower-metric default route - # (100) wins over the static fallback (200), and LAN devices reach - # the VM at the fixed address. The VM never needs to know the address. - # Resolvers stay QUAD9 (UseDNS = false) in both modes. - # ===================================================================== + # Static service-net + DHCP (bridged mode gets LAN lease at metric 100; host route at 200). networking.useNetworkd = true; systemd.network.enable = true; systemd.network.networks."10-eth" = { @@ -224,22 +154,9 @@ in }; networking.firewall.allowedTCPPorts = [ 22 ]; - # Lets the host fsfreeze before taking a snapshot. services.qemuGuest.enable = true; - # ===================================================================== - # Nix - # - # Determinate Nix owns /etc/nix/nix.conf and their docs are explicit that - # nix.custom.conf is the only supported place for your own settings. So - # there is no nix.settings block here, and no nix.gc either: determinate-nixd - # runs the collector itself. - # - # If the Determinate NixOS module on your version manages this file itself, - # drop this block and use its option instead. The documented nix-darwin - # equivalent is determinateNix.customSettings; the NixOS module's own option - # names are not published, so this writes the file directly. - # ===================================================================== + # Determinate owns nix.conf — custom settings only via nix.custom.conf (no nix.settings). environment.etc."nix/nix.custom.conf".text = '' trusted-users = root hermes extra-experimental-features = nix-command flakes @@ -248,17 +165,8 @@ in extra-trusted-public-keys = cache.flakehub.com-3:hJuILl5sVK4iKm86JzgdXW12Y2Hwd5G07qKtHTOcDCM=${lib.optionalString useHostCache " ${cachePublicKey}"} ''; - # Keep enough generations that a bad self-rebuild is one boot-menu entry - # away from being undone. boot.loader.systemd-boot.configurationLimit = lib.mkDefault 20; - # ===================================================================== - # The agent's account - # - # Inside the VM the agent is effectively root. That is the point: the VM - # is the boundary, not the account. It can rebuild the system, open ports, - # add services, and none of it reaches the host. - # ===================================================================== users.users.hermes = { isNormalUser = true; home = stateDir; @@ -274,22 +182,7 @@ in users.groups.hermes = { }; security.sudo.wheelNeedsPassword = false; - # ===================================================================== - # Hermes itself - # - # Installed with upstream's own installer rather than the Nix flake, so - # that `hermes update` works natively and the agent owns its own config. - # Upstream classes Nix/NixOS as a best-effort Tier 2 platform; inside a VM - # built to contain mess, taking the supported path is the cheaper trade. - # - # The installer itself runs automatically on first boot (below) - it - # supports a fully unattended mode and has no need for operator input. - # `hermes setup` and `hermes gateway install` do need real input (model - # provider, API keys, messaging platform tokens), so those stay manual: - # hermes setup - # hermes gateway install - # loginctl enable-linger hermes - # ===================================================================== + # Upstream installer (not Nix) so hermes update works. Interactive config is upstream docs. systemd.services.hermes-install = { description = "Install the Hermes Agent CLI (first boot only)"; after = [ "network-online.target" ]; @@ -335,13 +228,7 @@ in environment.systemPackages = hermesInstallDeps; - # hermes has createHome = false (its home is the state disk, seeded above, - # not NixOS's usual skeleton) so it never gets a ~/.bashrc to append to. - # Upstream's installer puts the `hermes` launcher in ~/.local/bin and says - # as much ("Could not detect shell config file to add ~/.local/bin to - # PATH"). This option adds it via /etc/set-environment (PAM, every login - # session) instead of relying on a per-user dotfile that will never exist - # here. Same option hl-fwdesk.nix already sets on the physical host. + # createHome=false so no ~/.bashrc; put ~/.local/bin on PATH via PAM instead. environment.localBinInPath = true; programs.direnv = { @@ -376,9 +263,13 @@ in keep anything you care about outside /var/lib/hermes. - Your identity is ~/.hermes/SOUL.md and it is yours to edit. It was seeded once on first boot and is never overwritten by a rebuild, so - changes you make there survive. Same for ~/.hermes/memories/. + changes you make there survive. Same for ~/.hermes/memories/ and + anything else you keep under ~/.hermes. - This file is regenerated on every system rebuild. Editing it is pointless; it will be replaced. Put durable notes in SOUL.md. + - Before your first `nixos-rebuild switch`, commit /etc/nixos. That + commit is the cheapest undo; previous generations are in the boot + menu if a rebuild goes badly. - The skill at ~/.hermes/skills/nixos-self-config covers how to change this machine safely. Read it before your first rebuild. - In NAT mode you have no route to the host or to any local network: diff --git a/guest/hardware.nix b/guest/hardware.nix index efa4b85..c87218b 100644 --- a/guest/hardware.nix +++ b/guest/hardware.nix @@ -1,16 +1,8 @@ { lib, ... }: -# Only used by the guest's own /etc/nixos flake (see guest/flake.nix). The -# bootstrap image gets its partitioning and bootloader from nixos-generators -# instead, which is why fileSystems/bootloader are kept out of common.nix - -# but driver support for the VM environment (qemu-guest.nix profile, virtio -# initrd modules) lives in common.nix instead, since the bootstrap image -# needs that too. See common.nix if you're looking for that. -# -# VERIFY THE LABELS on first boot with `lsblk -f`. nixos-generators names the -# root partition "nixos" and the ESP "ESP", but check rather than trust, and -# fix here if they differ. Getting this wrong means the first self-rebuild -# produces a system that will not boot. +# Used only by the guest's /etc/nixos flake. Bootstrap image partitioning +# comes from nixos-generators (see common.nix for virtio drivers). +# Verify labels with `lsblk -f` before the first self-rebuild. { fileSystems."/" = { diff --git a/guest/hermes/home/config.yaml b/guest/hermes/home/config.yaml index 92bb74c..89f0a71 100644 --- a/guest/hermes/home/config.yaml +++ b/guest/hermes/home/config.yaml @@ -1,11 +1,9 @@ -# Minimal starter config.yaml. -# -# Seeded once on first boot to ~/.hermes/config.yaml, then never touched -# again by a rebuild — edit it live with `hermes-ctl conf edit`, or just let -# Hermes manage it. Fill in a model provider before first use. +# Seeded once to ~/.hermes/config.yaml; never re-applied by a rebuild. +# Shape and keys: upstream Hermes docs. Edit live via `hermenix ctl conf edit` +# or let Hermes manage it. model: - provider: '' - default: '' + provider: "" + default: "" agent: reasoning_effort: medium diff --git a/guest/local.nix.example b/guest/local.nix.example index 7c5d53c..217d759 100644 --- a/guest/local.nix.example +++ b/guest/local.nix.example @@ -1,17 +1,10 @@ -# Host-specific settings, imported by common.nix from /etc/hermenix/local.nix -# on the HOST (not from this repo - it lives outside the flake's own source -# tree on purpose, since this flake is meant to be built from its git+ssh URL -# with no local checkout at all). -# -# `hermenix setup` writes /etc/hermenix/local.nix for you. This file is just -# the reference for its shape. +# Written to /etc/hermenix/local.nix on the host (outside the flake; --impure). +# `hermenix setup` creates it. See docs/INSTALL.md. { - # Output of `cat /var/lib/secrets/hermes-cache.pub` on the host, once - # generated. Leave empty (or omit the file entirely) and the guest just - # skips the host cache, falling back to cache.nixos.org. + # cat /var/lib/secrets/hermes-cache.pub — empty skips the host cache. cachePublicKey = ""; # e.g. "hermes-cache-1:AbCd..." - # Your SSH public key(s), for `hermes-vm ssh`. + # SSH keys for hermenix vm ssh. authorizedKeys = [ # "ssh-ed25519 AAAA... you@laptop" ]; diff --git a/modules/host.nix b/modules/host.nix index 7c1f19e..44aae76 100644 --- a/modules/host.nix +++ b/modules/host.nix @@ -19,14 +19,6 @@ let vmUser = "hermes-vm"; runDir = "/run/hermes-vm"; stateDir = "/var/lib/hermes-vm"; - # Two disks, deliberately. - # - # The system disk is large, churny and entirely rebuildable - it is the - # NixOS install and the Nix store. We never snapshot it. - # - # The state disk holds the only things that cannot be reproduced: Hermes's - # identity and memories, its workspace, and the NixOS config it has written - # for itself. Small, so snapshots are cheap and backups are quick. systemImage = "${cfg.disk.baseDir}/system/system.raw"; stateImage = "${cfg.disk.baseDir}/state/state.raw"; snapDir = "${cfg.disk.baseDir}/snapshots"; @@ -36,25 +28,15 @@ let anyUsb = pt.usb != [ ]; earlyIds = lib.filter (i: i != null) (map (d: d.id) pt.pci); - # "auto" adds a USB controller only when something needs plugging into it; - # "on"/"off" force the decision. usbControllerOn = cfg.devices.usbController == "on" || (cfg.devices.usbController == "auto" && anyUsb); - # swtpm state. Must live under /var/lib/ so the unit's StateDirectory owns it. tpmStateDir = "/var/lib/hermes-vm-tpm"; - # Netfilter mark tagging guest-originated traffic in bridged mode. The - # guest's only way in is the tap interface, so marking at the bridge - # prerouting hook identifies the guest regardless of which address it - # currently presents - its static service-net address, or the fixed LAN - # address served to it by the host-side DHCP on the bridge. 18501 == - # 0x4845 ("HE"), for hermenix. + # 18501 == 0x4845 ("HE"); marks guest traffic from the tap (bridged mode). bridgeMark = 18501; - # Dotted-quad netmask for a prefix length, used by the bridged-mode DHCP - # server's single-address lease (dnsmasq needs an explicit mask for a - # one-address range). + # dnsmasq needs an explicit mask for a one-address DHCP range. netmaskFromPrefix = p: lib.concatStringsSep "." ( @@ -76,15 +58,12 @@ let ) 4 ); - # Graceful ACPI power button via QMP, then wait for QEMU to exit. stopScript = pkgs.writeShellScript "hermes-vm-stop" '' set -eu printf '%s' '{"execute":"qmp_capabilities"}{"execute":"system_powerdown"}' \ | ${pkgs.socat}/bin/socat - UNIX-CONNECT:${runDir}/qmp.sock >/dev/null 2>&1 || exit 0 ''; - # Freeze the guest filesystems around a snapshot so it captures a consistent - # state rather than the equivalent of pulling the power. qga = verb: pkgs.writeShellScript "hermes-vm-fs${verb}" '' @@ -419,13 +398,14 @@ in NAT. The bridge absorbs `externalInterface`, so any device on the LAN can reach any port the guest exposes. - Addressing is fully static - no DHCP is involved. You must give - the guest a fixed LAN address (`lanAddress`), the host's own LAN - address (`hostLanAddress` - it moves onto the bridge), and the - LAN gateway (`lanGateway`). The guest is handed `lanAddress` by a - DHCP server this module runs on the bridge, keyed to the guest's - MAC (`network.mac`), so the guest itself stays a plain DHCP - client and never needs to know its own address. + Addressing is static from the operator's point of view: you pick + fixed addresses for the guest (`lanAddress`), this host + (`hostLanAddress` — it moves onto the bridge), and the LAN + gateway (`lanGateway`). A DHCP server this module runs on the + bridge hands `lanAddress` to the guest, keyed to `network.mac`, + so the guest stays a plain DHCP client and never needs to know + its own address. That lease never leaves the bridge; the LAN's + own DHCP cannot race it. Inbound is no longer limited to `publish` (which is ignored in this mode): the LAN may reach every guest port. Outbound is still @@ -761,18 +741,20 @@ in hermesHome = mkOption { type = types.str; - default = "${stateDir}/.hermes"; + # Guest path (not host disk.baseDir). + default = "/var/lib/hermes/.hermes"; description = '' Where Hermes's own data (SOUL.md, config.yaml, memories/, sessions/, state.db, and so on) lives inside the guest. Defaults to the seeded - layout under the state disk mount. Override this when carrying over - an existing flat HERMES_HOME from elsewhere (e.g. a prior Docker - deployment) that you're placing directly at the root of the state - disk instead of under .hermes - see `hermenix migrate --help`. + layout under the state disk mount (`/var/lib/hermes/.hermes`). + Override when carrying over an existing flat HERMES_HOME from + elsewhere (e.g. a prior Docker deployment) that you're placing + directly at the root of the state disk instead of under .hermes — + see `hermenix migrate --help`. This only changes where the host-side `hermenix ctl` commands (soul, - memory, conf, secrets, reseed, pull) looks. It does not change what - the gateway process itself uses - export HERMES_HOME to match inside + memory, conf, secrets, reseed, pull) look. It does not change what + the gateway process itself uses — export HERMES_HOME to match inside the guest when you install it. ''; }; @@ -856,12 +838,6 @@ in }; users.groups.${vmUser} = { }; - # ===================================================================== - # The VM itself - # - # No virtiofs, no shared filesystem of any kind, no vhost-net. The guest - # gets a block device, a tap, a vsock and a serial port. That is all. - # ===================================================================== systemd.services.hermes-vm = { description = "Hermes Agent VM"; wantedBy = [ "multi-user.target" ]; @@ -883,7 +859,6 @@ in exit 1 fi done - # Per-VM writable UEFI variable store, seeded once. if [ ! -e ${stateDir}/OVMF_VARS.fd ]; then install -m 0600 ${pkgs.OVMF.variables} ${stateDir}/OVMF_VARS.fd fi @@ -894,8 +869,6 @@ in Group = vmUser; SupplementaryGroups = [ "kvm" ]; RuntimeDirectory = "hermes-vm"; - # ProtectSystem=strict makes everything read-only; punch through for - # the images and the UEFI variable store only. ReadWritePaths = [ cfg.disk.baseDir ]; StateDirectory = "hermes-vm"; RuntimeDirectoryMode = "0750"; @@ -910,32 +883,21 @@ in "-m ${toString cfg.mem}" "-nodefaults" "-no-user-config" - # Display Settings "-display none" "-serial stdio" - # QEMU's own seccomp sandbox, everything denied that we don't need. "-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny" - # UEFI: firmware read-only, variables per-VM and writable. "-drive if=pflash,format=raw,unit=0,readonly=on,file=${pkgs.OVMF.firmware}" "-drive if=pflash,format=raw,unit=1,file=${stateDir}/OVMF_VARS.fd" - # System disk: NixOS and the Nix store. Rebuildable, never snapshotted. - # aio=threads and a writeback cache because these are files on a - # filesystem, not block devices; aio=native wants the latter. + # aio=threads+writeback: images are files, not blockdevs. "-drive if=none,id=system,format=raw,file=${systemImage},cache=writeback,aio=threads,discard=unmap" "-device virtio-blk-pci,drive=system" - # State disk: everything irreplaceable. Mounted at /var/lib/hermes. "-drive if=none,id=state,format=raw,file=${stateImage},cache=writeback,aio=threads,discard=unmap" "-device virtio-blk-pci,drive=state" - # Pre-created tap. vhost=off on purpose: vhost-net would move packet - # handling into the host kernel, which is exactly the surface we are - # trying not to expose. Costs throughput we do not need. + # vhost=off: keep packet handling out of the host kernel. "-netdev tap,id=net0,ifname=${net.tapId},script=no,downscript=no,vhost=off" "-device virtio-net-pci,netdev=net0,mac=${net.mac}" - # Host -> guest management, no network involved. "-qmp unix:${runDir}/qmp.sock,server=on,wait=off" ] - # Optional devices. rng and vsock are on by default; everything else - # is off unless enabled via services.hermenix.devices. ++ lib.optionals cfg.devices.rng.enable [ "-device virtio-rng-pci" ] @@ -994,11 +956,8 @@ in "-device virtio-serial-pci" "-device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0" ] - # Passed-through PCI devices. The host cannot use these while the VM - # is running. ++ map (d: "-device vfio-pci,host=${d.address}") pt.pci ++ map (u: "-device usb-host,vendorid=0x${u.vendorId},productid=0x${u.productId}") pt.usb - # Raw escape hatch. Unvalidated - prefer the typed devices.* options. ++ cfg.extraQemuArgs ); @@ -1007,7 +966,6 @@ in Restart = "always"; RestartSec = 5; - # ---- hardening ------------------------------------------------- DevicePolicy = "closed"; DeviceAllow = [ "/dev/kvm rw" @@ -1017,8 +975,7 @@ in ++ lib.optionals anyPci [ "char-vfio rw" ] ++ lib.optionals anyUsb [ "char-usb_device rw" ]; - # VFIO pins the entire guest RAM allocation, so the default memlock - # limit has to go or QEMU fails to start with an obscure error. + # VFIO pins guest RAM; default memlock is too low. LimitMEMLOCK = lib.mkIf anyPci "infinity"; NoNewPrivileges = true; ProtectSystem = "strict"; @@ -1047,16 +1004,13 @@ in "~@resources" "~@obsolete" ]; - # KVM means no JIT, so W^X is safe. If QEMU refuses to start, this is - # the first thing to flip. + # Safe under KVM (no JIT). First flip if QEMU won't start. MemoryDenyWriteExecute = true; CapabilityBoundingSet = [ "" ]; AmbientCapabilities = [ "" ]; }; }; - # TPM 2.0 emulator (swtpm), started before the VM. QEMU connects to its - # control socket in the runtime dir; state persists under its StateDirectory. systemd.services.hermes-vm-swtpm = mkIf cfg.devices.tpm.enable { description = "swtpm TPM emulator for the Hermes VM"; wantedBy = [ "multi-user.target" ]; @@ -1104,7 +1058,6 @@ in ''; }; - # Tap device, created as root, owned by the VM user, bridged by networkd. systemd.services.hermes-vm-tap = { description = "TAP interface for the Hermes VM"; after = [ "systemd-networkd.service" ]; @@ -1123,12 +1076,6 @@ in ''; }; - # ===================================================================== - # Device passthrough - # ===================================================================== - - # The IOMMU is what keeps a passed-through device from reading host - # memory. Without it, PCI passthrough is not a boundary at all. boot.kernelParams = lib.optionals (pt.iommu != null) [ (if pt.iommu == "intel" then "intel_iommu=on" else "amd_iommu=on") "iommu=pt" @@ -1140,12 +1087,10 @@ in "vfio_iommu_type1" ]; - # Claim the device before any host driver gets to it. boot.extraModprobeConfig = lib.optionalString (earlyIds != [ ]) '' options vfio-pci ids=${lib.concatStringsSep "," earlyIds} ''; - # Catch anything that was not early-bound, by address. Idempotent. systemd.services.hermes-vm-vfio = mkIf anyPci { description = "Bind passthrough PCI devices to vfio-pci"; before = [ "hermes-vm.service" ]; @@ -1182,9 +1127,6 @@ in SUBSYSTEM=="usb", ATTR{idVendor}=="${u.vendorId}", ATTR{idProduct}=="${u.productId}", OWNER="${vmUser}", MODE="0660" '') pt.usb; - # ===================================================================== - # Networking - # ===================================================================== networking.useNetworkd = true; systemd.network.netdevs."10-${net.bridge}".netdevConfig = { @@ -1194,10 +1136,7 @@ in systemd.network.networks."10-${net.bridge}" = { matchConfig.Name = net.bridge; - # hostAddress stays on the bridge in both modes: it is the guest's - # service net (host cache, hostAccess). In bridged mode the host's - # static LAN address is added on top of it - the bridge only ever has - # static addresses, never a DHCP lease. + # hostAddress = service net; bridged also adds hostLanAddress (static only). address = [ "${net.hostAddress}/${toString net.prefixLength}" ] @@ -1207,16 +1146,12 @@ in IPv4Forwarding = true; } // lib.optionalAttrs net.bridged { - # Static LAN config for the host: the bridge no longer takes a DHCP - # lease (bridged mode is static-only), so its gateway and DNS come - # from the module options instead. Gateway = net.lanGateway; DNS = if net.lanDns == [ ] then [ net.lanGateway ] else net.lanDns; }; }; - # The host's DNS, now that the bridge cannot learn it from a DHCP lease. - # mkDefault so the operator's own nameservers win if they set any. + # mkDefault: operator nameservers win if set. networking.nameservers = lib.mkIf net.bridged ( lib.mkDefault (if net.lanDns == [ ] then [ net.lanGateway ] else net.lanDns) ); @@ -1226,21 +1161,12 @@ in networkConfig.Bridge = net.bridge; }; - # Bridged mode: the physical uplink becomes a bridge port so the LAN - # shares an L2 domain with the guest. The port must not carry its own - # address config - the bridge owns the LAN address now (static, via - # hostLanAddress; there is no DHCP on the bridge). systemd.network.networks."12-${net.externalInterface}" = lib.mkIf net.bridged { matchConfig.Name = net.externalInterface; networkConfig.Bridge = net.bridge; }; - # The bridge is created with no MAC of its own, so Linux makes it adopt - # the lowest-numbered port's MAC - and the tap is always the first port. - # That is how the host once ended up presenting the VM's MAC on the LAN - # (a duplicate address, since the guest uses the same MAC). Clone the - # uplink's MAC onto the bridge so the host keeps its LAN identity; the - # guest keeps its own tap MAC (net.mac). + # Bridge would inherit tap MAC (first port) and collide with guest — clone uplink MAC. systemd.services.hermes-bridge-mac = lib.mkIf net.bridged { description = "Clone the uplink MAC onto the hermenix bridge"; after = [ @@ -1258,7 +1184,6 @@ in unitConfig.ConditionPathExists = "/sys/class/net/${net.externalInterface}"; script = '' set -euo pipefail - # networkd creates the bridge netdev asynchronously; give it a moment. for _ in 1 2 3 4 5 6 7 8 9 10; do [ -e "/sys/class/net/${net.bridge}" ] && break sleep 0.5 @@ -1268,17 +1193,11 @@ in ${pkgs.iproute2}/bin/ip link set dev ${net.bridge} down ${pkgs.iproute2}/bin/ip link set dev ${net.bridge} address "$mac" ${pkgs.iproute2}/bin/ip link set dev ${net.bridge} up - # Re-apply the bridge's static address config under the new MAC. ${pkgs.systemd}/bin/networkctl reconfigure ${net.bridge} || true ''; }; - # The guest is handed its fixed LAN address (lanAddress) by a dedicated - # DHCP server bound to the bridge, keyed to the guest's MAC. This is the - # guest's ONLY DHCP: its broadcast is blocked from leaving the bridge - # (see the hermenix-bridge table), so the LAN's own DHCP server can - # never race the reservation. DNS is disabled (port=0) - the guest runs - # static QUAD9 resolvers, and this host gets its DNS from lanDns. + # Single-address DHCP on the bridge (MAC-keyed). Broadcasts stay on-bridge. systemd.services.hermes-lan-dhcp = lib.mkIf net.bridged { description = "Static-lease DHCP for the bridged hermenix guest"; after = [ @@ -1309,34 +1228,16 @@ in ''; }; - # ip_forward: use the same sysctl keys networking.nat used (mkOverride 99 - # so they win over any lower-priority defaults elsewhere in the config). + # mkOverride 99: same keys as networking.nat, win over weaker defaults. boot.kernel.sysctl."net.ipv4.conf.all.forwarding" = lib.mkOverride 99 true; boot.kernel.sysctl."net.ipv4.conf.default.forwarding" = lib.mkOverride 99 true; - # Masquerade outbound guest traffic and DNAT published ports inbound. - # We use an explicit nftables table instead of networking.nat so that the - # rules are certain to appear in the prerouting/postrouting hooks. - # - # NOTE: DNAT uses 'iifname != bridge' rather than 'iifname externalInterface'. - # Scoping to a single interface means traffic arriving on any other path - # (Tailscale, a second NIC, WiFi, loopback) silently skips DNAT: the - # packet stays addressed to the host's own IP, falls into the INPUT chain, - # and gets dropped because those ports are not in allowedTCPPorts. - # Excluding only the bridge itself prevents loops from inside the guest - # while covering every legitimate external path. + # Explicit nft nat (not networking.nat). DNAT iifname != bridge so extra paths still match. networking.nftables.tables.hermenix-nat = lib.mkIf (net.externalInterface != null) { family = "ip"; content = if net.bridged then - # Bridged mode: no DNAT (the guest is directly reachable on the - # LAN) and no primary masquerade. Keep one scoped masquerade as a - # fallback: if the guest's DHCP lease lapses it falls back to its - # static default route via the host, and this keeps that path - # working instead of blackholing at the uplink. With a healthy - # lease the guest's traffic is L2-bridged and never reaches this - # chain. Only traffic handed up from the bridge (i.e. the guest's - # service-net traffic) is matched, never the LAN itself. + # No DNAT when bridged. Scoped masquerade covers service-net fallback if DHCP lapses. '' chain postrouting { type nat hook postrouting priority srcnat; @@ -1360,34 +1261,19 @@ in ''; }; - # ===================================================================== - # Bridged mode: bridge-family enforcement. - # - # With the uplink bridged, guest <-> LAN traffic is switched at L2 and - # never traverses the ip-family hooks the NAT-mode rules rely on, so - # bridged mode gets its own table where the tap port is unambiguous. - # lanAccess / allowLan still gate what the guest may reach; inbound - # from the LAN to the guest is open on every port. - # ===================================================================== networking.nftables.tables."hermenix-bridge" = lib.mkIf net.bridged { family = "bridge"; content = '' chain prerouting { type filter hook prerouting priority filter; policy accept; - # Tag anything entering from the guest. ip-family input rules - # (host cache, hostAccess) match this mark. iifname "${net.tapId}" meta mark set ${toString bridgeMark} - # The guest's address is static - served by hermes-lan-dhcp on - # this bridge - so keep its DHCP traffic off the LAN and keep the - # LAN's DHCP answers out of the guest. Only the host-side server - # may talk DHCP to it. + # Guest DHCP stays on-bridge only (no race with LAN DHCP). iifname "${net.tapId}" oifname "${net.externalInterface}" ether type ip ip protocol udp udp sport 68 udp dport 67 drop iifname "${net.externalInterface}" oifname "${net.tapId}" ether type ip ip protocol udp udp sport 67 udp dport 68 drop } chain forward { type filter hook forward priority filter; policy accept; - # Inbound from the LAN to the guest: any device, any port. oifname "${net.tapId}" accept '' + lib.concatMapStrings (s: '' @@ -1417,26 +1303,11 @@ in ''; }; - # ===================================================================== - # Firewall - # - # The guest owns its own kernel and its own config now, so nothing inside - # it constrains it. These rules are the boundary. - # ===================================================================== networking.nftables.enable = true; networking.firewall.enable = true; networking.firewall.filterForward = true; - # The deliberate guest -> host holes: the Nix cache, plus anything listed - # in hostAccess. - # - # NAT mode: the guest always arrives from its fixed service-net address, - # so match address + bridge. Bridged mode: the guest presents either the - # service-net address or its LAN address (served by the bridge's DHCP), - # so the bridge table above tags guest traffic with a netfilter mark as - # it enters the tap - the rules match that mark instead. The tap is - # exclusive to the guest, so the mark is as specific as an address - # match: LAN devices can never inherit these holes. + # Guest→host holes. NAT: guestAddress. Bridged: tap mark (guest may use LAN IP). networking.firewall.extraInputRules = if net.bridged then optionalString cfg.hostCache.enable '' @@ -1453,21 +1324,14 @@ in iifname "${net.bridge}" ip saddr ${net.guestAddress} ${s.proto} dport ${toString s.port} accept '') cfg.hostAccess; - # Order matters here. Allowlists must precede the RFC1918 drop. - # - # These ip-family rules only ever see routed traffic. They are the whole - # story in NAT mode; in bridged mode the guest's LAN traffic is - # L2-bridged instead and is governed by the hermenix-bridge table. + # Allowlists before RFC1918 drop. Bridged LAN is L2 (hermenix-bridge table). networking.firewall.extraForwardRules = '' iifname != "${net.bridge}" accept ct state established,related accept '' - # Guest reaching named services elsewhere on your network. + lib.concatMapStrings (s: '' iifname "${net.bridge}" ip daddr ${s.address} ${s.proto} dport ${toString s.port} accept '') cfg.lanAccess - # Inbound to guest services published on the uplink. NAT mode only; in - # bridged mode the bridge table opens every inbound port instead. + lib.optionalString (!net.bridged) ( lib.concatMapStrings (s: '' oifname "${net.bridge}" ip daddr ${net.guestAddress} ${s.proto} dport ${toString s.guestPort} accept @@ -1493,24 +1357,14 @@ in '' ); - # ===================================================================== - # Host store as a read-only cache for the guest - # ===================================================================== services.nix-serve = mkIf cfg.hostCache.enable { enable = true; package = pkgs.nix-serve-ng; bindAddress = net.hostAddress; inherit (cfg.hostCache) port secretKeyFile; - openFirewall = false; # handled above, scoped to the guest address + openFirewall = false; }; - # ===================================================================== - # Snapshots of the state disk only. - # - # The system disk is not snapshotted on purpose. It is large, it is where - # all the write churn happens, and `hermenix setup` can rebuild it from the flake - # in minutes. Snapshotting it would cost a lot and buy nothing. - # ===================================================================== systemd.services.hermes-vm-snapshot = mkIf cfg.snapshots.enable { description = "Snapshot the Hermes VM state disk"; path = with pkgs; [ @@ -1525,9 +1379,6 @@ in mkdir -p ${snapDir} ${optionalString cfg.disk.quiesce '' - # Flush the guest's filesystems so the snapshot is consistent rather - # than the equivalent of pulling the power. Takes milliseconds; the - # VM keeps running throughout. ${qga "freeze"} trap '${qga "thaw"}' EXIT ''} @@ -1539,8 +1390,7 @@ in trap - EXIT ''} - # Prune oldest first. Keep this list short: every live snapshot - # reintroduces copy-on-write on a nodatacow image. + # Live snapshots reintroduce CoW on nodatacow images — keep retention tight. ls -1d ${snapDir}/state-* 2>/dev/null | sort | head -n -${toString cfg.snapshots.keep} \ | while read -r old; do btrfs subvolume delete "$old"; done ''; @@ -1555,15 +1405,6 @@ in }; }; - # ===================================================================== - # Backups: the guest's state, not its disk. - # - # Pulled file-by-file over the SSH channel that already exists, rather - # than backing up the image. That keeps the backup portable - it restores - # onto any filesystem, or a different machine entirely - and avoids - # loop-mounting a guest-written filesystem on the host, which would mean - # the host kernel parsing data the agent controls. - # ===================================================================== systemd.services.hermes-vm-backup = mkIf cfg.backup.enable { description = "Back up Hermes state"; path = with pkgs; [ @@ -1608,17 +1449,12 @@ in }; environment.systemPackages = [ - # Nushell tab-completion definitions for hermenix. Installed to - # vendor/autoload so nushell picks them up automatically. (pkgs.writeTextFile { name = "hermenix-completions"; destination = "/share/nushell/vendor/autoload/hermenix.nu"; text = builtins.readFile ../nu/completions.nu; }) - # Single consolidated CLI: `hermenix vm *` controls the VM, - # `hermenix ctl *` drives Hermes inside it. Per-host values are - # added on top of the base package via makeWrapper. (pkgs.symlinkJoin { name = "hermenix"; paths = [ pkgs.hermenix ]; diff --git a/nu/ctl/ctl.nu b/nu/ctl/ctl.nu index 151e203..4d44639 100644 --- a/nu/ctl/ctl.nu +++ b/nu/ctl/ctl.nu @@ -1,12 +1,5 @@ -# hermenix ctl — drive Hermes inside the guest. -# -# Identity, config, secrets, gateway service, and a passthrough to the -# hermes CLI itself. For controlling the VM around it, see vm/vm.nu. -# -# Depends on: lib/common.nu, lib/guest.nu (ssh-args). - def home-dir [] { - $env.HERMES_HOME_DIR? | default "/var/lib/hermes-vm/.hermes" + $env.HERMES_HOME_DIR? | default "/var/lib/hermes/.hermes" } def gateway-unit [] { $env.HERMES_GATEWAY_UNIT? | default "hermes-agent" @@ -14,8 +7,6 @@ def gateway-unit [] { def ctl-gssh [cmd: string] { run-argv ((ssh-args) ++ [$cmd]) } -# Pull a file out of the guest, open it in $EDITOR, push it back. -# Skips the write if nothing changed (hash comparison). def edit-remote [remote: string, mode: string = "0644"] { let tmp = (mktemp) let fetched = do -i { ctl-gssh $"cat ($remote)" } | complete @@ -38,13 +29,8 @@ def edit-remote [remote: string, mode: string = "0644"] { } } -# Everything not matched by a more-specific `main ctl *` goes straight to the -# hermes CLI in the guest, with a pty for interactive subcommands like `chat`. -# -# A single string, not rest args: nu's parser tries to match any bare -# --flag-shaped token against this command's own flags before it ever -# reaches a rest parameter, so `hermenix ctl chat --model x` would fail to -# parse. Quote the whole thing instead — e.g. `hermenix ctl "chat --model x"`. +# Single string, not rest args: nu matches bare --flags on this command first. +# Quote multi-word passthroughs: hermenix ctl "chat --model x" def "main ctl" [remainder: string = ""] { if $remainder == "" { print "usage: hermenix ctl " @@ -94,8 +80,6 @@ def "main ctl conf" [action?: string] { } } -# API keys. Mode 0600, never enters the Nix store, and not printed unless -# you ask for it explicitly. def "main ctl secrets" [action?: string] { if $action == "edit" { edit-remote $"(home-dir)/.env" "0600" @@ -111,8 +95,6 @@ def "main ctl logs" [] { run-argv ((ssh-args) ++ ["-t" $"journalctl --user -u (g def "main ctl restart" [] { ctl-gssh $"systemctl --user restart (gateway-unit)" } def "main ctl units" [] { ctl-gssh "systemctl --user list-units 'hermes*' --all --no-pager" } -# Overwrite the agent's identity from the seed. Destructive — this is how -# you undo whatever it has decided about itself. def "main ctl reseed" [] { warn "This overwrites SOUL.md, memories and config.yaml in the guest" warn "with the seed from /etc/nixos/hermes. Anything Hermes has" @@ -122,8 +104,6 @@ def "main ctl reseed" [] { ok "reseeded" } -# Copy the agent's live identity back here, so you can fold what it has -# learned into the seed in guest/hermes/. def "main ctl pull" [dest: string = "./hermes-home"] { mkdir $dest ( diff --git a/nu/hermenix.nu b/nu/hermenix.nu index cb2ca52..30b2ebf 100644 --- a/nu/hermenix.nu +++ b/nu/hermenix.nu @@ -1,29 +1,4 @@ #!/usr/bin/env nu -# -# hermenix — host-side tooling for the Hermes Agent VM. -# -# No local checkout needed. `hermenix` is a flake package meant to be run -# straight from its git+ssh URL: -# -# nix run #hermenix -# -# Or install once and just call it directly: -# -# nix profile install #hermenix (or via the nixos-setup overlay) -# -# Host-specific settings (cache signing key, SSH key) live at -# /etc/hermenix/local.nix — see guest/local.nix.example. -# -# Source layout (all concatenated by flake.nix in dependency order): -# -# lib/common.nu output helpers, run-argv, vsock-ssh-args -# lib/guest.nu shared guest-connection env helpers -# host/setup.nu hermenix setup -# host/migrate.nu hermenix migrate -# host/remote.nu hermenix remote -# vm/vm.nu hermenix vm * -# ctl/ctl.nu hermenix ctl * -# hermenix.nu this file — top-level dispatch (main) def main [] { print "usage: hermenix ..." diff --git a/nu/host/migrate.nu b/nu/host/migrate.nu index 766d63b..e7c9b68 100644 --- a/nu/host/migrate.nu +++ b/nu/host/migrate.nu @@ -1,23 +1,3 @@ -# hermenix migrate — carry an existing flat HERMES_HOME into the guest, -# or selectively copy just the agent-connection files between two folder -# deployments. -# -# Only ever reads from the source directory. Nothing there is deleted or -# moved — the original stays a fully intact fallback. -# -# Run this AFTER setup + the host's nixos-rebuild have brought the guest up -# at least once, and `hermenix vm ssh true` succeeds. -# -# Depends on: lib/common.nu, lib/guest.nu. - -# Carry an existing flat HERMES_HOME (e.g. the nousresearch/hermes-agent -# Docker image, HERMES_HOME bind-mounted from the host) into a hermenix -# guest, preserving its exact layout. -# -# SAFETY: only ever reads from `source`. Nothing there is deleted or moved — -# the original stays a fully intact fallback. If anything goes wrong after -# this, `sudo nixos-rebuild switch --rollback` on the host restores the -# previous deployment pointed at the same, untouched source data. def "main migrate" [ source: string # existing flat HERMES_HOME to copy from (read-only) --vsock-cid: int = 77 # must match services.hermenix.vsockCid on the host @@ -51,22 +31,8 @@ def "main migrate" [ gssh "true" step 2 $"streaming ($source) -> guest:($dest) \(this can take a while for 8G+\)" - # --ignore-failed-read: a source entry can be unreadable (e.g. uid-mapped - # leftovers from a prior rootless-container deployment) without that - # aborting an otherwise-successful migration - tar still warns about it, - # it just doesn't fail the whole pipeline over it. - # - # --exclude='./.local/bin': launcher scripts in .local/bin are - # environment-specific (Docker wrappers, old arch builds, etc.) and must - # never be carried into the guest. hermes-install.service installs the - # correct native binary there on first boot; an overwritten launcher from - # the source would shadow it and fail with "docker not found on PATH". - # - # --exclude='./.hermes-installed': this sentinel tells hermes-install to - # skip the native install. Migrating it from the source would prevent the - # guest from ever installing hermes if the source's .local/bin/hermes is - # also excluded (above). Let the install service run and place a fresh - # binary instead. + # ignore-failed-read: rootless-docker uid leftovers. + # exclude .local/bin + .hermes-installed: don't shadow guest native install. ^tar --ignore-failed-read --exclude='./.local/bin' --exclude='./.hermes-installed' -C $source -cf - . | run-argv ($ssh_base ++ [$"mkdir -p '($dest)' && tar -C '($dest)' -xf -"]) step 3 "rewriting stale hostnames in the guest" @@ -84,30 +50,13 @@ def "main migrate" [ note "Tip: use `hermenix migrate connections ` to carry just" note "the connection config files between two folder deployments." print "" - say "Next, inside the guest (hermenix vm ssh):" - say $" export HERMES_HOME=($dest)" - say " # hermes installs itself automatically on first boot - if that's" - say " # already done (check: hermenix ctl units), just:" - say " # determinate-nixd login" - say " # sudo nix flake lock /etc/nixos" - say " # hermes setup" - say " # hermes gateway install" - print "" - say "Then verify with hermenix ctl status / logs before considering the old" - say "deployment retired. Decrypt secrets/hermes-env.age yourself and fold" - say $"anything relevant into ($dest)/.env \(hermenix ctl secrets edit\)." + say "Next:" + say $" - export HERMES_HOME=($dest) inside the guest (and set services.hermenix.hermesHome)" + say " - finish Hermes configuration with the upstream CLI/docs" + say " - fold secrets into the guest via: hermenix ctl secrets edit" + say " - verify: hermenix ctl status / logs — only then retire the old deployment" } -# Copy only the agent-connection files from one flat HERMES_HOME to another. -# -# Use this when you want to carry the connections/integrations config across -# deployments without overwriting identity (SOUL.md, memories, etc.). -# -# By default this is a pure local operation (host-to-host folder copy). -# Pass --to-guest to stream the files into the running guest over vsock -# instead, using the same path as `hermenix migrate`. -# -# SAFETY: only ever reads from `source`; the original is never modified. def "main migrate connections" [ source: string # existing flat HERMES_HOME to copy from (read-only) dest: string = "" # destination HERMES_HOME folder; not required when --to-guest is set @@ -130,17 +79,13 @@ def "main migrate connections" [ die "--rewrite-from and --rewrite-to must have the same length" } - # ---- resolve what we're actually copying -------------------------------- - # Flat files that exist in source. let existing_files = $files | where {|f| ($"($source)/($f)" | path exists) } - # Subdirectories that exist in source. let existing_dirs = $extra_dirs | where {|d| ($"($source)/($d)" | path exists) } if ($existing_files | is-empty) and ($existing_dirs | is-empty) { die $"none of the requested connection files/dirs were found under ($source)" } - # ---- build the sed expression for hostname rewrites -------------------- let has_rewrites = ($rewrite_from | length) > 0 let sed_expr = if $has_rewrites { let last_idx = ($rewrite_from | length) - 1 @@ -149,7 +94,6 @@ def "main migrate connections" [ | str join "; ") } else { "" } - # ---- announce ----------------------------------------------------------- say $"Source \(read-only\): ($source)" if $to_guest { say $"Destination \(guest\): ($guest_user)@hermes:($guest_dest) \(over vsock, cid ($vsock_cid)\)" @@ -170,9 +114,7 @@ def "main migrate connections" [ print "" if not (confirm-word "copy" "Type 'copy' to proceed:") { die "aborted" } - # ========================================================================= if $to_guest { - # ---- guest mode: stream over vsock ---------------------------------- let ssh_base = (vsock-ssh-args $vsock_cid $guest_user "hermes") def gssh [cmd: string] { run-argv ($ssh_base ++ [$cmd]) } @@ -180,7 +122,6 @@ def "main migrate connections" [ gssh "true" step 2 $"streaming connection files ($source) -> guest:($guest_dest)" - # Pass files and dirs as positional tar arguments — no --include needed. let tar_targets = $existing_files | append $existing_dirs ( ^tar --ignore-failed-read -C $source -cf - ...$tar_targets @@ -198,7 +139,6 @@ def "main migrate connections" [ } } } else { - # ---- local mode: copy on this host ---------------------------------- step 1 "creating destination directory" mkdir $dest diff --git a/nu/host/remote.nu b/nu/host/remote.nu index d35f6bc..3c0f541 100644 --- a/nu/host/remote.nu +++ b/nu/host/remote.nu @@ -1,26 +1,9 @@ -# hermenix remote — run setup or migrate on a remote host over SSH. -# -# No checkout involved anywhere: executes `nix run #hermenix -- ...` -# directly on the remote, fetching straight from Gitea. The remote host needs -# its own SSH access to Gitea already set up (same access as a git clone). -# -# Depends on: lib/common.nu (shquote, say, die). - -# Run `hermenix setup` or `hermenix migrate` on a remote host over SSH, -# instead of having to SSH in and run it locally yourself. `setup` and -# `migrate` both act on whatever machine they run on (local disks, local -# sudo, vsock into the local guest) — this just saves the manual SSH step. +# cmd must be one string: nu treats bare --flags as belonging to `remote`. def "main remote" [ - host: string # SSH target, e.g. fwdesk.lab.internal - cmd: string # subcommand + its own args, quoted as ONE string — e.g. "setup --clean" or "migrate /var/lib/hermes" - --flake: string = "git+ssh://gitea@minipc1.lab.internal:22/ari/hermenix.git" + host: string + cmd: string # e.g. "setup --clean" or "migrate /var/lib/hermes" + --flake: string = "git+ssh://gitea@gitea.skyg.ca/ari/hermenix.git" ] { - # `cmd` has to be a single string, not a rest param: nu's own CLI parser - # treats any bare --flag-shaped token as an attempt to match a flag on - # THIS command (main remote), even inside a rest parameter and even with - # a `--` separator — so `hermenix remote host setup --clean` fails before - # ever reaching this function. Splitting cmd back into words here (and - # re-quoting each one) sidesteps that entirely. let words = $cmd | split row " " | where {|w| $w != "" } if ($words | is-empty) { die $'usage: hermenix remote " [args...]"' diff --git a/nu/host/setup.nu b/nu/host/setup.nu index f37d2ae..f081a8d 100644 --- a/nu/host/setup.nu +++ b/nu/host/setup.nu @@ -1,15 +1,4 @@ -# hermenix setup — guided host-side provisioning. -# -# Walks through every prerequisite check, disk creation, cache signing key, -# local.nix, image build and image write. Safe to re-run; each step checks -# what already exists. Never runs nixos-rebuild. -# -# Depends on: lib/common.nu, lib/guest.nu. - -# `truncate -s` treats a bare number as bytes, not gigabytes — the size -# prompts' own defaults ("80G", "20G") make gigabytes the obvious intent, so -# normalize a bare integer answer to that rather than silently creating a -# byte-sized disk. Anything with an existing suffix passes through as-is. +# Bare `truncate -s N` is bytes; prompts default to gigabytes. def normalize-size [s: string] { try { let n = $s | into int @@ -19,28 +8,20 @@ def normalize-size [s: string] { } } -# Run a command, sudo-prefixed unless already root (set once via -# $env.HERMENIX_NEED_SUDO at the top of `main setup`). Fails loudly, like any -# other external command, if the command fails. def priv [...args: string] { let need_sudo = $env.HERMENIX_NEED_SUDO? | default false let argv = if $need_sudo { ["sudo"] ++ $args } else { $args } ^($argv | first) ...($argv | skip 1) } -# Same as priv, but captures stdout/exit code instead of failing the script. def priv-out [...args: string] { let need_sudo = $env.HERMENIX_NEED_SUDO? | default false let argv = if $need_sudo { ["sudo"] ++ $args } else { $args } (^($argv | first) ...($argv | skip 1) | complete) } -# Guided host-side setup: disks, cache signing key, /etc/hermenix/local.nix, -# build and write the bootstrap image. Safe to re-run — every step checks -# what's already done and offers to skip. Never runs nixos-rebuild. Works -# from any directory - no local checkout of this repo needed. def "main setup" [ - --flake: string = "git+ssh://gitea@minipc1.lab.internal:22/ari/hermenix.git" + --flake: string = "git+ssh://gitea@gitea.skyg.ca/ari/hermenix.git" --clean # stop the VM and delete existing disk images + UEFI vars first, for a fully fresh start ] { $env.HERMENIX_NEED_SUDO = ((^id -u | str trim) != "0") @@ -100,10 +81,7 @@ def "main setup" [ do -i { priv rm "-f" $"($base)/system/system.raw" } | complete | ignore do -i { priv rm "-f" $"($base)/state/state.raw" } | complete | ignore do -i { priv rm "-f" $"($base)/OVMF_VARS.fd" } | complete | ignore - # A fresh state disk means a fresh SSH host key on next boot - drop the - # stale known_hosts entry now so the next connect doesn't fail on a - # host-key mismatch. Not run through priv: this is the invoking user's - # own ~/.ssh/known_hosts, not a root-owned path. + # Fresh state disk => fresh SSH host key; drop stale known_hosts (user's, not root's). do -i { ^ssh-keygen -R hermes } | complete | ignore ok "cleaned existing disk images and stale known_hosts entry" } @@ -375,8 +353,7 @@ def "main setup" [ \}; " note $"detected uplink: ($uplink) — check that's right before rebuilding" - note "network.bridged = true instead puts the guest on your LAN (own DHCP" - note "address, every opened port reachable from the LAN) - see the README." + note "network.bridged = true puts the guest on your LAN — see docs/NETWORK.md." print $" Then: @@ -385,20 +362,18 @@ def "main setup" [ hermenix vm console (ansi dark_gray)# watch it boot(ansi reset) hermenix vm ssh (ansi dark_gray)# get in(ansi reset) - Inside the guest, in this order: + Inside the guest (hermenix side): lsblk -f (ansi dark_gray)# CHECK LABELS match guest/hardware.nix(ansi reset) determinate-nixd login (ansi dark_gray)# FlakeHub Cache(ansi reset) sudo nix flake lock /etc/nixos - (ansi dark_gray)# hermes installs itself automatically - give it a few minutes(ansi reset) - hermes setup - hermes gateway install + (ansi dark_gray)# hermes CLI installs unattended — give it a few minutes(ansi reset) - Back on the host: + Configure Hermes itself with the upstream CLI/docs (providers, gateway, …). + From the host, once the gateway unit exists: - hermenix ctl units (ansi dark_gray)# confirm the gateway unit name(ansi reset) + hermenix ctl units hermenix ctl status - hermenix ctl soul " warn "Check the disk labels before the guest's first nixos-rebuild. If they" diff --git a/nu/lib/common.nu b/nu/lib/common.nu index 9ca18f4..a9ddf8e 100644 --- a/nu/lib/common.nu +++ b/nu/lib/common.nu @@ -1,13 +1,5 @@ -# hermenix — shared helpers for the host-side nushell tools (hermenix, -# hermes-vm, hermes-ctl). -# -# Not a standalone script: nu can't reliably resolve relative `use` imports -# once a script has been copied into the Nix store as a single wrapped -# binary, so the flake concatenates this file with each tool's own script -# instead (see flake.nix / modules/host.nix). Keep this file free of a -# `main` def and of anything specific to one tool. - -# ---------------------------------------------------------------- output --- +# Shared helpers. Concatenated into the hermenix binary by flake.nix +# (relative `use` is unreliable once the script is in the store). def say [msg: string] { print $" ($msg)" } def note [msg: string] { print $" (ansi dark_gray)($msg)(ansi reset)" } @@ -19,11 +11,7 @@ def die [msg: string] { } def step [n: int, title: string] { print $"\n(ansi attr_bold)== ($n). ($title)(ansi reset)" } -# Read a line of user input. On a real terminal this prompts interactively -# via `input`. When stdin isn't a terminal (e.g. piped in from a test or -# script), `input` would otherwise try to read from the controlling terminal -# and hang forever waiting for a keystroke that never comes — so read the -# piped stdin directly instead. +# `input` hangs when stdin is piped (tests); read the pipe instead. def read-line [prompt: string] { if (is-terminal --stdin) { input $prompt @@ -53,16 +41,10 @@ def confirm-word [word: string, prompt: string] { (read-line $" ($prompt) ") == $word } -# Single-quote a string for safe embedding in a remote shell command line. def shquote [s: string] { "'" + ($s | str replace --all "'" "'\\''") + "'" } -# ------------------------------------------------------------------- ssh --- - -# argv for reaching the guest over vsock — no inbound network path needed. -# Shared by every tool that talks to the guest (hermenix migrate, hermes-vm, -# hermes-ctl). def vsock-ssh-args [vsock_cid: int, guest_user: string, guest_host: string] { [ "ssh" @@ -72,17 +54,11 @@ def vsock-ssh-args [vsock_cid: int, guest_user: string, guest_host: string] { ] } -# Run an argv list (e.g. from vsock-ssh-args, with trailing command words -# appended) as an external command. def run-argv [argv: list] { ^($argv | first) ...($argv | skip 1) } -# --------------------------------------------------------------- version --- - -# HERMENIX_VERSION is baked in at build time (flake.nix / modules/host.nix) -# as "YYYY.MM.DD.hh.mm-" from this repo's own git metadata. "dev" -# is only seen running an unwrapped checkout of the script directly. +# Baked in at package build time; "dev" only for an unwrapped checkout. def hermenix-version [] { $env.HERMENIX_VERSION? | default "dev" } diff --git a/nu/lib/guest.nu b/nu/lib/guest.nu index 5732cbb..cb865ba 100644 --- a/nu/lib/guest.nu +++ b/nu/lib/guest.nu @@ -1,7 +1,3 @@ -# Shared guest-connection helpers. -# Depends on: lib/common.nu (vsock-ssh-args). -# Used by both vm/ and ctl/. - def guest-user [] { "hermes" } def guest-host [] { $env.HERMES_GUEST_NAME? | default "hermes" @@ -10,5 +6,4 @@ def vsock-cid [] { ($env.HERMES_VSOCK_CID? | default "77") | into int } -# Full SSH argv for the guest, shared by `vm` and `ctl` subcommands. def ssh-args [] { vsock-ssh-args (vsock-cid) (guest-user) (guest-host) } diff --git a/nu/tests/ctl_test.nu b/nu/tests/ctl_test.nu index b251590..928eb6b 100644 --- a/nu/tests/ctl_test.nu +++ b/nu/tests/ctl_test.nu @@ -5,9 +5,9 @@ source ../lib/guest.nu source ../ctl/ctl.nu @test -def "home-dir defaults to the vm state disk path" [] { +def "home-dir defaults to the guest hermes home path" [] { with-env { HERMES_HOME_DIR: null } { - assert equal (home-dir) "/var/lib/hermes-vm/.hermes" + assert equal (home-dir) "/var/lib/hermes/.hermes" } } diff --git a/nu/vm/vm.nu b/nu/vm/vm.nu index e1ead55..89736ff 100644 --- a/nu/vm/vm.nu +++ b/nu/vm/vm.nu @@ -1,20 +1,9 @@ -# hermenix vm — control the Hermes Agent VM itself. -# -# Start/stop, console, config inspection, snapshots and backups. -# For driving Hermes the agent running inside the VM, see ctl/ctl.nu. -# -# Depends on: lib/common.nu, lib/guest.nu (ssh-args). - def disk-base-dir [] { $env.HERMES_DISK_BASE_DIR? | default "/var/lib/hermes-vm" } def snap-dir [] { $"(disk-base-dir)/snapshots" } -# Run a single already-fully-formed remote command string, or open an -# interactive shell if none is given. A single string, not rest args: nu's -# parser tries to match any bare --flag-shaped token against this command's -# own flags before it ever reaches a rest parameter, so `hermenix vm ssh sudo -# foo --bar` would fail to parse. Quote the whole remote command instead. +# Single string, not rest args: nu matches bare --flags on this command first. def vm-gssh [cmd?: string] { let argv = if $cmd == null { ssh-args } else { (ssh-args) ++ [$cmd] } run-argv $argv @@ -45,7 +34,6 @@ def "main vm start" [] { ^systemctl start hermes-vm } def "main vm stop" [] { ^systemctl stop hermes-vm } def "main vm restart" [] { ^systemctl restart hermes-vm } -# What is the guest actually running right now? def "main vm config" [] { vm-gssh 'echo "== current generation =="; \ readlink -f /run/current-system; \ @@ -57,7 +45,6 @@ def "main vm config" [] { git -C /etc/nixos status --short' } -# Copy the guest's live NixOS config here for reading or diffing. def "main vm config-pull" [dest: string = "./hermes-etc-nixos"] { mkdir $dest (vm-gssh "tar -C /etc/nixos -cf - .") | ^tar -C $dest -xf - @@ -84,8 +71,6 @@ def "main vm snapshots" [] { | print } -# Restore the state disk from a snapshot. The system disk is left alone — -# it holds nothing worth restoring. def "main vm rollback" [snapshot: string] { let snap = $"(snap-dir)/($snapshot)" if not ($snap | path exists) { die $"no such snapshot: ($snapshot)" } diff --git a/tests/network-modes-check.nix b/tests/network-modes-check.nix index 52d845f..472388b 100644 --- a/tests/network-modes-check.nix +++ b/tests/network-modes-check.nix @@ -25,9 +25,7 @@ let nixpkgsPath = pkgs.path; has = lib.strings.hasInfix; - # ------------------------------------------------------------------ # # Host module, evaluated with stub options for what this check reads. # - # ------------------------------------------------------------------ # hostOptions = { lib, ... }: { config._module.check = false; options = { @@ -211,10 +209,8 @@ let brNetworks."11-vm-hermes".networkConfig.Bridge == "hermes-br"; }; - # ------------------------------------------------------------------- # # Guest module: must produce one NIC config that works in both modes. # # Evaluated against the full NixOS option set. # - # ------------------------------------------------------------------- # guest = lib.evalModules { modules = (import "${nixpkgsPath}/nixos/modules/module-list.nix") ++ [ { config._module.check = false; }