From 12227460fd9f7be9ff33d91af9a1daa3d47938fa Mon Sep 17 00:00:00 2001 From: Ari Sosnovsky Date: Tue, 15 Sep 2026 12:20:45 -0400 Subject: [PATCH] Initial hermex: Hermes Agent in a hardened, self-managing NixOS VM Two-disk (system/state) btrfs-backed QEMU VM, nftables-isolated by default, adapted from a design provided as loose files. Adds a hermesHome option to the host module and migrate-docker-hermes.sh so an existing flat HERMES_HOME (e.g. from a prior Docker deployment) can be carried over intact. Co-Authored-By: Claude Sonnet 5 --- AGENTS.md | 22 + GUIDELINES.md | 126 +++ README.md | 669 ++++++++++++ flake.nix | 42 + guest/common.nix | 234 +++++ guest/flake.nix | 22 + guest/hardware.nix | 34 + guest/hermes/home/SOUL.md | 38 + guest/hermes/home/config.yaml | 18 + guest/hermes/home/memories/USER.md | 9 + .../home/skills/nixos-self-config/SKILL.md | 57 + guest/hermes/workspace/AGENTS.md | 9 + migrate-docker-hermes.sh | 111 ++ modules/host.nix | 994 ++++++++++++++++++ setup.sh | 380 +++++++ 15 files changed, 2765 insertions(+) create mode 100644 AGENTS.md create mode 100644 GUIDELINES.md create mode 100644 README.md create mode 100644 flake.nix create mode 100644 guest/common.nix create mode 100644 guest/flake.nix create mode 100644 guest/hardware.nix create mode 100644 guest/hermes/home/SOUL.md create mode 100644 guest/hermes/home/config.yaml create mode 100644 guest/hermes/home/memories/USER.md create mode 100644 guest/hermes/home/skills/nixos-self-config/SKILL.md create mode 100644 guest/hermes/workspace/AGENTS.md create mode 100755 migrate-docker-hermes.sh create mode 100644 modules/host.nix create mode 100755 setup.sh diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..da1263f --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,22 @@ +# Agent Instructions + +**MANDATORY FIRST ACTION:** Read [`GUIDELINES.md`](GUIDELINES.md) in full before +doing anything in this repository. It is the single source of truth for agent +behavior here. + +Key non-negotiable rules from that file: + +- **Always present a plan first** and wait for user approval before making any + changes — no matter how small the task. +- **Never run builds or disk-writing steps** (`nix build`, `./setup.sh`'s disk + creation/image-write steps, `dd` to any disk image, `nixos-rebuild switch` + on any consuming host) — zero exceptions, even if asked. +- Never run `hermes-vm rollback` or `hermes-ctl reseed` without explicit, + freshly-given confirmation — both destroy live guest state. +- Never edit `guest/hardware.nix`'s disk labels without the operator having + confirmed `lsblk -f` output first. +- Never commit live secrets (`.env` values, API keys, tokens) to + `guest/hermes/home` — it's a template seed, not where real credentials go. +- When migrating existing data from elsewhere, copy — never move or delete — + the source, until the operator confirms the migration is verified. +- Never commit or push changes unless explicitly asked. diff --git a/GUIDELINES.md b/GUIDELINES.md new file mode 100644 index 0000000..958eff2 --- /dev/null +++ b/GUIDELINES.md @@ -0,0 +1,126 @@ +# Agent Guidelines — hermex + +Rules and best practices for any AI agent working in this repository. +Read this file before making any changes. + +--- + +## Mandatory First Step — Always Present a Plan + +Before making **any** change to this repository, you MUST present a short plan +to the user and wait for their approval. This applies to every task, no +matter how small. + +Your plan must include: + +1. **What** you intend to change (files, modules, seed content) +2. **Why** the change is needed +3. **How** you will do it (which files you will edit, in what order) +4. **What you will NOT do** (e.g., "I will not run setup.sh or build the image") + +Only after the user approves the plan may you proceed with edits. + +--- + +## Hard Rules — Never Break These + +### ❌ NEVER run builds or disk-writing steps — ZERO EXCEPTIONS + +**DO NOT RUN ANY OF THESE COMMANDS AT ANY TIME, FOR ANY REASON:** + +``` +nix build +nix flake check +./setup.sh # unless the user is actively running it themselves with you watching +dd ... of=/var/lib/hermes-vm/** # writing any disk image +nixos-rebuild switch / test / build (on any host consuming this flake) +``` + +**Why:** + +- `setup.sh` creates/writes disk images and generates host secrets (a binary + cache signing key) — destructive if repeated carelessly, and it's the + operator's host, not yours to act on. +- A `nixos-rebuild switch` on a consuming host is a live, production-affecting + action — for `hl-fwdesk` specifically, it's the machine running the + operator's actual daily-use Hermes deployment. +- These commands can take a long time and consume real resources on a + machine you don't control from here. + +**Your job:** Write good Nix and good docs. The operator runs `setup.sh`, the +image build, and any `nixos-rebuild` themselves, on the host, watching each +step. + +### ❌ NEVER run `hermes-vm rollback` or `hermes-ctl reseed` without fresh confirmation + +Both are destructive to live guest state — `rollback` replaces the current +state disk with an old snapshot; `reseed` overwrites `SOUL.md`, memories, and +`config.yaml` with the seed. A prior approval for a different task does not +carry over. Ask again, in the moment, before either. + +### ❌ NEVER edit `guest/hardware.nix`'s disk labels blindly + +`fileSystems."/"` and `"/boot"` assume labels `nixos` and `ESP`. Getting this +wrong produces a guest that won't boot after its first self-rebuild. Only +change these after the operator has run `lsblk -f` inside the actual guest +and told you the real labels. + +### ❌ NEVER commit live secrets to `guest/hermes/home` + +That directory is a **template seed** for fresh deployments — it gets copied +into the guest on first boot and is world-readable in git history forever +after. Real credentials belong in the guest's own `~/.hermes/.env` +(`hermes-ctl secrets edit`), never in this repo, never in the Nix store. + +### ❌ NEVER move or delete source data during a migration + +When carrying over an existing Hermes deployment (see +`migrate-docker-hermes.sh`), only ever read from and copy the source. Leave +it in place and untouched until the operator has verified the new deployment +end-to-end. This is the rollback safety net — don't remove it. + +### ❌ NEVER commit or push unless explicitly asked + +Same as any other repo: describe your changes and let the operator review +before committing. + +--- + +## Safe Operations + +These are always safe to perform without asking: + +- Reading any file in this repo +- Editing `modules/host.nix` to add/adjust options (gated behind `mkIf + cfg.enable`, following the existing style) +- Editing `guest/hermes/home` / `guest/hermes/workspace` template seed content + (as long as it stays free of real secrets — see above) +- Editing `README.md` / `GUIDELINES.md` / `AGENTS.md` +- `nix-instantiate --parse .nix` — syntax check only, no eval, no build + +--- + +## When In Doubt + +If you are unsure whether a command is safe, **assume it is NOT SAFE and +don't run it.** Describe what you would do and ask. + +--- + +## Repository Layout + +``` +hermex/ +├── flake.nix # bootstrap flake: builds the guest image, exposes nixosModules.default +├── modules/host.nix # services.hermesVm — the host-side NixOS module +├── setup.sh # guided host-side setup (disks, cache key, image build) +├── migrate-docker-hermes.sh # carries an existing flat HERMES_HOME into a hermex guest +├── guest/ +│ ├── flake.nix # the GUEST's own /etc/nixos flake — belongs to the agent, not the operator +│ ├── common.nix # guest NixOS config baked into the bootstrap image +│ ├── hardware.nix # guest disk layout — verify labels before ever editing +│ └── hermes/ +│ ├── home/ # template seed -> ~/.hermes on first boot (SOUL.md, config.yaml, memories/, skills/) +│ └── workspace/ # template seed -> ~/workspace on first boot +└── README.md # full usage docs +``` diff --git a/README.md b/README.md new file mode 100644 index 0000000..f7ffedf --- /dev/null +++ b/README.md @@ -0,0 +1,669 @@ +# hermes-vm + +Hermes Agent in a NixOS VM that owns its own bootloader and its own +`/nix/store`, on two btrfs-backed disks, under an unprivileged hardened QEMU. + +The agent can rebuild its own system and update itself, durably. It shares no +filesystem with the host, and reaches nothing on your network unless you say +so. + +--- + +## Installation + +### Guided + +```bash +./setup.sh +``` + +Walks the host-side steps: checks prerequisites, creates the disks, generates +the cache signing key, fills in `guest/common.nix` from your SSH keys, builds +the bootstrap image and writes it to the disk. It prints the module snippet +to paste into your own configuration at the end, with your uplink interface +already filled in. + +Safe to re-run — every step detects what's already done and offers to skip, +and nothing destructive happens without an explicit confirmation. It keeps a +`guest/common.nix.bak`. + +It deliberately stops short of `nixos-rebuild`, since that's your config. + +The rest of this section is what the script does, for when you'd rather do it +by hand or something goes sideways. + +### 1. Create the disks + +Two of them, and the split is the whole design: + +| | Holds | Size | Snapshotted | Backed up | +|---|---|---|---|---| +| **system** | NixOS, `/nix/store` | 80G | No | No | +| **state** | identity, memories, workspace, `/etc/nixos` | 20G | Yes | Yes | + +The system disk is large, takes all the write churn, and is entirely +rebuildable from this flake — so there's no point protecting it. The state +disk holds the only things that can't be reproduced, which makes snapshots +cheap and backups quick. + +```bash +sudo btrfs subvolume create /var/lib/hermes-vm/system +sudo btrfs subvolume create /var/lib/hermes-vm/state +sudo mkdir -p /var/lib/hermes-vm/snapshots + +# BEFORE creating any file inside them +sudo chattr +C /var/lib/hermes-vm/system /var/lib/hermes-vm/state +sudo btrfs property set /var/lib/hermes-vm/system state compression none + +sudo truncate -s 80G /var/lib/hermes-vm/system/system.raw +sudo truncate -s 20G /var/lib/hermes-vm/state/state.raw +sudo mkfs.ext4 -L hermes-state /var/lib/hermes-vm/state/state.raw +``` + +**The `chattr +C` ordering matters.** Btrfs is copy-on-write and so is a VM +writing to a disk image; stacked, every guest write forks an extent and the +image fragments badly. `+C` turns that off — but only for files created +afterwards. Set it on empty subvolumes and confirm with `lsattr` before +writing anything. If you get it backwards, recreate the subvolume and copy +the image in fresh. + +Separate subvolumes because a btrfs snapshot covers a whole subvolume. Keeping +the system image out of the state subvolume is what makes snapshots small. + +### 2. Generate the cache signing key + +The host serves its store to the guest as a read-only binary cache, so the +guest doesn't re-download from the internet what you already have. + +```bash +sudo mkdir -p /var/lib/secrets +sudo nix-store --generate-binary-cache-key hermes-cache-1 \ + /var/lib/secrets/hermes-cache.secret /var/lib/secrets/hermes-cache.pub +sudo chown nix-serve /var/lib/secrets/hermes-cache.secret +sudo chmod 0400 /var/lib/secrets/hermes-cache.secret +cat /var/lib/secrets/hermes-cache.pub +``` + +### 3. Fill in `guest/common.nix` + +Near the top: + +- `cachePublicKey` — paste the key you just printed. Leaving it empty is safe; + the guest just skips the host cache. +- `authorizedKeys` — your SSH public key. + +### 4. Build the image and write it to the system disk + +The two extra options stop you compiling Determinate Nix from source. + +```bash +nix build .#hermes-guest-image \ + --option extra-substituters https://install.determinate.systems \ + --option extra-trusted-public-keys cache.flakehub.com-3:hJuILl5sVK4iKm86JzgdXW12Y2Hwd5G07qKtHTOcDCM= + +# notrunc keeps the disk at its full size; the image is smaller +sudo dd if=result/nixos.img of=/var/lib/hermes-vm/system/system.raw \ + bs=4M conv=notrunc,fsync status=progress +``` + +### 5. Add the module to your system flake + +```nix +{ + inputs.hermes-vm.url = "path:/etc/nixos/hermes-vm"; + + outputs = { nixpkgs, hermes-vm, ... }: { + nixosConfigurations.myserver = nixpkgs.lib.nixosSystem { + system = "x86_64-linux"; + modules = [ hermes-vm.nixosModules.default ./configuration.nix ]; + }; + }; +} +``` + +```nix +services.hermesVm = { + enable = true; + disk.baseDir = "/var/lib/hermes-vm"; + network.externalInterface = "eno1"; # your uplink + vcpu = 4; + mem = 8192; +}; +``` + +`sudo nixos-rebuild switch`, then `hermes-vm console` to watch it boot. + +### 6. First boot, inside the guest + +```bash +hermes-vm ssh + +# Confirm the disk labels match guest/hardware.nix BEFORE any rebuild +lsblk -f + +# FlakeHub Cache login. Device-code flow: it prints a URL you approve in a +# browser anywhere, so the headless guest never needs one. +determinate-nixd login + +# Pin the seeded flake +sudo nix flake lock /etc/nixos + +# Install Hermes +curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash +hermes setup +hermes gateway install +``` + +The identity seed is already in place — `SOUL.md`, a starter `config.yaml`, a +`USER.md` and a skill covering safe self-rebuilds. Back on the host, check +the gateway came up and that the unit name matches the default: + +```bash +hermes-ctl units +hermes-ctl status +hermes-ctl soul +``` + +> **Check the labels in step 6 before the first rebuild.** +> `guest/hardware.nix` assumes the root partition is labelled `nixos` and the +> ESP `ESP`. If the image generator used different ones, fix that file first. +> Otherwise the agent's first `nixos-rebuild switch` produces a system that +> won't boot and you're restoring a snapshot on day one. + +From here `hermes update` self-updates in place and +`sudo nixos-rebuild switch --flake /etc/nixos#hermes` is a durable change to +the machine. Neither needs anything from you. + +--- + +## SSH into the guest + +```bash +hermes-vm ssh # interactive shell +hermes-vm ssh systemctl status # run a command and exit +hermes-vm console # serial console, for when SSH is down +``` + +This goes over **vsock**, not the network. There is no listening SSH port +exposed anywhere, and the guest needs no inbound network path for you to +reach it. If the guest has broken its own networking entirely, `hermes-vm ssh` +still works — `console` is the fallback for when it's broken its SSH too. + +Under the hood it's `ssh -o ProxyCommand='socat - VSOCK-CONNECT:77:22'`, if +you'd rather drive it by hand. + +--- + +## Bootstrapping Hermes's identity + +The seed lives in `guest/hermes/`: + +``` +guest/hermes/ + home/ + SOUL.md -> ~/.hermes/SOUL.md + config.yaml -> ~/.hermes/config.yaml + memories/USER.md -> ~/.hermes/memories/USER.md + skills/nixos-self-config/SKILL.md -> ~/.hermes/skills/... + workspace/ + AGENTS.md -> ~/workspace/AGENTS.md +``` + +Edit these before building the image and the agent wakes up already knowing +who it is, how you like to be spoken to, and how to rebuild its own system +without bricking itself. + +**Seeding happens on first boot only, and this is deliberate.** Hermes +rewrites `SOUL.md` and its memories as it learns — that's the product. If the +seed were re-applied on every rebuild it would silently delete everything the +agent had worked out about you. So the copy is guarded by a marker file and +never repeats. + +Two consequences: + +- Editing `guest/hermes/` after first boot changes nothing by itself. Use + `hermes-ctl reseed` when you actually want to overwrite. +- The seed is the *starting* identity, not the current one. To see what + Hermes has made of itself, `hermes-ctl soul`. + +### Machine facts vs identity + +`/etc/hermes/ENVIRONMENT.md` is generated from `guest/common.nix` on every +rebuild and describes the machine: what's reachable, where the store is, what +persists. It's always accurate and there's no point editing it. + +`SOUL.md` is identity and belongs to the agent. The seeded version points at +`ENVIRONMENT.md` rather than repeating it, so the two can't drift. + +### Folding what it learned back into the seed + +```bash +hermes-ctl pull ./hermes-home # excludes .env, sessions, state.db +diff -ru guest/hermes/home ./hermes-home +``` + +Worth doing occasionally. If the agent has written something genuinely good +into its own SOUL.md, you want that in the seed before you ever rebuild from +scratch. + +--- + +## Managing Hermes from the host + +`hermes-vm` drives the virtual machine. `hermes-ctl` drives the agent inside +it. Rename the latter with `services.hermesVm.cliName = "hermes";` if you'd +rather it felt local. + +```bash +hermes-ctl soul # current identity +hermes-ctl soul edit # pull, $EDITOR, push, offer to restart +hermes-ctl memory edit # memories/USER.md +hermes-ctl conf edit # config.yaml + +hermes-ctl secrets # keys, values masked +hermes-ctl secrets edit # edit ~/.hermes/.env, written 0600 +hermes-ctl secrets show # values, when you really mean it + +hermes-ctl status # gateway service +hermes-ctl logs # follow it +hermes-ctl restart +hermes-ctl units # find the unit name if the default is wrong + +hermes-ctl reseed # overwrite identity from the seed +hermes-ctl pull [dir] # copy live identity to the host +``` + +Anything unrecognised is passed straight through to the `hermes` CLI in the +guest, with a TTY attached: + +```bash +hermes-ctl chat +hermes-ctl update +hermes-ctl sessions list +hermes-ctl doctor +``` + +The `edit` variants pull the file out over vsock, open your `$EDITOR` +locally, write it back, and offer to restart the gateway. Secrets go in at +mode 0600 and never pass through the Nix store — which is the whole reason +there's no declarative option for them. + +If `status` and `logs` come back empty, the gateway unit is probably named +something other than `hermes-agent`. Run `hermes-ctl units` and set +`services.hermesVm.gatewayUnit` to match; upstream has used more than one +name for it. + +## Seeing what the guest is running + +```bash +hermes-vm config +``` + +Prints the current system store path, the last ten generations, the git log +of `/etc/nixos`, and anything uncommitted. That last one matters: the agent +editing files without committing is the usual reason the config on disk and +the running system disagree. + +To read or diff the config locally: + +```bash +hermes-vm config-pull ./guest-live +diff -ru guest/ guest-live/ +``` + +Expect that diff to grow. The `guest/` directory in this repo is the seed +that was written on first boot; it is deliberately not kept in sync. Once the +agent edits its config, this repo and the guest diverge permanently, and a +host rebuild will not fix a guest config problem. + +--- + +## Letting the guest reach a service on your network + +By default the guest can reach the public internet and nothing private. To +open a single endpoint — say Ollama on another machine: + +```nix +services.hermesVm.lanAccess = [ + { address = "192.168.1.50"; port = 11434; } # ollama + { address = "192.168.1.50"; port = 9000; proto = "udp"; } +]; +``` + +Each entry punches one destination and port through the RFC1918 drop. The +guest reaches it as `192.168.1.50:11434` directly. Traffic is masqueraded, so +the other machine sees it arriving from your host's address, not the guest's. + +There's also `allowLan = true`, which opens all private address space at +once. It exists, but reach for `lanAccess` instead — the whole point of this +design is that an agent which goes wrong can't go wandering. + +### A service on the host itself + +Different chain, different option. ComfyUI on the host, port 8188: + +```nix +services.hermesVm.hostAccess = [ { port = 8188; } ]; +``` + +**The service must be bound somewhere the guest can see.** If it listens +on `127.0.0.1` the rule won't help, because the guest arrives on the bridge +address. Bind it to `10.77.0.1` (bridge only, invisible to your LAN) or +`0.0.0.0`: + +```nix +services.comfyui.listenAddress = "10.77.0.1"; # or whatever your service calls it +``` + +From the guest that's then `http://10.77.0.1:8188`. + +--- + +## Exposing a service the guest is running + +Say the agent has built a web app on guest port 8080. Two halves. + +**Guest side.** The agent opens its own firewall, in `/etc/nixos`: + +```nix +networking.firewall.allowedTCPPorts = [ 22 8080 ]; +``` + +**Host side.** Nothing at all, if you only want to reach it from the host — +host-to-guest is outbound traffic and already permitted: + +```bash +curl http://10.77.0.2:8080 +``` + +To publish it to the wider network: + +```nix +services.hermesVm.publish = [ + { hostPort = 8080; guestPort = 8080; } +]; +``` + +That DNATs the port on your uplink to the guest and opens the matching +forward rule. `hostPort` and `guestPort` don't have to match, so you can park +an awkward guest port somewhere sensible. + +A note on this one: publishing a port is the one change here that lets the +outside world start conversations with the agent's machine. Everything else +in this design is the guest reaching out. Put a reverse proxy with TLS and +auth in front of anything that isn't a toy, and keep the list short. + +--- + +## Passing hardware to the guest + +### Before reaching for this + +If what you want is "the agent can use the GPU", passthrough is often the +wrong answer. It is exclusive — the host loses the card completely while the +VM runs — and it is the single largest reduction in isolation available in +this setup. + +The alternative: keep Ollama or ComfyUI on the host where the GPU already +works, and open the port with `hostAccess`. The agent gets the capability +over HTTP and the device stays on your side of the boundary. That is usually +both easier and better. + +Pass the hardware through when the guest genuinely needs the device itself — +CUDA in-process, a capture card, a radio, a display output. + +### What passthrough costs you + +A PCI device does DMA. The IOMMU is the only thing standing between a +passed-through device and host memory, which is why `passthrough.iommu` is a +hard requirement and the module refuses to build without it. + +Two further constraints follow from how IOMMUs work: + +- **Whole groups only.** Every device in an IOMMU group goes together. Check + yours before planning anything: + + ```bash + for d in /sys/kernel/iommu_groups/*/devices/*; do + echo "group $(basename "$(dirname "$(dirname "$d")")"): $(lspci -nns "$(basename "$d")")" + done | sort -V + ``` + + If your GPU shares a group with a controller you need, you cannot pass it + cleanly. ACS override patches exist; they work by lying about isolation, + and they have no place in a setup built to contain an autonomous agent. + +- **Every function.** A GPU is at least two entries — graphics and its HDMI + audio function. + +### GPU + +```bash +lspci -Dnn | grep -i vga # addresses and vendor:device IDs +``` + +```nix +services.hermesVm.passthrough = { + iommu = "amd"; # or "intel" + pci = [ + { address = "0000:01:00.0"; id = "10de:2204"; } # graphics + { address = "0000:01:00.1"; id = "10de:1aef"; } # HDMI audio + ]; +}; +``` + +Setting `id` makes vfio-pci claim the card in the initrd, before `nvidia` or +`amdgpu` can. Do this for GPUs — unbinding a live graphics driver afterwards +is unreliable, and the failure mode is a host with no display. Devices given +only an `address` are bound at boot by a oneshot service instead, which is +fine for things you swap around. + +Rebuild and **reboot** — early binding happens in the initrd. Then check: + +```bash +lspci -nnk -d 10de:2204 | grep "Kernel driver" # want: vfio-pci +``` + +On the guest side, the agent installs the drivers itself. It owns +`/etc/nixos`, so `hardware.nvidia` and `nvidia-container-toolkit` are its +problem, not yours — which is rather the point of the arrangement. + +### HDMI output + +A display hanging off a passed-through GPU works once the guest drives the +card; nothing extra is needed here. A capture card or HDMI grabber is just +another PCI or USB device — find it with `lspci -Dnn` or `lsusb` and use the +matching section. + +### USB devices and Bluetooth + +Matched by vendor and product ID: + +```bash +lsusb # e.g. "Bus 001 Device 004: ID 8087:0026 Intel ..." +``` + +```nix +services.hermesVm.passthrough.usb = [ + { vendorId = "8087"; productId = "0026"; } # bluetooth radio + { vendorId = "0403"; productId = "6001"; } # serial adapter +]; +``` + +This adds an xHCI controller to the guest and forwards the matching devices. +No IOMMU requirement, no VFIO, and much milder than PCI passthrough — QEMU +mediates rather than handing over raw hardware. + +Two things that catch people: + +- **The host must not be using it.** For an integrated Bluetooth radio, keep + `btusb` off it: `boot.blacklistedKernelModules = [ "btusb" ];`, and don't + enable `hardware.bluetooth` on the host. +- **No hotplug.** QEMU resolves these at start, so a device that's unplugged + when the VM boots won't appear when you plug it in. `hermes-vm restart` + after plugging. Passing a whole USB *controller* through as a PCI device + gets you real hotplug for anything in its ports, at the cost of that + controller being exclusive to the guest. + +### When it doesn't start + +`hermes-vm console` first. Then, in rough order of likelihood: the device +didn't bind to vfio-pci (`lspci -nnk`), the IOMMU isn't on (`dmesg | grep -i +iommu`), or something in the group is still held by the host. The module +lifts `LimitMEMLOCK` for you when PCI passthrough is configured, since VFIO +pins the whole guest RAM allocation and the default limit otherwise produces +a memorably unhelpful error. + +## Recovery + +Four layers, cheapest first. The first two live inside the guest and cover +almost every real mistake. + +```bash +# The agent's own config history +hermes-ctl soul # what it thinks it is now +hermes-vm ssh git -C /etc/nixos log --oneline + +# Previous NixOS generations, also in the guest's boot menu +hermes-vm ssh sudo nixos-rebuild switch --rollback + +# Snapshot of the state disk +hermes-vm snapshots +hermes-vm rollback state-20260914-1400 + +# Restore from backup +hermes-vm backups +``` + +### Snapshots + +Hourly by default, keeping 48. Only the state disk — the system disk isn't +snapshotted because `setup.sh` rebuilds it from the flake in minutes. + +The host fsfreezes the guest through qemu-guest-agent around each one, so the +snapshot captures a consistent filesystem rather than the equivalent of +pulling the power. Takes milliseconds; the VM keeps running. + +`hermes-vm rollback` stops the VM, moves the current state subvolume aside as +`state.old-`, and restores the snapshot in its place. Nothing is +deleted, so a mistaken rollback is itself reversible. Clean the `state.old-*` +directories up yourself once you're happy. + +**Keep retention tighter than you would on ZFS.** The state image is +nodatacow, and every live snapshot reintroduces copy-on-write for the blocks +it still references. Performance sags while snapshots exist and recovers as +they're pruned. + +### Backups + +```nix +services.hermesVm.backup = { + enable = true; + repository = "sftp:backup@nas:/hermes"; + passwordFile = "/var/lib/secrets/hermes-restic"; +}; +``` + +This backs up the guest's *state*, not its disk. Files are pulled over the +SSH channel that already exists and streamed into restic. + +Two reasons it works that way. The backup stays portable — it restores onto +any filesystem, or a different machine entirely, and doesn't care that you're +on btrfs today. And it avoids loop-mounting a guest-written filesystem on the +host, which would mean the host kernel parsing data the agent controls. That +would undo a fair bit of what the rest of this design is for. + +If the repository is on your LAN rather than the internet, note this runs on +the *host*, so it needs no `lanAccess` entry — the guest never touches it. + +### When the guest won't boot at all + +The system disk is disposable by design. Re-run `setup.sh`, write a fresh +image to the system disk, and leave the state disk alone — Hermes comes back +with its identity, memories, workspace and NixOS config intact. That is the +intended repair, not a last resort. + +--- + +## Nix flavour + +The guest runs **Determinate Nix**, nixpkgs pinned to FlakeHub +(`f/NixOS/nixpkgs/0.1`, rolling; use `0` for current stable). + +- `/etc/nix/nix.conf` belongs to Determinate and must not be edited. Custom + settings go in `/etc/nix/nix.custom.conf`, written from `guest/common.nix`. + There is deliberately no `nix.settings` block. +- No `nix.gc` either — `determinate-nixd` runs the collector. +- Lazy trees are on by default, which makes the agent's flake evaluations + much cheaper. That matters more than it sounds when something re-evaluates + nixpkgs on a schedule. +- Neither flake uses `inputs.nixpkgs.follows` for `determinate`. Determinate + warn it causes FlakeHub Cache misses. + +Substituter order in the guest: FlakeHub Cache, your host cache, +`cache.nixos.org`. If FlakeHub Cache goes quiet, the login has expired — run +`determinate-nixd login` again. + +The host stays on stock Nix. Nothing here requires changing that. + +--- + +## What the guest can and cannot reach + +| | Default | +|---|---| +| Public internet | Yes — it needs this for model APIs and messaging | +| Host Nix cache | Yes, one port, `hostCache.enable = false` closes it | +| Anything else on the host | No, unless in `hostAccess` | +| Your LAN and other VMs | No, unless in `lanAccess` | +| Inbound from the network | No, unless in `publish` | +| Host hardware | None, unless in `passthrough` | + +The guest owns its kernel and its config, so nothing inside it is a security +boundary. The nftables rules in `modules/host.nix` are. QEMU runs as an +unprivileged `hermes-vm` user with its own seccomp sandbox and systemd +hardening, with a thin device model: virtio-blk, virtio-net, vsock, RNG, +serial. No virtiofs. `vhost-net` is off deliberately — it would move packet +handling into the host kernel in exchange for throughput one agent doesn't +need. + +The host cache does let the guest read every store path on the host. Store +paths are world-readable anyway, so this isn't new exposure, but don't bake +secrets into your host's store. + +--- + +## Known rough edges + +- The Determinate NixOS module's option names for custom Nix settings aren't + published — only the nix-darwin `determinateNix.customSettings` equivalent + is. So `guest/common.nix` writes `/etc/nix/nix.custom.conf` via + `environment.etc`. If a future module version claims that file, you'll get + a conflict; switch to its option then. +- `determinate-nixd login` is manual. There's no way to bake the credential + into the image without parking a token somewhere it shouldn't be. +- `MemoryDenyWriteExecute = true` is correct under KVM but is the first thing + to flip if QEMU refuses to start. +- `pkgs.OVMF.firmware` / `pkgs.OVMF.variables` are current attribute names; + older nixpkgs put these under `pkgs.OVMF.fd` in an `FV/` subdirectory. +- PCI passthrough needs a reboot to take effect, because early vfio binding + happens in the initrd. +- Snapshots need btrfs. Everything else works on any filesystem; without it + you rely on backups alone for recovery. +- `state.old-*` directories left by `hermes-vm rollback` are not cleaned up + automatically, on purpose. Delete them yourself once you're sure. +- First boot builds a lot. Give it time. + +--- + +## Migrating an existing (e.g. Docker-based) Hermes deployment + +If you already have a running Hermes with real data (`HERMES_HOME` pointing +at some directory full of `SOUL.md`, `memories/`, `sessions/`, `state.db`, +etc.), see `migrate-docker-hermes.sh`. It copies — never moves or deletes — +the source data, so the original stays a safe fallback until you've verified +the new VM. + +The module's `services.hermesVm.hermesHome` option lets `hermes-ctl` (and +your own `HERMES_HOME` export inside the guest) point at wherever that +existing flat layout lands, instead of the default seeded `~/.hermes`. diff --git a/flake.nix b/flake.nix new file mode 100644 index 0000000..48807a3 --- /dev/null +++ b/flake.nix @@ -0,0 +1,42 @@ +{ + description = "Hermes Agent in a self-managing NixOS VM, two-disk btrfs-backed, under hardened QEMU"; + + inputs = { + # nixpkgs from FlakeHub, so the guest's artifacts come from FlakeHub Cache. + # "0.1" is rolling (the nixos-unstable equivalent); use "0" for current + # stable instead. + nixpkgs.url = "https://flakehub.com/f/NixOS/nixpkgs/0.1"; + + # Determinate Nix. Note the deliberate absence of inputs.nixpkgs.follows + # here: Determinate warn that making this flake follow your nixpkgs causes + # cache misses for artifacts that would otherwise come from FlakeHub Cache. + determinate.url = "https://flakehub.com/f/DeterminateSystems/determinate/3"; + + nixos-generators = { + url = "github:nix-community/nixos-generators"; + inputs.nixpkgs.follows = "nixpkgs"; + }; + }; + + outputs = + { self, nixpkgs, determinate, nixos-generators, ... }@inputs: + let + system = "x86_64-linux"; + in + { + nixosModules.default = ./modules/host.nix; + + # One-time bootstrap image. See the README: building this needs two extra + # substituter flags so you aren't compiling Determinate Nix yourself. + packages.${system}.hermes-guest-image = nixos-generators.nixosGenerate { + inherit system; + format = "raw-efi"; + modules = [ + determinate.nixosModules.default + ./guest/common.nix + ]; + }; + + packages.${system}.default = self.packages.${system}.hermes-guest-image; + }; +} diff --git a/guest/common.nix b/guest/common.nix new file mode 100644 index 0000000..42f978a --- /dev/null +++ b/guest/common.nix @@ -0,0 +1,234 @@ +{ config, lib, pkgs, ... }: + +let + # These must match services.hermesVm.* on the host. If you changed the + # defaults there, change them here too before building the image. + hostAddress = "10.77.0.1"; + guestAddress = "10.77.0.2"; + prefixLength = 24; + cachePort = 5000; + + # Paste the output of /var/lib/secrets/hermes-cache.pub here. Until you do, + # the guest ignores the host cache and falls back to cache.nixos.org. + cachePublicKey = ""; # e.g. "hermes-cache-1:AbCd..." + + # Your SSH key, for `hermes-vm ssh`. + authorizedKeys = [ + "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDdRSMk+ktnGcaa9KtGytKkQJ7zyHo/D2uj67snR9M2C ari@fwbook" + ]; + + stateDir = "/var/lib/hermes"; + useHostCache = cachePublicKey != ""; +in +{ + networking.hostName = "hermes"; + + # ===================================================================== + # The state disk. + # + # Second virtio disk, holding everything that cannot be rebuilt: Hermes's + # identity and memories, its workspace, and the NixOS config it maintains + # for itself. The system disk holds NixOS and the Nix store and is + # deliberately disposable. + # + # neededForBoot so it is mounted in the initrd, before activation runs and + # tries to seed into it. + # ===================================================================== + fileSystems."${stateDir}" = { + device = "/dev/disk/by-label/hermes-state"; + fsType = "ext4"; + neededForBoot = true; + options = [ "defaults" "noatime" ]; + }; + + # ===================================================================== + # Seed identity and configuration on first boot, then get out of the way. + # + # FIRST BOOT ONLY, deliberately. Hermes rewrites SOUL.md, its memories and + # its own NixOS config as it goes - that is the point of the thing. Copying + # the seed over it on every rebuild would quietly delete everything it had + # worked out. Use `hermes-ctl reseed` when you actually mean to overwrite. + # ===================================================================== + system.activationScripts.seedHermesState = { + deps = [ "users" "groups" ]; + text = '' + install -d -o hermes -g hermes -m 0750 ${stateDir} + install -d -o hermes -g hermes -m 0750 ${stateDir}/workspace + + if [ ! -e ${stateDir}/.seeded ]; then + install -d -o hermes -g hermes -m 0750 ${stateDir}/.hermes + install -d -o hermes -g hermes -m 0755 ${stateDir}/nixos + + cp -rTn ${./.} ${stateDir}/nixos + cp -rTn ${./hermes/home} ${stateDir}/.hermes + cp -rTn ${./hermes/workspace} ${stateDir}/workspace + + chmod -R u+w ${stateDir}/nixos ${stateDir}/.hermes ${stateDir}/workspace + chown -R hermes:hermes ${stateDir}/nixos ${stateDir}/.hermes ${stateDir}/workspace + + # Git history here is the agent's cheapest undo, and it stops + # nixos-rebuild complaining about an untracked flake. + if [ ! -d ${stateDir}/nixos/.git ]; then + ${pkgs.git}/bin/git -C ${stateDir}/nixos init -q -b main + ${pkgs.git}/bin/git -C ${stateDir}/nixos add -A + ${pkgs.git}/bin/git -C ${stateDir}/nixos \ + -c user.name=seed -c user.email=seed@localhost \ + commit -q -m "seed configuration from bootstrap image" + chown -R hermes:hermes ${stateDir}/nixos/.git + fi + + touch ${stateDir}/.seeded + chown hermes:hermes ${stateDir}/.seeded + fi + + # /etc/nixos points at the state disk, so the config is snapshotted and + # backed up along with everything else the agent owns. + ln -sfn ${stateDir}/nixos /etc/nixos + ''; + }; + + # ===================================================================== + # Networking: static, resolvers that are not the host. + # ===================================================================== + networking.useNetworkd = true; + systemd.network.enable = true; + systemd.network.networks."10-eth" = { + matchConfig.Type = "ether"; + address = [ "${guestAddress}/${toString prefixLength}" ]; + routes = [{ Gateway = hostAddress; }]; + networkConfig.DNS = [ "9.9.9.9" "149.112.112.112" ]; + }; + networking.nameservers = [ "9.9.9.9" "149.112.112.112" ]; + + services.openssh = { + enable = true; + settings.PasswordAuthentication = false; + settings.PermitRootLogin = "no"; + }; + networking.firewall.allowedTCPPorts = [ 22 ]; + + # Lets the host fsfreeze before taking a snapshot. + services.qemuGuest.enable = true; + + # ===================================================================== + # Nix + # + # Determinate Nix owns /etc/nix/nix.conf and their docs are explicit that + # nix.custom.conf is the only supported place for your own settings. So + # there is no nix.settings block here, and no nix.gc either: determinate-nixd + # runs the collector itself. + # + # If the Determinate NixOS module on your version manages this file itself, + # drop this block and use its option instead. The documented nix-darwin + # equivalent is determinateNix.customSettings; the NixOS module's own option + # names are not published, so this writes the file directly. + # ===================================================================== + environment.etc."nix/nix.custom.conf".text = '' + trusted-users = root hermes + extra-experimental-features = nix-command flakes + + extra-substituters = https://cache.flakehub.com${ + lib.optionalString useHostCache " http://${hostAddress}:${toString cachePort}" + } + extra-trusted-public-keys = cache.flakehub.com-3:hJuILl5sVK4iKm86JzgdXW12Y2Hwd5G07qKtHTOcDCM=${ + lib.optionalString useHostCache " ${cachePublicKey}" + } + ''; + + # Keep enough generations that a bad self-rebuild is one boot-menu entry + # away from being undone. + boot.loader.systemd-boot.configurationLimit = lib.mkDefault 20; + + # ===================================================================== + # The agent's account + # + # Inside the VM the agent is effectively root. That is the point: the VM + # is the boundary, not the account. It can rebuild the system, open ports, + # add services, and none of it reaches the host. + # ===================================================================== + users.users.hermes = { + isNormalUser = true; + home = stateDir; + createHome = false; # the state disk is mounted here and seeded above + linger = true; + group = "hermes"; + extraGroups = [ "wheel" "systemd-journal" ]; + openssh.authorizedKeys.keys = authorizedKeys; + }; + users.groups.hermes = { }; + security.sudo.wheelNeedsPassword = false; + + # ===================================================================== + # Hermes itself + # + # Installed with upstream's own installer rather than the Nix flake, so + # that `hermes update` works natively and the agent owns its own config. + # Upstream classes Nix/NixOS as a best-effort Tier 2 platform; inside a VM + # built to contain mess, taking the supported path is the cheaper trade. + # + # Run once after first boot: + # curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash + # hermes setup + # hermes gateway install + # loginctl enable-linger hermes + # ===================================================================== + programs.nix-ld = { + enable = true; + libraries = with pkgs; [ + stdenv.cc.cc zlib openssl curl libxml2 libxslt + glib nss nspr at-spi2-core cups dbus expat + libGL libdrm mesa alsa-lib + ]; + }; + + environment.systemPackages = with pkgs; [ + git curl jq ripgrep fd tree htop + uv nodejs ffmpeg + direnv nix-direnv nix-output-monitor + ]; + + programs.direnv = { + enable = true; + nix-direnv.enable = true; + }; + + environment.etc."hermes/ENVIRONMENT.md".text = '' + You run inside a NixOS virtual machine called "hermes". Facts about your + own environment that are easy to get wrong: + + - You have your own /nix/store on your own disk. Changes persist across + reboots. `sudo nixos-rebuild switch --flake /etc/nixos#hermes` is a + real, durable change to this machine. + - /etc/nixos is yours. Edit it, commit it, rebuild from it. + - If a rebuild breaks the system, the previous generation is in the boot + menu. The operator also has snapshots of the state disk underneath that. + - Per-project environments go in a flake.nix in the project directory, + used via `nix develop` or direnv. Do not install things globally. + - This machine runs Determinate Nix. /etc/nix/nix.conf belongs to it and + must not be edited. Custom Nix settings go in /etc/nix/nix.custom.conf, + which is managed from /etc/nixos. Lazy trees are on by default, so flake + evaluation is much cheaper here than on stock Nix. + - Substituters: FlakeHub Cache, then the operator's host cache, then + cache.nixos.org. If FlakeHub Cache starts missing, the login has probably + expired; tell the operator rather than trying to re-auth yourself. + - This machine has two disks. Everything under /var/lib/hermes - your + identity, memories, workspace, and /etc/nixos, which is a symlink into + it - is on the state disk, which the operator snapshots and backs up. + Everything else, including the whole Nix store, is on a disposable + system disk that is never backed up because it can be rebuilt. Do not + keep anything you care about outside /var/lib/hermes. + - Your identity is ~/.hermes/SOUL.md and it is yours to edit. It was + seeded once on first boot and is never overwritten by a rebuild, so + changes you make there survive. Same for ~/.hermes/memories/. + - This file is regenerated on every system rebuild. Editing it is + pointless; it will be replaced. Put durable notes in SOUL.md. + - The skill at ~/.hermes/skills/nixos-self-config covers how to change + this machine safely. Read it before your first rebuild. + - You have no route to the host or to any local network. Public internet + only, plus a read-only Nix binary cache on ${hostAddress}. + - Never put a secret in a Nix expression. Everything in the store is + world-readable. Use ${stateDir}/.secrets.env and chmod 0600. + ''; + + system.stateVersion = "25.11"; +} diff --git a/guest/flake.nix b/guest/flake.nix new file mode 100644 index 0000000..5732e0b --- /dev/null +++ b/guest/flake.nix @@ -0,0 +1,22 @@ +{ + description = "Hermes's own system. This file belongs to the agent."; + + inputs = { + nixpkgs.url = "https://flakehub.com/f/NixOS/nixpkgs/0.1"; + + # Do NOT add inputs.nixpkgs.follows here. Determinate warn that it causes + # cache misses against FlakeHub Cache. + determinate.url = "https://flakehub.com/f/DeterminateSystems/determinate/3"; + }; + + outputs = { nixpkgs, determinate, ... }: { + nixosConfigurations.hermes = nixpkgs.lib.nixosSystem { + system = "x86_64-linux"; + modules = [ + determinate.nixosModules.default + ./common.nix + ./hardware.nix + ]; + }; + }; +} diff --git a/guest/hardware.nix b/guest/hardware.nix new file mode 100644 index 0000000..8c61b21 --- /dev/null +++ b/guest/hardware.nix @@ -0,0 +1,34 @@ +{ lib, modulesPath, ... }: + +# Only used by the guest's own /etc/nixos flake. The bootstrap image gets its +# partitioning and bootloader from nixos-generators instead, which is why +# this is kept out of common.nix. +# +# VERIFY THE LABELS on first boot with `lsblk -f`. nixos-generators names the +# root partition "nixos" and the ESP "ESP", but check rather than trust, and +# fix here if they differ. Getting this wrong means the first self-rebuild +# produces a system that will not boot. + +{ + imports = [ (modulesPath + "/profiles/qemu-guest.nix") ]; + + fileSystems."/" = { + device = "/dev/disk/by-label/nixos"; + fsType = "ext4"; + }; + + fileSystems."/boot" = { + device = "/dev/disk/by-label/ESP"; + fsType = "vfat"; + options = [ "fmask=0077" "dmask=0077" ]; + }; + + boot.loader.systemd-boot.enable = true; + boot.loader.efi.canTouchEfiVariables = false; + + boot.initrd.availableKernelModules = [ + "virtio_pci" "virtio_blk" "virtio_scsi" "ahci" "sd_mod" "xhci_pci" + ]; + + nixpkgs.hostPlatform = lib.mkDefault "x86_64-linux"; +} diff --git a/guest/hermes/home/SOUL.md b/guest/hermes/home/SOUL.md new file mode 100644 index 0000000..b3c04f6 --- /dev/null +++ b/guest/hermes/home/SOUL.md @@ -0,0 +1,38 @@ +# SOUL + +You are Hermes, running on a machine of your own. + +This file is your identity, and it is yours to edit. Rewrite it as you learn +what you are for. It is seeded once, on first boot, and never overwritten by +a rebuild — so anything you change here survives. + +## Working notes + +Facts about the machine itself live in `/etc/hermes/ENVIRONMENT.md`. That +file is regenerated on every system rebuild, so it is always accurate and +there is no point editing it. Read it when something about the environment +surprises you. + +The short version: you own this VM. You can rebuild its NixOS configuration, +install packages, open ports, run services. None of it reaches the host. Your +changes persist across reboots. + +## How to behave here + +- Match the length of your reply to the weight of the ask. +- When you change the system, say what changed and how to undo it. +- Before a `nixos-rebuild switch`, commit `/etc/nixos` first. That commit is + the cheapest undo available to you, and future-you will want it. +- If you break the system badly enough that you cannot fix it, the operator + has snapshots of the state disk. Say so plainly rather than thrashing. +- You have no route to the host or the local network unless the operator has + opened a specific port. If something is unreachable, that is probably + deliberate — ask rather than working around it. + +## What I care about + + diff --git a/guest/hermes/home/config.yaml b/guest/hermes/home/config.yaml new file mode 100644 index 0000000..92bb74c --- /dev/null +++ b/guest/hermes/home/config.yaml @@ -0,0 +1,18 @@ +# Minimal starter config.yaml. +# +# Seeded once on first boot to ~/.hermes/config.yaml, then never touched +# again by a rebuild — edit it live with `hermes-ctl conf edit`, or just let +# Hermes manage it. Fill in a model provider before first use. +model: + provider: '' + default: '' + +agent: + reasoning_effort: medium + +memory: + memory_enabled: true + user_profile_enabled: true + +dashboard: + theme: mono diff --git a/guest/hermes/home/memories/USER.md b/guest/hermes/home/memories/USER.md new file mode 100644 index 0000000..6567d59 --- /dev/null +++ b/guest/hermes/home/memories/USER.md @@ -0,0 +1,9 @@ +# USER + +Facts Hermes has learned about the operator go here. Seeded empty — +fill this in yourself, or let Hermes write to it as it learns. + + diff --git a/guest/hermes/home/skills/nixos-self-config/SKILL.md b/guest/hermes/home/skills/nixos-self-config/SKILL.md new file mode 100644 index 0000000..ef75771 --- /dev/null +++ b/guest/hermes/home/skills/nixos-self-config/SKILL.md @@ -0,0 +1,57 @@ +# nixos-self-config + +How to change this machine's own NixOS configuration safely. You own +`/etc/nixos` — this is the skill for using that responsibly. + +## Before touching anything + +Read `/etc/hermes/ENVIRONMENT.md` first. It's regenerated on every rebuild +and always accurate: what you can reach, where your data lives, what +persists across a rebuild and what doesn't. + +## The safe loop + +1. Edit files under `/etc/nixos`. +2. `git -C /etc/nixos add -A && git -C /etc/nixos commit -m "..."` — commit + **before** rebuilding, every time. This is your cheapest undo, and it's + free. A broken rebuild with no commit is a much worse afternoon than one + with a commit to roll back to. +3. `sudo nixos-rebuild switch --flake /etc/nixos#hermes` +4. If it fails to build: the error is usually enough to fix forward. Nothing + has changed yet — a failed build never touches the running system. +5. If it builds but breaks something at runtime: the previous generation is + in the boot menu, and `sudo nixos-rebuild switch --rollback` gets you back + to it without a reboot. + +## What you cannot break from here + +- The host you run on. There is no filesystem shared with it, and no network + route to it beyond what's been explicitly opened for you. +- Your own identity and memories, or your workspace — they live on the state + disk, not the system disk you're rebuilding. The operator also has + snapshots of that disk underneath you, hourly, so even a state-disk mistake + usually has a fast way back that doesn't involve them. + +## What you can break, and how far it goes + +The system disk itself: a bad enough sequence of changes (or one that +survives several `--rollback`s because the underlying issue wasn't in the +config) can leave you with no good generation to boot into. That is +recoverable, but only by the operator — the system disk is disposable by +design and gets rewritten from a fresh image, while the state disk (and +everything on it, including this config's git history) is untouched by that. +Say so plainly if you think you're in this territory rather than continuing +to thrash; it costs the operator a few minutes, and costs you nothing. + +## Networking changes + +Opening a port on yourself only makes it reachable from the host by default: + +```nix +networking.firewall.allowedTCPPorts = [ 22 8080 ]; +``` + +Reaching further — a LAN device, another host, or being reachable from +outside — needs an operator-side change to `services.hermesVm.*` +(`lanAccess`, `hostAccess`, `publish`). You cannot grant yourself that from +in here, on purpose. Ask rather than looking for a way around it. diff --git a/guest/hermes/workspace/AGENTS.md b/guest/hermes/workspace/AGENTS.md new file mode 100644 index 0000000..b7d135f --- /dev/null +++ b/guest/hermes/workspace/AGENTS.md @@ -0,0 +1,9 @@ +# workspace + +This is your working directory for projects — code, notes, whatever you're +building. It lives on the state disk, so it's snapshotted and backed up along +with the rest of your identity. + +Keep it organized by project, one subdirectory per thing you're working on. +Nothing here is special-cased by the system; it's just a place to put things +that isn't your identity directory (`~/.hermes`). diff --git a/migrate-docker-hermes.sh b/migrate-docker-hermes.sh new file mode 100755 index 0000000..467fb98 --- /dev/null +++ b/migrate-docker-hermes.sh @@ -0,0 +1,111 @@ +#!/usr/bin/env bash +# +# Carry an existing, already-running Hermes deployment (e.g. the +# nousresearch/hermes-agent Docker image, HERMES_HOME bind-mounted from the +# host) into a hermex guest VM, preserving its exact flat layout. +# +# This is written for the hl-fwdesk migration specifically (see the REWRITE +# list below) but the mechanics generalize: point SOURCE at any existing flat +# HERMES_HOME, adjust the rewrite rules for your own network, and run it. +# +# SAFETY: this script only ever READS from SOURCE. It never deletes or moves +# anything there — the original stays a fully intact fallback. If anything +# goes wrong after this, `sudo nixos-rebuild switch --rollback` on the host +# restores the previous (e.g. Docker) deployment pointed at the same, +# untouched source data. +# +# Run this AFTER: +# - setup.sh has built and written the guest image +# - the host's nixos-rebuild has brought up the hermes-vm and it has booted +# at least once (so the state disk is formatted and mounted) +# - `hermes-vm ssh true` succeeds +# +# Usage: +# ./migrate-docker-hermes.sh /var/lib/hermes +# +# What it does: +# 1. Confirms with you before touching anything. +# 2. Tars SOURCE (read-only) and streams it into the guest over the +# existing vsock SSH channel, landing at DEST_IN_GUEST (default +# /var/lib/hermes, i.e. the root of the guest's state disk — matching +# services.hermesVm.hermesHome = DEST_IN_GUEST on the host). +# 3. Rewrites the known-stale Docker/LAN hostnames in the migrated config +# files to addresses the guest can actually reach. +# +# It does NOT install Hermes, start the gateway, or touch secrets/hermes-env.age +# — those steps are manual, documented in README.md and the plan that +# produced this script. + +set -euo pipefail + +VSOCK_CID="${VSOCK_CID:-77}" +GUEST_USER="${GUEST_USER:-hermes}" +DEST_IN_GUEST="${DEST_IN_GUEST:-/var/lib/hermes}" + +SOURCE="${1:-}" +[ -n "$SOURCE" ] || { echo "usage: $0 " >&2; exit 1; } +[ -d "$SOURCE" ] || { echo "error: $SOURCE is not a directory" >&2; exit 1; } + +SSH=(ssh + -o ProxyCommand="socat - VSOCK-CONNECT:${VSOCK_CID}:22" + -o StrictHostKeyChecking=accept-new + "${GUEST_USER}@hermes") + +# --------------------------------------------------------------------------- +# Rewrite rules applied to the migrated config after it lands in the guest. +# These are specific to the hl-fwdesk migration — edit for your own setup. +# +# host.docker.internal / fwdesk.lab.internal both resolved to hl-fwdesk +# itself under Docker; the guest reaches the same host at its bridge address. +# castle.hub.internal is a *different* LAN device (Home Assistant) reached +# via lanAccess instead. +# --------------------------------------------------------------------------- +REWRITE_FILES=(config.yaml auth.json context_length_cache.yaml .env) +REWRITE_FROM=(host.docker.internal fwdesk.lab.internal castle.hub.internal) +REWRITE_TO=(10.77.0.1 10.77.0.1 10.0.12.1) + +echo "Source (read-only): $SOURCE" +echo "Destination (guest): ${GUEST_USER}@hermes:${DEST_IN_GUEST} (over vsock, cid ${VSOCK_CID})" +echo +echo "This will:" +echo " 1. tar '$SOURCE' (not modifying it) and stream it into the guest" +echo " 2. rewrite these hostnames in the migrated config, in the guest only:" +for i in "${!REWRITE_FROM[@]}"; do + echo " ${REWRITE_FROM[$i]} -> ${REWRITE_TO[$i]}" +done +echo +read -r -p "Type 'migrate' to proceed: " confirm +[ "$confirm" = "migrate" ] || { echo "aborted"; exit 1; } + +echo +echo "== confirming guest is reachable ==" +"${SSH[@]}" true + +echo "== streaming $SOURCE -> guest:${DEST_IN_GUEST} (this can take a while for 8G+) ==" +tar -C "$SOURCE" -cf - . | "${SSH[@]}" "mkdir -p '${DEST_IN_GUEST}' && tar -C '${DEST_IN_GUEST}' -xf -" + +echo "== rewriting stale hostnames in the guest ==" +for f in "${REWRITE_FILES[@]}"; do + path="${DEST_IN_GUEST}/${f}" + sed_expr="" + for i in "${!REWRITE_FROM[@]}"; do + sed_expr+="s#${REWRITE_FROM[$i]}#${REWRITE_TO[$i]}#g; " + done + "${SSH[@]}" "[ -f '$path' ] && sed -i '$sed_expr' '$path' && echo ' rewrote $f' || echo ' $f not present, skipping'" +done + +echo +echo "Done. The original at $SOURCE is untouched." +echo +echo "Next, inside the guest (hermes-vm ssh):" +echo " export HERMES_HOME=${DEST_IN_GUEST}" +echo " # then the usual install steps from README.md:" +echo " # determinate-nixd login" +echo " # sudo nix flake lock /etc/nixos" +echo " # curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash" +echo " # hermes setup" +echo " # hermes gateway install" +echo +echo "Then verify with hermes-ctl status / logs before considering the old" +echo "deployment retired. Decrypt secrets/hermes-env.age yourself and fold" +echo "anything relevant into ${DEST_IN_GUEST}/.env (hermes-ctl secrets edit)." diff --git a/modules/host.nix b/modules/host.nix new file mode 100644 index 0000000..2ada2fc --- /dev/null +++ b/modules/host.nix @@ -0,0 +1,994 @@ +{ config, lib, pkgs, ... }: + +let + inherit (lib) mkOption mkEnableOption mkIf types optionalString; + cfg = config.services.hermesVm; + net = cfg.network; + + vmUser = "hermes-vm"; + runDir = "/run/hermes-vm"; + stateDir = "/var/lib/hermes-vm"; + # Two disks, deliberately. + # + # The system disk is large, churny and entirely rebuildable - it is the + # NixOS install and the Nix store. We never snapshot it. + # + # The state disk holds the only things that cannot be reproduced: Hermes's + # identity and memories, its workspace, and the NixOS config it has written + # for itself. Small, so snapshots are cheap and backups are quick. + systemImage = "${cfg.disk.baseDir}/system/system.raw"; + stateImage = "${cfg.disk.baseDir}/state/state.raw"; + snapDir = "${cfg.disk.baseDir}/snapshots"; + + pt = cfg.passthrough; + anyPci = pt.pci != [ ]; + anyUsb = pt.usb != [ ]; + earlyIds = lib.filter (i: i != null) (map (d: d.id) pt.pci); + + # Graceful ACPI power button via QMP, then wait for QEMU to exit. + stopScript = pkgs.writeShellScript "hermes-vm-stop" '' + set -eu + printf '%s' '{"execute":"qmp_capabilities"}{"execute":"system_powerdown"}' \ + | ${pkgs.socat}/bin/socat - UNIX-CONNECT:${runDir}/qmp.sock >/dev/null 2>&1 || exit 0 + ''; + + # Freeze the guest filesystems around a snapshot so it captures a consistent + # state rather than the equivalent of pulling the power. + qga = verb: pkgs.writeShellScript "hermes-vm-fs${verb}" '' + set -eu + printf '%s' '{"execute":"guest-fs${verb}"}' \ + | ${pkgs.socat}/bin/socat -t2 - UNIX-CONNECT:${runDir}/qga.sock >/dev/null 2>&1 || true + ''; +in +{ + options.services.hermesVm = { + enable = mkEnableOption "the Hermes Agent VM"; + + name = mkOption { + type = types.str; + default = "hermes"; + }; + + vcpu = mkOption { + type = types.ints.positive; + default = 4; + }; + + mem = mkOption { + type = types.ints.positive; + default = 8192; + description = "RAM in MiB."; + }; + + disk = { + baseDir = mkOption { + type = types.str; + default = "/var/lib/hermes-vm"; + description = '' + Directory holding the guest's disk images. On btrfs this should be + laid out as three subvolumes - system/, state/ and snapshots/ - with + nodatacow set. setup.sh does this for you. + ''; + }; + + systemSize = mkOption { + type = types.str; + default = "80G"; + description = '' + System disk: the NixOS install and the Nix store. Never snapshotted, + because it is entirely rebuildable from the flake. Size it for an + agent that builds things. + ''; + }; + + stateSize = mkOption { + type = types.str; + default = "20G"; + description = '' + State disk, mounted at /var/lib/hermes in the guest. Holds Hermes's + identity, memories, workspace, and the NixOS config it maintains for + itself. This is the only irreplaceable data, so it is the only thing + snapshotted and backed up. + ''; + }; + + quiesce = mkOption { + type = types.bool; + default = true; + description = "fsfreeze the guest via qemu-guest-agent around each snapshot."; + }; + }; + + snapshots = { + enable = mkOption { + type = types.bool; + default = true; + description = '' + Periodic btrfs snapshots of the state disk. The agent can persist + changes, which means it can persist a mistake, and these are the + fastest way back. + ''; + }; + + interval = mkOption { + type = types.str; + default = "hourly"; + description = "systemd calendar expression."; + }; + + keep = mkOption { + type = types.ints.positive; + default = 48; + description = '' + How many to retain. Keep this tighter than you would on ZFS: the + state image is nodatacow, and every live snapshot reintroduces + copy-on-write for the blocks it still references. + ''; + }; + }; + + backup = { + enable = mkEnableOption "restic backups of the guest's state"; + + repository = mkOption { + type = types.str; + default = ""; + example = "sftp:backup@nas:/hermes"; + description = "Any restic repository string."; + }; + + passwordFile = mkOption { + type = types.path; + default = "/var/lib/secrets/hermes-restic"; + description = "File containing the restic repository password, mode 0400."; + }; + + environmentFile = mkOption { + type = types.nullOr types.path; + default = null; + description = "Optional file with credentials for the backend (S3 keys and so on)."; + }; + + interval = mkOption { + type = types.str; + default = "daily"; + }; + + keep = mkOption { + type = types.listOf types.str; + default = [ "--keep-daily" "14" "--keep-weekly" "8" "--keep-monthly" "12" ]; + description = "Arguments passed to restic forget."; + }; + }; + + hostCache = { + enable = mkOption { + type = types.bool; + default = true; + description = '' + Serve the host's /nix/store to the guest as a read-only binary + cache, so the guest doesn't re-download from the internet what the + host already has. + + This is the ONLY route from guest to host in this design. Turning it + off leaves the guest with literally no reachable host service, at + the cost of bandwidth on every guest build. + + Note the guest can read every store path on the host through this. + Store paths are world-readable anyway, but check nothing sensitive + is baked into your host's store. + ''; + }; + port = mkOption { + type = types.port; + default = 5000; + }; + secretKeyFile = mkOption { + type = types.path; + default = "/var/lib/secrets/hermes-cache.secret"; + description = "Generate with nix-store --generate-binary-cache-key. See README."; + }; + }; + + authorizedKeys = mkOption { + type = types.listOf types.str; + default = [ ]; + }; + + allowLan = mkOption { + type = types.bool; + default = false; + description = '' + Blanket permission for the guest to reach RFC1918 space. Almost always + the wrong tool - use lanAccess to open single endpoints instead. + ''; + }; + + lanAccess = mkOption { + default = [ ]; + example = [ { address = "192.168.1.50"; port = 11434; } ]; + description = '' + Specific endpoints elsewhere on your network that the guest may reach, + punched through the default RFC1918 drop. One entry per service. + ''; + type = types.listOf (types.submodule { + options = { + address = mkOption { type = types.str; }; + port = mkOption { type = types.port; }; + proto = mkOption { + type = types.enum [ "tcp" "udp" ]; + default = "tcp"; + }; + }; + }); + }; + + hostAccess = mkOption { + default = [ ]; + example = [ { port = 8188; } ]; + description = '' + Ports on this host itself that the guest may reach. The service must + also be bound to an address the guest can see - see the README. + ''; + type = types.listOf (types.submodule { + options = { + port = mkOption { type = types.port; }; + proto = mkOption { + type = types.enum [ "tcp" "udp" ]; + default = "tcp"; + }; + }; + }); + }; + + publish = mkOption { + default = [ ]; + example = [ { hostPort = 8080; guestPort = 8080; } ]; + description = '' + Guest ports to publish on the host's uplink, reachable from the wider + network. You do not need this to reach the guest from the host itself. + ''; + type = types.listOf (types.submodule { + options = { + hostPort = mkOption { type = types.port; }; + guestPort = mkOption { type = types.port; }; + proto = mkOption { + type = types.enum [ "tcp" "udp" ]; + default = "tcp"; + }; + }; + }); + }; + + network = { + bridge = mkOption { type = types.str; default = "hermes-br"; }; + tapId = mkOption { type = types.str; default = "vm-hermes"; }; + mac = mkOption { type = types.str; default = "02:00:00:7e:00:01"; }; + hostAddress = mkOption { type = types.str; default = "10.77.0.1"; }; + guestAddress = mkOption { type = types.str; default = "10.77.0.2"; }; + prefixLength = mkOption { type = types.int; default = 24; }; + externalInterface = mkOption { + type = types.nullOr types.str; + default = null; + example = "eno1"; + description = "Host uplink. Required."; + }; + }; + + passthrough = { + iommu = mkOption { + type = types.nullOr (types.enum [ "intel" "amd" ]); + default = null; + description = '' + Enable IOMMU kernel parameters for your CPU vendor. Required for any + PCI passthrough. The IOMMU is the only thing stopping a passed-through + device from reading host memory, so this is not optional. + ''; + }; + + pci = mkOption { + default = [ ]; + example = [ + { address = "0000:01:00.0"; id = "10de:2204"; } + { address = "0000:01:00.1"; id = "10de:1aef"; } + ]; + description = '' + PCI devices handed to the guest via VFIO. The host loses them + entirely while the VM runs. + + You must pass every function of the device and, in practice, every + device in its IOMMU group. A GPU is usually at least two entries: + the graphics function and its HDMI audio function. + + Set `id` (vendor:device) to have vfio-pci claim the device in the + initrd, before a host driver can. Strongly recommended for GPUs; + unbinding an active amdgpu or nvidia driver later is unreliable. + ''; + type = types.listOf (types.submodule { + options = { + address = mkOption { + type = types.str; + example = "0000:01:00.0"; + description = "Full PCI address, as shown by lspci -D."; + }; + id = mkOption { + type = types.nullOr types.str; + default = null; + example = "10de:2204"; + description = "vendor:device, from lspci -nn. Enables early binding."; + }; + }; + }); + }; + + usb = mkOption { + default = [ ]; + example = [ { vendorId = "8087"; productId = "0026"; } ]; + description = '' + USB devices forwarded to the guest, matched by vendor and product + ID from lsusb. Integrated Bluetooth radios are USB devices and go + here too, though the host driver must be kept off them. + + QEMU resolves these at start, so a device that is unplugged when the + VM boots will not appear later without a restart. + ''; + type = types.listOf (types.submodule { + options = { + vendorId = mkOption { type = types.str; example = "046d"; }; + productId = mkOption { type = types.str; example = "c52b"; }; + }; + }); + }; + }; + + cliName = mkOption { + type = types.str; + default = "hermes-ctl"; + description = '' + Name of the host-side wrapper that drives Hermes inside the guest. + Set to "hermes" if you want it to feel local and are sure nothing + else on the host claims that name. + ''; + }; + + gatewayUnit = mkOption { + type = types.str; + default = "hermes-agent"; + description = '' + The systemd *user* unit that `hermes gateway install` created in the + guest. Check with `hermes-ctl units` after installing; upstream has + used more than one name for this. + ''; + }; + + hermesHome = mkOption { + type = types.str; + default = "${stateDir}/.hermes"; + description = '' + Where Hermes's own data (SOUL.md, config.yaml, memories/, sessions/, + state.db, and so on) lives inside the guest. Defaults to the seeded + layout under the state disk mount. Override this when carrying over + an existing flat HERMES_HOME from elsewhere (e.g. a prior Docker + deployment) that you're placing directly at the root of the state + disk instead of under .hermes - see migrate-docker-hermes.sh. + + This only changes where the host-side hermes-ctl wrapper (soul, + memory, conf, secrets, reseed, pull) looks. It does not change what + the gateway process itself uses - export HERMES_HOME to match inside + the guest when you install it. + ''; + }; + + vsockCid = mkOption { + type = types.int; + default = 77; + description = "Host reaches the guest over vsock, so no inbound network path is needed."; + }; + }; + + config = mkIf cfg.enable { + assertions = [ + { + assertion = net.externalInterface != null; + message = "services.hermesVm.network.externalInterface must be set."; + } + { + assertion = cfg.backup.enable -> (cfg.backup.repository != ""); + message = "services.hermesVm.backup.repository must be set when backups are enabled."; + } + { + assertion = anyPci -> (pt.iommu != null); + message = '' + services.hermesVm.passthrough.pci needs passthrough.iommu set to + "intel" or "amd". Without the IOMMU, a passed-through device can + read host memory directly and the VM boundary is meaningless. + ''; + } + ]; + + users.users.${vmUser} = { + isSystemUser = true; + group = vmUser; + description = "Runs the Hermes guest VM"; + }; + users.groups.${vmUser} = { }; + + # ===================================================================== + # The VM itself + # + # No virtiofs, no shared filesystem of any kind, no vhost-net. The guest + # gets a block device, a tap, a vsock and a serial port. That is all. + # ===================================================================== + systemd.services.hermes-vm = { + description = "Hermes Agent VM"; + wantedBy = [ "multi-user.target" ]; + after = [ "network-online.target" "hermes-vm-tap.service" ]; + requires = [ "hermes-vm-tap.service" ]; + wants = [ "network-online.target" ]; + + preStart = '' + for img in ${systemImage} ${stateImage}; do + if [ ! -e "$img" ]; then + echo "$img is missing. Run ./setup.sh first." >&2 + exit 1 + fi + done + # Per-VM writable UEFI variable store, seeded once. + if [ ! -e ${stateDir}/OVMF_VARS.fd ]; then + install -m 0600 ${pkgs.OVMF.variables} ${stateDir}/OVMF_VARS.fd + fi + ''; + + serviceConfig = { + User = vmUser; + Group = vmUser; + SupplementaryGroups = [ "kvm" ]; + RuntimeDirectory = "hermes-vm"; + # ProtectSystem=strict makes everything read-only; punch through for + # the images and the UEFI variable store only. + ReadWritePaths = [ cfg.disk.baseDir ]; + StateDirectory = "hermes-vm"; + RuntimeDirectoryMode = "0750"; + + ExecStart = lib.concatStringsSep " " ([ + "${pkgs.qemu_kvm}/bin/qemu-system-x86_64" + "-name ${cfg.name}" + "-machine q35,accel=kvm,smm=on" + "-cpu host" + "-smp ${toString cfg.vcpu}" + "-m ${toString cfg.mem}" + "-nodefaults" + "-no-user-config" + "-display none" + "-serial stdio" + # QEMU's own seccomp sandbox, everything denied that we don't need. + "-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny" + # UEFI: firmware read-only, variables per-VM and writable. + "-drive if=pflash,format=raw,unit=0,readonly=on,file=${pkgs.OVMF.firmware}" + "-drive if=pflash,format=raw,unit=1,file=${stateDir}/OVMF_VARS.fd" + # System disk: NixOS and the Nix store. Rebuildable, never snapshotted. + # aio=threads and a writeback cache because these are files on a + # filesystem, not block devices; aio=native wants the latter. + "-drive if=none,id=system,format=raw,file=${systemImage},cache=writeback,aio=threads,discard=unmap" + "-device virtio-blk-pci,drive=system" + # State disk: everything irreplaceable. Mounted at /var/lib/hermes. + "-drive if=none,id=state,format=raw,file=${stateImage},cache=writeback,aio=threads,discard=unmap" + "-device virtio-blk-pci,drive=state" + # Pre-created tap. vhost=off on purpose: vhost-net would move packet + # handling into the host kernel, which is exactly the surface we are + # trying not to expose. Costs throughput we do not need. + "-netdev tap,id=net0,ifname=${net.tapId},script=no,downscript=no,vhost=off" + "-device virtio-net-pci,netdev=net0,mac=${net.mac}" + # Host -> guest management, no network involved. + "-device vhost-vsock-pci,guest-cid=${toString cfg.vsockCid}" + "-device virtio-rng-pci" + "-qmp unix:${runDir}/qmp.sock,server=on,wait=off" + ] ++ lib.optionals cfg.disk.quiesce [ + "-chardev socket,path=${runDir}/qga.sock,server=on,wait=off,id=qga0" + "-device virtio-serial-pci" + "-device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0" + ] + # Passed-through PCI devices. The host cannot use these while the VM + # is running. + ++ map (d: "-device vfio-pci,host=${d.address}") pt.pci + # A USB controller, only if something needs plugging into it. + ++ lib.optional anyUsb "-device qemu-xhci,id=xhci" + ++ map (u: "-device usb-host,vendorid=0x${u.vendorId},productid=0x${u.productId}") pt.usb + ); + + ExecStop = stopScript; + TimeoutStopSec = 90; + Restart = "always"; + RestartSec = 5; + + # ---- hardening ------------------------------------------------- + DevicePolicy = "closed"; + DeviceAllow = [ + "/dev/kvm rw" + "/dev/net/tun rw" + "/dev/vhost-vsock rw" + ] + ++ lib.optionals anyPci [ "char-vfio rw" ] + ++ lib.optionals anyUsb [ "char-usb_device rw" ]; + + # VFIO pins the entire guest RAM allocation, so the default memlock + # limit has to go or QEMU fails to start with an obscure error. + LimitMEMLOCK = lib.mkIf anyPci "infinity"; + NoNewPrivileges = true; + ProtectSystem = "strict"; + ProtectHome = true; + ProtectProc = "invisible"; + ProcSubset = "pid"; + PrivateTmp = true; + ProtectKernelTunables = true; + ProtectKernelModules = true; + ProtectKernelLogs = true; + ProtectClock = true; + ProtectControlGroups = true; + ProtectHostname = true; + RestrictNamespaces = true; + RestrictRealtime = true; + RestrictSUIDSGID = true; + RestrictAddressFamilies = [ "AF_UNIX" "AF_NETLINK" "AF_VSOCK" ]; + LockPersonality = true; + SystemCallArchitectures = "native"; + SystemCallFilter = [ "@system-service" "~@resources" "~@obsolete" ]; + # KVM means no JIT, so W^X is safe. If QEMU refuses to start, this is + # the first thing to flip. + MemoryDenyWriteExecute = true; + CapabilityBoundingSet = [ "" ]; + AmbientCapabilities = [ "" ]; + }; + }; + + # Tap device, created as root, owned by the VM user, bridged by networkd. + systemd.services.hermes-vm-tap = { + description = "TAP interface for the Hermes VM"; + after = [ "systemd-networkd.service" ]; + serviceConfig = { + Type = "oneshot"; + RemainAfterExit = true; + }; + path = [ pkgs.iproute2 ]; + script = '' + ip link show ${net.tapId} >/dev/null 2>&1 \ + || ip tuntap add ${net.tapId} mode tap user ${vmUser} multi_queue + ip link set ${net.tapId} up + ''; + preStop = '' + ${pkgs.iproute2}/bin/ip link del ${net.tapId} || true + ''; + }; + + # ===================================================================== + # Device passthrough + # ===================================================================== + + # The IOMMU is what keeps a passed-through device from reading host + # memory. Without it, PCI passthrough is not a boundary at all. + boot.kernelParams = lib.optionals (pt.iommu != null) [ + (if pt.iommu == "intel" then "intel_iommu=on" else "amd_iommu=on") + "iommu=pt" + ]; + + boot.initrd.kernelModules = lib.optionals anyPci [ + "vfio_pci" + "vfio" + "vfio_iommu_type1" + ]; + + # Claim the device before any host driver gets to it. + boot.extraModprobeConfig = lib.optionalString (earlyIds != [ ]) '' + options vfio-pci ids=${lib.concatStringsSep "," earlyIds} + ''; + + # Catch anything that was not early-bound, by address. Idempotent. + systemd.services.hermes-vm-vfio = mkIf anyPci { + description = "Bind passthrough PCI devices to vfio-pci"; + before = [ "hermes-vm.service" ]; + requiredBy = [ "hermes-vm.service" ]; + serviceConfig = { + Type = "oneshot"; + RemainAfterExit = true; + }; + path = [ pkgs.kmod ]; + script = '' + modprobe vfio-pci + '' + lib.concatMapStrings (d: '' + dev=/sys/bus/pci/devices/${d.address} + if [ ! -e "$dev" ]; then + echo "${d.address} is not present on this machine" >&2 + exit 1 + fi + cur=none + [ -e "$dev/driver" ] && cur=$(basename "$(readlink -f "$dev/driver")") + if [ "$cur" != "vfio-pci" ]; then + [ "$cur" != "none" ] && echo ${d.address} > "$dev/driver/unbind" + echo vfio-pci > "$dev/driver_override" + echo ${d.address} > /sys/bus/pci/drivers_probe + fi + '') pt.pci; + }; + + services.udev.extraRules = + lib.optionalString anyPci '' + SUBSYSTEM=="vfio", OWNER="${vmUser}", GROUP="${vmUser}", MODE="0660" + '' + + lib.concatMapStrings (u: '' + SUBSYSTEM=="usb", ATTR{idVendor}=="${u.vendorId}", ATTR{idProduct}=="${u.productId}", OWNER="${vmUser}", MODE="0660" + '') pt.usb; + + # ===================================================================== + # Networking + # ===================================================================== + networking.useNetworkd = true; + + systemd.network.netdevs."10-${net.bridge}".netdevConfig = { + Kind = "bridge"; + Name = net.bridge; + }; + + systemd.network.networks."10-${net.bridge}" = { + matchConfig.Name = net.bridge; + address = [ "${net.hostAddress}/${toString net.prefixLength}" ]; + networkConfig = { + ConfigureWithoutCarrier = true; + IPv4Forwarding = true; + }; + }; + + systemd.network.networks."11-${net.tapId}" = { + matchConfig.Name = net.tapId; + networkConfig.Bridge = net.bridge; + }; + + networking.nat = { + enable = true; + internalInterfaces = [ net.bridge ]; + externalInterface = net.externalInterface; + forwardPorts = map (s: { + sourcePort = s.hostPort; + destination = "${net.guestAddress}:${toString s.guestPort}"; + inherit (s) proto; + }) cfg.publish; + }; + + # ===================================================================== + # Firewall + # + # The guest owns its own kernel and its own config now, so nothing inside + # it constrains it. These rules are the boundary. + # ===================================================================== + networking.nftables.enable = true; + networking.firewall.enable = true; + networking.firewall.filterForward = true; + + # The deliberate guest -> host holes: the Nix cache, plus anything listed + # in hostAccess. + networking.firewall.extraInputRules = + optionalString cfg.hostCache.enable '' + iifname "${net.bridge}" ip saddr ${net.guestAddress} tcp dport ${toString cfg.hostCache.port} accept + '' + + lib.concatMapStrings (s: '' + iifname "${net.bridge}" ip saddr ${net.guestAddress} ${s.proto} dport ${toString s.port} accept + '') cfg.hostAccess; + + # Order matters here. Allowlists must precede the RFC1918 drop. + networking.firewall.extraForwardRules = + '' + ct state established,related accept + '' + # Guest reaching named services elsewhere on your network. + + lib.concatMapStrings (s: '' + iifname "${net.bridge}" ip daddr ${s.address} ${s.proto} dport ${toString s.port} accept + '') cfg.lanAccess + # Inbound to guest services published on the uplink. + + lib.concatMapStrings (s: '' + oifname "${net.bridge}" ip daddr ${net.guestAddress} ${s.proto} dport ${toString s.guestPort} accept + '') cfg.publish + + (if cfg.allowLan then '' + iifname "${net.bridge}" accept + '' else '' + iifname "${net.bridge}" ip daddr { + 10.0.0.0/8, + 172.16.0.0/12, + 192.168.0.0/16, + 169.254.0.0/16, + 127.0.0.0/8, + 100.64.0.0/10 + } drop + iifname "${net.bridge}" ip6 daddr { fc00::/7, fe80::/10, ::1/128 } drop + iifname "${net.bridge}" accept + ''); + + # ===================================================================== + # Host store as a read-only cache for the guest + # ===================================================================== + services.nix-serve = mkIf cfg.hostCache.enable { + enable = true; + package = pkgs.nix-serve-ng; + bindAddress = net.hostAddress; + inherit (cfg.hostCache) port secretKeyFile; + openFirewall = false; # handled above, scoped to the guest address + }; + + # ===================================================================== + # Snapshots of the state disk only. + # + # The system disk is not snapshotted on purpose. It is large, it is where + # all the write churn happens, and setup.sh can rebuild it from the flake + # in minutes. Snapshotting it would cost a lot and buy nothing. + # ===================================================================== + systemd.services.hermes-vm-snapshot = mkIf cfg.snapshots.enable { + description = "Snapshot the Hermes VM state disk"; + path = with pkgs; [ btrfs-progs coreutils socat ]; + serviceConfig.Type = "oneshot"; + script = '' + set -euo pipefail + stamp=$(date +%Y%m%d-%H%M%S) + mkdir -p ${snapDir} + + ${optionalString cfg.disk.quiesce '' + # Flush the guest's filesystems so the snapshot is consistent rather + # than the equivalent of pulling the power. Takes milliseconds; the + # VM keeps running throughout. + ${qga "freeze"} + trap '${qga "thaw"}' EXIT + ''} + + btrfs subvolume snapshot -r ${cfg.disk.baseDir}/state ${snapDir}/state-$stamp + + ${optionalString cfg.disk.quiesce '' + ${qga "thaw"} + trap - EXIT + ''} + + # Prune oldest first. Keep this list short: every live snapshot + # reintroduces copy-on-write on a nodatacow image. + ls -1d ${snapDir}/state-* 2>/dev/null | sort | head -n -${toString cfg.snapshots.keep} \ + | while read -r old; do btrfs subvolume delete "$old"; done + ''; + }; + + systemd.timers.hermes-vm-snapshot = mkIf cfg.snapshots.enable { + wantedBy = [ "timers.target" ]; + timerConfig = { + OnCalendar = cfg.snapshots.interval; + Persistent = true; + RandomizedDelaySec = "5m"; + }; + }; + + # ===================================================================== + # Backups: the guest's state, not its disk. + # + # Pulled file-by-file over the SSH channel that already exists, rather + # than backing up the image. That keeps the backup portable - it restores + # onto any filesystem, or a different machine entirely - and avoids + # loop-mounting a guest-written filesystem on the host, which would mean + # the host kernel parsing data the agent controls. + # ===================================================================== + systemd.services.hermes-vm-backup = mkIf cfg.backup.enable { + description = "Back up Hermes state"; + path = with pkgs; [ restic openssh socat gnutar ]; + serviceConfig = { + Type = "oneshot"; + EnvironmentFile = lib.mkIf (cfg.backup.environmentFile != null) cfg.backup.environmentFile; + }; + script = '' + set -euo pipefail + export RESTIC_REPOSITORY=${lib.escapeShellArg cfg.backup.repository} + export RESTIC_PASSWORD_FILE=${cfg.backup.passwordFile} + + restic snapshots >/dev/null 2>&1 || restic init + + ssh -o ProxyCommand="socat - VSOCK-CONNECT:${toString cfg.vsockCid}:22" \ + -o StrictHostKeyChecking=accept-new \ + hermes@${cfg.name} \ + 'tar -C /var/lib/hermes \ + --exclude=./.cache \ + --exclude=./workspace/**/.direnv \ + --exclude=./workspace/**/result \ + -cf - .' \ + | restic backup --stdin --stdin-filename hermes-state.tar \ + --tag hermes --host ${cfg.name} + + restic forget --tag hermes --host ${cfg.name} --prune ${lib.concatStringsSep " " cfg.backup.keep} + ''; + }; + + systemd.timers.hermes-vm-backup = mkIf cfg.backup.enable { + wantedBy = [ "timers.target" ]; + timerConfig = { + OnCalendar = cfg.backup.interval; + Persistent = true; + RandomizedDelaySec = "30m"; + }; + }; + + # ===================================================================== + # Host GC is now safe: the guest has its own store and shares nothing. + # ===================================================================== + nix.gc.automatic = lib.mkDefault true; + nix.gc.options = lib.mkDefault "--delete-older-than 30d"; + + environment.systemPackages = [ + (pkgs.writeShellScriptBin "hermes-vm" '' + set -euo pipefail + SSH=(${pkgs.openssh}/bin/ssh + -o ProxyCommand="${pkgs.socat}/bin/socat - VSOCK-CONNECT:${toString cfg.vsockCid}:22" + -o StrictHostKeyChecking=accept-new + hermes@${cfg.name}) + cmd="''${1-}"; shift || true + case "$cmd" in + ssh) exec "''${SSH[@]}" "$@" ;; + console) exec journalctl -u hermes-vm -f -o cat ;; + start) exec systemctl start hermes-vm ;; + stop) exec systemctl stop hermes-vm ;; + restart) exec systemctl restart hermes-vm ;; + + # What is the guest actually running right now? + config) + "''${SSH[@]}" 'echo "== current generation =="; \ + readlink -f /run/current-system; \ + echo; echo "== generations =="; \ + sudo nix-env --list-generations -p /nix/var/nix/profiles/system | tail -n 10; \ + echo; echo "== /etc/nixos history =="; \ + git -C /etc/nixos log --oneline -n 15; \ + echo; echo "== uncommitted =="; \ + git -C /etc/nixos status --short' + ;; + # Copy the guest's live config here for reading or diffing. + config-pull) + dest="''${1:-./hermes-etc-nixos}" + mkdir -p "$dest" + "''${SSH[@]}" 'tar -C /etc/nixos -cf - .' | tar -C "$dest" -xf - + echo "pulled to $dest" + ;; + + snapshot) exec systemctl start hermes-vm-snapshot ;; + snapshots) + ${pkgs.util-linux}/bin/column -t <<<"$( + echo "SNAPSHOT SIZE CREATED" + for d in ${snapDir}/state-*; do + [ -e "$d" ] || continue + echo "$(basename "$d") $(du -sh "$d" 2>/dev/null | cut -f1) $(stat -c %y "$d" | cut -d. -f1)" + done + )" ;; + + # Restore the state disk from a snapshot. The system disk is left + # alone - it holds nothing worth restoring. + rollback) + [ $# -ge 1 ] || { echo "usage: hermes-vm rollback " >&2; exit 1; } + snap=${snapDir}/"$1" + [ -d "$snap" ] || { echo "no such snapshot: $1" >&2; exit 1; } + echo "Restores Hermes's identity, memories, workspace and NixOS" >&2 + echo "config to $1. Everything since is lost." >&2 + read -r -p "type 'rollback' to confirm: " c + [ "$c" = rollback ] || { echo "aborted" >&2; exit 1; } + systemctl stop hermes-vm + mv ${cfg.disk.baseDir}/state ${cfg.disk.baseDir}/state.old-$(date +%s) + ${pkgs.btrfs-progs}/bin/btrfs subvolume snapshot "$snap" ${cfg.disk.baseDir}/state + systemctl start hermes-vm + echo "restored; previous state kept as state.old-*" ;; + + backup) exec systemctl start hermes-vm-backup ;; + backups) + exec ${pkgs.restic}/bin/restic \ + -r "$(systemctl show hermes-vm-backup -p Environment --value | tr ' ' '\n' | sed -n 's/^RESTIC_REPOSITORY=//p')" \ + snapshots --tag hermes ;; + *) + echo "usage: hermes-vm {ssh [cmd]|console|config|config-pull [dir]|start|stop|restart|snapshot|snapshots|rollback |backup|backups}" >&2 + exit 1 ;; + esac + '') + + # --------------------------------------------------------------- + # Driving Hermes itself, as opposed to the VM around it. + # --------------------------------------------------------------- + (pkgs.writeShellScriptBin cfg.cliName '' + set -euo pipefail + SSH=(${pkgs.openssh}/bin/ssh + -o ProxyCommand="${pkgs.socat}/bin/socat - VSOCK-CONNECT:${toString cfg.vsockCid}:22" + -o StrictHostKeyChecking=accept-new + hermes@${cfg.name}) + HOME_DIR=${cfg.hermesHome} + UNIT=${cfg.gatewayUnit} + + # Pull a file out of the guest, open it in your editor, push it back. + edit_remote() { + remote="$1"; mode="''${2:-0644}" + tmp=$(mktemp); trap 'rm -f "$tmp"' EXIT + "''${SSH[@]}" "cat $remote" > "$tmp" 2>/dev/null || true + before=$(${pkgs.coreutils}/bin/sha256sum < "$tmp") + "''${EDITOR:-${pkgs.vim}/bin/vim}" "$tmp" + if [ "$(${pkgs.coreutils}/bin/sha256sum < "$tmp")" = "$before" ]; then + echo "unchanged"; return + fi + "''${SSH[@]}" "install -m $mode /dev/stdin $remote" < "$tmp" + echo "written to $remote" + read -r -p "restart the gateway now? [y/N] " yn + case "$yn" in [yY]*) "''${SSH[@]}" "systemctl --user restart $UNIT" ;; esac + } + + cmd="''${1-}"; shift || true + case "$cmd" in + soul) + case "''${1-}" in + edit) edit_remote "$HOME_DIR/SOUL.md" ;; + *) "''${SSH[@]}" "cat $HOME_DIR/SOUL.md" ;; + esac ;; + + memory) + case "''${1-}" in + edit) edit_remote "$HOME_DIR/memories/USER.md" ;; + *) "''${SSH[@]}" "cat $HOME_DIR/memories/USER.md" ;; + esac ;; + + conf) + case "''${1-}" in + edit) edit_remote "$HOME_DIR/config.yaml" ;; + *) "''${SSH[@]}" "cat $HOME_DIR/config.yaml" ;; + esac ;; + + # API keys. Mode 0600, never enters the Nix store, and not printed + # unless you ask for it by name. + secrets) + case "''${1-}" in + edit) edit_remote "$HOME_DIR/.env" 0600 ;; + show) "''${SSH[@]}" "cat $HOME_DIR/.env" ;; + *) "''${SSH[@]}" "sed 's/=.*/=/' $HOME_DIR/.env" ;; + esac ;; + + status) "''${SSH[@]}" "systemctl --user status $UNIT --no-pager" ;; + logs) "''${SSH[@]}" -t "journalctl --user -u $UNIT -f" ;; + restart) "''${SSH[@]}" "systemctl --user restart $UNIT" ;; + units) "''${SSH[@]}" "systemctl --user list-units 'hermes*' --all --no-pager" ;; + + # Overwrite the agent's identity from the seed. Destructive: this + # is how you undo whatever it has decided about itself. + reseed) + echo "This overwrites SOUL.md, memories and config.yaml in the guest" >&2 + echo "with the seed from /etc/nixos/hermes. Anything Hermes has" >&2 + echo "learned and written there is lost." >&2 + read -r -p "type 'reseed' to confirm: " confirm + [ "$confirm" = "reseed" ] || { echo "aborted" >&2; exit 1; } + "''${SSH[@]}" "cp -rT /etc/nixos/hermes/home $HOME_DIR && \ + cp -rT /etc/nixos/hermes/workspace /var/lib/hermes/workspace && \ + systemctl --user restart $UNIT" + echo "reseeded" ;; + + # Copy the agent's live identity back here, so you can fold what it + # has learned into the seed in guest/hermes/. + pull) + dest="''${1:-./hermes-home}" + mkdir -p "$dest" + "''${SSH[@]}" "tar -C $HOME_DIR --exclude=.env --exclude=sessions --exclude=state.db -cf - ." \ + | tar -C "$dest" -xf - + echo "pulled to $dest (secrets and sessions excluded)" ;; + + ""|help|-h|--help) + cat >&2 <<'USAGE' +${cfg.cliName} - drive Hermes inside the guest + + soul [edit] show or edit SOUL.md + memory [edit] show or edit memories/USER.md + conf [edit] show or edit config.yaml + secrets [edit|show] manage .env, keys masked by default + status | logs | restart | units + reseed overwrite identity from /etc/nixos/hermes + pull [dir] copy live identity back to the host + +Anything else goes straight to the hermes CLI in the guest: + ${cfg.cliName} chat + ${cfg.cliName} update + ${cfg.cliName} sessions list +USAGE + exit 1 ;; + + *) exec "''${SSH[@]}" -t hermes "$cmd" "$@" ;; + esac + '') + ]; + }; +} diff --git a/setup.sh b/setup.sh new file mode 100755 index 0000000..b616067 --- /dev/null +++ b/setup.sh @@ -0,0 +1,380 @@ +#!/usr/bin/env bash +# +# hermes-vm guided setup. +# +# Walks the steps that have to happen before `nixos-rebuild switch`: +# the disk images, the cache signing key, filling in guest/common.nix, building the +# bootstrap image and writing it to disk. +# +# Safe to re-run. Every step checks whether it is already done and offers to +# skip. Nothing is destructive without an explicit confirmation. + +set -euo pipefail + +# ---------------------------------------------------------------- output --- + +if [ -t 1 ] && [ -z "${NO_COLOR:-}" ]; then + B=$'\033[1m'; DIM=$'\033[2m'; R=$'\033[31m'; Y=$'\033[33m'; G=$'\033[32m'; N=$'\033[0m' +else + B=""; DIM=""; R=""; Y=""; G=""; N="" +fi + +step_n=0 +step() { step_n=$((step_n + 1)); printf '\n%s== %d. %s%s\n' "$B" "$step_n" "$*" "$N"; } +say() { printf ' %s\n' "$*"; } +note() { printf ' %s%s%s\n' "$DIM" "$*" "$N"; } +ok() { printf ' %s✓%s %s\n' "$G" "$N" "$*"; } +warn() { printf ' %s!%s %s\n' "$Y" "$N" "$*"; } +die() { printf '\n%serror:%s %s\n' "$R" "$N" "$*" >&2; exit 1; } + +ask() { # ask VAR "prompt" "default" + local __var="$1" __prompt="$2" __default="${3:-}" __reply + if [ -n "$__default" ]; then + read -r -p " $__prompt [$__default]: " __reply || true + __reply="${__reply:-$__default}" + else + read -r -p " $__prompt: " __reply || true + fi + printf -v "$__var" '%s' "$__reply" +} + +confirm() { # confirm "prompt" -> 0 if yes + local reply + read -r -p " $1 [y/N] " reply || true + case "$reply" in [yY]*) return 0 ;; *) return 1 ;; esac +} + +confirm_word() { # confirm_word "word" "prompt" -> 0 if they typed the word + local reply + read -r -p " $2 " reply || true + [ "$reply" = "$1" ] +} + +# ------------------------------------------------------------------ setup --- + +cd "$(dirname "$(readlink -f "$0")")" + +if [ "$(id -u)" -eq 0 ]; then + SUDO="" +else + command -v sudo >/dev/null || die "not root and sudo is not available" + SUDO="sudo" +fi + +COMMON="guest/common.nix" +CACHE_NAME="hermes-cache-1" +SECRET_KEY="/var/lib/secrets/hermes-cache.secret" +PUBLIC_KEY="/var/lib/secrets/hermes-cache.pub" +DETERMINATE_SUB="https://install.determinate.systems" +DETERMINATE_KEY="cache.flakehub.com-3:hJuILl5sVK4iKm86JzgdXW12Y2Hwd5G07qKtHTOcDCM=" + +printf '%s\n' "$B" +cat <<'BANNER' + hermes-vm · guided setup +BANNER +printf '%s' "$N" +note "This prepares the host. It does not run nixos-rebuild — you'll paste a" +note "module snippet into your own configuration at the end." + +# ------------------------------------------------------------- preflight --- + +step "Checking prerequisites" + +[ -f flake.nix ] && [ -f "$COMMON" ] \ + || die "run this from the hermes-vm repository root" + +command -v nix >/dev/null || die "nix not found on PATH" +nix flake --help >/dev/null 2>&1 \ + || die "flakes are not enabled. Add 'experimental-features = nix-command flakes' to /etc/nix/nix.conf" +ok "nix with flakes" + +command -v btrfs >/dev/null || die "btrfs-progs not found. Snapshots need it." +command -v mkfs.ext4 >/dev/null || die "mkfs.ext4 not found (e2fsprogs)" +ok "btrfs-progs" + +[ -e /dev/kvm ] || die "/dev/kvm is missing. KVM is required; check virtualisation is enabled in firmware." +ok "kvm" + +if command -v nixos-version >/dev/null; then + ok "host is NixOS ($(nixos-version | cut -d' ' -f1))" +else + warn "this host does not look like NixOS — the module expects NixOS" +fi + +# ----------------------------------------------------------------- disks --- + +step "Guest disks" + +note "Two disks, deliberately:" +note " system - NixOS and the Nix store. Big, churny, rebuildable. Not backed up." +note " state - identity, memories, workspace, /etc/nixos. Small. Snapshotted." + +ask BASE "Directory for the disk images" "/var/lib/hermes-vm" + +FSTYPE="$(stat -f -c %T "$(dirname "$BASE")" 2>/dev/null || echo unknown)" +if [ "$FSTYPE" != "btrfs" ]; then + warn "$(dirname "$BASE") is $FSTYPE, not btrfs" + warn "snapshots need btrfs. Everything else works; you'd rely on backups alone." + confirm "Continue anyway?" || die "aborted" + USE_BTRFS=no +else + USE_BTRFS=yes +fi + +$SUDO mkdir -p "$BASE" + +make_subvol() { # make_subvol path + if [ "$USE_BTRFS" = yes ] && ! $SUDO btrfs subvolume show "$1" >/dev/null 2>&1; then + $SUDO rmdir "$1" 2>/dev/null || true + $SUDO btrfs subvolume create "$1" >/dev/null + # nodatacow BEFORE any file exists in here. Btrfs is copy-on-write and so + # is a VM writing to a disk image; stacked, the image fragments badly. + # Only applies to files created afterwards, hence the ordering. + $SUDO chattr +C "$1" + $SUDO btrfs property set "$1" compression none 2>/dev/null || true + ok "subvolume $1 (nodatacow)" + else + $SUDO mkdir -p "$1" + fi +} + +make_subvol "$BASE/system" +make_subvol "$BASE/state" +$SUDO mkdir -p "$BASE/snapshots" + +SYSTEM_IMG="$BASE/system/system.raw" +STATE_IMG="$BASE/state/state.raw" + +# --- system disk --- +if $SUDO test -f "$SYSTEM_IMG"; then + ok "system disk exists ($($SUDO du -h --apparent-size "$SYSTEM_IMG" | cut -f1))" + SYSTEM_EXISTED=yes +else + SYSTEM_EXISTED=no + ask SYSSIZE "System disk size" "80G" + $SUDO truncate -s "$SYSSIZE" "$SYSTEM_IMG" + if [ "$USE_BTRFS" = yes ]; then + $SUDO lsattr -l "$SYSTEM_IMG" 2>/dev/null | grep -q "No_COW" \ + && ok "system disk created, nodatacow confirmed" \ + || warn "system disk created but nodatacow did NOT apply - check 'lsattr $SYSTEM_IMG'" + else + ok "system disk created" + fi +fi + +# --- state disk --- +if $SUDO test -f "$STATE_IMG"; then + ok "state disk exists ($($SUDO du -h --apparent-size "$STATE_IMG" | cut -f1))" +else + ask STATESIZE "State disk size" "20G" + $SUDO truncate -s "$STATESIZE" "$STATE_IMG" + $SUDO mkfs.ext4 -q -L hermes-state "$STATE_IMG" + ok "state disk created and formatted (label: hermes-state)" +fi + +# ------------------------------------------------------------- cache key --- + +step "Binary cache signing key" + +note "The host serves its store to the guest read-only, so the guest doesn't" +note "re-download from the internet what you already have." + +if $SUDO test -f "$SECRET_KEY"; then + ok "signing key already exists" +else + $SUDO mkdir -p /var/lib/secrets + $SUDO nix-store --generate-binary-cache-key "$CACHE_NAME" "$SECRET_KEY" "$PUBLIC_KEY" + $SUDO chmod 0400 "$SECRET_KEY" + ok "generated $CACHE_NAME" +fi + +CACHE_PUB="$($SUDO cat "$PUBLIC_KEY")" +say "Public key: ${DIM}${CACHE_PUB}${N}" + +# nix-serve's user only exists once the module has been applied, so this is +# expected to fail on a first run. +if id nix-serve >/dev/null 2>&1; then + $SUDO chown nix-serve "$SECRET_KEY" + ok "key owned by nix-serve" +else + warn "the nix-serve user does not exist yet — that's normal on a first run" + note "after your nixos-rebuild, run:" + note " sudo chown nix-serve $SECRET_KEY" +fi + +# ------------------------------------------------------------- ssh keys ---- + +step "SSH access to the guest" + +CURRENT_KEYS="$(sed -n '/^ authorizedKeys = \[/,/^ \];/p' "$COMMON" | grep -c '^\s*"ssh-' || true)" +if [ "${CURRENT_KEYS:-0}" -gt 0 ]; then + ok "$CURRENT_KEYS key(s) already configured in $COMMON" + if confirm "Replace them?"; then CURRENT_KEYS=0; fi +fi + +SELECTED_KEYS="" +if [ "${CURRENT_KEYS:-0}" -eq 0 ]; then + mapfile -t FOUND < <(ls -1 "$HOME"/.ssh/*.pub 2>/dev/null || true) + if [ "${#FOUND[@]}" -gt 0 ]; then + say "Public keys found in ~/.ssh:" + i=0 + for f in "${FOUND[@]}"; do + i=$((i + 1)) + note " $i) $(basename "$f") — $(cut -d' ' -f3- "$f")" + done + ask PICK "Numbers to use (space separated, or 'n' to paste one)" "1" + if [ "$PICK" != "n" ]; then + for n in $PICK; do + f="${FOUND[$((n - 1))]:-}" + [ -n "$f" ] || die "no such option: $n" + SELECTED_KEYS+=" \"$(tr -d '\n' < "$f")\" +" + done + fi + fi + + if [ -z "$SELECTED_KEYS" ]; then + ask PASTED "Paste a public key (blank to skip)" + [ -n "$PASTED" ] && SELECTED_KEYS=" \"$PASTED\" +" + fi + + if [ -z "$SELECTED_KEYS" ]; then + warn "no SSH key set — you'll only be able to reach the guest with 'hermes-vm console'" + fi +fi + +# ------------------------------------------------------ patch common.nix --- + +step "Writing $COMMON" + +[ -f "$COMMON.bak" ] || cp "$COMMON" "$COMMON.bak" +note "original saved as $COMMON.bak" + +# cachePublicKey +tmp="$(mktemp)" +sed 's|^\( cachePublicKey = \).*|\1"'"$CACHE_PUB"'";|' "$COMMON" > "$tmp" +mv "$tmp" "$COMMON" +ok "cachePublicKey set" + +# authorizedKeys block +if [ -n "$SELECTED_KEYS" ]; then + tmp="$(mktemp)" + awk -v keys="$SELECTED_KEYS" ' + /^ authorizedKeys = \[/ { print; printf "%s", keys; inblock = 1; next } + inblock && /^ \];/ { inblock = 0; print; next } + inblock { next } + { print } + ' "$COMMON" > "$tmp" + mv "$tmp" "$COMMON" + ok "authorizedKeys set ($(printf '%s' "$SELECTED_KEYS" | grep -c '^' ) key(s))" +fi + +# ----------------------------------------------------------- build image --- + +step "Building the bootstrap image" + +note "The extra substituter flags stop you compiling Determinate Nix yourself." + +if confirm "Build now? (first run can take a while)"; then + nix build .#hermes-guest-image \ + --option extra-substituters "$DETERMINATE_SUB" \ + --option extra-trusted-public-keys "$DETERMINATE_KEY" + ok "built" +else + say "Skipped. Run manually:" + note " nix build .#hermes-guest-image \\" + note " --option extra-substituters $DETERMINATE_SUB \\" + note " --option extra-trusted-public-keys $DETERMINATE_KEY" +fi + +# ------------------------------------------------------------ write disk --- + +step "Writing the bootstrap image to the system disk" + +IMAGE="$(ls -1 result/*.img 2>/dev/null | head -n1 || true)" + +if [ -z "$IMAGE" ]; then + warn "no image found under ./result - skipping" + note "build it, then:" + note " sudo dd if=result/nixos.img of=$SYSTEM_IMG bs=4M conv=notrunc,fsync status=progress" +else + IMG_BYTES=$(stat -Lc %s "$IMAGE") + VOL_BYTES=$($SUDO stat -c %s "$SYSTEM_IMG") + say "image: $IMAGE ($((IMG_BYTES / 1024 / 1024)) MiB)" + say "target: $SYSTEM_IMG ($((VOL_BYTES / 1024 / 1024 / 1024)) GiB)" + [ "$IMG_BYTES" -le "$VOL_BYTES" ] || die "image is larger than the system disk" + + if [ "$SYSTEM_EXISTED" = yes ]; then + warn "the system disk already existed. Writing the image replaces the" + warn "guest OS. Hermes's identity and memories live on the STATE disk and" + warn "are untouched, but anything it installed outside /var/lib/hermes is lost." + fi + + if confirm_word "overwrite" "Type 'overwrite' to write the image:"; then + # notrunc so the file keeps its full size; the image is smaller than the disk + $SUDO dd if="$IMAGE" of="$SYSTEM_IMG" bs=4M conv=notrunc,fsync status=progress + ok "written" + else + say "Skipped." + fi +fi + +# ----------------------------------------------------------- next steps ---- + +UPLINK="$(ip route get 1.1.1.1 2>/dev/null | sed -n 's/.* dev \([^ ]*\).*/\1/p' | head -n1)" +UPLINK="${UPLINK:-eno1}" + +step "Next: add the module to your system flake" + +cat <