921 lines
34 KiB
Markdown
921 lines
34 KiB
Markdown
# hermenix
|
|
|
|
Hermes Agent in a NixOS VM that owns its own bootloader and its own
|
|
`/nix/store`, on two btrfs-backed disks, under an unprivileged hardened QEMU.
|
|
|
|
The agent can rebuild its own system and update itself, durably. It shares no
|
|
filesystem with the host, and reaches nothing on your network unless you say
|
|
so.
|
|
|
|
One CLI, `hermenix`, with five subcommands:
|
|
|
|
| | For |
|
|
|---|---|
|
|
| `hermenix setup` | one-off host provisioning — disks, cache key, image build |
|
|
| `hermenix migrate <src>` | carry an existing flat `HERMES_HOME` into the guest |
|
|
| `hermenix remote <host> "<cmd>"` | run `setup`/`migrate` on a remote host over SSH |
|
|
| `hermenix vm <sub>` | control the VM itself — start/stop/console/snapshots/backups |
|
|
| `hermenix ctl <sub>` | drive the agent inside the VM — identity/config/secrets/gateway |
|
|
|
|
---
|
|
|
|
## Installation
|
|
|
|
### Guided
|
|
|
|
The `hermenix` CLI (`nu/hermenix.nu`) is a flake package — no checkout needed
|
|
anywhere. Run it straight from its Gitea URL:
|
|
|
|
```bash
|
|
nix run git+ssh://gitea@minipc1.lab.internal:22/ari/hermenix.git#hermenix -- setup
|
|
```
|
|
|
|
(Or `nix profile install git+ssh://gitea@minipc1.lab.internal:22/ari/hermenix.git#hermenix`
|
|
once, then just `hermenix setup` from here on — or get it declaratively via
|
|
the nixos-setup overlay, same as any other package. Any of these work from
|
|
any directory, on any host with SSH access to Gitea.)
|
|
|
|
Walks the host-side steps: checks prerequisites, creates the disks, generates
|
|
the cache signing key, writes `/etc/hermenix/local.nix` from your SSH keys,
|
|
builds the bootstrap image and writes it to the disk. It prints the module
|
|
snippet to paste into your own configuration at the end, with your uplink
|
|
interface already filled in.
|
|
|
|
Safe to re-run — every step detects what's already done and offers to skip,
|
|
and nothing destructive happens without an explicit confirmation.
|
|
|
|
It deliberately stops short of `nixos-rebuild`, since that's your config.
|
|
|
|
The rest of this section is what the command does, for when you'd rather do
|
|
it by hand or something goes sideways.
|
|
|
|
### 1. Create the disks
|
|
|
|
Two of them, and the split is the whole design:
|
|
|
|
| | Holds | Size | Snapshotted | Backed up |
|
|
|---|---|---|---|---|
|
|
| **system** | NixOS, `/nix/store` | 80G | No | No |
|
|
| **state** | identity, memories, workspace, `/etc/nixos` | 20G | Yes | Yes |
|
|
|
|
The system disk is large, takes all the write churn, and is entirely
|
|
rebuildable from this flake — so there's no point protecting it. The state
|
|
disk holds the only things that can't be reproduced, which makes snapshots
|
|
cheap and backups quick.
|
|
|
|
```bash
|
|
sudo btrfs subvolume create /var/lib/hermes-vm/system
|
|
sudo btrfs subvolume create /var/lib/hermes-vm/state
|
|
sudo mkdir -p /var/lib/hermes-vm/snapshots
|
|
|
|
# BEFORE creating any file inside them
|
|
sudo chattr +C /var/lib/hermes-vm/system /var/lib/hermes-vm/state
|
|
sudo btrfs property set /var/lib/hermes-vm/system compression none
|
|
sudo btrfs property set /var/lib/hermes-vm/state compression none
|
|
|
|
sudo truncate -s 80G /var/lib/hermes-vm/system/system.raw
|
|
sudo truncate -s 20G /var/lib/hermes-vm/state/state.raw
|
|
sudo mkfs.ext4 -L hermes-state /var/lib/hermes-vm/state/state.raw
|
|
```
|
|
|
|
**The `chattr +C` ordering matters.** Btrfs is copy-on-write and so is a VM
|
|
writing to a disk image; stacked, every guest write forks an extent and the
|
|
image fragments badly. `+C` turns that off — but only for files created
|
|
afterwards. Set it on empty subvolumes and confirm with `lsattr` before
|
|
writing anything. If you get it backwards, recreate the subvolume and copy
|
|
the image in fresh.
|
|
|
|
Separate subvolumes because a btrfs snapshot covers a whole subvolume. Keeping
|
|
the system image out of the state subvolume is what makes snapshots small.
|
|
|
|
### 2. Generate the cache signing key
|
|
|
|
The host serves its store to the guest as a read-only binary cache, so the
|
|
guest doesn't re-download from the internet what you already have.
|
|
|
|
```bash
|
|
sudo mkdir -p /var/lib/secrets
|
|
sudo nix-store --generate-binary-cache-key hermes-cache-1 \
|
|
/var/lib/secrets/hermes-cache.secret /var/lib/secrets/hermes-cache.pub
|
|
sudo chown nix-serve /var/lib/secrets/hermes-cache.secret
|
|
sudo chmod 0400 /var/lib/secrets/hermes-cache.secret
|
|
cat /var/lib/secrets/hermes-cache.pub
|
|
```
|
|
|
|
### 3. Write `/etc/hermenix/local.nix`
|
|
|
|
Host-specific settings live here, not in this repo — see
|
|
`guest/local.nix.example` for the reference shape:
|
|
|
|
```nix
|
|
{
|
|
cachePublicKey = "hermes-cache-1:AbCd..."; # the key you just printed; leave
|
|
# empty and the guest just skips
|
|
# the host cache
|
|
authorizedKeys = [ "ssh-ed25519 AAAA... you@laptop" ];
|
|
}
|
|
```
|
|
|
|
`sudo mkdir -p /etc/hermenix` first if it doesn't exist.
|
|
|
|
### 4. Build the image and write it to the system disk
|
|
|
|
The two extra substituter options stop you compiling Determinate Nix from
|
|
source. `--impure` is needed because `guest/common.nix` reads
|
|
`/etc/hermenix/local.nix` from outside this flake's own source tree —
|
|
deliberately, so the flake itself stays generic and shareable while your
|
|
settings stay on your host.
|
|
|
|
```bash
|
|
nix build --impure git+ssh://gitea@minipc1.lab.internal:22/ari/hermenix.git#hermes-guest-image \
|
|
--option extra-substituters https://install.determinate.systems \
|
|
--option extra-trusted-public-keys cache.flakehub.com-3:hJuILl5sVK4iKm86JzgdXW12Y2Hwd5G07qKtHTOcDCM=
|
|
|
|
# notrunc keeps the disk at its full size; the image is smaller
|
|
sudo dd if=result/nixos.img of=/var/lib/hermes-vm/system/system.raw \
|
|
bs=4M conv=notrunc,fsync status=progress
|
|
```
|
|
|
|
(Building from a local checkout instead — e.g. while developing this flake —
|
|
works the same way with `.` in place of the git URL.)
|
|
|
|
### 5. Add the module to your system flake
|
|
|
|
```nix
|
|
{
|
|
inputs.hermenix.url = "git+ssh://gitea@minipc1.lab.internal:22/ari/hermenix.git";
|
|
|
|
outputs = { nixpkgs, hermenix, ... }: {
|
|
nixosConfigurations.myserver = nixpkgs.lib.nixosSystem {
|
|
system = "x86_64-linux";
|
|
modules = [ hermenix.nixosModules.default ./configuration.nix ];
|
|
};
|
|
};
|
|
}
|
|
```
|
|
|
|
```nix
|
|
services.hermenix = {
|
|
enable = true;
|
|
disk.baseDir = "/var/lib/hermes-vm";
|
|
network.externalInterface = "eno1"; # your uplink
|
|
vcpu = 4;
|
|
mem = 8192;
|
|
};
|
|
```
|
|
|
|
`sudo nixos-rebuild switch`, then `hermenix vm console` to watch it boot.
|
|
|
|
### 6. First boot, inside the guest
|
|
|
|
```bash
|
|
hermenix vm ssh
|
|
|
|
# Confirm the disk labels match guest/hardware.nix BEFORE any rebuild
|
|
lsblk -f
|
|
|
|
# FlakeHub Cache login. Device-code flow: it prints a URL you approve in a
|
|
# browser anywhere, so the headless guest never needs one.
|
|
determinate-nixd login
|
|
|
|
# Pin the seeded flake
|
|
sudo nix flake lock /etc/nixos
|
|
```
|
|
|
|
The Hermes CLI installs itself automatically on first boot (`systemd.services.hermes-install`
|
|
in `common.nix`) — it runs unattended, so there's nothing to run for that
|
|
part. Give it a few minutes (it's compiling native Node modules); watch it
|
|
with `journalctl --user -u hermes-install -f` if you want to see it happen.
|
|
Once it's done:
|
|
|
|
```bash
|
|
hermes setup
|
|
hermes gateway install
|
|
```
|
|
|
|
These two need real input (model provider, API keys, messaging platform
|
|
tokens), so they stay manual.
|
|
|
|
The identity seed is already in place — `SOUL.md`, a starter `config.yaml`, a
|
|
`USER.md` and a skill covering safe self-rebuilds. Back on the host, check
|
|
the gateway came up and that the unit name matches the default:
|
|
|
|
```bash
|
|
hermenix ctl units
|
|
hermenix ctl status
|
|
hermenix ctl soul
|
|
```
|
|
|
|
> **Check the labels in step 6 before the first rebuild.**
|
|
> `guest/hardware.nix` assumes the root partition is labelled `nixos` and the
|
|
> ESP `ESP`. If the image generator used different ones, fix that file first.
|
|
> Otherwise the agent's first `nixos-rebuild switch` produces a system that
|
|
> won't boot and you're restoring a snapshot on day one.
|
|
|
|
From here `hermes update` self-updates in place and
|
|
`sudo nixos-rebuild switch --flake /etc/nixos#hermes` is a durable change to
|
|
the machine. Neither needs anything from you.
|
|
|
|
---
|
|
|
|
## Running it on a remote host
|
|
|
|
`setup` and `migrate` both act on whatever machine they run on — local
|
|
disks, local sudo, vsock into the local guest. There's no way around SSHing
|
|
into the target host first. `hermenix remote` does that SSH step for you:
|
|
|
|
```bash
|
|
hermenix remote fwdesk.lab.internal setup
|
|
hermenix remote fwdesk.lab.internal "setup --clean"
|
|
hermenix remote fwdesk.lab.internal "migrate /var/lib/hermes"
|
|
```
|
|
|
|
Quote the subcommand and its own flags/args as **one string** whenever
|
|
there's more than just the bare subcommand name — nu's own argument parser
|
|
treats any bare `--flag`-shaped word as an attempt to match a flag on
|
|
`remote` itself, so `hermenix remote host setup --clean` (unquoted) fails
|
|
before it ever reaches the SSH step.
|
|
|
|
It runs `nix run <flake>#hermenix -- <the rest of the command>` directly on
|
|
the remote host — no checkout involved, on either end. The remote host does
|
|
need its own SSH access to Gitea (the same access a `git clone` would need).
|
|
Override the flake URL with `--flake` if you're pointing at a fork or a
|
|
branch.
|
|
|
|
It connects with `ssh -t`, which allocates a real pty — required for the
|
|
interactive prompts in `setup`/`migrate` to work at all over the connection.
|
|
|
|
---
|
|
|
|
## SSH into the guest
|
|
|
|
```bash
|
|
hermenix vm ssh # interactive shell
|
|
hermenix vm ssh "systemctl status" # run a command and exit
|
|
hermenix vm console # follow the VM's boot/journal output
|
|
```
|
|
|
|
`ssh` takes its remote command as a single string, not separate words — quote
|
|
anything past a single bare word (nu's own parser otherwise tries to match
|
|
`--flag`-shaped tokens against `vm ssh`'s own flags, and errors on extra
|
|
positional words either way).
|
|
|
|
This goes over **vsock**, not the network. There is no listening SSH port
|
|
exposed anywhere, and the guest needs no inbound network path for you to
|
|
reach it. `console` is just `journalctl -u hermes-vm -f` on the host — a
|
|
read-only follow of the VM unit's journal, useful for watching it boot or
|
|
diagnosing why SSH won't come up, but it can't take input. There's currently
|
|
no interactive fallback for when the guest itself is unresponsive.
|
|
|
|
Under the hood `ssh` is `ssh -o ProxyCommand='socat - VSOCK-CONNECT:<cid>:22'`
|
|
(cid defaults to 77, `services.hermenix.vsockCid`), if you'd rather drive it
|
|
by hand.
|
|
|
|
The VM itself is a systemd unit: `hermenix vm start`/`stop`/`restart` control
|
|
it directly, same as `systemctl {start,stop,restart} hermes-vm` on the host.
|
|
|
|
---
|
|
|
|
## Bootstrapping Hermes's identity
|
|
|
|
The seed lives in `guest/hermes/`:
|
|
|
|
```
|
|
guest/hermes/
|
|
home/
|
|
SOUL.md -> ~/.hermes/SOUL.md
|
|
config.yaml -> ~/.hermes/config.yaml
|
|
memories/USER.md -> ~/.hermes/memories/USER.md
|
|
skills/nixos-self-config/SKILL.md -> ~/.hermes/skills/...
|
|
workspace/
|
|
AGENTS.md -> ~/workspace/AGENTS.md
|
|
```
|
|
|
|
Edit these before building the image and the agent wakes up already knowing
|
|
who it is, how you like to be spoken to, and how to rebuild its own system
|
|
without bricking itself.
|
|
|
|
**Seeding happens on first boot only, and this is deliberate.** Hermes
|
|
rewrites `SOUL.md` and its memories as it learns — that's the product. If the
|
|
seed were re-applied on every rebuild it would silently delete everything the
|
|
agent had worked out about you. So the copy is guarded by a marker file and
|
|
never repeats.
|
|
|
|
Two consequences:
|
|
|
|
- Editing `guest/hermes/` after first boot changes nothing by itself. Use
|
|
`hermenix ctl reseed` when you actually want to overwrite.
|
|
- The seed is the *starting* identity, not the current one. To see what
|
|
Hermes has made of itself, `hermenix ctl soul`.
|
|
|
|
### Machine facts vs identity
|
|
|
|
`/etc/hermes/ENVIRONMENT.md` is generated from `guest/common.nix` on every
|
|
rebuild and describes the machine: what's reachable, where the store is, what
|
|
persists. It's always accurate and there's no point editing it.
|
|
|
|
`SOUL.md` is identity and belongs to the agent. The seeded version points at
|
|
`ENVIRONMENT.md` rather than repeating it, so the two can't drift.
|
|
|
|
### Folding what it learned back into the seed
|
|
|
|
```bash
|
|
hermenix ctl pull ./hermes-home # excludes .env, sessions, state.db
|
|
diff -ru guest/hermes/home ./hermes-home
|
|
```
|
|
|
|
Worth doing occasionally. If the agent has written something genuinely good
|
|
into its own SOUL.md, you want that in the seed before you ever rebuild from
|
|
scratch.
|
|
|
|
---
|
|
|
|
## Managing Hermes from the host
|
|
|
|
`hermenix vm` drives the virtual machine. `hermenix ctl` drives the agent
|
|
inside it.
|
|
|
|
```bash
|
|
hermenix ctl soul # current identity
|
|
hermenix ctl soul edit # pull, $EDITOR, push, offer to restart
|
|
hermenix ctl memory edit # memories/USER.md
|
|
hermenix ctl conf edit # config.yaml
|
|
|
|
hermenix ctl secrets # keys, values masked
|
|
hermenix ctl secrets edit # edit hermesHome's .env, written 0600
|
|
hermenix ctl secrets show # values, when you really mean it
|
|
|
|
hermenix ctl status # gateway service
|
|
hermenix ctl logs # follow it
|
|
hermenix ctl restart
|
|
hermenix ctl units # find the unit name if the default is wrong
|
|
|
|
hermenix ctl reseed # overwrite identity from the seed
|
|
hermenix ctl pull [dir] # copy live identity to the host
|
|
```
|
|
|
|
Anything unrecognised is passed straight through to the `hermes` CLI in the
|
|
guest, with a TTY attached — quote it as one string if it's more than a
|
|
single word (same rule as `vm ssh` above):
|
|
|
|
```bash
|
|
hermenix ctl chat
|
|
hermenix ctl update
|
|
hermenix ctl "sessions list"
|
|
hermenix ctl doctor
|
|
```
|
|
|
|
The `edit` variants pull the file out over vsock, open your `$EDITOR`
|
|
locally, write it back, and offer to restart the gateway. Secrets go in at
|
|
mode 0600 and never pass through the Nix store — which is the whole reason
|
|
there's no declarative option for them.
|
|
|
|
If `status` and `logs` come back empty, the gateway unit is probably named
|
|
something other than `hermes-agent`. Run `hermenix ctl units` and set
|
|
`services.hermenix.gatewayUnit` to match; upstream has used more than one
|
|
name for it.
|
|
|
|
## Seeing what the guest is running
|
|
|
|
```bash
|
|
hermenix vm config
|
|
```
|
|
|
|
Prints the current system store path, the last ten generations, the git log
|
|
of `/etc/nixos`, and anything uncommitted. That last one matters: the agent
|
|
editing files without committing is the usual reason the config on disk and
|
|
the running system disagree.
|
|
|
|
To read or diff the config locally:
|
|
|
|
```bash
|
|
hermenix vm config-pull ./guest-live
|
|
diff -ru guest/ guest-live/
|
|
```
|
|
|
|
Expect that diff to grow. The `guest/` directory in this repo is the seed
|
|
that was written on first boot; it is deliberately not kept in sync. Once the
|
|
agent edits its config, this repo and the guest diverge permanently, and a
|
|
host rebuild will not fix a guest config problem.
|
|
|
|
---
|
|
|
|
## Letting the guest reach a service on your network
|
|
|
|
By default the guest can reach the public internet and nothing private. To
|
|
open a single endpoint — say Ollama on another machine:
|
|
|
|
```nix
|
|
services.hermenix.lanAccess = [
|
|
{ address = "192.168.1.50"; port = 11434; } # ollama
|
|
{ address = "192.168.1.50"; port = 9000; proto = "udp"; }
|
|
];
|
|
```
|
|
|
|
Each entry punches one destination and port through the RFC1918 drop. The
|
|
guest reaches it as `192.168.1.50:11434` directly. Traffic is masqueraded, so
|
|
the other machine sees it arriving from your host's address, not the guest's.
|
|
|
|
There's also `allowLan = true`, which opens all private address space at
|
|
once. It exists, but reach for `lanAccess` instead — the whole point of this
|
|
design is that an agent which goes wrong can't go wandering.
|
|
|
|
### A service on the host itself
|
|
|
|
Different chain, different option. ComfyUI on the host, port 8188:
|
|
|
|
```nix
|
|
services.hermenix.hostAccess = [ { port = 8188; } ];
|
|
```
|
|
|
|
**The service must be bound somewhere the guest can see.** If it listens
|
|
on `127.0.0.1` the rule won't help, because the guest arrives on the bridge
|
|
address. Bind it to `10.77.0.1` (bridge only, invisible to your LAN) or
|
|
`0.0.0.0`:
|
|
|
|
```nix
|
|
services.comfyui.listenAddress = "10.77.0.1"; # or whatever your service calls it
|
|
```
|
|
|
|
From the guest that's then `http://10.77.0.1:8188`.
|
|
|
|
---
|
|
|
|
## Exposing a service the guest is running
|
|
|
|
Say the agent has built a web app on guest port 8080. Two halves.
|
|
|
|
**Guest side.** The agent opens its own firewall, in `/etc/nixos`:
|
|
|
|
```nix
|
|
networking.firewall.allowedTCPPorts = [ 22 8080 ];
|
|
```
|
|
|
|
**Host side.** Nothing at all, if you only want to reach it from the host —
|
|
host-to-guest is outbound traffic and already permitted:
|
|
|
|
```bash
|
|
curl http://10.77.0.2:8080
|
|
```
|
|
|
|
To publish it to the wider network:
|
|
|
|
```nix
|
|
services.hermenix.publish = [
|
|
{ hostPort = 8080; guestPort = 8080; }
|
|
];
|
|
```
|
|
|
|
That DNATs the port on your uplink to the guest and opens the matching
|
|
forward rule. `hostPort` and `guestPort` don't have to match, so you can park
|
|
an awkward guest port somewhere sensible.
|
|
|
|
A note on this one: publishing a port is the one change here that lets the
|
|
outside world start conversations with the agent's machine. Everything else
|
|
in this design is the guest reaching out. Put a reverse proxy with TLS and
|
|
auth in front of anything that isn't a toy, and keep the list short.
|
|
|
|
---
|
|
|
|
## Passing hardware to the guest
|
|
|
|
### Before reaching for this
|
|
|
|
If what you want is "the agent can use the GPU", passthrough is often the
|
|
wrong answer. It is exclusive — the host loses the card completely while the
|
|
VM runs — and it is the single largest reduction in isolation available in
|
|
this setup.
|
|
|
|
The alternative: keep Ollama or ComfyUI on the host where the GPU already
|
|
works, and open the port with `hostAccess`. The agent gets the capability
|
|
over HTTP and the device stays on your side of the boundary. That is usually
|
|
both easier and better.
|
|
|
|
Pass the hardware through when the guest genuinely needs the device itself —
|
|
CUDA in-process, a capture card, a radio, a display output.
|
|
|
|
### What passthrough costs you
|
|
|
|
A PCI device does DMA. The IOMMU is the only thing standing between a
|
|
passed-through device and host memory, which is why `passthrough.iommu` is a
|
|
hard requirement and the module refuses to build without it.
|
|
|
|
Two further constraints follow from how IOMMUs work:
|
|
|
|
- **Whole groups only.** Every device in an IOMMU group goes together. Check
|
|
yours before planning anything:
|
|
|
|
```bash
|
|
for d in /sys/kernel/iommu_groups/*/devices/*; do
|
|
echo "group $(basename "$(dirname "$(dirname "$d")")"): $(lspci -nns "$(basename "$d")")"
|
|
done | sort -V
|
|
```
|
|
|
|
If your GPU shares a group with a controller you need, you cannot pass it
|
|
cleanly. ACS override patches exist; they work by lying about isolation,
|
|
and they have no place in a setup built to contain an autonomous agent.
|
|
|
|
- **Every function.** A GPU is at least two entries — graphics and its HDMI
|
|
audio function.
|
|
|
|
### GPU
|
|
|
|
```bash
|
|
lspci -Dnn | grep -i vga # addresses and vendor:device IDs
|
|
```
|
|
|
|
```nix
|
|
services.hermenix.passthrough = {
|
|
iommu = "amd"; # or "intel"
|
|
pci = [
|
|
{ address = "0000:01:00.0"; id = "10de:2204"; } # graphics
|
|
{ address = "0000:01:00.1"; id = "10de:1aef"; } # HDMI audio
|
|
];
|
|
};
|
|
```
|
|
|
|
Setting `id` makes vfio-pci claim the card in the initrd, before `nvidia` or
|
|
`amdgpu` can. Do this for GPUs — unbinding a live graphics driver afterwards
|
|
is unreliable, and the failure mode is a host with no display. Devices given
|
|
only an `address` are bound at boot by a oneshot service instead, which is
|
|
fine for things you swap around.
|
|
|
|
Rebuild and **reboot** — early binding happens in the initrd. Then check:
|
|
|
|
```bash
|
|
lspci -nnk -d 10de:2204 | grep "Kernel driver" # want: vfio-pci
|
|
```
|
|
|
|
On the guest side, the agent installs the drivers itself. It owns
|
|
`/etc/nixos`, so `hardware.nvidia` and `nvidia-container-toolkit` are its
|
|
problem, not yours — which is rather the point of the arrangement.
|
|
|
|
### HDMI output
|
|
|
|
A display hanging off a passed-through GPU works once the guest drives the
|
|
card; nothing extra is needed here. A capture card or HDMI grabber is just
|
|
another PCI or USB device — find it with `lspci -Dnn` or `lsusb` and use the
|
|
matching section.
|
|
|
|
### USB devices and Bluetooth
|
|
|
|
Matched by vendor and product ID:
|
|
|
|
```bash
|
|
lsusb # e.g. "Bus 001 Device 004: ID 8087:0026 Intel ..."
|
|
```
|
|
|
|
```nix
|
|
services.hermenix.passthrough.usb = [
|
|
{ vendorId = "8087"; productId = "0026"; } # bluetooth radio
|
|
{ vendorId = "0403"; productId = "6001"; } # serial adapter
|
|
];
|
|
```
|
|
|
|
This adds an xHCI controller to the guest and forwards the matching devices.
|
|
No IOMMU requirement, no VFIO, and much milder than PCI passthrough — QEMU
|
|
mediates rather than handing over raw hardware.
|
|
|
|
Two things that catch people:
|
|
|
|
- **The host must not be using it.** For an integrated Bluetooth radio, keep
|
|
`btusb` off it: `boot.blacklistedKernelModules = [ "btusb" ];`, and don't
|
|
enable `hardware.bluetooth` on the host.
|
|
- **No hotplug.** QEMU resolves these at start, so a device that's unplugged
|
|
when the VM boots won't appear when you plug it in. `hermenix vm restart`
|
|
after plugging. Passing a whole USB *controller* through as a PCI device
|
|
gets you real hotplug for anything in its ports, at the cost of that
|
|
controller being exclusive to the guest.
|
|
|
|
### When it doesn't start
|
|
|
|
`hermenix vm console` first. Then, in rough order of likelihood: the device
|
|
didn't bind to vfio-pci (`lspci -nnk`), the IOMMU isn't on (`dmesg | grep -i
|
|
iommu`), or something in the group is still held by the host. The module
|
|
lifts `LimitMEMLOCK` for you when PCI passthrough is configured, since VFIO
|
|
pins the whole guest RAM allocation and the default limit otherwise produces
|
|
a memorably unhelpful error.
|
|
|
|
## Recovery
|
|
|
|
Four layers, cheapest first. The first two live inside the guest and cover
|
|
almost every real mistake.
|
|
|
|
```bash
|
|
# The agent's own config history
|
|
hermenix ctl soul # what it thinks it is now
|
|
hermenix vm ssh "git -C /etc/nixos log --oneline"
|
|
|
|
# Previous NixOS generations, also in the guest's boot menu
|
|
hermenix vm ssh "sudo nixos-rebuild switch --rollback"
|
|
|
|
# Snapshot of the state disk
|
|
hermenix vm snapshot # trigger one now, e.g. before a risky change
|
|
hermenix vm snapshots
|
|
hermenix vm rollback state-20260914-1400 # asks you to type 'rollback' to confirm
|
|
|
|
# Restore from backup
|
|
hermenix vm backup # trigger one now
|
|
hermenix vm backups
|
|
```
|
|
|
|
### Snapshots
|
|
|
|
Hourly by default, keeping 48. Only the state disk — the system disk isn't
|
|
snapshotted because `hermenix setup` rebuilds it from the flake in minutes.
|
|
|
|
The host fsfreezes the guest through qemu-guest-agent around each one, so the
|
|
snapshot captures a consistent filesystem rather than the equivalent of
|
|
pulling the power. Takes milliseconds; the VM keeps running.
|
|
|
|
`hermenix vm rollback` stops the VM, moves the current state subvolume aside as
|
|
`state.old-<timestamp>`, and restores the snapshot in its place. Nothing is
|
|
deleted, so a mistaken rollback is itself reversible. Clean the `state.old-*`
|
|
directories up yourself once you're happy.
|
|
|
|
**Keep retention tighter than you would on ZFS.** The state image is
|
|
nodatacow, and every live snapshot reintroduces copy-on-write for the blocks
|
|
it still references. Performance sags while snapshots exist and recovers as
|
|
they're pruned.
|
|
|
|
### Backups
|
|
|
|
```nix
|
|
services.hermenix.backup = {
|
|
enable = true;
|
|
repository = "sftp:backup@nas:/hermes";
|
|
passwordFile = "/var/lib/secrets/hermes-restic";
|
|
};
|
|
```
|
|
|
|
This backs up the guest's *state*, not its disk. Files are pulled over the
|
|
SSH channel that already exists and streamed into restic.
|
|
|
|
Two reasons it works that way. The backup stays portable — it restores onto
|
|
any filesystem, or a different machine entirely, and doesn't care that you're
|
|
on btrfs today. And it avoids loop-mounting a guest-written filesystem on the
|
|
host, which would mean the host kernel parsing data the agent controls. That
|
|
would undo a fair bit of what the rest of this design is for.
|
|
|
|
If the repository is on your LAN rather than the internet, note this runs on
|
|
the *host*, so it needs no `lanAccess` entry — the guest never touches it.
|
|
|
|
### When the guest won't boot at all
|
|
|
|
The system disk is disposable by design. Re-run `hermenix setup`, write a
|
|
fresh image to the system disk, and leave the state disk alone — Hermes comes
|
|
back with its identity, memories, workspace and NixOS config intact. That is
|
|
the intended repair, not a last resort.
|
|
|
|
### Starting completely fresh
|
|
|
|
`hermenix setup --clean` stops the VM, deletes the existing
|
|
`system.raw`/`state.raw`/UEFI vars, and drops the stale `known_hosts` entry
|
|
for the guest (a fresh state disk means a fresh SSH host key on next boot,
|
|
so the old entry would otherwise fail with a host-key-mismatch on your next
|
|
connect) — then walks through setup again for a genuinely clean first boot,
|
|
rather than the disk-exists, skip-ahead path. It leaves the btrfs
|
|
subvolumes and any snapshots alone (only the files inside them), and asks
|
|
you to type `clean` to confirm before touching anything.
|
|
|
|
---
|
|
|
|
## Nix flavour
|
|
|
|
The guest runs **Determinate Nix**, nixpkgs pinned to FlakeHub
|
|
(`f/NixOS/nixpkgs/0.1`, rolling; use `0` for current stable).
|
|
|
|
- `/etc/nix/nix.conf` belongs to Determinate and must not be edited. Custom
|
|
settings go in `/etc/nix/nix.custom.conf`, written from `guest/common.nix`.
|
|
There is deliberately no `nix.settings` block.
|
|
- No `nix.gc` either — `determinate-nixd` runs the collector.
|
|
- Lazy trees are on by default, which makes the agent's flake evaluations
|
|
much cheaper. That matters more than it sounds when something re-evaluates
|
|
nixpkgs on a schedule.
|
|
- Neither flake uses `inputs.nixpkgs.follows` for `determinate`. Determinate
|
|
warn it causes FlakeHub Cache misses.
|
|
|
|
Substituter order in the guest: FlakeHub Cache, your host cache,
|
|
`cache.nixos.org`. If FlakeHub Cache goes quiet, the login has expired — run
|
|
`determinate-nixd login` again.
|
|
|
|
The host stays on stock Nix. Nothing here requires changing that.
|
|
|
|
---
|
|
|
|
## What the guest can and cannot reach
|
|
|
|
| | Default |
|
|
|---|---|
|
|
| Public internet | Yes — it needs this for model APIs and messaging |
|
|
| Host Nix cache | Yes, one port, `hostCache.enable = false` closes it |
|
|
| Anything else on the host | No, unless in `hostAccess` |
|
|
| Your LAN and other VMs | No, unless in `lanAccess` |
|
|
| Inbound from the network | No, unless in `publish` |
|
|
| Host hardware | None, unless in `passthrough` |
|
|
|
|
The guest owns its kernel and its config, so nothing inside it is a security
|
|
boundary. The nftables rules in `modules/host.nix` are. QEMU runs as an
|
|
unprivileged `hermes-vm` user with its own seccomp sandbox and systemd
|
|
hardening, with a thin device model: virtio-blk, virtio-net, vsock, RNG,
|
|
serial. No virtiofs. `vhost-net` is off deliberately — it would move packet
|
|
handling into the host kernel in exchange for throughput one agent doesn't
|
|
need.
|
|
|
|
The host cache does let the guest read every store path on the host. Store
|
|
paths are world-readable anyway, so this isn't new exposure, but don't bake
|
|
secrets into your host's store.
|
|
|
|
---
|
|
|
|
## Known rough edges
|
|
|
|
- The Determinate NixOS module's option names for custom Nix settings aren't
|
|
published — only the nix-darwin `determinateNix.customSettings` equivalent
|
|
is. So `guest/common.nix` writes `/etc/nix/nix.custom.conf` via
|
|
`environment.etc`. If a future module version claims that file, you'll get
|
|
a conflict; switch to its option then.
|
|
- `determinate-nixd login` is manual. There's no way to bake the credential
|
|
into the image without parking a token somewhere it shouldn't be.
|
|
- `MemoryDenyWriteExecute = true` is correct under KVM but is the first thing
|
|
to flip if QEMU refuses to start.
|
|
- `pkgs.OVMF.firmware` / `pkgs.OVMF.variables` are current attribute names;
|
|
older nixpkgs put these under `pkgs.OVMF.fd` in an `FV/` subdirectory.
|
|
- PCI passthrough needs a reboot to take effect, because early vfio binding
|
|
happens in the initrd.
|
|
- Snapshots need btrfs. Everything else works on any filesystem; without it
|
|
you rely on backups alone for recovery.
|
|
- `state.old-*` directories left by `hermenix vm rollback` are not cleaned up
|
|
automatically, on purpose. Delete them yourself once you're sure.
|
|
- First boot builds a lot. Give it time.
|
|
- `services.hermenix.hermesHome` currently defaults to a path that doesn't
|
|
exist in the guest (it's built from the host's disk-image directory, not
|
|
the guest's mounted state disk) — set it explicitly on every deployment,
|
|
migrated or not. See "Migrating an existing deployment" below for the
|
|
value to use.
|
|
|
|
---
|
|
|
|
## Migrating an existing (e.g. Docker-based) Hermes deployment
|
|
|
|
If you already have a running Hermes with real data (`HERMES_HOME` pointing
|
|
at some directory full of `SOUL.md`, `memories/`, `sessions/`, `state.db`,
|
|
etc.):
|
|
|
|
```bash
|
|
nix run git+ssh://gitea@minipc1.lab.internal:22/ari/hermenix.git#hermenix -- migrate /path/to/existing/HERMES_HOME
|
|
```
|
|
|
|
Run `... -- migrate --help` for the full set of options
|
|
(vsock CID, guest user, destination path, hostname rewrites). It copies —
|
|
never moves or deletes — the source data, so the original stays a safe
|
|
fallback until you've verified the new VM. To run it from a different
|
|
machine than the one hosting the VM, see `hermenix remote` above — the same
|
|
`migrate /path/...` argument goes in the quoted command string.
|
|
|
|
The module's `services.hermenix.hermesHome` option lets `hermenix ctl` (and
|
|
your own `HERMES_HOME` export inside the guest) point at wherever that
|
|
existing flat layout lands, instead of the default seeded `.hermes` under the
|
|
state disk.
|
|
|
|
**Set `hermesHome` explicitly, even if you're not migrating.** The option's
|
|
shipped default doesn't currently resolve to a real path inside the guest —
|
|
it's built from the host's own disk-image directory rather than the guest's
|
|
mounted state disk, so `hermenix ctl soul`/`conf`/`secrets`/etc. will fail
|
|
against it unhelpfully until you override it. For the default seeded layout:
|
|
|
|
```nix
|
|
services.hermenix.hermesHome = "/var/lib/hermes/.hermes";
|
|
```
|
|
|
|
For a migrated flat layout, use whatever `--dest` you passed to `migrate`
|
|
(default `/var/lib/hermes`).
|
|
|
|
### Don't run it with `sudo`
|
|
|
|
`migrate` (like `hermenix vm ssh` and `hermenix ctl`) reaches the guest with
|
|
a plain `ssh` call over vsock, authenticating with whatever key or agent the
|
|
invoking user already has — the same identity that's in the guest's
|
|
`authorizedKeys`. Prefixing the command with `sudo` runs that `ssh` as
|
|
**root** instead, which has no matching key, and you'll hit:
|
|
|
|
```
|
|
hermes@hermes: Permission denied (publickey,keyboard-interactive).
|
|
```
|
|
|
|
If `hermenix vm ssh` works for you as your normal user, `migrate` doesn't
|
|
need anything more privileged — run it the same way.
|
|
|
|
### If some of the source can't be read
|
|
|
|
Old flat `HERMES_HOME` directories, especially ones left behind by a
|
|
rootless-Docker deployment, sometimes contain a stray entry owned by a
|
|
container-internal uid with no mapping on the host — unreadable by anyone
|
|
outside the container. `migrate` tars the source with `--ignore-failed-read`,
|
|
so a warning like this is expected and non-fatal:
|
|
|
|
```
|
|
tar: ./.hermes: Cannot open: Permission denied
|
|
```
|
|
|
|
Everything else still gets copied and the run still completes, including the
|
|
hostname rewrite step. If you actually need what's in the unreadable entry,
|
|
grab it separately, as root, before or after the main migration — e.g.:
|
|
|
|
```bash
|
|
sudo tar -C /path/to/HERMES_HOME/.hermes -cf - . \
|
|
| hermenix vm ssh 'mkdir -p /var/lib/hermes/.hermes-old && tar -C /var/lib/hermes/.hermes-old -xf -'
|
|
```
|
|
|
|
### `hermes setup` fails with "docker not found on PATH. Cannot route to container"
|
|
|
|
`hermenix migrate` automatically excludes `.local/bin` and `.hermes-installed`
|
|
from the migration tar, so this should not happen with current versions.
|
|
If you see it after a migration with an older build, the source's
|
|
`.local/bin/hermes` (a Docker-routing launcher from the previous environment)
|
|
was copied into the guest and is shadowing the native install.
|
|
|
|
Fix inside the guest:
|
|
|
|
```bash
|
|
# Remove the stale launcher
|
|
rm -f ~/.local/bin/hermes
|
|
|
|
# Reinstall the native hermes binary
|
|
curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh \
|
|
| bash -s -- --non-interactive --skip-setup
|
|
|
|
# Verify it's a real binary, not a shell script
|
|
file ~/.local/bin/hermes
|
|
|
|
# Now proceed with setup
|
|
hermes setup
|
|
```
|
|
|
|
If `hermes-install.service` never ran because `.hermes-installed` was also
|
|
migrated from the source, you may instead let systemd do it:
|
|
|
|
```bash
|
|
rm -f ~/.local/bin/hermes ~/.hermes-installed
|
|
sudo systemctl start hermes-install
|
|
journalctl -fu hermes-install # watch until complete
|
|
hermes setup
|
|
```
|
|
|
|
|
|
## Hacking on this repo
|
|
|
|
The devshell provides nushell, nufmt, and pre-commit hooks. Enter it once:
|
|
|
|
```bash
|
|
direnv allow
|
|
```
|
|
|
|
On first entry devenv installs the pre-commit hooks automatically. Hooks run
|
|
on every commit, including:
|
|
|
|
- **nufmt** — formats all `*.nu` files in place
|
|
- **gen-completions** — reruns `nu/gen-completions.nu` and stages
|
|
`nu/completions.nu` whenever a usage block in the source files changes
|
|
- **nu-tests** — runs the CLI test suite (see below) whenever anything
|
|
under `nu/` changes
|
|
|
|
### Testing
|
|
|
|
Two test suites. Neither is ever run for you by an agent working in this
|
|
repo — they're for you (or CI) to run.
|
|
|
|
**CLI tests** (`nu/tests/`), via [nutest](https://github.com/vyadh/nutest):
|
|
|
|
```bash
|
|
runtest # everything (bin/ is on PATH inside the devShell)
|
|
runtest vm_test # just nu/tests/vm_test.nu
|
|
runtest-debug vm_test # load vm_test.nu into an interactive nu session instead
|
|
```
|
|
|
|
Covers the CLI's pure helper functions (path/env defaults, `shquote`,
|
|
`normalize-size`, the `gen-completions.nu` parser) and a smoke test that the
|
|
whole `hermenix`/`vm`/`ctl` subcommand tree parses and dispatches correctly
|
|
— without ever invoking a subcommand's real body (no ssh, no disk writes,
|
|
no `nix build`).
|
|
|
|
**Guest VM test** (`guest/vm-test.nix`): boots `guest/common.nix` — the same
|
|
module set `hermes-guest-image` is built from — in an ephemeral, throwaway
|
|
NixOS test VM. No real disk under `/var/lib/hermes-vm/**` is ever touched:
|
|
|
|
```bash
|
|
runvmtest # plain pass/fail (nix build .#checks.x86_64-linux.guest-vm --impure -L)
|
|
runvmtest-debug # build the interactive driver and drop into it (ptpython REPL)
|
|
```
|
|
|
|
Checks first-boot seeding (`~/.hermes`, `~/workspace`, the `/etc/nixos`
|
|
symlink), that seeding is idempotent across a reboot (the whole point of
|
|
the `.seeded` marker), the `hermes` user/sudo setup, and SSH. It
|
|
deliberately does not cover the real `hermes-install` script (no internet
|
|
route inside the test VM) or `modules/host.nix`'s own `hermes-vm` QEMU
|
|
service — both stay manual, operator-run checks.
|
|
|
|
`runvmtest-debug` gives you `start_all()`, `machine.wait_for_unit(...)`, and
|
|
`machine.shell_interact()` (a live shell inside the guest) to step through a
|
|
failure by hand instead of just getting pass/fail.
|