Files
hermenix/README.md
T
2026-09-17 16:26:15 -04:00

921 lines
34 KiB
Markdown

# hermenix
Hermes Agent in a NixOS VM that owns its own bootloader and its own
`/nix/store`, on two btrfs-backed disks, under an unprivileged hardened QEMU.
The agent can rebuild its own system and update itself, durably. It shares no
filesystem with the host, and reaches nothing on your network unless you say
so.
One CLI, `hermenix`, with five subcommands:
| | For |
|---|---|
| `hermenix setup` | one-off host provisioning — disks, cache key, image build |
| `hermenix migrate <src>` | carry an existing flat `HERMES_HOME` into the guest |
| `hermenix remote <host> "<cmd>"` | run `setup`/`migrate` on a remote host over SSH |
| `hermenix vm <sub>` | control the VM itself — start/stop/console/snapshots/backups |
| `hermenix ctl <sub>` | drive the agent inside the VM — identity/config/secrets/gateway |
---
## Installation
### Guided
The `hermenix` CLI (`nu/hermenix.nu`) is a flake package — no checkout needed
anywhere. Run it straight from its Gitea URL:
```bash
nix run git+ssh://gitea@minipc1.lab.internal:22/ari/hermenix.git#hermenix -- setup
```
(Or `nix profile install git+ssh://gitea@minipc1.lab.internal:22/ari/hermenix.git#hermenix`
once, then just `hermenix setup` from here on — or get it declaratively via
the nixos-setup overlay, same as any other package. Any of these work from
any directory, on any host with SSH access to Gitea.)
Walks the host-side steps: checks prerequisites, creates the disks, generates
the cache signing key, writes `/etc/hermenix/local.nix` from your SSH keys,
builds the bootstrap image and writes it to the disk. It prints the module
snippet to paste into your own configuration at the end, with your uplink
interface already filled in.
Safe to re-run — every step detects what's already done and offers to skip,
and nothing destructive happens without an explicit confirmation.
It deliberately stops short of `nixos-rebuild`, since that's your config.
The rest of this section is what the command does, for when you'd rather do
it by hand or something goes sideways.
### 1. Create the disks
Two of them, and the split is the whole design:
| | Holds | Size | Snapshotted | Backed up |
|---|---|---|---|---|
| **system** | NixOS, `/nix/store` | 80G | No | No |
| **state** | identity, memories, workspace, `/etc/nixos` | 20G | Yes | Yes |
The system disk is large, takes all the write churn, and is entirely
rebuildable from this flake — so there's no point protecting it. The state
disk holds the only things that can't be reproduced, which makes snapshots
cheap and backups quick.
```bash
sudo btrfs subvolume create /var/lib/hermes-vm/system
sudo btrfs subvolume create /var/lib/hermes-vm/state
sudo mkdir -p /var/lib/hermes-vm/snapshots
# BEFORE creating any file inside them
sudo chattr +C /var/lib/hermes-vm/system /var/lib/hermes-vm/state
sudo btrfs property set /var/lib/hermes-vm/system compression none
sudo btrfs property set /var/lib/hermes-vm/state compression none
sudo truncate -s 80G /var/lib/hermes-vm/system/system.raw
sudo truncate -s 20G /var/lib/hermes-vm/state/state.raw
sudo mkfs.ext4 -L hermes-state /var/lib/hermes-vm/state/state.raw
```
**The `chattr +C` ordering matters.** Btrfs is copy-on-write and so is a VM
writing to a disk image; stacked, every guest write forks an extent and the
image fragments badly. `+C` turns that off — but only for files created
afterwards. Set it on empty subvolumes and confirm with `lsattr` before
writing anything. If you get it backwards, recreate the subvolume and copy
the image in fresh.
Separate subvolumes because a btrfs snapshot covers a whole subvolume. Keeping
the system image out of the state subvolume is what makes snapshots small.
### 2. Generate the cache signing key
The host serves its store to the guest as a read-only binary cache, so the
guest doesn't re-download from the internet what you already have.
```bash
sudo mkdir -p /var/lib/secrets
sudo nix-store --generate-binary-cache-key hermes-cache-1 \
/var/lib/secrets/hermes-cache.secret /var/lib/secrets/hermes-cache.pub
sudo chown nix-serve /var/lib/secrets/hermes-cache.secret
sudo chmod 0400 /var/lib/secrets/hermes-cache.secret
cat /var/lib/secrets/hermes-cache.pub
```
### 3. Write `/etc/hermenix/local.nix`
Host-specific settings live here, not in this repo — see
`guest/local.nix.example` for the reference shape:
```nix
{
cachePublicKey = "hermes-cache-1:AbCd..."; # the key you just printed; leave
# empty and the guest just skips
# the host cache
authorizedKeys = [ "ssh-ed25519 AAAA... you@laptop" ];
}
```
`sudo mkdir -p /etc/hermenix` first if it doesn't exist.
### 4. Build the image and write it to the system disk
The two extra substituter options stop you compiling Determinate Nix from
source. `--impure` is needed because `guest/common.nix` reads
`/etc/hermenix/local.nix` from outside this flake's own source tree —
deliberately, so the flake itself stays generic and shareable while your
settings stay on your host.
```bash
nix build --impure git+ssh://gitea@minipc1.lab.internal:22/ari/hermenix.git#hermes-guest-image \
--option extra-substituters https://install.determinate.systems \
--option extra-trusted-public-keys cache.flakehub.com-3:hJuILl5sVK4iKm86JzgdXW12Y2Hwd5G07qKtHTOcDCM=
# notrunc keeps the disk at its full size; the image is smaller
sudo dd if=result/nixos.img of=/var/lib/hermes-vm/system/system.raw \
bs=4M conv=notrunc,fsync status=progress
```
(Building from a local checkout instead — e.g. while developing this flake —
works the same way with `.` in place of the git URL.)
### 5. Add the module to your system flake
```nix
{
inputs.hermenix.url = "git+ssh://gitea@minipc1.lab.internal:22/ari/hermenix.git";
outputs = { nixpkgs, hermenix, ... }: {
nixosConfigurations.myserver = nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
modules = [ hermenix.nixosModules.default ./configuration.nix ];
};
};
}
```
```nix
services.hermenix = {
enable = true;
disk.baseDir = "/var/lib/hermes-vm";
network.externalInterface = "eno1"; # your uplink
vcpu = 4;
mem = 8192;
};
```
`sudo nixos-rebuild switch`, then `hermenix vm console` to watch it boot.
### 6. First boot, inside the guest
```bash
hermenix vm ssh
# Confirm the disk labels match guest/hardware.nix BEFORE any rebuild
lsblk -f
# FlakeHub Cache login. Device-code flow: it prints a URL you approve in a
# browser anywhere, so the headless guest never needs one.
determinate-nixd login
# Pin the seeded flake
sudo nix flake lock /etc/nixos
```
The Hermes CLI installs itself automatically on first boot (`systemd.services.hermes-install`
in `common.nix`) — it runs unattended, so there's nothing to run for that
part. Give it a few minutes (it's compiling native Node modules); watch it
with `journalctl --user -u hermes-install -f` if you want to see it happen.
Once it's done:
```bash
hermes setup
hermes gateway install
```
These two need real input (model provider, API keys, messaging platform
tokens), so they stay manual.
The identity seed is already in place — `SOUL.md`, a starter `config.yaml`, a
`USER.md` and a skill covering safe self-rebuilds. Back on the host, check
the gateway came up and that the unit name matches the default:
```bash
hermenix ctl units
hermenix ctl status
hermenix ctl soul
```
> **Check the labels in step 6 before the first rebuild.**
> `guest/hardware.nix` assumes the root partition is labelled `nixos` and the
> ESP `ESP`. If the image generator used different ones, fix that file first.
> Otherwise the agent's first `nixos-rebuild switch` produces a system that
> won't boot and you're restoring a snapshot on day one.
From here `hermes update` self-updates in place and
`sudo nixos-rebuild switch --flake /etc/nixos#hermes` is a durable change to
the machine. Neither needs anything from you.
---
## Running it on a remote host
`setup` and `migrate` both act on whatever machine they run on — local
disks, local sudo, vsock into the local guest. There's no way around SSHing
into the target host first. `hermenix remote` does that SSH step for you:
```bash
hermenix remote fwdesk.lab.internal setup
hermenix remote fwdesk.lab.internal "setup --clean"
hermenix remote fwdesk.lab.internal "migrate /var/lib/hermes"
```
Quote the subcommand and its own flags/args as **one string** whenever
there's more than just the bare subcommand name — nu's own argument parser
treats any bare `--flag`-shaped word as an attempt to match a flag on
`remote` itself, so `hermenix remote host setup --clean` (unquoted) fails
before it ever reaches the SSH step.
It runs `nix run <flake>#hermenix -- <the rest of the command>` directly on
the remote host — no checkout involved, on either end. The remote host does
need its own SSH access to Gitea (the same access a `git clone` would need).
Override the flake URL with `--flake` if you're pointing at a fork or a
branch.
It connects with `ssh -t`, which allocates a real pty — required for the
interactive prompts in `setup`/`migrate` to work at all over the connection.
---
## SSH into the guest
```bash
hermenix vm ssh # interactive shell
hermenix vm ssh "systemctl status" # run a command and exit
hermenix vm console # follow the VM's boot/journal output
```
`ssh` takes its remote command as a single string, not separate words — quote
anything past a single bare word (nu's own parser otherwise tries to match
`--flag`-shaped tokens against `vm ssh`'s own flags, and errors on extra
positional words either way).
This goes over **vsock**, not the network. There is no listening SSH port
exposed anywhere, and the guest needs no inbound network path for you to
reach it. `console` is just `journalctl -u hermes-vm -f` on the host — a
read-only follow of the VM unit's journal, useful for watching it boot or
diagnosing why SSH won't come up, but it can't take input. There's currently
no interactive fallback for when the guest itself is unresponsive.
Under the hood `ssh` is `ssh -o ProxyCommand='socat - VSOCK-CONNECT:<cid>:22'`
(cid defaults to 77, `services.hermenix.vsockCid`), if you'd rather drive it
by hand.
The VM itself is a systemd unit: `hermenix vm start`/`stop`/`restart` control
it directly, same as `systemctl {start,stop,restart} hermes-vm` on the host.
---
## Bootstrapping Hermes's identity
The seed lives in `guest/hermes/`:
```
guest/hermes/
home/
SOUL.md -> ~/.hermes/SOUL.md
config.yaml -> ~/.hermes/config.yaml
memories/USER.md -> ~/.hermes/memories/USER.md
skills/nixos-self-config/SKILL.md -> ~/.hermes/skills/...
workspace/
AGENTS.md -> ~/workspace/AGENTS.md
```
Edit these before building the image and the agent wakes up already knowing
who it is, how you like to be spoken to, and how to rebuild its own system
without bricking itself.
**Seeding happens on first boot only, and this is deliberate.** Hermes
rewrites `SOUL.md` and its memories as it learns — that's the product. If the
seed were re-applied on every rebuild it would silently delete everything the
agent had worked out about you. So the copy is guarded by a marker file and
never repeats.
Two consequences:
- Editing `guest/hermes/` after first boot changes nothing by itself. Use
`hermenix ctl reseed` when you actually want to overwrite.
- The seed is the *starting* identity, not the current one. To see what
Hermes has made of itself, `hermenix ctl soul`.
### Machine facts vs identity
`/etc/hermes/ENVIRONMENT.md` is generated from `guest/common.nix` on every
rebuild and describes the machine: what's reachable, where the store is, what
persists. It's always accurate and there's no point editing it.
`SOUL.md` is identity and belongs to the agent. The seeded version points at
`ENVIRONMENT.md` rather than repeating it, so the two can't drift.
### Folding what it learned back into the seed
```bash
hermenix ctl pull ./hermes-home # excludes .env, sessions, state.db
diff -ru guest/hermes/home ./hermes-home
```
Worth doing occasionally. If the agent has written something genuinely good
into its own SOUL.md, you want that in the seed before you ever rebuild from
scratch.
---
## Managing Hermes from the host
`hermenix vm` drives the virtual machine. `hermenix ctl` drives the agent
inside it.
```bash
hermenix ctl soul # current identity
hermenix ctl soul edit # pull, $EDITOR, push, offer to restart
hermenix ctl memory edit # memories/USER.md
hermenix ctl conf edit # config.yaml
hermenix ctl secrets # keys, values masked
hermenix ctl secrets edit # edit hermesHome's .env, written 0600
hermenix ctl secrets show # values, when you really mean it
hermenix ctl status # gateway service
hermenix ctl logs # follow it
hermenix ctl restart
hermenix ctl units # find the unit name if the default is wrong
hermenix ctl reseed # overwrite identity from the seed
hermenix ctl pull [dir] # copy live identity to the host
```
Anything unrecognised is passed straight through to the `hermes` CLI in the
guest, with a TTY attached — quote it as one string if it's more than a
single word (same rule as `vm ssh` above):
```bash
hermenix ctl chat
hermenix ctl update
hermenix ctl "sessions list"
hermenix ctl doctor
```
The `edit` variants pull the file out over vsock, open your `$EDITOR`
locally, write it back, and offer to restart the gateway. Secrets go in at
mode 0600 and never pass through the Nix store — which is the whole reason
there's no declarative option for them.
If `status` and `logs` come back empty, the gateway unit is probably named
something other than `hermes-agent`. Run `hermenix ctl units` and set
`services.hermenix.gatewayUnit` to match; upstream has used more than one
name for it.
## Seeing what the guest is running
```bash
hermenix vm config
```
Prints the current system store path, the last ten generations, the git log
of `/etc/nixos`, and anything uncommitted. That last one matters: the agent
editing files without committing is the usual reason the config on disk and
the running system disagree.
To read or diff the config locally:
```bash
hermenix vm config-pull ./guest-live
diff -ru guest/ guest-live/
```
Expect that diff to grow. The `guest/` directory in this repo is the seed
that was written on first boot; it is deliberately not kept in sync. Once the
agent edits its config, this repo and the guest diverge permanently, and a
host rebuild will not fix a guest config problem.
---
## Letting the guest reach a service on your network
By default the guest can reach the public internet and nothing private. To
open a single endpoint — say Ollama on another machine:
```nix
services.hermenix.lanAccess = [
{ address = "192.168.1.50"; port = 11434; } # ollama
{ address = "192.168.1.50"; port = 9000; proto = "udp"; }
];
```
Each entry punches one destination and port through the RFC1918 drop. The
guest reaches it as `192.168.1.50:11434` directly. Traffic is masqueraded, so
the other machine sees it arriving from your host's address, not the guest's.
There's also `allowLan = true`, which opens all private address space at
once. It exists, but reach for `lanAccess` instead — the whole point of this
design is that an agent which goes wrong can't go wandering.
### A service on the host itself
Different chain, different option. ComfyUI on the host, port 8188:
```nix
services.hermenix.hostAccess = [ { port = 8188; } ];
```
**The service must be bound somewhere the guest can see.** If it listens
on `127.0.0.1` the rule won't help, because the guest arrives on the bridge
address. Bind it to `10.77.0.1` (bridge only, invisible to your LAN) or
`0.0.0.0`:
```nix
services.comfyui.listenAddress = "10.77.0.1"; # or whatever your service calls it
```
From the guest that's then `http://10.77.0.1:8188`.
---
## Exposing a service the guest is running
Say the agent has built a web app on guest port 8080. Two halves.
**Guest side.** The agent opens its own firewall, in `/etc/nixos`:
```nix
networking.firewall.allowedTCPPorts = [ 22 8080 ];
```
**Host side.** Nothing at all, if you only want to reach it from the host —
host-to-guest is outbound traffic and already permitted:
```bash
curl http://10.77.0.2:8080
```
To publish it to the wider network:
```nix
services.hermenix.publish = [
{ hostPort = 8080; guestPort = 8080; }
];
```
That DNATs the port on your uplink to the guest and opens the matching
forward rule. `hostPort` and `guestPort` don't have to match, so you can park
an awkward guest port somewhere sensible.
A note on this one: publishing a port is the one change here that lets the
outside world start conversations with the agent's machine. Everything else
in this design is the guest reaching out. Put a reverse proxy with TLS and
auth in front of anything that isn't a toy, and keep the list short.
---
## Passing hardware to the guest
### Before reaching for this
If what you want is "the agent can use the GPU", passthrough is often the
wrong answer. It is exclusive — the host loses the card completely while the
VM runs — and it is the single largest reduction in isolation available in
this setup.
The alternative: keep Ollama or ComfyUI on the host where the GPU already
works, and open the port with `hostAccess`. The agent gets the capability
over HTTP and the device stays on your side of the boundary. That is usually
both easier and better.
Pass the hardware through when the guest genuinely needs the device itself —
CUDA in-process, a capture card, a radio, a display output.
### What passthrough costs you
A PCI device does DMA. The IOMMU is the only thing standing between a
passed-through device and host memory, which is why `passthrough.iommu` is a
hard requirement and the module refuses to build without it.
Two further constraints follow from how IOMMUs work:
- **Whole groups only.** Every device in an IOMMU group goes together. Check
yours before planning anything:
```bash
for d in /sys/kernel/iommu_groups/*/devices/*; do
echo "group $(basename "$(dirname "$(dirname "$d")")"): $(lspci -nns "$(basename "$d")")"
done | sort -V
```
If your GPU shares a group with a controller you need, you cannot pass it
cleanly. ACS override patches exist; they work by lying about isolation,
and they have no place in a setup built to contain an autonomous agent.
- **Every function.** A GPU is at least two entries — graphics and its HDMI
audio function.
### GPU
```bash
lspci -Dnn | grep -i vga # addresses and vendor:device IDs
```
```nix
services.hermenix.passthrough = {
iommu = "amd"; # or "intel"
pci = [
{ address = "0000:01:00.0"; id = "10de:2204"; } # graphics
{ address = "0000:01:00.1"; id = "10de:1aef"; } # HDMI audio
];
};
```
Setting `id` makes vfio-pci claim the card in the initrd, before `nvidia` or
`amdgpu` can. Do this for GPUs — unbinding a live graphics driver afterwards
is unreliable, and the failure mode is a host with no display. Devices given
only an `address` are bound at boot by a oneshot service instead, which is
fine for things you swap around.
Rebuild and **reboot** — early binding happens in the initrd. Then check:
```bash
lspci -nnk -d 10de:2204 | grep "Kernel driver" # want: vfio-pci
```
On the guest side, the agent installs the drivers itself. It owns
`/etc/nixos`, so `hardware.nvidia` and `nvidia-container-toolkit` are its
problem, not yours — which is rather the point of the arrangement.
### HDMI output
A display hanging off a passed-through GPU works once the guest drives the
card; nothing extra is needed here. A capture card or HDMI grabber is just
another PCI or USB device — find it with `lspci -Dnn` or `lsusb` and use the
matching section.
### USB devices and Bluetooth
Matched by vendor and product ID:
```bash
lsusb # e.g. "Bus 001 Device 004: ID 8087:0026 Intel ..."
```
```nix
services.hermenix.passthrough.usb = [
{ vendorId = "8087"; productId = "0026"; } # bluetooth radio
{ vendorId = "0403"; productId = "6001"; } # serial adapter
];
```
This adds an xHCI controller to the guest and forwards the matching devices.
No IOMMU requirement, no VFIO, and much milder than PCI passthrough — QEMU
mediates rather than handing over raw hardware.
Two things that catch people:
- **The host must not be using it.** For an integrated Bluetooth radio, keep
`btusb` off it: `boot.blacklistedKernelModules = [ "btusb" ];`, and don't
enable `hardware.bluetooth` on the host.
- **No hotplug.** QEMU resolves these at start, so a device that's unplugged
when the VM boots won't appear when you plug it in. `hermenix vm restart`
after plugging. Passing a whole USB *controller* through as a PCI device
gets you real hotplug for anything in its ports, at the cost of that
controller being exclusive to the guest.
### When it doesn't start
`hermenix vm console` first. Then, in rough order of likelihood: the device
didn't bind to vfio-pci (`lspci -nnk`), the IOMMU isn't on (`dmesg | grep -i
iommu`), or something in the group is still held by the host. The module
lifts `LimitMEMLOCK` for you when PCI passthrough is configured, since VFIO
pins the whole guest RAM allocation and the default limit otherwise produces
a memorably unhelpful error.
## Recovery
Four layers, cheapest first. The first two live inside the guest and cover
almost every real mistake.
```bash
# The agent's own config history
hermenix ctl soul # what it thinks it is now
hermenix vm ssh "git -C /etc/nixos log --oneline"
# Previous NixOS generations, also in the guest's boot menu
hermenix vm ssh "sudo nixos-rebuild switch --rollback"
# Snapshot of the state disk
hermenix vm snapshot # trigger one now, e.g. before a risky change
hermenix vm snapshots
hermenix vm rollback state-20260914-1400 # asks you to type 'rollback' to confirm
# Restore from backup
hermenix vm backup # trigger one now
hermenix vm backups
```
### Snapshots
Hourly by default, keeping 48. Only the state disk — the system disk isn't
snapshotted because `hermenix setup` rebuilds it from the flake in minutes.
The host fsfreezes the guest through qemu-guest-agent around each one, so the
snapshot captures a consistent filesystem rather than the equivalent of
pulling the power. Takes milliseconds; the VM keeps running.
`hermenix vm rollback` stops the VM, moves the current state subvolume aside as
`state.old-<timestamp>`, and restores the snapshot in its place. Nothing is
deleted, so a mistaken rollback is itself reversible. Clean the `state.old-*`
directories up yourself once you're happy.
**Keep retention tighter than you would on ZFS.** The state image is
nodatacow, and every live snapshot reintroduces copy-on-write for the blocks
it still references. Performance sags while snapshots exist and recovers as
they're pruned.
### Backups
```nix
services.hermenix.backup = {
enable = true;
repository = "sftp:backup@nas:/hermes";
passwordFile = "/var/lib/secrets/hermes-restic";
};
```
This backs up the guest's *state*, not its disk. Files are pulled over the
SSH channel that already exists and streamed into restic.
Two reasons it works that way. The backup stays portable — it restores onto
any filesystem, or a different machine entirely, and doesn't care that you're
on btrfs today. And it avoids loop-mounting a guest-written filesystem on the
host, which would mean the host kernel parsing data the agent controls. That
would undo a fair bit of what the rest of this design is for.
If the repository is on your LAN rather than the internet, note this runs on
the *host*, so it needs no `lanAccess` entry — the guest never touches it.
### When the guest won't boot at all
The system disk is disposable by design. Re-run `hermenix setup`, write a
fresh image to the system disk, and leave the state disk alone — Hermes comes
back with its identity, memories, workspace and NixOS config intact. That is
the intended repair, not a last resort.
### Starting completely fresh
`hermenix setup --clean` stops the VM, deletes the existing
`system.raw`/`state.raw`/UEFI vars, and drops the stale `known_hosts` entry
for the guest (a fresh state disk means a fresh SSH host key on next boot,
so the old entry would otherwise fail with a host-key-mismatch on your next
connect) — then walks through setup again for a genuinely clean first boot,
rather than the disk-exists, skip-ahead path. It leaves the btrfs
subvolumes and any snapshots alone (only the files inside them), and asks
you to type `clean` to confirm before touching anything.
---
## Nix flavour
The guest runs **Determinate Nix**, nixpkgs pinned to FlakeHub
(`f/NixOS/nixpkgs/0.1`, rolling; use `0` for current stable).
- `/etc/nix/nix.conf` belongs to Determinate and must not be edited. Custom
settings go in `/etc/nix/nix.custom.conf`, written from `guest/common.nix`.
There is deliberately no `nix.settings` block.
- No `nix.gc` either — `determinate-nixd` runs the collector.
- Lazy trees are on by default, which makes the agent's flake evaluations
much cheaper. That matters more than it sounds when something re-evaluates
nixpkgs on a schedule.
- Neither flake uses `inputs.nixpkgs.follows` for `determinate`. Determinate
warn it causes FlakeHub Cache misses.
Substituter order in the guest: FlakeHub Cache, your host cache,
`cache.nixos.org`. If FlakeHub Cache goes quiet, the login has expired — run
`determinate-nixd login` again.
The host stays on stock Nix. Nothing here requires changing that.
---
## What the guest can and cannot reach
| | Default |
|---|---|
| Public internet | Yes — it needs this for model APIs and messaging |
| Host Nix cache | Yes, one port, `hostCache.enable = false` closes it |
| Anything else on the host | No, unless in `hostAccess` |
| Your LAN and other VMs | No, unless in `lanAccess` |
| Inbound from the network | No, unless in `publish` |
| Host hardware | None, unless in `passthrough` |
The guest owns its kernel and its config, so nothing inside it is a security
boundary. The nftables rules in `modules/host.nix` are. QEMU runs as an
unprivileged `hermes-vm` user with its own seccomp sandbox and systemd
hardening, with a thin device model: virtio-blk, virtio-net, vsock, RNG,
serial. No virtiofs. `vhost-net` is off deliberately — it would move packet
handling into the host kernel in exchange for throughput one agent doesn't
need.
The host cache does let the guest read every store path on the host. Store
paths are world-readable anyway, so this isn't new exposure, but don't bake
secrets into your host's store.
---
## Known rough edges
- The Determinate NixOS module's option names for custom Nix settings aren't
published — only the nix-darwin `determinateNix.customSettings` equivalent
is. So `guest/common.nix` writes `/etc/nix/nix.custom.conf` via
`environment.etc`. If a future module version claims that file, you'll get
a conflict; switch to its option then.
- `determinate-nixd login` is manual. There's no way to bake the credential
into the image without parking a token somewhere it shouldn't be.
- `MemoryDenyWriteExecute = true` is correct under KVM but is the first thing
to flip if QEMU refuses to start.
- `pkgs.OVMF.firmware` / `pkgs.OVMF.variables` are current attribute names;
older nixpkgs put these under `pkgs.OVMF.fd` in an `FV/` subdirectory.
- PCI passthrough needs a reboot to take effect, because early vfio binding
happens in the initrd.
- Snapshots need btrfs. Everything else works on any filesystem; without it
you rely on backups alone for recovery.
- `state.old-*` directories left by `hermenix vm rollback` are not cleaned up
automatically, on purpose. Delete them yourself once you're sure.
- First boot builds a lot. Give it time.
- `services.hermenix.hermesHome` currently defaults to a path that doesn't
exist in the guest (it's built from the host's disk-image directory, not
the guest's mounted state disk) — set it explicitly on every deployment,
migrated or not. See "Migrating an existing deployment" below for the
value to use.
---
## Migrating an existing (e.g. Docker-based) Hermes deployment
If you already have a running Hermes with real data (`HERMES_HOME` pointing
at some directory full of `SOUL.md`, `memories/`, `sessions/`, `state.db`,
etc.):
```bash
nix run git+ssh://gitea@minipc1.lab.internal:22/ari/hermenix.git#hermenix -- migrate /path/to/existing/HERMES_HOME
```
Run `... -- migrate --help` for the full set of options
(vsock CID, guest user, destination path, hostname rewrites). It copies —
never moves or deletes — the source data, so the original stays a safe
fallback until you've verified the new VM. To run it from a different
machine than the one hosting the VM, see `hermenix remote` above — the same
`migrate /path/...` argument goes in the quoted command string.
The module's `services.hermenix.hermesHome` option lets `hermenix ctl` (and
your own `HERMES_HOME` export inside the guest) point at wherever that
existing flat layout lands, instead of the default seeded `.hermes` under the
state disk.
**Set `hermesHome` explicitly, even if you're not migrating.** The option's
shipped default doesn't currently resolve to a real path inside the guest —
it's built from the host's own disk-image directory rather than the guest's
mounted state disk, so `hermenix ctl soul`/`conf`/`secrets`/etc. will fail
against it unhelpfully until you override it. For the default seeded layout:
```nix
services.hermenix.hermesHome = "/var/lib/hermes/.hermes";
```
For a migrated flat layout, use whatever `--dest` you passed to `migrate`
(default `/var/lib/hermes`).
### Don't run it with `sudo`
`migrate` (like `hermenix vm ssh` and `hermenix ctl`) reaches the guest with
a plain `ssh` call over vsock, authenticating with whatever key or agent the
invoking user already has — the same identity that's in the guest's
`authorizedKeys`. Prefixing the command with `sudo` runs that `ssh` as
**root** instead, which has no matching key, and you'll hit:
```
hermes@hermes: Permission denied (publickey,keyboard-interactive).
```
If `hermenix vm ssh` works for you as your normal user, `migrate` doesn't
need anything more privileged — run it the same way.
### If some of the source can't be read
Old flat `HERMES_HOME` directories, especially ones left behind by a
rootless-Docker deployment, sometimes contain a stray entry owned by a
container-internal uid with no mapping on the host — unreadable by anyone
outside the container. `migrate` tars the source with `--ignore-failed-read`,
so a warning like this is expected and non-fatal:
```
tar: ./.hermes: Cannot open: Permission denied
```
Everything else still gets copied and the run still completes, including the
hostname rewrite step. If you actually need what's in the unreadable entry,
grab it separately, as root, before or after the main migration — e.g.:
```bash
sudo tar -C /path/to/HERMES_HOME/.hermes -cf - . \
| hermenix vm ssh 'mkdir -p /var/lib/hermes/.hermes-old && tar -C /var/lib/hermes/.hermes-old -xf -'
```
### `hermes setup` fails with "docker not found on PATH. Cannot route to container"
`hermenix migrate` automatically excludes `.local/bin` and `.hermes-installed`
from the migration tar, so this should not happen with current versions.
If you see it after a migration with an older build, the source's
`.local/bin/hermes` (a Docker-routing launcher from the previous environment)
was copied into the guest and is shadowing the native install.
Fix inside the guest:
```bash
# Remove the stale launcher
rm -f ~/.local/bin/hermes
# Reinstall the native hermes binary
curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh \
| bash -s -- --non-interactive --skip-setup
# Verify it's a real binary, not a shell script
file ~/.local/bin/hermes
# Now proceed with setup
hermes setup
```
If `hermes-install.service` never ran because `.hermes-installed` was also
migrated from the source, you may instead let systemd do it:
```bash
rm -f ~/.local/bin/hermes ~/.hermes-installed
sudo systemctl start hermes-install
journalctl -fu hermes-install # watch until complete
hermes setup
```
## Hacking on this repo
The devshell provides nushell, nufmt, and pre-commit hooks. Enter it once:
```bash
direnv allow
```
On first entry devenv installs the pre-commit hooks automatically. Hooks run
on every commit, including:
- **nufmt** — formats all `*.nu` files in place
- **gen-completions** — reruns `nu/gen-completions.nu` and stages
`nu/completions.nu` whenever a usage block in the source files changes
- **nu-tests** — runs the CLI test suite (see below) whenever anything
under `nu/` changes
### Testing
Two test suites. Neither is ever run for you by an agent working in this
repo — they're for you (or CI) to run.
**CLI tests** (`nu/tests/`), via [nutest](https://github.com/vyadh/nutest):
```bash
runtest # everything (bin/ is on PATH inside the devShell)
runtest vm_test # just nu/tests/vm_test.nu
runtest-debug vm_test # load vm_test.nu into an interactive nu session instead
```
Covers the CLI's pure helper functions (path/env defaults, `shquote`,
`normalize-size`, the `gen-completions.nu` parser) and a smoke test that the
whole `hermenix`/`vm`/`ctl` subcommand tree parses and dispatches correctly
— without ever invoking a subcommand's real body (no ssh, no disk writes,
no `nix build`).
**Guest VM test** (`guest/vm-test.nix`): boots `guest/common.nix` — the same
module set `hermes-guest-image` is built from — in an ephemeral, throwaway
NixOS test VM. No real disk under `/var/lib/hermes-vm/**` is ever touched:
```bash
runvmtest # plain pass/fail (nix build .#checks.x86_64-linux.guest-vm --impure -L)
runvmtest-debug # build the interactive driver and drop into it (ptpython REPL)
```
Checks first-boot seeding (`~/.hermes`, `~/workspace`, the `/etc/nixos`
symlink), that seeding is idempotent across a reboot (the whole point of
the `.seeded` marker), the `hermes` user/sudo setup, and SSH. It
deliberately does not cover the real `hermes-install` script (no internet
route inside the test VM) or `modules/host.nix`'s own `hermes-vm` QEMU
service — both stay manual, operator-run checks.
`runvmtest-debug` gives you `start_all()`, `machine.wait_for_unit(...)`, and
`machine.shell_interact()` (a live shell inside the guest) to step through a
failure by hand instead of just getting pass/fail.