2026-09-23 14:29:25 -04:00
2026-09-23 13:24:07 -04:00
2026-09-23 13:24:07 -04:00
2026-09-23 13:24:07 -04:00
2026-09-23 13:24:07 -04:00
2026-09-23 13:24:07 -04:00
2026-09-17 15:16:29 -04:00
2026-09-23 13:24:07 -04:00
2026-09-16 22:09:42 -04:00
2026-09-17 15:16:29 -04:00
2026-09-17 15:16:29 -04:00
2026-09-17 14:14:14 -04:00
2026-09-23 13:24:07 -04:00
2026-09-17 15:16:29 -04:00
2026-09-23 14:29:25 -04:00

Hermenix

Hermes Agent in a NixOS VM that owns its own bootloader and its own /nix/store, on two btrfs-backed disks, under an unprivileged hardened QEMU.

THE ULTIMATE WAY TO RUN HERMES!

The agent can rebuild its own system and update itself, durably. It shares no filesystem with the host, and reaches nothing on your network unless you say so.

One CLI, hermenix, with five subcommands:

For
hermenix setup one-off host provisioning — disks, cache key, image build
hermenix migrate <src> carry an existing flat HERMES_HOME into the guest
hermenix remote <host> "<cmd>" run setup/migrate on a remote host over SSH
hermenix vm <sub> control the VM itself — start/stop/console/snapshots/backups
hermenix ctl <sub> drive the agent inside the VM — identity/config/secrets/gateway

Installation

Guided

The hermenix CLI (nu/hermenix.nu) is a flake package — no checkout needed anywhere. Run it straight from its Gitea URL:

nix run git+ssh://gitea@minipc1.lab.internal:22/ari/hermenix.git#hermenix -- setup

(Or nix profile install git+ssh://gitea@minipc1.lab.internal:22/ari/hermenix.git#hermenix once, then just hermenix setup from here on — or get it declaratively via the nixos-setup overlay, same as any other package. Any of these work from any directory, on any host with SSH access to Gitea.)

Walks the host-side steps: checks prerequisites, creates the disks, generates the cache signing key, writes /etc/hermenix/local.nix from your SSH keys, builds the bootstrap image and writes it to the disk. It prints the module snippet to paste into your own configuration at the end, with your uplink interface already filled in.

Safe to re-run — every step detects what's already done and offers to skip, and nothing destructive happens without an explicit confirmation.

It deliberately stops short of nixos-rebuild, since that's your config.

The rest of this section is what the command does, for when you'd rather do it by hand or something goes sideways.

1. Create the disks

Two of them, and the split is the whole design:

Holds Size Snapshotted Backed up
system NixOS, /nix/store 80G No No
state identity, memories, workspace, /etc/nixos 20G Yes Yes

The system disk is large, takes all the write churn, and is entirely rebuildable from this flake — so there's no point protecting it. The state disk holds the only things that can't be reproduced, which makes snapshots cheap and backups quick.

sudo btrfs subvolume create /var/lib/hermes-vm/system
sudo btrfs subvolume create /var/lib/hermes-vm/state
sudo mkdir -p /var/lib/hermes-vm/snapshots

# BEFORE creating any file inside them
sudo chattr +C /var/lib/hermes-vm/system /var/lib/hermes-vm/state
sudo btrfs property set /var/lib/hermes-vm/system compression none
sudo btrfs property set /var/lib/hermes-vm/state compression none

sudo truncate -s 80G /var/lib/hermes-vm/system/system.raw
sudo truncate -s 20G /var/lib/hermes-vm/state/state.raw
sudo mkfs.ext4 -L hermes-state /var/lib/hermes-vm/state/state.raw

The chattr +C ordering matters. Btrfs is copy-on-write and so is a VM writing to a disk image; stacked, every guest write forks an extent and the image fragments badly. +C turns that off — but only for files created afterwards. Set it on empty subvolumes and confirm with lsattr before writing anything. If you get it backwards, recreate the subvolume and copy the image in fresh.

Separate subvolumes because a btrfs snapshot covers a whole subvolume. Keeping the system image out of the state subvolume is what makes snapshots small.

2. Generate the cache signing key

The host serves its store to the guest as a read-only binary cache, so the guest doesn't re-download from the internet what you already have.

sudo mkdir -p /var/lib/secrets
sudo nix-store --generate-binary-cache-key hermes-cache-1 \
  /var/lib/secrets/hermes-cache.secret /var/lib/secrets/hermes-cache.pub
sudo chown nix-serve /var/lib/secrets/hermes-cache.secret
sudo chmod 0400 /var/lib/secrets/hermes-cache.secret
cat /var/lib/secrets/hermes-cache.pub

3. Write /etc/hermenix/local.nix

Host-specific settings live here, not in this repo — see guest/local.nix.example for the reference shape:

{
  cachePublicKey = "hermes-cache-1:AbCd..."; # the key you just printed; leave
                                              # empty and the guest just skips
                                              # the host cache
  authorizedKeys = [ "ssh-ed25519 AAAA... you@laptop" ];
}

sudo mkdir -p /etc/hermenix first if it doesn't exist.

4. Build the image and write it to the system disk

The two extra substituter options stop you compiling Determinate Nix from source. --impure is needed because guest/common.nix reads /etc/hermenix/local.nix from outside this flake's own source tree — deliberately, so the flake itself stays generic and shareable while your settings stay on your host.

nix build --impure git+ssh://gitea@minipc1.lab.internal:22/ari/hermenix.git#hermes-guest-image \
  --option extra-substituters https://install.determinate.systems \
  --option extra-trusted-public-keys cache.flakehub.com-3:hJuILl5sVK4iKm86JzgdXW12Y2Hwd5G07qKtHTOcDCM=

# notrunc keeps the disk at its full size; the image is smaller
sudo dd if=result/nixos.img of=/var/lib/hermes-vm/system/system.raw \
  bs=4M conv=notrunc,fsync status=progress

(Building from a local checkout instead — e.g. while developing this flake — works the same way with . in place of the git URL.)

5. Add the module to your system flake

{
  inputs.hermenix.url = "git+ssh://gitea@minipc1.lab.internal:22/ari/hermenix.git";

  outputs = { nixpkgs, hermenix, ... }: {
    nixosConfigurations.myserver = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules = [ hermenix.nixosModules.default ./configuration.nix ];
    };
  };
}
services.hermenix = {
  enable = true;
  disk.baseDir = "/var/lib/hermes-vm";
  network.externalInterface = "eno1";   # your uplink
  # network.bridged = true;             # guest takes its own LAN IP via DHCP;
  #                                     # every port it opens is then LAN-reachable
  vcpu = 4;
  mem = 8192;
};

sudo nixos-rebuild switch, then hermenix vm console to watch it boot.

6. First boot, inside the guest

hermenix vm ssh

# Confirm the disk labels match guest/hardware.nix BEFORE any rebuild
lsblk -f

# FlakeHub Cache login. Device-code flow: it prints a URL you approve in a
# browser anywhere, so the headless guest never needs one.
determinate-nixd login

# Pin the seeded flake
sudo nix flake lock /etc/nixos

The Hermes CLI installs itself automatically on first boot (systemd.services.hermes-install in common.nix) — it runs unattended, so there's nothing to run for that part. Give it a few minutes (it's compiling native Node modules); watch it with journalctl --user -u hermes-install -f if you want to see it happen. Once it's done:

hermes setup
hermes gateway install

These two need real input (model provider, API keys, messaging platform tokens), so they stay manual.

The identity seed is already in place — SOUL.md, a starter config.yaml, a USER.md and a skill covering safe self-rebuilds. Back on the host, check the gateway came up and that the unit name matches the default:

hermenix ctl units
hermenix ctl status
hermenix ctl soul

Check the labels in step 6 before the first rebuild. guest/hardware.nix assumes the root partition is labelled nixos and the ESP ESP. If the image generator used different ones, fix that file first. Otherwise the agent's first nixos-rebuild switch produces a system that won't boot and you're restoring a snapshot on day one.

From here hermes update self-updates in place and sudo nixos-rebuild switch --flake /etc/nixos#hermes is a durable change to the machine. Neither needs anything from you.


Running it on a remote host

setup and migrate both act on whatever machine they run on — local disks, local sudo, vsock into the local guest. There's no way around SSHing into the target host first. hermenix remote does that SSH step for you:

hermenix remote fwdesk.lab.internal setup
hermenix remote fwdesk.lab.internal "setup --clean"
hermenix remote fwdesk.lab.internal "migrate /var/lib/hermes"

Quote the subcommand and its own flags/args as one string whenever there's more than just the bare subcommand name — nu's own argument parser treats any bare --flag-shaped word as an attempt to match a flag on remote itself, so hermenix remote host setup --clean (unquoted) fails before it ever reaches the SSH step.

It runs nix run <flake>#hermenix -- <the rest of the command> directly on the remote host — no checkout involved, on either end. The remote host does need its own SSH access to Gitea (the same access a git clone would need). Override the flake URL with --flake if you're pointing at a fork or a branch.

It connects with ssh -t, which allocates a real pty — required for the interactive prompts in setup/migrate to work at all over the connection.


SSH into the guest

hermenix vm ssh                       # interactive shell
hermenix vm ssh "systemctl status"    # run a command and exit
hermenix vm console                   # follow the VM's boot/journal output

ssh takes its remote command as a single string, not separate words — quote anything past a single bare word (nu's own parser otherwise tries to match --flag-shaped tokens against vm ssh's own flags, and errors on extra positional words either way).

This goes over vsock, not the network. There is no listening SSH port exposed anywhere, and the guest needs no inbound network path for you to reach it. console is just journalctl -u hermes-vm -f on the host — a read-only follow of the VM unit's journal, useful for watching it boot or diagnosing why SSH won't come up, but it can't take input. There's currently no interactive fallback for when the guest itself is unresponsive.

Under the hood ssh is ssh -o ProxyCommand='socat - VSOCK-CONNECT:<cid>:22' (cid defaults to 77, services.hermenix.vsockCid), if you'd rather drive it by hand.

The VM itself is a systemd unit: hermenix vm start/stop/restart control it directly, same as systemctl {start,stop,restart} hermes-vm on the host.


Bootstrapping Hermes's identity

The seed lives in guest/hermes/:

guest/hermes/
  home/
    SOUL.md                              -> ~/.hermes/SOUL.md
    config.yaml                          -> ~/.hermes/config.yaml
    memories/USER.md                     -> ~/.hermes/memories/USER.md
    skills/nixos-self-config/SKILL.md    -> ~/.hermes/skills/...
  workspace/
    AGENTS.md                            -> ~/workspace/AGENTS.md

Edit these before building the image and the agent wakes up already knowing who it is, how you like to be spoken to, and how to rebuild its own system without bricking itself.

Seeding happens on first boot only, and this is deliberate. Hermes rewrites SOUL.md and its memories as it learns — that's the product. If the seed were re-applied on every rebuild it would silently delete everything the agent had worked out about you. So the copy is guarded by a marker file and never repeats.

Two consequences:

  • Editing guest/hermes/ after first boot changes nothing by itself. Use hermenix ctl reseed when you actually want to overwrite.
  • The seed is the starting identity, not the current one. To see what Hermes has made of itself, hermenix ctl soul.

Machine facts vs identity

/etc/hermes/ENVIRONMENT.md is generated from guest/common.nix on every rebuild and describes the machine: what's reachable, where the store is, what persists. It's always accurate and there's no point editing it.

SOUL.md is identity and belongs to the agent. The seeded version points at ENVIRONMENT.md rather than repeating it, so the two can't drift.

Folding what it learned back into the seed

hermenix ctl pull ./hermes-home        # excludes .env, sessions, state.db
diff -ru guest/hermes/home ./hermes-home

Worth doing occasionally. If the agent has written something genuinely good into its own SOUL.md, you want that in the seed before you ever rebuild from scratch.


Managing Hermes from the host

hermenix vm drives the virtual machine. hermenix ctl drives the agent inside it.

hermenix ctl soul                  # current identity
hermenix ctl soul edit             # pull, $EDITOR, push, offer to restart
hermenix ctl memory edit           # memories/USER.md
hermenix ctl conf edit             # config.yaml

hermenix ctl secrets               # keys, values masked
hermenix ctl secrets edit          # edit hermesHome's .env, written 0600
hermenix ctl secrets show          # values, when you really mean it

hermenix ctl status                # gateway service
hermenix ctl logs                  # follow it
hermenix ctl restart
hermenix ctl units                 # find the unit name if the default is wrong

hermenix ctl reseed                # overwrite identity from the seed
hermenix ctl pull [dir]            # copy live identity to the host

Anything unrecognised is passed straight through to the hermes CLI in the guest, with a TTY attached — quote it as one string if it's more than a single word (same rule as vm ssh above):

hermenix ctl chat
hermenix ctl update
hermenix ctl "sessions list"
hermenix ctl doctor

The edit variants pull the file out over vsock, open your $EDITOR locally, write it back, and offer to restart the gateway. Secrets go in at mode 0600 and never pass through the Nix store — which is the whole reason there's no declarative option for them.

If status and logs come back empty, the gateway unit is probably named something other than hermes-agent. Run hermenix ctl units and set services.hermenix.gatewayUnit to match; upstream has used more than one name for it.

Seeing what the guest is running

hermenix vm config

Prints the current system store path, the last ten generations, the git log of /etc/nixos, and anything uncommitted. That last one matters: the agent editing files without committing is the usual reason the config on disk and the running system disagree.

To read or diff the config locally:

hermenix vm config-pull ./guest-live
diff -ru guest/ guest-live/

Expect that diff to grow. The guest/ directory in this repo is the seed that was written on first boot; it is deliberately not kept in sync. Once the agent edits its config, this repo and the guest diverge permanently, and a host rebuild will not fix a guest config problem.


Letting the guest reach a service on your network

By default the guest can reach the public internet and nothing private. To open a single endpoint — say Ollama on another machine:

services.hermenix.lanAccess = [
  { address = "192.168.1.50"; port = 11434; }     # ollama
  { address = "192.168.1.50"; port = 9000; proto = "udp"; }
];

Each entry punches one destination and port through the RFC1918 drop. The guest reaches it as 192.168.1.50:11434 directly. Traffic is masqueraded, so the other machine sees it arriving from your host's address, not the guest's.

There's also allowLan = true, which opens all private address space at once. It exists, but reach for lanAccess instead — the whole point of this design is that an agent which goes wrong can't go wandering.

A service on the host itself

Different chain, different option. ComfyUI on the host, port 8188:

services.hermenix.hostAccess = [ { port = 8188; } ];

The service must be bound somewhere the guest can see. If it listens on 127.0.0.1 the rule won't help, because the guest arrives on the bridge address. Bind it to 10.77.0.1 (bridge only, invisible to your LAN) or 0.0.0.0:

services.comfyui.listenAddress = "10.77.0.1";   # or whatever your service calls it

From the guest that's then http://10.77.0.1:8188.


Exposing a service the guest is running

Say the agent has built a web app on guest port 8080. Two halves.

Guest side. The agent opens its own firewall, in /etc/nixos:

networking.firewall.allowedTCPPorts = [ 22 8080 ];

Host side. Nothing at all, if you only want to reach it from the host — host-to-guest is outbound traffic and already permitted:

curl http://10.77.0.2:8080

To publish it to the wider network:

services.hermenix.publish = [
  { hostPort = 8080; guestPort = 8080; }
];

That DNATs the port on your uplink to the guest and opens the matching forward rule. hostPort and guestPort don't have to match, so you can park an awkward guest port somewhere sensible.

A note on this one: publishing a port is the one change here that lets the outside world start conversations with the agent's machine. Everything else in this design is the guest reaching out. Put a reverse proxy with TLS and auth in front of anything that isn't a toy, and keep the list short.


Putting the guest directly on your LAN

The default setup puts the guest on an isolated bridge behind the host's NAT, reachable from outside only through publish. This section is for when you want the VM to be a first-class device on your LAN instead: its own address from the LAN's DHCP, reachable on any port from any device on the network.

Enable it:

services.hermenix.network.bridged = true;

In one sentence, what that does: the bridge absorbs your uplink (network.externalInterface), the guest and the host share the LAN, the guest takes a DHCP lease, and inbound traffic to the guest is open on every port. Outbound is still gated by lanAccess / allowLan, and hostAccess plus the host Nix cache still work over the service net at 10.77.0.1.

1. Ready the guest first (one-time, if the image is older than this repo)

The guest keeps a static service-net address (10.77.0.2/24) for talking to the host and adds DHCP on the same NIC, so one config works in both modes. That is already what guest/common.nix does; if your guest image predates it, apply the equivalent change to /etc/nixos/common.nix inside the VM:

systemd.network.networks."10-eth" = {
  matchConfig.Type = "ether";
  address = [ "10.77.0.2/24" ];              # service net to the host
  routes = [ { Gateway = "10.77.0.1"; Metric = 200; } ];  # NAT-mode fallback
  networkConfig.DHCP = "ipv4";               # LAN lease in bridged mode
  dhcpV4Config = {                            # DHCP default route wins (100)
    UseDNS = false;
    RouteMetric = 100;
  };
};

cd /etc/nixos && git apply -p1 < patch, commit, and sudo nixos-rebuild switch --flake /etc/nixos#hermes. In NAT mode the DHCP attempt fails quietly (there is no server on the isolated bridge), so check nothing broke: ip -4 addr still shows 10.77.0.2 and ip route still goes via 10.77.0.1.

2. Rebuild the host with the toggle on

Set services.hermenix.network.bridged = true alongside network.externalInterface, then sudo nixos-rebuild switch. On rebuild the host's uplink becomes a bridge port, the host takes a fresh DHCP lease on the bridge, and the guest's DHCP is answered by the LAN router. The network drops briefly while the host reconfigures - plan for it.

3. Verify

From the host, hermenix vm ssh. Inside the guest, ip -4 addr shows the service net and a LAN address, and ip route shows a default route via your LAN gateway. From any other device on the LAN:

ssh hermes@<guest-lan-ip>        # and any other port the guest opens

The guest's own firewall decides what answers - by default it only opens port 22, so the agent must open whatever it wants LAN-reachable. Egress is still gated too: from the guest, 192.168.1.50:11434 only works if it is in lanAccess (or allowLan = true).

Gotchas

  • The LAN needs a working DHCP server.
  • publish does nothing in this mode - every port is reachable, so its entries are ignored. Drop them if you come back to NAT mode.
  • The host's LAN IP comes from a fresh DHCP lease on the bridge, and the bridge MAC can differ from the uplink's - the host's address may change after the switch. Reserve it in the DHCP server if that matters.
  • Any network config you had for the uplink interface is replaced by its bridge-port role while bridged (this module's networkd unit for it takes priority over most others).

Switching back

Set bridged = false and rebuild the host. The guest's DHCP client finds no server on the isolated bridge and settles onto its static route via the host again; no guest-side change needed.


Passing hardware to the guest

Before reaching for this

If what you want is "the agent can use the GPU", passthrough is often the wrong answer. It is exclusive — the host loses the card completely while the VM runs — and it is the single largest reduction in isolation available in this setup.

The alternative: keep Ollama or ComfyUI on the host where the GPU already works, and open the port with hostAccess. The agent gets the capability over HTTP and the device stays on your side of the boundary. That is usually both easier and better.

Pass the hardware through when the guest genuinely needs the device itself — CUDA in-process, a capture card, a radio, a display output.

What passthrough costs you

A PCI device does DMA. The IOMMU is the only thing standing between a passed-through device and host memory, which is why passthrough.iommu is a hard requirement and the module refuses to build without it.

Two further constraints follow from how IOMMUs work:

  • Whole groups only. Every device in an IOMMU group goes together. Check yours before planning anything:

    for d in /sys/kernel/iommu_groups/*/devices/*; do
      echo "group $(basename "$(dirname "$(dirname "$d")")"): $(lspci -nns "$(basename "$d")")"
    done | sort -V
    

    If your GPU shares a group with a controller you need, you cannot pass it cleanly. ACS override patches exist; they work by lying about isolation, and they have no place in a setup built to contain an autonomous agent.

  • Every function. A GPU is at least two entries — graphics and its HDMI audio function.

GPU

lspci -Dnn | grep -i vga        # addresses and vendor:device IDs
services.hermenix.passthrough = {
  iommu = "amd";                                     # or "intel"
  pci = [
    { address = "0000:01:00.0"; id = "10de:2204"; }  # graphics
    { address = "0000:01:00.1"; id = "10de:1aef"; }  # HDMI audio
  ];
};

Setting id makes vfio-pci claim the card in the initrd, before nvidia or amdgpu can. Do this for GPUs — unbinding a live graphics driver afterwards is unreliable, and the failure mode is a host with no display. Devices given only an address are bound at boot by a oneshot service instead, which is fine for things you swap around.

Rebuild and reboot — early binding happens in the initrd. Then check:

lspci -nnk -d 10de:2204 | grep "Kernel driver"   # want: vfio-pci

On the guest side, the agent installs the drivers itself. It owns /etc/nixos, so hardware.nvidia and nvidia-container-toolkit are its problem, not yours — which is rather the point of the arrangement.

HDMI output

A display hanging off a passed-through GPU works once the guest drives the card; nothing extra is needed here. A capture card or HDMI grabber is just another PCI or USB device — find it with lspci -Dnn or lsusb and use the matching section.

USB devices and Bluetooth

Matched by vendor and product ID:

lsusb        # e.g. "Bus 001 Device 004: ID 8087:0026 Intel ..."
services.hermenix.passthrough.usb = [
  { vendorId = "8087"; productId = "0026"; }   # bluetooth radio
  { vendorId = "0403"; productId = "6001"; }   # serial adapter
];

This adds an xHCI controller to the guest and forwards the matching devices. No IOMMU requirement, no VFIO, and much milder than PCI passthrough — QEMU mediates rather than handing over raw hardware.

Two things that catch people:

  • The host must not be using it. For an integrated Bluetooth radio, keep btusb off it: boot.blacklistedKernelModules = [ "btusb" ];, and don't enable hardware.bluetooth on the host.
  • No hotplug. QEMU resolves these at start, so a device that's unplugged when the VM boots won't appear when you plug it in. hermenix vm restart after plugging. Passing a whole USB controller through as a PCI device gets you real hotplug for anything in its ports, at the cost of that controller being exclusive to the guest.

When it doesn't start

hermenix vm console first. Then, in rough order of likelihood: the device didn't bind to vfio-pci (lspci -nnk), the IOMMU isn't on (dmesg | grep -i iommu), or something in the group is still held by the host. The module lifts LimitMEMLOCK for you when PCI passthrough is configured, since VFIO pins the whole guest RAM allocation and the default limit otherwise produces a memorably unhelpful error.

Recovery

Four layers, cheapest first. The first two live inside the guest and cover almost every real mistake.

# The agent's own config history
hermenix ctl soul                          # what it thinks it is now
hermenix vm ssh "git -C /etc/nixos log --oneline"

# Previous NixOS generations, also in the guest's boot menu
hermenix vm ssh "sudo nixos-rebuild switch --rollback"

# Snapshot of the state disk
hermenix vm snapshot                       # trigger one now, e.g. before a risky change
hermenix vm snapshots
hermenix vm rollback state-20260914-1400   # asks you to type 'rollback' to confirm

# Restore from backup
hermenix vm backup                         # trigger one now
hermenix vm backups

Snapshots

Hourly by default, keeping 48. Only the state disk — the system disk isn't snapshotted because hermenix setup rebuilds it from the flake in minutes.

The host fsfreezes the guest through qemu-guest-agent around each one, so the snapshot captures a consistent filesystem rather than the equivalent of pulling the power. Takes milliseconds; the VM keeps running.

hermenix vm rollback stops the VM, moves the current state subvolume aside as state.old-<timestamp>, and restores the snapshot in its place. Nothing is deleted, so a mistaken rollback is itself reversible. Clean the state.old-* directories up yourself once you're happy.

Keep retention tighter than you would on ZFS. The state image is nodatacow, and every live snapshot reintroduces copy-on-write for the blocks it still references. Performance sags while snapshots exist and recovers as they're pruned.

Backups

services.hermenix.backup = {
  enable = true;
  repository = "sftp:backup@nas:/hermes";
  passwordFile = "/var/lib/secrets/hermes-restic";
};

This backs up the guest's state, not its disk. Files are pulled over the SSH channel that already exists and streamed into restic.

Two reasons it works that way. The backup stays portable — it restores onto any filesystem, or a different machine entirely, and doesn't care that you're on btrfs today. And it avoids loop-mounting a guest-written filesystem on the host, which would mean the host kernel parsing data the agent controls. That would undo a fair bit of what the rest of this design is for.

If the repository is on your LAN rather than the internet, note this runs on the host, so it needs no lanAccess entry — the guest never touches it.

When the guest won't boot at all

The system disk is disposable by design. Re-run hermenix setup, write a fresh image to the system disk, and leave the state disk alone — Hermes comes back with its identity, memories, workspace and NixOS config intact. That is the intended repair, not a last resort.

Starting completely fresh

hermenix setup --clean stops the VM, deletes the existing system.raw/state.raw/UEFI vars, and drops the stale known_hosts entry for the guest (a fresh state disk means a fresh SSH host key on next boot, so the old entry would otherwise fail with a host-key-mismatch on your next connect) — then walks through setup again for a genuinely clean first boot, rather than the disk-exists, skip-ahead path. It leaves the btrfs subvolumes and any snapshots alone (only the files inside them), and asks you to type clean to confirm before touching anything.


Nix flavour

The guest runs Determinate Nix, nixpkgs pinned to FlakeHub (f/NixOS/nixpkgs/0.1, rolling; use 0 for current stable).

  • /etc/nix/nix.conf belongs to Determinate and must not be edited. Custom settings go in /etc/nix/nix.custom.conf, written from guest/common.nix. There is deliberately no nix.settings block.
  • No nix.gc either — determinate-nixd runs the collector.
  • Lazy trees are on by default, which makes the agent's flake evaluations much cheaper. That matters more than it sounds when something re-evaluates nixpkgs on a schedule.
  • Neither flake uses inputs.nixpkgs.follows for determinate. Determinate warn it causes FlakeHub Cache misses.

Substituter order in the guest: FlakeHub Cache, your host cache, cache.nixos.org. If FlakeHub Cache goes quiet, the login has expired — run determinate-nixd login again.

The host stays on stock Nix. Nothing here requires changing that.


What the guest can and cannot reach

Default
Public internet Yes — it needs this for model APIs and messaging
Host Nix cache Yes, one port, hostCache.enable = false closes it
Anything else on the host No, unless in hostAccess
Your LAN and other VMs No, unless in lanAccess
Inbound from the network No, unless in publish
Host hardware None, unless in passthrough

With network.bridged = true the "Inbound from the network" row changes: with the guest on the LAN, every port it opens is reachable from the LAN; publish stops doing anything. The outbound rows are unchanged - the lanAccess / allowLan gates still apply.

The guest owns its kernel and its config, so nothing inside it is a security boundary. The nftables rules in modules/host.nix are. QEMU runs as an unprivileged hermes-vm user with its own seccomp sandbox and systemd hardening, with a thin device model: virtio-blk, virtio-net, vsock, RNG, serial. No virtiofs. vhost-net is off deliberately — it would move packet handling into the host kernel in exchange for throughput one agent doesn't need.

The host cache does let the guest read every store path on the host. Store paths are world-readable anyway, so this isn't new exposure, but don't bake secrets into your host's store.


Known rough edges

  • The Determinate NixOS module's option names for custom Nix settings aren't published — only the nix-darwin determinateNix.customSettings equivalent is. So guest/common.nix writes /etc/nix/nix.custom.conf via environment.etc. If a future module version claims that file, you'll get a conflict; switch to its option then.
  • determinate-nixd login is manual. There's no way to bake the credential into the image without parking a token somewhere it shouldn't be.
  • MemoryDenyWriteExecute = true is correct under KVM but is the first thing to flip if QEMU refuses to start.
  • pkgs.OVMF.firmware / pkgs.OVMF.variables are current attribute names; older nixpkgs put these under pkgs.OVMF.fd in an FV/ subdirectory.
  • PCI passthrough needs a reboot to take effect, because early vfio binding happens in the initrd.
  • Snapshots need btrfs. Everything else works on any filesystem; without it you rely on backups alone for recovery.
  • state.old-* directories left by hermenix vm rollback are not cleaned up automatically, on purpose. Delete them yourself once you're sure.
  • First boot builds a lot. Give it time.
  • services.hermenix.hermesHome currently defaults to a path that doesn't exist in the guest (it's built from the host's disk-image directory, not the guest's mounted state disk) — set it explicitly on every deployment, migrated or not. See "Migrating an existing deployment" below for the value to use.

Migrating an existing (e.g. Docker-based) Hermes deployment

If you already have a running Hermes with real data (HERMES_HOME pointing at some directory full of SOUL.md, memories/, sessions/, state.db, etc.):

nix run git+ssh://gitea@minipc1.lab.internal:22/ari/hermenix.git#hermenix -- migrate /path/to/existing/HERMES_HOME

Run ... -- migrate --help for the full set of options (vsock CID, guest user, destination path, hostname rewrites). It copies — never moves or deletes — the source data, so the original stays a safe fallback until you've verified the new VM. To run it from a different machine than the one hosting the VM, see hermenix remote above — the same migrate /path/... argument goes in the quoted command string.

The module's services.hermenix.hermesHome option lets hermenix ctl (and your own HERMES_HOME export inside the guest) point at wherever that existing flat layout lands, instead of the default seeded .hermes under the state disk.

Set hermesHome explicitly, even if you're not migrating. The option's shipped default doesn't currently resolve to a real path inside the guest — it's built from the host's own disk-image directory rather than the guest's mounted state disk, so hermenix ctl soul/conf/secrets/etc. will fail against it unhelpfully until you override it. For the default seeded layout:

services.hermenix.hermesHome = "/var/lib/hermes/.hermes";

For a migrated flat layout, use whatever --dest you passed to migrate (default /var/lib/hermes).

Don't run it with sudo

migrate (like hermenix vm ssh and hermenix ctl) reaches the guest with a plain ssh call over vsock, authenticating with whatever key or agent the invoking user already has — the same identity that's in the guest's authorizedKeys. Prefixing the command with sudo runs that ssh as root instead, which has no matching key, and you'll hit:

hermes@hermes: Permission denied (publickey,keyboard-interactive).

If hermenix vm ssh works for you as your normal user, migrate doesn't need anything more privileged — run it the same way.

If some of the source can't be read

Old flat HERMES_HOME directories, especially ones left behind by a rootless-Docker deployment, sometimes contain a stray entry owned by a container-internal uid with no mapping on the host — unreadable by anyone outside the container. migrate tars the source with --ignore-failed-read, so a warning like this is expected and non-fatal:

tar: ./.hermes: Cannot open: Permission denied

Everything else still gets copied and the run still completes, including the hostname rewrite step. If you actually need what's in the unreadable entry, grab it separately, as root, before or after the main migration — e.g.:

sudo tar -C /path/to/HERMES_HOME/.hermes -cf - . \
  | hermenix vm ssh 'mkdir -p /var/lib/hermes/.hermes-old && tar -C /var/lib/hermes/.hermes-old -xf -'

hermes setup fails with "docker not found on PATH. Cannot route to container"

hermenix migrate automatically excludes .local/bin and .hermes-installed from the migration tar, so this should not happen with current versions. If you see it after a migration with an older build, the source's .local/bin/hermes (a Docker-routing launcher from the previous environment) was copied into the guest and is shadowing the native install.

Fix inside the guest:

# Remove the stale launcher
rm -f ~/.local/bin/hermes

# Reinstall the native hermes binary
curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh \
  | bash -s -- --non-interactive --skip-setup

# Verify it's a real binary, not a shell script
file ~/.local/bin/hermes

# Now proceed with setup
hermes setup

If hermes-install.service never ran because .hermes-installed was also migrated from the source, you may instead let systemd do it:

rm -f ~/.local/bin/hermes ~/.hermes-installed
sudo systemctl start hermes-install
journalctl -fu hermes-install   # watch until complete
hermes setup

Hacking on this repo

The devshell provides nushell, nufmt, and pre-commit hooks. Enter it once:

direnv allow

On first entry devenv installs the pre-commit hooks automatically. Hooks run on every commit, including:

  • nufmt — formats all *.nu files in place
  • gen-completions — reruns nu/gen-completions.nu and stages nu/completions.nu whenever a usage block in the source files changes
  • nu-tests — runs the CLI test suite (see below) whenever anything under nu/ changes

Testing

Three test suites. None is ever run for you by an agent working in this repo — they're for you (or CI) to run.

CLI tests (nu/tests/), via nutest:

runtest              # everything (bin/ is on PATH inside the devShell)
runtest vm_test       # just nu/tests/vm_test.nu
runtest-debug vm_test # load vm_test.nu into an interactive nu session instead

Covers the CLI's pure helper functions (path/env defaults, shquote, normalize-size, the gen-completions.nu parser) and a smoke test that the whole hermenix/vm/ctl subcommand tree parses and dispatches correctly — without ever invoking a subcommand's real body (no ssh, no disk writes, no nix build).

Guest VM tests (guest/vm-tests/test_unbridge.nix / guest/vm-tests/test_bridge.nix): boot guest/common.nix — the same module set hermes-guest-image is built from — in ephemeral, throwaway NixOS test VMs. No real disk under /var/lib/hermes-vm/** is ever touched:

runvmtest                    # NAT/isolated-bridge mode (plain pass/fail)
runvmtest guest-vm-bridged   # bridged mode with a simulated LAN
runvmtest-debug              # interactive driver REPL (ptpython) for the default suite

Both tests boot the guest the same way and share every check that must hold in both modes — first-boot seeding (~/.hermes, ~/workspace, the /etc/nixos symlink), that seeding is idempotent across a reboot (the whole point of the .seeded marker), the hermes user/sudo setup, and SSH. That shared node config and assertion script are defined once in guest/vm-tests/common.nix and imported by both, so nothing can drift between the two modes' common checks.

Each test then adds the checks that only make sense for its mode:

  • guest-vm-unbridged — the isolated bridge has no DHCP server, so it asserts the guest stands on its static config alone: exactly the one 10.77.0.2/24 address and exactly one default route, via the host.
  • guest-vm-bridged — a second node simulates the LAN (a dnsmasq DHCP server, a throwaway HTTP service, and a plain device). It asserts the guest takes a DHCP lease beside the static service-net address, keeps both default routes (the LAN gateway at metric 100 and the host fallback at metric 200), can reach a service on the LAN, and that a LAN device can connect to a port the guest exposes (the guest's own firewall still decides what answers). The host-side half of bridged mode — the bridge itself, the mark-based rules — is covered statically by the check below.

Both deliberately do not cover the real hermes-install script (no internet route inside the test VM) or modules/host.nix's own hermes-vm QEMU service — those stay manual, operator-run checks.

runvmtest-debug gives you start_all(), machine.wait_for_unit(...), and machine.shell_interact() (a live shell inside the guest) to step through a failure by hand instead of just getting pass/fail.

Module evaluation checks (tests/network-modes-check.nix): a static check — no build, no VM boot — that evaluates modules/host.nix in both networking modes (NAT and network.bridged = true) plus guest/common.nix, and asserts on the generated nftables rulesets, the firewall input/forward rules, the networkd units, and the guest's DHCP/route-metric config:

nix build .#checks.x86_64-linux.network-modes

Catches regressions in the bridged-mode flag without booting anything. If it fails, the assertion that broke is named in the error.

S
Description
Hermes + NixOs Opinionated Setup
Readme
774 KiB
0 Stars 1 Watchers 0 Forks
Languages
Nix 64.4%
Nushell 35.2%
Nu 0.4%