doc clean

This commit is contained in:
ari committed 2026-10-09 14:35:12 -04:00
1 parent c2a7f94821
commit 30427e3975
34 files changed
+1119 -1758

No files matched your search

+101
View File
@@ -0,0 +1,101 @@
# CLI reference
One binary, `hermenix`:
| | For |
| -------------------------------- | ---------------------------------------------------------- |
| `hermenix setup` | host provisioning — disks, cache key, image |
| `hermenix migrate <src>` | existing flat `HERMES_HOME` → guest |
| `hermenix migrate connections` | connection files only |
| `hermenix remote <host> "<cmd>"` | `setup`/`migrate` over SSH |
| `hermenix vm <sub>` | VM lifecycle, snapshots, backups |
| `hermenix ctl <sub>` | host-side bridge into the guest (files, unit, passthrough) |
| `hermenix version` | packaged version string |
Quote multi-word remote/passthrough args as one string —
[CONCEPTS.md](CONCEPTS.md#quote-multi-word-subcommands).
Setup: [INSTALL.md](INSTALL.md). Migrate: [MIGRATE.md](MIGRATE.md).
---
## `hermenix vm`
```bash
hermenix vm ssh # interactive shell (vsock)
hermenix vm ssh "systemctl status" # run a command and exit
hermenix vm console # journalctl -u hermes-vm -f (read-only)
hermenix vm start | stop | restart
hermenix vm config # guest generation, history, git status
hermenix vm config-pull [dir] # copy guest /etc/nixos here
hermenix vm snapshot | snapshots
hermenix vm rollback <snap> # type 'rollback' to confirm
hermenix vm backup | backups
```
`console` is follow-only (no input). No interactive fallback if the guest is
dead.
Repo `guest/` is the **seed**; live guest `/etc/nixos` diverges after the
agent edits it. `config-pull` is for reading/diffing — a host rebuild will not
fix a guest config problem.
---
## `hermenix ctl`
Host-side helpers over vsock. They know where Hermes's files live in _this_
VM layout and how to reach the gateway unit — they are not a second Hermes
manual. For Hermes features (chat, sessions, providers, skills, …) use the
upstream CLI or [Hermes Agent docs](https://github.com/NousResearch/hermes-agent); unknown `ctl` words are
passed through to `hermes` in the guest.
```bash
# Files under hermesHome (default /var/lib/hermes/.hermes)
hermenix ctl soul [edit]
hermenix ctl memory [edit] # memories/USER.md if present
hermenix ctl conf [edit] # config.yaml
hermenix ctl secrets [edit|show] # .env, mode 0600; masked by default
# Gateway user unit (name: services.hermenix.gatewayUnit)
hermenix ctl status | logs | restart
hermenix ctl units
# Seed / fold-back (destructive where noted)
hermenix ctl reseed # overwrite from /etc/nixos/hermes seed
hermenix ctl pull [dir] # live home → host (no .env/sessions/state.db)
```
Passthrough examples (Hermes's own commands):
```bash
hermenix ctl chat
hermenix ctl update
hermenix ctl "sessions list"
hermenix ctl doctor
```
`edit` pulls over vsock, opens local `$EDITOR`, writes back, offers restart.
Secrets never enter the Nix store — no declarative option for them.
If `status`/`logs` are empty, check `hermenix ctl units` and set
`services.hermenix.gatewayUnit` to the real unit name.
`services.hermenix.hermesHome` is where `ctl` looks. Override for a flat
migrated layout; export `HERMES_HOME` inside the guest for the gateway
process if it differs from the default.
---
## Env baked in by the host module
| Env | Option | Default |
| ---------------------- | -------------------------------- | ------------------------- |
| `HERMES_VSOCK_CID` | `services.hermenix.vsockCid` | `77` |
| `HERMES_GUEST_NAME` | `services.hermenix.name` | `hermes` |
| `HERMES_DISK_BASE_DIR` | `services.hermenix.disk.baseDir` | `/var/lib/hermes-vm` |
| `HERMES_HOME_DIR` | `services.hermenix.hermesHome` | `/var/lib/hermes/.hermes` |
| `HERMES_GATEWAY_UNIT` | `services.hermenix.gatewayUnit` | `hermes-agent` |
+98
View File
@@ -0,0 +1,98 @@
# Concepts
Shared model for everything else in these docs. Read once; other pages assume it.
## Two disks
| | Holds | Size (default) | Snapshotted | Backed up |
| ---------- | ---------------------------------------------- | -------------- | ----------- | --------- |
| **system** | NixOS, `/nix/store` | 80G | No | No |
| **state** | identity, workspace, `/etc/nixos`, Hermes home | 20G | Yes | Yes |
System is large, churny, rebuildable from this flake. State holds what cannot
be reproduced — snapshots stay cheap.
Host images under `services.hermenix.disk.baseDir` (default `/var/lib/hermes-vm`):
```
$baseDir/system/system.raw
$baseDir/state/state.raw
$baseDir/snapshots/
```
Guest: state disk at **`/var/lib/hermes`**. Hermes home defaults to
`/var/lib/hermes/.hermes` (`~/.hermes` for user `hermes`). `/etc/nixos` →
`/var/lib/hermes/nixos`.
What belongs _inside_ Hermes home (SOUL, memories, skills, sessions, …) is
defined by Hermes itself — see [upstream docs](https://github.com/NousResearch/hermes-agent). hermenix only
decides which disk it lives on and that the seed is applied once.
## The boundary is the host
The guest owns its kernel and config. Nothing _inside_ the guest is a security
boundary; `modules/host.nix` nftables are. QEMU runs as unprivileged
`hermes-vm` with seccomp + systemd hardening. Thin device model: virtio-blk,
virtio-net, vsock, RNG, serial. No virtiofs. `vhost-net` off on purpose.
| | Default |
| ------------------------- | --------------------------------------------------- |
| Public internet | Yes |
| Host Nix cache | Yes, one port; `hostCache.enable = false` closes it |
| Anything else on the host | No, unless `hostAccess` |
| LAN / other VMs | No, unless `lanAccess` |
| Inbound from the network | No, unless `publish` (NAT mode) |
| Host hardware | None, unless `passthrough` |
`network.bridged = true`: LAN inbound = every port the guest opens; `publish`
ignored. Outbound gates unchanged. [NETWORK.md](NETWORK.md).
## Seed once
`guest/hermes/` is a **template** copied into the guest on first boot only
(`/var/lib/hermes/.seeded`):
```
guest/hermes/
home/ → ~/.hermes (SOUL.md, config.yaml, skills/, …)
workspace/ → ~/workspace
```
Hermes owns and rewrites its home after that — re-seeding on every rebuild
would wipe learned state. hermenix consequences:
- Editing `guest/hermes/` post-first-boot does nothing alone →
`hermenix ctl reseed` (destructive; needs fresh confirmation).
- Live files: `hermenix ctl soul` / `conf` / … or `hermenix ctl pull`.
- Machine facts (not identity): `/etc/hermes/ENVIRONMENT.md`, regenerated
every guest rebuild from `guest/common.nix`.
## vsock, not network SSH
Host → guest (`vm ssh`, `ctl`, backups) uses **vsock** (default CID 77). No
management SSH port on the network path.
```
ssh -o ProxyCommand='socat - VSOCK-CONNECT:<cid>:22' hermes@hermes
```
## Determinate Nix in the guest
Guest: **Determinate Nix**, nixpkgs on FlakeHub. Host: stock Nix.
- Do not edit `/etc/nix/nix.conf` — custom settings in
`/etc/nix/nix.custom.conf` (from `guest/common.nix`).
- No guest `nix.settings` / `nix.gc`; `determinate-nixd` collects.
- Substituters: FlakeHub Cache → host cache → `cache.nixos.org`.
- Do not set `inputs.nixpkgs.follows` on `determinate` (cache misses).
## Quote multi-word subcommands
nu treats bare `--flag` tokens as flags on the _outer_ command. Quote anything
past a single bare word:
```bash
hermenix remote fwdesk.lab.internal "setup --clean"
hermenix vm ssh "systemctl status"
hermenix ctl "sessions list"
```
+82
View File
@@ -0,0 +1,82 @@
# Hacking on this repo
## Devshell
```bash
direnv allow
```
Provides nushell, nufmt, nixfmt, and pre-commit hooks (installed on first
entry):
- **treefmt** — nixfmt / deadnix / statix / nufmt
- **nu-check** — parse-check `*.nu`
- **gen-completions** — regenerates `nu/completions.nu` from usage blocks
- **nu-tests** — `bin/runtest` when anything under `nu/` changes
## Layout
See [GUIDELINES.md](../GUIDELINES.md#repository-layout). The CLI is several
nushell files concatenated by `flake.nix` in dependency order
(`lib/` → `host/` → `vm/` → `ctl/` → `hermenix.nu`).
## Testing
Three suites. **Agents may run them only through the wrappers below** — never
via bare `nix build` / `nix flake check`. See [GUIDELINES.md](../GUIDELINES.md).
### CLI tests (`nu/tests/`, nutest)
```bash
runtest # everything (bin/ on PATH in the devShell)
runtest vm_test
runtest-debug vm_test
```
Pure helpers + smoke that the subcommand tree parses — no ssh, no disk writes,
no `nix build`.
### Guest VM tests (`guest/vm-tests/`)
Boot `guest/common.nix` in ephemeral NixOS test VMs. Never touches
`/var/lib/hermes-vm/**`:
```bash
runvmtest # NAT / isolated-bridge (default)
runvmtest guest-vm-bridged # bridged mode with simulated LAN
runvmtest-debug # interactive driver REPL
```
Shared seeding/SSH/user assertions live in `guest/vm-tests/common.nix`.
Mode-specific checks:
- **unbridged** — static `10.77.0.2/24` only, one default route via host
- **bridged** — DHCP lease beside service net, dual default routes, LAN
reachability both ways
Does not cover real `hermes-install` (no internet in the test VM) or the host
`hermes-vm` QEMU unit.
### Module evaluation checks
Static — no VM boot. Evaluates `modules/host.nix` in both network modes plus
`guest/common.nix`, asserts on nftables / networkd / guest DHCP metrics.
Operator/CI (not the agent-safe wrapper path today):
```bash
nix build .#checks.x86_64-linux.network-modes
```
Agents: describe the check and let the operator run it, unless/until a
`bin/` wrapper exists for it.
## Parsing the CLI without running it
```bash
nu nu/hermenix.nu --help
nu nu/hermenix.nu setup --help
# etc.
```
Nix syntax only: `nix-instantiate --parse path/to/file.nix`.
+117
View File
@@ -0,0 +1,117 @@
# Install
## Guided (recommended)
The `hermenix` CLI is a flake package — no checkout needed:
```bash
nix run git+ssh://gitea@gitea.skyg.ca/ari/hermenix.git#hermenix -- setup
```
Or install once:
```bash
nix profile install git+ssh://gitea@gitea.skyg.ca/ari/hermenix.git#hermenix
hermenix setup
```
`setup` walks host-side steps: prerequisites, disks, cache signing key,
`/etc/hermenix/local.nix` from your SSH keys, bootstrap image build, write to
the system disk. It prints the module snippet to paste into your configuration
(uplink filled in).
Safe to re-run — steps skip what's done; nothing destructive without
confirmation. It never runs `nixos-rebuild` (that's your config).
Hand-run equivalent: [MANUAL_SETUP.md](MANUAL_SETUP.md).
## Add the module
```nix
# flake.nix
{
inputs.hermenix.url = "git+ssh://gitea@gitea.skyg.ca/ari/hermenix.git";
outputs = { nixpkgs, hermenix, ... }: {
nixosConfigurations.myserver = nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
modules = [ hermenix.nixosModules.default ./configuration.nix ];
};
};
}
```
```nix
# configuration.nix
services.hermenix = {
enable = true;
disk.baseDir = "/var/lib/hermes-vm";
network.externalInterface = "eno1"; # your uplink
# network.bridged = true; # see docs/NETWORK.md
vcpu = 4;
mem = 8192;
};
```
```bash
sudo nixos-rebuild switch
hermenix vm console # watch it boot
```
## First boot (hermenix side)
```bash
hermenix vm ssh
# Labels must match guest/hardware.nix BEFORE any guest rebuild
lsblk -f
# FlakeHub Cache (device-code flow — approve URL from any browser)
determinate-nixd login
sudo nix flake lock /etc/nixos
```
The Hermes CLI installs unattended on first boot (`hermes-install.service`;
a few minutes for native modules). After that, configure Hermes itself with
its own CLI — provider, keys, gateway, etc. That is **upstream Hermes**, not
hermenix; see the [Hermes Agent docs](https://github.com/NousResearch/hermes-agent).
From the host you can drive the guest install/gateway unit once it exists:
```bash
hermenix ctl units
hermenix ctl status
```
> **Check labels before the first guest `nixos-rebuild`.**
> `guest/hardware.nix` assumes root `nixos` and ESP `ESP`. Wrong labels →
> unbootable system after the agent's first self-rebuild.
Guest NixOS changes are durable via
`sudo nixos-rebuild switch --flake /etc/nixos#hermes` inside the VM. Hermes
self-updates are upstream (`hermes update`); use `hermenix ctl` only as a
vsock passthrough if you prefer running them from the host.
## Remote host
`setup` / `migrate` act on the machine they run on. `hermenix remote` SSHs:
```bash
hermenix remote fwdesk.lab.internal setup
hermenix remote fwdesk.lab.internal "setup --clean"
hermenix remote fwdesk.lab.internal "migrate /var/lib/hermes"
```
Quote subcommand + flags as **one string** when more than a bare name —
[CONCEPTS.md](CONCEPTS.md#quote-multi-word-subcommands).
Runs `nix run <flake>#hermenix -- …` on the remote (no checkout). Remote needs
SSH access to Gitea. Override URL with `--flake`. Uses `ssh -t` for prompts.
## Next
- [CLI.md](CLI.md) — day-to-day `vm` / `ctl`
- [NETWORK.md](NETWORK.md) — allowlists / bridged
- [MIGRATE.md](MIGRATE.md) — existing `HERMES_HOME`
- [RECOVERY.md](RECOVERY.md) — snapshots / unbootable guest
+84
View File
@@ -0,0 +1,84 @@
# Manual setup
What `hermenix setup` does, step by step, for hand-running or when something
goes sideways. Prefer the guided command in [INSTALL.md](INSTALL.md) when you
can.
## 1. Create the disks
Two disks — design in [CONCEPTS.md](CONCEPTS.md#two-disks).
```bash
sudo btrfs subvolume create /var/lib/hermes-vm/system
sudo btrfs subvolume create /var/lib/hermes-vm/state
sudo mkdir -p /var/lib/hermes-vm/snapshots
# BEFORE creating any file inside them
sudo chattr +C /var/lib/hermes-vm/system /var/lib/hermes-vm/state
sudo btrfs property set /var/lib/hermes-vm/system compression none
sudo btrfs property set /var/lib/hermes-vm/state compression none
sudo truncate -s 80G /var/lib/hermes-vm/system/system.raw
sudo truncate -s 20G /var/lib/hermes-vm/state/state.raw
sudo mkfs.ext4 -L hermes-state /var/lib/hermes-vm/state/state.raw
```
**`chattr +C` ordering matters.** Btrfs CoW stacked under a VM disk image
fragments badly. `+C` disables that — but only for files created _after_.
Set it on empty subvolumes and confirm with `lsattr` before writing anything.
Wrong order → recreate the subvolume and copy the image in fresh.
Separate subvolumes so a btrfs snapshot of state does not include the system
image.
## 2. Generate the cache signing key
The host serves its store to the guest as a read-only binary cache.
```bash
sudo mkdir -p /var/lib/secrets
sudo nix-store --generate-binary-cache-key hermes-cache-1 \
/var/lib/secrets/hermes-cache.secret /var/lib/secrets/hermes-cache.pub
sudo chown nix-serve /var/lib/secrets/hermes-cache.secret
sudo chmod 0400 /var/lib/secrets/hermes-cache.secret
cat /var/lib/secrets/hermes-cache.pub
```
## 3. Write `/etc/hermenix/local.nix`
Host-specific settings live here, not in this repo — see
`guest/local.nix.example`:
```nix
{
cachePublicKey = "hermes-cache-1:AbCd..."; # key you just printed; empty = skip host cache
authorizedKeys = [ "ssh-ed25519 AAAA... you@laptop" ];
}
```
`sudo mkdir -p /etc/hermenix` first if needed.
## 4. Build the image and write it to the system disk
`--impure` is required because `guest/common.nix` reads
`/etc/hermenix/local.nix` outside the flake source tree. The extra substituter
flags avoid compiling Determinate Nix from source. `setup` also passes
`--no-write-lock-file --refresh` when building from the git URL — mirror that
if you're doing the same by hand.
```bash
nix build --impure --no-write-lock-file --refresh \
git+ssh://gitea@gitea.skyg.ca/ari/hermenix.git#hermes-guest-image \
--option extra-substituters https://install.determinate.systems \
--option extra-trusted-public-keys cache.flakehub.com-3:hJuILl5sVK4iKm86JzgdXW12Y2Hwd5G07qKtHTOcDCM=
# notrunc keeps the disk at its full size; the image is smaller
sudo dd if=result/nixos.img of=/var/lib/hermes-vm/system/system.raw \
bs=4M conv=notrunc,fsync status=progress
```
From a local checkout, use `.` instead of the git URL (still `--impure`).
## 5. Module + first boot
Continue from [INSTALL.md](INSTALL.md#add-the-module).
+74
View File
@@ -0,0 +1,74 @@
# Migrate an existing deployment
Copy an existing flat `HERMES_HOME` into the guest after hermenix setup + host
`nixos-rebuild` and `hermenix vm ssh true` work:
```bash
nix run git+ssh://gitea@gitea.skyg.ca/ari/hermenix.git#hermenix -- \
migrate /path/to/existing/HERMES_HOME
```
`migrate --help` for vsock CID, guest user, `--dest`, hostname rewrites.
**Copy only — never move or delete the source** until the new side is verified.
Remote: [INSTALL.md](INSTALL.md#remote-host) — put `migrate /path/...` in the
quoted command string.
What a `HERMES_HOME` contains and how Hermes uses it is upstream territory —
[Hermes Agent](https://github.com/NousResearch/hermes-agent). This page is only the hermenix carry path.
## `hermesHome`
`services.hermenix.hermesHome` is where `hermenix ctl` looks in the guest.
| Layout | Value |
| -------------------------- | -------------------------------------------- |
| Seeded `.hermes` (default) | `/var/lib/hermes/.hermes` |
| Flat at state root | `migrate --dest` (default `/var/lib/hermes`) |
Export `HERMES_HOME` inside the guest to the same path for the gateway process.
## Don't use `sudo`
`migrate` uses your user's ssh-over-vsock key (same as `authorizedKeys`).
`sudo` → root has no key → `Permission denied (publickey)`. If
`hermenix vm ssh` works as you, run `migrate` the same way.
## `hermenix migrate connections`
Connection/integration files only (no identity wipe), local or `--to-guest`:
```bash
hermenix migrate connections /old/HERMES_HOME ./staging
hermenix migrate connections /old/HERMES_HOME --to-guest
```
Defaults: files `config.yaml`, `auth.json`, `.env`; dir `connections/`.
Rewrites apply in the destination only.
## Troubleshooting
### Unreadable source entries
Rootless-Docker leftovers may be unreadable; `tar --ignore-failed-read` makes
that non-fatal. Pull those paths as root separately if needed:
```bash
sudo tar -C /path/to/HERMES_HOME/.hermes -cf - . \
| hermenix vm ssh 'mkdir -p /var/lib/hermes/.hermes-old && tar -C /var/lib/hermes/.hermes-old -xf -'
```
### Stale Docker launcher after an old migrate
Current `migrate` excludes `.local/bin` and `.hermes-installed`. If an older
run left a Docker-routing `~/.local/bin/hermes` ("docker not found on PATH"):
```bash
rm -f ~/.local/bin/hermes ~/.hermes-installed
sudo systemctl start hermes-install # or reinstall per upstream docs
journalctl -fu hermes-install
```
Then finish Hermes configuration with the upstream CLI
([docs](https://github.com/NousResearch/hermes-agent)), not with hermenix-specific steps.
+120
View File
@@ -0,0 +1,120 @@
# Network
Default model and reachability table: [CONCEPTS.md](CONCEPTS.md#the-boundary-is-the-host).
## Mode matrix
| | NAT (default) | Bridged (`network.bridged = true`) |
| ------------------------------ | ------------------------------------- | ---------------------------------------- |
| Guest address | `10.77.0.2` on isolated bridge | service net **plus** fixed LAN address |
| Inbound from LAN / internet | only via `publish` | every port the guest opens |
| Guest → public internet | yes (masqueraded) | yes (L2 / LAN gateway) |
| Guest → private LAN | `lanAccess` / `allowLan` only | same gates |
| Guest → host services | `hostAccess` + host cache on `10.77.0.1` | same over service net |
| `publish` | DNAT on uplink | **ignored** |
## Guest → a service on your LAN
```nix
services.hermenix.lanAccess = [
{ address = "192.168.1.50"; port = 11434; } # ollama
{ address = "192.168.1.50"; port = 9000; proto = "udp"; }
];
```
One destination:port per entry, punched through the RFC1918 drop. Traffic is
masqueraded; the peer sees the host's address.
`allowLan = true` opens all private space at once. Prefer `lanAccess` — an
agent that goes wrong should not wander.
## Guest → a service on the host
```nix
services.hermenix.hostAccess = [ { port = 8188; } ];
```
The service must bind somewhere the guest can see. `127.0.0.1` will not work
(guest arrives on the bridge). Bind to `10.77.0.1` (bridge only) or
`0.0.0.0`:
```nix
services.comfyui.listenAddress = "10.77.0.1";
```
From the guest: `http://10.77.0.1:8188`.
Often better than GPU passthrough when you only need the capability over HTTP
— see [PASSTHROUGH.md](PASSTHROUGH.md).
## Expose a guest service
**Guest side** — agent opens its firewall in `/etc/nixos`:
```nix
networking.firewall.allowedTCPPorts = [ 22 8080 ];
```
**Host only** — already allowed (outbound host→guest):
```bash
curl http://10.77.0.2:8080
```
**Wider network** (NAT mode):
```nix
services.hermenix.publish = [
{ hostPort = 8080; guestPort = 8080; }
];
```
DNATs the uplink port and opens the forward rule. Ports need not match.
Publishing is the one change that lets the outside world start conversations
with the agent's machine — put TLS/auth in front of anything that isn't a toy.
## Bridged mode (guest on the LAN)
Guest becomes a first-class LAN device: fixed address, reachable on any port it
opens, outbound still gated.
```nix
services.hermenix.network = {
bridged = true;
externalInterface = "enp191s0";
lanAddress = "10.0.12.50"; # guest's fixed LAN address
hostLanAddress = "10.0.12.10"; # this host's LAN address
lanGateway = "10.0.12.1";
lanPrefixLength = 24;
# lanDns = [ "1.1.1.1" ]; # optional; defaults to lanGateway
};
```
The bridge absorbs the uplink. The host-side DHCP on the bridge reserves
`lanAddress` for the guest MAC; the guest stays a plain DHCP client. The
bridge clones the uplink MAC so the host keeps its LAN identity. Guest DHCP
broadcast never leaves the bridge.
### Switching over
1. Pick free addresses for guest, host (use the host's existing LAN address if
reserved), and gateway. Keep them outside the DHCP pool or reserve them.
2. Set options above, `sudo nixos-rebuild switch` (brief network drop).
3. Verify: `hermenix vm ssh`; inside, `ip -4 addr` shows `10.77.0.2` **and**
the LAN address. From another LAN device:
`ssh hermes@<guest-lan-address>`.
### Switching back
`bridged = false` and rebuild. Guest DHCP finds no server on the isolated
bridge and falls back to the static route via the host. No guest-side change.
### Gotchas
- `hostLanAddress` is a static LAN claim — reserve it.
- Uplink network config is replaced by the bridge-port role while bridged.
- Older guest images without DHCP-on-NIC in `guest/common.nix` won't take a
bridged address; upgrade the image or apply that config inside the VM first.
- Drop `publish` entries if you return to NAT mode; they do nothing while
bridged.
+124
View File
@@ -0,0 +1,124 @@
# Hardware passthrough & virtual devices
## Before reaching for passthrough
If the goal is "the agent can use the GPU", passthrough is often wrong. It is
exclusive (host loses the card while the VM runs) and is the largest isolation
reduction in this setup.
Prefer: keep Ollama/ComfyUI on the host, open the port with `hostAccess`
([NETWORK.md](NETWORK.md#guest--a-service-on-the-host)). Pass hardware through
when the guest genuinely needs the device — CUDA in-process, capture card,
radio, display output.
## What PCI passthrough costs
A PCI device does DMA. The IOMMU is the only thing between it and host memory,
so `passthrough.iommu` is required; the module refuses to build without it.
- **Whole IOMMU groups only.** Check before planning:
```bash
for d in /sys/kernel/iommu_groups/*/devices/*; do
echo "group $(basename "$(dirname "$(dirname "$d")")"): $(lspci -nns "$(basename "$d")")"
done | sort -V
```
ACS override patches lie about isolation — not appropriate here.
- **Every function.** A GPU is at least graphics + HDMI audio.
## GPU
```bash
lspci -Dnn | grep -i vga
```
```nix
services.hermenix.passthrough = {
iommu = "amd"; # or "intel"
pci = [
{ address = "0000:01:00.0"; id = "10de:2204"; } # graphics
{ address = "0000:01:00.1"; id = "10de:1aef"; } # HDMI audio
];
};
```
Setting `id` makes vfio-pci claim the card in the initrd before `nvidia` /
`amdgpu`. Do this for GPUs. Address-only devices bind at boot via a oneshot
instead.
Rebuild and **reboot**, then:
```bash
lspci -nnk -d 10de:2204 | grep "Kernel driver" # want: vfio-pci
```
Guest-side drivers are the agent's problem (`/etc/nixos`).
## USB / Bluetooth
```bash
lsusb
```
```nix
services.hermenix.passthrough.usb = [
{ vendorId = "8087"; productId = "0026"; }
{ vendorId = "0403"; productId = "6001"; }
];
```
QEMU mediates (no IOMMU/VFIO). Host must not be using the device (`btusb`
blacklist, no `hardware.bluetooth` for an integrated radio). No hotplug —
`hermenix vm restart` after plugging. Whole USB controller as PCI gives real
hotplug at the cost of exclusivity.
## When the VM won't start
`hermenix vm console` first. Then: device not on vfio-pci (`lspci -nnk`),
IOMMU off (`dmesg | grep -i iommu`), or something in the group still held by
the host. The module raises `LimitMEMLOCK` when PCI passthrough is configured.
---
## Virtual devices
`services.hermenix.devices` attaches optional QEMU devices (no host hardware):
| Option | Default | Attaches |
| ------------------------- | ---------- | ----------------------------------------------------- |
| `devices.rng.enable` | `true` | virtio entropy |
| `devices.vsock.enable` | `true` | host↔guest vsock (`ctl`, backups) |
| `devices.video.enable` | `false` | virtual display / render node |
| `devices.video.model` | `"virtio"` | `virtio` (render node) or legacy VGA |
| `devices.balloon.enable` | `false` | virtio memory balloon |
| `devices.watchdog.enable` | `false` | `i6300esb`/`ib700` |
| `devices.input.enable` | `false` | virtio keyboard/tablet |
| `devices.audio.enable` | `false` | `intel-hda`/`virtio-sound` (see caveat) |
| `devices.tpm.enable` | `false` | TPM 2.0 via swtpm |
| `devices.cdroms` | `[]` | read-only CD images |
| `devices.extraDisks` | `[]` | extra raw disks |
| `devices.usbController` | `"auto"` | `qemu-xhci` iff `passthrough.usb` set |
`extraQemuArgs` appends raw QEMU args (unvalidated). Prefer typed options.
### Render node
```nix
services.hermenix.devices.video.enable = true;
```
Gives `/dev/dri/card0` and `/dev/dri/renderD128` (`virtio` model only).
Console stays serial.
### TPM
`devices.tpm.enable = true` starts swtpm; state under
`/var/lib/hermes-vm-tpm` by default.
### Audio caveat
`hermes-vm.service` is hardened (`ProtectHome`, `PrivateTmp`, dedicated user)
and cannot reach PipeWire without loosening that. Leave audio off unless you
mean to relax the unit.
+63
View File
@@ -0,0 +1,63 @@
# Recovery
Cheapest first. Layers 1–2 are inside the guest; 3–4 are host-side hermenix.
```bash
# Guest NixOS history
hermenix vm ssh "git -C /etc/nixos log --oneline"
hermenix vm ssh "sudo nixos-rebuild switch --rollback" # also in boot menu
# State-disk snapshot
hermenix vm snapshot
hermenix vm snapshots
hermenix vm rollback state-20260914-1400 # type 'rollback' to confirm
# Restic backup of state files
hermenix vm backup
hermenix vm backups
```
Identity files under Hermes home can be inspected with `hermenix ctl soul`
etc.; restoring _Hermes_ app state is mostly "roll the state disk" or restore
from backup — not a separate product flow.
## Snapshots
Hourly, keep 48, **state disk only**. System disk: re-image with
`hermenix setup`.
fsfreeze via qemu-guest-agent around each shot. `rollback` stops the VM, moves
current state to `state.old-<timestamp>`, restores snapshot — reversible until
you delete `state.old-*`.
Keep retention tighter than on ZFS: state image is nodatacow; live snapshots
reintroduce CoW on referenced blocks.
## Backups
```nix
services.hermenix.backup = {
enable = true;
repository = "sftp:backup@nas:/hermes";
passwordFile = "/var/lib/secrets/hermes-restic";
};
```
State **files** over vsock/SSH into restic — not the raw disk. Portable;
avoids loop-mounting agent-controlled FS on the host. Runs on the host (no
`lanAccess` needed for a LAN repo).
## Guest won't boot
Re-image **system** only (`hermenix setup` write step); leave state disk.
Identity + `/etc/nixos` + workspace return with it.
## Starting completely fresh
```bash
hermenix setup --clean
```
Stops VM; deletes `system.raw` / `state.raw` / UEFI vars; drops stale
`known_hosts` for the guest host key; walks setup again. Subvolumes/snapshots
kept. Type `clean` to confirm.
+21
View File
@@ -0,0 +1,21 @@
# Known rough edges
- The Determinate NixOS module's option names for custom Nix settings aren't
published — only the nix-darwin `determinateNix.customSettings` equivalent
is. So `guest/common.nix` writes `/etc/nix/nix.custom.conf` via
`environment.etc`. If a future module version claims that file, switch to
its option.
- `determinate-nixd login` is manual. No safe way to bake the credential into
the image.
- `MemoryDenyWriteExecute = true` is correct under KVM but is the first thing
to flip if QEMU refuses to start.
- `pkgs.OVMF.firmware` / `pkgs.OVMF.variables` are current attribute names;
older nixpkgs put these under `pkgs.OVMF.fd` in an `FV/` subdirectory.
- PCI passthrough needs a reboot (early vfio binding in initrd).
- Snapshots need btrfs. Everything else works on any filesystem; without btrfs
you rely on backups alone.
- `state.old-*` left by `hermenix vm rollback` are not cleaned automatically.
Delete them once you're sure.
- First boot builds a lot. Give it time.
- `devices.audio.enable` fights unit hardening — see
[PASSTHROUGH.md](PASSTHROUGH.md#audio-caveat).