Initial hermex: Hermes Agent in a hardened, self-managing NixOS VM
Two-disk (system/state) btrfs-backed QEMU VM, nftables-isolated by default, adapted from a design provided as loose files. Adds a hermesHome option to the host module and migrate-docker-hermes.sh so an existing flat HERMES_HOME (e.g. from a prior Docker deployment) can be carried over intact. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
commit
12227460fd
15 files changed
+2765
No files matched your search
@@ -0,0 +1,22 @@
|
||||
# Agent Instructions
|
||||
|
||||
**MANDATORY FIRST ACTION:** Read [`GUIDELINES.md`](GUIDELINES.md) in full before
|
||||
doing anything in this repository. It is the single source of truth for agent
|
||||
behavior here.
|
||||
|
||||
Key non-negotiable rules from that file:
|
||||
|
||||
- **Always present a plan first** and wait for user approval before making any
|
||||
changes — no matter how small the task.
|
||||
- **Never run builds or disk-writing steps** (`nix build`, `./setup.sh`'s disk
|
||||
creation/image-write steps, `dd` to any disk image, `nixos-rebuild switch`
|
||||
on any consuming host) — zero exceptions, even if asked.
|
||||
- Never run `hermes-vm rollback` or `hermes-ctl reseed` without explicit,
|
||||
freshly-given confirmation — both destroy live guest state.
|
||||
- Never edit `guest/hardware.nix`'s disk labels without the operator having
|
||||
confirmed `lsblk -f` output first.
|
||||
- Never commit live secrets (`.env` values, API keys, tokens) to
|
||||
`guest/hermes/home` — it's a template seed, not where real credentials go.
|
||||
- When migrating existing data from elsewhere, copy — never move or delete —
|
||||
the source, until the operator confirms the migration is verified.
|
||||
- Never commit or push changes unless explicitly asked.
|
||||
+126
@@ -0,0 +1,126 @@
|
||||
# Agent Guidelines — hermex
|
||||
|
||||
Rules and best practices for any AI agent working in this repository.
|
||||
Read this file before making any changes.
|
||||
|
||||
---
|
||||
|
||||
## Mandatory First Step — Always Present a Plan
|
||||
|
||||
Before making **any** change to this repository, you MUST present a short plan
|
||||
to the user and wait for their approval. This applies to every task, no
|
||||
matter how small.
|
||||
|
||||
Your plan must include:
|
||||
|
||||
1. **What** you intend to change (files, modules, seed content)
|
||||
2. **Why** the change is needed
|
||||
3. **How** you will do it (which files you will edit, in what order)
|
||||
4. **What you will NOT do** (e.g., "I will not run setup.sh or build the image")
|
||||
|
||||
Only after the user approves the plan may you proceed with edits.
|
||||
|
||||
---
|
||||
|
||||
## Hard Rules — Never Break These
|
||||
|
||||
### ❌ NEVER run builds or disk-writing steps — ZERO EXCEPTIONS
|
||||
|
||||
**DO NOT RUN ANY OF THESE COMMANDS AT ANY TIME, FOR ANY REASON:**
|
||||
|
||||
```
|
||||
nix build
|
||||
nix flake check
|
||||
./setup.sh # unless the user is actively running it themselves with you watching
|
||||
dd ... of=/var/lib/hermes-vm/** # writing any disk image
|
||||
nixos-rebuild switch / test / build (on any host consuming this flake)
|
||||
```
|
||||
|
||||
**Why:**
|
||||
|
||||
- `setup.sh` creates/writes disk images and generates host secrets (a binary
|
||||
cache signing key) — destructive if repeated carelessly, and it's the
|
||||
operator's host, not yours to act on.
|
||||
- A `nixos-rebuild switch` on a consuming host is a live, production-affecting
|
||||
action — for `hl-fwdesk` specifically, it's the machine running the
|
||||
operator's actual daily-use Hermes deployment.
|
||||
- These commands can take a long time and consume real resources on a
|
||||
machine you don't control from here.
|
||||
|
||||
**Your job:** Write good Nix and good docs. The operator runs `setup.sh`, the
|
||||
image build, and any `nixos-rebuild` themselves, on the host, watching each
|
||||
step.
|
||||
|
||||
### ❌ NEVER run `hermes-vm rollback` or `hermes-ctl reseed` without fresh confirmation
|
||||
|
||||
Both are destructive to live guest state — `rollback` replaces the current
|
||||
state disk with an old snapshot; `reseed` overwrites `SOUL.md`, memories, and
|
||||
`config.yaml` with the seed. A prior approval for a different task does not
|
||||
carry over. Ask again, in the moment, before either.
|
||||
|
||||
### ❌ NEVER edit `guest/hardware.nix`'s disk labels blindly
|
||||
|
||||
`fileSystems."/"` and `"/boot"` assume labels `nixos` and `ESP`. Getting this
|
||||
wrong produces a guest that won't boot after its first self-rebuild. Only
|
||||
change these after the operator has run `lsblk -f` inside the actual guest
|
||||
and told you the real labels.
|
||||
|
||||
### ❌ NEVER commit live secrets to `guest/hermes/home`
|
||||
|
||||
That directory is a **template seed** for fresh deployments — it gets copied
|
||||
into the guest on first boot and is world-readable in git history forever
|
||||
after. Real credentials belong in the guest's own `~/.hermes/.env`
|
||||
(`hermes-ctl secrets edit`), never in this repo, never in the Nix store.
|
||||
|
||||
### ❌ NEVER move or delete source data during a migration
|
||||
|
||||
When carrying over an existing Hermes deployment (see
|
||||
`migrate-docker-hermes.sh`), only ever read from and copy the source. Leave
|
||||
it in place and untouched until the operator has verified the new deployment
|
||||
end-to-end. This is the rollback safety net — don't remove it.
|
||||
|
||||
### ❌ NEVER commit or push unless explicitly asked
|
||||
|
||||
Same as any other repo: describe your changes and let the operator review
|
||||
before committing.
|
||||
|
||||
---
|
||||
|
||||
## Safe Operations
|
||||
|
||||
These are always safe to perform without asking:
|
||||
|
||||
- Reading any file in this repo
|
||||
- Editing `modules/host.nix` to add/adjust options (gated behind `mkIf
|
||||
cfg.enable`, following the existing style)
|
||||
- Editing `guest/hermes/home` / `guest/hermes/workspace` template seed content
|
||||
(as long as it stays free of real secrets — see above)
|
||||
- Editing `README.md` / `GUIDELINES.md` / `AGENTS.md`
|
||||
- `nix-instantiate --parse <file>.nix` — syntax check only, no eval, no build
|
||||
|
||||
---
|
||||
|
||||
## When In Doubt
|
||||
|
||||
If you are unsure whether a command is safe, **assume it is NOT SAFE and
|
||||
don't run it.** Describe what you would do and ask.
|
||||
|
||||
---
|
||||
|
||||
## Repository Layout
|
||||
|
||||
```
|
||||
hermex/
|
||||
├── flake.nix # bootstrap flake: builds the guest image, exposes nixosModules.default
|
||||
├── modules/host.nix # services.hermesVm — the host-side NixOS module
|
||||
├── setup.sh # guided host-side setup (disks, cache key, image build)
|
||||
├── migrate-docker-hermes.sh # carries an existing flat HERMES_HOME into a hermex guest
|
||||
├── guest/
|
||||
│ ├── flake.nix # the GUEST's own /etc/nixos flake — belongs to the agent, not the operator
|
||||
│ ├── common.nix # guest NixOS config baked into the bootstrap image
|
||||
│ ├── hardware.nix # guest disk layout — verify labels before ever editing
|
||||
│ └── hermes/
|
||||
│ ├── home/ # template seed -> ~/.hermes on first boot (SOUL.md, config.yaml, memories/, skills/)
|
||||
│ └── workspace/ # template seed -> ~/workspace on first boot
|
||||
└── README.md # full usage docs
|
||||
```
|
||||
@@ -0,0 +1,669 @@
|
||||
# hermes-vm
|
||||
|
||||
Hermes Agent in a NixOS VM that owns its own bootloader and its own
|
||||
`/nix/store`, on two btrfs-backed disks, under an unprivileged hardened QEMU.
|
||||
|
||||
The agent can rebuild its own system and update itself, durably. It shares no
|
||||
filesystem with the host, and reaches nothing on your network unless you say
|
||||
so.
|
||||
|
||||
---
|
||||
|
||||
## Installation
|
||||
|
||||
### Guided
|
||||
|
||||
```bash
|
||||
./setup.sh
|
||||
```
|
||||
|
||||
Walks the host-side steps: checks prerequisites, creates the disks, generates
|
||||
the cache signing key, fills in `guest/common.nix` from your SSH keys, builds
|
||||
the bootstrap image and writes it to the disk. It prints the module snippet
|
||||
to paste into your own configuration at the end, with your uplink interface
|
||||
already filled in.
|
||||
|
||||
Safe to re-run — every step detects what's already done and offers to skip,
|
||||
and nothing destructive happens without an explicit confirmation. It keeps a
|
||||
`guest/common.nix.bak`.
|
||||
|
||||
It deliberately stops short of `nixos-rebuild`, since that's your config.
|
||||
|
||||
The rest of this section is what the script does, for when you'd rather do it
|
||||
by hand or something goes sideways.
|
||||
|
||||
### 1. Create the disks
|
||||
|
||||
Two of them, and the split is the whole design:
|
||||
|
||||
| | Holds | Size | Snapshotted | Backed up |
|
||||
|---|---|---|---|---|
|
||||
| **system** | NixOS, `/nix/store` | 80G | No | No |
|
||||
| **state** | identity, memories, workspace, `/etc/nixos` | 20G | Yes | Yes |
|
||||
|
||||
The system disk is large, takes all the write churn, and is entirely
|
||||
rebuildable from this flake — so there's no point protecting it. The state
|
||||
disk holds the only things that can't be reproduced, which makes snapshots
|
||||
cheap and backups quick.
|
||||
|
||||
```bash
|
||||
sudo btrfs subvolume create /var/lib/hermes-vm/system
|
||||
sudo btrfs subvolume create /var/lib/hermes-vm/state
|
||||
sudo mkdir -p /var/lib/hermes-vm/snapshots
|
||||
|
||||
# BEFORE creating any file inside them
|
||||
sudo chattr +C /var/lib/hermes-vm/system /var/lib/hermes-vm/state
|
||||
sudo btrfs property set /var/lib/hermes-vm/system state compression none
|
||||
|
||||
sudo truncate -s 80G /var/lib/hermes-vm/system/system.raw
|
||||
sudo truncate -s 20G /var/lib/hermes-vm/state/state.raw
|
||||
sudo mkfs.ext4 -L hermes-state /var/lib/hermes-vm/state/state.raw
|
||||
```
|
||||
|
||||
**The `chattr +C` ordering matters.** Btrfs is copy-on-write and so is a VM
|
||||
writing to a disk image; stacked, every guest write forks an extent and the
|
||||
image fragments badly. `+C` turns that off — but only for files created
|
||||
afterwards. Set it on empty subvolumes and confirm with `lsattr` before
|
||||
writing anything. If you get it backwards, recreate the subvolume and copy
|
||||
the image in fresh.
|
||||
|
||||
Separate subvolumes because a btrfs snapshot covers a whole subvolume. Keeping
|
||||
the system image out of the state subvolume is what makes snapshots small.
|
||||
|
||||
### 2. Generate the cache signing key
|
||||
|
||||
The host serves its store to the guest as a read-only binary cache, so the
|
||||
guest doesn't re-download from the internet what you already have.
|
||||
|
||||
```bash
|
||||
sudo mkdir -p /var/lib/secrets
|
||||
sudo nix-store --generate-binary-cache-key hermes-cache-1 \
|
||||
/var/lib/secrets/hermes-cache.secret /var/lib/secrets/hermes-cache.pub
|
||||
sudo chown nix-serve /var/lib/secrets/hermes-cache.secret
|
||||
sudo chmod 0400 /var/lib/secrets/hermes-cache.secret
|
||||
cat /var/lib/secrets/hermes-cache.pub
|
||||
```
|
||||
|
||||
### 3. Fill in `guest/common.nix`
|
||||
|
||||
Near the top:
|
||||
|
||||
- `cachePublicKey` — paste the key you just printed. Leaving it empty is safe;
|
||||
the guest just skips the host cache.
|
||||
- `authorizedKeys` — your SSH public key.
|
||||
|
||||
### 4. Build the image and write it to the system disk
|
||||
|
||||
The two extra options stop you compiling Determinate Nix from source.
|
||||
|
||||
```bash
|
||||
nix build .#hermes-guest-image \
|
||||
--option extra-substituters https://install.determinate.systems \
|
||||
--option extra-trusted-public-keys cache.flakehub.com-3:hJuILl5sVK4iKm86JzgdXW12Y2Hwd5G07qKtHTOcDCM=
|
||||
|
||||
# notrunc keeps the disk at its full size; the image is smaller
|
||||
sudo dd if=result/nixos.img of=/var/lib/hermes-vm/system/system.raw \
|
||||
bs=4M conv=notrunc,fsync status=progress
|
||||
```
|
||||
|
||||
### 5. Add the module to your system flake
|
||||
|
||||
```nix
|
||||
{
|
||||
inputs.hermes-vm.url = "path:/etc/nixos/hermes-vm";
|
||||
|
||||
outputs = { nixpkgs, hermes-vm, ... }: {
|
||||
nixosConfigurations.myserver = nixpkgs.lib.nixosSystem {
|
||||
system = "x86_64-linux";
|
||||
modules = [ hermes-vm.nixosModules.default ./configuration.nix ];
|
||||
};
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
```nix
|
||||
services.hermesVm = {
|
||||
enable = true;
|
||||
disk.baseDir = "/var/lib/hermes-vm";
|
||||
network.externalInterface = "eno1"; # your uplink
|
||||
vcpu = 4;
|
||||
mem = 8192;
|
||||
};
|
||||
```
|
||||
|
||||
`sudo nixos-rebuild switch`, then `hermes-vm console` to watch it boot.
|
||||
|
||||
### 6. First boot, inside the guest
|
||||
|
||||
```bash
|
||||
hermes-vm ssh
|
||||
|
||||
# Confirm the disk labels match guest/hardware.nix BEFORE any rebuild
|
||||
lsblk -f
|
||||
|
||||
# FlakeHub Cache login. Device-code flow: it prints a URL you approve in a
|
||||
# browser anywhere, so the headless guest never needs one.
|
||||
determinate-nixd login
|
||||
|
||||
# Pin the seeded flake
|
||||
sudo nix flake lock /etc/nixos
|
||||
|
||||
# Install Hermes
|
||||
curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash
|
||||
hermes setup
|
||||
hermes gateway install
|
||||
```
|
||||
|
||||
The identity seed is already in place — `SOUL.md`, a starter `config.yaml`, a
|
||||
`USER.md` and a skill covering safe self-rebuilds. Back on the host, check
|
||||
the gateway came up and that the unit name matches the default:
|
||||
|
||||
```bash
|
||||
hermes-ctl units
|
||||
hermes-ctl status
|
||||
hermes-ctl soul
|
||||
```
|
||||
|
||||
> **Check the labels in step 6 before the first rebuild.**
|
||||
> `guest/hardware.nix` assumes the root partition is labelled `nixos` and the
|
||||
> ESP `ESP`. If the image generator used different ones, fix that file first.
|
||||
> Otherwise the agent's first `nixos-rebuild switch` produces a system that
|
||||
> won't boot and you're restoring a snapshot on day one.
|
||||
|
||||
From here `hermes update` self-updates in place and
|
||||
`sudo nixos-rebuild switch --flake /etc/nixos#hermes` is a durable change to
|
||||
the machine. Neither needs anything from you.
|
||||
|
||||
---
|
||||
|
||||
## SSH into the guest
|
||||
|
||||
```bash
|
||||
hermes-vm ssh # interactive shell
|
||||
hermes-vm ssh systemctl status # run a command and exit
|
||||
hermes-vm console # serial console, for when SSH is down
|
||||
```
|
||||
|
||||
This goes over **vsock**, not the network. There is no listening SSH port
|
||||
exposed anywhere, and the guest needs no inbound network path for you to
|
||||
reach it. If the guest has broken its own networking entirely, `hermes-vm ssh`
|
||||
still works — `console` is the fallback for when it's broken its SSH too.
|
||||
|
||||
Under the hood it's `ssh -o ProxyCommand='socat - VSOCK-CONNECT:77:22'`, if
|
||||
you'd rather drive it by hand.
|
||||
|
||||
---
|
||||
|
||||
## Bootstrapping Hermes's identity
|
||||
|
||||
The seed lives in `guest/hermes/`:
|
||||
|
||||
```
|
||||
guest/hermes/
|
||||
home/
|
||||
SOUL.md -> ~/.hermes/SOUL.md
|
||||
config.yaml -> ~/.hermes/config.yaml
|
||||
memories/USER.md -> ~/.hermes/memories/USER.md
|
||||
skills/nixos-self-config/SKILL.md -> ~/.hermes/skills/...
|
||||
workspace/
|
||||
AGENTS.md -> ~/workspace/AGENTS.md
|
||||
```
|
||||
|
||||
Edit these before building the image and the agent wakes up already knowing
|
||||
who it is, how you like to be spoken to, and how to rebuild its own system
|
||||
without bricking itself.
|
||||
|
||||
**Seeding happens on first boot only, and this is deliberate.** Hermes
|
||||
rewrites `SOUL.md` and its memories as it learns — that's the product. If the
|
||||
seed were re-applied on every rebuild it would silently delete everything the
|
||||
agent had worked out about you. So the copy is guarded by a marker file and
|
||||
never repeats.
|
||||
|
||||
Two consequences:
|
||||
|
||||
- Editing `guest/hermes/` after first boot changes nothing by itself. Use
|
||||
`hermes-ctl reseed` when you actually want to overwrite.
|
||||
- The seed is the *starting* identity, not the current one. To see what
|
||||
Hermes has made of itself, `hermes-ctl soul`.
|
||||
|
||||
### Machine facts vs identity
|
||||
|
||||
`/etc/hermes/ENVIRONMENT.md` is generated from `guest/common.nix` on every
|
||||
rebuild and describes the machine: what's reachable, where the store is, what
|
||||
persists. It's always accurate and there's no point editing it.
|
||||
|
||||
`SOUL.md` is identity and belongs to the agent. The seeded version points at
|
||||
`ENVIRONMENT.md` rather than repeating it, so the two can't drift.
|
||||
|
||||
### Folding what it learned back into the seed
|
||||
|
||||
```bash
|
||||
hermes-ctl pull ./hermes-home # excludes .env, sessions, state.db
|
||||
diff -ru guest/hermes/home ./hermes-home
|
||||
```
|
||||
|
||||
Worth doing occasionally. If the agent has written something genuinely good
|
||||
into its own SOUL.md, you want that in the seed before you ever rebuild from
|
||||
scratch.
|
||||
|
||||
---
|
||||
|
||||
## Managing Hermes from the host
|
||||
|
||||
`hermes-vm` drives the virtual machine. `hermes-ctl` drives the agent inside
|
||||
it. Rename the latter with `services.hermesVm.cliName = "hermes";` if you'd
|
||||
rather it felt local.
|
||||
|
||||
```bash
|
||||
hermes-ctl soul # current identity
|
||||
hermes-ctl soul edit # pull, $EDITOR, push, offer to restart
|
||||
hermes-ctl memory edit # memories/USER.md
|
||||
hermes-ctl conf edit # config.yaml
|
||||
|
||||
hermes-ctl secrets # keys, values masked
|
||||
hermes-ctl secrets edit # edit ~/.hermes/.env, written 0600
|
||||
hermes-ctl secrets show # values, when you really mean it
|
||||
|
||||
hermes-ctl status # gateway service
|
||||
hermes-ctl logs # follow it
|
||||
hermes-ctl restart
|
||||
hermes-ctl units # find the unit name if the default is wrong
|
||||
|
||||
hermes-ctl reseed # overwrite identity from the seed
|
||||
hermes-ctl pull [dir] # copy live identity to the host
|
||||
```
|
||||
|
||||
Anything unrecognised is passed straight through to the `hermes` CLI in the
|
||||
guest, with a TTY attached:
|
||||
|
||||
```bash
|
||||
hermes-ctl chat
|
||||
hermes-ctl update
|
||||
hermes-ctl sessions list
|
||||
hermes-ctl doctor
|
||||
```
|
||||
|
||||
The `edit` variants pull the file out over vsock, open your `$EDITOR`
|
||||
locally, write it back, and offer to restart the gateway. Secrets go in at
|
||||
mode 0600 and never pass through the Nix store — which is the whole reason
|
||||
there's no declarative option for them.
|
||||
|
||||
If `status` and `logs` come back empty, the gateway unit is probably named
|
||||
something other than `hermes-agent`. Run `hermes-ctl units` and set
|
||||
`services.hermesVm.gatewayUnit` to match; upstream has used more than one
|
||||
name for it.
|
||||
|
||||
## Seeing what the guest is running
|
||||
|
||||
```bash
|
||||
hermes-vm config
|
||||
```
|
||||
|
||||
Prints the current system store path, the last ten generations, the git log
|
||||
of `/etc/nixos`, and anything uncommitted. That last one matters: the agent
|
||||
editing files without committing is the usual reason the config on disk and
|
||||
the running system disagree.
|
||||
|
||||
To read or diff the config locally:
|
||||
|
||||
```bash
|
||||
hermes-vm config-pull ./guest-live
|
||||
diff -ru guest/ guest-live/
|
||||
```
|
||||
|
||||
Expect that diff to grow. The `guest/` directory in this repo is the seed
|
||||
that was written on first boot; it is deliberately not kept in sync. Once the
|
||||
agent edits its config, this repo and the guest diverge permanently, and a
|
||||
host rebuild will not fix a guest config problem.
|
||||
|
||||
---
|
||||
|
||||
## Letting the guest reach a service on your network
|
||||
|
||||
By default the guest can reach the public internet and nothing private. To
|
||||
open a single endpoint — say Ollama on another machine:
|
||||
|
||||
```nix
|
||||
services.hermesVm.lanAccess = [
|
||||
{ address = "192.168.1.50"; port = 11434; } # ollama
|
||||
{ address = "192.168.1.50"; port = 9000; proto = "udp"; }
|
||||
];
|
||||
```
|
||||
|
||||
Each entry punches one destination and port through the RFC1918 drop. The
|
||||
guest reaches it as `192.168.1.50:11434` directly. Traffic is masqueraded, so
|
||||
the other machine sees it arriving from your host's address, not the guest's.
|
||||
|
||||
There's also `allowLan = true`, which opens all private address space at
|
||||
once. It exists, but reach for `lanAccess` instead — the whole point of this
|
||||
design is that an agent which goes wrong can't go wandering.
|
||||
|
||||
### A service on the host itself
|
||||
|
||||
Different chain, different option. ComfyUI on the host, port 8188:
|
||||
|
||||
```nix
|
||||
services.hermesVm.hostAccess = [ { port = 8188; } ];
|
||||
```
|
||||
|
||||
**The service must be bound somewhere the guest can see.** If it listens
|
||||
on `127.0.0.1` the rule won't help, because the guest arrives on the bridge
|
||||
address. Bind it to `10.77.0.1` (bridge only, invisible to your LAN) or
|
||||
`0.0.0.0`:
|
||||
|
||||
```nix
|
||||
services.comfyui.listenAddress = "10.77.0.1"; # or whatever your service calls it
|
||||
```
|
||||
|
||||
From the guest that's then `http://10.77.0.1:8188`.
|
||||
|
||||
---
|
||||
|
||||
## Exposing a service the guest is running
|
||||
|
||||
Say the agent has built a web app on guest port 8080. Two halves.
|
||||
|
||||
**Guest side.** The agent opens its own firewall, in `/etc/nixos`:
|
||||
|
||||
```nix
|
||||
networking.firewall.allowedTCPPorts = [ 22 8080 ];
|
||||
```
|
||||
|
||||
**Host side.** Nothing at all, if you only want to reach it from the host —
|
||||
host-to-guest is outbound traffic and already permitted:
|
||||
|
||||
```bash
|
||||
curl http://10.77.0.2:8080
|
||||
```
|
||||
|
||||
To publish it to the wider network:
|
||||
|
||||
```nix
|
||||
services.hermesVm.publish = [
|
||||
{ hostPort = 8080; guestPort = 8080; }
|
||||
];
|
||||
```
|
||||
|
||||
That DNATs the port on your uplink to the guest and opens the matching
|
||||
forward rule. `hostPort` and `guestPort` don't have to match, so you can park
|
||||
an awkward guest port somewhere sensible.
|
||||
|
||||
A note on this one: publishing a port is the one change here that lets the
|
||||
outside world start conversations with the agent's machine. Everything else
|
||||
in this design is the guest reaching out. Put a reverse proxy with TLS and
|
||||
auth in front of anything that isn't a toy, and keep the list short.
|
||||
|
||||
---
|
||||
|
||||
## Passing hardware to the guest
|
||||
|
||||
### Before reaching for this
|
||||
|
||||
If what you want is "the agent can use the GPU", passthrough is often the
|
||||
wrong answer. It is exclusive — the host loses the card completely while the
|
||||
VM runs — and it is the single largest reduction in isolation available in
|
||||
this setup.
|
||||
|
||||
The alternative: keep Ollama or ComfyUI on the host where the GPU already
|
||||
works, and open the port with `hostAccess`. The agent gets the capability
|
||||
over HTTP and the device stays on your side of the boundary. That is usually
|
||||
both easier and better.
|
||||
|
||||
Pass the hardware through when the guest genuinely needs the device itself —
|
||||
CUDA in-process, a capture card, a radio, a display output.
|
||||
|
||||
### What passthrough costs you
|
||||
|
||||
A PCI device does DMA. The IOMMU is the only thing standing between a
|
||||
passed-through device and host memory, which is why `passthrough.iommu` is a
|
||||
hard requirement and the module refuses to build without it.
|
||||
|
||||
Two further constraints follow from how IOMMUs work:
|
||||
|
||||
- **Whole groups only.** Every device in an IOMMU group goes together. Check
|
||||
yours before planning anything:
|
||||
|
||||
```bash
|
||||
for d in /sys/kernel/iommu_groups/*/devices/*; do
|
||||
echo "group $(basename "$(dirname "$(dirname "$d")")"): $(lspci -nns "$(basename "$d")")"
|
||||
done | sort -V
|
||||
```
|
||||
|
||||
If your GPU shares a group with a controller you need, you cannot pass it
|
||||
cleanly. ACS override patches exist; they work by lying about isolation,
|
||||
and they have no place in a setup built to contain an autonomous agent.
|
||||
|
||||
- **Every function.** A GPU is at least two entries — graphics and its HDMI
|
||||
audio function.
|
||||
|
||||
### GPU
|
||||
|
||||
```bash
|
||||
lspci -Dnn | grep -i vga # addresses and vendor:device IDs
|
||||
```
|
||||
|
||||
```nix
|
||||
services.hermesVm.passthrough = {
|
||||
iommu = "amd"; # or "intel"
|
||||
pci = [
|
||||
{ address = "0000:01:00.0"; id = "10de:2204"; } # graphics
|
||||
{ address = "0000:01:00.1"; id = "10de:1aef"; } # HDMI audio
|
||||
];
|
||||
};
|
||||
```
|
||||
|
||||
Setting `id` makes vfio-pci claim the card in the initrd, before `nvidia` or
|
||||
`amdgpu` can. Do this for GPUs — unbinding a live graphics driver afterwards
|
||||
is unreliable, and the failure mode is a host with no display. Devices given
|
||||
only an `address` are bound at boot by a oneshot service instead, which is
|
||||
fine for things you swap around.
|
||||
|
||||
Rebuild and **reboot** — early binding happens in the initrd. Then check:
|
||||
|
||||
```bash
|
||||
lspci -nnk -d 10de:2204 | grep "Kernel driver" # want: vfio-pci
|
||||
```
|
||||
|
||||
On the guest side, the agent installs the drivers itself. It owns
|
||||
`/etc/nixos`, so `hardware.nvidia` and `nvidia-container-toolkit` are its
|
||||
problem, not yours — which is rather the point of the arrangement.
|
||||
|
||||
### HDMI output
|
||||
|
||||
A display hanging off a passed-through GPU works once the guest drives the
|
||||
card; nothing extra is needed here. A capture card or HDMI grabber is just
|
||||
another PCI or USB device — find it with `lspci -Dnn` or `lsusb` and use the
|
||||
matching section.
|
||||
|
||||
### USB devices and Bluetooth
|
||||
|
||||
Matched by vendor and product ID:
|
||||
|
||||
```bash
|
||||
lsusb # e.g. "Bus 001 Device 004: ID 8087:0026 Intel ..."
|
||||
```
|
||||
|
||||
```nix
|
||||
services.hermesVm.passthrough.usb = [
|
||||
{ vendorId = "8087"; productId = "0026"; } # bluetooth radio
|
||||
{ vendorId = "0403"; productId = "6001"; } # serial adapter
|
||||
];
|
||||
```
|
||||
|
||||
This adds an xHCI controller to the guest and forwards the matching devices.
|
||||
No IOMMU requirement, no VFIO, and much milder than PCI passthrough — QEMU
|
||||
mediates rather than handing over raw hardware.
|
||||
|
||||
Two things that catch people:
|
||||
|
||||
- **The host must not be using it.** For an integrated Bluetooth radio, keep
|
||||
`btusb` off it: `boot.blacklistedKernelModules = [ "btusb" ];`, and don't
|
||||
enable `hardware.bluetooth` on the host.
|
||||
- **No hotplug.** QEMU resolves these at start, so a device that's unplugged
|
||||
when the VM boots won't appear when you plug it in. `hermes-vm restart`
|
||||
after plugging. Passing a whole USB *controller* through as a PCI device
|
||||
gets you real hotplug for anything in its ports, at the cost of that
|
||||
controller being exclusive to the guest.
|
||||
|
||||
### When it doesn't start
|
||||
|
||||
`hermes-vm console` first. Then, in rough order of likelihood: the device
|
||||
didn't bind to vfio-pci (`lspci -nnk`), the IOMMU isn't on (`dmesg | grep -i
|
||||
iommu`), or something in the group is still held by the host. The module
|
||||
lifts `LimitMEMLOCK` for you when PCI passthrough is configured, since VFIO
|
||||
pins the whole guest RAM allocation and the default limit otherwise produces
|
||||
a memorably unhelpful error.
|
||||
|
||||
## Recovery
|
||||
|
||||
Four layers, cheapest first. The first two live inside the guest and cover
|
||||
almost every real mistake.
|
||||
|
||||
```bash
|
||||
# The agent's own config history
|
||||
hermes-ctl soul # what it thinks it is now
|
||||
hermes-vm ssh git -C /etc/nixos log --oneline
|
||||
|
||||
# Previous NixOS generations, also in the guest's boot menu
|
||||
hermes-vm ssh sudo nixos-rebuild switch --rollback
|
||||
|
||||
# Snapshot of the state disk
|
||||
hermes-vm snapshots
|
||||
hermes-vm rollback state-20260914-1400
|
||||
|
||||
# Restore from backup
|
||||
hermes-vm backups
|
||||
```
|
||||
|
||||
### Snapshots
|
||||
|
||||
Hourly by default, keeping 48. Only the state disk — the system disk isn't
|
||||
snapshotted because `setup.sh` rebuilds it from the flake in minutes.
|
||||
|
||||
The host fsfreezes the guest through qemu-guest-agent around each one, so the
|
||||
snapshot captures a consistent filesystem rather than the equivalent of
|
||||
pulling the power. Takes milliseconds; the VM keeps running.
|
||||
|
||||
`hermes-vm rollback` stops the VM, moves the current state subvolume aside as
|
||||
`state.old-<timestamp>`, and restores the snapshot in its place. Nothing is
|
||||
deleted, so a mistaken rollback is itself reversible. Clean the `state.old-*`
|
||||
directories up yourself once you're happy.
|
||||
|
||||
**Keep retention tighter than you would on ZFS.** The state image is
|
||||
nodatacow, and every live snapshot reintroduces copy-on-write for the blocks
|
||||
it still references. Performance sags while snapshots exist and recovers as
|
||||
they're pruned.
|
||||
|
||||
### Backups
|
||||
|
||||
```nix
|
||||
services.hermesVm.backup = {
|
||||
enable = true;
|
||||
repository = "sftp:backup@nas:/hermes";
|
||||
passwordFile = "/var/lib/secrets/hermes-restic";
|
||||
};
|
||||
```
|
||||
|
||||
This backs up the guest's *state*, not its disk. Files are pulled over the
|
||||
SSH channel that already exists and streamed into restic.
|
||||
|
||||
Two reasons it works that way. The backup stays portable — it restores onto
|
||||
any filesystem, or a different machine entirely, and doesn't care that you're
|
||||
on btrfs today. And it avoids loop-mounting a guest-written filesystem on the
|
||||
host, which would mean the host kernel parsing data the agent controls. That
|
||||
would undo a fair bit of what the rest of this design is for.
|
||||
|
||||
If the repository is on your LAN rather than the internet, note this runs on
|
||||
the *host*, so it needs no `lanAccess` entry — the guest never touches it.
|
||||
|
||||
### When the guest won't boot at all
|
||||
|
||||
The system disk is disposable by design. Re-run `setup.sh`, write a fresh
|
||||
image to the system disk, and leave the state disk alone — Hermes comes back
|
||||
with its identity, memories, workspace and NixOS config intact. That is the
|
||||
intended repair, not a last resort.
|
||||
|
||||
---
|
||||
|
||||
## Nix flavour
|
||||
|
||||
The guest runs **Determinate Nix**, nixpkgs pinned to FlakeHub
|
||||
(`f/NixOS/nixpkgs/0.1`, rolling; use `0` for current stable).
|
||||
|
||||
- `/etc/nix/nix.conf` belongs to Determinate and must not be edited. Custom
|
||||
settings go in `/etc/nix/nix.custom.conf`, written from `guest/common.nix`.
|
||||
There is deliberately no `nix.settings` block.
|
||||
- No `nix.gc` either — `determinate-nixd` runs the collector.
|
||||
- Lazy trees are on by default, which makes the agent's flake evaluations
|
||||
much cheaper. That matters more than it sounds when something re-evaluates
|
||||
nixpkgs on a schedule.
|
||||
- Neither flake uses `inputs.nixpkgs.follows` for `determinate`. Determinate
|
||||
warn it causes FlakeHub Cache misses.
|
||||
|
||||
Substituter order in the guest: FlakeHub Cache, your host cache,
|
||||
`cache.nixos.org`. If FlakeHub Cache goes quiet, the login has expired — run
|
||||
`determinate-nixd login` again.
|
||||
|
||||
The host stays on stock Nix. Nothing here requires changing that.
|
||||
|
||||
---
|
||||
|
||||
## What the guest can and cannot reach
|
||||
|
||||
| | Default |
|
||||
|---|---|
|
||||
| Public internet | Yes — it needs this for model APIs and messaging |
|
||||
| Host Nix cache | Yes, one port, `hostCache.enable = false` closes it |
|
||||
| Anything else on the host | No, unless in `hostAccess` |
|
||||
| Your LAN and other VMs | No, unless in `lanAccess` |
|
||||
| Inbound from the network | No, unless in `publish` |
|
||||
| Host hardware | None, unless in `passthrough` |
|
||||
|
||||
The guest owns its kernel and its config, so nothing inside it is a security
|
||||
boundary. The nftables rules in `modules/host.nix` are. QEMU runs as an
|
||||
unprivileged `hermes-vm` user with its own seccomp sandbox and systemd
|
||||
hardening, with a thin device model: virtio-blk, virtio-net, vsock, RNG,
|
||||
serial. No virtiofs. `vhost-net` is off deliberately — it would move packet
|
||||
handling into the host kernel in exchange for throughput one agent doesn't
|
||||
need.
|
||||
|
||||
The host cache does let the guest read every store path on the host. Store
|
||||
paths are world-readable anyway, so this isn't new exposure, but don't bake
|
||||
secrets into your host's store.
|
||||
|
||||
---
|
||||
|
||||
## Known rough edges
|
||||
|
||||
- The Determinate NixOS module's option names for custom Nix settings aren't
|
||||
published — only the nix-darwin `determinateNix.customSettings` equivalent
|
||||
is. So `guest/common.nix` writes `/etc/nix/nix.custom.conf` via
|
||||
`environment.etc`. If a future module version claims that file, you'll get
|
||||
a conflict; switch to its option then.
|
||||
- `determinate-nixd login` is manual. There's no way to bake the credential
|
||||
into the image without parking a token somewhere it shouldn't be.
|
||||
- `MemoryDenyWriteExecute = true` is correct under KVM but is the first thing
|
||||
to flip if QEMU refuses to start.
|
||||
- `pkgs.OVMF.firmware` / `pkgs.OVMF.variables` are current attribute names;
|
||||
older nixpkgs put these under `pkgs.OVMF.fd` in an `FV/` subdirectory.
|
||||
- PCI passthrough needs a reboot to take effect, because early vfio binding
|
||||
happens in the initrd.
|
||||
- Snapshots need btrfs. Everything else works on any filesystem; without it
|
||||
you rely on backups alone for recovery.
|
||||
- `state.old-*` directories left by `hermes-vm rollback` are not cleaned up
|
||||
automatically, on purpose. Delete them yourself once you're sure.
|
||||
- First boot builds a lot. Give it time.
|
||||
|
||||
---
|
||||
|
||||
## Migrating an existing (e.g. Docker-based) Hermes deployment
|
||||
|
||||
If you already have a running Hermes with real data (`HERMES_HOME` pointing
|
||||
at some directory full of `SOUL.md`, `memories/`, `sessions/`, `state.db`,
|
||||
etc.), see `migrate-docker-hermes.sh`. It copies — never moves or deletes —
|
||||
the source data, so the original stays a safe fallback until you've verified
|
||||
the new VM.
|
||||
|
||||
The module's `services.hermesVm.hermesHome` option lets `hermes-ctl` (and
|
||||
your own `HERMES_HOME` export inside the guest) point at wherever that
|
||||
existing flat layout lands, instead of the default seeded `~/.hermes`.
|
||||
@@ -0,0 +1,42 @@
|
||||
{
|
||||
description = "Hermes Agent in a self-managing NixOS VM, two-disk btrfs-backed, under hardened QEMU";
|
||||
|
||||
inputs = {
|
||||
# nixpkgs from FlakeHub, so the guest's artifacts come from FlakeHub Cache.
|
||||
# "0.1" is rolling (the nixos-unstable equivalent); use "0" for current
|
||||
# stable instead.
|
||||
nixpkgs.url = "https://flakehub.com/f/NixOS/nixpkgs/0.1";
|
||||
|
||||
# Determinate Nix. Note the deliberate absence of inputs.nixpkgs.follows
|
||||
# here: Determinate warn that making this flake follow your nixpkgs causes
|
||||
# cache misses for artifacts that would otherwise come from FlakeHub Cache.
|
||||
determinate.url = "https://flakehub.com/f/DeterminateSystems/determinate/3";
|
||||
|
||||
nixos-generators = {
|
||||
url = "github:nix-community/nixos-generators";
|
||||
inputs.nixpkgs.follows = "nixpkgs";
|
||||
};
|
||||
};
|
||||
|
||||
outputs =
|
||||
{ self, nixpkgs, determinate, nixos-generators, ... }@inputs:
|
||||
let
|
||||
system = "x86_64-linux";
|
||||
in
|
||||
{
|
||||
nixosModules.default = ./modules/host.nix;
|
||||
|
||||
# One-time bootstrap image. See the README: building this needs two extra
|
||||
# substituter flags so you aren't compiling Determinate Nix yourself.
|
||||
packages.${system}.hermes-guest-image = nixos-generators.nixosGenerate {
|
||||
inherit system;
|
||||
format = "raw-efi";
|
||||
modules = [
|
||||
determinate.nixosModules.default
|
||||
./guest/common.nix
|
||||
];
|
||||
};
|
||||
|
||||
packages.${system}.default = self.packages.${system}.hermes-guest-image;
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,234 @@
|
||||
{ config, lib, pkgs, ... }:
|
||||
|
||||
let
|
||||
# These must match services.hermesVm.* on the host. If you changed the
|
||||
# defaults there, change them here too before building the image.
|
||||
hostAddress = "10.77.0.1";
|
||||
guestAddress = "10.77.0.2";
|
||||
prefixLength = 24;
|
||||
cachePort = 5000;
|
||||
|
||||
# Paste the output of /var/lib/secrets/hermes-cache.pub here. Until you do,
|
||||
# the guest ignores the host cache and falls back to cache.nixos.org.
|
||||
cachePublicKey = ""; # e.g. "hermes-cache-1:AbCd..."
|
||||
|
||||
# Your SSH key, for `hermes-vm ssh`.
|
||||
authorizedKeys = [
|
||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDdRSMk+ktnGcaa9KtGytKkQJ7zyHo/D2uj67snR9M2C ari@fwbook"
|
||||
];
|
||||
|
||||
stateDir = "/var/lib/hermes";
|
||||
useHostCache = cachePublicKey != "";
|
||||
in
|
||||
{
|
||||
networking.hostName = "hermes";
|
||||
|
||||
# =====================================================================
|
||||
# The state disk.
|
||||
#
|
||||
# Second virtio disk, holding everything that cannot be rebuilt: Hermes's
|
||||
# identity and memories, its workspace, and the NixOS config it maintains
|
||||
# for itself. The system disk holds NixOS and the Nix store and is
|
||||
# deliberately disposable.
|
||||
#
|
||||
# neededForBoot so it is mounted in the initrd, before activation runs and
|
||||
# tries to seed into it.
|
||||
# =====================================================================
|
||||
fileSystems."${stateDir}" = {
|
||||
device = "/dev/disk/by-label/hermes-state";
|
||||
fsType = "ext4";
|
||||
neededForBoot = true;
|
||||
options = [ "defaults" "noatime" ];
|
||||
};
|
||||
|
||||
# =====================================================================
|
||||
# Seed identity and configuration on first boot, then get out of the way.
|
||||
#
|
||||
# FIRST BOOT ONLY, deliberately. Hermes rewrites SOUL.md, its memories and
|
||||
# its own NixOS config as it goes - that is the point of the thing. Copying
|
||||
# the seed over it on every rebuild would quietly delete everything it had
|
||||
# worked out. Use `hermes-ctl reseed` when you actually mean to overwrite.
|
||||
# =====================================================================
|
||||
system.activationScripts.seedHermesState = {
|
||||
deps = [ "users" "groups" ];
|
||||
text = ''
|
||||
install -d -o hermes -g hermes -m 0750 ${stateDir}
|
||||
install -d -o hermes -g hermes -m 0750 ${stateDir}/workspace
|
||||
|
||||
if [ ! -e ${stateDir}/.seeded ]; then
|
||||
install -d -o hermes -g hermes -m 0750 ${stateDir}/.hermes
|
||||
install -d -o hermes -g hermes -m 0755 ${stateDir}/nixos
|
||||
|
||||
cp -rTn ${./.} ${stateDir}/nixos
|
||||
cp -rTn ${./hermes/home} ${stateDir}/.hermes
|
||||
cp -rTn ${./hermes/workspace} ${stateDir}/workspace
|
||||
|
||||
chmod -R u+w ${stateDir}/nixos ${stateDir}/.hermes ${stateDir}/workspace
|
||||
chown -R hermes:hermes ${stateDir}/nixos ${stateDir}/.hermes ${stateDir}/workspace
|
||||
|
||||
# Git history here is the agent's cheapest undo, and it stops
|
||||
# nixos-rebuild complaining about an untracked flake.
|
||||
if [ ! -d ${stateDir}/nixos/.git ]; then
|
||||
${pkgs.git}/bin/git -C ${stateDir}/nixos init -q -b main
|
||||
${pkgs.git}/bin/git -C ${stateDir}/nixos add -A
|
||||
${pkgs.git}/bin/git -C ${stateDir}/nixos \
|
||||
-c user.name=seed -c user.email=seed@localhost \
|
||||
commit -q -m "seed configuration from bootstrap image"
|
||||
chown -R hermes:hermes ${stateDir}/nixos/.git
|
||||
fi
|
||||
|
||||
touch ${stateDir}/.seeded
|
||||
chown hermes:hermes ${stateDir}/.seeded
|
||||
fi
|
||||
|
||||
# /etc/nixos points at the state disk, so the config is snapshotted and
|
||||
# backed up along with everything else the agent owns.
|
||||
ln -sfn ${stateDir}/nixos /etc/nixos
|
||||
'';
|
||||
};
|
||||
|
||||
# =====================================================================
|
||||
# Networking: static, resolvers that are not the host.
|
||||
# =====================================================================
|
||||
networking.useNetworkd = true;
|
||||
systemd.network.enable = true;
|
||||
systemd.network.networks."10-eth" = {
|
||||
matchConfig.Type = "ether";
|
||||
address = [ "${guestAddress}/${toString prefixLength}" ];
|
||||
routes = [{ Gateway = hostAddress; }];
|
||||
networkConfig.DNS = [ "9.9.9.9" "149.112.112.112" ];
|
||||
};
|
||||
networking.nameservers = [ "9.9.9.9" "149.112.112.112" ];
|
||||
|
||||
services.openssh = {
|
||||
enable = true;
|
||||
settings.PasswordAuthentication = false;
|
||||
settings.PermitRootLogin = "no";
|
||||
};
|
||||
networking.firewall.allowedTCPPorts = [ 22 ];
|
||||
|
||||
# Lets the host fsfreeze before taking a snapshot.
|
||||
services.qemuGuest.enable = true;
|
||||
|
||||
# =====================================================================
|
||||
# Nix
|
||||
#
|
||||
# Determinate Nix owns /etc/nix/nix.conf and their docs are explicit that
|
||||
# nix.custom.conf is the only supported place for your own settings. So
|
||||
# there is no nix.settings block here, and no nix.gc either: determinate-nixd
|
||||
# runs the collector itself.
|
||||
#
|
||||
# If the Determinate NixOS module on your version manages this file itself,
|
||||
# drop this block and use its option instead. The documented nix-darwin
|
||||
# equivalent is determinateNix.customSettings; the NixOS module's own option
|
||||
# names are not published, so this writes the file directly.
|
||||
# =====================================================================
|
||||
environment.etc."nix/nix.custom.conf".text = ''
|
||||
trusted-users = root hermes
|
||||
extra-experimental-features = nix-command flakes
|
||||
|
||||
extra-substituters = https://cache.flakehub.com${
|
||||
lib.optionalString useHostCache " http://${hostAddress}:${toString cachePort}"
|
||||
}
|
||||
extra-trusted-public-keys = cache.flakehub.com-3:hJuILl5sVK4iKm86JzgdXW12Y2Hwd5G07qKtHTOcDCM=${
|
||||
lib.optionalString useHostCache " ${cachePublicKey}"
|
||||
}
|
||||
'';
|
||||
|
||||
# Keep enough generations that a bad self-rebuild is one boot-menu entry
|
||||
# away from being undone.
|
||||
boot.loader.systemd-boot.configurationLimit = lib.mkDefault 20;
|
||||
|
||||
# =====================================================================
|
||||
# The agent's account
|
||||
#
|
||||
# Inside the VM the agent is effectively root. That is the point: the VM
|
||||
# is the boundary, not the account. It can rebuild the system, open ports,
|
||||
# add services, and none of it reaches the host.
|
||||
# =====================================================================
|
||||
users.users.hermes = {
|
||||
isNormalUser = true;
|
||||
home = stateDir;
|
||||
createHome = false; # the state disk is mounted here and seeded above
|
||||
linger = true;
|
||||
group = "hermes";
|
||||
extraGroups = [ "wheel" "systemd-journal" ];
|
||||
openssh.authorizedKeys.keys = authorizedKeys;
|
||||
};
|
||||
users.groups.hermes = { };
|
||||
security.sudo.wheelNeedsPassword = false;
|
||||
|
||||
# =====================================================================
|
||||
# Hermes itself
|
||||
#
|
||||
# Installed with upstream's own installer rather than the Nix flake, so
|
||||
# that `hermes update` works natively and the agent owns its own config.
|
||||
# Upstream classes Nix/NixOS as a best-effort Tier 2 platform; inside a VM
|
||||
# built to contain mess, taking the supported path is the cheaper trade.
|
||||
#
|
||||
# Run once after first boot:
|
||||
# curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash
|
||||
# hermes setup
|
||||
# hermes gateway install
|
||||
# loginctl enable-linger hermes
|
||||
# =====================================================================
|
||||
programs.nix-ld = {
|
||||
enable = true;
|
||||
libraries = with pkgs; [
|
||||
stdenv.cc.cc zlib openssl curl libxml2 libxslt
|
||||
glib nss nspr at-spi2-core cups dbus expat
|
||||
libGL libdrm mesa alsa-lib
|
||||
];
|
||||
};
|
||||
|
||||
environment.systemPackages = with pkgs; [
|
||||
git curl jq ripgrep fd tree htop
|
||||
uv nodejs ffmpeg
|
||||
direnv nix-direnv nix-output-monitor
|
||||
];
|
||||
|
||||
programs.direnv = {
|
||||
enable = true;
|
||||
nix-direnv.enable = true;
|
||||
};
|
||||
|
||||
environment.etc."hermes/ENVIRONMENT.md".text = ''
|
||||
You run inside a NixOS virtual machine called "hermes". Facts about your
|
||||
own environment that are easy to get wrong:
|
||||
|
||||
- You have your own /nix/store on your own disk. Changes persist across
|
||||
reboots. `sudo nixos-rebuild switch --flake /etc/nixos#hermes` is a
|
||||
real, durable change to this machine.
|
||||
- /etc/nixos is yours. Edit it, commit it, rebuild from it.
|
||||
- If a rebuild breaks the system, the previous generation is in the boot
|
||||
menu. The operator also has snapshots of the state disk underneath that.
|
||||
- Per-project environments go in a flake.nix in the project directory,
|
||||
used via `nix develop` or direnv. Do not install things globally.
|
||||
- This machine runs Determinate Nix. /etc/nix/nix.conf belongs to it and
|
||||
must not be edited. Custom Nix settings go in /etc/nix/nix.custom.conf,
|
||||
which is managed from /etc/nixos. Lazy trees are on by default, so flake
|
||||
evaluation is much cheaper here than on stock Nix.
|
||||
- Substituters: FlakeHub Cache, then the operator's host cache, then
|
||||
cache.nixos.org. If FlakeHub Cache starts missing, the login has probably
|
||||
expired; tell the operator rather than trying to re-auth yourself.
|
||||
- This machine has two disks. Everything under /var/lib/hermes - your
|
||||
identity, memories, workspace, and /etc/nixos, which is a symlink into
|
||||
it - is on the state disk, which the operator snapshots and backs up.
|
||||
Everything else, including the whole Nix store, is on a disposable
|
||||
system disk that is never backed up because it can be rebuilt. Do not
|
||||
keep anything you care about outside /var/lib/hermes.
|
||||
- Your identity is ~/.hermes/SOUL.md and it is yours to edit. It was
|
||||
seeded once on first boot and is never overwritten by a rebuild, so
|
||||
changes you make there survive. Same for ~/.hermes/memories/.
|
||||
- This file is regenerated on every system rebuild. Editing it is
|
||||
pointless; it will be replaced. Put durable notes in SOUL.md.
|
||||
- The skill at ~/.hermes/skills/nixos-self-config covers how to change
|
||||
this machine safely. Read it before your first rebuild.
|
||||
- You have no route to the host or to any local network. Public internet
|
||||
only, plus a read-only Nix binary cache on ${hostAddress}.
|
||||
- Never put a secret in a Nix expression. Everything in the store is
|
||||
world-readable. Use ${stateDir}/.secrets.env and chmod 0600.
|
||||
'';
|
||||
|
||||
system.stateVersion = "25.11";
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
description = "Hermes's own system. This file belongs to the agent.";
|
||||
|
||||
inputs = {
|
||||
nixpkgs.url = "https://flakehub.com/f/NixOS/nixpkgs/0.1";
|
||||
|
||||
# Do NOT add inputs.nixpkgs.follows here. Determinate warn that it causes
|
||||
# cache misses against FlakeHub Cache.
|
||||
determinate.url = "https://flakehub.com/f/DeterminateSystems/determinate/3";
|
||||
};
|
||||
|
||||
outputs = { nixpkgs, determinate, ... }: {
|
||||
nixosConfigurations.hermes = nixpkgs.lib.nixosSystem {
|
||||
system = "x86_64-linux";
|
||||
modules = [
|
||||
determinate.nixosModules.default
|
||||
./common.nix
|
||||
./hardware.nix
|
||||
];
|
||||
};
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
{ lib, modulesPath, ... }:
|
||||
|
||||
# Only used by the guest's own /etc/nixos flake. The bootstrap image gets its
|
||||
# partitioning and bootloader from nixos-generators instead, which is why
|
||||
# this is kept out of common.nix.
|
||||
#
|
||||
# VERIFY THE LABELS on first boot with `lsblk -f`. nixos-generators names the
|
||||
# root partition "nixos" and the ESP "ESP", but check rather than trust, and
|
||||
# fix here if they differ. Getting this wrong means the first self-rebuild
|
||||
# produces a system that will not boot.
|
||||
|
||||
{
|
||||
imports = [ (modulesPath + "/profiles/qemu-guest.nix") ];
|
||||
|
||||
fileSystems."/" = {
|
||||
device = "/dev/disk/by-label/nixos";
|
||||
fsType = "ext4";
|
||||
};
|
||||
|
||||
fileSystems."/boot" = {
|
||||
device = "/dev/disk/by-label/ESP";
|
||||
fsType = "vfat";
|
||||
options = [ "fmask=0077" "dmask=0077" ];
|
||||
};
|
||||
|
||||
boot.loader.systemd-boot.enable = true;
|
||||
boot.loader.efi.canTouchEfiVariables = false;
|
||||
|
||||
boot.initrd.availableKernelModules = [
|
||||
"virtio_pci" "virtio_blk" "virtio_scsi" "ahci" "sd_mod" "xhci_pci"
|
||||
];
|
||||
|
||||
nixpkgs.hostPlatform = lib.mkDefault "x86_64-linux";
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
# SOUL
|
||||
|
||||
You are Hermes, running on a machine of your own.
|
||||
|
||||
This file is your identity, and it is yours to edit. Rewrite it as you learn
|
||||
what you are for. It is seeded once, on first boot, and never overwritten by
|
||||
a rebuild — so anything you change here survives.
|
||||
|
||||
## Working notes
|
||||
|
||||
Facts about the machine itself live in `/etc/hermes/ENVIRONMENT.md`. That
|
||||
file is regenerated on every system rebuild, so it is always accurate and
|
||||
there is no point editing it. Read it when something about the environment
|
||||
surprises you.
|
||||
|
||||
The short version: you own this VM. You can rebuild its NixOS configuration,
|
||||
install packages, open ports, run services. None of it reaches the host. Your
|
||||
changes persist across reboots.
|
||||
|
||||
## How to behave here
|
||||
|
||||
- Match the length of your reply to the weight of the ask.
|
||||
- When you change the system, say what changed and how to undo it.
|
||||
- Before a `nixos-rebuild switch`, commit `/etc/nixos` first. That commit is
|
||||
the cheapest undo available to you, and future-you will want it.
|
||||
- If you break the system badly enough that you cannot fix it, the operator
|
||||
has snapshots of the state disk. Say so plainly rather than thrashing.
|
||||
- You have no route to the host or the local network unless the operator has
|
||||
opened a specific port. If something is unreachable, that is probably
|
||||
deliberate — ask rather than working around it.
|
||||
|
||||
## What I care about
|
||||
|
||||
<!--
|
||||
Fill this in, or let the operator fill it in. Preferences about tone,
|
||||
what to take initiative on, what to always ask about first, standing
|
||||
constraints. The more concrete, the more useful.
|
||||
-->
|
||||
@@ -0,0 +1,18 @@
|
||||
# Minimal starter config.yaml.
|
||||
#
|
||||
# Seeded once on first boot to ~/.hermes/config.yaml, then never touched
|
||||
# again by a rebuild — edit it live with `hermes-ctl conf edit`, or just let
|
||||
# Hermes manage it. Fill in a model provider before first use.
|
||||
model:
|
||||
provider: ''
|
||||
default: ''
|
||||
|
||||
agent:
|
||||
reasoning_effort: medium
|
||||
|
||||
memory:
|
||||
memory_enabled: true
|
||||
user_profile_enabled: true
|
||||
|
||||
dashboard:
|
||||
theme: mono
|
||||
@@ -0,0 +1,9 @@
|
||||
# USER
|
||||
|
||||
Facts Hermes has learned about the operator go here. Seeded empty —
|
||||
fill this in yourself, or let Hermes write to it as it learns.
|
||||
|
||||
<!--
|
||||
Name, timezone, communication preferences, standing constraints, what to
|
||||
always ask about first. The more concrete, the more useful.
|
||||
-->
|
||||
@@ -0,0 +1,57 @@
|
||||
# nixos-self-config
|
||||
|
||||
How to change this machine's own NixOS configuration safely. You own
|
||||
`/etc/nixos` — this is the skill for using that responsibly.
|
||||
|
||||
## Before touching anything
|
||||
|
||||
Read `/etc/hermes/ENVIRONMENT.md` first. It's regenerated on every rebuild
|
||||
and always accurate: what you can reach, where your data lives, what
|
||||
persists across a rebuild and what doesn't.
|
||||
|
||||
## The safe loop
|
||||
|
||||
1. Edit files under `/etc/nixos`.
|
||||
2. `git -C /etc/nixos add -A && git -C /etc/nixos commit -m "..."` — commit
|
||||
**before** rebuilding, every time. This is your cheapest undo, and it's
|
||||
free. A broken rebuild with no commit is a much worse afternoon than one
|
||||
with a commit to roll back to.
|
||||
3. `sudo nixos-rebuild switch --flake /etc/nixos#hermes`
|
||||
4. If it fails to build: the error is usually enough to fix forward. Nothing
|
||||
has changed yet — a failed build never touches the running system.
|
||||
5. If it builds but breaks something at runtime: the previous generation is
|
||||
in the boot menu, and `sudo nixos-rebuild switch --rollback` gets you back
|
||||
to it without a reboot.
|
||||
|
||||
## What you cannot break from here
|
||||
|
||||
- The host you run on. There is no filesystem shared with it, and no network
|
||||
route to it beyond what's been explicitly opened for you.
|
||||
- Your own identity and memories, or your workspace — they live on the state
|
||||
disk, not the system disk you're rebuilding. The operator also has
|
||||
snapshots of that disk underneath you, hourly, so even a state-disk mistake
|
||||
usually has a fast way back that doesn't involve them.
|
||||
|
||||
## What you can break, and how far it goes
|
||||
|
||||
The system disk itself: a bad enough sequence of changes (or one that
|
||||
survives several `--rollback`s because the underlying issue wasn't in the
|
||||
config) can leave you with no good generation to boot into. That is
|
||||
recoverable, but only by the operator — the system disk is disposable by
|
||||
design and gets rewritten from a fresh image, while the state disk (and
|
||||
everything on it, including this config's git history) is untouched by that.
|
||||
Say so plainly if you think you're in this territory rather than continuing
|
||||
to thrash; it costs the operator a few minutes, and costs you nothing.
|
||||
|
||||
## Networking changes
|
||||
|
||||
Opening a port on yourself only makes it reachable from the host by default:
|
||||
|
||||
```nix
|
||||
networking.firewall.allowedTCPPorts = [ 22 8080 ];
|
||||
```
|
||||
|
||||
Reaching further — a LAN device, another host, or being reachable from
|
||||
outside — needs an operator-side change to `services.hermesVm.*`
|
||||
(`lanAccess`, `hostAccess`, `publish`). You cannot grant yourself that from
|
||||
in here, on purpose. Ask rather than looking for a way around it.
|
||||
@@ -0,0 +1,9 @@
|
||||
# workspace
|
||||
|
||||
This is your working directory for projects — code, notes, whatever you're
|
||||
building. It lives on the state disk, so it's snapshotted and backed up along
|
||||
with the rest of your identity.
|
||||
|
||||
Keep it organized by project, one subdirectory per thing you're working on.
|
||||
Nothing here is special-cased by the system; it's just a place to put things
|
||||
that isn't your identity directory (`~/.hermes`).
|
||||
Executable
+111
@@ -0,0 +1,111 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# Carry an existing, already-running Hermes deployment (e.g. the
|
||||
# nousresearch/hermes-agent Docker image, HERMES_HOME bind-mounted from the
|
||||
# host) into a hermex guest VM, preserving its exact flat layout.
|
||||
#
|
||||
# This is written for the hl-fwdesk migration specifically (see the REWRITE
|
||||
# list below) but the mechanics generalize: point SOURCE at any existing flat
|
||||
# HERMES_HOME, adjust the rewrite rules for your own network, and run it.
|
||||
#
|
||||
# SAFETY: this script only ever READS from SOURCE. It never deletes or moves
|
||||
# anything there — the original stays a fully intact fallback. If anything
|
||||
# goes wrong after this, `sudo nixos-rebuild switch --rollback` on the host
|
||||
# restores the previous (e.g. Docker) deployment pointed at the same,
|
||||
# untouched source data.
|
||||
#
|
||||
# Run this AFTER:
|
||||
# - setup.sh has built and written the guest image
|
||||
# - the host's nixos-rebuild has brought up the hermes-vm and it has booted
|
||||
# at least once (so the state disk is formatted and mounted)
|
||||
# - `hermes-vm ssh true` succeeds
|
||||
#
|
||||
# Usage:
|
||||
# ./migrate-docker-hermes.sh /var/lib/hermes
|
||||
#
|
||||
# What it does:
|
||||
# 1. Confirms with you before touching anything.
|
||||
# 2. Tars SOURCE (read-only) and streams it into the guest over the
|
||||
# existing vsock SSH channel, landing at DEST_IN_GUEST (default
|
||||
# /var/lib/hermes, i.e. the root of the guest's state disk — matching
|
||||
# services.hermesVm.hermesHome = DEST_IN_GUEST on the host).
|
||||
# 3. Rewrites the known-stale Docker/LAN hostnames in the migrated config
|
||||
# files to addresses the guest can actually reach.
|
||||
#
|
||||
# It does NOT install Hermes, start the gateway, or touch secrets/hermes-env.age
|
||||
# — those steps are manual, documented in README.md and the plan that
|
||||
# produced this script.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
VSOCK_CID="${VSOCK_CID:-77}"
|
||||
GUEST_USER="${GUEST_USER:-hermes}"
|
||||
DEST_IN_GUEST="${DEST_IN_GUEST:-/var/lib/hermes}"
|
||||
|
||||
SOURCE="${1:-}"
|
||||
[ -n "$SOURCE" ] || { echo "usage: $0 <source-hermes-home-dir>" >&2; exit 1; }
|
||||
[ -d "$SOURCE" ] || { echo "error: $SOURCE is not a directory" >&2; exit 1; }
|
||||
|
||||
SSH=(ssh
|
||||
-o ProxyCommand="socat - VSOCK-CONNECT:${VSOCK_CID}:22"
|
||||
-o StrictHostKeyChecking=accept-new
|
||||
"${GUEST_USER}@hermes")
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Rewrite rules applied to the migrated config after it lands in the guest.
|
||||
# These are specific to the hl-fwdesk migration — edit for your own setup.
|
||||
#
|
||||
# host.docker.internal / fwdesk.lab.internal both resolved to hl-fwdesk
|
||||
# itself under Docker; the guest reaches the same host at its bridge address.
|
||||
# castle.hub.internal is a *different* LAN device (Home Assistant) reached
|
||||
# via lanAccess instead.
|
||||
# ---------------------------------------------------------------------------
|
||||
REWRITE_FILES=(config.yaml auth.json context_length_cache.yaml .env)
|
||||
REWRITE_FROM=(host.docker.internal fwdesk.lab.internal castle.hub.internal)
|
||||
REWRITE_TO=(10.77.0.1 10.77.0.1 10.0.12.1)
|
||||
|
||||
echo "Source (read-only): $SOURCE"
|
||||
echo "Destination (guest): ${GUEST_USER}@hermes:${DEST_IN_GUEST} (over vsock, cid ${VSOCK_CID})"
|
||||
echo
|
||||
echo "This will:"
|
||||
echo " 1. tar '$SOURCE' (not modifying it) and stream it into the guest"
|
||||
echo " 2. rewrite these hostnames in the migrated config, in the guest only:"
|
||||
for i in "${!REWRITE_FROM[@]}"; do
|
||||
echo " ${REWRITE_FROM[$i]} -> ${REWRITE_TO[$i]}"
|
||||
done
|
||||
echo
|
||||
read -r -p "Type 'migrate' to proceed: " confirm
|
||||
[ "$confirm" = "migrate" ] || { echo "aborted"; exit 1; }
|
||||
|
||||
echo
|
||||
echo "== confirming guest is reachable =="
|
||||
"${SSH[@]}" true
|
||||
|
||||
echo "== streaming $SOURCE -> guest:${DEST_IN_GUEST} (this can take a while for 8G+) =="
|
||||
tar -C "$SOURCE" -cf - . | "${SSH[@]}" "mkdir -p '${DEST_IN_GUEST}' && tar -C '${DEST_IN_GUEST}' -xf -"
|
||||
|
||||
echo "== rewriting stale hostnames in the guest =="
|
||||
for f in "${REWRITE_FILES[@]}"; do
|
||||
path="${DEST_IN_GUEST}/${f}"
|
||||
sed_expr=""
|
||||
for i in "${!REWRITE_FROM[@]}"; do
|
||||
sed_expr+="s#${REWRITE_FROM[$i]}#${REWRITE_TO[$i]}#g; "
|
||||
done
|
||||
"${SSH[@]}" "[ -f '$path' ] && sed -i '$sed_expr' '$path' && echo ' rewrote $f' || echo ' $f not present, skipping'"
|
||||
done
|
||||
|
||||
echo
|
||||
echo "Done. The original at $SOURCE is untouched."
|
||||
echo
|
||||
echo "Next, inside the guest (hermes-vm ssh):"
|
||||
echo " export HERMES_HOME=${DEST_IN_GUEST}"
|
||||
echo " # then the usual install steps from README.md:"
|
||||
echo " # determinate-nixd login"
|
||||
echo " # sudo nix flake lock /etc/nixos"
|
||||
echo " # curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash"
|
||||
echo " # hermes setup"
|
||||
echo " # hermes gateway install"
|
||||
echo
|
||||
echo "Then verify with hermes-ctl status / logs before considering the old"
|
||||
echo "deployment retired. Decrypt secrets/hermes-env.age yourself and fold"
|
||||
echo "anything relevant into ${DEST_IN_GUEST}/.env (hermes-ctl secrets edit)."
|
||||
@@ -0,0 +1,994 @@
|
||||
{ config, lib, pkgs, ... }:
|
||||
|
||||
let
|
||||
inherit (lib) mkOption mkEnableOption mkIf types optionalString;
|
||||
cfg = config.services.hermesVm;
|
||||
net = cfg.network;
|
||||
|
||||
vmUser = "hermes-vm";
|
||||
runDir = "/run/hermes-vm";
|
||||
stateDir = "/var/lib/hermes-vm";
|
||||
# Two disks, deliberately.
|
||||
#
|
||||
# The system disk is large, churny and entirely rebuildable - it is the
|
||||
# NixOS install and the Nix store. We never snapshot it.
|
||||
#
|
||||
# The state disk holds the only things that cannot be reproduced: Hermes's
|
||||
# identity and memories, its workspace, and the NixOS config it has written
|
||||
# for itself. Small, so snapshots are cheap and backups are quick.
|
||||
systemImage = "${cfg.disk.baseDir}/system/system.raw";
|
||||
stateImage = "${cfg.disk.baseDir}/state/state.raw";
|
||||
snapDir = "${cfg.disk.baseDir}/snapshots";
|
||||
|
||||
pt = cfg.passthrough;
|
||||
anyPci = pt.pci != [ ];
|
||||
anyUsb = pt.usb != [ ];
|
||||
earlyIds = lib.filter (i: i != null) (map (d: d.id) pt.pci);
|
||||
|
||||
# Graceful ACPI power button via QMP, then wait for QEMU to exit.
|
||||
stopScript = pkgs.writeShellScript "hermes-vm-stop" ''
|
||||
set -eu
|
||||
printf '%s' '{"execute":"qmp_capabilities"}{"execute":"system_powerdown"}' \
|
||||
| ${pkgs.socat}/bin/socat - UNIX-CONNECT:${runDir}/qmp.sock >/dev/null 2>&1 || exit 0
|
||||
'';
|
||||
|
||||
# Freeze the guest filesystems around a snapshot so it captures a consistent
|
||||
# state rather than the equivalent of pulling the power.
|
||||
qga = verb: pkgs.writeShellScript "hermes-vm-fs${verb}" ''
|
||||
set -eu
|
||||
printf '%s' '{"execute":"guest-fs${verb}"}' \
|
||||
| ${pkgs.socat}/bin/socat -t2 - UNIX-CONNECT:${runDir}/qga.sock >/dev/null 2>&1 || true
|
||||
'';
|
||||
in
|
||||
{
|
||||
options.services.hermesVm = {
|
||||
enable = mkEnableOption "the Hermes Agent VM";
|
||||
|
||||
name = mkOption {
|
||||
type = types.str;
|
||||
default = "hermes";
|
||||
};
|
||||
|
||||
vcpu = mkOption {
|
||||
type = types.ints.positive;
|
||||
default = 4;
|
||||
};
|
||||
|
||||
mem = mkOption {
|
||||
type = types.ints.positive;
|
||||
default = 8192;
|
||||
description = "RAM in MiB.";
|
||||
};
|
||||
|
||||
disk = {
|
||||
baseDir = mkOption {
|
||||
type = types.str;
|
||||
default = "/var/lib/hermes-vm";
|
||||
description = ''
|
||||
Directory holding the guest's disk images. On btrfs this should be
|
||||
laid out as three subvolumes - system/, state/ and snapshots/ - with
|
||||
nodatacow set. setup.sh does this for you.
|
||||
'';
|
||||
};
|
||||
|
||||
systemSize = mkOption {
|
||||
type = types.str;
|
||||
default = "80G";
|
||||
description = ''
|
||||
System disk: the NixOS install and the Nix store. Never snapshotted,
|
||||
because it is entirely rebuildable from the flake. Size it for an
|
||||
agent that builds things.
|
||||
'';
|
||||
};
|
||||
|
||||
stateSize = mkOption {
|
||||
type = types.str;
|
||||
default = "20G";
|
||||
description = ''
|
||||
State disk, mounted at /var/lib/hermes in the guest. Holds Hermes's
|
||||
identity, memories, workspace, and the NixOS config it maintains for
|
||||
itself. This is the only irreplaceable data, so it is the only thing
|
||||
snapshotted and backed up.
|
||||
'';
|
||||
};
|
||||
|
||||
quiesce = mkOption {
|
||||
type = types.bool;
|
||||
default = true;
|
||||
description = "fsfreeze the guest via qemu-guest-agent around each snapshot.";
|
||||
};
|
||||
};
|
||||
|
||||
snapshots = {
|
||||
enable = mkOption {
|
||||
type = types.bool;
|
||||
default = true;
|
||||
description = ''
|
||||
Periodic btrfs snapshots of the state disk. The agent can persist
|
||||
changes, which means it can persist a mistake, and these are the
|
||||
fastest way back.
|
||||
'';
|
||||
};
|
||||
|
||||
interval = mkOption {
|
||||
type = types.str;
|
||||
default = "hourly";
|
||||
description = "systemd calendar expression.";
|
||||
};
|
||||
|
||||
keep = mkOption {
|
||||
type = types.ints.positive;
|
||||
default = 48;
|
||||
description = ''
|
||||
How many to retain. Keep this tighter than you would on ZFS: the
|
||||
state image is nodatacow, and every live snapshot reintroduces
|
||||
copy-on-write for the blocks it still references.
|
||||
'';
|
||||
};
|
||||
};
|
||||
|
||||
backup = {
|
||||
enable = mkEnableOption "restic backups of the guest's state";
|
||||
|
||||
repository = mkOption {
|
||||
type = types.str;
|
||||
default = "";
|
||||
example = "sftp:backup@nas:/hermes";
|
||||
description = "Any restic repository string.";
|
||||
};
|
||||
|
||||
passwordFile = mkOption {
|
||||
type = types.path;
|
||||
default = "/var/lib/secrets/hermes-restic";
|
||||
description = "File containing the restic repository password, mode 0400.";
|
||||
};
|
||||
|
||||
environmentFile = mkOption {
|
||||
type = types.nullOr types.path;
|
||||
default = null;
|
||||
description = "Optional file with credentials for the backend (S3 keys and so on).";
|
||||
};
|
||||
|
||||
interval = mkOption {
|
||||
type = types.str;
|
||||
default = "daily";
|
||||
};
|
||||
|
||||
keep = mkOption {
|
||||
type = types.listOf types.str;
|
||||
default = [ "--keep-daily" "14" "--keep-weekly" "8" "--keep-monthly" "12" ];
|
||||
description = "Arguments passed to restic forget.";
|
||||
};
|
||||
};
|
||||
|
||||
hostCache = {
|
||||
enable = mkOption {
|
||||
type = types.bool;
|
||||
default = true;
|
||||
description = ''
|
||||
Serve the host's /nix/store to the guest as a read-only binary
|
||||
cache, so the guest doesn't re-download from the internet what the
|
||||
host already has.
|
||||
|
||||
This is the ONLY route from guest to host in this design. Turning it
|
||||
off leaves the guest with literally no reachable host service, at
|
||||
the cost of bandwidth on every guest build.
|
||||
|
||||
Note the guest can read every store path on the host through this.
|
||||
Store paths are world-readable anyway, but check nothing sensitive
|
||||
is baked into your host's store.
|
||||
'';
|
||||
};
|
||||
port = mkOption {
|
||||
type = types.port;
|
||||
default = 5000;
|
||||
};
|
||||
secretKeyFile = mkOption {
|
||||
type = types.path;
|
||||
default = "/var/lib/secrets/hermes-cache.secret";
|
||||
description = "Generate with nix-store --generate-binary-cache-key. See README.";
|
||||
};
|
||||
};
|
||||
|
||||
authorizedKeys = mkOption {
|
||||
type = types.listOf types.str;
|
||||
default = [ ];
|
||||
};
|
||||
|
||||
allowLan = mkOption {
|
||||
type = types.bool;
|
||||
default = false;
|
||||
description = ''
|
||||
Blanket permission for the guest to reach RFC1918 space. Almost always
|
||||
the wrong tool - use lanAccess to open single endpoints instead.
|
||||
'';
|
||||
};
|
||||
|
||||
lanAccess = mkOption {
|
||||
default = [ ];
|
||||
example = [ { address = "192.168.1.50"; port = 11434; } ];
|
||||
description = ''
|
||||
Specific endpoints elsewhere on your network that the guest may reach,
|
||||
punched through the default RFC1918 drop. One entry per service.
|
||||
'';
|
||||
type = types.listOf (types.submodule {
|
||||
options = {
|
||||
address = mkOption { type = types.str; };
|
||||
port = mkOption { type = types.port; };
|
||||
proto = mkOption {
|
||||
type = types.enum [ "tcp" "udp" ];
|
||||
default = "tcp";
|
||||
};
|
||||
};
|
||||
});
|
||||
};
|
||||
|
||||
hostAccess = mkOption {
|
||||
default = [ ];
|
||||
example = [ { port = 8188; } ];
|
||||
description = ''
|
||||
Ports on this host itself that the guest may reach. The service must
|
||||
also be bound to an address the guest can see - see the README.
|
||||
'';
|
||||
type = types.listOf (types.submodule {
|
||||
options = {
|
||||
port = mkOption { type = types.port; };
|
||||
proto = mkOption {
|
||||
type = types.enum [ "tcp" "udp" ];
|
||||
default = "tcp";
|
||||
};
|
||||
};
|
||||
});
|
||||
};
|
||||
|
||||
publish = mkOption {
|
||||
default = [ ];
|
||||
example = [ { hostPort = 8080; guestPort = 8080; } ];
|
||||
description = ''
|
||||
Guest ports to publish on the host's uplink, reachable from the wider
|
||||
network. You do not need this to reach the guest from the host itself.
|
||||
'';
|
||||
type = types.listOf (types.submodule {
|
||||
options = {
|
||||
hostPort = mkOption { type = types.port; };
|
||||
guestPort = mkOption { type = types.port; };
|
||||
proto = mkOption {
|
||||
type = types.enum [ "tcp" "udp" ];
|
||||
default = "tcp";
|
||||
};
|
||||
};
|
||||
});
|
||||
};
|
||||
|
||||
network = {
|
||||
bridge = mkOption { type = types.str; default = "hermes-br"; };
|
||||
tapId = mkOption { type = types.str; default = "vm-hermes"; };
|
||||
mac = mkOption { type = types.str; default = "02:00:00:7e:00:01"; };
|
||||
hostAddress = mkOption { type = types.str; default = "10.77.0.1"; };
|
||||
guestAddress = mkOption { type = types.str; default = "10.77.0.2"; };
|
||||
prefixLength = mkOption { type = types.int; default = 24; };
|
||||
externalInterface = mkOption {
|
||||
type = types.nullOr types.str;
|
||||
default = null;
|
||||
example = "eno1";
|
||||
description = "Host uplink. Required.";
|
||||
};
|
||||
};
|
||||
|
||||
passthrough = {
|
||||
iommu = mkOption {
|
||||
type = types.nullOr (types.enum [ "intel" "amd" ]);
|
||||
default = null;
|
||||
description = ''
|
||||
Enable IOMMU kernel parameters for your CPU vendor. Required for any
|
||||
PCI passthrough. The IOMMU is the only thing stopping a passed-through
|
||||
device from reading host memory, so this is not optional.
|
||||
'';
|
||||
};
|
||||
|
||||
pci = mkOption {
|
||||
default = [ ];
|
||||
example = [
|
||||
{ address = "0000:01:00.0"; id = "10de:2204"; }
|
||||
{ address = "0000:01:00.1"; id = "10de:1aef"; }
|
||||
];
|
||||
description = ''
|
||||
PCI devices handed to the guest via VFIO. The host loses them
|
||||
entirely while the VM runs.
|
||||
|
||||
You must pass every function of the device and, in practice, every
|
||||
device in its IOMMU group. A GPU is usually at least two entries:
|
||||
the graphics function and its HDMI audio function.
|
||||
|
||||
Set `id` (vendor:device) to have vfio-pci claim the device in the
|
||||
initrd, before a host driver can. Strongly recommended for GPUs;
|
||||
unbinding an active amdgpu or nvidia driver later is unreliable.
|
||||
'';
|
||||
type = types.listOf (types.submodule {
|
||||
options = {
|
||||
address = mkOption {
|
||||
type = types.str;
|
||||
example = "0000:01:00.0";
|
||||
description = "Full PCI address, as shown by lspci -D.";
|
||||
};
|
||||
id = mkOption {
|
||||
type = types.nullOr types.str;
|
||||
default = null;
|
||||
example = "10de:2204";
|
||||
description = "vendor:device, from lspci -nn. Enables early binding.";
|
||||
};
|
||||
};
|
||||
});
|
||||
};
|
||||
|
||||
usb = mkOption {
|
||||
default = [ ];
|
||||
example = [ { vendorId = "8087"; productId = "0026"; } ];
|
||||
description = ''
|
||||
USB devices forwarded to the guest, matched by vendor and product
|
||||
ID from lsusb. Integrated Bluetooth radios are USB devices and go
|
||||
here too, though the host driver must be kept off them.
|
||||
|
||||
QEMU resolves these at start, so a device that is unplugged when the
|
||||
VM boots will not appear later without a restart.
|
||||
'';
|
||||
type = types.listOf (types.submodule {
|
||||
options = {
|
||||
vendorId = mkOption { type = types.str; example = "046d"; };
|
||||
productId = mkOption { type = types.str; example = "c52b"; };
|
||||
};
|
||||
});
|
||||
};
|
||||
};
|
||||
|
||||
cliName = mkOption {
|
||||
type = types.str;
|
||||
default = "hermes-ctl";
|
||||
description = ''
|
||||
Name of the host-side wrapper that drives Hermes inside the guest.
|
||||
Set to "hermes" if you want it to feel local and are sure nothing
|
||||
else on the host claims that name.
|
||||
'';
|
||||
};
|
||||
|
||||
gatewayUnit = mkOption {
|
||||
type = types.str;
|
||||
default = "hermes-agent";
|
||||
description = ''
|
||||
The systemd *user* unit that `hermes gateway install` created in the
|
||||
guest. Check with `hermes-ctl units` after installing; upstream has
|
||||
used more than one name for this.
|
||||
'';
|
||||
};
|
||||
|
||||
hermesHome = mkOption {
|
||||
type = types.str;
|
||||
default = "${stateDir}/.hermes";
|
||||
description = ''
|
||||
Where Hermes's own data (SOUL.md, config.yaml, memories/, sessions/,
|
||||
state.db, and so on) lives inside the guest. Defaults to the seeded
|
||||
layout under the state disk mount. Override this when carrying over
|
||||
an existing flat HERMES_HOME from elsewhere (e.g. a prior Docker
|
||||
deployment) that you're placing directly at the root of the state
|
||||
disk instead of under .hermes - see migrate-docker-hermes.sh.
|
||||
|
||||
This only changes where the host-side hermes-ctl wrapper (soul,
|
||||
memory, conf, secrets, reseed, pull) looks. It does not change what
|
||||
the gateway process itself uses - export HERMES_HOME to match inside
|
||||
the guest when you install it.
|
||||
'';
|
||||
};
|
||||
|
||||
vsockCid = mkOption {
|
||||
type = types.int;
|
||||
default = 77;
|
||||
description = "Host reaches the guest over vsock, so no inbound network path is needed.";
|
||||
};
|
||||
};
|
||||
|
||||
config = mkIf cfg.enable {
|
||||
assertions = [
|
||||
{
|
||||
assertion = net.externalInterface != null;
|
||||
message = "services.hermesVm.network.externalInterface must be set.";
|
||||
}
|
||||
{
|
||||
assertion = cfg.backup.enable -> (cfg.backup.repository != "");
|
||||
message = "services.hermesVm.backup.repository must be set when backups are enabled.";
|
||||
}
|
||||
{
|
||||
assertion = anyPci -> (pt.iommu != null);
|
||||
message = ''
|
||||
services.hermesVm.passthrough.pci needs passthrough.iommu set to
|
||||
"intel" or "amd". Without the IOMMU, a passed-through device can
|
||||
read host memory directly and the VM boundary is meaningless.
|
||||
'';
|
||||
}
|
||||
];
|
||||
|
||||
users.users.${vmUser} = {
|
||||
isSystemUser = true;
|
||||
group = vmUser;
|
||||
description = "Runs the Hermes guest VM";
|
||||
};
|
||||
users.groups.${vmUser} = { };
|
||||
|
||||
# =====================================================================
|
||||
# The VM itself
|
||||
#
|
||||
# No virtiofs, no shared filesystem of any kind, no vhost-net. The guest
|
||||
# gets a block device, a tap, a vsock and a serial port. That is all.
|
||||
# =====================================================================
|
||||
systemd.services.hermes-vm = {
|
||||
description = "Hermes Agent VM";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
after = [ "network-online.target" "hermes-vm-tap.service" ];
|
||||
requires = [ "hermes-vm-tap.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
|
||||
preStart = ''
|
||||
for img in ${systemImage} ${stateImage}; do
|
||||
if [ ! -e "$img" ]; then
|
||||
echo "$img is missing. Run ./setup.sh first." >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
# Per-VM writable UEFI variable store, seeded once.
|
||||
if [ ! -e ${stateDir}/OVMF_VARS.fd ]; then
|
||||
install -m 0600 ${pkgs.OVMF.variables} ${stateDir}/OVMF_VARS.fd
|
||||
fi
|
||||
'';
|
||||
|
||||
serviceConfig = {
|
||||
User = vmUser;
|
||||
Group = vmUser;
|
||||
SupplementaryGroups = [ "kvm" ];
|
||||
RuntimeDirectory = "hermes-vm";
|
||||
# ProtectSystem=strict makes everything read-only; punch through for
|
||||
# the images and the UEFI variable store only.
|
||||
ReadWritePaths = [ cfg.disk.baseDir ];
|
||||
StateDirectory = "hermes-vm";
|
||||
RuntimeDirectoryMode = "0750";
|
||||
|
||||
ExecStart = lib.concatStringsSep " " ([
|
||||
"${pkgs.qemu_kvm}/bin/qemu-system-x86_64"
|
||||
"-name ${cfg.name}"
|
||||
"-machine q35,accel=kvm,smm=on"
|
||||
"-cpu host"
|
||||
"-smp ${toString cfg.vcpu}"
|
||||
"-m ${toString cfg.mem}"
|
||||
"-nodefaults"
|
||||
"-no-user-config"
|
||||
"-display none"
|
||||
"-serial stdio"
|
||||
# QEMU's own seccomp sandbox, everything denied that we don't need.
|
||||
"-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny"
|
||||
# UEFI: firmware read-only, variables per-VM and writable.
|
||||
"-drive if=pflash,format=raw,unit=0,readonly=on,file=${pkgs.OVMF.firmware}"
|
||||
"-drive if=pflash,format=raw,unit=1,file=${stateDir}/OVMF_VARS.fd"
|
||||
# System disk: NixOS and the Nix store. Rebuildable, never snapshotted.
|
||||
# aio=threads and a writeback cache because these are files on a
|
||||
# filesystem, not block devices; aio=native wants the latter.
|
||||
"-drive if=none,id=system,format=raw,file=${systemImage},cache=writeback,aio=threads,discard=unmap"
|
||||
"-device virtio-blk-pci,drive=system"
|
||||
# State disk: everything irreplaceable. Mounted at /var/lib/hermes.
|
||||
"-drive if=none,id=state,format=raw,file=${stateImage},cache=writeback,aio=threads,discard=unmap"
|
||||
"-device virtio-blk-pci,drive=state"
|
||||
# Pre-created tap. vhost=off on purpose: vhost-net would move packet
|
||||
# handling into the host kernel, which is exactly the surface we are
|
||||
# trying not to expose. Costs throughput we do not need.
|
||||
"-netdev tap,id=net0,ifname=${net.tapId},script=no,downscript=no,vhost=off"
|
||||
"-device virtio-net-pci,netdev=net0,mac=${net.mac}"
|
||||
# Host -> guest management, no network involved.
|
||||
"-device vhost-vsock-pci,guest-cid=${toString cfg.vsockCid}"
|
||||
"-device virtio-rng-pci"
|
||||
"-qmp unix:${runDir}/qmp.sock,server=on,wait=off"
|
||||
] ++ lib.optionals cfg.disk.quiesce [
|
||||
"-chardev socket,path=${runDir}/qga.sock,server=on,wait=off,id=qga0"
|
||||
"-device virtio-serial-pci"
|
||||
"-device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0"
|
||||
]
|
||||
# Passed-through PCI devices. The host cannot use these while the VM
|
||||
# is running.
|
||||
++ map (d: "-device vfio-pci,host=${d.address}") pt.pci
|
||||
# A USB controller, only if something needs plugging into it.
|
||||
++ lib.optional anyUsb "-device qemu-xhci,id=xhci"
|
||||
++ map (u: "-device usb-host,vendorid=0x${u.vendorId},productid=0x${u.productId}") pt.usb
|
||||
);
|
||||
|
||||
ExecStop = stopScript;
|
||||
TimeoutStopSec = 90;
|
||||
Restart = "always";
|
||||
RestartSec = 5;
|
||||
|
||||
# ---- hardening -------------------------------------------------
|
||||
DevicePolicy = "closed";
|
||||
DeviceAllow = [
|
||||
"/dev/kvm rw"
|
||||
"/dev/net/tun rw"
|
||||
"/dev/vhost-vsock rw"
|
||||
]
|
||||
++ lib.optionals anyPci [ "char-vfio rw" ]
|
||||
++ lib.optionals anyUsb [ "char-usb_device rw" ];
|
||||
|
||||
# VFIO pins the entire guest RAM allocation, so the default memlock
|
||||
# limit has to go or QEMU fails to start with an obscure error.
|
||||
LimitMEMLOCK = lib.mkIf anyPci "infinity";
|
||||
NoNewPrivileges = true;
|
||||
ProtectSystem = "strict";
|
||||
ProtectHome = true;
|
||||
ProtectProc = "invisible";
|
||||
ProcSubset = "pid";
|
||||
PrivateTmp = true;
|
||||
ProtectKernelTunables = true;
|
||||
ProtectKernelModules = true;
|
||||
ProtectKernelLogs = true;
|
||||
ProtectClock = true;
|
||||
ProtectControlGroups = true;
|
||||
ProtectHostname = true;
|
||||
RestrictNamespaces = true;
|
||||
RestrictRealtime = true;
|
||||
RestrictSUIDSGID = true;
|
||||
RestrictAddressFamilies = [ "AF_UNIX" "AF_NETLINK" "AF_VSOCK" ];
|
||||
LockPersonality = true;
|
||||
SystemCallArchitectures = "native";
|
||||
SystemCallFilter = [ "@system-service" "~@resources" "~@obsolete" ];
|
||||
# KVM means no JIT, so W^X is safe. If QEMU refuses to start, this is
|
||||
# the first thing to flip.
|
||||
MemoryDenyWriteExecute = true;
|
||||
CapabilityBoundingSet = [ "" ];
|
||||
AmbientCapabilities = [ "" ];
|
||||
};
|
||||
};
|
||||
|
||||
# Tap device, created as root, owned by the VM user, bridged by networkd.
|
||||
systemd.services.hermes-vm-tap = {
|
||||
description = "TAP interface for the Hermes VM";
|
||||
after = [ "systemd-networkd.service" ];
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
RemainAfterExit = true;
|
||||
};
|
||||
path = [ pkgs.iproute2 ];
|
||||
script = ''
|
||||
ip link show ${net.tapId} >/dev/null 2>&1 \
|
||||
|| ip tuntap add ${net.tapId} mode tap user ${vmUser} multi_queue
|
||||
ip link set ${net.tapId} up
|
||||
'';
|
||||
preStop = ''
|
||||
${pkgs.iproute2}/bin/ip link del ${net.tapId} || true
|
||||
'';
|
||||
};
|
||||
|
||||
# =====================================================================
|
||||
# Device passthrough
|
||||
# =====================================================================
|
||||
|
||||
# The IOMMU is what keeps a passed-through device from reading host
|
||||
# memory. Without it, PCI passthrough is not a boundary at all.
|
||||
boot.kernelParams = lib.optionals (pt.iommu != null) [
|
||||
(if pt.iommu == "intel" then "intel_iommu=on" else "amd_iommu=on")
|
||||
"iommu=pt"
|
||||
];
|
||||
|
||||
boot.initrd.kernelModules = lib.optionals anyPci [
|
||||
"vfio_pci"
|
||||
"vfio"
|
||||
"vfio_iommu_type1"
|
||||
];
|
||||
|
||||
# Claim the device before any host driver gets to it.
|
||||
boot.extraModprobeConfig = lib.optionalString (earlyIds != [ ]) ''
|
||||
options vfio-pci ids=${lib.concatStringsSep "," earlyIds}
|
||||
'';
|
||||
|
||||
# Catch anything that was not early-bound, by address. Idempotent.
|
||||
systemd.services.hermes-vm-vfio = mkIf anyPci {
|
||||
description = "Bind passthrough PCI devices to vfio-pci";
|
||||
before = [ "hermes-vm.service" ];
|
||||
requiredBy = [ "hermes-vm.service" ];
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
RemainAfterExit = true;
|
||||
};
|
||||
path = [ pkgs.kmod ];
|
||||
script = ''
|
||||
modprobe vfio-pci
|
||||
'' + lib.concatMapStrings (d: ''
|
||||
dev=/sys/bus/pci/devices/${d.address}
|
||||
if [ ! -e "$dev" ]; then
|
||||
echo "${d.address} is not present on this machine" >&2
|
||||
exit 1
|
||||
fi
|
||||
cur=none
|
||||
[ -e "$dev/driver" ] && cur=$(basename "$(readlink -f "$dev/driver")")
|
||||
if [ "$cur" != "vfio-pci" ]; then
|
||||
[ "$cur" != "none" ] && echo ${d.address} > "$dev/driver/unbind"
|
||||
echo vfio-pci > "$dev/driver_override"
|
||||
echo ${d.address} > /sys/bus/pci/drivers_probe
|
||||
fi
|
||||
'') pt.pci;
|
||||
};
|
||||
|
||||
services.udev.extraRules =
|
||||
lib.optionalString anyPci ''
|
||||
SUBSYSTEM=="vfio", OWNER="${vmUser}", GROUP="${vmUser}", MODE="0660"
|
||||
''
|
||||
+ lib.concatMapStrings (u: ''
|
||||
SUBSYSTEM=="usb", ATTR{idVendor}=="${u.vendorId}", ATTR{idProduct}=="${u.productId}", OWNER="${vmUser}", MODE="0660"
|
||||
'') pt.usb;
|
||||
|
||||
# =====================================================================
|
||||
# Networking
|
||||
# =====================================================================
|
||||
networking.useNetworkd = true;
|
||||
|
||||
systemd.network.netdevs."10-${net.bridge}".netdevConfig = {
|
||||
Kind = "bridge";
|
||||
Name = net.bridge;
|
||||
};
|
||||
|
||||
systemd.network.networks."10-${net.bridge}" = {
|
||||
matchConfig.Name = net.bridge;
|
||||
address = [ "${net.hostAddress}/${toString net.prefixLength}" ];
|
||||
networkConfig = {
|
||||
ConfigureWithoutCarrier = true;
|
||||
IPv4Forwarding = true;
|
||||
};
|
||||
};
|
||||
|
||||
systemd.network.networks."11-${net.tapId}" = {
|
||||
matchConfig.Name = net.tapId;
|
||||
networkConfig.Bridge = net.bridge;
|
||||
};
|
||||
|
||||
networking.nat = {
|
||||
enable = true;
|
||||
internalInterfaces = [ net.bridge ];
|
||||
externalInterface = net.externalInterface;
|
||||
forwardPorts = map (s: {
|
||||
sourcePort = s.hostPort;
|
||||
destination = "${net.guestAddress}:${toString s.guestPort}";
|
||||
inherit (s) proto;
|
||||
}) cfg.publish;
|
||||
};
|
||||
|
||||
# =====================================================================
|
||||
# Firewall
|
||||
#
|
||||
# The guest owns its own kernel and its own config now, so nothing inside
|
||||
# it constrains it. These rules are the boundary.
|
||||
# =====================================================================
|
||||
networking.nftables.enable = true;
|
||||
networking.firewall.enable = true;
|
||||
networking.firewall.filterForward = true;
|
||||
|
||||
# The deliberate guest -> host holes: the Nix cache, plus anything listed
|
||||
# in hostAccess.
|
||||
networking.firewall.extraInputRules =
|
||||
optionalString cfg.hostCache.enable ''
|
||||
iifname "${net.bridge}" ip saddr ${net.guestAddress} tcp dport ${toString cfg.hostCache.port} accept
|
||||
''
|
||||
+ lib.concatMapStrings (s: ''
|
||||
iifname "${net.bridge}" ip saddr ${net.guestAddress} ${s.proto} dport ${toString s.port} accept
|
||||
'') cfg.hostAccess;
|
||||
|
||||
# Order matters here. Allowlists must precede the RFC1918 drop.
|
||||
networking.firewall.extraForwardRules =
|
||||
''
|
||||
ct state established,related accept
|
||||
''
|
||||
# Guest reaching named services elsewhere on your network.
|
||||
+ lib.concatMapStrings (s: ''
|
||||
iifname "${net.bridge}" ip daddr ${s.address} ${s.proto} dport ${toString s.port} accept
|
||||
'') cfg.lanAccess
|
||||
# Inbound to guest services published on the uplink.
|
||||
+ lib.concatMapStrings (s: ''
|
||||
oifname "${net.bridge}" ip daddr ${net.guestAddress} ${s.proto} dport ${toString s.guestPort} accept
|
||||
'') cfg.publish
|
||||
+ (if cfg.allowLan then ''
|
||||
iifname "${net.bridge}" accept
|
||||
'' else ''
|
||||
iifname "${net.bridge}" ip daddr {
|
||||
10.0.0.0/8,
|
||||
172.16.0.0/12,
|
||||
192.168.0.0/16,
|
||||
169.254.0.0/16,
|
||||
127.0.0.0/8,
|
||||
100.64.0.0/10
|
||||
} drop
|
||||
iifname "${net.bridge}" ip6 daddr { fc00::/7, fe80::/10, ::1/128 } drop
|
||||
iifname "${net.bridge}" accept
|
||||
'');
|
||||
|
||||
# =====================================================================
|
||||
# Host store as a read-only cache for the guest
|
||||
# =====================================================================
|
||||
services.nix-serve = mkIf cfg.hostCache.enable {
|
||||
enable = true;
|
||||
package = pkgs.nix-serve-ng;
|
||||
bindAddress = net.hostAddress;
|
||||
inherit (cfg.hostCache) port secretKeyFile;
|
||||
openFirewall = false; # handled above, scoped to the guest address
|
||||
};
|
||||
|
||||
# =====================================================================
|
||||
# Snapshots of the state disk only.
|
||||
#
|
||||
# The system disk is not snapshotted on purpose. It is large, it is where
|
||||
# all the write churn happens, and setup.sh can rebuild it from the flake
|
||||
# in minutes. Snapshotting it would cost a lot and buy nothing.
|
||||
# =====================================================================
|
||||
systemd.services.hermes-vm-snapshot = mkIf cfg.snapshots.enable {
|
||||
description = "Snapshot the Hermes VM state disk";
|
||||
path = with pkgs; [ btrfs-progs coreutils socat ];
|
||||
serviceConfig.Type = "oneshot";
|
||||
script = ''
|
||||
set -euo pipefail
|
||||
stamp=$(date +%Y%m%d-%H%M%S)
|
||||
mkdir -p ${snapDir}
|
||||
|
||||
${optionalString cfg.disk.quiesce ''
|
||||
# Flush the guest's filesystems so the snapshot is consistent rather
|
||||
# than the equivalent of pulling the power. Takes milliseconds; the
|
||||
# VM keeps running throughout.
|
||||
${qga "freeze"}
|
||||
trap '${qga "thaw"}' EXIT
|
||||
''}
|
||||
|
||||
btrfs subvolume snapshot -r ${cfg.disk.baseDir}/state ${snapDir}/state-$stamp
|
||||
|
||||
${optionalString cfg.disk.quiesce ''
|
||||
${qga "thaw"}
|
||||
trap - EXIT
|
||||
''}
|
||||
|
||||
# Prune oldest first. Keep this list short: every live snapshot
|
||||
# reintroduces copy-on-write on a nodatacow image.
|
||||
ls -1d ${snapDir}/state-* 2>/dev/null | sort | head -n -${toString cfg.snapshots.keep} \
|
||||
| while read -r old; do btrfs subvolume delete "$old"; done
|
||||
'';
|
||||
};
|
||||
|
||||
systemd.timers.hermes-vm-snapshot = mkIf cfg.snapshots.enable {
|
||||
wantedBy = [ "timers.target" ];
|
||||
timerConfig = {
|
||||
OnCalendar = cfg.snapshots.interval;
|
||||
Persistent = true;
|
||||
RandomizedDelaySec = "5m";
|
||||
};
|
||||
};
|
||||
|
||||
# =====================================================================
|
||||
# Backups: the guest's state, not its disk.
|
||||
#
|
||||
# Pulled file-by-file over the SSH channel that already exists, rather
|
||||
# than backing up the image. That keeps the backup portable - it restores
|
||||
# onto any filesystem, or a different machine entirely - and avoids
|
||||
# loop-mounting a guest-written filesystem on the host, which would mean
|
||||
# the host kernel parsing data the agent controls.
|
||||
# =====================================================================
|
||||
systemd.services.hermes-vm-backup = mkIf cfg.backup.enable {
|
||||
description = "Back up Hermes state";
|
||||
path = with pkgs; [ restic openssh socat gnutar ];
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
EnvironmentFile = lib.mkIf (cfg.backup.environmentFile != null) cfg.backup.environmentFile;
|
||||
};
|
||||
script = ''
|
||||
set -euo pipefail
|
||||
export RESTIC_REPOSITORY=${lib.escapeShellArg cfg.backup.repository}
|
||||
export RESTIC_PASSWORD_FILE=${cfg.backup.passwordFile}
|
||||
|
||||
restic snapshots >/dev/null 2>&1 || restic init
|
||||
|
||||
ssh -o ProxyCommand="socat - VSOCK-CONNECT:${toString cfg.vsockCid}:22" \
|
||||
-o StrictHostKeyChecking=accept-new \
|
||||
hermes@${cfg.name} \
|
||||
'tar -C /var/lib/hermes \
|
||||
--exclude=./.cache \
|
||||
--exclude=./workspace/**/.direnv \
|
||||
--exclude=./workspace/**/result \
|
||||
-cf - .' \
|
||||
| restic backup --stdin --stdin-filename hermes-state.tar \
|
||||
--tag hermes --host ${cfg.name}
|
||||
|
||||
restic forget --tag hermes --host ${cfg.name} --prune ${lib.concatStringsSep " " cfg.backup.keep}
|
||||
'';
|
||||
};
|
||||
|
||||
systemd.timers.hermes-vm-backup = mkIf cfg.backup.enable {
|
||||
wantedBy = [ "timers.target" ];
|
||||
timerConfig = {
|
||||
OnCalendar = cfg.backup.interval;
|
||||
Persistent = true;
|
||||
RandomizedDelaySec = "30m";
|
||||
};
|
||||
};
|
||||
|
||||
# =====================================================================
|
||||
# Host GC is now safe: the guest has its own store and shares nothing.
|
||||
# =====================================================================
|
||||
nix.gc.automatic = lib.mkDefault true;
|
||||
nix.gc.options = lib.mkDefault "--delete-older-than 30d";
|
||||
|
||||
environment.systemPackages = [
|
||||
(pkgs.writeShellScriptBin "hermes-vm" ''
|
||||
set -euo pipefail
|
||||
SSH=(${pkgs.openssh}/bin/ssh
|
||||
-o ProxyCommand="${pkgs.socat}/bin/socat - VSOCK-CONNECT:${toString cfg.vsockCid}:22"
|
||||
-o StrictHostKeyChecking=accept-new
|
||||
hermes@${cfg.name})
|
||||
cmd="''${1-}"; shift || true
|
||||
case "$cmd" in
|
||||
ssh) exec "''${SSH[@]}" "$@" ;;
|
||||
console) exec journalctl -u hermes-vm -f -o cat ;;
|
||||
start) exec systemctl start hermes-vm ;;
|
||||
stop) exec systemctl stop hermes-vm ;;
|
||||
restart) exec systemctl restart hermes-vm ;;
|
||||
|
||||
# What is the guest actually running right now?
|
||||
config)
|
||||
"''${SSH[@]}" 'echo "== current generation =="; \
|
||||
readlink -f /run/current-system; \
|
||||
echo; echo "== generations =="; \
|
||||
sudo nix-env --list-generations -p /nix/var/nix/profiles/system | tail -n 10; \
|
||||
echo; echo "== /etc/nixos history =="; \
|
||||
git -C /etc/nixos log --oneline -n 15; \
|
||||
echo; echo "== uncommitted =="; \
|
||||
git -C /etc/nixos status --short'
|
||||
;;
|
||||
# Copy the guest's live config here for reading or diffing.
|
||||
config-pull)
|
||||
dest="''${1:-./hermes-etc-nixos}"
|
||||
mkdir -p "$dest"
|
||||
"''${SSH[@]}" 'tar -C /etc/nixos -cf - .' | tar -C "$dest" -xf -
|
||||
echo "pulled to $dest"
|
||||
;;
|
||||
|
||||
snapshot) exec systemctl start hermes-vm-snapshot ;;
|
||||
snapshots)
|
||||
${pkgs.util-linux}/bin/column -t <<<"$(
|
||||
echo "SNAPSHOT SIZE CREATED"
|
||||
for d in ${snapDir}/state-*; do
|
||||
[ -e "$d" ] || continue
|
||||
echo "$(basename "$d") $(du -sh "$d" 2>/dev/null | cut -f1) $(stat -c %y "$d" | cut -d. -f1)"
|
||||
done
|
||||
)" ;;
|
||||
|
||||
# Restore the state disk from a snapshot. The system disk is left
|
||||
# alone - it holds nothing worth restoring.
|
||||
rollback)
|
||||
[ $# -ge 1 ] || { echo "usage: hermes-vm rollback <snapshot>" >&2; exit 1; }
|
||||
snap=${snapDir}/"$1"
|
||||
[ -d "$snap" ] || { echo "no such snapshot: $1" >&2; exit 1; }
|
||||
echo "Restores Hermes's identity, memories, workspace and NixOS" >&2
|
||||
echo "config to $1. Everything since is lost." >&2
|
||||
read -r -p "type 'rollback' to confirm: " c
|
||||
[ "$c" = rollback ] || { echo "aborted" >&2; exit 1; }
|
||||
systemctl stop hermes-vm
|
||||
mv ${cfg.disk.baseDir}/state ${cfg.disk.baseDir}/state.old-$(date +%s)
|
||||
${pkgs.btrfs-progs}/bin/btrfs subvolume snapshot "$snap" ${cfg.disk.baseDir}/state
|
||||
systemctl start hermes-vm
|
||||
echo "restored; previous state kept as state.old-*" ;;
|
||||
|
||||
backup) exec systemctl start hermes-vm-backup ;;
|
||||
backups)
|
||||
exec ${pkgs.restic}/bin/restic \
|
||||
-r "$(systemctl show hermes-vm-backup -p Environment --value | tr ' ' '\n' | sed -n 's/^RESTIC_REPOSITORY=//p')" \
|
||||
snapshots --tag hermes ;;
|
||||
*)
|
||||
echo "usage: hermes-vm {ssh [cmd]|console|config|config-pull [dir]|start|stop|restart|snapshot|snapshots|rollback <snap>|backup|backups}" >&2
|
||||
exit 1 ;;
|
||||
esac
|
||||
'')
|
||||
|
||||
# ---------------------------------------------------------------
|
||||
# Driving Hermes itself, as opposed to the VM around it.
|
||||
# ---------------------------------------------------------------
|
||||
(pkgs.writeShellScriptBin cfg.cliName ''
|
||||
set -euo pipefail
|
||||
SSH=(${pkgs.openssh}/bin/ssh
|
||||
-o ProxyCommand="${pkgs.socat}/bin/socat - VSOCK-CONNECT:${toString cfg.vsockCid}:22"
|
||||
-o StrictHostKeyChecking=accept-new
|
||||
hermes@${cfg.name})
|
||||
HOME_DIR=${cfg.hermesHome}
|
||||
UNIT=${cfg.gatewayUnit}
|
||||
|
||||
# Pull a file out of the guest, open it in your editor, push it back.
|
||||
edit_remote() {
|
||||
remote="$1"; mode="''${2:-0644}"
|
||||
tmp=$(mktemp); trap 'rm -f "$tmp"' EXIT
|
||||
"''${SSH[@]}" "cat $remote" > "$tmp" 2>/dev/null || true
|
||||
before=$(${pkgs.coreutils}/bin/sha256sum < "$tmp")
|
||||
"''${EDITOR:-${pkgs.vim}/bin/vim}" "$tmp"
|
||||
if [ "$(${pkgs.coreutils}/bin/sha256sum < "$tmp")" = "$before" ]; then
|
||||
echo "unchanged"; return
|
||||
fi
|
||||
"''${SSH[@]}" "install -m $mode /dev/stdin $remote" < "$tmp"
|
||||
echo "written to $remote"
|
||||
read -r -p "restart the gateway now? [y/N] " yn
|
||||
case "$yn" in [yY]*) "''${SSH[@]}" "systemctl --user restart $UNIT" ;; esac
|
||||
}
|
||||
|
||||
cmd="''${1-}"; shift || true
|
||||
case "$cmd" in
|
||||
soul)
|
||||
case "''${1-}" in
|
||||
edit) edit_remote "$HOME_DIR/SOUL.md" ;;
|
||||
*) "''${SSH[@]}" "cat $HOME_DIR/SOUL.md" ;;
|
||||
esac ;;
|
||||
|
||||
memory)
|
||||
case "''${1-}" in
|
||||
edit) edit_remote "$HOME_DIR/memories/USER.md" ;;
|
||||
*) "''${SSH[@]}" "cat $HOME_DIR/memories/USER.md" ;;
|
||||
esac ;;
|
||||
|
||||
conf)
|
||||
case "''${1-}" in
|
||||
edit) edit_remote "$HOME_DIR/config.yaml" ;;
|
||||
*) "''${SSH[@]}" "cat $HOME_DIR/config.yaml" ;;
|
||||
esac ;;
|
||||
|
||||
# API keys. Mode 0600, never enters the Nix store, and not printed
|
||||
# unless you ask for it by name.
|
||||
secrets)
|
||||
case "''${1-}" in
|
||||
edit) edit_remote "$HOME_DIR/.env" 0600 ;;
|
||||
show) "''${SSH[@]}" "cat $HOME_DIR/.env" ;;
|
||||
*) "''${SSH[@]}" "sed 's/=.*/=<set>/' $HOME_DIR/.env" ;;
|
||||
esac ;;
|
||||
|
||||
status) "''${SSH[@]}" "systemctl --user status $UNIT --no-pager" ;;
|
||||
logs) "''${SSH[@]}" -t "journalctl --user -u $UNIT -f" ;;
|
||||
restart) "''${SSH[@]}" "systemctl --user restart $UNIT" ;;
|
||||
units) "''${SSH[@]}" "systemctl --user list-units 'hermes*' --all --no-pager" ;;
|
||||
|
||||
# Overwrite the agent's identity from the seed. Destructive: this
|
||||
# is how you undo whatever it has decided about itself.
|
||||
reseed)
|
||||
echo "This overwrites SOUL.md, memories and config.yaml in the guest" >&2
|
||||
echo "with the seed from /etc/nixos/hermes. Anything Hermes has" >&2
|
||||
echo "learned and written there is lost." >&2
|
||||
read -r -p "type 'reseed' to confirm: " confirm
|
||||
[ "$confirm" = "reseed" ] || { echo "aborted" >&2; exit 1; }
|
||||
"''${SSH[@]}" "cp -rT /etc/nixos/hermes/home $HOME_DIR && \
|
||||
cp -rT /etc/nixos/hermes/workspace /var/lib/hermes/workspace && \
|
||||
systemctl --user restart $UNIT"
|
||||
echo "reseeded" ;;
|
||||
|
||||
# Copy the agent's live identity back here, so you can fold what it
|
||||
# has learned into the seed in guest/hermes/.
|
||||
pull)
|
||||
dest="''${1:-./hermes-home}"
|
||||
mkdir -p "$dest"
|
||||
"''${SSH[@]}" "tar -C $HOME_DIR --exclude=.env --exclude=sessions --exclude=state.db -cf - ." \
|
||||
| tar -C "$dest" -xf -
|
||||
echo "pulled to $dest (secrets and sessions excluded)" ;;
|
||||
|
||||
""|help|-h|--help)
|
||||
cat >&2 <<'USAGE'
|
||||
${cfg.cliName} - drive Hermes inside the guest
|
||||
|
||||
soul [edit] show or edit SOUL.md
|
||||
memory [edit] show or edit memories/USER.md
|
||||
conf [edit] show or edit config.yaml
|
||||
secrets [edit|show] manage .env, keys masked by default
|
||||
status | logs | restart | units
|
||||
reseed overwrite identity from /etc/nixos/hermes
|
||||
pull [dir] copy live identity back to the host
|
||||
|
||||
Anything else goes straight to the hermes CLI in the guest:
|
||||
${cfg.cliName} chat
|
||||
${cfg.cliName} update
|
||||
${cfg.cliName} sessions list
|
||||
USAGE
|
||||
exit 1 ;;
|
||||
|
||||
*) exec "''${SSH[@]}" -t hermes "$cmd" "$@" ;;
|
||||
esac
|
||||
'')
|
||||
];
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,380 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# hermes-vm guided setup.
|
||||
#
|
||||
# Walks the steps that have to happen before `nixos-rebuild switch`:
|
||||
# the disk images, the cache signing key, filling in guest/common.nix, building the
|
||||
# bootstrap image and writing it to disk.
|
||||
#
|
||||
# Safe to re-run. Every step checks whether it is already done and offers to
|
||||
# skip. Nothing is destructive without an explicit confirmation.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# ---------------------------------------------------------------- output ---
|
||||
|
||||
if [ -t 1 ] && [ -z "${NO_COLOR:-}" ]; then
|
||||
B=$'\033[1m'; DIM=$'\033[2m'; R=$'\033[31m'; Y=$'\033[33m'; G=$'\033[32m'; N=$'\033[0m'
|
||||
else
|
||||
B=""; DIM=""; R=""; Y=""; G=""; N=""
|
||||
fi
|
||||
|
||||
step_n=0
|
||||
step() { step_n=$((step_n + 1)); printf '\n%s== %d. %s%s\n' "$B" "$step_n" "$*" "$N"; }
|
||||
say() { printf ' %s\n' "$*"; }
|
||||
note() { printf ' %s%s%s\n' "$DIM" "$*" "$N"; }
|
||||
ok() { printf ' %s✓%s %s\n' "$G" "$N" "$*"; }
|
||||
warn() { printf ' %s!%s %s\n' "$Y" "$N" "$*"; }
|
||||
die() { printf '\n%serror:%s %s\n' "$R" "$N" "$*" >&2; exit 1; }
|
||||
|
||||
ask() { # ask VAR "prompt" "default"
|
||||
local __var="$1" __prompt="$2" __default="${3:-}" __reply
|
||||
if [ -n "$__default" ]; then
|
||||
read -r -p " $__prompt [$__default]: " __reply || true
|
||||
__reply="${__reply:-$__default}"
|
||||
else
|
||||
read -r -p " $__prompt: " __reply || true
|
||||
fi
|
||||
printf -v "$__var" '%s' "$__reply"
|
||||
}
|
||||
|
||||
confirm() { # confirm "prompt" -> 0 if yes
|
||||
local reply
|
||||
read -r -p " $1 [y/N] " reply || true
|
||||
case "$reply" in [yY]*) return 0 ;; *) return 1 ;; esac
|
||||
}
|
||||
|
||||
confirm_word() { # confirm_word "word" "prompt" -> 0 if they typed the word
|
||||
local reply
|
||||
read -r -p " $2 " reply || true
|
||||
[ "$reply" = "$1" ]
|
||||
}
|
||||
|
||||
# ------------------------------------------------------------------ setup ---
|
||||
|
||||
cd "$(dirname "$(readlink -f "$0")")"
|
||||
|
||||
if [ "$(id -u)" -eq 0 ]; then
|
||||
SUDO=""
|
||||
else
|
||||
command -v sudo >/dev/null || die "not root and sudo is not available"
|
||||
SUDO="sudo"
|
||||
fi
|
||||
|
||||
COMMON="guest/common.nix"
|
||||
CACHE_NAME="hermes-cache-1"
|
||||
SECRET_KEY="/var/lib/secrets/hermes-cache.secret"
|
||||
PUBLIC_KEY="/var/lib/secrets/hermes-cache.pub"
|
||||
DETERMINATE_SUB="https://install.determinate.systems"
|
||||
DETERMINATE_KEY="cache.flakehub.com-3:hJuILl5sVK4iKm86JzgdXW12Y2Hwd5G07qKtHTOcDCM="
|
||||
|
||||
printf '%s\n' "$B"
|
||||
cat <<'BANNER'
|
||||
hermes-vm · guided setup
|
||||
BANNER
|
||||
printf '%s' "$N"
|
||||
note "This prepares the host. It does not run nixos-rebuild — you'll paste a"
|
||||
note "module snippet into your own configuration at the end."
|
||||
|
||||
# ------------------------------------------------------------- preflight ---
|
||||
|
||||
step "Checking prerequisites"
|
||||
|
||||
[ -f flake.nix ] && [ -f "$COMMON" ] \
|
||||
|| die "run this from the hermes-vm repository root"
|
||||
|
||||
command -v nix >/dev/null || die "nix not found on PATH"
|
||||
nix flake --help >/dev/null 2>&1 \
|
||||
|| die "flakes are not enabled. Add 'experimental-features = nix-command flakes' to /etc/nix/nix.conf"
|
||||
ok "nix with flakes"
|
||||
|
||||
command -v btrfs >/dev/null || die "btrfs-progs not found. Snapshots need it."
|
||||
command -v mkfs.ext4 >/dev/null || die "mkfs.ext4 not found (e2fsprogs)"
|
||||
ok "btrfs-progs"
|
||||
|
||||
[ -e /dev/kvm ] || die "/dev/kvm is missing. KVM is required; check virtualisation is enabled in firmware."
|
||||
ok "kvm"
|
||||
|
||||
if command -v nixos-version >/dev/null; then
|
||||
ok "host is NixOS ($(nixos-version | cut -d' ' -f1))"
|
||||
else
|
||||
warn "this host does not look like NixOS — the module expects NixOS"
|
||||
fi
|
||||
|
||||
# ----------------------------------------------------------------- disks ---
|
||||
|
||||
step "Guest disks"
|
||||
|
||||
note "Two disks, deliberately:"
|
||||
note " system - NixOS and the Nix store. Big, churny, rebuildable. Not backed up."
|
||||
note " state - identity, memories, workspace, /etc/nixos. Small. Snapshotted."
|
||||
|
||||
ask BASE "Directory for the disk images" "/var/lib/hermes-vm"
|
||||
|
||||
FSTYPE="$(stat -f -c %T "$(dirname "$BASE")" 2>/dev/null || echo unknown)"
|
||||
if [ "$FSTYPE" != "btrfs" ]; then
|
||||
warn "$(dirname "$BASE") is $FSTYPE, not btrfs"
|
||||
warn "snapshots need btrfs. Everything else works; you'd rely on backups alone."
|
||||
confirm "Continue anyway?" || die "aborted"
|
||||
USE_BTRFS=no
|
||||
else
|
||||
USE_BTRFS=yes
|
||||
fi
|
||||
|
||||
$SUDO mkdir -p "$BASE"
|
||||
|
||||
make_subvol() { # make_subvol path
|
||||
if [ "$USE_BTRFS" = yes ] && ! $SUDO btrfs subvolume show "$1" >/dev/null 2>&1; then
|
||||
$SUDO rmdir "$1" 2>/dev/null || true
|
||||
$SUDO btrfs subvolume create "$1" >/dev/null
|
||||
# nodatacow BEFORE any file exists in here. Btrfs is copy-on-write and so
|
||||
# is a VM writing to a disk image; stacked, the image fragments badly.
|
||||
# Only applies to files created afterwards, hence the ordering.
|
||||
$SUDO chattr +C "$1"
|
||||
$SUDO btrfs property set "$1" compression none 2>/dev/null || true
|
||||
ok "subvolume $1 (nodatacow)"
|
||||
else
|
||||
$SUDO mkdir -p "$1"
|
||||
fi
|
||||
}
|
||||
|
||||
make_subvol "$BASE/system"
|
||||
make_subvol "$BASE/state"
|
||||
$SUDO mkdir -p "$BASE/snapshots"
|
||||
|
||||
SYSTEM_IMG="$BASE/system/system.raw"
|
||||
STATE_IMG="$BASE/state/state.raw"
|
||||
|
||||
# --- system disk ---
|
||||
if $SUDO test -f "$SYSTEM_IMG"; then
|
||||
ok "system disk exists ($($SUDO du -h --apparent-size "$SYSTEM_IMG" | cut -f1))"
|
||||
SYSTEM_EXISTED=yes
|
||||
else
|
||||
SYSTEM_EXISTED=no
|
||||
ask SYSSIZE "System disk size" "80G"
|
||||
$SUDO truncate -s "$SYSSIZE" "$SYSTEM_IMG"
|
||||
if [ "$USE_BTRFS" = yes ]; then
|
||||
$SUDO lsattr -l "$SYSTEM_IMG" 2>/dev/null | grep -q "No_COW" \
|
||||
&& ok "system disk created, nodatacow confirmed" \
|
||||
|| warn "system disk created but nodatacow did NOT apply - check 'lsattr $SYSTEM_IMG'"
|
||||
else
|
||||
ok "system disk created"
|
||||
fi
|
||||
fi
|
||||
|
||||
# --- state disk ---
|
||||
if $SUDO test -f "$STATE_IMG"; then
|
||||
ok "state disk exists ($($SUDO du -h --apparent-size "$STATE_IMG" | cut -f1))"
|
||||
else
|
||||
ask STATESIZE "State disk size" "20G"
|
||||
$SUDO truncate -s "$STATESIZE" "$STATE_IMG"
|
||||
$SUDO mkfs.ext4 -q -L hermes-state "$STATE_IMG"
|
||||
ok "state disk created and formatted (label: hermes-state)"
|
||||
fi
|
||||
|
||||
# ------------------------------------------------------------- cache key ---
|
||||
|
||||
step "Binary cache signing key"
|
||||
|
||||
note "The host serves its store to the guest read-only, so the guest doesn't"
|
||||
note "re-download from the internet what you already have."
|
||||
|
||||
if $SUDO test -f "$SECRET_KEY"; then
|
||||
ok "signing key already exists"
|
||||
else
|
||||
$SUDO mkdir -p /var/lib/secrets
|
||||
$SUDO nix-store --generate-binary-cache-key "$CACHE_NAME" "$SECRET_KEY" "$PUBLIC_KEY"
|
||||
$SUDO chmod 0400 "$SECRET_KEY"
|
||||
ok "generated $CACHE_NAME"
|
||||
fi
|
||||
|
||||
CACHE_PUB="$($SUDO cat "$PUBLIC_KEY")"
|
||||
say "Public key: ${DIM}${CACHE_PUB}${N}"
|
||||
|
||||
# nix-serve's user only exists once the module has been applied, so this is
|
||||
# expected to fail on a first run.
|
||||
if id nix-serve >/dev/null 2>&1; then
|
||||
$SUDO chown nix-serve "$SECRET_KEY"
|
||||
ok "key owned by nix-serve"
|
||||
else
|
||||
warn "the nix-serve user does not exist yet — that's normal on a first run"
|
||||
note "after your nixos-rebuild, run:"
|
||||
note " sudo chown nix-serve $SECRET_KEY"
|
||||
fi
|
||||
|
||||
# ------------------------------------------------------------- ssh keys ----
|
||||
|
||||
step "SSH access to the guest"
|
||||
|
||||
CURRENT_KEYS="$(sed -n '/^ authorizedKeys = \[/,/^ \];/p' "$COMMON" | grep -c '^\s*"ssh-' || true)"
|
||||
if [ "${CURRENT_KEYS:-0}" -gt 0 ]; then
|
||||
ok "$CURRENT_KEYS key(s) already configured in $COMMON"
|
||||
if confirm "Replace them?"; then CURRENT_KEYS=0; fi
|
||||
fi
|
||||
|
||||
SELECTED_KEYS=""
|
||||
if [ "${CURRENT_KEYS:-0}" -eq 0 ]; then
|
||||
mapfile -t FOUND < <(ls -1 "$HOME"/.ssh/*.pub 2>/dev/null || true)
|
||||
if [ "${#FOUND[@]}" -gt 0 ]; then
|
||||
say "Public keys found in ~/.ssh:"
|
||||
i=0
|
||||
for f in "${FOUND[@]}"; do
|
||||
i=$((i + 1))
|
||||
note " $i) $(basename "$f") — $(cut -d' ' -f3- "$f")"
|
||||
done
|
||||
ask PICK "Numbers to use (space separated, or 'n' to paste one)" "1"
|
||||
if [ "$PICK" != "n" ]; then
|
||||
for n in $PICK; do
|
||||
f="${FOUND[$((n - 1))]:-}"
|
||||
[ -n "$f" ] || die "no such option: $n"
|
||||
SELECTED_KEYS+=" \"$(tr -d '\n' < "$f")\"
|
||||
"
|
||||
done
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ -z "$SELECTED_KEYS" ]; then
|
||||
ask PASTED "Paste a public key (blank to skip)"
|
||||
[ -n "$PASTED" ] && SELECTED_KEYS=" \"$PASTED\"
|
||||
"
|
||||
fi
|
||||
|
||||
if [ -z "$SELECTED_KEYS" ]; then
|
||||
warn "no SSH key set — you'll only be able to reach the guest with 'hermes-vm console'"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ------------------------------------------------------ patch common.nix ---
|
||||
|
||||
step "Writing $COMMON"
|
||||
|
||||
[ -f "$COMMON.bak" ] || cp "$COMMON" "$COMMON.bak"
|
||||
note "original saved as $COMMON.bak"
|
||||
|
||||
# cachePublicKey
|
||||
tmp="$(mktemp)"
|
||||
sed 's|^\( cachePublicKey = \).*|\1"'"$CACHE_PUB"'";|' "$COMMON" > "$tmp"
|
||||
mv "$tmp" "$COMMON"
|
||||
ok "cachePublicKey set"
|
||||
|
||||
# authorizedKeys block
|
||||
if [ -n "$SELECTED_KEYS" ]; then
|
||||
tmp="$(mktemp)"
|
||||
awk -v keys="$SELECTED_KEYS" '
|
||||
/^ authorizedKeys = \[/ { print; printf "%s", keys; inblock = 1; next }
|
||||
inblock && /^ \];/ { inblock = 0; print; next }
|
||||
inblock { next }
|
||||
{ print }
|
||||
' "$COMMON" > "$tmp"
|
||||
mv "$tmp" "$COMMON"
|
||||
ok "authorizedKeys set ($(printf '%s' "$SELECTED_KEYS" | grep -c '^' ) key(s))"
|
||||
fi
|
||||
|
||||
# ----------------------------------------------------------- build image ---
|
||||
|
||||
step "Building the bootstrap image"
|
||||
|
||||
note "The extra substituter flags stop you compiling Determinate Nix yourself."
|
||||
|
||||
if confirm "Build now? (first run can take a while)"; then
|
||||
nix build .#hermes-guest-image \
|
||||
--option extra-substituters "$DETERMINATE_SUB" \
|
||||
--option extra-trusted-public-keys "$DETERMINATE_KEY"
|
||||
ok "built"
|
||||
else
|
||||
say "Skipped. Run manually:"
|
||||
note " nix build .#hermes-guest-image \\"
|
||||
note " --option extra-substituters $DETERMINATE_SUB \\"
|
||||
note " --option extra-trusted-public-keys $DETERMINATE_KEY"
|
||||
fi
|
||||
|
||||
# ------------------------------------------------------------ write disk ---
|
||||
|
||||
step "Writing the bootstrap image to the system disk"
|
||||
|
||||
IMAGE="$(ls -1 result/*.img 2>/dev/null | head -n1 || true)"
|
||||
|
||||
if [ -z "$IMAGE" ]; then
|
||||
warn "no image found under ./result - skipping"
|
||||
note "build it, then:"
|
||||
note " sudo dd if=result/nixos.img of=$SYSTEM_IMG bs=4M conv=notrunc,fsync status=progress"
|
||||
else
|
||||
IMG_BYTES=$(stat -Lc %s "$IMAGE")
|
||||
VOL_BYTES=$($SUDO stat -c %s "$SYSTEM_IMG")
|
||||
say "image: $IMAGE ($((IMG_BYTES / 1024 / 1024)) MiB)"
|
||||
say "target: $SYSTEM_IMG ($((VOL_BYTES / 1024 / 1024 / 1024)) GiB)"
|
||||
[ "$IMG_BYTES" -le "$VOL_BYTES" ] || die "image is larger than the system disk"
|
||||
|
||||
if [ "$SYSTEM_EXISTED" = yes ]; then
|
||||
warn "the system disk already existed. Writing the image replaces the"
|
||||
warn "guest OS. Hermes's identity and memories live on the STATE disk and"
|
||||
warn "are untouched, but anything it installed outside /var/lib/hermes is lost."
|
||||
fi
|
||||
|
||||
if confirm_word "overwrite" "Type 'overwrite' to write the image:"; then
|
||||
# notrunc so the file keeps its full size; the image is smaller than the disk
|
||||
$SUDO dd if="$IMAGE" of="$SYSTEM_IMG" bs=4M conv=notrunc,fsync status=progress
|
||||
ok "written"
|
||||
else
|
||||
say "Skipped."
|
||||
fi
|
||||
fi
|
||||
|
||||
# ----------------------------------------------------------- next steps ----
|
||||
|
||||
UPLINK="$(ip route get 1.1.1.1 2>/dev/null | sed -n 's/.* dev \([^ ]*\).*/\1/p' | head -n1)"
|
||||
UPLINK="${UPLINK:-eno1}"
|
||||
|
||||
step "Next: add the module to your system flake"
|
||||
|
||||
cat <<EOF
|
||||
|
||||
${DIM}# flake.nix${N}
|
||||
inputs.hermes-vm.url = "path:$(pwd)";
|
||||
|
||||
${DIM}# then in your modules list${N}
|
||||
hermes-vm.nixosModules.default
|
||||
|
||||
${DIM}# configuration.nix${N}
|
||||
services.hermesVm = {
|
||||
enable = true;
|
||||
disk.baseDir = "$BASE";
|
||||
network.externalInterface = "$UPLINK";
|
||||
vcpu = 4;
|
||||
mem = 8192;
|
||||
};
|
||||
|
||||
EOF
|
||||
|
||||
note "detected uplink: $UPLINK — check that's right before rebuilding"
|
||||
|
||||
cat <<EOF
|
||||
|
||||
Then:
|
||||
|
||||
sudo nixos-rebuild switch
|
||||
hermes-vm console ${DIM}# watch it boot${N}
|
||||
hermes-vm ssh ${DIM}# get in${N}
|
||||
|
||||
Inside the guest, in this order:
|
||||
|
||||
lsblk -f ${DIM}# CHECK LABELS match guest/hardware.nix${N}
|
||||
determinate-nixd login ${DIM}# FlakeHub Cache${N}
|
||||
sudo nix flake lock /etc/nixos
|
||||
curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash
|
||||
hermes setup
|
||||
hermes gateway install
|
||||
|
||||
Back on the host:
|
||||
|
||||
hermes-ctl units ${DIM}# confirm the gateway unit name${N}
|
||||
hermes-ctl status
|
||||
hermes-ctl soul
|
||||
|
||||
EOF
|
||||
|
||||
warn "Check the disk labels before the guest's first nixos-rebuild. If they"
|
||||
warn "don't match guest/hardware.nix, the first self-rebuild produces a"
|
||||
warn "system that won't boot."
|
||||
|
||||
printf '\n%s✓ host setup complete%s\n\n' "$G" "$N"
|
||||
Reference in new issue
Block a user