doc clean
This commit is contained in:
1 parent
c2a7f94821
commit
30427e3975
34 files changed
+1119
-1758
No files matched your search
+101
@@ -0,0 +1,101 @@
|
||||
# CLI reference
|
||||
|
||||
One binary, `hermenix`:
|
||||
|
||||
| | For |
|
||||
| -------------------------------- | ---------------------------------------------------------- |
|
||||
| `hermenix setup` | host provisioning — disks, cache key, image |
|
||||
| `hermenix migrate <src>` | existing flat `HERMES_HOME` → guest |
|
||||
| `hermenix migrate connections` | connection files only |
|
||||
| `hermenix remote <host> "<cmd>"` | `setup`/`migrate` over SSH |
|
||||
| `hermenix vm <sub>` | VM lifecycle, snapshots, backups |
|
||||
| `hermenix ctl <sub>` | host-side bridge into the guest (files, unit, passthrough) |
|
||||
| `hermenix version` | packaged version string |
|
||||
|
||||
Quote multi-word remote/passthrough args as one string —
|
||||
[CONCEPTS.md](CONCEPTS.md#quote-multi-word-subcommands).
|
||||
|
||||
Setup: [INSTALL.md](INSTALL.md). Migrate: [MIGRATE.md](MIGRATE.md).
|
||||
|
||||
---
|
||||
|
||||
## `hermenix vm`
|
||||
|
||||
```bash
|
||||
hermenix vm ssh # interactive shell (vsock)
|
||||
hermenix vm ssh "systemctl status" # run a command and exit
|
||||
hermenix vm console # journalctl -u hermes-vm -f (read-only)
|
||||
hermenix vm start | stop | restart
|
||||
|
||||
hermenix vm config # guest generation, history, git status
|
||||
hermenix vm config-pull [dir] # copy guest /etc/nixos here
|
||||
|
||||
hermenix vm snapshot | snapshots
|
||||
hermenix vm rollback <snap> # type 'rollback' to confirm
|
||||
|
||||
hermenix vm backup | backups
|
||||
```
|
||||
|
||||
`console` is follow-only (no input). No interactive fallback if the guest is
|
||||
dead.
|
||||
|
||||
Repo `guest/` is the **seed**; live guest `/etc/nixos` diverges after the
|
||||
agent edits it. `config-pull` is for reading/diffing — a host rebuild will not
|
||||
fix a guest config problem.
|
||||
|
||||
---
|
||||
|
||||
## `hermenix ctl`
|
||||
|
||||
Host-side helpers over vsock. They know where Hermes's files live in _this_
|
||||
VM layout and how to reach the gateway unit — they are not a second Hermes
|
||||
manual. For Hermes features (chat, sessions, providers, skills, …) use the
|
||||
upstream CLI or [Hermes Agent docs](https://github.com/NousResearch/hermes-agent); unknown `ctl` words are
|
||||
passed through to `hermes` in the guest.
|
||||
|
||||
```bash
|
||||
# Files under hermesHome (default /var/lib/hermes/.hermes)
|
||||
hermenix ctl soul [edit]
|
||||
hermenix ctl memory [edit] # memories/USER.md if present
|
||||
hermenix ctl conf [edit] # config.yaml
|
||||
hermenix ctl secrets [edit|show] # .env, mode 0600; masked by default
|
||||
|
||||
# Gateway user unit (name: services.hermenix.gatewayUnit)
|
||||
hermenix ctl status | logs | restart
|
||||
hermenix ctl units
|
||||
|
||||
# Seed / fold-back (destructive where noted)
|
||||
hermenix ctl reseed # overwrite from /etc/nixos/hermes seed
|
||||
hermenix ctl pull [dir] # live home → host (no .env/sessions/state.db)
|
||||
```
|
||||
|
||||
Passthrough examples (Hermes's own commands):
|
||||
|
||||
```bash
|
||||
hermenix ctl chat
|
||||
hermenix ctl update
|
||||
hermenix ctl "sessions list"
|
||||
hermenix ctl doctor
|
||||
```
|
||||
|
||||
`edit` pulls over vsock, opens local `$EDITOR`, writes back, offers restart.
|
||||
Secrets never enter the Nix store — no declarative option for them.
|
||||
|
||||
If `status`/`logs` are empty, check `hermenix ctl units` and set
|
||||
`services.hermenix.gatewayUnit` to the real unit name.
|
||||
|
||||
`services.hermenix.hermesHome` is where `ctl` looks. Override for a flat
|
||||
migrated layout; export `HERMES_HOME` inside the guest for the gateway
|
||||
process if it differs from the default.
|
||||
|
||||
---
|
||||
|
||||
## Env baked in by the host module
|
||||
|
||||
| Env | Option | Default |
|
||||
| ---------------------- | -------------------------------- | ------------------------- |
|
||||
| `HERMES_VSOCK_CID` | `services.hermenix.vsockCid` | `77` |
|
||||
| `HERMES_GUEST_NAME` | `services.hermenix.name` | `hermes` |
|
||||
| `HERMES_DISK_BASE_DIR` | `services.hermenix.disk.baseDir` | `/var/lib/hermes-vm` |
|
||||
| `HERMES_HOME_DIR` | `services.hermenix.hermesHome` | `/var/lib/hermes/.hermes` |
|
||||
| `HERMES_GATEWAY_UNIT` | `services.hermenix.gatewayUnit` | `hermes-agent` |
|
||||
@@ -0,0 +1,98 @@
|
||||
# Concepts
|
||||
|
||||
Shared model for everything else in these docs. Read once; other pages assume it.
|
||||
|
||||
## Two disks
|
||||
|
||||
| | Holds | Size (default) | Snapshotted | Backed up |
|
||||
| ---------- | ---------------------------------------------- | -------------- | ----------- | --------- |
|
||||
| **system** | NixOS, `/nix/store` | 80G | No | No |
|
||||
| **state** | identity, workspace, `/etc/nixos`, Hermes home | 20G | Yes | Yes |
|
||||
|
||||
System is large, churny, rebuildable from this flake. State holds what cannot
|
||||
be reproduced — snapshots stay cheap.
|
||||
|
||||
Host images under `services.hermenix.disk.baseDir` (default `/var/lib/hermes-vm`):
|
||||
|
||||
```
|
||||
$baseDir/system/system.raw
|
||||
$baseDir/state/state.raw
|
||||
$baseDir/snapshots/
|
||||
```
|
||||
|
||||
Guest: state disk at **`/var/lib/hermes`**. Hermes home defaults to
|
||||
`/var/lib/hermes/.hermes` (`~/.hermes` for user `hermes`). `/etc/nixos` →
|
||||
`/var/lib/hermes/nixos`.
|
||||
|
||||
What belongs _inside_ Hermes home (SOUL, memories, skills, sessions, …) is
|
||||
defined by Hermes itself — see [upstream docs](https://github.com/NousResearch/hermes-agent). hermenix only
|
||||
decides which disk it lives on and that the seed is applied once.
|
||||
|
||||
## The boundary is the host
|
||||
|
||||
The guest owns its kernel and config. Nothing _inside_ the guest is a security
|
||||
boundary; `modules/host.nix` nftables are. QEMU runs as unprivileged
|
||||
`hermes-vm` with seccomp + systemd hardening. Thin device model: virtio-blk,
|
||||
virtio-net, vsock, RNG, serial. No virtiofs. `vhost-net` off on purpose.
|
||||
|
||||
| | Default |
|
||||
| ------------------------- | --------------------------------------------------- |
|
||||
| Public internet | Yes |
|
||||
| Host Nix cache | Yes, one port; `hostCache.enable = false` closes it |
|
||||
| Anything else on the host | No, unless `hostAccess` |
|
||||
| LAN / other VMs | No, unless `lanAccess` |
|
||||
| Inbound from the network | No, unless `publish` (NAT mode) |
|
||||
| Host hardware | None, unless `passthrough` |
|
||||
|
||||
`network.bridged = true`: LAN inbound = every port the guest opens; `publish`
|
||||
ignored. Outbound gates unchanged. [NETWORK.md](NETWORK.md).
|
||||
|
||||
## Seed once
|
||||
|
||||
`guest/hermes/` is a **template** copied into the guest on first boot only
|
||||
(`/var/lib/hermes/.seeded`):
|
||||
|
||||
```
|
||||
guest/hermes/
|
||||
home/ → ~/.hermes (SOUL.md, config.yaml, skills/, …)
|
||||
workspace/ → ~/workspace
|
||||
```
|
||||
|
||||
Hermes owns and rewrites its home after that — re-seeding on every rebuild
|
||||
would wipe learned state. hermenix consequences:
|
||||
|
||||
- Editing `guest/hermes/` post-first-boot does nothing alone →
|
||||
`hermenix ctl reseed` (destructive; needs fresh confirmation).
|
||||
- Live files: `hermenix ctl soul` / `conf` / … or `hermenix ctl pull`.
|
||||
- Machine facts (not identity): `/etc/hermes/ENVIRONMENT.md`, regenerated
|
||||
every guest rebuild from `guest/common.nix`.
|
||||
|
||||
## vsock, not network SSH
|
||||
|
||||
Host → guest (`vm ssh`, `ctl`, backups) uses **vsock** (default CID 77). No
|
||||
management SSH port on the network path.
|
||||
|
||||
```
|
||||
ssh -o ProxyCommand='socat - VSOCK-CONNECT:<cid>:22' hermes@hermes
|
||||
```
|
||||
|
||||
## Determinate Nix in the guest
|
||||
|
||||
Guest: **Determinate Nix**, nixpkgs on FlakeHub. Host: stock Nix.
|
||||
|
||||
- Do not edit `/etc/nix/nix.conf` — custom settings in
|
||||
`/etc/nix/nix.custom.conf` (from `guest/common.nix`).
|
||||
- No guest `nix.settings` / `nix.gc`; `determinate-nixd` collects.
|
||||
- Substituters: FlakeHub Cache → host cache → `cache.nixos.org`.
|
||||
- Do not set `inputs.nixpkgs.follows` on `determinate` (cache misses).
|
||||
|
||||
## Quote multi-word subcommands
|
||||
|
||||
nu treats bare `--flag` tokens as flags on the _outer_ command. Quote anything
|
||||
past a single bare word:
|
||||
|
||||
```bash
|
||||
hermenix remote fwdesk.lab.internal "setup --clean"
|
||||
hermenix vm ssh "systemctl status"
|
||||
hermenix ctl "sessions list"
|
||||
```
|
||||
@@ -0,0 +1,82 @@
|
||||
# Hacking on this repo
|
||||
|
||||
## Devshell
|
||||
|
||||
```bash
|
||||
direnv allow
|
||||
```
|
||||
|
||||
Provides nushell, nufmt, nixfmt, and pre-commit hooks (installed on first
|
||||
entry):
|
||||
|
||||
- **treefmt** — nixfmt / deadnix / statix / nufmt
|
||||
- **nu-check** — parse-check `*.nu`
|
||||
- **gen-completions** — regenerates `nu/completions.nu` from usage blocks
|
||||
- **nu-tests** — `bin/runtest` when anything under `nu/` changes
|
||||
|
||||
## Layout
|
||||
|
||||
See [GUIDELINES.md](../GUIDELINES.md#repository-layout). The CLI is several
|
||||
nushell files concatenated by `flake.nix` in dependency order
|
||||
(`lib/` → `host/` → `vm/` → `ctl/` → `hermenix.nu`).
|
||||
|
||||
## Testing
|
||||
|
||||
Three suites. **Agents may run them only through the wrappers below** — never
|
||||
via bare `nix build` / `nix flake check`. See [GUIDELINES.md](../GUIDELINES.md).
|
||||
|
||||
### CLI tests (`nu/tests/`, nutest)
|
||||
|
||||
```bash
|
||||
runtest # everything (bin/ on PATH in the devShell)
|
||||
runtest vm_test
|
||||
runtest-debug vm_test
|
||||
```
|
||||
|
||||
Pure helpers + smoke that the subcommand tree parses — no ssh, no disk writes,
|
||||
no `nix build`.
|
||||
|
||||
### Guest VM tests (`guest/vm-tests/`)
|
||||
|
||||
Boot `guest/common.nix` in ephemeral NixOS test VMs. Never touches
|
||||
`/var/lib/hermes-vm/**`:
|
||||
|
||||
```bash
|
||||
runvmtest # NAT / isolated-bridge (default)
|
||||
runvmtest guest-vm-bridged # bridged mode with simulated LAN
|
||||
runvmtest-debug # interactive driver REPL
|
||||
```
|
||||
|
||||
Shared seeding/SSH/user assertions live in `guest/vm-tests/common.nix`.
|
||||
Mode-specific checks:
|
||||
|
||||
- **unbridged** — static `10.77.0.2/24` only, one default route via host
|
||||
- **bridged** — DHCP lease beside service net, dual default routes, LAN
|
||||
reachability both ways
|
||||
|
||||
Does not cover real `hermes-install` (no internet in the test VM) or the host
|
||||
`hermes-vm` QEMU unit.
|
||||
|
||||
### Module evaluation checks
|
||||
|
||||
Static — no VM boot. Evaluates `modules/host.nix` in both network modes plus
|
||||
`guest/common.nix`, asserts on nftables / networkd / guest DHCP metrics.
|
||||
|
||||
Operator/CI (not the agent-safe wrapper path today):
|
||||
|
||||
```bash
|
||||
nix build .#checks.x86_64-linux.network-modes
|
||||
```
|
||||
|
||||
Agents: describe the check and let the operator run it, unless/until a
|
||||
`bin/` wrapper exists for it.
|
||||
|
||||
## Parsing the CLI without running it
|
||||
|
||||
```bash
|
||||
nu nu/hermenix.nu --help
|
||||
nu nu/hermenix.nu setup --help
|
||||
# etc.
|
||||
```
|
||||
|
||||
Nix syntax only: `nix-instantiate --parse path/to/file.nix`.
|
||||
+117
@@ -0,0 +1,117 @@
|
||||
# Install
|
||||
|
||||
## Guided (recommended)
|
||||
|
||||
The `hermenix` CLI is a flake package — no checkout needed:
|
||||
|
||||
```bash
|
||||
nix run git+ssh://gitea@gitea.skyg.ca/ari/hermenix.git#hermenix -- setup
|
||||
```
|
||||
|
||||
Or install once:
|
||||
|
||||
```bash
|
||||
nix profile install git+ssh://gitea@gitea.skyg.ca/ari/hermenix.git#hermenix
|
||||
hermenix setup
|
||||
```
|
||||
|
||||
`setup` walks host-side steps: prerequisites, disks, cache signing key,
|
||||
`/etc/hermenix/local.nix` from your SSH keys, bootstrap image build, write to
|
||||
the system disk. It prints the module snippet to paste into your configuration
|
||||
(uplink filled in).
|
||||
|
||||
Safe to re-run — steps skip what's done; nothing destructive without
|
||||
confirmation. It never runs `nixos-rebuild` (that's your config).
|
||||
|
||||
Hand-run equivalent: [MANUAL_SETUP.md](MANUAL_SETUP.md).
|
||||
|
||||
## Add the module
|
||||
|
||||
```nix
|
||||
# flake.nix
|
||||
{
|
||||
inputs.hermenix.url = "git+ssh://gitea@gitea.skyg.ca/ari/hermenix.git";
|
||||
|
||||
outputs = { nixpkgs, hermenix, ... }: {
|
||||
nixosConfigurations.myserver = nixpkgs.lib.nixosSystem {
|
||||
system = "x86_64-linux";
|
||||
modules = [ hermenix.nixosModules.default ./configuration.nix ];
|
||||
};
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
```nix
|
||||
# configuration.nix
|
||||
services.hermenix = {
|
||||
enable = true;
|
||||
disk.baseDir = "/var/lib/hermes-vm";
|
||||
network.externalInterface = "eno1"; # your uplink
|
||||
# network.bridged = true; # see docs/NETWORK.md
|
||||
vcpu = 4;
|
||||
mem = 8192;
|
||||
};
|
||||
```
|
||||
|
||||
```bash
|
||||
sudo nixos-rebuild switch
|
||||
hermenix vm console # watch it boot
|
||||
```
|
||||
|
||||
## First boot (hermenix side)
|
||||
|
||||
```bash
|
||||
hermenix vm ssh
|
||||
|
||||
# Labels must match guest/hardware.nix BEFORE any guest rebuild
|
||||
lsblk -f
|
||||
|
||||
# FlakeHub Cache (device-code flow — approve URL from any browser)
|
||||
determinate-nixd login
|
||||
|
||||
sudo nix flake lock /etc/nixos
|
||||
```
|
||||
|
||||
The Hermes CLI installs unattended on first boot (`hermes-install.service`;
|
||||
a few minutes for native modules). After that, configure Hermes itself with
|
||||
its own CLI — provider, keys, gateway, etc. That is **upstream Hermes**, not
|
||||
hermenix; see the [Hermes Agent docs](https://github.com/NousResearch/hermes-agent).
|
||||
|
||||
From the host you can drive the guest install/gateway unit once it exists:
|
||||
|
||||
```bash
|
||||
hermenix ctl units
|
||||
hermenix ctl status
|
||||
```
|
||||
|
||||
> **Check labels before the first guest `nixos-rebuild`.**
|
||||
> `guest/hardware.nix` assumes root `nixos` and ESP `ESP`. Wrong labels →
|
||||
> unbootable system after the agent's first self-rebuild.
|
||||
|
||||
Guest NixOS changes are durable via
|
||||
`sudo nixos-rebuild switch --flake /etc/nixos#hermes` inside the VM. Hermes
|
||||
self-updates are upstream (`hermes update`); use `hermenix ctl` only as a
|
||||
vsock passthrough if you prefer running them from the host.
|
||||
|
||||
## Remote host
|
||||
|
||||
`setup` / `migrate` act on the machine they run on. `hermenix remote` SSHs:
|
||||
|
||||
```bash
|
||||
hermenix remote fwdesk.lab.internal setup
|
||||
hermenix remote fwdesk.lab.internal "setup --clean"
|
||||
hermenix remote fwdesk.lab.internal "migrate /var/lib/hermes"
|
||||
```
|
||||
|
||||
Quote subcommand + flags as **one string** when more than a bare name —
|
||||
[CONCEPTS.md](CONCEPTS.md#quote-multi-word-subcommands).
|
||||
|
||||
Runs `nix run <flake>#hermenix -- …` on the remote (no checkout). Remote needs
|
||||
SSH access to Gitea. Override URL with `--flake`. Uses `ssh -t` for prompts.
|
||||
|
||||
## Next
|
||||
|
||||
- [CLI.md](CLI.md) — day-to-day `vm` / `ctl`
|
||||
- [NETWORK.md](NETWORK.md) — allowlists / bridged
|
||||
- [MIGRATE.md](MIGRATE.md) — existing `HERMES_HOME`
|
||||
- [RECOVERY.md](RECOVERY.md) — snapshots / unbootable guest
|
||||
@@ -0,0 +1,84 @@
|
||||
# Manual setup
|
||||
|
||||
What `hermenix setup` does, step by step, for hand-running or when something
|
||||
goes sideways. Prefer the guided command in [INSTALL.md](INSTALL.md) when you
|
||||
can.
|
||||
|
||||
## 1. Create the disks
|
||||
|
||||
Two disks — design in [CONCEPTS.md](CONCEPTS.md#two-disks).
|
||||
|
||||
```bash
|
||||
sudo btrfs subvolume create /var/lib/hermes-vm/system
|
||||
sudo btrfs subvolume create /var/lib/hermes-vm/state
|
||||
sudo mkdir -p /var/lib/hermes-vm/snapshots
|
||||
|
||||
# BEFORE creating any file inside them
|
||||
sudo chattr +C /var/lib/hermes-vm/system /var/lib/hermes-vm/state
|
||||
sudo btrfs property set /var/lib/hermes-vm/system compression none
|
||||
sudo btrfs property set /var/lib/hermes-vm/state compression none
|
||||
|
||||
sudo truncate -s 80G /var/lib/hermes-vm/system/system.raw
|
||||
sudo truncate -s 20G /var/lib/hermes-vm/state/state.raw
|
||||
sudo mkfs.ext4 -L hermes-state /var/lib/hermes-vm/state/state.raw
|
||||
```
|
||||
|
||||
**`chattr +C` ordering matters.** Btrfs CoW stacked under a VM disk image
|
||||
fragments badly. `+C` disables that — but only for files created _after_.
|
||||
Set it on empty subvolumes and confirm with `lsattr` before writing anything.
|
||||
Wrong order → recreate the subvolume and copy the image in fresh.
|
||||
|
||||
Separate subvolumes so a btrfs snapshot of state does not include the system
|
||||
image.
|
||||
|
||||
## 2. Generate the cache signing key
|
||||
|
||||
The host serves its store to the guest as a read-only binary cache.
|
||||
|
||||
```bash
|
||||
sudo mkdir -p /var/lib/secrets
|
||||
sudo nix-store --generate-binary-cache-key hermes-cache-1 \
|
||||
/var/lib/secrets/hermes-cache.secret /var/lib/secrets/hermes-cache.pub
|
||||
sudo chown nix-serve /var/lib/secrets/hermes-cache.secret
|
||||
sudo chmod 0400 /var/lib/secrets/hermes-cache.secret
|
||||
cat /var/lib/secrets/hermes-cache.pub
|
||||
```
|
||||
|
||||
## 3. Write `/etc/hermenix/local.nix`
|
||||
|
||||
Host-specific settings live here, not in this repo — see
|
||||
`guest/local.nix.example`:
|
||||
|
||||
```nix
|
||||
{
|
||||
cachePublicKey = "hermes-cache-1:AbCd..."; # key you just printed; empty = skip host cache
|
||||
authorizedKeys = [ "ssh-ed25519 AAAA... you@laptop" ];
|
||||
}
|
||||
```
|
||||
|
||||
`sudo mkdir -p /etc/hermenix` first if needed.
|
||||
|
||||
## 4. Build the image and write it to the system disk
|
||||
|
||||
`--impure` is required because `guest/common.nix` reads
|
||||
`/etc/hermenix/local.nix` outside the flake source tree. The extra substituter
|
||||
flags avoid compiling Determinate Nix from source. `setup` also passes
|
||||
`--no-write-lock-file --refresh` when building from the git URL — mirror that
|
||||
if you're doing the same by hand.
|
||||
|
||||
```bash
|
||||
nix build --impure --no-write-lock-file --refresh \
|
||||
git+ssh://gitea@gitea.skyg.ca/ari/hermenix.git#hermes-guest-image \
|
||||
--option extra-substituters https://install.determinate.systems \
|
||||
--option extra-trusted-public-keys cache.flakehub.com-3:hJuILl5sVK4iKm86JzgdXW12Y2Hwd5G07qKtHTOcDCM=
|
||||
|
||||
# notrunc keeps the disk at its full size; the image is smaller
|
||||
sudo dd if=result/nixos.img of=/var/lib/hermes-vm/system/system.raw \
|
||||
bs=4M conv=notrunc,fsync status=progress
|
||||
```
|
||||
|
||||
From a local checkout, use `.` instead of the git URL (still `--impure`).
|
||||
|
||||
## 5. Module + first boot
|
||||
|
||||
Continue from [INSTALL.md](INSTALL.md#add-the-module).
|
||||
@@ -0,0 +1,74 @@
|
||||
# Migrate an existing deployment
|
||||
|
||||
Copy an existing flat `HERMES_HOME` into the guest after hermenix setup + host
|
||||
`nixos-rebuild` and `hermenix vm ssh true` work:
|
||||
|
||||
```bash
|
||||
nix run git+ssh://gitea@gitea.skyg.ca/ari/hermenix.git#hermenix -- \
|
||||
migrate /path/to/existing/HERMES_HOME
|
||||
```
|
||||
|
||||
`migrate --help` for vsock CID, guest user, `--dest`, hostname rewrites.
|
||||
|
||||
**Copy only — never move or delete the source** until the new side is verified.
|
||||
|
||||
Remote: [INSTALL.md](INSTALL.md#remote-host) — put `migrate /path/...` in the
|
||||
quoted command string.
|
||||
|
||||
What a `HERMES_HOME` contains and how Hermes uses it is upstream territory —
|
||||
[Hermes Agent](https://github.com/NousResearch/hermes-agent). This page is only the hermenix carry path.
|
||||
|
||||
## `hermesHome`
|
||||
|
||||
`services.hermenix.hermesHome` is where `hermenix ctl` looks in the guest.
|
||||
|
||||
| Layout | Value |
|
||||
| -------------------------- | -------------------------------------------- |
|
||||
| Seeded `.hermes` (default) | `/var/lib/hermes/.hermes` |
|
||||
| Flat at state root | `migrate --dest` (default `/var/lib/hermes`) |
|
||||
|
||||
Export `HERMES_HOME` inside the guest to the same path for the gateway process.
|
||||
|
||||
## Don't use `sudo`
|
||||
|
||||
`migrate` uses your user's ssh-over-vsock key (same as `authorizedKeys`).
|
||||
`sudo` → root has no key → `Permission denied (publickey)`. If
|
||||
`hermenix vm ssh` works as you, run `migrate` the same way.
|
||||
|
||||
## `hermenix migrate connections`
|
||||
|
||||
Connection/integration files only (no identity wipe), local or `--to-guest`:
|
||||
|
||||
```bash
|
||||
hermenix migrate connections /old/HERMES_HOME ./staging
|
||||
hermenix migrate connections /old/HERMES_HOME --to-guest
|
||||
```
|
||||
|
||||
Defaults: files `config.yaml`, `auth.json`, `.env`; dir `connections/`.
|
||||
Rewrites apply in the destination only.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Unreadable source entries
|
||||
|
||||
Rootless-Docker leftovers may be unreadable; `tar --ignore-failed-read` makes
|
||||
that non-fatal. Pull those paths as root separately if needed:
|
||||
|
||||
```bash
|
||||
sudo tar -C /path/to/HERMES_HOME/.hermes -cf - . \
|
||||
| hermenix vm ssh 'mkdir -p /var/lib/hermes/.hermes-old && tar -C /var/lib/hermes/.hermes-old -xf -'
|
||||
```
|
||||
|
||||
### Stale Docker launcher after an old migrate
|
||||
|
||||
Current `migrate` excludes `.local/bin` and `.hermes-installed`. If an older
|
||||
run left a Docker-routing `~/.local/bin/hermes` ("docker not found on PATH"):
|
||||
|
||||
```bash
|
||||
rm -f ~/.local/bin/hermes ~/.hermes-installed
|
||||
sudo systemctl start hermes-install # or reinstall per upstream docs
|
||||
journalctl -fu hermes-install
|
||||
```
|
||||
|
||||
Then finish Hermes configuration with the upstream CLI
|
||||
([docs](https://github.com/NousResearch/hermes-agent)), not with hermenix-specific steps.
|
||||
+120
@@ -0,0 +1,120 @@
|
||||
# Network
|
||||
|
||||
Default model and reachability table: [CONCEPTS.md](CONCEPTS.md#the-boundary-is-the-host).
|
||||
|
||||
## Mode matrix
|
||||
|
||||
| | NAT (default) | Bridged (`network.bridged = true`) |
|
||||
| ------------------------------ | ------------------------------------- | ---------------------------------------- |
|
||||
| Guest address | `10.77.0.2` on isolated bridge | service net **plus** fixed LAN address |
|
||||
| Inbound from LAN / internet | only via `publish` | every port the guest opens |
|
||||
| Guest → public internet | yes (masqueraded) | yes (L2 / LAN gateway) |
|
||||
| Guest → private LAN | `lanAccess` / `allowLan` only | same gates |
|
||||
| Guest → host services | `hostAccess` + host cache on `10.77.0.1` | same over service net |
|
||||
| `publish` | DNAT on uplink | **ignored** |
|
||||
|
||||
## Guest → a service on your LAN
|
||||
|
||||
```nix
|
||||
services.hermenix.lanAccess = [
|
||||
{ address = "192.168.1.50"; port = 11434; } # ollama
|
||||
{ address = "192.168.1.50"; port = 9000; proto = "udp"; }
|
||||
];
|
||||
```
|
||||
|
||||
One destination:port per entry, punched through the RFC1918 drop. Traffic is
|
||||
masqueraded; the peer sees the host's address.
|
||||
|
||||
`allowLan = true` opens all private space at once. Prefer `lanAccess` — an
|
||||
agent that goes wrong should not wander.
|
||||
|
||||
## Guest → a service on the host
|
||||
|
||||
```nix
|
||||
services.hermenix.hostAccess = [ { port = 8188; } ];
|
||||
```
|
||||
|
||||
The service must bind somewhere the guest can see. `127.0.0.1` will not work
|
||||
(guest arrives on the bridge). Bind to `10.77.0.1` (bridge only) or
|
||||
`0.0.0.0`:
|
||||
|
||||
```nix
|
||||
services.comfyui.listenAddress = "10.77.0.1";
|
||||
```
|
||||
|
||||
From the guest: `http://10.77.0.1:8188`.
|
||||
|
||||
Often better than GPU passthrough when you only need the capability over HTTP
|
||||
— see [PASSTHROUGH.md](PASSTHROUGH.md).
|
||||
|
||||
## Expose a guest service
|
||||
|
||||
**Guest side** — agent opens its firewall in `/etc/nixos`:
|
||||
|
||||
```nix
|
||||
networking.firewall.allowedTCPPorts = [ 22 8080 ];
|
||||
```
|
||||
|
||||
**Host only** — already allowed (outbound host→guest):
|
||||
|
||||
```bash
|
||||
curl http://10.77.0.2:8080
|
||||
```
|
||||
|
||||
**Wider network** (NAT mode):
|
||||
|
||||
```nix
|
||||
services.hermenix.publish = [
|
||||
{ hostPort = 8080; guestPort = 8080; }
|
||||
];
|
||||
```
|
||||
|
||||
DNATs the uplink port and opens the forward rule. Ports need not match.
|
||||
Publishing is the one change that lets the outside world start conversations
|
||||
with the agent's machine — put TLS/auth in front of anything that isn't a toy.
|
||||
|
||||
## Bridged mode (guest on the LAN)
|
||||
|
||||
Guest becomes a first-class LAN device: fixed address, reachable on any port it
|
||||
opens, outbound still gated.
|
||||
|
||||
```nix
|
||||
services.hermenix.network = {
|
||||
bridged = true;
|
||||
externalInterface = "enp191s0";
|
||||
|
||||
lanAddress = "10.0.12.50"; # guest's fixed LAN address
|
||||
hostLanAddress = "10.0.12.10"; # this host's LAN address
|
||||
lanGateway = "10.0.12.1";
|
||||
lanPrefixLength = 24;
|
||||
# lanDns = [ "1.1.1.1" ]; # optional; defaults to lanGateway
|
||||
};
|
||||
```
|
||||
|
||||
The bridge absorbs the uplink. The host-side DHCP on the bridge reserves
|
||||
`lanAddress` for the guest MAC; the guest stays a plain DHCP client. The
|
||||
bridge clones the uplink MAC so the host keeps its LAN identity. Guest DHCP
|
||||
broadcast never leaves the bridge.
|
||||
|
||||
### Switching over
|
||||
|
||||
1. Pick free addresses for guest, host (use the host's existing LAN address if
|
||||
reserved), and gateway. Keep them outside the DHCP pool or reserve them.
|
||||
2. Set options above, `sudo nixos-rebuild switch` (brief network drop).
|
||||
3. Verify: `hermenix vm ssh`; inside, `ip -4 addr` shows `10.77.0.2` **and**
|
||||
the LAN address. From another LAN device:
|
||||
`ssh hermes@<guest-lan-address>`.
|
||||
|
||||
### Switching back
|
||||
|
||||
`bridged = false` and rebuild. Guest DHCP finds no server on the isolated
|
||||
bridge and falls back to the static route via the host. No guest-side change.
|
||||
|
||||
### Gotchas
|
||||
|
||||
- `hostLanAddress` is a static LAN claim — reserve it.
|
||||
- Uplink network config is replaced by the bridge-port role while bridged.
|
||||
- Older guest images without DHCP-on-NIC in `guest/common.nix` won't take a
|
||||
bridged address; upgrade the image or apply that config inside the VM first.
|
||||
- Drop `publish` entries if you return to NAT mode; they do nothing while
|
||||
bridged.
|
||||
@@ -0,0 +1,124 @@
|
||||
# Hardware passthrough & virtual devices
|
||||
|
||||
## Before reaching for passthrough
|
||||
|
||||
If the goal is "the agent can use the GPU", passthrough is often wrong. It is
|
||||
exclusive (host loses the card while the VM runs) and is the largest isolation
|
||||
reduction in this setup.
|
||||
|
||||
Prefer: keep Ollama/ComfyUI on the host, open the port with `hostAccess`
|
||||
([NETWORK.md](NETWORK.md#guest--a-service-on-the-host)). Pass hardware through
|
||||
when the guest genuinely needs the device — CUDA in-process, capture card,
|
||||
radio, display output.
|
||||
|
||||
## What PCI passthrough costs
|
||||
|
||||
A PCI device does DMA. The IOMMU is the only thing between it and host memory,
|
||||
so `passthrough.iommu` is required; the module refuses to build without it.
|
||||
|
||||
- **Whole IOMMU groups only.** Check before planning:
|
||||
|
||||
```bash
|
||||
for d in /sys/kernel/iommu_groups/*/devices/*; do
|
||||
echo "group $(basename "$(dirname "$(dirname "$d")")"): $(lspci -nns "$(basename "$d")")"
|
||||
done | sort -V
|
||||
```
|
||||
|
||||
ACS override patches lie about isolation — not appropriate here.
|
||||
|
||||
- **Every function.** A GPU is at least graphics + HDMI audio.
|
||||
|
||||
## GPU
|
||||
|
||||
```bash
|
||||
lspci -Dnn | grep -i vga
|
||||
```
|
||||
|
||||
```nix
|
||||
services.hermenix.passthrough = {
|
||||
iommu = "amd"; # or "intel"
|
||||
pci = [
|
||||
{ address = "0000:01:00.0"; id = "10de:2204"; } # graphics
|
||||
{ address = "0000:01:00.1"; id = "10de:1aef"; } # HDMI audio
|
||||
];
|
||||
};
|
||||
```
|
||||
|
||||
Setting `id` makes vfio-pci claim the card in the initrd before `nvidia` /
|
||||
`amdgpu`. Do this for GPUs. Address-only devices bind at boot via a oneshot
|
||||
instead.
|
||||
|
||||
Rebuild and **reboot**, then:
|
||||
|
||||
```bash
|
||||
lspci -nnk -d 10de:2204 | grep "Kernel driver" # want: vfio-pci
|
||||
```
|
||||
|
||||
Guest-side drivers are the agent's problem (`/etc/nixos`).
|
||||
|
||||
## USB / Bluetooth
|
||||
|
||||
```bash
|
||||
lsusb
|
||||
```
|
||||
|
||||
```nix
|
||||
services.hermenix.passthrough.usb = [
|
||||
{ vendorId = "8087"; productId = "0026"; }
|
||||
{ vendorId = "0403"; productId = "6001"; }
|
||||
];
|
||||
```
|
||||
|
||||
QEMU mediates (no IOMMU/VFIO). Host must not be using the device (`btusb`
|
||||
blacklist, no `hardware.bluetooth` for an integrated radio). No hotplug —
|
||||
`hermenix vm restart` after plugging. Whole USB controller as PCI gives real
|
||||
hotplug at the cost of exclusivity.
|
||||
|
||||
## When the VM won't start
|
||||
|
||||
`hermenix vm console` first. Then: device not on vfio-pci (`lspci -nnk`),
|
||||
IOMMU off (`dmesg | grep -i iommu`), or something in the group still held by
|
||||
the host. The module raises `LimitMEMLOCK` when PCI passthrough is configured.
|
||||
|
||||
---
|
||||
|
||||
## Virtual devices
|
||||
|
||||
`services.hermenix.devices` attaches optional QEMU devices (no host hardware):
|
||||
|
||||
| Option | Default | Attaches |
|
||||
| ------------------------- | ---------- | ----------------------------------------------------- |
|
||||
| `devices.rng.enable` | `true` | virtio entropy |
|
||||
| `devices.vsock.enable` | `true` | host↔guest vsock (`ctl`, backups) |
|
||||
| `devices.video.enable` | `false` | virtual display / render node |
|
||||
| `devices.video.model` | `"virtio"` | `virtio` (render node) or legacy VGA |
|
||||
| `devices.balloon.enable` | `false` | virtio memory balloon |
|
||||
| `devices.watchdog.enable` | `false` | `i6300esb`/`ib700` |
|
||||
| `devices.input.enable` | `false` | virtio keyboard/tablet |
|
||||
| `devices.audio.enable` | `false` | `intel-hda`/`virtio-sound` (see caveat) |
|
||||
| `devices.tpm.enable` | `false` | TPM 2.0 via swtpm |
|
||||
| `devices.cdroms` | `[]` | read-only CD images |
|
||||
| `devices.extraDisks` | `[]` | extra raw disks |
|
||||
| `devices.usbController` | `"auto"` | `qemu-xhci` iff `passthrough.usb` set |
|
||||
|
||||
`extraQemuArgs` appends raw QEMU args (unvalidated). Prefer typed options.
|
||||
|
||||
### Render node
|
||||
|
||||
```nix
|
||||
services.hermenix.devices.video.enable = true;
|
||||
```
|
||||
|
||||
Gives `/dev/dri/card0` and `/dev/dri/renderD128` (`virtio` model only).
|
||||
Console stays serial.
|
||||
|
||||
### TPM
|
||||
|
||||
`devices.tpm.enable = true` starts swtpm; state under
|
||||
`/var/lib/hermes-vm-tpm` by default.
|
||||
|
||||
### Audio caveat
|
||||
|
||||
`hermes-vm.service` is hardened (`ProtectHome`, `PrivateTmp`, dedicated user)
|
||||
and cannot reach PipeWire without loosening that. Leave audio off unless you
|
||||
mean to relax the unit.
|
||||
@@ -0,0 +1,63 @@
|
||||
# Recovery
|
||||
|
||||
Cheapest first. Layers 1–2 are inside the guest; 3–4 are host-side hermenix.
|
||||
|
||||
```bash
|
||||
# Guest NixOS history
|
||||
hermenix vm ssh "git -C /etc/nixos log --oneline"
|
||||
hermenix vm ssh "sudo nixos-rebuild switch --rollback" # also in boot menu
|
||||
|
||||
# State-disk snapshot
|
||||
hermenix vm snapshot
|
||||
hermenix vm snapshots
|
||||
hermenix vm rollback state-20260914-1400 # type 'rollback' to confirm
|
||||
|
||||
# Restic backup of state files
|
||||
hermenix vm backup
|
||||
hermenix vm backups
|
||||
```
|
||||
|
||||
Identity files under Hermes home can be inspected with `hermenix ctl soul`
|
||||
etc.; restoring _Hermes_ app state is mostly "roll the state disk" or restore
|
||||
from backup — not a separate product flow.
|
||||
|
||||
## Snapshots
|
||||
|
||||
Hourly, keep 48, **state disk only**. System disk: re-image with
|
||||
`hermenix setup`.
|
||||
|
||||
fsfreeze via qemu-guest-agent around each shot. `rollback` stops the VM, moves
|
||||
current state to `state.old-<timestamp>`, restores snapshot — reversible until
|
||||
you delete `state.old-*`.
|
||||
|
||||
Keep retention tighter than on ZFS: state image is nodatacow; live snapshots
|
||||
reintroduce CoW on referenced blocks.
|
||||
|
||||
## Backups
|
||||
|
||||
```nix
|
||||
services.hermenix.backup = {
|
||||
enable = true;
|
||||
repository = "sftp:backup@nas:/hermes";
|
||||
passwordFile = "/var/lib/secrets/hermes-restic";
|
||||
};
|
||||
```
|
||||
|
||||
State **files** over vsock/SSH into restic — not the raw disk. Portable;
|
||||
avoids loop-mounting agent-controlled FS on the host. Runs on the host (no
|
||||
`lanAccess` needed for a LAN repo).
|
||||
|
||||
## Guest won't boot
|
||||
|
||||
Re-image **system** only (`hermenix setup` write step); leave state disk.
|
||||
Identity + `/etc/nixos` + workspace return with it.
|
||||
|
||||
## Starting completely fresh
|
||||
|
||||
```bash
|
||||
hermenix setup --clean
|
||||
```
|
||||
|
||||
Stops VM; deletes `system.raw` / `state.raw` / UEFI vars; drops stale
|
||||
`known_hosts` for the guest host key; walks setup again. Subvolumes/snapshots
|
||||
kept. Type `clean` to confirm.
|
||||
@@ -0,0 +1,21 @@
|
||||
# Known rough edges
|
||||
|
||||
- The Determinate NixOS module's option names for custom Nix settings aren't
|
||||
published — only the nix-darwin `determinateNix.customSettings` equivalent
|
||||
is. So `guest/common.nix` writes `/etc/nix/nix.custom.conf` via
|
||||
`environment.etc`. If a future module version claims that file, switch to
|
||||
its option.
|
||||
- `determinate-nixd login` is manual. No safe way to bake the credential into
|
||||
the image.
|
||||
- `MemoryDenyWriteExecute = true` is correct under KVM but is the first thing
|
||||
to flip if QEMU refuses to start.
|
||||
- `pkgs.OVMF.firmware` / `pkgs.OVMF.variables` are current attribute names;
|
||||
older nixpkgs put these under `pkgs.OVMF.fd` in an `FV/` subdirectory.
|
||||
- PCI passthrough needs a reboot (early vfio binding in initrd).
|
||||
- Snapshots need btrfs. Everything else works on any filesystem; without btrfs
|
||||
you rely on backups alone.
|
||||
- `state.old-*` left by `hermenix vm rollback` are not cleaned automatically.
|
||||
Delete them once you're sure.
|
||||
- First boot builds a lot. Give it time.
|
||||
- `devices.audio.enable` fights unit hardening — see
|
||||
[PASSTHROUGH.md](PASSTHROUGH.md#audio-caveat).
|
||||
Reference in new issue
Block a user