fixing test failures

This commit is contained in:
ari committed 2026-09-23 13:24:07 -04:00
1 parent c8ff8b1f25
commit 42e903909c
14 files changed
+823 -111

No files matched your search

+163 -35
View File
@@ -9,13 +9,13 @@ so.
One CLI, `hermenix`, with five subcommands:
| | For |
|---|---|
| `hermenix setup` | one-off host provisioning — disks, cache key, image build |
| `hermenix migrate <src>` | carry an existing flat `HERMES_HOME` into the guest |
| `hermenix remote <host> "<cmd>"` | run `setup`/`migrate` on a remote host over SSH |
| `hermenix vm <sub>` | control the VM itself — start/stop/console/snapshots/backups |
| `hermenix ctl <sub>` | drive the agent inside the VM — identity/config/secrets/gateway |
| | For |
| -------------------------------- | --------------------------------------------------------------- |
| `hermenix setup` | one-off host provisioning — disks, cache key, image build |
| `hermenix migrate <src>` | carry an existing flat `HERMES_HOME` into the guest |
| `hermenix remote <host> "<cmd>"` | run `setup`/`migrate` on a remote host over SSH |
| `hermenix vm <sub>` | control the VM itself — start/stop/console/snapshots/backups |
| `hermenix ctl <sub>` | drive the agent inside the VM — identity/config/secrets/gateway |
---
@@ -53,10 +53,10 @@ it by hand or something goes sideways.
Two of them, and the split is the whole design:
| | Holds | Size | Snapshotted | Backed up |
|---|---|---|---|---|
| **system** | NixOS, `/nix/store` | 80G | No | No |
| **state** | identity, memories, workspace, `/etc/nixos` | 20G | Yes | Yes |
| | Holds | Size | Snapshotted | Backed up |
| ---------- | ------------------------------------------- | ---- | ----------- | --------- |
| **system** | NixOS, `/nix/store` | 80G | No | No |
| **state** | identity, memories, workspace, `/etc/nixos` | 20G | Yes | Yes |
The system disk is large, takes all the write churn, and is entirely
rebuildable from this flake — so there's no point protecting it. The state
@@ -159,6 +159,8 @@ services.hermenix = {
enable = true;
disk.baseDir = "/var/lib/hermes-vm";
network.externalInterface = "eno1"; # your uplink
# network.bridged = true; # guest takes its own LAN IP via DHCP;
# # every port it opens is then LAN-reachable
vcpu = 4;
mem = 8192;
};
@@ -305,7 +307,7 @@ Two consequences:
- Editing `guest/hermes/` after first boot changes nothing by itself. Use
`hermenix ctl reseed` when you actually want to overwrite.
- The seed is the *starting* identity, not the current one. To see what
- The seed is the _starting_ identity, not the current one. To see what
Hermes has made of itself, `hermenix ctl soul`.
### Machine facts vs identity
@@ -477,6 +479,94 @@ auth in front of anything that isn't a toy, and keep the list short.
---
## Putting the guest directly on your LAN
The default setup puts the guest on an isolated bridge behind the host's NAT,
reachable from outside only through `publish`. This section is for when you
want the VM to be a first-class device on your LAN instead: its own address
from the LAN's DHCP, reachable on any port from any device on the network.
Enable it:
```nix
services.hermenix.network.bridged = true;
```
In one sentence, what that does: the bridge absorbs your uplink
(`network.externalInterface`), the guest and the host share the LAN, the
guest takes a DHCP lease, and inbound traffic to the guest is open on every
port. Outbound is still gated by `lanAccess` / `allowLan`, and `hostAccess`
plus the host Nix cache still work over the service net at `10.77.0.1`.
### 1. Ready the guest first (one-time, if the image is older than this repo)
The guest keeps a static service-net address (`10.77.0.2/24`) for talking to
the host and adds DHCP on the same NIC, so one config works in both modes.
That is already what `guest/common.nix` does; if your guest image predates
it, apply the equivalent change to `/etc/nixos/common.nix` inside the VM:
```nix
systemd.network.networks."10-eth" = {
matchConfig.Type = "ether";
address = [ "10.77.0.2/24" ]; # service net to the host
routes = [ { Gateway = "10.77.0.1"; Metric = 200; } ]; # NAT-mode fallback
networkConfig.DHCP = "ipv4"; # LAN lease in bridged mode
dhcpV4Config = { # DHCP default route wins (100)
UseDNS = false;
RouteMetric = 100;
};
};
```
`cd /etc/nixos && git apply -p1 < patch`, commit, and
`sudo nixos-rebuild switch --flake /etc/nixos#hermes`. In NAT mode the DHCP
attempt fails quietly (there is no server on the isolated bridge), so check
nothing broke: `ip -4 addr` still shows `10.77.0.2` and `ip route` still
goes via `10.77.0.1`.
### 2. Rebuild the host with the toggle on
Set `services.hermenix.network.bridged = true` alongside
`network.externalInterface`, then `sudo nixos-rebuild switch`. On rebuild
the host's uplink becomes a bridge port, the host takes a fresh DHCP lease
on the bridge, and the guest's DHCP is answered by the LAN router. The
network drops briefly while the host reconfigures - plan for it.
### 3. Verify
From the host, `hermenix vm ssh`. Inside the guest, `ip -4 addr` shows the
service net _and_ a LAN address, and `ip route` shows a default route via
your LAN gateway. From any other device on the LAN:
```bash
ssh hermes@<guest-lan-ip> # and any other port the guest opens
```
The guest's own firewall decides what answers - by default it only opens
port 22, so the agent must open whatever it wants LAN-reachable. Egress is
still gated too: from the guest, `192.168.1.50:11434` only works if it is
in `lanAccess` (or `allowLan = true`).
### Gotchas
- The LAN needs a working DHCP server.
- `publish` does nothing in this mode - every port is reachable, so its
entries are ignored. Drop them if you come back to NAT mode.
- The host's LAN IP comes from a fresh DHCP lease on the bridge, and the
bridge MAC can differ from the uplink's - the host's address may change
after the switch. Reserve it in the DHCP server if that matters.
- Any network config you had for the uplink interface is replaced by its
bridge-port role while bridged (this module's networkd unit for it takes
priority over most others).
### Switching back
Set `bridged = false` and rebuild the host. The guest's DHCP client finds
no server on the isolated bridge and settles onto its static route via the
host again; no guest-side change needed.
---
## Passing hardware to the guest
### Before reaching for this
@@ -583,7 +673,7 @@ Two things that catch people:
enable `hardware.bluetooth` on the host.
- **No hotplug.** QEMU resolves these at start, so a device that's unplugged
when the VM boots won't appear when you plug it in. `hermenix vm restart`
after plugging. Passing a whole USB *controller* through as a PCI device
after plugging. Passing a whole USB _controller_ through as a PCI device
gets you real hotplug for anything in its ports, at the cost of that
controller being exclusive to the guest.
@@ -648,7 +738,7 @@ services.hermenix.backup = {
};
```
This backs up the guest's *state*, not its disk. Files are pulled over the
This backs up the guest's _state_, not its disk. Files are pulled over the
SSH channel that already exists and streamed into restic.
Two reasons it works that way. The backup stays portable — it restores onto
@@ -658,7 +748,7 @@ host, which would mean the host kernel parsing data the agent controls. That
would undo a fair bit of what the rest of this design is for.
If the repository is on your LAN rather than the internet, note this runs on
the *host*, so it needs no `lanAccess` entry — the guest never touches it.
the _host_, so it needs no `lanAccess` entry — the guest never touches it.
### When the guest won't boot at all
@@ -705,14 +795,19 @@ The host stays on stock Nix. Nothing here requires changing that.
## What the guest can and cannot reach
| | Default |
|---|---|
| Public internet | Yes — it needs this for model APIs and messaging |
| Host Nix cache | Yes, one port, `hostCache.enable = false` closes it |
| Anything else on the host | No, unless in `hostAccess` |
| Your LAN and other VMs | No, unless in `lanAccess` |
| Inbound from the network | No, unless in `publish` |
| Host hardware | None, unless in `passthrough` |
| | Default |
| ------------------------- | --------------------------------------------------- |
| Public internet | Yes — it needs this for model APIs and messaging |
| Host Nix cache | Yes, one port, `hostCache.enable = false` closes it |
| Anything else on the host | No, unless in `hostAccess` |
| Your LAN and other VMs | No, unless in `lanAccess` |
| Inbound from the network | No, unless in `publish` |
| Host hardware | None, unless in `passthrough` |
With `network.bridged = true` the "Inbound from the network" row changes:
with the guest on the LAN, every port it opens is reachable from the LAN;
`publish` stops doing anything. The outbound rows are unchanged - the
`lanAccess` / `allowLan` gates still apply.
The guest owns its kernel and its config, so nothing inside it is a security
boundary. The nftables rules in `modules/host.nix` are. QEMU runs as an
@@ -862,7 +957,6 @@ journalctl -fu hermes-install # watch until complete
hermes setup
```
## Hacking on this repo
The devshell provides nushell, nufmt, and pre-commit hooks. Enter it once:
@@ -882,7 +976,7 @@ on every commit, including:
### Testing
Two test suites. Neither is ever run for you by an agent working in this
Three test suites. None is ever run for you by an agent working in this
repo — they're for you (or CI) to run.
**CLI tests** (`nu/tests/`), via [nutest](https://github.com/vyadh/nutest):
@@ -899,22 +993,56 @@ whole `hermenix`/`vm`/`ctl` subcommand tree parses and dispatches correctly
— without ever invoking a subcommand's real body (no ssh, no disk writes,
no `nix build`).
**Guest VM test** (`guest/vm-test.nix`): boots `guest/common.nix` — the same
module set `hermes-guest-image` is built from — in an ephemeral, throwaway
NixOS test VM. No real disk under `/var/lib/hermes-vm/**` is ever touched:
**Guest VM tests** (`guest/vm-tests/test_unbridge.nix` / `guest/vm-tests/test_bridge.nix`):
boot `guest/common.nix` — the same module set `hermes-guest-image` is built
from — in ephemeral, throwaway NixOS test VMs. No real disk under
`/var/lib/hermes-vm/**` is ever touched:
```bash
runvmtest # plain pass/fail (nix build .#checks.x86_64-linux.guest-vm --impure -L)
runvmtest-debug # build the interactive driver and drop into it (ptpython REPL)
runvmtest # NAT/isolated-bridge mode (plain pass/fail)
runvmtest guest-vm-bridged # bridged mode with a simulated LAN
runvmtest-debug # interactive driver REPL (ptpython) for the default suite
```
Checks first-boot seeding (`~/.hermes`, `~/workspace`, the `/etc/nixos`
symlink), that seeding is idempotent across a reboot (the whole point of
the `.seeded` marker), the `hermes` user/sudo setup, and SSH. It
deliberately does not cover the real `hermes-install` script (no internet
Both tests boot the guest the same way and share every check that must hold
in both modes — first-boot seeding (`~/.hermes`, `~/workspace`, the
`/etc/nixos` symlink), that seeding is idempotent across a reboot (the whole
point of the `.seeded` marker), the `hermes` user/sudo setup, and SSH. That
shared node config and assertion script are defined once in
`guest/vm-tests/common.nix` and imported by both, so nothing can drift
between the two modes' common checks.
Each test then adds the checks that only make sense for its mode:
- `guest-vm-unbridged` — the isolated bridge has no DHCP server, so it
asserts the guest stands on its static config alone: exactly the one
`10.77.0.2/24` address and exactly one default route, via the host.
- `guest-vm-bridged` — a second node simulates the LAN (a dnsmasq DHCP
server, a throwaway HTTP service, and a plain device). It asserts the
guest takes a DHCP lease beside the static service-net address, keeps
both default routes (the LAN gateway at metric 100 and the host fallback
at metric 200), can reach a service on the LAN, and that a LAN device can
connect to a port the guest exposes (the guest's own firewall still
decides what answers). The host-side half of bridged mode — the bridge
itself, the mark-based rules — is covered statically by the check below.
Both deliberately do not cover the real `hermes-install` script (no internet
route inside the test VM) or `modules/host.nix`'s own `hermes-vm` QEMU
service — both stay manual, operator-run checks.
service — those stay manual, operator-run checks.
`runvmtest-debug` gives you `start_all()`, `machine.wait_for_unit(...)`, and
`machine.shell_interact()` (a live shell inside the guest) to step through a
failure by hand instead of just getting pass/fail.
**Module evaluation checks** (`tests/network-modes-check.nix`): a static
check — no build, no VM boot — that evaluates `modules/host.nix` in both
networking modes (NAT and `network.bridged = true`) plus `guest/common.nix`,
and asserts on the generated nftables rulesets, the firewall input/forward
rules, the networkd units, and the guest's DHCP/route-metric config:
```bash
nix build .#checks.x86_64-linux.network-modes
```
Catches regressions in the bridged-mode flag without booting anything. If
it fails, the assertion that broke is named in the error.