fixing test failures

This commit is contained in:
ari committed 2026-09-23 13:24:07 -04:00
1 parent c8ff8b1f25
commit 42e903909c
14 files changed
+791 -79

No files matched your search

+3
View File
@@ -25,3 +25,6 @@ devenv.local.yaml
# pre-commit
.pre-commit-config.yaml
# tmp
.tmp
+147 -19
View File
@@ -10,7 +10,7 @@ so.
One CLI, `hermenix`, with five subcommands:
| | For |
|---|---|
| -------------------------------- | --------------------------------------------------------------- |
| `hermenix setup` | one-off host provisioning — disks, cache key, image build |
| `hermenix migrate <src>` | carry an existing flat `HERMES_HOME` into the guest |
| `hermenix remote <host> "<cmd>"` | run `setup`/`migrate` on a remote host over SSH |
@@ -54,7 +54,7 @@ it by hand or something goes sideways.
Two of them, and the split is the whole design:
| | Holds | Size | Snapshotted | Backed up |
|---|---|---|---|---|
| ---------- | ------------------------------------------- | ---- | ----------- | --------- |
| **system** | NixOS, `/nix/store` | 80G | No | No |
| **state** | identity, memories, workspace, `/etc/nixos` | 20G | Yes | Yes |
@@ -159,6 +159,8 @@ services.hermenix = {
enable = true;
disk.baseDir = "/var/lib/hermes-vm";
network.externalInterface = "eno1"; # your uplink
# network.bridged = true; # guest takes its own LAN IP via DHCP;
# # every port it opens is then LAN-reachable
vcpu = 4;
mem = 8192;
};
@@ -305,7 +307,7 @@ Two consequences:
- Editing `guest/hermes/` after first boot changes nothing by itself. Use
`hermenix ctl reseed` when you actually want to overwrite.
- The seed is the *starting* identity, not the current one. To see what
- The seed is the _starting_ identity, not the current one. To see what
Hermes has made of itself, `hermenix ctl soul`.
### Machine facts vs identity
@@ -477,6 +479,94 @@ auth in front of anything that isn't a toy, and keep the list short.
---
## Putting the guest directly on your LAN
The default setup puts the guest on an isolated bridge behind the host's NAT,
reachable from outside only through `publish`. This section is for when you
want the VM to be a first-class device on your LAN instead: its own address
from the LAN's DHCP, reachable on any port from any device on the network.
Enable it:
```nix
services.hermenix.network.bridged = true;
```
In one sentence, what that does: the bridge absorbs your uplink
(`network.externalInterface`), the guest and the host share the LAN, the
guest takes a DHCP lease, and inbound traffic to the guest is open on every
port. Outbound is still gated by `lanAccess` / `allowLan`, and `hostAccess`
plus the host Nix cache still work over the service net at `10.77.0.1`.
### 1. Ready the guest first (one-time, if the image is older than this repo)
The guest keeps a static service-net address (`10.77.0.2/24`) for talking to
the host and adds DHCP on the same NIC, so one config works in both modes.
That is already what `guest/common.nix` does; if your guest image predates
it, apply the equivalent change to `/etc/nixos/common.nix` inside the VM:
```nix
systemd.network.networks."10-eth" = {
matchConfig.Type = "ether";
address = [ "10.77.0.2/24" ]; # service net to the host
routes = [ { Gateway = "10.77.0.1"; Metric = 200; } ]; # NAT-mode fallback
networkConfig.DHCP = "ipv4"; # LAN lease in bridged mode
dhcpV4Config = { # DHCP default route wins (100)
UseDNS = false;
RouteMetric = 100;
};
};
```
`cd /etc/nixos && git apply -p1 < patch`, commit, and
`sudo nixos-rebuild switch --flake /etc/nixos#hermes`. In NAT mode the DHCP
attempt fails quietly (there is no server on the isolated bridge), so check
nothing broke: `ip -4 addr` still shows `10.77.0.2` and `ip route` still
goes via `10.77.0.1`.
### 2. Rebuild the host with the toggle on
Set `services.hermenix.network.bridged = true` alongside
`network.externalInterface`, then `sudo nixos-rebuild switch`. On rebuild
the host's uplink becomes a bridge port, the host takes a fresh DHCP lease
on the bridge, and the guest's DHCP is answered by the LAN router. The
network drops briefly while the host reconfigures - plan for it.
### 3. Verify
From the host, `hermenix vm ssh`. Inside the guest, `ip -4 addr` shows the
service net _and_ a LAN address, and `ip route` shows a default route via
your LAN gateway. From any other device on the LAN:
```bash
ssh hermes@<guest-lan-ip> # and any other port the guest opens
```
The guest's own firewall decides what answers - by default it only opens
port 22, so the agent must open whatever it wants LAN-reachable. Egress is
still gated too: from the guest, `192.168.1.50:11434` only works if it is
in `lanAccess` (or `allowLan = true`).
### Gotchas
- The LAN needs a working DHCP server.
- `publish` does nothing in this mode - every port is reachable, so its
entries are ignored. Drop them if you come back to NAT mode.
- The host's LAN IP comes from a fresh DHCP lease on the bridge, and the
bridge MAC can differ from the uplink's - the host's address may change
after the switch. Reserve it in the DHCP server if that matters.
- Any network config you had for the uplink interface is replaced by its
bridge-port role while bridged (this module's networkd unit for it takes
priority over most others).
### Switching back
Set `bridged = false` and rebuild the host. The guest's DHCP client finds
no server on the isolated bridge and settles onto its static route via the
host again; no guest-side change needed.
---
## Passing hardware to the guest
### Before reaching for this
@@ -583,7 +673,7 @@ Two things that catch people:
enable `hardware.bluetooth` on the host.
- **No hotplug.** QEMU resolves these at start, so a device that's unplugged
when the VM boots won't appear when you plug it in. `hermenix vm restart`
after plugging. Passing a whole USB *controller* through as a PCI device
after plugging. Passing a whole USB _controller_ through as a PCI device
gets you real hotplug for anything in its ports, at the cost of that
controller being exclusive to the guest.
@@ -648,7 +738,7 @@ services.hermenix.backup = {
};
```
This backs up the guest's *state*, not its disk. Files are pulled over the
This backs up the guest's _state_, not its disk. Files are pulled over the
SSH channel that already exists and streamed into restic.
Two reasons it works that way. The backup stays portable — it restores onto
@@ -658,7 +748,7 @@ host, which would mean the host kernel parsing data the agent controls. That
would undo a fair bit of what the rest of this design is for.
If the repository is on your LAN rather than the internet, note this runs on
the *host*, so it needs no `lanAccess` entry — the guest never touches it.
the _host_, so it needs no `lanAccess` entry — the guest never touches it.
### When the guest won't boot at all
@@ -706,7 +796,7 @@ The host stays on stock Nix. Nothing here requires changing that.
## What the guest can and cannot reach
| | Default |
|---|---|
| ------------------------- | --------------------------------------------------- |
| Public internet | Yes — it needs this for model APIs and messaging |
| Host Nix cache | Yes, one port, `hostCache.enable = false` closes it |
| Anything else on the host | No, unless in `hostAccess` |
@@ -714,6 +804,11 @@ The host stays on stock Nix. Nothing here requires changing that.
| Inbound from the network | No, unless in `publish` |
| Host hardware | None, unless in `passthrough` |
With `network.bridged = true` the "Inbound from the network" row changes:
with the guest on the LAN, every port it opens is reachable from the LAN;
`publish` stops doing anything. The outbound rows are unchanged - the
`lanAccess` / `allowLan` gates still apply.
The guest owns its kernel and its config, so nothing inside it is a security
boundary. The nftables rules in `modules/host.nix` are. QEMU runs as an
unprivileged `hermes-vm` user with its own seccomp sandbox and systemd
@@ -862,7 +957,6 @@ journalctl -fu hermes-install # watch until complete
hermes setup
```
## Hacking on this repo
The devshell provides nushell, nufmt, and pre-commit hooks. Enter it once:
@@ -882,7 +976,7 @@ on every commit, including:
### Testing
Two test suites. Neither is ever run for you by an agent working in this
Three test suites. None is ever run for you by an agent working in this
repo — they're for you (or CI) to run.
**CLI tests** (`nu/tests/`), via [nutest](https://github.com/vyadh/nutest):
@@ -899,22 +993,56 @@ whole `hermenix`/`vm`/`ctl` subcommand tree parses and dispatches correctly
— without ever invoking a subcommand's real body (no ssh, no disk writes,
no `nix build`).
**Guest VM test** (`guest/vm-test.nix`): boots `guest/common.nix` — the same
module set `hermes-guest-image` is built from — in an ephemeral, throwaway
NixOS test VM. No real disk under `/var/lib/hermes-vm/**` is ever touched:
**Guest VM tests** (`guest/vm-tests/test_unbridge.nix` / `guest/vm-tests/test_bridge.nix`):
boot `guest/common.nix` — the same module set `hermes-guest-image` is built
from — in ephemeral, throwaway NixOS test VMs. No real disk under
`/var/lib/hermes-vm/**` is ever touched:
```bash
runvmtest # plain pass/fail (nix build .#checks.x86_64-linux.guest-vm --impure -L)
runvmtest-debug # build the interactive driver and drop into it (ptpython REPL)
runvmtest # NAT/isolated-bridge mode (plain pass/fail)
runvmtest guest-vm-bridged # bridged mode with a simulated LAN
runvmtest-debug # interactive driver REPL (ptpython) for the default suite
```
Checks first-boot seeding (`~/.hermes`, `~/workspace`, the `/etc/nixos`
symlink), that seeding is idempotent across a reboot (the whole point of
the `.seeded` marker), the `hermes` user/sudo setup, and SSH. It
deliberately does not cover the real `hermes-install` script (no internet
Both tests boot the guest the same way and share every check that must hold
in both modes — first-boot seeding (`~/.hermes`, `~/workspace`, the
`/etc/nixos` symlink), that seeding is idempotent across a reboot (the whole
point of the `.seeded` marker), the `hermes` user/sudo setup, and SSH. That
shared node config and assertion script are defined once in
`guest/vm-tests/common.nix` and imported by both, so nothing can drift
between the two modes' common checks.
Each test then adds the checks that only make sense for its mode:
- `guest-vm-unbridged` — the isolated bridge has no DHCP server, so it
asserts the guest stands on its static config alone: exactly the one
`10.77.0.2/24` address and exactly one default route, via the host.
- `guest-vm-bridged` — a second node simulates the LAN (a dnsmasq DHCP
server, a throwaway HTTP service, and a plain device). It asserts the
guest takes a DHCP lease beside the static service-net address, keeps
both default routes (the LAN gateway at metric 100 and the host fallback
at metric 200), can reach a service on the LAN, and that a LAN device can
connect to a port the guest exposes (the guest's own firewall still
decides what answers). The host-side half of bridged mode — the bridge
itself, the mark-based rules — is covered statically by the check below.
Both deliberately do not cover the real `hermes-install` script (no internet
route inside the test VM) or `modules/host.nix`'s own `hermes-vm` QEMU
service — both stay manual, operator-run checks.
service — those stay manual, operator-run checks.
`runvmtest-debug` gives you `start_all()`, `machine.wait_for_unit(...)`, and
`machine.shell_interact()` (a live shell inside the guest) to step through a
failure by hand instead of just getting pass/fail.
**Module evaluation checks** (`tests/network-modes-check.nix`): a static
check — no build, no VM boot — that evaluates `modules/host.nix` in both
networking modes (NAT and `network.bridged = true`) plus `guest/common.nix`,
and asserts on the generated nftables rulesets, the firewall input/forward
rules, the networkd units, and the guest's DHCP/route-metric config:
```bash
nix build .#checks.x86_64-linux.network-modes
```
Catches regressions in the bridged-mode flag without booting anything. If
it fails, the assertion that broke is named in the error.
+8 -5
View File
@@ -1,22 +1,25 @@
#!/usr/bin/env nu
#
# runvmtest — build and run the guest-config VM test (guest/vm-test.nix).
# runvmtest — build and run the guest-config VM tests. The mode-independent
# assertions (seeding, idempotency, account, SSH) are shared between the two
# tests via guest/vm-tests/common.nix; each test adds its own mode's
# networking checks.
#
# Plain pass/fail: the build fails if any testScript assertion fails. For
# stepping through it interactively instead, use runvmtest-debug.
#
# --impure is required because guest/common.nix reads
# /etc/hermenix/local.nix if present, same as the real hermes-guest-image
# build — a pre-existing property of common.nix, not something this test
# introduces.
# build — a pre-existing property of common.nix, not something the tests
# introduce.
#
# Not run for you automatically: this needs a real build, which agents in
# this repo don't run (see GUIDELINES.md). You (or CI) run this by hand.
def main [
suite?: string # which check to run: guest-vm (default) or migrate-connections-vm
suite?: string # which check to run: guest-vm-unbridged (default), guest-vm-bridged, or migrate-connections-vm
] {
let check = $suite | default "guest-vm"
let check = $suite | default "guest-vm-unbridged"
let attr = $".#checks.x86_64-linux.($check)"
print $"running ($attr)"
^nix build $attr --impure -L
+4 -4
View File
@@ -1,6 +1,6 @@
#!/usr/bin/env nu
#
# runvmtest-debug — build the guest-config VM test's interactive driver and
# runvmtest-debug — build a guest-config VM test's interactive driver and
# drop into it, instead of just running it for pass/fail (see runvmtest).
#
# Opens a Python (ptpython) REPL with the test's node defined but not yet
@@ -8,16 +8,16 @@
# start_all()
# machine.wait_for_unit("multi-user.target")
# machine.shell_interact() # live shell inside the guest, over the console
# machine.succeed("...") # run any assertion from guest/vm-test.nix by hand
# machine.succeed("...") # run any assertion from the guest-vm tests by hand
#
# Same --impure requirement as runvmtest, same reason (guest/common.nix
# reads /etc/hermenix/local.nix if present). Not run for you automatically
# — needs a real build, which agents in this repo don't run.
def main [
suite?: string # which check to debug: guest-vm (default) or migrate-connections-vm
suite?: string # which check to debug: guest-vm-unbridged (default), guest-vm-bridged, or migrate-connections-vm
] {
let check = $suite | default "guest-vm"
let check = $suite | default "guest-vm-unbridged"
let attr = $".#checks.x86_64-linux.($check).driverInteractive"
print $"building interactive driver for ($attr)"
^nix build $attr --impure -L
+16 -6
View File
@@ -118,13 +118,23 @@
};
# Never run by an agent — operator/CI only:
# nix build .#checks.x86_64-linux.guest-vm --impure -L
checks.${system} = {
guest-vm = import ./guest/vm-test.nix { inherit pkgs determinate; };
migrate-connections-vm = import ./guest/migrate-connections-test.nix {
inherit pkgs;
# nix build .#checks.x86_64-linux.guest-vm-unbridged --impure -L
# nix build .#checks.x86_64-linux.guest-vm-bridged --impure -L
# nix build .#checks.x86_64-linux.migrate-connections-vm --impure -L
#
# The guest VM tests (guest/vm-tests/) share their node config and
# mode-independent assertions via guest/vm-tests/common.nix; each test
# adds the checks that only apply to its mode. default.nix aggregates
# them into this attrset, which the static network-modes check joins.
checks.${system} =
(import ./guest/vm-tests {
inherit pkgs determinate;
hermenix = self.packages.${system}.hermenix;
};
})
// {
# No build, no VM boot: evaluates host.nix in both networking modes
# and guest/common.nix, and asserts on the generated rulesets/units.
network-modes = import ./tests/network-modes-check.nix { inherit pkgs; };
};
};
}
+28 -5
View File
@@ -177,19 +177,39 @@ in
};
# =====================================================================
# Networking: static, resolvers that are not the host.
# Networking: a static service-net address for talking to the host (nix
# cache, hostAccess), plus DHCP so an operator can later switch this VM
# onto the LAN (services.hermenix.network.bridged = true on the host).
# - NAT mode: no DHCP server on the host bridge; the static default
# route via the host is the only one and carries all traffic.
# - bridged mode: the LAN's DHCP leases this NIC a real LAN address;
# that lease's lower-metric default route (100) wins over the static
# fallback (200), and LAN devices reach the VM at the lease address.
# Resolvers stay QUAD9 (UseDNS = false) in both modes.
# =====================================================================
networking.useNetworkd = true;
systemd.network.enable = true;
systemd.network.networks."10-eth" = {
matchConfig.Type = "ether";
address = [ "${guestAddress}/${toString prefixLength}" ];
routes = [ { Gateway = hostAddress; } ];
networkConfig.DNS = [
routes = [
{
Gateway = hostAddress;
Metric = 200;
}
];
networkConfig = {
DHCP = "ipv4";
DNS = [
"9.9.9.9"
"149.112.112.112"
];
};
dhcpV4Config = {
UseDNS = false;
RouteMetric = 100;
};
};
networking.nameservers = [
"9.9.9.9"
"149.112.112.112"
@@ -359,8 +379,11 @@ in
pointless; it will be replaced. Put durable notes in SOUL.md.
- The skill at ~/.hermes/skills/nixos-self-config covers how to change
this machine safely. Read it before your first rebuild.
- You have no route to the host or to any local network. Public internet
only, plus a read-only Nix binary cache on ${hostAddress}.
- In NAT mode you have no route to the host or to any local network:
public internet only, plus a read-only Nix binary cache on
${hostAddress}. In bridged mode (network.bridged = true on the host)
you take a LAN address from DHCP instead, LAN devices can reach any
port you open, and lanAccess/allowLan still limit what you can reach.
- Never put a secret in a Nix expression. Everything in the store is
world-readable. Use ${stateDir}/.secrets.env and chmod 0600.
'';
+39 -36
View File
@@ -1,42 +1,42 @@
# Guest-config tier VM test: boots guest/common.nix (the same module set
# used to build hermes-guest-image, see flake.nix) in an ephemeral,
# throwaway NixOS test VM, and checks seeding/services. This never touches
# a real disk under /var/lib/hermes-vm/** and never runs `hermenix setup`.
# Shared building blocks for the two guest-config VM tests in this
# directory: test_unbridge.nix and test_bridge.nix.
#
# guest/hardware.nix (the real disk-label file) is intentionally not
# imported here, same as hermes-guest-image's own build — nixos-generators
# and, here, the NixOS test framework both supply their own bootloader and
# root filesystem instead.
# The guest node's configuration and every assertion that must hold in both
# networking modes are defined here, once; each mode test imports this file,
# uses guestNode as its machine, and appends only the checks that apply to
# its mode (the isolated-bridge NAT setup vs. the DHCP-served LAN bridge).
#
# Both tests boot guest/common.nix — the same module set hermes-guest-image
# is built from — in an ephemeral, throwaway NixOS test VM. No real disk
# under /var/lib/hermes-vm/** is ever touched, and `hermenix setup` is never
# run. guest/hardware.nix (the real disk-label file) is intentionally not
# imported, same as hermes-guest-image's own build — nixos-generators and,
# here, the NixOS test framework both supply their own bootloader and root
# filesystem instead.
#
# Build with (never run by an agent, only by the operator/CI):
# nix build .#checks.x86_64-linux.guest-vm --impure -L
# nix build .#checks.x86_64-linux.guest-vm-unbridged --impure -L
# nix build .#checks.x86_64-linux.guest-vm-bridged --impure -L
#
# --impure is required because guest/common.nix reads /etc/hermenix/local.nix
# if present (see its own comment) — a pre-existing property of common.nix,
# not something introduced by this test.
{ pkgs, determinate }:
# not something the tests introduce.
{ determinate }:
pkgs.testers.runNixOSTest {
name = "hermenix-guest-config";
# This test boots, reboots once, and runs only quick local checks — a few
# minutes is generous. Bound it far below the framework's 1-hour default so
# a genuine hang (boot failure, a blocked prompt, a stuck reboot) fails
# fast instead of burning a full hour before being reported.
globalTimeout = 600;
nodes.machine =
{
# The guest node, exactly as both tests boot it.
guestNode =
{ lib, ... }:
{
imports = [
determinate.nixosModules.default
../guest/common.nix
../common.nix
];
# A real, separate, auto-formatted virtual disk for the state dir —
# not tmpfs. tmpfs would not survive the in-test reboot below, which
# would make the idempotency check below pass for the wrong reason
# (nothing to re-seed over, rather than correctly skipping re-seeding).
# would make the idempotency check pass for the wrong reason (nothing
# to re-seed over, rather than correctly skipping re-seeding).
virtualisation.emptyDiskImages = [ 1024 ];
fileSystems."/var/lib/hermes" = lib.mkForce {
device = "/dev/vdb";
@@ -45,6 +45,16 @@ pkgs.testers.runNixOSTest {
neededForBoot = true;
};
# The test framework's default topology is not the real guest's: it
# adds a SLiRP user-mode NIC (10.0.2.0/24, with its own DHCP server)
# alongside the shared vlan, and the guest's networkd config matches
# any ethernet NIC. Strip the user-mode NIC and keep only the shared
# vlan (which udev names eth0) so the test machine has the production
# shape: one virtual NIC, and no stray DHCP server to confuse the
# mode-specific networking assertions.
virtualisation.qemu.networkingOptions = lib.mkForce [ ];
virtualisation.vlans = [ 1 ];
# nixosTest VMs have no route to the internet by design, so the real
# hermes-install body (a third-party curl | bash installer) can't run
# hermetically here. Stubbed to just flag itself done; the real
@@ -54,7 +64,9 @@ pkgs.testers.runNixOSTest {
'';
};
testScript = ''
# Assertions that hold in both modes: first-boot seeding, seeding
# idempotency across a reboot, the agent's account, and SSH.
commonScript = ''
# allow_reboot=True: without it, the VM starts with qemu's -no-reboot,
# so the guest-initiated machine.reboot() below would just kill the VM
# instead of restarting it, and the backdoor shell would never
@@ -84,18 +96,9 @@ pkgs.testers.runNixOSTest {
machine.succeed("groups hermes | grep -q wheel")
machine.succeed("su - hermes -c 'sudo -n true'")
# SSH is up.
# SSH is up (checked over the test backdoor, so it is independent of
# whatever address configuration the mode under test produces).
machine.wait_for_unit("sshd.service")
machine.wait_for_open_port(22)
# NOTE: not checking /etc/nix/nix.custom.conf's content here. `nix eval`
# on environment.etc."nix/nix.custom.conf".text confirms guest/common.nix
# itself renders the expected (no-host-cache) content, but the actual
# booted guest's copy of that file instead holds the NixOS test
# framework's own injected nix.settings (substituters = cache.nixos.org,
# trusted-users = root, system-features = nixos-test ...) — needed so
# nested Nix operations inside the test VM work at all. That's an
# artifact of running under nixosTest, not something guest/common.nix or
# this test controls, so it's not a reliable thing to assert on here.
'';
}
+27
View File
@@ -0,0 +1,27 @@
# guest/vm-tests/default.nix — the granular guest VM test suite.
#
# Aggregates the three runNixOSTest checks into a single import surface for
# flake.nix's checks.${system}. common.nix holds the guest node config and
# the mode-independent assertions shared by the two host-networking-mode
# tests; test_unbridge.nix / test_bridge.nix add each mode's own checks, and
# test_migrate_connections.nix covers `hermenix migrate connections`.
#
# Run by the operator/CI, never by an agent (see GUIDELINES.md):
# nix build .#checks.x86_64-linux.guest-vm-unbridged --impure -L
# nix build .#checks.x86_64-linux.guest-vm-bridged --impure -L
# nix build .#checks.x86_64-linux.migrate-connections-vm --impure -L
#
# --impure is required because guest/common.nix reads
# /etc/hermenix/local.nix if present (see its own comment) — a pre-existing
# property of common.nix, not something the tests introduce.
{
pkgs,
determinate,
hermenix,
}:
{
guest-vm-unbridged = import ./test_unbridge.nix { inherit pkgs determinate; };
guest-vm-bridged = import ./test_bridge.nix { inherit pkgs determinate; };
migrate-connections-vm = import ./test_migrate_connections.nix { inherit pkgs hermenix; };
}
+124
View File
@@ -0,0 +1,124 @@
# Guest-config VM test, bridged mode: the guest on a LAN served by DHCP,
# with a second node acting as the LAN (a DHCP server, a throwaway HTTP
# service, and a plain device that connects back to the guest). This is the
# guest-facing half of services.hermenix.network.bridged = true: the module's
# own bridge/enforcement rules are covered statically by
# tests/network-modes-check.nix.
#
# The guest node and the mode-independent assertions (seeding, idempotency,
# account, SSH) are shared with guest-vm-unbridged via
# common.nix.
#
# Build with (never run by an agent, only by the operator/CI):
# nix build .#checks.x86_64-linux.guest-vm-bridged --impure -L
{ pkgs, determinate }:
let
common = import ./common.nix { inherit determinate; };
in
pkgs.testers.runNixOSTest {
name = "hermenix-guest-config-bridged";
# Boots once, reboots once, and waits on DHCP lease negotiation plus a few
# cross-node checks - still only minutes. Bound it far below the
# framework's 1-hour default so a genuine hang fails fast.
globalTimeout = 900;
nodes = {
machine = common.guestNode;
# The "LAN": serves DHCP to the guest and hosts a throwaway HTTP
# service, standing in for the LAN router and another LAN device. Its
# firewall is off so the guest<->LAN checks exercise the network path
# itself, not this throwaway node's own firewall.
lan =
{ pkgs, lib, ... }:
{
networking.firewall.enable = false;
# Same production-shape networking as the guest node: kill the test
# framework's SLiRP user-mode NIC so the only NIC left is the one
# shared with the guest.
virtualisation.qemu.networkingOptions = lib.mkForce [ ];
virtualisation.vlans = [ 1 ];
# The framework's own name-based static assignment missed the
# shared NIC when this suite actually ran (the NIC came up as eth1,
# not the eth0 the framework expected), and real interface names
# vary anyway - and node numbering is attr order, which is not
# something to trust. The shared NIC's MAC is deterministic
# (52:54:00:12:<vlan>:<node>, see qemu-common.nix), so pin the
# address in networkd by MAC. Two candidate rules cover either node
# numbering; the one that matches this node's actual NIC wins, and
# nothing depends on an interface name.
networking.useNetworkd = true;
systemd.network.networks."10-hermes-lan-1" = {
matchConfig.MACAddress = "52:54:00:12:01:01"; # vlan 1, node #1
address = [ "192.168.1.2/24" ];
networkConfig.LinkLocalAddressing = "no";
};
systemd.network.networks."10-hermes-lan-2" = {
matchConfig.MACAddress = "52:54:00:12:01:02"; # vlan 1, node #2
address = [ "192.168.1.2/24" ];
networkConfig.LinkLocalAddressing = "no";
};
# DHCP only (port = 0 disables dnsmasq's DNS half; the guest keeps
# its own QUAD9 resolvers regardless). Bound to the node's static
# vlan address rather than an interface name - the framework's
# interface naming is not something this test should depend on.
services.dnsmasq = {
enable = true;
settings = {
port = 0;
listen-address = "192.168.1.2";
bind-interfaces = true;
dhcp-range = "192.168.1.100,192.168.1.200,12h";
dhcp-option = [
"3,192.168.1.2" # default gateway -> this node
"6,192.168.1.2" # DNS -> this node
];
};
};
environment.systemPackages = [ pkgs.python3 ];
};
};
testScript = ''
# Bring the LAN up early so the guest's DHCP client (which retries for
# as long as it runs) can get a lease as soon as the guest is online.
lan.start()
''
+ common.commonScript
+ ''
# --- bridged-mode networking ---------------------------------------
# The LAN's DHCP gave the guest a lease alongside its static
# service-net address. (Both test nodes share one vlan, standing in for
# the bridged uplink on a real host.)
machine.wait_until_succeeds("ip -4 route show default | grep -q 'metric 100'")
addrs = machine.succeed("ip -4 -o addr show | grep -v '^1: lo'").strip().splitlines()
guest_lan_ip = None
for line in addrs:
ip = line.split()[3].split("/")[0]
if not ip.startswith("10.77."):
guest_lan_ip = ip
assert any("10.77.0.2/24" in line for line in addrs), f"static service-net address missing in bridged mode: {addrs}"
assert guest_lan_ip is not None, f"guest never took a LAN DHCP lease: {addrs}"
# Both default routes are in place: the LAN's (metric 100, in use) and
# the host service-net fallback (metric 200) that NAT mode relies on.
machine.succeed("ip -4 route show default | grep -q 'default via 192.168.1.2'")
machine.succeed("ip -4 route show default | grep -q 'default via 10.77.0.1'")
# Guest -> LAN: the guest reaches a service on the LAN node.
lan.succeed("systemd-run --collect -u lanhttpd python3 -m http.server 8000 --bind 0.0.0.0")
machine.wait_until_succeeds("curl -fsS http://192.168.1.2:8000/ >/dev/null")
# LAN -> guest: a plain device on the LAN can connect to a port the
# guest exposes - the guest's own firewall still decides what answers
# (here, sshd on 22).
lan.wait_until_succeeds(f"bash -c 'exec 3<>/dev/tcp/{guest_lan_ip}/22'")
'';
}
+42
View File
@@ -0,0 +1,42 @@
# Guest-config VM test, NAT (unbridged) mode: guest/common.nix as it sits on
# the host's isolated bridge, where there is no DHCP server and the guest
# must be fully served by its static service-net configuration.
#
# The guest node and the mode-independent assertions (seeding, idempotency,
# account, SSH) are shared with guest-vm-bridged via common.nix;
# the networking assertions here only make sense in this mode.
#
# Build with (never run by an agent, only by the operator/CI):
# nix build .#checks.x86_64-linux.guest-vm-unbridged --impure -L
{ pkgs, determinate }:
let
common = import ./common.nix { inherit determinate; };
in
pkgs.testers.runNixOSTest {
name = "hermenix-guest-config-unbridged";
# Boots once, reboots once, and runs only quick local checks — a few
# minutes is generous. Bound it far below the framework's 1-hour default so
# a genuine hang (boot failure, a blocked prompt, a stuck reboot) fails
# fast instead of burning a full hour before being reported.
globalTimeout = 600;
nodes.machine = common.guestNode;
testScript = common.commonScript + ''
# --- NAT-mode networking -------------------------------------------
# The isolated vlan has no DHCP server, so the guest's NIC must be
# self-sufficient: exactly the one static service-net address, and
# exactly one default route - via the host - because the DHCP client
# found nobody to talk to and contributed nothing. The interface name
# is deliberately not hard-coded (the real guest's NIC name varies too;
# only the address layout is asserted).
addrs = machine.succeed("ip -4 -o addr show | grep -v '^1: lo'").strip().splitlines()
assert len(addrs) == 1, f"expected only the static service-net address, got: {addrs}"
assert "10.77.0.2/24" in addrs[0], f"static service-net address missing: {addrs}"
machine.succeed("test \"$(ip -4 route show default | wc -l)\" -eq 1")
machine.succeed("ip -4 route show default | grep -q 'default via 10.77.0.1'")
'';
}
+136 -4
View File
@@ -36,6 +36,13 @@ let
anyUsb = pt.usb != [ ];
earlyIds = lib.filter (i: i != null) (map (d: d.id) pt.pci);
# Netfilter mark tagging guest-originated traffic in bridged mode. The
# guest's only way in is the tap interface, so marking at the bridge
# prerouting hook is as specific as an address match - and it is the only
# way to identify the guest at the ip-family layer, where its DHCP-issued
# LAN address is not statically known. 18501 == 0x4845 ("HE"), for hermenix.
bridgeMark = 18501;
# Graceful ACPI power button via QMP, then wait for QEMU to exit.
stopScript = pkgs.writeShellScript "hermes-vm-stop" ''
set -eu
@@ -330,6 +337,26 @@ in
type = types.int;
default = 24;
};
bridged = mkOption {
type = types.bool;
default = false;
description = ''
Put the guest directly on your LAN instead of behind the host's
NAT. The bridge absorbs `externalInterface`, and the guest takes
an address from your LAN's DHCP, so any device on the LAN can
reach any port the guest exposes.
Inbound is no longer limited to `publish` (which is ignored in
this mode): the LAN may reach every guest port. Outbound is still
governed by `lanAccess` / `allowLan`, and `hostAccess` plus the
host cache still work over the host-guest service net
(`hostAddress` stays configured on the bridge).
Requires a working DHCP server on the LAN, and a guest that runs
a DHCP client on its NIC (see guest/common.nix - this repo's
guest does).
'';
};
externalInterface = mkOption {
type = types.nullOr types.str;
default = null;
@@ -734,11 +761,15 @@ in
systemd.network.networks."10-${net.bridge}" = {
matchConfig.Name = net.bridge;
# hostAddress stays on the bridge in both modes: it is the guest's
# service net (host cache, hostAccess). In bridged mode the LAN DHCP
# lease is added on top of it.
address = [ "${net.hostAddress}/${toString net.prefixLength}" ];
networkConfig = {
ConfigureWithoutCarrier = true;
IPv4Forwarding = true;
};
}
// lib.optionalAttrs net.bridged { DHCP = "ipv4"; };
};
systemd.network.networks."11-${net.tapId}" = {
@@ -746,6 +777,17 @@ in
networkConfig.Bridge = net.bridge;
};
# Bridged mode: the physical uplink becomes a bridge port so the LAN
# shares an L2 domain with the guest and the guest's DHCP is answered
# by the LAN itself. The port must not carry its own address config -
# the bridge owns the LAN address now. Note the bridge takes a fresh
# DHCP lease, and because its MAC can differ from the uplink's, the
# host's LAN IP may change after the first switch to bridged mode.
systemd.network.networks."12-${net.externalInterface}" = lib.mkIf net.bridged {
matchConfig.Name = net.externalInterface;
networkConfig.Bridge = net.bridge;
};
# ip_forward: use the same sysctl keys networking.nat used (mkOverride 99
# so they win over any lower-priority defaults elsewhere in the config).
boot.kernel.sysctl."net.ipv4.conf.all.forwarding" = lib.mkOverride 99 true;
@@ -764,7 +806,24 @@ in
# while covering every legitimate external path.
networking.nftables.tables.hermenix-nat = lib.mkIf (net.externalInterface != null) {
family = "ip";
content = ''
content =
if net.bridged then
# Bridged mode: no DNAT (the guest is directly reachable on the
# LAN) and no primary masquerade. Keep one scoped masquerade as a
# fallback: if the guest's DHCP lease lapses it falls back to its
# static default route via the host, and this keeps that path
# working instead of blackholing at the uplink. With a healthy
# lease the guest's traffic is L2-bridged and never reaches this
# chain. Only traffic handed up from the bridge (i.e. the guest's
# service-net traffic) is matched, never the LAN itself.
''
chain postrouting {
type nat hook postrouting priority srcnat;
iifname "${net.bridge}" oifname "${net.externalInterface}" masquerade
}
''
else
''
chain prerouting {
type nat hook prerouting priority dstnat;
''
@@ -780,6 +839,56 @@ in
'';
};
# =====================================================================
# Bridged mode: bridge-family enforcement.
#
# With the uplink bridged, guest <-> LAN traffic is switched at L2 and
# never traverses the ip-family hooks the NAT-mode rules rely on, so
# bridged mode gets its own table where the tap port is unambiguous.
# lanAccess / allowLan still gate what the guest may reach; inbound
# from the LAN to the guest is open on every port.
# =====================================================================
networking.nftables.tables."hermenix-bridge" = lib.mkIf net.bridged {
family = "bridge";
content = ''
chain prerouting {
type filter hook prerouting priority filter; policy accept;
# Tag anything entering from the guest. ip-family input rules
# (host cache, hostAccess) match this mark.
iifname "${net.tapId}" meta mark set ${toString bridgeMark}
}
chain forward {
type filter hook forward priority filter; policy accept;
# Inbound from the LAN to the guest: any device, any port.
oifname "${net.tapId}" accept
''
+ lib.concatMapStrings (s: ''
iifname "${net.tapId}" ip daddr ${s.address} ${s.proto} dport ${toString s.port} accept
'') cfg.lanAccess
+ (
if cfg.allowLan then
''
iifname "${net.tapId}" accept
''
else
''
iifname "${net.tapId}" ip daddr {
10.0.0.0/8,
172.16.0.0/12,
192.168.0.0/16,
169.254.0.0/16,
127.0.0.0/8,
100.64.0.0/10
} drop
iifname "${net.tapId}" ip6 daddr { fc00::/7, fe80::/10, ::1/128 } drop
iifname "${net.tapId}" accept
''
)
+ ''
}
'';
};
# =====================================================================
# Firewall
#
@@ -792,7 +901,23 @@ in
# The deliberate guest -> host holes: the Nix cache, plus anything listed
# in hostAccess.
#
# NAT mode: the guest always arrives from its fixed service-net address,
# so match address + bridge. Bridged mode: the guest's LAN address comes
# from DHCP and is not known here, so the bridge table above tags guest
# traffic with a netfilter mark as it enters the tap - the rules match
# that mark instead. The tap is exclusive to the guest, so the mark is
# as specific as an address match: LAN devices can never inherit these
# holes.
networking.firewall.extraInputRules =
if net.bridged then
optionalString cfg.hostCache.enable ''
meta mark ${toString bridgeMark} tcp dport ${toString cfg.hostCache.port} accept
''
+ lib.concatMapStrings (s: ''
meta mark ${toString bridgeMark} ${s.proto} dport ${toString s.port} accept
'') cfg.hostAccess
else
optionalString cfg.hostCache.enable ''
iifname "${net.bridge}" ip saddr ${net.guestAddress} tcp dport ${toString cfg.hostCache.port} accept
''
@@ -801,6 +926,10 @@ in
'') cfg.hostAccess;
# Order matters here. Allowlists must precede the RFC1918 drop.
#
# These ip-family rules only ever see routed traffic. They are the whole
# story in NAT mode; in bridged mode the guest's LAN traffic is
# L2-bridged instead and is governed by the hermenix-bridge table.
networking.firewall.extraForwardRules = ''
iifname != "${net.bridge}" accept
ct state established,related accept
@@ -809,10 +938,13 @@ in
+ lib.concatMapStrings (s: ''
iifname "${net.bridge}" ip daddr ${s.address} ${s.proto} dport ${toString s.port} accept
'') cfg.lanAccess
# Inbound to guest services published on the uplink.
+ lib.concatMapStrings (s: ''
# Inbound to guest services published on the uplink. NAT mode only; in
# bridged mode the bridge table opens every inbound port instead.
+ lib.optionalString (!net.bridged) (
lib.concatMapStrings (s: ''
oifname "${net.bridge}" ip daddr ${net.guestAddress} ${s.proto} dport ${toString s.guestPort} accept
'') cfg.publish
)
+ (
if cfg.allowLan then
''
+2
View File
@@ -375,6 +375,8 @@ def "main setup" [
\};
"
note $"detected uplink: ($uplink) — check that's right before rebuilding"
note "network.bridged = true instead puts the guest on your LAN (own DHCP"
note "address, every opened port reachable from the LAN) - see the README."
print $"
Then:
+215
View File
@@ -0,0 +1,215 @@
# Static evaluation check for the host module's two networking modes (NAT
# and bridged) and the guest's mode-agnostic NIC configuration.
#
# Nothing here is built or booted: the check evaluates services.hermenix
# (modules/host.nix) in both modes plus guest/common.nix, and asserts on the
# networking configuration the modules produce - the nftables rulesets, the
# ip-family input/forward rules, the systemd-networkd units, and the guest's
# DHCP / route-metric setup. Any failing assertion fails the check listing
# what broke.
#
# Run by the operator/CI, never by an agent:
# nix build .#checks.x86_64-linux.network-modes
#
# The host evaluations use minimal stub options for the NixOS modules the
# host module's config feeds (networking.nftables, networking.firewall,
# systemd.network) because this check only reads those outputs back; every
# other option host.nix writes is out of scope and passed through with
# _module.check = false. The guest evaluation is different: guest/common.nix
# runs against the full NixOS option set (module-list.nix) because its real
# imports (qemu-guest profile and friends) need it.
{ pkgs }:
let
inherit (pkgs) lib;
nixpkgsPath = pkgs.path;
has = lib.strings.hasInfix;
# ------------------------------------------------------------------ #
# Host module, evaluated with stub options for what this check reads. #
# ------------------------------------------------------------------ #
hostOptions = { lib, ... }: {
config._module.check = false;
options = {
networking = {
useNetworkd = lib.mkOption {
type = lib.types.bool;
default = false;
};
nftables = {
enable = lib.mkOption {
type = lib.types.bool;
default = false;
};
tables = lib.mkOption {
type = lib.types.attrsOf lib.types.anything;
default = { };
};
};
firewall = {
enable = lib.mkOption {
type = lib.types.bool;
default = true;
};
filterForward = lib.mkOption {
type = lib.types.bool;
default = false;
};
extraInputRules = lib.mkOption {
type = lib.types.str;
default = "";
};
extraForwardRules = lib.mkOption {
type = lib.types.str;
default = "";
};
};
};
systemd.network = {
networks = lib.mkOption {
type = lib.types.attrsOf lib.types.anything;
default = { };
};
netdevs = lib.mkOption {
type = lib.types.attrsOf lib.types.anything;
default = { };
};
};
};
};
# The one scenario every assertion is made against.
evalHost =
bridged:
lib.evalModules {
modules = [
hostOptions
../modules/host.nix
{
services.hermenix = {
enable = true;
network = {
externalInterface = "eno1";
inherit bridged;
};
allowLan = false;
publish = [
{
hostPort = 8080;
guestPort = 8080;
}
];
lanAccess = [
{
address = "192.168.1.50";
port = 11434;
}
];
hostAccess = [ { port = 8188; } ];
};
}
];
specialArgs = { inherit pkgs; };
};
nat = evalHost false;
br = evalHost true;
natNft = nat.config.networking.nftables.tables;
brNft = br.config.networking.nftables.tables;
natNat = natNft."hermenix-nat" or { };
brNat = brNft."hermenix-nat" or { };
brBr = brNft."hermenix-bridge" or { };
natNetworks = nat.config.systemd.network.networks;
brNetworks = br.config.systemd.network.networks;
natInput = nat.config.networking.firewall.extraInputRules;
natForward = nat.config.networking.firewall.extraForwardRules;
brInput = br.config.networking.firewall.extraInputRules;
brForward = br.config.networking.firewall.extraForwardRules;
hostResults = {
# --- NAT mode: the original behaviour has to be unchanged ----------
"nat: no bridged-mode bridge table" = !(natNft ? "hermenix-bridge");
"nat: publish is DNATed to the guest" = has "dnat to 10.77.0.2:8080" natNat.content;
"nat: outbound masquerades on the uplink" = has "masquerade" natNat.content;
"nat: input rules match the fixed service-net address" =
has "ip saddr 10.77.0.2 tcp dport 5000 accept" natInput && !(has "meta mark" natInput);
"nat: published port has a forward accept" =
has "ip daddr 10.77.0.2 tcp dport 8080 accept" natForward;
"nat: bridge keeps the static service-net address, no LAN DHCP" =
natNetworks."10-hermes-br".address == [ "10.77.0.1/24" ]
&& (natNetworks."10-hermes-br".networkConfig.DHCP or null) == null;
"nat: uplink is not a bridge port" = !(natNetworks ? "12-eno1");
"nat: tap is a bridge port" = natNetworks."11-vm-hermes".networkConfig.Bridge == "hermes-br";
"nat: bridge netdev exists" =
nat.config.systemd.network.netdevs."10-hermes-br".netdevConfig.Kind == "bridge";
# --- bridged mode: the guest is directly on the LAN ----------------
"bridged: bridge-family table exists" = brNft ? "hermenix-bridge" && brBr.family == "bridge";
"bridged: guest traffic is marked as it enters the tap" = has "meta mark set 18501" brBr.content;
"bridged: inbound from the LAN is open on every port" =
has "oifname \"vm-hermes\" accept" brBr.content;
"bridged: lanAccess still gates guest egress" =
has "ip daddr 192.168.1.50 tcp dport 11434 accept" brBr.content;
"bridged: private-space egress is dropped unless allowlisted" =
has "192.168.0.0/16" brBr.content && has "fc00::/7" brBr.content;
"bridged: no DNAT for publish" =
!(has "dnat to" brNat.content) && !(has "chain prerouting" brNat.content);
"bridged: scoped fallback masquerade remains" = has "masquerade" brNat.content;
"bridged: input rules match the mark, not the unknown DHCP address" =
has "meta mark 18501 tcp dport 5000 accept" brInput
&& has "meta mark 18501 tcp dport 8188 accept" brInput
&& !(has "ip saddr 10.77.0.2" brInput);
"bridged: publish forward rules are dropped" =
!(has "ip daddr 10.77.0.2 tcp dport 8080 accept" brForward);
"bridged: bridge keeps the service net and takes LAN DHCP" =
brNetworks."10-hermes-br".address == [ "10.77.0.1/24" ]
&& brNetworks."10-hermes-br".networkConfig.DHCP == "ipv4";
"bridged: uplink is a bridge port" = brNetworks."12-eno1".networkConfig.Bridge == "hermes-br";
"bridged: tap is still a bridge port" =
brNetworks."11-vm-hermes".networkConfig.Bridge == "hermes-br";
};
# ------------------------------------------------------------------- #
# Guest module: must produce one NIC config that works in both modes. #
# Evaluated against the full NixOS option set. #
# ------------------------------------------------------------------- #
guest = lib.evalModules {
modules = (import "${nixpkgsPath}/nixos/modules/module-list.nix") ++ [
{ config._module.check = false; }
../guest/common.nix
];
specialArgs = {
inherit pkgs;
modulesPath = "${nixpkgsPath}/nixos/modules";
};
};
gEth = guest.config.systemd.network.networks."10-eth";
gRoute = builtins.elemAt gEth.routes 0;
guestResults = {
"guest: NIC asks for a DHCP lease (bridged mode)" = gEth.networkConfig.DHCP == "ipv4";
"guest: service-net address stays static (both modes)" = gEth.address == [ "10.77.0.2/24" ];
"guest: static default route via the host is the NAT fallback" =
gRoute.Gateway == "10.77.0.1" && gRoute.Metric == 200;
"guest: DHCP default route wins (metric 100)" = gEth.dhcpV4Config.RouteMetric == 100;
"guest: resolvers stay QUAD9, DHCP DNS is ignored" =
gEth.dhcpV4Config.UseDNS == false
&&
gEth.networkConfig.DNS == [
"9.9.9.9"
"149.112.112.112"
];
};
results = hostResults // guestResults;
failing = builtins.filter (n: !results.${n}) (builtins.attrNames results);
in
assert lib.assertMsg (failing == [ ]) ''
hermenix network-modes check FAILED:
${lib.concatMapStringsSep "\n" (n: " - " + n) failing}
'';
pkgs.runCommand "hermenix-network-modes-check" { } "touch $out"