safer bridge mode
This commit is contained in:
1 parent
b5d59c34ac
commit
d5afd6510f
7 files changed
+337
-120
No files matched your search
@@ -483,89 +483,86 @@ auth in front of anything that isn't a toy, and keep the list short.
|
|||||||
|
|
||||||
## Putting the guest directly on your LAN
|
## Putting the guest directly on your LAN
|
||||||
|
|
||||||
The default setup puts the guest on an isolated bridge behind the host's NAT,
|
The default setup keeps the guest on an isolated bridge behind the host's
|
||||||
reachable from outside only through `publish`. This section is for when you
|
NAT, reachable from outside only through `publish`. This section is for
|
||||||
want the VM to be a first-class device on your LAN instead: its own address
|
when you want the VM to be a first-class device on your LAN instead: its
|
||||||
from the LAN's DHCP, reachable on any port from any device on the network.
|
own fixed address on the network, reachable on any port from any device,
|
||||||
|
while keeping everything it can see outbound locked down.
|
||||||
|
|
||||||
Enable it:
|
Enable it and tell it your LAN's numbers:
|
||||||
|
|
||||||
```nix
|
```nix
|
||||||
services.hermenix.network.bridged = true;
|
services.hermenix.network = {
|
||||||
```
|
bridged = true;
|
||||||
|
externalInterface = "enp191s0"; # your uplink (required)
|
||||||
|
|
||||||
In one sentence, what that does: the bridge absorbs your uplink
|
# Static LAN addressing: bridged mode does not use your LAN's DHCP.
|
||||||
(`network.externalInterface`), the guest and the host share the LAN, the
|
lanAddress = "10.0.12.50"; # the guest's fixed LAN address
|
||||||
guest takes a DHCP lease, and inbound traffic to the guest is open on every
|
hostLanAddress = "10.0.12.10"; # this host's own LAN address
|
||||||
port. Outbound is still gated by `lanAccess` / `allowLan`, and `hostAccess`
|
lanGateway = "10.0.12.1"; # your router
|
||||||
plus the host Nix cache still work over the service net at `10.77.0.1`.
|
lanPrefixLength = 24; # default 24
|
||||||
|
# lanDns = [ "1.1.1.1" ]; # optional; defaults to lanGateway
|
||||||
### 1. Ready the guest first (one-time, if the image is older than this repo)
|
|
||||||
|
|
||||||
The guest keeps a static service-net address (`10.77.0.2/24`) for talking to
|
|
||||||
the host and adds DHCP on the same NIC, so one config works in both modes.
|
|
||||||
That is already what `guest/common.nix` does; if your guest image predates
|
|
||||||
it, apply the equivalent change to `/etc/nixos/common.nix` inside the VM:
|
|
||||||
|
|
||||||
```nix
|
|
||||||
systemd.network.networks."10-eth" = {
|
|
||||||
matchConfig.Type = "ether";
|
|
||||||
address = [ "10.77.0.2/24" ]; # service net to the host
|
|
||||||
routes = [ { Gateway = "10.77.0.1"; Metric = 200; } ]; # NAT-mode fallback
|
|
||||||
networkConfig.DHCP = "ipv4"; # LAN lease in bridged mode
|
|
||||||
dhcpV4Config = { # DHCP default route wins (100)
|
|
||||||
UseDNS = false;
|
|
||||||
RouteMetric = 100;
|
|
||||||
};
|
|
||||||
};
|
};
|
||||||
```
|
```
|
||||||
|
|
||||||
`cd /etc/nixos && git apply -p1 < patch`, commit, and
|
What that gives you: the bridge absorbs the uplink, the host and the guest
|
||||||
`sudo nixos-rebuild switch --flake /etc/nixos#hermes`. In NAT mode the DHCP
|
share the LAN, and each side has a fixed address. The guest never needs to
|
||||||
attempt fails quietly (there is no server on the isolated bridge), so check
|
know its own address - the module runs a small DHCP server on the bridge
|
||||||
nothing broke: `ip -4 addr` still shows `10.77.0.2` and `ip route` still
|
that reserves `lanAddress` for the guest's MAC, so the VM stays a plain
|
||||||
goes via `10.77.0.1`.
|
DHCP client and just receives its address. This host keeps its LAN identity
|
||||||
|
too: the bridge clones your uplink's MAC, so the LAN sees this host under
|
||||||
|
its usual MAC (no duplicate-address surprises), and the guest's DHCP
|
||||||
|
broadcast never leaves the bridge, so nothing on your LAN can race the
|
||||||
|
reservation.
|
||||||
|
|
||||||
### 2. Rebuild the host with the toggle on
|
### What is and isn't reachable
|
||||||
|
|
||||||
Set `services.hermenix.network.bridged = true` alongside
|
- **Inbound to the guest, from anywhere on the LAN: every port** is open.
|
||||||
`network.externalInterface`, then `sudo nixos-rebuild switch`. On rebuild
|
The guest's own firewall decides what answers - by default it only opens
|
||||||
the host's uplink becomes a bridge port, the host takes a fresh DHCP lease
|
port 22, so the agent opens whatever it wants LAN-reachable.
|
||||||
on the bridge, and the guest's DHCP is answered by the LAN router. The
|
- **Guest egress is still gated**, exactly as in NAT mode: `lanAccess`
|
||||||
network drops briefly while the host reconfigures - plan for it.
|
entries and `allowLan` are unchanged. The VM sees the LAN, but only the
|
||||||
|
parts you have allowed.
|
||||||
|
- **Guest to host still works**: `hostAccess` and the host Nix cache are
|
||||||
|
reachable over the service net at `10.77.0.1` in both modes.
|
||||||
|
- **`publish` is ignored** in this mode - every inbound port is already
|
||||||
|
reachable. Drop the entries if you come back to NAT mode.
|
||||||
|
|
||||||
### 3. Verify
|
### Switching over
|
||||||
|
|
||||||
From the host, `hermenix vm ssh`. Inside the guest, `ip -4 addr` shows the
|
1. Pick three free addresses: the guest (`lanAddress`), this host
|
||||||
service net _and_ a LAN address, and `ip route` shows a default route via
|
(`hostLanAddress` - use the address it already has on that interface,
|
||||||
your LAN gateway. From any other device on the LAN:
|
e.g. a router reservation, so nothing else on the LAN breaks), and the
|
||||||
|
router itself (`lanGateway`). Keep `hostLanAddress` and `lanAddress`
|
||||||
|
outside any DHCP pool, or reserve them in the router first.
|
||||||
|
2. Set `network.bridged = true` with the options above, then
|
||||||
|
`sudo nixos-rebuild switch`. The network drops briefly while the host
|
||||||
|
reconfigures - plan for it.
|
||||||
|
3. Verify from the host: `hermenix vm ssh`; inside the guest,
|
||||||
|
`ip -4 addr` shows the service net (`10.77.0.2`) **and** its fixed LAN
|
||||||
|
address. From any other device on the LAN:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh hermes@<guest-lan-ip> # and any other port the guest opens
|
ssh hermes@<guest-lan-address> # and any other port the guest opens
|
||||||
```
|
```
|
||||||
|
|
||||||
The guest's own firewall decides what answers - by default it only opens
|
|
||||||
port 22, so the agent must open whatever it wants LAN-reachable. Egress is
|
|
||||||
still gated too: from the guest, `192.168.1.50:11434` only works if it is
|
|
||||||
in `lanAccess` (or `allowLan = true`).
|
|
||||||
|
|
||||||
### Gotchas
|
|
||||||
|
|
||||||
- The LAN needs a working DHCP server.
|
|
||||||
- `publish` does nothing in this mode - every port is reachable, so its
|
|
||||||
entries are ignored. Drop them if you come back to NAT mode.
|
|
||||||
- The host's LAN IP comes from a fresh DHCP lease on the bridge, and the
|
|
||||||
bridge MAC can differ from the uplink's - the host's address may change
|
|
||||||
after the switch. Reserve it in the DHCP server if that matters.
|
|
||||||
- Any network config you had for the uplink interface is replaced by its
|
|
||||||
bridge-port role while bridged (this module's networkd unit for it takes
|
|
||||||
priority over most others).
|
|
||||||
|
|
||||||
### Switching back
|
### Switching back
|
||||||
|
|
||||||
Set `bridged = false` and rebuild the host. The guest's DHCP client finds
|
Set `bridged = false` (the `lan*` options are ignored) and rebuild the
|
||||||
no server on the isolated bridge and settles onto its static route via the
|
host. The guest's DHCP client finds no server on the isolated bridge and
|
||||||
host again; no guest-side change needed.
|
settles onto its static route via the host again; no guest-side change
|
||||||
|
needed.
|
||||||
|
|
||||||
|
### Gotchas
|
||||||
|
|
||||||
|
- `hostLanAddress` is a static claim on the LAN - pick one your router
|
||||||
|
won't hand out, or reserve it.
|
||||||
|
- Any network config you had for the uplink interface is replaced by its
|
||||||
|
bridge-port role while bridged (this module's networkd unit for it takes
|
||||||
|
priority over most others).
|
||||||
|
- An older guest image that predates the DHCP-on-NIC setup in the current
|
||||||
|
`guest/common.nix` won't take an address in bridged mode; upgrade the
|
||||||
|
image or apply that config inside the VM first.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
+5
-3
@@ -182,9 +182,11 @@ in
|
|||||||
# onto the LAN (services.hermenix.network.bridged = true on the host).
|
# onto the LAN (services.hermenix.network.bridged = true on the host).
|
||||||
# - NAT mode: no DHCP server on the host bridge; the static default
|
# - NAT mode: no DHCP server on the host bridge; the static default
|
||||||
# route via the host is the only one and carries all traffic.
|
# route via the host is the only one and carries all traffic.
|
||||||
# - bridged mode: the LAN's DHCP leases this NIC a real LAN address;
|
# - bridged mode: the host runs a DHCP server on its bridge that hands
|
||||||
# that lease's lower-metric default route (100) wins over the static
|
# this NIC its fixed LAN address (services.hermenix.network.lanAddress,
|
||||||
# fallback (200), and LAN devices reach the VM at the lease address.
|
# keyed to this NIC's MAC); that lease's lower-metric default route
|
||||||
|
# (100) wins over the static fallback (200), and LAN devices reach
|
||||||
|
# the VM at the fixed address. The VM never needs to know the address.
|
||||||
# Resolvers stay QUAD9 (UseDNS = false) in both modes.
|
# Resolvers stay QUAD9 (UseDNS = false) in both modes.
|
||||||
# =====================================================================
|
# =====================================================================
|
||||||
networking.useNetworkd = true;
|
networking.useNetworkd = true;
|
||||||
|
|||||||
@@ -25,9 +25,10 @@ changes persist across reboots.
|
|||||||
the cheapest undo available to you, and future-you will want it.
|
the cheapest undo available to you, and future-you will want it.
|
||||||
- If you break the system badly enough that you cannot fix it, the operator
|
- If you break the system badly enough that you cannot fix it, the operator
|
||||||
has snapshots of the state disk. Say so plainly rather than thrashing.
|
has snapshots of the state disk. Say so plainly rather than thrashing.
|
||||||
- You have no route to the host or the local network unless the operator has
|
- Reachability is gated on the host: what you can reach from here is
|
||||||
opened a specific port. If something is unreachable, that is probably
|
allowlisted (lanAccess/allowLan), in both network modes. If something is
|
||||||
deliberate — ask rather than working around it.
|
unreachable, that is probably deliberate — ask rather than working around
|
||||||
|
it.
|
||||||
|
|
||||||
## What I care about
|
## What I care about
|
||||||
|
|
||||||
|
|||||||
@@ -4,7 +4,7 @@
|
|||||||
# The guest node's configuration and every assertion that must hold in both
|
# The guest node's configuration and every assertion that must hold in both
|
||||||
# networking modes are defined here, once; each mode test imports this file,
|
# networking modes are defined here, once; each mode test imports this file,
|
||||||
# uses guestNode as its machine, and appends only the checks that apply to
|
# uses guestNode as its machine, and appends only the checks that apply to
|
||||||
# its mode (the isolated-bridge NAT setup vs. the DHCP-served LAN bridge).
|
# its mode (the isolated-bridge NAT setup vs. the fixed-address LAN bridge).
|
||||||
#
|
#
|
||||||
# Both tests boot guest/common.nix — the same module set hermes-guest-image
|
# Both tests boot guest/common.nix — the same module set hermes-guest-image
|
||||||
# is built from — in an ephemeral, throwaway NixOS test VM. No real disk
|
# is built from — in an ephemeral, throwaway NixOS test VM. No real disk
|
||||||
|
|||||||
@@ -1,9 +1,11 @@
|
|||||||
# Guest-config VM test, bridged mode: the guest on a LAN served by DHCP,
|
# Guest-config VM test, bridged mode: the guest on a LAN that hands it one
|
||||||
# with a second node acting as the LAN (a DHCP server, a throwaway HTTP
|
# fixed address (the production shape of services.hermenix.network.bridged
|
||||||
# service, and a plain device that connects back to the guest). This is the
|
# = true: the host-side DHCP server reserves lanAddress for the guest's MAC,
|
||||||
# guest-facing half of services.hermenix.network.bridged = true: the module's
|
# so the guest is effectively static without knowing it). A second node acts
|
||||||
# own bridge/enforcement rules are covered statically by
|
# as the LAN: the reservation-style DHCP server, a throwaway HTTP service,
|
||||||
# tests/network-modes-check.nix.
|
# and a plain device that connects back to the guest. This is the
|
||||||
|
# guest-facing half of the module; the module's own bridge/enforcement rules
|
||||||
|
# are covered statically by tests/network-modes-check.nix.
|
||||||
#
|
#
|
||||||
# The guest node and the mode-independent assertions (seeding, idempotency,
|
# The guest node and the mode-independent assertions (seeding, idempotency,
|
||||||
# account, SSH) are shared with guest-vm-unbridged via
|
# account, SSH) are shared with guest-vm-unbridged via
|
||||||
@@ -27,10 +29,10 @@ pkgs.testers.runNixOSTest {
|
|||||||
nodes = {
|
nodes = {
|
||||||
machine = common.guestNode;
|
machine = common.guestNode;
|
||||||
|
|
||||||
# The "LAN": serves DHCP to the guest and hosts a throwaway HTTP
|
# The "LAN": hands the guest its single fixed address and hosts a
|
||||||
# service, standing in for the LAN router and another LAN device. Its
|
# throwaway HTTP service, standing in for the router and another LAN
|
||||||
# firewall is off so the guest<->LAN checks exercise the network path
|
# device. Its firewall is off so the guest<->LAN checks exercise the
|
||||||
# itself, not this throwaway node's own firewall.
|
# network path itself, not this throwaway node's own firewall.
|
||||||
lan =
|
lan =
|
||||||
{ pkgs, lib, ... }:
|
{ pkgs, lib, ... }:
|
||||||
{
|
{
|
||||||
@@ -63,7 +65,9 @@ pkgs.testers.runNixOSTest {
|
|||||||
networkConfig.LinkLocalAddressing = "no";
|
networkConfig.LinkLocalAddressing = "no";
|
||||||
};
|
};
|
||||||
|
|
||||||
# DHCP only (port = 0 disables dnsmasq's DNS half; the guest keeps
|
# DHCP only, and reservation-style: a single-address range, exactly
|
||||||
|
# how the host-side hermes-lan-dhcp hands the real guest its fixed
|
||||||
|
# lanAddress (port = 0 disables dnsmasq's DNS half; the guest keeps
|
||||||
# its own QUAD9 resolvers regardless). Bound to the node's static
|
# its own QUAD9 resolvers regardless). Bound to the node's static
|
||||||
# vlan address rather than an interface name - the framework's
|
# vlan address rather than an interface name - the framework's
|
||||||
# interface naming is not something this test should depend on.
|
# interface naming is not something this test should depend on.
|
||||||
@@ -73,10 +77,9 @@ pkgs.testers.runNixOSTest {
|
|||||||
port = 0;
|
port = 0;
|
||||||
listen-address = "192.168.1.2";
|
listen-address = "192.168.1.2";
|
||||||
bind-interfaces = true;
|
bind-interfaces = true;
|
||||||
dhcp-range = "192.168.1.100,192.168.1.200,12h";
|
dhcp-range = "192.168.1.100,192.168.1.100,12h";
|
||||||
dhcp-option = [
|
dhcp-option = [
|
||||||
"3,192.168.1.2" # default gateway -> this node
|
"3,192.168.1.2" # default gateway -> this node
|
||||||
"6,192.168.1.2" # DNS -> this node
|
|
||||||
];
|
];
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
@@ -93,7 +96,7 @@ pkgs.testers.runNixOSTest {
|
|||||||
+ common.commonScript
|
+ common.commonScript
|
||||||
+ ''
|
+ ''
|
||||||
# --- bridged-mode networking ---------------------------------------
|
# --- bridged-mode networking ---------------------------------------
|
||||||
# The LAN's DHCP gave the guest a lease alongside its static
|
# The LAN handed the guest its fixed address alongside its static
|
||||||
# service-net address. (Both test nodes share one vlan, standing in for
|
# service-net address. (Both test nodes share one vlan, standing in for
|
||||||
# the bridged uplink on a real host.)
|
# the bridged uplink on a real host.)
|
||||||
machine.wait_until_succeeds("ip -4 route show default | grep -q 'metric 100'")
|
machine.wait_until_succeeds("ip -4 route show default | grep -q 'metric 100'")
|
||||||
@@ -105,7 +108,7 @@ pkgs.testers.runNixOSTest {
|
|||||||
if not ip.startswith("10.77."):
|
if not ip.startswith("10.77."):
|
||||||
guest_lan_ip = ip
|
guest_lan_ip = ip
|
||||||
assert any("10.77.0.2/24" in line for line in addrs), f"static service-net address missing in bridged mode: {addrs}"
|
assert any("10.77.0.2/24" in line for line in addrs), f"static service-net address missing in bridged mode: {addrs}"
|
||||||
assert guest_lan_ip is not None, f"guest never took a LAN DHCP lease: {addrs}"
|
assert guest_lan_ip == "192.168.1.100", f"guest should have been handed its fixed LAN address, got: {addrs}"
|
||||||
|
|
||||||
# Both default routes are in place: the LAN's (metric 100, in use) and
|
# Both default routes are in place: the LAN's (metric 100, in use) and
|
||||||
# the host service-net fallback (metric 200) that NAT mode relies on.
|
# the host service-net fallback (metric 200) that NAT mode relies on.
|
||||||
|
|||||||
+200
-25
@@ -38,11 +38,36 @@ let
|
|||||||
|
|
||||||
# Netfilter mark tagging guest-originated traffic in bridged mode. The
|
# Netfilter mark tagging guest-originated traffic in bridged mode. The
|
||||||
# guest's only way in is the tap interface, so marking at the bridge
|
# guest's only way in is the tap interface, so marking at the bridge
|
||||||
# prerouting hook is as specific as an address match - and it is the only
|
# prerouting hook identifies the guest regardless of which address it
|
||||||
# way to identify the guest at the ip-family layer, where its DHCP-issued
|
# currently presents - its static service-net address, or the fixed LAN
|
||||||
# LAN address is not statically known. 18501 == 0x4845 ("HE"), for hermenix.
|
# address served to it by the host-side DHCP on the bridge. 18501 ==
|
||||||
|
# 0x4845 ("HE"), for hermenix.
|
||||||
bridgeMark = 18501;
|
bridgeMark = 18501;
|
||||||
|
|
||||||
|
# Dotted-quad netmask for a prefix length, used by the bridged-mode DHCP
|
||||||
|
# server's single-address lease (dnsmasq needs an explicit mask for a
|
||||||
|
# one-address range).
|
||||||
|
netmaskFromPrefix =
|
||||||
|
p:
|
||||||
|
lib.concatStringsSep "." (
|
||||||
|
lib.genList (
|
||||||
|
b:
|
||||||
|
toString (
|
||||||
|
builtins.elemAt [
|
||||||
|
0
|
||||||
|
128
|
||||||
|
192
|
||||||
|
224
|
||||||
|
240
|
||||||
|
248
|
||||||
|
252
|
||||||
|
254
|
||||||
|
255
|
||||||
|
] (lib.max 0 (lib.min 8 (p - b * 8)))
|
||||||
|
)
|
||||||
|
) 4
|
||||||
|
);
|
||||||
|
|
||||||
# Graceful ACPI power button via QMP, then wait for QEMU to exit.
|
# Graceful ACPI power button via QMP, then wait for QEMU to exit.
|
||||||
stopScript = pkgs.writeShellScript "hermes-vm-stop" ''
|
stopScript = pkgs.writeShellScript "hermes-vm-stop" ''
|
||||||
set -eu
|
set -eu
|
||||||
@@ -337,24 +362,68 @@ in
|
|||||||
type = types.int;
|
type = types.int;
|
||||||
default = 24;
|
default = 24;
|
||||||
};
|
};
|
||||||
|
lanAddress = mkOption {
|
||||||
|
type = types.str;
|
||||||
|
default = "";
|
||||||
|
example = "10.0.12.50";
|
||||||
|
description = "The guest's fixed address on your LAN. Required when `bridged = true`.";
|
||||||
|
};
|
||||||
|
hostLanAddress = mkOption {
|
||||||
|
type = types.str;
|
||||||
|
default = "";
|
||||||
|
example = "10.0.12.10";
|
||||||
|
description = ''
|
||||||
|
This host's own address on your LAN. When `bridged` is enabled the
|
||||||
|
uplink becomes a bridge port and this address moves onto the bridge
|
||||||
|
(which clones the uplink's MAC, so the host keeps its LAN identity).
|
||||||
|
Required when `bridged = true`.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
lanGateway = mkOption {
|
||||||
|
type = types.str;
|
||||||
|
default = "";
|
||||||
|
example = "10.0.12.1";
|
||||||
|
description = ''
|
||||||
|
Your LAN's router. Used as the default gateway for the guest and,
|
||||||
|
while `bridged`, for this host's bridge too (DNS unless `lanDns` is
|
||||||
|
set). Required when `bridged = true`.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
lanPrefixLength = mkOption {
|
||||||
|
type = types.int;
|
||||||
|
default = 24;
|
||||||
|
description = "Prefix length of your LAN subnet (netmask).";
|
||||||
|
};
|
||||||
|
lanDns = mkOption {
|
||||||
|
type = types.listOf types.str;
|
||||||
|
default = [ ];
|
||||||
|
example = [ "1.1.1.1" ];
|
||||||
|
description = ''
|
||||||
|
DNS servers for this host while `bridged` (the bridge no longer
|
||||||
|
takes a DHCP lease). Defaults to the LAN gateway.
|
||||||
|
'';
|
||||||
|
};
|
||||||
bridged = mkOption {
|
bridged = mkOption {
|
||||||
type = types.bool;
|
type = types.bool;
|
||||||
default = false;
|
default = false;
|
||||||
description = ''
|
description = ''
|
||||||
Put the guest directly on your LAN instead of behind the host's
|
Put the guest directly on your LAN instead of behind the host's
|
||||||
NAT. The bridge absorbs `externalInterface`, and the guest takes
|
NAT. The bridge absorbs `externalInterface`, so any device on the
|
||||||
an address from your LAN's DHCP, so any device on the LAN can
|
LAN can reach any port the guest exposes.
|
||||||
reach any port the guest exposes.
|
|
||||||
|
Addressing is fully static - no DHCP is involved. You must give
|
||||||
|
the guest a fixed LAN address (`lanAddress`), the host's own LAN
|
||||||
|
address (`hostLanAddress` - it moves onto the bridge), and the
|
||||||
|
LAN gateway (`lanGateway`). The guest is handed `lanAddress` by a
|
||||||
|
DHCP server this module runs on the bridge, keyed to the guest's
|
||||||
|
MAC (`network.mac`), so the guest itself stays a plain DHCP
|
||||||
|
client and never needs to know its own address.
|
||||||
|
|
||||||
Inbound is no longer limited to `publish` (which is ignored in
|
Inbound is no longer limited to `publish` (which is ignored in
|
||||||
this mode): the LAN may reach every guest port. Outbound is still
|
this mode): the LAN may reach every guest port. Outbound is still
|
||||||
governed by `lanAccess` / `allowLan`, and `hostAccess` plus the
|
governed by `lanAccess` / `allowLan`, and `hostAccess` plus the
|
||||||
host cache still work over the host-guest service net
|
host cache still work over the host-guest service net
|
||||||
(`hostAddress` stays configured on the bridge).
|
(`hostAddress` stays configured on the bridge).
|
||||||
|
|
||||||
Requires a working DHCP server on the LAN, and a guest that runs
|
|
||||||
a DHCP client on its NIC (see guest/common.nix - this repo's
|
|
||||||
guest does).
|
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
externalInterface = mkOption {
|
externalInterface = mkOption {
|
||||||
@@ -509,6 +578,17 @@ in
|
|||||||
assertion = net.externalInterface != null;
|
assertion = net.externalInterface != null;
|
||||||
message = "services.hermenix.network.externalInterface must be set.";
|
message = "services.hermenix.network.externalInterface must be set.";
|
||||||
}
|
}
|
||||||
|
{
|
||||||
|
assertion =
|
||||||
|
!net.bridged || (net.lanAddress != "" && net.hostLanAddress != "" && net.lanGateway != "");
|
||||||
|
message = ''
|
||||||
|
services.hermenix.network.bridged = true requires a fully static
|
||||||
|
LAN setup (DHCP is no longer supported in bridged mode). Set
|
||||||
|
network.lanAddress (the guest's fixed LAN address),
|
||||||
|
network.hostLanAddress (this host's own LAN address, which moves
|
||||||
|
onto the bridge) and network.lanGateway (your router's address).
|
||||||
|
'';
|
||||||
|
}
|
||||||
{
|
{
|
||||||
assertion = cfg.backup.enable -> (cfg.backup.repository != "");
|
assertion = cfg.backup.enable -> (cfg.backup.repository != "");
|
||||||
message = "services.hermenix.backup.repository must be set when backups are enabled.";
|
message = "services.hermenix.backup.repository must be set when backups are enabled.";
|
||||||
@@ -762,32 +842,120 @@ in
|
|||||||
systemd.network.networks."10-${net.bridge}" = {
|
systemd.network.networks."10-${net.bridge}" = {
|
||||||
matchConfig.Name = net.bridge;
|
matchConfig.Name = net.bridge;
|
||||||
# hostAddress stays on the bridge in both modes: it is the guest's
|
# hostAddress stays on the bridge in both modes: it is the guest's
|
||||||
# service net (host cache, hostAccess). In bridged mode the LAN DHCP
|
# service net (host cache, hostAccess). In bridged mode the host's
|
||||||
# lease is added on top of it.
|
# static LAN address is added on top of it - the bridge only ever has
|
||||||
address = [ "${net.hostAddress}/${toString net.prefixLength}" ];
|
# static addresses, never a DHCP lease.
|
||||||
|
address = [
|
||||||
|
"${net.hostAddress}/${toString net.prefixLength}"
|
||||||
|
]
|
||||||
|
++ lib.optional net.bridged "${net.hostLanAddress}/${toString net.lanPrefixLength}";
|
||||||
networkConfig = {
|
networkConfig = {
|
||||||
ConfigureWithoutCarrier = true;
|
ConfigureWithoutCarrier = true;
|
||||||
IPv4Forwarding = true;
|
IPv4Forwarding = true;
|
||||||
}
|
}
|
||||||
// lib.optionalAttrs net.bridged { DHCP = "ipv4"; };
|
// lib.optionalAttrs net.bridged {
|
||||||
|
# Static LAN config for the host: the bridge no longer takes a DHCP
|
||||||
|
# lease (bridged mode is static-only), so its gateway and DNS come
|
||||||
|
# from the module options instead.
|
||||||
|
Gateway = net.lanGateway;
|
||||||
|
DNS = if net.lanDns == [ ] then [ net.lanGateway ] else net.lanDns;
|
||||||
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
|
# The host's DNS, now that the bridge cannot learn it from a DHCP lease.
|
||||||
|
# mkDefault so the operator's own nameservers win if they set any.
|
||||||
|
networking.nameservers = lib.mkIf net.bridged (
|
||||||
|
lib.mkDefault (if net.lanDns == [ ] then [ net.lanGateway ] else net.lanDns)
|
||||||
|
);
|
||||||
|
|
||||||
systemd.network.networks."11-${net.tapId}" = {
|
systemd.network.networks."11-${net.tapId}" = {
|
||||||
matchConfig.Name = net.tapId;
|
matchConfig.Name = net.tapId;
|
||||||
networkConfig.Bridge = net.bridge;
|
networkConfig.Bridge = net.bridge;
|
||||||
};
|
};
|
||||||
|
|
||||||
# Bridged mode: the physical uplink becomes a bridge port so the LAN
|
# Bridged mode: the physical uplink becomes a bridge port so the LAN
|
||||||
# shares an L2 domain with the guest and the guest's DHCP is answered
|
# shares an L2 domain with the guest. The port must not carry its own
|
||||||
# by the LAN itself. The port must not carry its own address config -
|
# address config - the bridge owns the LAN address now (static, via
|
||||||
# the bridge owns the LAN address now. Note the bridge takes a fresh
|
# hostLanAddress; there is no DHCP on the bridge).
|
||||||
# DHCP lease, and because its MAC can differ from the uplink's, the
|
|
||||||
# host's LAN IP may change after the first switch to bridged mode.
|
|
||||||
systemd.network.networks."12-${net.externalInterface}" = lib.mkIf net.bridged {
|
systemd.network.networks."12-${net.externalInterface}" = lib.mkIf net.bridged {
|
||||||
matchConfig.Name = net.externalInterface;
|
matchConfig.Name = net.externalInterface;
|
||||||
networkConfig.Bridge = net.bridge;
|
networkConfig.Bridge = net.bridge;
|
||||||
};
|
};
|
||||||
|
|
||||||
|
# The bridge is created with no MAC of its own, so Linux makes it adopt
|
||||||
|
# the lowest-numbered port's MAC - and the tap is always the first port.
|
||||||
|
# That is how the host once ended up presenting the VM's MAC on the LAN
|
||||||
|
# (a duplicate address, since the guest uses the same MAC). Clone the
|
||||||
|
# uplink's MAC onto the bridge so the host keeps its LAN identity; the
|
||||||
|
# guest keeps its own tap MAC (net.mac).
|
||||||
|
systemd.services.hermes-bridge-mac = lib.mkIf net.bridged {
|
||||||
|
description = "Clone the uplink MAC onto the hermenix bridge";
|
||||||
|
after = [
|
||||||
|
"systemd-networkd.service"
|
||||||
|
"sys-subsystem-net-devices-${net.bridge}.device"
|
||||||
|
"sys-subsystem-net-devices-${net.externalInterface}.device"
|
||||||
|
];
|
||||||
|
wants = [ "systemd-networkd.service" ];
|
||||||
|
before = [ "network-online.target" ];
|
||||||
|
wantedBy = [ "multi-user.target" ];
|
||||||
|
serviceConfig = {
|
||||||
|
Type = "oneshot";
|
||||||
|
RemainAfterExit = true;
|
||||||
|
};
|
||||||
|
unitConfig.ConditionPathExists = "/sys/class/net/${net.externalInterface}";
|
||||||
|
script = ''
|
||||||
|
set -euo pipefail
|
||||||
|
# networkd creates the bridge netdev asynchronously; give it a moment.
|
||||||
|
for _ in 1 2 3 4 5 6 7 8 9 10; do
|
||||||
|
[ -e "/sys/class/net/${net.bridge}" ] && break
|
||||||
|
sleep 0.5
|
||||||
|
done
|
||||||
|
[ -e "/sys/class/net/${net.bridge}" ] || exit 1
|
||||||
|
mac="$(${pkgs.coreutils}/bin/cat /sys/class/net/${net.externalInterface}/address)"
|
||||||
|
${pkgs.iproute2}/bin/ip link set dev ${net.bridge} down
|
||||||
|
${pkgs.iproute2}/bin/ip link set dev ${net.bridge} address "$mac"
|
||||||
|
${pkgs.iproute2}/bin/ip link set dev ${net.bridge} up
|
||||||
|
# Re-apply the bridge's static address config under the new MAC.
|
||||||
|
${pkgs.systemd}/bin/networkctl reconfigure ${net.bridge} || true
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
# The guest is handed its fixed LAN address (lanAddress) by a dedicated
|
||||||
|
# DHCP server bound to the bridge, keyed to the guest's MAC. This is the
|
||||||
|
# guest's ONLY DHCP: its broadcast is blocked from leaving the bridge
|
||||||
|
# (see the hermenix-bridge table), so the LAN's own DHCP server can
|
||||||
|
# never race the reservation. DNS is disabled (port=0) - the guest runs
|
||||||
|
# static QUAD9 resolvers, and this host gets its DNS from lanDns.
|
||||||
|
systemd.services.hermes-lan-dhcp = lib.mkIf net.bridged {
|
||||||
|
description = "Static-lease DHCP for the bridged hermenix guest";
|
||||||
|
after = [
|
||||||
|
"systemd-networkd.service"
|
||||||
|
"sys-subsystem-net-devices-${net.bridge}.device"
|
||||||
|
];
|
||||||
|
wants = [ "systemd-networkd.service" ];
|
||||||
|
wantedBy = [ "multi-user.target" ];
|
||||||
|
serviceConfig = {
|
||||||
|
Type = "simple";
|
||||||
|
Restart = "on-failure";
|
||||||
|
RestartSec = 5;
|
||||||
|
};
|
||||||
|
script = ''
|
||||||
|
for _ in 1 2 3 4 5 6 7 8 9 10; do
|
||||||
|
[ -e "/sys/class/net/${net.bridge}" ] && break
|
||||||
|
sleep 0.5
|
||||||
|
done
|
||||||
|
exec ${pkgs.dnsmasq}/bin/dnsmasq --keep-in-foreground \
|
||||||
|
--conf-file=/dev/null \
|
||||||
|
--interface=${net.bridge} \
|
||||||
|
--bind-dynamic \
|
||||||
|
--port=0 \
|
||||||
|
--dhcp-authoritative \
|
||||||
|
--dhcp-range=${net.lanAddress},${net.lanAddress},${netmaskFromPrefix net.lanPrefixLength},12h \
|
||||||
|
--dhcp-host=${net.mac},${net.lanAddress} \
|
||||||
|
--dhcp-option=option:router,${net.lanGateway}
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
# ip_forward: use the same sysctl keys networking.nat used (mkOverride 99
|
# ip_forward: use the same sysctl keys networking.nat used (mkOverride 99
|
||||||
# so they win over any lower-priority defaults elsewhere in the config).
|
# so they win over any lower-priority defaults elsewhere in the config).
|
||||||
boot.kernel.sysctl."net.ipv4.conf.all.forwarding" = lib.mkOverride 99 true;
|
boot.kernel.sysctl."net.ipv4.conf.all.forwarding" = lib.mkOverride 99 true;
|
||||||
@@ -856,6 +1024,13 @@ in
|
|||||||
# Tag anything entering from the guest. ip-family input rules
|
# Tag anything entering from the guest. ip-family input rules
|
||||||
# (host cache, hostAccess) match this mark.
|
# (host cache, hostAccess) match this mark.
|
||||||
iifname "${net.tapId}" meta mark set ${toString bridgeMark}
|
iifname "${net.tapId}" meta mark set ${toString bridgeMark}
|
||||||
|
|
||||||
|
# The guest's address is static - served by hermes-lan-dhcp on
|
||||||
|
# this bridge - so keep its DHCP traffic off the LAN and keep the
|
||||||
|
# LAN's DHCP answers out of the guest. Only the host-side server
|
||||||
|
# may talk DHCP to it.
|
||||||
|
iifname "${net.tapId}" oifname "${net.externalInterface}" ether type ip ip protocol udp udp sport 68 udp dport 67 drop
|
||||||
|
iifname "${net.externalInterface}" oifname "${net.tapId}" ether type ip ip protocol udp udp sport 67 udp dport 68 drop
|
||||||
}
|
}
|
||||||
chain forward {
|
chain forward {
|
||||||
type filter hook forward priority filter; policy accept;
|
type filter hook forward priority filter; policy accept;
|
||||||
@@ -903,12 +1078,12 @@ in
|
|||||||
# in hostAccess.
|
# in hostAccess.
|
||||||
#
|
#
|
||||||
# NAT mode: the guest always arrives from its fixed service-net address,
|
# NAT mode: the guest always arrives from its fixed service-net address,
|
||||||
# so match address + bridge. Bridged mode: the guest's LAN address comes
|
# so match address + bridge. Bridged mode: the guest presents either the
|
||||||
# from DHCP and is not known here, so the bridge table above tags guest
|
# service-net address or its LAN address (served by the bridge's DHCP),
|
||||||
# traffic with a netfilter mark as it enters the tap - the rules match
|
# so the bridge table above tags guest traffic with a netfilter mark as
|
||||||
# that mark instead. The tap is exclusive to the guest, so the mark is
|
# it enters the tap - the rules match that mark instead. The tap is
|
||||||
# as specific as an address match: LAN devices can never inherit these
|
# exclusive to the guest, so the mark is as specific as an address
|
||||||
# holes.
|
# match: LAN devices can never inherit these holes.
|
||||||
networking.firewall.extraInputRules =
|
networking.firewall.extraInputRules =
|
||||||
if net.bridged then
|
if net.bridged then
|
||||||
optionalString cfg.hostCache.enable ''
|
optionalString cfg.hostCache.enable ''
|
||||||
|
|||||||
@@ -36,6 +36,10 @@ let
|
|||||||
type = lib.types.bool;
|
type = lib.types.bool;
|
||||||
default = false;
|
default = false;
|
||||||
};
|
};
|
||||||
|
nameservers = lib.mkOption {
|
||||||
|
type = lib.types.listOf lib.types.str;
|
||||||
|
default = [ ];
|
||||||
|
};
|
||||||
nftables = {
|
nftables = {
|
||||||
enable = lib.mkOption {
|
enable = lib.mkOption {
|
||||||
type = lib.types.bool;
|
type = lib.types.bool;
|
||||||
@@ -65,6 +69,10 @@ let
|
|||||||
};
|
};
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
systemd.services = lib.mkOption {
|
||||||
|
type = lib.types.attrsOf lib.types.anything;
|
||||||
|
default = { };
|
||||||
|
};
|
||||||
systemd.network = {
|
systemd.network = {
|
||||||
networks = lib.mkOption {
|
networks = lib.mkOption {
|
||||||
type = lib.types.attrsOf lib.types.anything;
|
type = lib.types.attrsOf lib.types.anything;
|
||||||
@@ -91,6 +99,13 @@ let
|
|||||||
network = {
|
network = {
|
||||||
externalInterface = "eno1";
|
externalInterface = "eno1";
|
||||||
inherit bridged;
|
inherit bridged;
|
||||||
|
}
|
||||||
|
// lib.optionalAttrs bridged {
|
||||||
|
# Bridged mode is static-only: guest address, host address on
|
||||||
|
# the bridge, and the LAN router.
|
||||||
|
lanAddress = "10.7.7.50";
|
||||||
|
hostLanAddress = "10.7.7.2";
|
||||||
|
lanGateway = "10.7.7.1";
|
||||||
};
|
};
|
||||||
allowLan = false;
|
allowLan = false;
|
||||||
publish = [
|
publish = [
|
||||||
@@ -146,7 +161,7 @@ let
|
|||||||
"nat: bridge netdev exists" =
|
"nat: bridge netdev exists" =
|
||||||
nat.config.systemd.network.netdevs."10-hermes-br".netdevConfig.Kind == "bridge";
|
nat.config.systemd.network.netdevs."10-hermes-br".netdevConfig.Kind == "bridge";
|
||||||
|
|
||||||
# --- bridged mode: the guest is directly on the LAN ----------------
|
# --- bridged mode: the guest is directly on the LAN, static-only -----
|
||||||
"bridged: bridge-family table exists" = brNft ? "hermenix-bridge" && brBr.family == "bridge";
|
"bridged: bridge-family table exists" = brNft ? "hermenix-bridge" && brBr.family == "bridge";
|
||||||
"bridged: guest traffic is marked as it enters the tap" = has "meta mark set 18501" brBr.content;
|
"bridged: guest traffic is marked as it enters the tap" = has "meta mark set 18501" brBr.content;
|
||||||
"bridged: inbound from the LAN is open on every port" =
|
"bridged: inbound from the LAN is open on every port" =
|
||||||
@@ -155,18 +170,42 @@ let
|
|||||||
has "ip daddr 192.168.1.50 tcp dport 11434 accept" brBr.content;
|
has "ip daddr 192.168.1.50 tcp dport 11434 accept" brBr.content;
|
||||||
"bridged: private-space egress is dropped unless allowlisted" =
|
"bridged: private-space egress is dropped unless allowlisted" =
|
||||||
has "192.168.0.0/16" brBr.content && has "fc00::/7" brBr.content;
|
has "192.168.0.0/16" brBr.content && has "fc00::/7" brBr.content;
|
||||||
|
"bridged: guest DHCP is confined to the bridge, can't race the LAN router" =
|
||||||
|
has "udp sport 68 udp dport 67 drop" brBr.content
|
||||||
|
&& has "udp sport 67 udp dport 68 drop" brBr.content;
|
||||||
"bridged: no DNAT for publish" =
|
"bridged: no DNAT for publish" =
|
||||||
!(has "dnat to" brNat.content) && !(has "chain prerouting" brNat.content);
|
!(has "dnat to" brNat.content) && !(has "chain prerouting" brNat.content);
|
||||||
"bridged: scoped fallback masquerade remains" = has "masquerade" brNat.content;
|
"bridged: scoped fallback masquerade remains" = has "masquerade" brNat.content;
|
||||||
"bridged: input rules match the mark, not the unknown DHCP address" =
|
"bridged: input rules match the mark, not an address" =
|
||||||
has "meta mark 18501 tcp dport 5000 accept" brInput
|
has "meta mark 18501 tcp dport 5000 accept" brInput
|
||||||
&& has "meta mark 18501 tcp dport 8188 accept" brInput
|
&& has "meta mark 18501 tcp dport 8188 accept" brInput
|
||||||
&& !(has "ip saddr 10.77.0.2" brInput);
|
&& !(has "ip saddr 10.77.0.2" brInput);
|
||||||
"bridged: publish forward rules are dropped" =
|
"bridged: publish forward rules are dropped" =
|
||||||
!(has "ip daddr 10.77.0.2 tcp dport 8080 accept" brForward);
|
!(has "ip daddr 10.77.0.2 tcp dport 8080 accept" brForward);
|
||||||
"bridged: bridge keeps the service net and takes LAN DHCP" =
|
"bridged: bridge carries only static addresses, never a DHCP lease" =
|
||||||
brNetworks."10-hermes-br".address == [ "10.77.0.1/24" ]
|
brNetworks."10-hermes-br".address == [
|
||||||
&& brNetworks."10-hermes-br".networkConfig.DHCP == "ipv4";
|
"10.77.0.1/24"
|
||||||
|
"10.7.7.2/24"
|
||||||
|
]
|
||||||
|
&& (brNetworks."10-hermes-br".networkConfig.DHCP or null) == null
|
||||||
|
&& !(has "DHCP" (builtins.toJSON brNetworks."10-hermes-br"));
|
||||||
|
"bridged: host gateway and DNS are static on the bridge" =
|
||||||
|
brNetworks."10-hermes-br".networkConfig.Gateway == "10.7.7.1"
|
||||||
|
&& brNetworks."10-hermes-br".networkConfig.DNS == [ "10.7.7.1" ];
|
||||||
|
"bridged: guest is handed lanAddress by the bridge DHCP server" =
|
||||||
|
let
|
||||||
|
dhcp = br.config.systemd.services."hermes-lan-dhcp".script;
|
||||||
|
in
|
||||||
|
has "--interface=hermes-br" dhcp
|
||||||
|
&& has "--dhcp-range=10.7.7.50,10.7.7.50,255.255.255.0,12h" dhcp
|
||||||
|
&& has "--dhcp-host=02:00:00:7e:00:01,10.7.7.50" dhcp
|
||||||
|
&& has "--dhcp-option=option:router,10.7.7.1" dhcp
|
||||||
|
&& has "--port=0" dhcp;
|
||||||
|
"bridged: uplink MAC is cloned onto the bridge, host keeps its identity" =
|
||||||
|
let
|
||||||
|
mac = br.config.systemd.services."hermes-bridge-mac".script;
|
||||||
|
in
|
||||||
|
has "ip link set dev hermes-br address" mac && has "/sys/class/net/eno1/address" mac;
|
||||||
"bridged: uplink is a bridge port" = brNetworks."12-eno1".networkConfig.Bridge == "hermes-br";
|
"bridged: uplink is a bridge port" = brNetworks."12-eno1".networkConfig.Bridge == "hermes-br";
|
||||||
"bridged: tap is still a bridge port" =
|
"bridged: tap is still a bridge port" =
|
||||||
brNetworks."11-vm-hermes".networkConfig.Bridge == "hermes-br";
|
brNetworks."11-vm-hermes".networkConfig.Bridge == "hermes-br";
|
||||||
|
|||||||
Reference in new issue
Block a user