Files
2026-10-09 14:35:12 -04:00

125 lines
4.3 KiB
Markdown

# Hardware passthrough & virtual devices
## Before reaching for passthrough
If the goal is "the agent can use the GPU", passthrough is often wrong. It is
exclusive (host loses the card while the VM runs) and is the largest isolation
reduction in this setup.
Prefer: keep Ollama/ComfyUI on the host, open the port with `hostAccess`
([NETWORK.md](NETWORK.md#guest--a-service-on-the-host)). Pass hardware through
when the guest genuinely needs the device — CUDA in-process, capture card,
radio, display output.
## What PCI passthrough costs
A PCI device does DMA. The IOMMU is the only thing between it and host memory,
so `passthrough.iommu` is required; the module refuses to build without it.
- **Whole IOMMU groups only.** Check before planning:
```bash
for d in /sys/kernel/iommu_groups/*/devices/*; do
echo "group $(basename "$(dirname "$(dirname "$d")")"): $(lspci -nns "$(basename "$d")")"
done | sort -V
```
ACS override patches lie about isolation — not appropriate here.
- **Every function.** A GPU is at least graphics + HDMI audio.
## GPU
```bash
lspci -Dnn | grep -i vga
```
```nix
services.hermenix.passthrough = {
iommu = "amd"; # or "intel"
pci = [
{ address = "0000:01:00.0"; id = "10de:2204"; } # graphics
{ address = "0000:01:00.1"; id = "10de:1aef"; } # HDMI audio
];
};
```
Setting `id` makes vfio-pci claim the card in the initrd before `nvidia` /
`amdgpu`. Do this for GPUs. Address-only devices bind at boot via a oneshot
instead.
Rebuild and **reboot**, then:
```bash
lspci -nnk -d 10de:2204 | grep "Kernel driver" # want: vfio-pci
```
Guest-side drivers are the agent's problem (`/etc/nixos`).
## USB / Bluetooth
```bash
lsusb
```
```nix
services.hermenix.passthrough.usb = [
{ vendorId = "8087"; productId = "0026"; }
{ vendorId = "0403"; productId = "6001"; }
];
```
QEMU mediates (no IOMMU/VFIO). Host must not be using the device (`btusb`
blacklist, no `hardware.bluetooth` for an integrated radio). No hotplug —
`hermenix vm restart` after plugging. Whole USB controller as PCI gives real
hotplug at the cost of exclusivity.
## When the VM won't start
`hermenix vm console` first. Then: device not on vfio-pci (`lspci -nnk`),
IOMMU off (`dmesg | grep -i iommu`), or something in the group still held by
the host. The module raises `LimitMEMLOCK` when PCI passthrough is configured.
---
## Virtual devices
`services.hermenix.devices` attaches optional QEMU devices (no host hardware):
| Option | Default | Attaches |
| ------------------------- | ---------- | ----------------------------------------------------- |
| `devices.rng.enable` | `true` | virtio entropy |
| `devices.vsock.enable` | `true` | host↔guest vsock (`ctl`, backups) |
| `devices.video.enable` | `false` | virtual display / render node |
| `devices.video.model` | `"virtio"` | `virtio` (render node) or legacy VGA |
| `devices.balloon.enable` | `false` | virtio memory balloon |
| `devices.watchdog.enable` | `false` | `i6300esb`/`ib700` |
| `devices.input.enable` | `false` | virtio keyboard/tablet |
| `devices.audio.enable` | `false` | `intel-hda`/`virtio-sound` (see caveat) |
| `devices.tpm.enable` | `false` | TPM 2.0 via swtpm |
| `devices.cdroms` | `[]` | read-only CD images |
| `devices.extraDisks` | `[]` | extra raw disks |
| `devices.usbController` | `"auto"` | `qemu-xhci` iff `passthrough.usb` set |
`extraQemuArgs` appends raw QEMU args (unvalidated). Prefer typed options.
### Render node
```nix
services.hermenix.devices.video.enable = true;
```
Gives `/dev/dri/card0` and `/dev/dri/renderD128` (`virtio` model only).
Console stays serial.
### TPM
`devices.tpm.enable = true` starts swtpm; state under
`/var/lib/hermes-vm-tpm` by default.
### Audio caveat
`hermes-vm.service` is hardened (`ProtectHome`, `PrivateTmp`, dedicated user)
and cannot reach PipeWire without loosening that. Leave audio off unless you
mean to relax the unit.