125 lines
4.3 KiB
Markdown
125 lines
4.3 KiB
Markdown
# Hardware passthrough & virtual devices
|
|
|
|
## Before reaching for passthrough
|
|
|
|
If the goal is "the agent can use the GPU", passthrough is often wrong. It is
|
|
exclusive (host loses the card while the VM runs) and is the largest isolation
|
|
reduction in this setup.
|
|
|
|
Prefer: keep Ollama/ComfyUI on the host, open the port with `hostAccess`
|
|
([NETWORK.md](NETWORK.md#guest--a-service-on-the-host)). Pass hardware through
|
|
when the guest genuinely needs the device — CUDA in-process, capture card,
|
|
radio, display output.
|
|
|
|
## What PCI passthrough costs
|
|
|
|
A PCI device does DMA. The IOMMU is the only thing between it and host memory,
|
|
so `passthrough.iommu` is required; the module refuses to build without it.
|
|
|
|
- **Whole IOMMU groups only.** Check before planning:
|
|
|
|
```bash
|
|
for d in /sys/kernel/iommu_groups/*/devices/*; do
|
|
echo "group $(basename "$(dirname "$(dirname "$d")")"): $(lspci -nns "$(basename "$d")")"
|
|
done | sort -V
|
|
```
|
|
|
|
ACS override patches lie about isolation — not appropriate here.
|
|
|
|
- **Every function.** A GPU is at least graphics + HDMI audio.
|
|
|
|
## GPU
|
|
|
|
```bash
|
|
lspci -Dnn | grep -i vga
|
|
```
|
|
|
|
```nix
|
|
services.hermenix.passthrough = {
|
|
iommu = "amd"; # or "intel"
|
|
pci = [
|
|
{ address = "0000:01:00.0"; id = "10de:2204"; } # graphics
|
|
{ address = "0000:01:00.1"; id = "10de:1aef"; } # HDMI audio
|
|
];
|
|
};
|
|
```
|
|
|
|
Setting `id` makes vfio-pci claim the card in the initrd before `nvidia` /
|
|
`amdgpu`. Do this for GPUs. Address-only devices bind at boot via a oneshot
|
|
instead.
|
|
|
|
Rebuild and **reboot**, then:
|
|
|
|
```bash
|
|
lspci -nnk -d 10de:2204 | grep "Kernel driver" # want: vfio-pci
|
|
```
|
|
|
|
Guest-side drivers are the agent's problem (`/etc/nixos`).
|
|
|
|
## USB / Bluetooth
|
|
|
|
```bash
|
|
lsusb
|
|
```
|
|
|
|
```nix
|
|
services.hermenix.passthrough.usb = [
|
|
{ vendorId = "8087"; productId = "0026"; }
|
|
{ vendorId = "0403"; productId = "6001"; }
|
|
];
|
|
```
|
|
|
|
QEMU mediates (no IOMMU/VFIO). Host must not be using the device (`btusb`
|
|
blacklist, no `hardware.bluetooth` for an integrated radio). No hotplug —
|
|
`hermenix vm restart` after plugging. Whole USB controller as PCI gives real
|
|
hotplug at the cost of exclusivity.
|
|
|
|
## When the VM won't start
|
|
|
|
`hermenix vm console` first. Then: device not on vfio-pci (`lspci -nnk`),
|
|
IOMMU off (`dmesg | grep -i iommu`), or something in the group still held by
|
|
the host. The module raises `LimitMEMLOCK` when PCI passthrough is configured.
|
|
|
|
---
|
|
|
|
## Virtual devices
|
|
|
|
`services.hermenix.devices` attaches optional QEMU devices (no host hardware):
|
|
|
|
| Option | Default | Attaches |
|
|
| ------------------------- | ---------- | ----------------------------------------------------- |
|
|
| `devices.rng.enable` | `true` | virtio entropy |
|
|
| `devices.vsock.enable` | `true` | host↔guest vsock (`ctl`, backups) |
|
|
| `devices.video.enable` | `false` | virtual display / render node |
|
|
| `devices.video.model` | `"virtio"` | `virtio` (render node) or legacy VGA |
|
|
| `devices.balloon.enable` | `false` | virtio memory balloon |
|
|
| `devices.watchdog.enable` | `false` | `i6300esb`/`ib700` |
|
|
| `devices.input.enable` | `false` | virtio keyboard/tablet |
|
|
| `devices.audio.enable` | `false` | `intel-hda`/`virtio-sound` (see caveat) |
|
|
| `devices.tpm.enable` | `false` | TPM 2.0 via swtpm |
|
|
| `devices.cdroms` | `[]` | read-only CD images |
|
|
| `devices.extraDisks` | `[]` | extra raw disks |
|
|
| `devices.usbController` | `"auto"` | `qemu-xhci` iff `passthrough.usb` set |
|
|
|
|
`extraQemuArgs` appends raw QEMU args (unvalidated). Prefer typed options.
|
|
|
|
### Render node
|
|
|
|
```nix
|
|
services.hermenix.devices.video.enable = true;
|
|
```
|
|
|
|
Gives `/dev/dri/card0` and `/dev/dri/renderD128` (`virtio` model only).
|
|
Console stays serial.
|
|
|
|
### TPM
|
|
|
|
`devices.tpm.enable = true` starts swtpm; state under
|
|
`/var/lib/hermes-vm-tpm` by default.
|
|
|
|
### Audio caveat
|
|
|
|
`hermes-vm.service` is hardened (`ProtectHome`, `PrivateTmp`, dedicated user)
|
|
and cannot reach PipeWire without loosening that. Leave audio off unless you
|
|
mean to relax the unit.
|