Files
hermenix/docs/PASSTHROUGH.md
T
2026-10-09 14:35:12 -04:00

4.3 KiB

Hardware passthrough & virtual devices

Before reaching for passthrough

If the goal is "the agent can use the GPU", passthrough is often wrong. It is exclusive (host loses the card while the VM runs) and is the largest isolation reduction in this setup.

Prefer: keep Ollama/ComfyUI on the host, open the port with hostAccess (NETWORK.md). Pass hardware through when the guest genuinely needs the device — CUDA in-process, capture card, radio, display output.

What PCI passthrough costs

A PCI device does DMA. The IOMMU is the only thing between it and host memory, so passthrough.iommu is required; the module refuses to build without it.

  • Whole IOMMU groups only. Check before planning:

    for d in /sys/kernel/iommu_groups/*/devices/*; do
      echo "group $(basename "$(dirname "$(dirname "$d")")"): $(lspci -nns "$(basename "$d")")"
    done | sort -V
    

    ACS override patches lie about isolation — not appropriate here.

  • Every function. A GPU is at least graphics + HDMI audio.

GPU

lspci -Dnn | grep -i vga
services.hermenix.passthrough = {
  iommu = "amd";   # or "intel"
  pci = [
    { address = "0000:01:00.0"; id = "10de:2204"; }  # graphics
    { address = "0000:01:00.1"; id = "10de:1aef"; }  # HDMI audio
  ];
};

Setting id makes vfio-pci claim the card in the initrd before nvidia / amdgpu. Do this for GPUs. Address-only devices bind at boot via a oneshot instead.

Rebuild and reboot, then:

lspci -nnk -d 10de:2204 | grep "Kernel driver"   # want: vfio-pci

Guest-side drivers are the agent's problem (/etc/nixos).

USB / Bluetooth

lsusb
services.hermenix.passthrough.usb = [
  { vendorId = "8087"; productId = "0026"; }
  { vendorId = "0403"; productId = "6001"; }
];

QEMU mediates (no IOMMU/VFIO). Host must not be using the device (btusb blacklist, no hardware.bluetooth for an integrated radio). No hotplug — hermenix vm restart after plugging. Whole USB controller as PCI gives real hotplug at the cost of exclusivity.

When the VM won't start

hermenix vm console first. Then: device not on vfio-pci (lspci -nnk), IOMMU off (dmesg | grep -i iommu), or something in the group still held by the host. The module raises LimitMEMLOCK when PCI passthrough is configured.


Virtual devices

services.hermenix.devices attaches optional QEMU devices (no host hardware):

Option Default Attaches
devices.rng.enable true virtio entropy
devices.vsock.enable true host↔guest vsock (ctl, backups)
devices.video.enable false virtual display / render node
devices.video.model "virtio" virtio (render node) or legacy VGA
devices.balloon.enable false virtio memory balloon
devices.watchdog.enable false i6300esb/ib700
devices.input.enable false virtio keyboard/tablet
devices.audio.enable false intel-hda/virtio-sound (see caveat)
devices.tpm.enable false TPM 2.0 via swtpm
devices.cdroms [] read-only CD images
devices.extraDisks [] extra raw disks
devices.usbController "auto" qemu-xhci iff passthrough.usb set

extraQemuArgs appends raw QEMU args (unvalidated). Prefer typed options.

Render node

services.hermenix.devices.video.enable = true;

Gives /dev/dri/card0 and /dev/dri/renderD128 (virtio model only). Console stays serial.

TPM

devices.tpm.enable = true starts swtpm; state under /var/lib/hermes-vm-tpm by default.

Audio caveat

hermes-vm.service is hardened (ProtectHome, PrivateTmp, dedicated user) and cannot reach PipeWire without loosening that. Leave audio off unless you mean to relax the unit.