Files
hyprland-headless-stream-pl…/docs/ARCHITECTURE.md
T
Hermenix 7190960120 capture: encode JPEG in-process too, so the live stream keeps working
PNG-only was a real defect, not a nice-to-have: `hhstream_live.server` serves
`multipart/x-mixed-replace` with `image/jpeg` parts and its own capture default is
`grim -c -t jpeg -q 80 -`, so switching this backend on would have broken video
streaming while looking faster in a benchmark.

JPEG now goes through libjpeg-turbo by ctypes, using its `TJPF_BGRX` input format
-- which is exactly the compositor's B, G, R, X memory order, so the frame goes
into the encoder with no copy and no colour conversion. Measured on this VM
against the real compositor, the deployment's own settings (Virtual-1, 1280x720,
jpeg q80, 200 frames each, two runs):

                        p50 ms   fps    our CPU   compositor CPU
    in-process           76.2   13.1     2.1 ms      51.5 ms
    grim -c -t jpeg     137.0    7.3     0.5 ms      80.2 ms
                                                          (second run: 84/144,
                                                           55.1/79.8 -> 1.71x)

So ~1.7-1.8x the frame rate and ~1.5x less of the compositor's CPU, with frames
of the same order (188 KB against grim's 222 KB). PNG stays available through the
stdlib for a deployment that would rather not carry the library.

Both segfault-class traps from the spike are handled and documented: every
argtype is declared (an undeclared pitch is a crash, not a wrong number), and
`*jpegBuf` is reset to NULL after `tjFree` (turbojpeg reads a non-NULL pointer as
a caller buffer, so the next frame would write into freed memory). The decoder is
here because a lossy codec can only be tested by round-tripping pixels -- the new
suite encodes solid blocks whose channels differ enough that a swap is
unmistakable, and includes a padded stride.

Packaging: libjpeg-turbo is reached through ctypes, so nothing links it and the
path must reach the loader -- the console scripts get a wrapper with it on
LD_LIBRARY_PATH, and the unit check exports it too. A missing library is reported
at startup rather than once per frame: a server that is up but can never produce
a frame is worse than one that refuses to start.

Verified end to end: server with `--capture wayland:Virtual-1 --format jpeg`,
client Hello `image/jpeg`, three frames read, each magic `ffd8`.
2026-10-11 22:58:28 +00:00

157 lines
10 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Architecture
```
┌──────────────────────────────────────────┐
│ hyprland-headless.service │
│ Hyprland 0.56.2, no physical monitor │
│ private XDG_RUNTIME_DIR, as `user` │
│ outputs come from `outputs[]` │
│ + one headless output per session │
└───┬───────────┬───────────────┬──────────┘
│ │ │
IPC: request / │ │ Wayland │ Wayland
event sockets │ │ session │ (virtual pointer)
│ ▼ │
│ ┌────────────────────────┴─────────┐
│ │ hyprland-headless-stream.service │
│ │ hhstream-server │
│ │ capture ──▶ one frame slot │
│ │ accept ──▶ thread per client│
│ └───────────────┬──────────────────┘
│ │ AF_UNIX, mode 0660
│ ▼
│ many clients ── hhstream-client / any reader
│
┌────────────┴──────────────┐ ┌──────────────────────────┐
│ hhstream-sessions.service │─────▶│ hhstream-pointer.service │
│ hhstreamd │ │ one Wayland connection │
│ registry, containers, │ JSON │ one virtual pointer │
│ the pointer lease │ over │ per output │
└────────────┬──────────────┘ a └──────────────────────────┘
│ private socket
│ AF_UNIX control.sock, mode 0660
▼
hhstream session ... (the CLI an agent drives)
```
Five pieces, deliberately separate:
1. **The compositor** (`Hyprland`, unchanged upstream) renders into virtual
outputs. It knows nothing about frames on a socket.
2. **The frame server** (`hhstream`) captures, holds exactly one frame, and
publishes it. It owns the socket, the framing and the drop policy.
3. **The client** is any process that can open a Unix socket and read the
contract's byte format. The shipped `hhstream-client` is a convenience, not a
requirement.
4. **The session broker** (`hhstreamd`) starts allowlisted applications as
tracked sessions, gives each one its own headless output and workspace (a
*container*), reads the compositor back over its own IPC, and owns the
pointer lease. It is the only part that keeps state across a restart.
5. **The pointer driver** (`hhstream-pointer`) is a Wayland client of the same
session that creates one `zwlr_virtual_pointer_v1` per output. It holds no
lease and no session state: the broker decides, the driver executes. It is
a separate process because a Wayland client needs its own connection and a
driver blocked on the compositor must not stall session lifecycle.
## Why containers are outputs
A session's container is a headless **output**, not just a workspace, and the
two consequences are the reason:
- `grim -o hh-<id>` captures exactly one session, so one agent's screenshot
cannot contain another's window;
- `zwlr_virtual_pointer_manager_v1` version 2 creates a pointer *bound to an
output*, so a warp to (0.5, 0.5) is the middle of that session's screen and
not the middle of the desktop. Version 1 is refused rather than used,
because its coordinates are global and every click would land somewhere
plausible and wrong.
The cursor itself is singular — there is one seat — so pointer access is
serialised by a broker-wide **lease**: one holder, a TTL the broker enforces
with its own sweeper, and buttons lifted on release or expiry. The lease
disciplines these sessions; it does not enforce against a hostile client,
which can create its own virtual pointer (that is what sandboxing, not the
lease, would fix).
## Why one frame, not a queue
The server keeps a single slot holding the most recent frame. Each client thread
sends whatever is in that slot when it is ready. A client that cannot keep up
therefore misses frames instead of forcing the server to buffer them: memory use
does not grow with the number of stalled clients, and a stalled client cannot
delay the others. See `docs/CONTRACT.md` for exactly what a client observes.
## Capture
Capturing is a separate program (by default `grim`, a wlr-screencopy client) run
inside the compositor's session and pointed at stdout. That choice is what keeps
`hhstream` free of Wayland and GPU code — the awkward part lives in a tool that
already exists and is already tested — and it is why the capture command is a
module option: tests inject a fake, and a different backend is a swap rather than
a rewrite.
That swap has now happened once. `captureBackend = "wayland"` captures in-process
(`src/hhstream/wayland_capture.py`): one long-lived `zwlr-screencopy-v1` client
with **one shared-memory buffer reused for every frame**, instead of a `grim`
process per frame. Measured at 1280×720 over 200 frames, that is ~1.5× the frame
rate (78 ms against 117 ms per frame) and — the number that matters on this
machine — roughly **half the compositor's CPU per frame** (67 ms against 116 ms),
because `grim` makes the compositor import a new buffer every time. It is
deliberately the *deprecated* protocol: `ext-image-copy-capture-v1` measured
identically and appears in a Hyprland segfault stack (`create_frame` →
`CScreenshareFrame::transform`). The price of the in-process route is that the
package now carries Wayland protocol bindings of its own (`src/hhstream/_protocol/`,
vendored the same way `hhstream_pointer` already vendors its two) and that it can
publish **JPEG** (through libjpeg-turbo) as well as PNG — and JPEG is the one
that matters, because the live viewer serves `image/jpeg` parts and a PNG-only
backend would have broken video streaming while looking faster on a benchmark.
The library is reached through `ctypes` and has to be on `LD_LIBRARY_PATH`, which
the package's wrapper arranges. Both costs are why `command` is still the default:
switching is the operator's call, not something a merge should decide.
Comments in the spike that produced these numbers live in the branch's
`capture-speed` worktree; the honest caveat that came with them is that this box
is saturated by *rendering* whatever is on screen, so the win is in how much room
the compositor keeps, not in headroom that did not exist.
## The hardware requirement, stated plainly
`Hyprland` 0.56.2 asks aquamarine for its **headless backend as mandatory**
(`src/Compositor.cpp`) and does not accept the DRM backend if it is
unavailable. Aquamarine's headless backend still needs an EGL/GBM **render
node**; there is no software-only path that works without one. Consequently a
host with no graphics device at all (`/dev/dri` missing) cannot run this, and
Pyprland fails with `CBackend::create() failed!` before any of its managers come
up.
So the requirements are:
- a render node, i.e. `/dev/dri/renderD*` — from a real GPU, a virtual GPU
(`virtio-gpu`, `virtio-gpu-gl`, QXL) or a NixOS test VM's own QEMU display;
- the `video`/`render` groups for whichever user runs it (the module sets them).
## Testing, and what each layer proves
| Layer | Proves |
|---|---|
| `tests/unit` | the framing, the capture wrapper and the server's client handling, with no compositor and no GPU |
| `checks.vm-basic` | the module actually starts both units, the socket appears with the right owner/mode, the Hello parses and two frames arrive — with a fake capture program |
| `checks.vm-hyprland` | the real thing: headless Hyprland comes up in the VM and `grim` captures a frame of the configured size |
| `checks.vm-sessions` | the session broker side: a real application runs as a tracked session on the compositor, the compositor's IPC sees its window, the idle reaper fires for real (and does not fire when the session is touched), and a broker restart adopts its sessions back |
| `checks.vm-ipc` | both IPC sockets exist at the names the contract freezes and fit in `sun_path`; `hyprctl` answers; and a subscriber on the **event** socket receives an `openwindow` for a window the test creates — the one assertion that can tell a working event socket from a silently truncated one |
| `checks.vm-containers` | one `hh-*` output per session at the container mode, the application's window on it, a second window of the same application beside the first, an unmatched window quarantined instead of landing on whoever had focus, `grim -o` isolating one container, and teardown removing the output and releasing its workspace slot |
| `checks.vm-pointer` | the driver binds a version-2 virtual pointer; acquire warps into the container and restores from the bookmark; a click arrives at the application, which reports the coordinates it was delivered at; a second acquire is BUSY naming the holder; an expired lease is swept and its held button lifted; a broker restart leaves the lock free, the bookmark intact and the driver reachable |
The three session checks share one guest (`checks/lib/session-machine.nix`) and
one set of test-script helpers (`checks/lib/session-prelude.nix`); the helpers
contain no assertions, so each check's script is still a complete account of
what it proves.
`vm-hyprland` is the only check that can fail for reasons outside this repo, so
it is kept separate from `vm-basic` on purpose: when it fails, the compositor's
log is the answer and the other two layers are already known good.
**Run the VM checks one at a time.** Each boots a guest with a
software-rendered compositor; two at once on a two-core machine make the
timing-sensitive assertions flake, and a flaky green is worth less than a slow
one.