PNG-only was a real defect, not a nice-to-have: `hhstream_live.server` serves
`multipart/x-mixed-replace` with `image/jpeg` parts and its own capture default is
`grim -c -t jpeg -q 80 -`, so switching this backend on would have broken video
streaming while looking faster in a benchmark.
JPEG now goes through libjpeg-turbo by ctypes, using its `TJPF_BGRX` input format
-- which is exactly the compositor's B, G, R, X memory order, so the frame goes
into the encoder with no copy and no colour conversion. Measured on this VM
against the real compositor, the deployment's own settings (Virtual-1, 1280x720,
jpeg q80, 200 frames each, two runs):
p50 ms fps our CPU compositor CPU
in-process 76.2 13.1 2.1 ms 51.5 ms
grim -c -t jpeg 137.0 7.3 0.5 ms 80.2 ms
(second run: 84/144,
55.1/79.8 -> 1.71x)
So ~1.7-1.8x the frame rate and ~1.5x less of the compositor's CPU, with frames
of the same order (188 KB against grim's 222 KB). PNG stays available through the
stdlib for a deployment that would rather not carry the library.
Both segfault-class traps from the spike are handled and documented: every
argtype is declared (an undeclared pitch is a crash, not a wrong number), and
`*jpegBuf` is reset to NULL after `tjFree` (turbojpeg reads a non-NULL pointer as
a caller buffer, so the next frame would write into freed memory). The decoder is
here because a lossy codec can only be tested by round-tripping pixels -- the new
suite encodes solid blocks whose channels differ enough that a swap is
unmistakable, and includes a padded stride.
Packaging: libjpeg-turbo is reached through ctypes, so nothing links it and the
path must reach the loader -- the console scripts get a wrapper with it on
LD_LIBRARY_PATH, and the unit check exports it too. A missing library is reported
at startup rather than once per frame: a server that is up but can never produce
a frame is worse than one that refuses to start.
Verified end to end: server with `--capture wayland:Virtual-1 --format jpeg`,
client Hello `image/jpeg`, three frames read, each magic `ffd8`.
157 lines
10 KiB
Markdown
157 lines
10 KiB
Markdown
# Architecture
|
||
|
||
```
|
||
┌──────────────────────────────────────────┐
|
||
│ hyprland-headless.service │
|
||
│ Hyprland 0.56.2, no physical monitor │
|
||
│ private XDG_RUNTIME_DIR, as `user` │
|
||
│ outputs come from `outputs[]` │
|
||
│ + one headless output per session │
|
||
└───┬───────────┬───────────────┬──────────┘
|
||
│ │ │
|
||
IPC: request / │ │ Wayland │ Wayland
|
||
event sockets │ │ session │ (virtual pointer)
|
||
│ ▼ │
|
||
│ ┌────────────────────────┴─────────┐
|
||
│ │ hyprland-headless-stream.service │
|
||
│ │ hhstream-server │
|
||
│ │ capture ──▶ one frame slot │
|
||
│ │ accept ──▶ thread per client│
|
||
│ └───────────────┬──────────────────┘
|
||
│ │ AF_UNIX, mode 0660
|
||
│ ▼
|
||
│ many clients ── hhstream-client / any reader
|
||
│
|
||
┌────────────┴──────────────┐ ┌──────────────────────────┐
|
||
│ hhstream-sessions.service │─────▶│ hhstream-pointer.service │
|
||
│ hhstreamd │ │ one Wayland connection │
|
||
│ registry, containers, │ JSON │ one virtual pointer │
|
||
│ the pointer lease │ over │ per output │
|
||
└────────────┬──────────────┘ a └──────────────────────────┘
|
||
│ private socket
|
||
│ AF_UNIX control.sock, mode 0660
|
||
▼
|
||
hhstream session ... (the CLI an agent drives)
|
||
```
|
||
|
||
Five pieces, deliberately separate:
|
||
|
||
1. **The compositor** (`Hyprland`, unchanged upstream) renders into virtual
|
||
outputs. It knows nothing about frames on a socket.
|
||
2. **The frame server** (`hhstream`) captures, holds exactly one frame, and
|
||
publishes it. It owns the socket, the framing and the drop policy.
|
||
3. **The client** is any process that can open a Unix socket and read the
|
||
contract's byte format. The shipped `hhstream-client` is a convenience, not a
|
||
requirement.
|
||
4. **The session broker** (`hhstreamd`) starts allowlisted applications as
|
||
tracked sessions, gives each one its own headless output and workspace (a
|
||
*container*), reads the compositor back over its own IPC, and owns the
|
||
pointer lease. It is the only part that keeps state across a restart.
|
||
5. **The pointer driver** (`hhstream-pointer`) is a Wayland client of the same
|
||
session that creates one `zwlr_virtual_pointer_v1` per output. It holds no
|
||
lease and no session state: the broker decides, the driver executes. It is
|
||
a separate process because a Wayland client needs its own connection and a
|
||
driver blocked on the compositor must not stall session lifecycle.
|
||
|
||
## Why containers are outputs
|
||
|
||
A session's container is a headless **output**, not just a workspace, and the
|
||
two consequences are the reason:
|
||
|
||
- `grim -o hh-<id>` captures exactly one session, so one agent's screenshot
|
||
cannot contain another's window;
|
||
- `zwlr_virtual_pointer_manager_v1` version 2 creates a pointer *bound to an
|
||
output*, so a warp to (0.5, 0.5) is the middle of that session's screen and
|
||
not the middle of the desktop. Version 1 is refused rather than used,
|
||
because its coordinates are global and every click would land somewhere
|
||
plausible and wrong.
|
||
|
||
The cursor itself is singular — there is one seat — so pointer access is
|
||
serialised by a broker-wide **lease**: one holder, a TTL the broker enforces
|
||
with its own sweeper, and buttons lifted on release or expiry. The lease
|
||
disciplines these sessions; it does not enforce against a hostile client,
|
||
which can create its own virtual pointer (that is what sandboxing, not the
|
||
lease, would fix).
|
||
|
||
## Why one frame, not a queue
|
||
|
||
The server keeps a single slot holding the most recent frame. Each client thread
|
||
sends whatever is in that slot when it is ready. A client that cannot keep up
|
||
therefore misses frames instead of forcing the server to buffer them: memory use
|
||
does not grow with the number of stalled clients, and a stalled client cannot
|
||
delay the others. See `docs/CONTRACT.md` for exactly what a client observes.
|
||
|
||
## Capture
|
||
|
||
Capturing is a separate program (by default `grim`, a wlr-screencopy client) run
|
||
inside the compositor's session and pointed at stdout. That choice is what keeps
|
||
`hhstream` free of Wayland and GPU code — the awkward part lives in a tool that
|
||
already exists and is already tested — and it is why the capture command is a
|
||
module option: tests inject a fake, and a different backend is a swap rather than
|
||
a rewrite.
|
||
|
||
That swap has now happened once. `captureBackend = "wayland"` captures in-process
|
||
(`src/hhstream/wayland_capture.py`): one long-lived `zwlr-screencopy-v1` client
|
||
with **one shared-memory buffer reused for every frame**, instead of a `grim`
|
||
process per frame. Measured at 1280×720 over 200 frames, that is ~1.5× the frame
|
||
rate (78 ms against 117 ms per frame) and — the number that matters on this
|
||
machine — roughly **half the compositor's CPU per frame** (67 ms against 116 ms),
|
||
because `grim` makes the compositor import a new buffer every time. It is
|
||
deliberately the *deprecated* protocol: `ext-image-copy-capture-v1` measured
|
||
identically and appears in a Hyprland segfault stack (`create_frame` →
|
||
`CScreenshareFrame::transform`). The price of the in-process route is that the
|
||
package now carries Wayland protocol bindings of its own (`src/hhstream/_protocol/`,
|
||
vendored the same way `hhstream_pointer` already vendors its two) and that it can
|
||
publish **JPEG** (through libjpeg-turbo) as well as PNG — and JPEG is the one
|
||
that matters, because the live viewer serves `image/jpeg` parts and a PNG-only
|
||
backend would have broken video streaming while looking faster on a benchmark.
|
||
The library is reached through `ctypes` and has to be on `LD_LIBRARY_PATH`, which
|
||
the package's wrapper arranges. Both costs are why `command` is still the default:
|
||
switching is the operator's call, not something a merge should decide.
|
||
|
||
Comments in the spike that produced these numbers live in the branch's
|
||
`capture-speed` worktree; the honest caveat that came with them is that this box
|
||
is saturated by *rendering* whatever is on screen, so the win is in how much room
|
||
the compositor keeps, not in headroom that did not exist.
|
||
|
||
## The hardware requirement, stated plainly
|
||
|
||
`Hyprland` 0.56.2 asks aquamarine for its **headless backend as mandatory**
|
||
(`src/Compositor.cpp`) and does not accept the DRM backend if it is
|
||
unavailable. Aquamarine's headless backend still needs an EGL/GBM **render
|
||
node**; there is no software-only path that works without one. Consequently a
|
||
host with no graphics device at all (`/dev/dri` missing) cannot run this, and
|
||
Pyprland fails with `CBackend::create() failed!` before any of its managers come
|
||
up.
|
||
|
||
So the requirements are:
|
||
|
||
- a render node, i.e. `/dev/dri/renderD*` — from a real GPU, a virtual GPU
|
||
(`virtio-gpu`, `virtio-gpu-gl`, QXL) or a NixOS test VM's own QEMU display;
|
||
- the `video`/`render` groups for whichever user runs it (the module sets them).
|
||
|
||
## Testing, and what each layer proves
|
||
|
||
| Layer | Proves |
|
||
|---|---|
|
||
| `tests/unit` | the framing, the capture wrapper and the server's client handling, with no compositor and no GPU |
|
||
| `checks.vm-basic` | the module actually starts both units, the socket appears with the right owner/mode, the Hello parses and two frames arrive — with a fake capture program |
|
||
| `checks.vm-hyprland` | the real thing: headless Hyprland comes up in the VM and `grim` captures a frame of the configured size |
|
||
| `checks.vm-sessions` | the session broker side: a real application runs as a tracked session on the compositor, the compositor's IPC sees its window, the idle reaper fires for real (and does not fire when the session is touched), and a broker restart adopts its sessions back |
|
||
| `checks.vm-ipc` | both IPC sockets exist at the names the contract freezes and fit in `sun_path`; `hyprctl` answers; and a subscriber on the **event** socket receives an `openwindow` for a window the test creates — the one assertion that can tell a working event socket from a silently truncated one |
|
||
| `checks.vm-containers` | one `hh-*` output per session at the container mode, the application's window on it, a second window of the same application beside the first, an unmatched window quarantined instead of landing on whoever had focus, `grim -o` isolating one container, and teardown removing the output and releasing its workspace slot |
|
||
| `checks.vm-pointer` | the driver binds a version-2 virtual pointer; acquire warps into the container and restores from the bookmark; a click arrives at the application, which reports the coordinates it was delivered at; a second acquire is BUSY naming the holder; an expired lease is swept and its held button lifted; a broker restart leaves the lock free, the bookmark intact and the driver reachable |
|
||
|
||
The three session checks share one guest (`checks/lib/session-machine.nix`) and
|
||
one set of test-script helpers (`checks/lib/session-prelude.nix`); the helpers
|
||
contain no assertions, so each check's script is still a complete account of
|
||
what it proves.
|
||
|
||
`vm-hyprland` is the only check that can fail for reasons outside this repo, so
|
||
it is kept separate from `vm-basic` on purpose: when it fails, the compositor's
|
||
log is the answer and the other two layers are already known good.
|
||
|
||
**Run the VM checks one at a time.** Each boots a guest with a
|
||
software-rendered compositor; two at once on a two-core machine make the
|
||
timing-sensitive assertions flake, and a flaky green is worth less than a slow
|
||
one. |