The frame server could only capture by running a program once per frame
(grim). That works, but ~95% of a grim frame is waiting -- fork, connect,
handshake, round trip -- not encoding, and grim makes the compositor import
a *new* shared-memory buffer for every single frame.
This adds `--capture wayland[:OUTPUT]`: one long-lived zwlr-screencopy-v1
client per server, with one shm buffer reused for the whole run, capturing
inside the process. Measured here at 1280x720 (spike capture-speed, 200
frames; live re-check on Virtual-1, 60 frames):
grim (subprocess) 117.3 ms/frame 8.1 fps compositor 116.4 ms
grim-rs (Rust crate) 124.6 ms/frame 7.5 fps compositor 123.1 ms
this backend 78.0 ms/frame 13.9 fps compositor 67.5 ms
live, PNG both sides: this 128.5 ms/frame against grim's 389.3 ms (3.0x)
Three findings, each of which shaped the code:
* The win is buffer reuse, not language. grim and grim-rs both hand the
compositor a fresh buffer per frame, which is why their compositor CPU is
nearly double ours. The per-frame allocate-copy-swizzle that grim-rs pays
was measured in isolation at 2.4 ms of *client* time; the rest of its
deficit lands inside the compositor.
* ext-image-copy-capture-v1 is no faster (78.0 vs 77.7 ms) and its
create_frame path appears in a Hyprland segfault stack, so this keeps the
deprecated protocol grim uses. The switch is one constant when that crash
is fixed.
* PNG only, deliberately: PNG is the one encoding this package produces
without a dependency, so --format jpeg is refused rather than publishing a
Hello whose MIME type the frames do not have. A PNG frame is ~2.1 MB
against ~120 KB for JPEG q80, which is why the command backend stays the
default and why a libjpeg-turbo encoder (1.7 ms/frame, measured) is the
obvious follow-up.
The timeout is the load-bearing part of the client: pywayland only reads
events from read()/dispatch(block=True), neither of which takes a timeout,
and the server captures in a worker thread where SIGALRM is unavailable. It
therefore polls the display fd itself, so a compositor that stops answering
produces a CaptureError instead of a capture thread that never returns.
The protocol bindings are vendored under src/hhstream/_protocol/ exactly the
way hhstream_pointer already vendors its two, with regenerate.py recording
where each XML came from. No new dependency: pywayland was already required
by the input driver.
Contract: the new server flag and the two new module options are documented
in docs/CONTRACT.md, and docs/ARCHITECTURE.md's prediction that a native
screencopy client would be "a swap rather than a rewrite" is now the record
of what the swap cost.
Unit suite: 977 passed, 2 skipped, 0 failed (nix build -L .#checks.unit).
The live numbers were taken on this VM against the real compositor.
153 lines
10 KiB
Markdown
153 lines
10 KiB
Markdown
# Architecture
|
||
|
||
```
|
||
┌──────────────────────────────────────────┐
|
||
│ hyprland-headless.service │
|
||
│ Hyprland 0.56.2, no physical monitor │
|
||
│ private XDG_RUNTIME_DIR, as `user` │
|
||
│ outputs come from `outputs[]` │
|
||
│ + one headless output per session │
|
||
└───┬───────────┬───────────────┬──────────┘
|
||
│ │ │
|
||
IPC: request / │ │ Wayland │ Wayland
|
||
event sockets │ │ session │ (virtual pointer)
|
||
│ ▼ │
|
||
│ ┌────────────────────────┴─────────┐
|
||
│ │ hyprland-headless-stream.service │
|
||
│ │ hhstream-server │
|
||
│ │ capture ──▶ one frame slot │
|
||
│ │ accept ──▶ thread per client│
|
||
│ └───────────────┬──────────────────┘
|
||
│ │ AF_UNIX, mode 0660
|
||
│ ▼
|
||
│ many clients ── hhstream-client / any reader
|
||
│
|
||
┌────────────┴──────────────┐ ┌──────────────────────────┐
|
||
│ hhstream-sessions.service │─────▶│ hhstream-pointer.service │
|
||
│ hhstreamd │ │ one Wayland connection │
|
||
│ registry, containers, │ JSON │ one virtual pointer │
|
||
│ the pointer lease │ over │ per output │
|
||
└────────────┬──────────────┘ a └──────────────────────────┘
|
||
│ private socket
|
||
│ AF_UNIX control.sock, mode 0660
|
||
▼
|
||
hhstream session ... (the CLI an agent drives)
|
||
```
|
||
|
||
Five pieces, deliberately separate:
|
||
|
||
1. **The compositor** (`Hyprland`, unchanged upstream) renders into virtual
|
||
outputs. It knows nothing about frames on a socket.
|
||
2. **The frame server** (`hhstream`) captures, holds exactly one frame, and
|
||
publishes it. It owns the socket, the framing and the drop policy.
|
||
3. **The client** is any process that can open a Unix socket and read the
|
||
contract's byte format. The shipped `hhstream-client` is a convenience, not a
|
||
requirement.
|
||
4. **The session broker** (`hhstreamd`) starts allowlisted applications as
|
||
tracked sessions, gives each one its own headless output and workspace (a
|
||
*container*), reads the compositor back over its own IPC, and owns the
|
||
pointer lease. It is the only part that keeps state across a restart.
|
||
5. **The pointer driver** (`hhstream-pointer`) is a Wayland client of the same
|
||
session that creates one `zwlr_virtual_pointer_v1` per output. It holds no
|
||
lease and no session state: the broker decides, the driver executes. It is
|
||
a separate process because a Wayland client needs its own connection and a
|
||
driver blocked on the compositor must not stall session lifecycle.
|
||
|
||
## Why containers are outputs
|
||
|
||
A session's container is a headless **output**, not just a workspace, and the
|
||
two consequences are the reason:
|
||
|
||
- `grim -o hh-<id>` captures exactly one session, so one agent's screenshot
|
||
cannot contain another's window;
|
||
- `zwlr_virtual_pointer_manager_v1` version 2 creates a pointer *bound to an
|
||
output*, so a warp to (0.5, 0.5) is the middle of that session's screen and
|
||
not the middle of the desktop. Version 1 is refused rather than used,
|
||
because its coordinates are global and every click would land somewhere
|
||
plausible and wrong.
|
||
|
||
The cursor itself is singular — there is one seat — so pointer access is
|
||
serialised by a broker-wide **lease**: one holder, a TTL the broker enforces
|
||
with its own sweeper, and buttons lifted on release or expiry. The lease
|
||
disciplines these sessions; it does not enforce against a hostile client,
|
||
which can create its own virtual pointer (that is what sandboxing, not the
|
||
lease, would fix).
|
||
|
||
## Why one frame, not a queue
|
||
|
||
The server keeps a single slot holding the most recent frame. Each client thread
|
||
sends whatever is in that slot when it is ready. A client that cannot keep up
|
||
therefore misses frames instead of forcing the server to buffer them: memory use
|
||
does not grow with the number of stalled clients, and a stalled client cannot
|
||
delay the others. See `docs/CONTRACT.md` for exactly what a client observes.
|
||
|
||
## Capture
|
||
|
||
Capturing is a separate program (by default `grim`, a wlr-screencopy client) run
|
||
inside the compositor's session and pointed at stdout. That choice is what keeps
|
||
`hhstream` free of Wayland and GPU code — the awkward part lives in a tool that
|
||
already exists and is already tested — and it is why the capture command is a
|
||
module option: tests inject a fake, and a different backend is a swap rather than
|
||
a rewrite.
|
||
|
||
That swap has now happened once. `captureBackend = "wayland"` captures in-process
|
||
(`src/hhstream/wayland_capture.py`): one long-lived `zwlr-screencopy-v1` client
|
||
with **one shared-memory buffer reused for every frame**, instead of a `grim`
|
||
process per frame. Measured at 1280×720 over 200 frames, that is ~1.5× the frame
|
||
rate (78 ms against 117 ms per frame) and — the number that matters on this
|
||
machine — roughly **half the compositor's CPU per frame** (67 ms against 116 ms),
|
||
because `grim` makes the compositor import a new buffer every time. It is
|
||
deliberately the *deprecated* protocol: `ext-image-copy-capture-v1` measured
|
||
identically and appears in a Hyprland segfault stack (`create_frame` →
|
||
`CScreenshareFrame::transform`). The price of the in-process route is that the
|
||
package now carries Wayland protocol bindings of its own (`src/hhstream/_protocol/`,
|
||
vendored the same way `hhstream_pointer` already vendors its two) and that it can
|
||
only publish **PNG**, since PNG is the one format it can encode without a
|
||
dependency. Both costs are why `command` is still the default.
|
||
|
||
Comments in the spike that produced these numbers live in the branch's
|
||
`capture-speed` worktree; the honest caveat that came with them is that this box
|
||
is saturated by *rendering* whatever is on screen, so the win is in how much room
|
||
the compositor keeps, not in headroom that did not exist.
|
||
|
||
## The hardware requirement, stated plainly
|
||
|
||
`Hyprland` 0.56.2 asks aquamarine for its **headless backend as mandatory**
|
||
(`src/Compositor.cpp`) and does not accept the DRM backend if it is
|
||
unavailable. Aquamarine's headless backend still needs an EGL/GBM **render
|
||
node**; there is no software-only path that works without one. Consequently a
|
||
host with no graphics device at all (`/dev/dri` missing) cannot run this, and
|
||
Pyprland fails with `CBackend::create() failed!` before any of its managers come
|
||
up.
|
||
|
||
So the requirements are:
|
||
|
||
- a render node, i.e. `/dev/dri/renderD*` — from a real GPU, a virtual GPU
|
||
(`virtio-gpu`, `virtio-gpu-gl`, QXL) or a NixOS test VM's own QEMU display;
|
||
- the `video`/`render` groups for whichever user runs it (the module sets them).
|
||
|
||
## Testing, and what each layer proves
|
||
|
||
| Layer | Proves |
|
||
|---|---|
|
||
| `tests/unit` | the framing, the capture wrapper and the server's client handling, with no compositor and no GPU |
|
||
| `checks.vm-basic` | the module actually starts both units, the socket appears with the right owner/mode, the Hello parses and two frames arrive — with a fake capture program |
|
||
| `checks.vm-hyprland` | the real thing: headless Hyprland comes up in the VM and `grim` captures a frame of the configured size |
|
||
| `checks.vm-sessions` | the session broker side: a real application runs as a tracked session on the compositor, the compositor's IPC sees its window, the idle reaper fires for real (and does not fire when the session is touched), and a broker restart adopts its sessions back |
|
||
| `checks.vm-ipc` | both IPC sockets exist at the names the contract freezes and fit in `sun_path`; `hyprctl` answers; and a subscriber on the **event** socket receives an `openwindow` for a window the test creates — the one assertion that can tell a working event socket from a silently truncated one |
|
||
| `checks.vm-containers` | one `hh-*` output per session at the container mode, the application's window on it, a second window of the same application beside the first, an unmatched window quarantined instead of landing on whoever had focus, `grim -o` isolating one container, and teardown removing the output and releasing its workspace slot |
|
||
| `checks.vm-pointer` | the driver binds a version-2 virtual pointer; acquire warps into the container and restores from the bookmark; a click arrives at the application, which reports the coordinates it was delivered at; a second acquire is BUSY naming the holder; an expired lease is swept and its held button lifted; a broker restart leaves the lock free, the bookmark intact and the driver reachable |
|
||
|
||
The three session checks share one guest (`checks/lib/session-machine.nix`) and
|
||
one set of test-script helpers (`checks/lib/session-prelude.nix`); the helpers
|
||
contain no assertions, so each check's script is still a complete account of
|
||
what it proves.
|
||
|
||
`vm-hyprland` is the only check that can fail for reasons outside this repo, so
|
||
it is kept separate from `vm-basic` on purpose: when it fails, the compositor's
|
||
log is the answer and the other two layers are already known good.
|
||
|
||
**Run the VM checks one at a time.** Each boots a guest with a
|
||
software-rendered compositor; two at once on a two-core machine make the
|
||
timing-sensitive assertions flake, and a flaky green is worth less than a slow
|
||
one. |