Files
hyprland-headless-stream-pl…/spikes/capture-speed/README.md
T
Hermenix 36efa316fd spike: the verdict, the table, the chart -- in-process pywayland wins on zwlr-screencopy
Four routes measured on one frozen contract at one session's container size, with
RAM and CPU as first-class results: in-process pywayland is ~1.5x faster (78 vs
117 ms), uses ~2.8x less of our own CPU, and nearly halves the compositor's
per-frame CPU (67 vs 116 ms). grim-rs buys nothing and costs 746 MB RSS.
ext-image-copy-capture measures identical to screencopy and appears in a
compositor crash stack, so the deprecated-but-working protocol is the pick.

Also fixes the client's unclean exit (pywayland teardown segfaulted after the
result was printed, 16 times, one batch coinciding with a compositor crash).
2026-10-11 17:08:13 +00:00

164 lines
9.8 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# capture-speed — can we drop `grim`, and what would it buy?
**Question** (the operator's): which capture route renders images faster for live
streaming the workspaces to the web and for agent eyes — the shipped `grim`
subprocess, the Rust `grim-rs` crate, or a raw `pywayland` client? And what does
each cost in RAM and CPU?
**Answer**: in-process wins, decisively and cheaply — **~1.5× the frame rate,
~2.8× less of our own CPU, and ~42% less CPU *inside the compositor*** — and
neither Rust nor the newer capture protocol is needed to get it. `grim-rs` is no
faster than the subprocess it would replace and costs **746 MB** of RSS to grim's
18 MB.
## Method
One frozen contract (`HARNESS.md`), one runner (`bench.py`), one target per pass.
- **Target**: a Chromium session on a **CSS-animated** page (no JavaScript, no
timers) so the screen genuinely changes; frames verified to differ between
captures before any run. The still half of the contract lives in `results/still/`
and `results/final-still/` (partial — see "What went wrong").
- **Cells**: one session's container at 1280×720, `raw` and `jpeg q80`, **200
measured frames per run, 2 runs each** (the contract asks 300; 200 is stated
here rather than hidden).
- **Baseline**: `grim` exactly as it ships — one process per frame, fork and
encode inside the timed span.
- **Resources**: producer CPU/RSS (`getrusage`), machine CPU (`/proc/stat`), the
**compositor's own** CPU (`/proc/<Hyprland>/stat`), and an idle **control**
sample with nothing capturing, so an animating page's cost is not attributed
to a route.
- **No GPU**: Hyprland renders through software (kms_swrast/llvmpipe). No route
was tuned to a dma-buf path, and none is claimed. Each producer reports which
protocol actually served its frames.
## Head-to-head — 1280×720, moving screen, one session, sessions run sequentially
| route | mode | p50 ms | fps | own CPU ms | machine CPU ms | **compositor CPU ms** | machine % | **RSS MB** |
|---|---|---|---|---|---|---|---|---|
| `grim` (subprocess, ships today) | jpeg | 117.3 | 8.1 | 5.03 | 247.9 | **116.4** | 199.1 | **18.5** |
| `grim` | raw | 121.4 | 7.9 | 3.86 | 255.2 | 118.4 | 199.4 | 20.5 |
| `grim-rs` (Rust crate) | jpeg | 124.6 | 7.5 | 6.60 | 266.7 | 123.1 | 199.2 | **746.3** |
| `grim-rs` | raw | 120.1 | 7.8 | 1.59 | 254.1 | 119.2 | 198.9 | 746.1 |
| **`pywayland` screencopy** | jpeg | **78.0** | **13.9** | **1.78** | **143.0** | **67.5** | 199.1 | 29.3 |
| `pywayland` screencopy | raw | 76.7 | 14.2 | 0.33 | 140.4 | 67.1 | 198.7 | 24.4 |
| `pywayland` ext-image-copy-capture | jpeg | 77.7 | 14.0 | 1.79 | 142.2 | 67.2 | 199.3 | 29.1 |
| `pywayland` ext-image-copy-capture | raw | 73.5 | 14.4 | 0.31 | 138.5 | 66.2 | 198.9 | 24.5 |
Idle control — nothing capturing at all: **199.1% of one core**, the compositor
alone at 94.1%. Frames are 105–130 KB as JPEG, 2.76 MB as raw PPM.
### What the table says
1. **In-process is ~1.5× faster and much cheaper**: 78 ms vs 117 ms, 13.9 fps vs
8.1, our own CPU 1.8 ms vs 5.0 ms per frame.
2. **The win that matters is the compositor's, not ours**: 67 ms vs 116 ms of
compositor CPU *per frame*. On a 2-core box that is the difference between the
desktop being starved and merely busy.
3. **`grim-rs` is not a candidate**: slightly slower than the subprocess (124.6
vs 117.3 ms), *higher* own CPU (6.6 vs 5.0 ms), and **746 MB RSS** — 40× grim.
Its own notes explain the speed: it allocates a fresh shm buffer and copies
into a `Vec` per capture. The memory figure is reproducible across runs.
4. **screencopy vs ext-image-copy-capture: no measurable difference** (78.0 vs
77.7 ms; compositor 67.5 vs 67.2 ms). The newer protocol buys nothing here, and
its frame-creation path is where one compositor crash landed — so the
recommendation is the older, deprecated-but-working protocol, which is also
what `grim` uses (apples to apples).
5. **The box is saturated in every cell (≈199% of one core)** — and the idle
control is *also* 199%. Rendering the animating screen, not capturing it, is
what fills this machine. No capture change fixes that; what changes is how much
room the compositor has left.
## Where the time actually goes (the finding that reframes it)
`grim` at 1280×720 JPEG q80 spends ~95% of each frame **waiting** — on the fork,
the connect, the compositor's round trip — not encoding: JPEG q80 of a 1280×720
frame costs **1.7 ms** in-process. So the available win is in *not paying the
round trip per frame*, which is exactly what a persistent in-process client does —
and why screencopy and ext-image-copy-capture land within noise of each other.
## What went wrong (each item cost real time; none is hidden)
- **The compositor crashed three times** during measurement (`.Hyprland-wrapped`
SEGV in `CScreenshareFrame::transform()` reached via
`ext-image-copy-capture`'s `create_frame`, plus twice in `libaquamarine`),
taking the stream/sessions/pointer units and both session outputs with it. When
that happened, cells failed and `hhstream status` hung for minutes. **The ext
stack could not be reproduced in isolation** (5 clean frames afterwards), so
this is recorded as *unstable under sustained capture load*, not as "the modern
protocol is broken".
- **Our own client segfaulted on exit 16 times** (`python3.14 … segfault …
libwayland-client.so`) — pywayland/CFFI teardown, *after* the result printed, so
green runs reported exit 139. Fixed with `os._exit(0)` after flushing. A batch
of these coincided in time with a compositor crash, so the fix may matter
beyond tidiness.
- **Two SIGTERM kills of the benchmark** (exit 143, no error line): a 40-minute
`nix-shell` tree does not belong in an agent background process. Moved to a
transient systemd unit, and the runner made resumable.
- **`grim -o NAME` must precede the trailing `-`**, or grim exits 0 having written
**637 bytes of nothing**.
- **grim's only uncompressed format is `ppm`**; `pbm`/`pnm`/`rgb`/`bmp` are
rejected. Raw is *not* cheaper than JPEG for it — writing 2.76 MB down a pipe
costs what encoding costs.
- **turbojpeg reuses a non-NULL `*jpegBuf`**: freeing it and leaving the pointer
makes the next call write into freed memory (heap-corrupting segfault with no
stack trace near the encoder).
- **`display.dispatch(block=False)` never reads the socket** — it processes only
what is already read, so a wait built on it times out against a healthy
compositor. The read is `dispatch(block=True)`.
- **`ext-image-capture-source-v1` has one manager per source kind**
(`ext_output_…`, `ext_foreign_toplevel_…`). Looking for a generic
`ext_image_capture_source_manager_v1` finds nothing and reads as "the modern
protocol is unavailable" — a wrong conclusion this spike drew once. The
toplevel variant means **single-window capture is available**, which matters
for agent eyes.
- **`--all` cells silently changed geometry** (2560→3840) once a second session
existed, so they are excluded from this table rather than averaged.
- **A `--resume` keyed on file existence skips failed cells too**, because a
failure also writes a file; it now requires a parseable result line.
## Verdict: VALIDATED — with a caveat that changes the shape of the answer
- **Replace `grim` with an in-process `pywayland` client on `zwlr-screencopy`**
for the one-session case (live streaming a workspace, agent eyes). ~1.5×
faster, ~2.8× less own CPU, and — the number that decides it — **the
compositor's per-frame CPU nearly halves**.
- **Do not adopt `grim-rs`**: no speed win, 40× the memory.
- **Do not adopt `ext-image-copy-capture` yet** on this Hyprland build: no
measured gain, and its create-frame path appears in a compositor crash stack.
- **Keep raw/ppm out of the wire path**: it is not cheaper than JPEG anywhere
measured.
- **The caveat**: on this software-rendered 2-core VM the machine is saturated by
*rendering* whatever is on screen, in every configuration including idle. The
frame-rate win is real, but no capture change buys headroom — it changes how
much of a saturated box the compositor keeps.
## What a production version would take
- Replace the subprocess behind `src/hhstream/capture.py`'s `CommandCapture` with
a pywayland screencopy client: ~250 lines, vendored protocol XML (the pattern
already used by `src/hhstream_pointer/_protocol/`), and **no new dependency** —
`pywayland` is already a dependency of the package, so the stdlib-only policy
for the frame socket, CLI and broker is untouched.
- JPEG encoding then needs libjpeg-turbo via ctypes (stdlib `zlib` PNG is the
fallback, at 2–3× the bytes) — that *is* a genuine new dependency for that
module, and the one thing worth arguing about in review.
- Single-window capture comes free via
`ext_foreign_toplevel_image_capture_source_manager_v1`; worth a follow-up spike
once the compositor's stability under sustained capture is understood.
## Files
| File | What it is |
|---|---|
| `HARNESS.md` | the frozen measurement contract (argv, JSON, timing boundary, resource fields) |
| `bench.py` | the runner: every route through every cell, raw JSON kept, table derived from the files, `--resume` |
| `producers/grim_producer.py` | route 000, the baseline |
| `producers/wayland_producer.py` | route 001b, raw `pywayland` (screencopy *and* ext-image-copy-capture) |
| `producers/resources.py` | RAM/CPU sampling shared by the Python producers |
| `001a-grim-rs/` | route 001a, the Rust `grim-rs` producer (Rust toolchain confined to its own `shell.nix`) |
| `run-final.sh`, `run-remaining.sh` | the pass drivers, meant to run in a transient systemd unit |
| `chart.py` | the two charts (`python3 chart.py results/final-motion`) |
| `results/final-motion/` | the table above, its chart, and every run's raw stdout/stderr |
| `shell.nix` | the spike's environment — deliberately not in the repo flake |