Four routes measured on one frozen contract at one session's container size, with RAM and CPU as first-class results: in-process pywayland is ~1.5x faster (78 vs 117 ms), uses ~2.8x less of our own CPU, and nearly halves the compositor's per-frame CPU (67 vs 116 ms). grim-rs buys nothing and costs 746 MB RSS. ext-image-copy-capture measures identical to screencopy and appears in a compositor crash stack, so the deprecated-but-working protocol is the pick. Also fixes the client's unclean exit (pywayland teardown segfaulted after the result was printed, 16 times, one batch coinciding with a compositor crash).
164 lines
9.8 KiB
Markdown
164 lines
9.8 KiB
Markdown
# capture-speed — can we drop `grim`, and what would it buy?
|
||
|
||
**Question** (the operator's): which capture route renders images faster for live
|
||
streaming the workspaces to the web and for agent eyes — the shipped `grim`
|
||
subprocess, the Rust `grim-rs` crate, or a raw `pywayland` client? And what does
|
||
each cost in RAM and CPU?
|
||
|
||
**Answer**: in-process wins, decisively and cheaply — **~1.5× the frame rate,
|
||
~2.8× less of our own CPU, and ~42% less CPU *inside the compositor*** — and
|
||
neither Rust nor the newer capture protocol is needed to get it. `grim-rs` is no
|
||
faster than the subprocess it would replace and costs **746 MB** of RSS to grim's
|
||
18 MB.
|
||
|
||
## Method
|
||
|
||
One frozen contract (`HARNESS.md`), one runner (`bench.py`), one target per pass.
|
||
|
||
- **Target**: a Chromium session on a **CSS-animated** page (no JavaScript, no
|
||
timers) so the screen genuinely changes; frames verified to differ between
|
||
captures before any run. The still half of the contract lives in `results/still/`
|
||
and `results/final-still/` (partial — see "What went wrong").
|
||
- **Cells**: one session's container at 1280×720, `raw` and `jpeg q80`, **200
|
||
measured frames per run, 2 runs each** (the contract asks 300; 200 is stated
|
||
here rather than hidden).
|
||
- **Baseline**: `grim` exactly as it ships — one process per frame, fork and
|
||
encode inside the timed span.
|
||
- **Resources**: producer CPU/RSS (`getrusage`), machine CPU (`/proc/stat`), the
|
||
**compositor's own** CPU (`/proc/<Hyprland>/stat`), and an idle **control**
|
||
sample with nothing capturing, so an animating page's cost is not attributed
|
||
to a route.
|
||
- **No GPU**: Hyprland renders through software (kms_swrast/llvmpipe). No route
|
||
was tuned to a dma-buf path, and none is claimed. Each producer reports which
|
||
protocol actually served its frames.
|
||
|
||
## Head-to-head — 1280×720, moving screen, one session, sessions run sequentially
|
||
|
||
| route | mode | p50 ms | fps | own CPU ms | machine CPU ms | **compositor CPU ms** | machine % | **RSS MB** |
|
||
|---|---|---|---|---|---|---|---|---|
|
||
| `grim` (subprocess, ships today) | jpeg | 117.3 | 8.1 | 5.03 | 247.9 | **116.4** | 199.1 | **18.5** |
|
||
| `grim` | raw | 121.4 | 7.9 | 3.86 | 255.2 | 118.4 | 199.4 | 20.5 |
|
||
| `grim-rs` (Rust crate) | jpeg | 124.6 | 7.5 | 6.60 | 266.7 | 123.1 | 199.2 | **746.3** |
|
||
| `grim-rs` | raw | 120.1 | 7.8 | 1.59 | 254.1 | 119.2 | 198.9 | 746.1 |
|
||
| **`pywayland` screencopy** | jpeg | **78.0** | **13.9** | **1.78** | **143.0** | **67.5** | 199.1 | 29.3 |
|
||
| `pywayland` screencopy | raw | 76.7 | 14.2 | 0.33 | 140.4 | 67.1 | 198.7 | 24.4 |
|
||
| `pywayland` ext-image-copy-capture | jpeg | 77.7 | 14.0 | 1.79 | 142.2 | 67.2 | 199.3 | 29.1 |
|
||
| `pywayland` ext-image-copy-capture | raw | 73.5 | 14.4 | 0.31 | 138.5 | 66.2 | 198.9 | 24.5 |
|
||
|
||
Idle control — nothing capturing at all: **199.1% of one core**, the compositor
|
||
alone at 94.1%. Frames are 105–130 KB as JPEG, 2.76 MB as raw PPM.
|
||
|
||
### What the table says
|
||
|
||
1. **In-process is ~1.5× faster and much cheaper**: 78 ms vs 117 ms, 13.9 fps vs
|
||
8.1, our own CPU 1.8 ms vs 5.0 ms per frame.
|
||
2. **The win that matters is the compositor's, not ours**: 67 ms vs 116 ms of
|
||
compositor CPU *per frame*. On a 2-core box that is the difference between the
|
||
desktop being starved and merely busy.
|
||
3. **`grim-rs` is not a candidate**: slightly slower than the subprocess (124.6
|
||
vs 117.3 ms), *higher* own CPU (6.6 vs 5.0 ms), and **746 MB RSS** — 40× grim.
|
||
Its own notes explain the speed: it allocates a fresh shm buffer and copies
|
||
into a `Vec` per capture. The memory figure is reproducible across runs.
|
||
4. **screencopy vs ext-image-copy-capture: no measurable difference** (78.0 vs
|
||
77.7 ms; compositor 67.5 vs 67.2 ms). The newer protocol buys nothing here, and
|
||
its frame-creation path is where one compositor crash landed — so the
|
||
recommendation is the older, deprecated-but-working protocol, which is also
|
||
what `grim` uses (apples to apples).
|
||
5. **The box is saturated in every cell (≈199% of one core)** — and the idle
|
||
control is *also* 199%. Rendering the animating screen, not capturing it, is
|
||
what fills this machine. No capture change fixes that; what changes is how much
|
||
room the compositor has left.
|
||
|
||
## Where the time actually goes (the finding that reframes it)
|
||
|
||
`grim` at 1280×720 JPEG q80 spends ~95% of each frame **waiting** — on the fork,
|
||
the connect, the compositor's round trip — not encoding: JPEG q80 of a 1280×720
|
||
frame costs **1.7 ms** in-process. So the available win is in *not paying the
|
||
round trip per frame*, which is exactly what a persistent in-process client does —
|
||
and why screencopy and ext-image-copy-capture land within noise of each other.
|
||
|
||
## What went wrong (each item cost real time; none is hidden)
|
||
|
||
- **The compositor crashed three times** during measurement (`.Hyprland-wrapped`
|
||
SEGV in `CScreenshareFrame::transform()` reached via
|
||
`ext-image-copy-capture`'s `create_frame`, plus twice in `libaquamarine`),
|
||
taking the stream/sessions/pointer units and both session outputs with it. When
|
||
that happened, cells failed and `hhstream status` hung for minutes. **The ext
|
||
stack could not be reproduced in isolation** (5 clean frames afterwards), so
|
||
this is recorded as *unstable under sustained capture load*, not as "the modern
|
||
protocol is broken".
|
||
- **Our own client segfaulted on exit 16 times** (`python3.14 … segfault …
|
||
libwayland-client.so`) — pywayland/CFFI teardown, *after* the result printed, so
|
||
green runs reported exit 139. Fixed with `os._exit(0)` after flushing. A batch
|
||
of these coincided in time with a compositor crash, so the fix may matter
|
||
beyond tidiness.
|
||
- **Two SIGTERM kills of the benchmark** (exit 143, no error line): a 40-minute
|
||
`nix-shell` tree does not belong in an agent background process. Moved to a
|
||
transient systemd unit, and the runner made resumable.
|
||
- **`grim -o NAME` must precede the trailing `-`**, or grim exits 0 having written
|
||
**637 bytes of nothing**.
|
||
- **grim's only uncompressed format is `ppm`**; `pbm`/`pnm`/`rgb`/`bmp` are
|
||
rejected. Raw is *not* cheaper than JPEG for it — writing 2.76 MB down a pipe
|
||
costs what encoding costs.
|
||
- **turbojpeg reuses a non-NULL `*jpegBuf`**: freeing it and leaving the pointer
|
||
makes the next call write into freed memory (heap-corrupting segfault with no
|
||
stack trace near the encoder).
|
||
- **`display.dispatch(block=False)` never reads the socket** — it processes only
|
||
what is already read, so a wait built on it times out against a healthy
|
||
compositor. The read is `dispatch(block=True)`.
|
||
- **`ext-image-capture-source-v1` has one manager per source kind**
|
||
(`ext_output_…`, `ext_foreign_toplevel_…`). Looking for a generic
|
||
`ext_image_capture_source_manager_v1` finds nothing and reads as "the modern
|
||
protocol is unavailable" — a wrong conclusion this spike drew once. The
|
||
toplevel variant means **single-window capture is available**, which matters
|
||
for agent eyes.
|
||
- **`--all` cells silently changed geometry** (2560→3840) once a second session
|
||
existed, so they are excluded from this table rather than averaged.
|
||
- **A `--resume` keyed on file existence skips failed cells too**, because a
|
||
failure also writes a file; it now requires a parseable result line.
|
||
|
||
## Verdict: VALIDATED — with a caveat that changes the shape of the answer
|
||
|
||
- **Replace `grim` with an in-process `pywayland` client on `zwlr-screencopy`**
|
||
for the one-session case (live streaming a workspace, agent eyes). ~1.5×
|
||
faster, ~2.8× less own CPU, and — the number that decides it — **the
|
||
compositor's per-frame CPU nearly halves**.
|
||
- **Do not adopt `grim-rs`**: no speed win, 40× the memory.
|
||
- **Do not adopt `ext-image-copy-capture` yet** on this Hyprland build: no
|
||
measured gain, and its create-frame path appears in a compositor crash stack.
|
||
- **Keep raw/ppm out of the wire path**: it is not cheaper than JPEG anywhere
|
||
measured.
|
||
- **The caveat**: on this software-rendered 2-core VM the machine is saturated by
|
||
*rendering* whatever is on screen, in every configuration including idle. The
|
||
frame-rate win is real, but no capture change buys headroom — it changes how
|
||
much of a saturated box the compositor keeps.
|
||
|
||
## What a production version would take
|
||
|
||
- Replace the subprocess behind `src/hhstream/capture.py`'s `CommandCapture` with
|
||
a pywayland screencopy client: ~250 lines, vendored protocol XML (the pattern
|
||
already used by `src/hhstream_pointer/_protocol/`), and **no new dependency** —
|
||
`pywayland` is already a dependency of the package, so the stdlib-only policy
|
||
for the frame socket, CLI and broker is untouched.
|
||
- JPEG encoding then needs libjpeg-turbo via ctypes (stdlib `zlib` PNG is the
|
||
fallback, at 2–3× the bytes) — that *is* a genuine new dependency for that
|
||
module, and the one thing worth arguing about in review.
|
||
- Single-window capture comes free via
|
||
`ext_foreign_toplevel_image_capture_source_manager_v1`; worth a follow-up spike
|
||
once the compositor's stability under sustained capture is understood.
|
||
|
||
## Files
|
||
|
||
| File | What it is |
|
||
|---|---|
|
||
| `HARNESS.md` | the frozen measurement contract (argv, JSON, timing boundary, resource fields) |
|
||
| `bench.py` | the runner: every route through every cell, raw JSON kept, table derived from the files, `--resume` |
|
||
| `producers/grim_producer.py` | route 000, the baseline |
|
||
| `producers/wayland_producer.py` | route 001b, raw `pywayland` (screencopy *and* ext-image-copy-capture) |
|
||
| `producers/resources.py` | RAM/CPU sampling shared by the Python producers |
|
||
| `001a-grim-rs/` | route 001a, the Rust `grim-rs` producer (Rust toolchain confined to its own `shell.nix`) |
|
||
| `run-final.sh`, `run-remaining.sh` | the pass drivers, meant to run in a transient systemd unit |
|
||
| `chart.py` | the two charts (`python3 chart.py results/final-motion`) |
|
||
| `results/final-motion/` | the table above, its chart, and every run's raw stdout/stderr |
|
||
| `shell.nix` | the spike's environment — deliberately not in the repo flake |
|