# capture-speed — can we drop `grim`, and what would it buy? **Question** (the operator's): which capture route renders images faster for live streaming the workspaces to the web and for agent eyes — the shipped `grim` subprocess, the Rust `grim-rs` crate, or a raw `pywayland` client? And what does each cost in RAM and CPU? **Answer**: in-process wins, decisively and cheaply — **~1.5× the frame rate, ~2.8× less of our own CPU, and ~42% less CPU *inside the compositor*** — and neither Rust nor the newer capture protocol is needed to get it. `grim-rs` is no faster than the subprocess it would replace and costs **746 MB** of RSS to grim's 18 MB. ## Method One frozen contract (`HARNESS.md`), one runner (`bench.py`), one target per pass. - **Target**: a Chromium session on a **CSS-animated** page (no JavaScript, no timers) so the screen genuinely changes; frames verified to differ between captures before any run. The still half of the contract lives in `results/still/` and `results/final-still/` (partial — see "What went wrong"). - **Cells**: one session's container at 1280×720, `raw` and `jpeg q80`, **200 measured frames per run, 2 runs each** (the contract asks 300; 200 is stated here rather than hidden). - **Baseline**: `grim` exactly as it ships — one process per frame, fork and encode inside the timed span. - **Resources**: producer CPU/RSS (`getrusage`), machine CPU (`/proc/stat`), the **compositor's own** CPU (`/proc//stat`), and an idle **control** sample with nothing capturing, so an animating page's cost is not attributed to a route. - **No GPU**: Hyprland renders through software (kms_swrast/llvmpipe). No route was tuned to a dma-buf path, and none is claimed. Each producer reports which protocol actually served its frames. ## Head-to-head — 1280×720, moving screen, one session, sessions run sequentially | route | mode | p50 ms | fps | own CPU ms | machine CPU ms | **compositor CPU ms** | machine % | **RSS MB** | |---|---|---|---|---|---|---|---|---| | `grim` (subprocess, ships today) | jpeg | 117.3 | 8.1 | 5.03 | 247.9 | **116.4** | 199.1 | **18.5** | | `grim` | raw | 121.4 | 7.9 | 3.86 | 255.2 | 118.4 | 199.4 | 20.5 | | `grim-rs` (Rust crate) | jpeg | 124.6 | 7.5 | 6.60 | 266.7 | 123.1 | 199.2 | **746.3** | | `grim-rs` | raw | 120.1 | 7.8 | 1.59 | 254.1 | 119.2 | 198.9 | 746.1 | | **`pywayland` screencopy** | jpeg | **78.0** | **13.9** | **1.78** | **143.0** | **67.5** | 199.1 | 29.3 | | `pywayland` screencopy | raw | 76.7 | 14.2 | 0.33 | 140.4 | 67.1 | 198.7 | 24.4 | | `pywayland` ext-image-copy-capture | jpeg | 77.7 | 14.0 | 1.79 | 142.2 | 67.2 | 199.3 | 29.1 | | `pywayland` ext-image-copy-capture | raw | 73.5 | 14.4 | 0.31 | 138.5 | 66.2 | 198.9 | 24.5 | Idle control — nothing capturing at all: **199.1% of one core**, the compositor alone at 94.1%. Frames are 105–130 KB as JPEG, 2.76 MB as raw PPM. ### What the table says 1. **In-process is ~1.5× faster and much cheaper**: 78 ms vs 117 ms, 13.9 fps vs 8.1, our own CPU 1.8 ms vs 5.0 ms per frame. 2. **The win that matters is the compositor's, not ours**: 67 ms vs 116 ms of compositor CPU *per frame*. On a 2-core box that is the difference between the desktop being starved and merely busy. 3. **`grim-rs` is not a candidate**: slightly slower than the subprocess (124.6 vs 117.3 ms), *higher* own CPU (6.6 vs 5.0 ms), and **746 MB RSS** — 40× grim. Its own notes explain the speed: it allocates a fresh shm buffer and copies into a `Vec` per capture. The memory figure is reproducible across runs. 4. **screencopy vs ext-image-copy-capture: no measurable difference** (78.0 vs 77.7 ms; compositor 67.5 vs 67.2 ms). The newer protocol buys nothing here, and its frame-creation path is where one compositor crash landed — so the recommendation is the older, deprecated-but-working protocol, which is also what `grim` uses (apples to apples). 5. **The box is saturated in every cell (≈199% of one core)** — and the idle control is *also* 199%. Rendering the animating screen, not capturing it, is what fills this machine. No capture change fixes that; what changes is how much room the compositor has left. ## Where the time actually goes (the finding that reframes it) `grim` at 1280×720 JPEG q80 spends ~95% of each frame **waiting** — on the fork, the connect, the compositor's round trip — not encoding: JPEG q80 of a 1280×720 frame costs **1.7 ms** in-process. So the available win is in *not paying the round trip per frame*, which is exactly what a persistent in-process client does — and why screencopy and ext-image-copy-capture land within noise of each other. ## What went wrong (each item cost real time; none is hidden) - **The compositor crashed three times** during measurement (`.Hyprland-wrapped` SEGV in `CScreenshareFrame::transform()` reached via `ext-image-copy-capture`'s `create_frame`, plus twice in `libaquamarine`), taking the stream/sessions/pointer units and both session outputs with it. When that happened, cells failed and `hhstream status` hung for minutes. **The ext stack could not be reproduced in isolation** (5 clean frames afterwards), so this is recorded as *unstable under sustained capture load*, not as "the modern protocol is broken". - **Our own client segfaulted on exit 16 times** (`python3.14 … segfault … libwayland-client.so`) — pywayland/CFFI teardown, *after* the result printed, so green runs reported exit 139. Fixed with `os._exit(0)` after flushing. A batch of these coincided in time with a compositor crash, so the fix may matter beyond tidiness. - **Two SIGTERM kills of the benchmark** (exit 143, no error line): a 40-minute `nix-shell` tree does not belong in an agent background process. Moved to a transient systemd unit, and the runner made resumable. - **`grim -o NAME` must precede the trailing `-`**, or grim exits 0 having written **637 bytes of nothing**. - **grim's only uncompressed format is `ppm`**; `pbm`/`pnm`/`rgb`/`bmp` are rejected. Raw is *not* cheaper than JPEG for it — writing 2.76 MB down a pipe costs what encoding costs. - **turbojpeg reuses a non-NULL `*jpegBuf`**: freeing it and leaving the pointer makes the next call write into freed memory (heap-corrupting segfault with no stack trace near the encoder). - **`display.dispatch(block=False)` never reads the socket** — it processes only what is already read, so a wait built on it times out against a healthy compositor. The read is `dispatch(block=True)`. - **`ext-image-capture-source-v1` has one manager per source kind** (`ext_output_…`, `ext_foreign_toplevel_…`). Looking for a generic `ext_image_capture_source_manager_v1` finds nothing and reads as "the modern protocol is unavailable" — a wrong conclusion this spike drew once. The toplevel variant means **single-window capture is available**, which matters for agent eyes. - **`--all` cells silently changed geometry** (2560→3840) once a second session existed, so they are excluded from this table rather than averaged. - **A `--resume` keyed on file existence skips failed cells too**, because a failure also writes a file; it now requires a parseable result line. ## Verdict: VALIDATED — with a caveat that changes the shape of the answer - **Replace `grim` with an in-process `pywayland` client on `zwlr-screencopy`** for the one-session case (live streaming a workspace, agent eyes). ~1.5× faster, ~2.8× less own CPU, and — the number that decides it — **the compositor's per-frame CPU nearly halves**. - **Do not adopt `grim-rs`**: no speed win, 40× the memory. - **Do not adopt `ext-image-copy-capture` yet** on this Hyprland build: no measured gain, and its create-frame path appears in a compositor crash stack. - **Keep raw/ppm out of the wire path**: it is not cheaper than JPEG anywhere measured. - **The caveat**: on this software-rendered 2-core VM the machine is saturated by *rendering* whatever is on screen, in every configuration including idle. The frame-rate win is real, but no capture change buys headroom — it changes how much of a saturated box the compositor keeps. ## What a production version would take - Replace the subprocess behind `src/hhstream/capture.py`'s `CommandCapture` with a pywayland screencopy client: ~250 lines, vendored protocol XML (the pattern already used by `src/hhstream_pointer/_protocol/`), and **no new dependency** — `pywayland` is already a dependency of the package, so the stdlib-only policy for the frame socket, CLI and broker is untouched. - JPEG encoding then needs libjpeg-turbo via ctypes (stdlib `zlib` PNG is the fallback, at 2–3× the bytes) — that *is* a genuine new dependency for that module, and the one thing worth arguing about in review. - Single-window capture comes free via `ext_foreign_toplevel_image_capture_source_manager_v1`; worth a follow-up spike once the compositor's stability under sustained capture is understood. ## Files | File | What it is | |---|---| | `HARNESS.md` | the frozen measurement contract (argv, JSON, timing boundary, resource fields) | | `bench.py` | the runner: every route through every cell, raw JSON kept, table derived from the files, `--resume` | | `producers/grim_producer.py` | route 000, the baseline | | `producers/wayland_producer.py` | route 001b, raw `pywayland` (screencopy *and* ext-image-copy-capture) | | `producers/resources.py` | RAM/CPU sampling shared by the Python producers | | `001a-grim-rs/` | route 001a, the Rust `grim-rs` producer (Rust toolchain confined to its own `shell.nix`) | | `run-final.sh`, `run-remaining.sh` | the pass drivers, meant to run in a transient systemd unit | | `chart.py` | the two charts (`python3 chart.py results/final-motion`) | | `results/final-motion/` | the table above, its chart, and every run's raw stdout/stderr | | `shell.nix` | the spike's environment — deliberately not in the repo flake |