Files
hyprland-headless-stream-pl…/HARNESS.md
T
Hermenix da14048010 spike: measure grim, raw pywayland and grim-rs on one contract, with RAM and CPU
The question is which capture route to ship for live streaming and agent eyes.
Speed alone does not decide it, so every route now also reports producer CPU and
RSS, machine CPU from /proc/stat, the compositor's own CPU, and a per-frame
buffer footprint -- all sampled around the measured loop, with an idle control
sample so an animating page's cost is not attributed to the capture.

The baseline forks a process per frame and spends 95% of each frame waiting, so
the interesting finding is not which route is faster but which one leaves the
compositor able to do its job: in-process acquisition roughly halves the
compositor's per-frame CPU even though the machine saturates either way.
2026-10-11 15:45:09 +00:00

137 lines
7.1 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# The measurement contract — frozen
Two people are building capture producers in two worktrees. If each invents its
own timing, the comparison is worthless. So this file is the interface, and it is
**frozen**: implement to it exactly, do not improve it.
## The producer command
Every route is a command that captures frames and reports its own timings as one
JSON object. Same argv, same output, whatever language it is written in.
```
<producer> --output <wl_output_name> --frames <N> --warmup <W> \
--mode {raw|jpeg} --json
<producer> --all --frames <N> --warmup <W> --mode {raw|jpeg} --json
```
| Flag | Meaning |
|---|---|
| `--output NAME` | capture exactly this `wl_output` (e.g. `hh-<session-id>`, or the main output) |
| `--all` | capture **every** output and stitch them, left to right, into one image — this is the 2560×720 geometry `hhstream` publishes today. Mutually exclusive with `--output`. |
| `--frames N` | **measured** frames (after warm-up) |
| `--warmup W` | discarded frames, taken first, on the same connection |
| `--mode raw` | the frame is complete in a userspace buffer; **no encode, no format conversion** |
| `--mode jpeg` | the frame is complete as JPEG bytes |
| `--json` | print the result object (below) as the **last line of stdout** |
Anything else the producer wants to say goes to **stderr**. Stdout carries
exactly one JSON object, so a runner can parse it without guessing.
## What is timed
**One frame = from just before the capture is requested, to the moment that
frame's bytes are complete in the producer's own userspace buffer.**
- `--mode raw`: stop when the compositor's frame is ready and the buffer is
mapped and readable. Do **not** convert ARGB→RGB, do not apply transforms, do
not touch the pixels. Reads that a consumer would do (stride, checksum) are
outside the measurement.
- `--mode jpeg`: stop when the encoded bytes are complete in the buffer, however
you encode them (library, subprocess, hardware). Encoding is inside the
measurement, because encoding is what a live stream pays.
- For the **subprocess baseline** (`grim`), the timed span is the whole
`fork`+`exec`+run+read: spawn, write to a pipe, read to EOF. That is the honest
cost of the route we are trying to beat, and the fork cost is the point.
- Exclude warm-up frames from every statistic. Use a monotonic clock.
- The first frame after a connection is established is a warm-up frame by
definition — allocate buffers and settle the protocol before measuring.
## The result object
```json
{
"route": "grim-subprocess", // short id of the implementation
"language": "python", // python | rust
"protocol": "zwlr-screencopy-v1", // what actually served the frames
"output": "hh-chromium-yrfadn", // or "all"
"geometry": [1280, 720],
"mode": "jpeg",
"warmup": 10,
"frames": 300,
"frame_ms": { "min": 0.0, "p50": 0.0, "mean": 0.0, "p95": 0.0, "max": 0.0 },
"fps_sustained": 0.0, // frames / (sum of frame_ms / 1000)
"wall_ms": 0.0, // total wall time of the measured run
"cpu_ms_per_frame": null, // process CPU time / frames, or null if unknown
"bytes_p50": null, // frame size in bytes (jpeg mode), else null
"persistent": true, // one long-lived connection, or a process per frame
"notes": "" // anything a reader must know to read the numbers
}
```
### Resource fields (added 2026-10-11 — the operator's requirement)
Speed alone does not decide this. A route that hits 15 fps by burning a core is
not a route we can ship on a machine that also renders. Five more fields are
**required**, measured over the same measured window as `frame_ms`:
```json
"rss_peak_mb": 0.0, // peak RSS of the producer (children included for subprocess routes)
"rss_peak_kb_raw": 0, // the unrounded value, so nobody re-derives a rounded one
"system_cpu_ms_per_frame": 0.0, // every process on the machine, from /proc/stat
"compositor_cpu_ms_per_frame": 0.0, // the compositor's own utime+stime, from /proc/<hyprland>/stat
"machine_cpu_pct": 0.0, // system_cpu over the window, as % of one core (can exceed 100)
"frames_bytes_allocated_p50": null, // bytes of buffer the route holds per frame (raw mode: the whole frame)
```
- `system_cpu_ms_per_frame` is the honest "what this costs the machine" number,
and it is the one that catches a route which is fast only because the
compositor did extra work. Report it even when it is unflattering.
- `compositor_cpu_ms_per_frame` separates "the capture is expensive" from "the
capture made the compositor expensive". Read `/proc/<pid>/stat` fields 14+15
(utime+stime, in clock ticks — divide by `os.sysconf("SC_CLK_TCK")`). If the
file is unreadable, say so in `notes` rather than reporting zero.
- `rss_peak_mb`: `resource.getrusage(RUSAGE_SELF).ru_maxrss` for a persistent
route; `RUSAGE_CHILDREN` for a subprocess-per-frame route (it is the peak of
the children, which is what actually matters there). ru_maxrss is in **KiB**
on Linux, despite the name.
- These are measured *around the measured loop only* — warm-up excluded — so
they line up with `frame_ms`.
### The conditions a run is reported under
- `protocol` must name what really ran, not what you hoped: if
`ext-image-copy-capture-v1` is unavailable and you fell back to
`zwlr-screencopy-v1`, say so. If you had to fall back at runtime, say that too.
- `persistent` separates "one connection, N frames" from "a process per frame" —
the two are not comparable and the table must not mix them silently.
- `cpu_ms_per_frame` is process CPU (`/proc/self/stat` utime+stime, or
`getrusage`) divided by frames. A route that is fast by pinning a core is not
fast; report it.
- Numbers with no measurement behind them stay `null`. Never estimate a field.
## The conditions a run is reported under
The runner runs every route under identical conditions and records them:
1. **Geometry**: `--output <one container>` (1280×720) and `--all` (2560×720).
2. **Screen**: a *still* pass and a *moving* pass. Motion is made by a session
printing in a loop or a page being scrolled; the runner starts it, waits for
it to be visibly moving, and destroys it afterwards. A still screen can make a
damage-driven capture wait — if that happens, it is a **finding** and goes in
the verdict, not in a footnote.
3. **Mode**: `raw` and `jpeg`.
4. **Repetition**: every cell twice; report both runs, not just the better one.
## The rule that keeps this honest
**The baseline is `grim` as it runs today.** Whatever beats it, beats that, and
the verdict has to say by how much — or that nothing did. A route that is only
faster in a mode the live stream does not use (raw pixels, when the wire carries
JPEG) must be labelled as such rather than presented as a win.
And the environment caveat that goes in every verdict: **this VM has no GPU.**
Hyprland renders through software (kms_swrast / llvmpipe). Zero-copy dma-buf
paths cannot help here, so a route must not be tuned to take one; say what a
GPU machine would change, but do not claim it as measured.