capture: encode JPEG in-process too, so the live stream keeps working

PNG-only was a real defect, not a nice-to-have: `hhstream_live.server` serves
`multipart/x-mixed-replace` with `image/jpeg` parts and its own capture default is
`grim -c -t jpeg -q 80 -`, so switching this backend on would have broken video
streaming while looking faster in a benchmark.

JPEG now goes through libjpeg-turbo by ctypes, using its `TJPF_BGRX` input format
-- which is exactly the compositor's B, G, R, X memory order, so the frame goes
into the encoder with no copy and no colour conversion. Measured on this VM
against the real compositor, the deployment's own settings (Virtual-1, 1280x720,
jpeg q80, 200 frames each, two runs):

                        p50 ms   fps    our CPU   compositor CPU
    in-process           76.2   13.1     2.1 ms      51.5 ms
    grim -c -t jpeg     137.0    7.3     0.5 ms      80.2 ms
                                                          (second run: 84/144,
                                                           55.1/79.8 -> 1.71x)

So ~1.7-1.8x the frame rate and ~1.5x less of the compositor's CPU, with frames
of the same order (188 KB against grim's 222 KB). PNG stays available through the
stdlib for a deployment that would rather not carry the library.

Both segfault-class traps from the spike are handled and documented: every
argtype is declared (an undeclared pitch is a crash, not a wrong number), and
`*jpegBuf` is reset to NULL after `tjFree` (turbojpeg reads a non-NULL pointer as
a caller buffer, so the next frame would write into freed memory). The decoder is
here because a lossy codec can only be tested by round-tripping pixels -- the new
suite encodes solid blocks whose channels differ enough that a swap is
unmistakable, and includes a padded stride.

Packaging: libjpeg-turbo is reached through ctypes, so nothing links it and the
path must reach the loader -- the console scripts get a wrapper with it on
LD_LIBRARY_PATH, and the unit check exports it too. A missing library is reported
at startup rather than once per frame: a server that is up but can never produce
a frame is worse than one that refuses to start.

Verified end to end: server with `--capture wayland:Virtual-1 --format jpeg`,
client Hello `image/jpeg`, three frames read, each magic `ffd8`.
This commit is contained in:
Hermenix committed 2026-10-11 22:58:28 +00:00
1 parent a813e0b4f8
commit 7190960120
10 files changed
+553 -53

No files matched your search

+10
View File
@@ -13,6 +13,7 @@
# the suite runs in a build.
{
lib,
libjpeg_turbo,
runCommand,
python3,
hhstream,
@@ -26,6 +27,8 @@ let
# whether a bad string types a prefix.
pytestEnv = python3.withPackages (ps: [ ps.pytest ps.pywayland ps.xkbcommon ]);
# Only the suite itself, so an editor's droppings or a local .pytest_cache
# cannot change the derivation's hash.
testSrc = lib.cleanSourceWith {
@@ -62,6 +65,13 @@ runCommand "hhstream-unit"
# than letting pytest do it makes a packaging problem distinguishable from a
# test failure.
export PYTHONPATH=${hhstream}/${python3.sitePackages}
# `tests/unit/test_jpeg.py` round-trips a real frame through libjpeg-turbo: a
# lossy codec can only be checked by comparing pixels, so the suite needs the
# library -- and it needs the *loader* to find it, because the encoder reaches
# it through `ctypes` and nothing links it. Same reason nix/package.nix wraps
# the console scripts with this path.
export LD_LIBRARY_PATH=${libjpeg_turbo}/lib
export PYTHONDONTWRITEBYTECODE=1
export HOME=$PWD
python3 -c 'import hhstream; print("testing hhstream", hhstream.__version__, "from", hhstream.__file__)'
+6 -2
View File
@@ -102,8 +102,12 @@ identically and appears in a Hyprland segfault stack (`create_frame` →
`CScreenshareFrame::transform`). The price of the in-process route is that the
package now carries Wayland protocol bindings of its own (`src/hhstream/_protocol/`,
vendored the same way `hhstream_pointer` already vendors its two) and that it can
only publish **PNG**, since PNG is the one format it can encode without a
dependency. Both costs are why `command` is still the default.
publish **JPEG** (through libjpeg-turbo) as well as PNG — and JPEG is the one
that matters, because the live viewer serves `image/jpeg` parts and a PNG-only
backend would have broken video streaming while looking faster on a benchmark.
The library is reached through `ctypes` and has to be on `LD_LIBRARY_PATH`, which
the package's wrapper arranges. Both costs are why `command` is still the default:
switching is the operator's call, not something a merge should decide.
Comments in the spike that produced these numbers live in the branch's
`capture-speed` worktree; the honest caveat that came with them is that this box
+8 -5
View File
@@ -94,7 +94,7 @@ Namespace: `services.hyprland-headless-stream`
| `hyprlandPackage` | package | `pkgs.hyprland` | which compositor |
| `package` | package | the flake's own build | the frame server/client |
| `captureCommand` | list of str | `[ "grim" … ]` | override the capture *command* — used by tests |
| `captureBackend` | `"command"` \| `"wayland"` | `"command"` | capture by command or in-process (`zwlr-screencopy-v1`); `wayland` needs `format = "png"` |
| `captureBackend` | `"command"` \| `"wayland"` | `"command"` | capture by command or in-process (`zwlr-screencopy-v1`); both `format` values work |
| `captureOutput` | str or null | `null` | output to capture by name when `captureBackend = "wayland"`; null = the first one |
Services (names are part of the contract, tests refer to them):
@@ -111,6 +111,7 @@ status as the wire format. Exactly this shape, and no others:
```
hhstream-server --socket <socketPath> --fps <frameRate> --format <png|jpeg> \
[--capture-quality <1..100>] \
[--capture-command <token> ...] [--capture wayland[:NAME]]
```
@@ -124,10 +125,12 @@ hhstream-server --socket <socketPath> --fps <frameRate> --format <png|jpeg> \
- `--capture` selects a **named backend** instead of a program: `wayland`
captures the compositor's first output in-process, `wayland:NAME` the output
Hyprland calls `NAME`. It is exclusive with `--capture-command` (the backend
replaces the command; passing both is a caller error). It requires
`--format png`, and the server refuses `jpeg` rather than publish a Hello whose
MIME type the frames do not have — the in-process client can encode PNG without
a dependency and nothing else.
replaces the command; passing both is a caller error). It honours both
`--format` values: `png` through the stdlib, `jpeg` through libjpeg-turbo. If
libjpeg-turbo is missing and `--format jpeg` was asked for, the server **refuses
to start** rather than serving a socket it can never produce a frame for.
- `--capture-quality` is the JPEG quality for the in-process backend (1..100,
default 80, matching the default command's `-q 80`); `--format png` ignores it.
- There is deliberately **no** `--width` / `--height`: the Hello reports the
captured frame's real size (see above).
- The socket's **mode** (0660) is the server's business; its **ownership** is
+8 -19
View File
@@ -418,11 +418,14 @@ in {
per frame, because it reuses one shared-memory buffer instead of making
`grim` build a new one per frame.
It publishes **PNG only**: PNG is the one encoding this package can do
without a dependency, so {option}`format` must be `png` when this is set.
That is a real trade -- a JPEG frame is ~105-130 KB against ~580 KB for
the same frame as PNG -- and it is the reason the default is still
`command`.
Both of the {option}`format` values work, and in practice this means
`jpeg`: the live viewer serves `image/jpeg` parts, so a PNG-only backend
would break video streaming. JPEG is encoded in-process through
libjpeg-turbo at the same quality the default `grim` command uses
(~1.7 ms a frame, ~120 KB), and PNG remains available through the stdlib
for a deployment that would rather not carry that library (~2.1 MB a
frame). The default is still `command` only because switching backends
is the operator's call, not because anything is missing.
'';
};
@@ -822,20 +825,6 @@ in {
assertion = lib.hasPrefix "/" cfg.socketPath;
message = "services.hyprland-headless-stream.socketPath must be an absolute path.";
}
# The Hello carries the frame's MIME type, and the in-process backend can
# only produce PNG. A jpeg setting here would publish a format the frames
# do not have, so it is refused at build time rather than mis-decoded by
# every client at run time.
{
assertion = cfg.captureBackend != "wayland" || cfg.format == "png";
message = ''
services.hyprland-headless-stream.captureBackend = "wayland" publishes
PNG only, but format = "${cfg.format}".
JPEG in-process would need an encoder this package does not carry yet.
Set format = "png", or go back to captureBackend = "command".
'';
}
# Structural, not cosmetic: see the `runtimeDirName` comment above. The
# ceiling exists so the compositor's *event* socket still fits in
# `sun_path`, and the symptom of breaking it is silent -- the request
+19 -1
View File
@@ -1,5 +1,6 @@
{
lib,
libjpeg_turbo,
python3,
...
}:
@@ -22,7 +23,17 @@ python3.pkgs.buildPythonApplication {
# hhstream_pointer._protocol.regenerate for how to refresh them -- including
# the note that virtual-keyboard-unstable-v1.xml is not in wayland-protocols
# or wlr-protocols in this nixpkgs and comes from pkgs.hyprland.src.
dependencies = [ python3.pkgs.pywayland python3.pkgs.xkbcommon ];
dependencies = [ python3.pkgs.pywayland python3.pkgs.xkbcommon libjpeg_turbo ];
# The in-process jpeg encoder reaches libjpeg-turbo through `ctypes`, so the
# library has to be *findable at run time*, which a propagated build input
# alone does not achieve: nothing links it, so nothing puts its directory on
# LD_LIBRARY_PATH. The wrapper is what makes `ctypes.util.find_library` succeed
# inside the unit -- and it is the same library grim already links, so the
# frames stay byte-compatible with the existing stream.
makeWrapperArgs = [
"--prefix LD_LIBRARY_PATH : ${lib.makeLibraryPath [ libjpeg_turbo ]}"
];
# Unit tests are wired up in the flake's `checks` so they can run without
# installing the package; see docs/CONTRACT.md.
@@ -38,6 +49,13 @@ python3.pkgs.buildPythonApplication {
# The web app: importing the server pulls in the router, the publisher and
# the pages.
"hhstream_live.server"
# The capture backend and its vendored protocol bindings: importing the
# backend runs the scanner-generated `_gen_c()` for wl_shm, wl_output and
# zwlr-screencopy-manager-v1, which is the same cheap proof the pointer
# driver gets above.
"hhstream.wayland_capture"
"hhstream._protocol.wlr_screencopy_unstable_v1"
"hhstream.jpeg"
];
# `pythonImportsCheck` only *imports*, and `hhstream_live.phrase` reads its
+219
View File
@@ -0,0 +1,219 @@
"""JPEG encoding and decoding through libjpeg-turbo, via `ctypes`.
Why this exists at all: the live viewer (`hhstream_live.server`) serves
`multipart/x-mixed-replace` with parts labelled `image/jpeg`, and its own capture
default is `grim -c -t jpeg -q 80 -`. The in-process capture backend has to
produce the same thing or video streaming breaks -- so PNG-only was never an
option, and this is the half of the work that makes the backend usable.
libjpeg-turbo rather than a Python binding because it is what `grim` already
links: the frames stay byte-compatible with the existing stream, the dependency is
a library rather than a wheel, and measured on this machine the encode costs
~1.7 ms for a 1280x720 frame -- against ~30-60 ms for the stdlib PNG path, and
~2.1 MB against ~120 KB on the wire.
**`TJPF_BGRX` is the whole reason there is no colour conversion here.** A
`wl_shm` frame arrives as B, G, R, X in memory -- which is exactly libjpeg-turbo's
`TJPF_BGRX` input format (3). Naming it means the compositor's buffer goes straight
into the encoder with no per-frame swizzle, which is the ~2.2 ms the spike
measured for doing that conversion in Python.
Two traps, both of which cost a real debugging session and are the reason this
file is chatty:
1. **Declare every `argtypes`.** Without them ctypes passes arguments through a
default signature and `tjCompress2` reads the wrong width/pitch -- a segfault
that takes the whole server down, not a wrong byte count.
2. **Reset `*jpegBuf` to NULL after `tjFree`.** turbojpeg treats a non-NULL
`*jpegBuf` as a caller-supplied destination, so leaving the freed pointer in
place makes the *next* frame write into freed memory: a heap-corrupting
segfault several frames later, with no stack trace anywhere near the encoder.
The decoder is here because the unit tests need to prove the encoder round-trips
without adding an image library to the test environment -- and it is the only
honest way to check a lossy codec: compare pixels, not a file header.
"""
from __future__ import annotations
import ctypes
import ctypes.util
import os
from typing import Final
#: From turbojpeg.h: memory order B, G, R, X -- what a `wl_shm` frame already is.
TJPF_BGRX: Final = 3
#: 4:2:0 chroma subsampling, the default `grim -q 80` effectively produces.
TJSAMP_420: Final = 2
#: What the deployment's capture command uses (`grim -c -t jpeg -q 80 -`).
DEFAULT_QUALITY: Final = 80
#: Tried in order when no explicit path is given. Env override first, because a
#: nix wrapper is the natural place to point this at the exact library it built
#: against; then whatever the loader can find (which is how the wrapper's
#: `LD_LIBRARY_PATH` reaches ctypes).
_LIBRARY_NAMES: Final = ("libturbojpeg.so.0", "libturbojpeg.so", "turbojpeg")
class JPEGError(RuntimeError):
"""libjpeg-turbo is missing, or refused the frame."""
def _load_library(path: str | None = None) -> ctypes.CDLL:
candidate = path or os.environ.get("TURBOJPEG_PATH")
if candidate:
try:
return ctypes.CDLL(candidate)
except OSError as exc:
raise JPEGError(f"cannot load libjpeg-turbo from {candidate}: {exc}") from exc
found = ctypes.util.find_library("turbojpeg")
for name in filter(None, (found, *_LIBRARY_NAMES)):
try:
return ctypes.CDLL(name)
except OSError:
continue
raise JPEGError(
"libjpeg-turbo is not available; the in-process backend needs it for JPEG "
"frames (set TURBOJPEG_PATH, or use --format png)"
)
class TurboJPEG:
"""One encoder and one decoder handle, kept for the process's lifetime.
Handle reuse is deliberate: `tjInitCompress` allocates, and doing that per
frame is the kind of per-frame work this whole backend exists to remove.
"""
def __init__(self, path: str | None = None) -> None:
self.lib = _load_library(path)
self._declare_signatures()
self._compress_handle = self.lib.tjInitCompress()
if not self._compress_handle:
raise JPEGError("tjInitCompress failed")
self._decompress_handle = None
self._out = ctypes.c_void_p()
self._size = ctypes.c_ulong()
def _declare_signatures(self) -> None:
"""Trap 1: every argument typed, or the call reads the wrong pitch."""
lib = self.lib
lib.tjInitCompress.argtypes = []
lib.tjInitCompress.restype = ctypes.c_void_p
lib.tjDestroy.argtypes = [ctypes.c_void_p]
lib.tjDestroy.restype = ctypes.c_int
lib.tjFree.argtypes = [ctypes.c_void_p]
lib.tjFree.restype = None
lib.tjCompress2.argtypes = [
ctypes.c_void_p, # tjhandle
ctypes.c_void_p, # srcBuf
ctypes.c_int, # width
ctypes.c_int, # pitch (bytes per row)
ctypes.c_int, # height
ctypes.c_int, # pixelFormat
ctypes.POINTER(ctypes.c_void_p), # *jpegBuf (out)
ctypes.POINTER(ctypes.c_ulong), # *jpegSize (out)
ctypes.c_int, # jpegSubsamp
ctypes.c_int, # jpegQual
ctypes.c_int, # flags
]
lib.tjCompress2.restype = ctypes.c_int
lib.tjInitDecompress.argtypes = []
lib.tjInitDecompress.restype = ctypes.c_void_p
lib.tjDecompressHeader3.argtypes = [
ctypes.c_void_p, # tjhandle
ctypes.c_void_p, # jpegBuf
ctypes.c_ulong, # jpegSize
ctypes.POINTER(ctypes.c_int), # *width
ctypes.POINTER(ctypes.c_int), # *height
ctypes.POINTER(ctypes.c_int), # *jpegSubsamp
ctypes.POINTER(ctypes.c_int), # *jpegColorspace
]
lib.tjDecompressHeader3.restype = ctypes.c_int
lib.tjDecompress2.argtypes = [
ctypes.c_void_p, # tjhandle
ctypes.c_void_p, # jpegBuf
ctypes.c_ulong, # jpegSize
ctypes.c_void_p, # dstBuf
ctypes.c_int, # width
ctypes.c_int, # pitch
ctypes.c_int, # height
ctypes.c_int, # pixelFormat
ctypes.c_int, # flags
]
lib.tjDecompress2.restype = ctypes.c_int
def compress(self, address: int, width: int, height: int, pitch: int,
quality: int = DEFAULT_QUALITY) -> bytes:
"""Encode one BGRX frame held at *address* (a mapped shm buffer)."""
if not 0 < quality <= 100:
raise ValueError(f"JPEG quality must be 1..100, got {quality!r}")
rc = self.lib.tjCompress2(
ctypes.c_void_p(self._compress_handle),
ctypes.c_void_p(address),
ctypes.c_int(width),
ctypes.c_int(pitch),
ctypes.c_int(height),
ctypes.c_int(TJPF_BGRX),
ctypes.byref(self._out),
ctypes.byref(self._size),
ctypes.c_int(TJSAMP_420),
ctypes.c_int(quality),
0,
)
if rc != 0:
raise JPEGError(f"tjCompress2 failed (code {rc})")
try:
return ctypes.string_at(self._out, self._size.value)
finally:
self.lib.tjFree(self._out)
# Trap 2: a non-NULL *jpegBuf means "caller's buffer" to turbojpeg, so
# the freed pointer must not be left behind.
self._out = ctypes.c_void_p()
self._size = ctypes.c_ulong()
def decompress(self, data: bytes) -> tuple[int, int, bytes]:
"""Decode a JPEG to (width, height, BGRX pixels). For tests and tools."""
if self._decompress_handle is None:
self._decompress_handle = self.lib.tjInitDecompress()
if not self._decompress_handle:
raise JPEGError("tjInitDecompress failed")
source = ctypes.create_string_buffer(data, len(data))
return self._decompress(ctypes.cast(source, ctypes.c_void_p), len(data))
def _decompress(self, pointer: ctypes.c_void_p, size: int) -> tuple[int, int, bytes]:
# Four separate `c_int` objects, not a `(c_int * 4)()` array: iterating an
# array yields plain Python ints, and `byref` refuses those -- so the
# out-parameters of the header call have to be real ctypes cells.
frame_width = ctypes.c_int()
frame_height = ctypes.c_int()
subsamp = ctypes.c_int()
colorspace = ctypes.c_int()
rc = self.lib.tjDecompressHeader3(
ctypes.c_void_p(self._decompress_handle), pointer, ctypes.c_ulong(size),
ctypes.byref(frame_width), ctypes.byref(frame_height),
ctypes.byref(subsamp), ctypes.byref(colorspace),
)
if rc != 0:
raise JPEGError(f"tjDecompressHeader3 failed (code {rc})")
width, height = frame_width.value, frame_height.value
pitch = width * 4
out = ctypes.create_string_buffer(pitch * height)
rc = self.lib.tjDecompress2(
ctypes.c_void_p(self._decompress_handle), pointer, ctypes.c_ulong(size),
ctypes.cast(out, ctypes.c_void_p), ctypes.c_int(width), ctypes.c_int(pitch),
ctypes.c_int(height), ctypes.c_int(TJPF_BGRX), 0,
)
if rc != 0:
raise JPEGError(f"tjDecompress2 failed (code {rc})")
return width, height, out.raw
def close(self) -> None:
for name in ("_compress_handle", "_decompress_handle"):
handle = getattr(self, name, None)
if handle:
self.lib.tjDestroy(ctypes.c_void_p(handle))
setattr(self, name, None)
__all__ = ["TurboJPEG", "JPEGError", "DEFAULT_QUALITY", "TJPF_BGRX", "TJSAMP_420"]
+34 -17
View File
@@ -323,8 +323,14 @@ def build_parser() -> argparse.ArgumentParser:
parser.add_argument(
_CAPTURE_BACKEND_FLAG, metavar="BACKEND", default=None,
help="capture in-process instead of by command: 'wayland' to capture the "
"first output, 'wayland:NAME' to capture that output by name. "
"Only PNG can be produced this way, so it needs --format png.",
"first output, 'wayland:NAME' to capture that output by name. Both "
"--format png and --format jpeg are supported; jpeg needs "
"libjpeg-turbo on the machine.",
)
parser.add_argument(
"--capture-quality", type=_quality, default=80, metavar="N",
help="JPEG quality for the in-process backend (default: %(default)s); "
"ignored by --format png. Matches the default capture command's -q.",
)
parser.add_argument(
_CAPTURE_FLAG, action="append", dest="capture_command", metavar="TOKEN",
@@ -345,7 +351,7 @@ def main(argv: Iterable[str] | None = None) -> int:
capture_command = args.capture_command or _default_capture_command(args.format)
capture: CaptureBackend = CommandCapture(capture_command)
else:
capture = _build_capture_backend(args.capture, args.format)
capture = _build_capture_backend(args.capture, args.format, args.capture_quality)
if capture is None:
return 2
server = FrameServer(
@@ -369,22 +375,25 @@ def _default_capture_command(fmt: str) -> list[str]:
return ["grim", "-t", fmt, "-"]
def _build_capture_backend(spec: str, fmt: str) -> CaptureBackend | None:
def _build_capture_backend(spec: str, fmt: str, quality: int) -> CaptureBackend | None:
"""Turn `--capture BACKEND` into a backend, or log why it cannot be one.
The in-process backend publishes PNG, because PNG is the one format this
package can produce without a dependency (see `png.encode_rgb`). Rather than
advertise a MIME type the frames do not have -- the Hello carries the format,
and a client that trusts it would mis-decode every frame -- this refuses the
combination and says what to do instead.
Both formats the contract allows are produced in-process: PNG through the
stdlib (`png.encode_rgb`), JPEG through libjpeg-turbo (`jpeg.TurboJPEG`).
JPEG is not optional in practice -- the live viewer serves
`multipart/x-mixed-replace` with `image/jpeg` parts -- so a missing
libjpeg-turbo is reported here, at startup, rather than once per frame: a
server that is up but can never produce a frame is worse than one that
refuses to start, because the socket still accepts connections.
"""
if fmt != "png":
log.error(
"the in-process capture backend publishes PNG only, but --format %s "
"was asked for; JPEG in-process needs an encoder this package does "
"not carry yet, so use --format png or the command backend", fmt,
)
return None
if fmt == "jpeg":
from .jpeg import JPEGError, TurboJPEG
try:
TurboJPEG().close()
except JPEGError as exc:
log.error("cannot capture as jpeg: %s", exc)
return None
if spec == "wayland":
output = None
elif spec.startswith("wayland:"):
@@ -402,7 +411,15 @@ def _build_capture_backend(spec: str, fmt: str) -> CaptureBackend | None:
# backend (still the default) should not need pywayland importable at all.
from .wayland_capture import WaylandCapture
return WaylandCapture(output=output)
return WaylandCapture(output=output, codec=fmt, quality=quality)
def _quality(text: str) -> int:
"""JPEG quality, matching grim's `-q`: 1..100, and refused outside it."""
value = int(text)
if not 1 <= value <= 100:
raise argparse.ArgumentTypeError(f"must be 1..100, got {text!r}")
return value
def _positive_int(text: str) -> int:
+39 -2
View File
@@ -29,7 +29,14 @@ Three things follow, and each one shaped the code below:
favour of ``ext-image-copy-capture-v1``, which was measured as identical
(78.0 vs 77.7 ms, compositor 67.5 vs 67.2) and which appears in a Hyprland
segfault stack (``CScreenshareFrame::transform()`` reached from
``create_frame``). When that crash is fixed, the switch is a constant here.
`create_frame`. When that crash is fixed, the switch is a constant here.
4. **Both codecs, because the stream decides.** The live viewer serves
`multipart/x-mixed-replace` with `image/jpeg` parts, so a PNG-only backend
would have broken video streaming while looking faster in a benchmark. JPEG
goes through libjpeg-turbo's `TJPF_BGRX` input format, which is exactly the
compositor's memory order -- no conversion, ~1.7 ms a frame, ~120 KB.
PNG stays available (stdlib `zlib`) for deployments that would rather not
carry the library, at ~2.1 MB a frame.
**Why the wait is a poll loop and not a signal.** The frame server captures in a
worker thread (``FrameServer._capture_loop``), so ``SIGALRM`` -- the obvious way to
@@ -44,6 +51,7 @@ and actually reads.
from __future__ import annotations
import ctypes
import logging
import mmap
import os
@@ -58,6 +66,7 @@ from ._protocol.wlr_screencopy_unstable_v1.zwlr_screencopy_manager_v1 import (
ZwlrScreencopyManagerV1,
)
from .capture import CaptureError
from .jpeg import DEFAULT_QUALITY, TurboJPEG
from .png import encode_rgb
log = logging.getLogger(__name__)
@@ -87,6 +96,9 @@ class _ShmBuffer:
self.fd = os.memfd_create("hhstream-frame", os.MFD_CLOEXEC)
os.ftruncate(self.fd, self.size)
self.map = mmap.mmap(self.fd, self.size, prot=mmap.PROT_READ | mmap.PROT_WRITE)
#: Address of the mapping, for the ctypes encoder. Kept, not re-derived:
#: `addressof(from_buffer(...))` per frame is pure overhead.
self.address = ctypes.addressof(ctypes.c_char.from_buffer(self.map))
self.pool = shm.create_pool(self.fd, self.size)
self.buffer = self.pool.create_buffer(0, width, height, stride, fmt)
@@ -117,9 +129,14 @@ class WaylandCapture:
DEFAULT_TIMEOUT = 5.0
def __init__(self, output: str | None = None, timeout: float = DEFAULT_TIMEOUT,
display: str | None = None, level: int = 1) -> None:
display: str | None = None, level: int = 1, codec: str = "png",
quality: int = DEFAULT_QUALITY) -> None:
if timeout is not None and timeout <= 0:
raise ValueError(f"capture timeout must be positive, got {timeout!r}")
if codec not in ("png", "jpeg"):
raise ValueError(f"unknown codec {codec!r}; expected 'png' or 'jpeg'")
self.codec = codec
self.quality = quality
self.output_name = output
self.timeout = timeout
#: `None` means "whatever WAYLAND_DISPLAY says", which is how the module
@@ -138,6 +155,9 @@ class WaylandCapture:
self._ready: bool | None = None
self._failure: str | None = None
self._advertised: tuple[int, int, int, int] | None = None
#: Built on the first jpeg frame, not in __init__: a png-only deployment
#: must not need libjpeg-turbo on the box at all.
self._encoder: TurboJPEG | None = None
# -- the two-method backend interface ---------------------------------
@@ -158,6 +178,8 @@ class WaylandCapture:
self._copy_frame()
self._wait_for_ready()
assert self._buffer is not None
if self.codec == "jpeg":
return self._encode_jpeg(self._buffer)
return encode_rgb(self._buffer.width, self._buffer.height,
self._buffer.stride, memoryview(self._buffer.map),
level=self.level)
@@ -167,9 +189,24 @@ class WaylandCapture:
self.close()
raise CaptureError(f"wayland capture failed: {exc}") from exc
def _encode_jpeg(self, buffer: _ShmBuffer) -> bytes:
"""Encode the mapped buffer in place -- no copy, no colour conversion.
`TJPF_BGRX` tells libjpeg-turbo that the compositor's B,G,R,X layout is
already the format it wants, which is what keeps this at ~1.7 ms a frame
instead of paying a Python swizzle first.
"""
if self._encoder is None:
self._encoder = TurboJPEG()
return self._encoder.compress(buffer.address, buffer.width, buffer.height,
buffer.stride, quality=self.quality)
def close(self) -> None:
"""Drop the connection and the buffer. Safe to call twice, and after a
failure -- it is the recovery path, not just a shutdown path."""
if self._encoder is not None:
self._encoder.close()
self._encoder = None
if self._buffer is not None:
try:
self._buffer.destroy()
+164
View File
@@ -0,0 +1,164 @@
"""`hhstream.jpeg`: the libjpeg-turbo binding, checked by round-tripping pixels.
A lossy codec cannot be tested by looking at a file header, so these tests encode a
raster and decode it again through the same library. The raster is chosen to fail
loudly on the two mistakes that matter here: uniformly filled blocks (JPEG
reproduces a solid block almost exactly, so a tolerance of a few counts is honest)
whose channels differ enough that a BGR/RGB mix-up is unmistakable, and a padded
stride, because assuming `width * 4` is the bug grim shipped in 1.5.
This is also the only test of the pixel-format claim the backend rests on:
`TJPF_BGRX` exists precisely so the compositor's B, G, R, X buffer can go into the
encoder unconverted. If that were the wrong format, the decoded block colours come
back swapped and these tests say so.
"""
from __future__ import annotations
import ctypes
import pytest
from hhstream.jpeg import DEFAULT_QUALITY, JPEGError, TurboJPEG
BLOCK = 16
def blocks() -> list[tuple[int, int, int]]:
"""Four colours whose channels differ enough that a swap cannot hide."""
return [(200, 30, 30), (30, 200, 30), (30, 30, 200), (140, 120, 100)]
def as_rgb(colour: tuple[int, int, int]) -> tuple[int, int, int]:
"""`blocks()` writes memory order (B, G, R); assertions read (R, G, B).
Getting these two the wrong way round is indistinguishable from a wrong pixel
format in the encoder -- which is exactly what this file exists to detect, so
the conversion is named rather than done inline.
"""
blue, green, red = colour
return red, green, blue
def colour_at(x: int, y: int, width: int) -> tuple[int, int, int]:
"""The block colour at a pixel, wrapping so any frame size is covered.
One formula for both the raster and the assertions: they used to index the
four colours independently, which ran off the end of the list for anything
bigger than a 2x2 grid of blocks.
"""
column = x // BLOCK
row = y // BLOCK
return blocks()[((row * (width // BLOCK)) + column) % len(blocks())]
def raster(width: int = 64, height: int = 64, stride: int | None = None) -> tuple[bytes, int]:
"""A BGRX frame of solid blocks; returns (bytes, stride)."""
stride = stride if stride is not None else width * 4
out = bytearray()
for y in range(height):
row = bytearray()
for x in range(width):
b, g, r = colour_at(x, y, width)
row += bytes((b, g, r, 0x00))
row += b"\x77" * (stride - width * 4) # padding, if any
out += row
return bytes(out), stride
def address_of(pixels: bytes) -> int:
return ctypes.addressof(ctypes.create_string_buffer(pixels, len(pixels)))
def centre_colour(pixels: bytes, stride: int, x: int, y: int) -> tuple[int, int, int]:
offset = y * stride + x * 4
blue, green, red, _ = pixels[offset:offset + 4]
return red, green, blue
def test_a_frame_round_trips_with_its_channels_in_the_right_place():
enc = TurboJPEG()
try:
pixels, stride = raster()
encoded = enc.compress(address_of(pixels), 64, 64, stride)
width, height, decoded = enc.decompress(encoded)
assert (width, height) == (64, 64)
for column, row in ((0, 0), (1, 0), (2, 0), (0, 1), (3, 3)):
x, y = column * BLOCK + BLOCK // 2, row * BLOCK + BLOCK // 2
want = as_rgb(colour_at(x, y, 64))
got = centre_colour(decoded, width * 4, x, y)
assert got == pytest.approx(want, abs=12), (
f"block ({row}, {column}) came back as {got}, expected ~{want} "
"-- a swapped channel order or a wrong pixel format"
)
# Being close is not enough on its own: assert the channels are
# distinguishable, so a swap could not pass as "close enough".
assert max(want) - min(want) > 24, "pick colours a swap would expose"
finally:
enc.close()
def test_a_padded_stride_is_honoured():
"""The padding bytes are deliberately not black; leaking them changes the
last column of every row."""
enc = TurboJPEG()
try:
pixels, stride = raster(64, 64, stride=64 * 4 + 32)
assert stride != 64 * 4
encoded = enc.compress(address_of(pixels), 64, 64, stride)
_, _, decoded = enc.decompress(encoded)
got = centre_colour(decoded, 64 * 4, 8, 8)
assert got == pytest.approx(as_rgb(colour_at(8, 8, 64)), abs=12), got
finally:
enc.close()
def test_quality_trades_bytes_for_fidelity():
enc = TurboJPEG()
try:
pixels, stride = raster()
small = enc.compress(address_of(pixels), 64, 64, stride, quality=20)
large = enc.compress(address_of(pixels), 64, 64, stride, quality=95)
assert len(small) < len(large)
finally:
enc.close()
def test_the_output_is_really_a_jpeg():
enc = TurboJPEG()
try:
pixels, stride = raster()
encoded = enc.compress(address_of(pixels), 64, 64, stride)
assert encoded.startswith(b"\xff\xd8"), "missing SOI marker"
assert b"\xff\xc0" in encoded, "missing SOF0 marker"
assert encoded.endswith(b"\xff\xd9"), "missing EOI marker"
finally:
enc.close()
def test_the_default_quality_is_the_one_the_capture_command_used():
"""`grim -c -t jpeg -q 80 -` is the deployment's command; a swap must not
silently change the frames' size or quality."""
assert DEFAULT_QUALITY == 80
@pytest.mark.parametrize("quality", [0, -1, 101])
def test_an_impossible_quality_is_refused(quality):
enc = TurboJPEG()
try:
pixels, stride = raster()
with pytest.raises(ValueError, match="1..100"):
enc.compress(address_of(pixels), 64, 64, stride, quality=quality)
finally:
enc.close()
def test_a_library_that_is_not_there_is_a_clear_error_not_a_crash():
with pytest.raises(JPEGError, match="cannot load libjpeg-turbo"):
TurboJPEG(path="/nonexistent/libturbojpeg.so")
def test_close_is_safe_twice():
enc = TurboJPEG()
enc.close()
enc.close()
+46 -7
View File
@@ -99,7 +99,7 @@ def test_a_timeout_of_zero_or_less_is_refused_at_construction():
)
def test_the_backend_spec_selects_an_output_without_connecting(spec, expected_output):
"""Constructing the backend must not open a connection -- grab() does."""
backend = _build_capture_backend(spec, "png")
backend = _build_capture_backend(spec, "png", 80)
assert isinstance(backend, WaylandCapture)
assert backend.output_name == expected_output
assert backend._display is None, "the connection belongs to grab(), not to __init__"
@@ -109,14 +109,41 @@ def test_the_backend_spec_selects_an_output_without_connecting(spec, expected_ou
def test_an_unknown_backend_spec_is_refused_rather_than_guessed(spec, caplog):
"""`wayland:a:b` is not an output name, and guessing would capture the wrong
output silently."""
assert _build_capture_backend(spec, "png") is None
assert _build_capture_backend(spec, "png", 80) is None
def test_jpeg_is_refused_rather_than_advertised_in_the_hello(caplog):
"""The Hello carries the MIME type; publishing jpeg over PNG frames would
mis-decode every frame in every client, so it is refused up front."""
assert _build_capture_backend("wayland", "jpeg") is None
assert "PNG only" in caplog.text
def test_jpeg_is_what_the_backend_uses_when_jpeg_is_asked_for():
"""The live viewer serves image/jpeg parts, so this is the format that has to
work -- a PNG-only backend would break video streaming."""
backend = _build_capture_backend("wayland", "jpeg", 80)
assert isinstance(backend, WaylandCapture)
assert (backend.codec, backend.quality) == ("jpeg", 80)
def test_png_never_needs_the_jpeg_library(monkeypatch):
"""A deployment that chose png must not require libjpeg-turbo on the box."""
import hhstream.jpeg as jpeg
def explode(*_args, **_kwargs):
raise jpeg.JPEGError("no libjpeg-turbo anywhere")
monkeypatch.setattr(jpeg, "TurboJPEG", explode)
backend = _build_capture_backend("wayland", "png", 80)
assert isinstance(backend, WaylandCapture)
assert backend.codec == "png"
def test_a_missing_jpeg_library_refuses_to_start_rather_than_never_serving(monkeypatch, caplog):
"""A server that is up but can never produce a frame is worse than one that
refuses to start: its socket still accepts connections."""
import hhstream.jpeg as jpeg
def explode(*_args, **_kwargs):
raise jpeg.JPEGError("no libjpeg-turbo anywhere")
monkeypatch.setattr(jpeg, "TurboJPEG", explode)
assert _build_capture_backend("wayland", "jpeg", 80) is None
assert "cannot capture as jpeg" in caplog.text
def test_the_backend_flag_survives_the_token_folding_that_capture_tokens_need():
@@ -125,6 +152,18 @@ def test_the_backend_flag_survives_the_token_folding_that_capture_tokens_need():
assert build_parser().parse_args(folded).capture == "wayland:Virtual-1"
def test_quality_defaults_to_match_the_command_backend():
"""`grim -q 80` is what the deployment used, so a swap must not change it."""
args = build_parser().parse_args(["--capture", "wayland"])
assert args.capture_quality == 80
@pytest.mark.parametrize("bad", ["0", "101", "-5"])
def test_an_impossible_quality_is_refused(bad):
with pytest.raises(SystemExit):
build_parser().parse_args(["--capture-quality", bad])
def test_no_backend_flag_still_means_the_command_backend():
args = build_parser().parse_args(["--socket", "/tmp/x.sock"])
assert args.capture is None