diff --git a/README.dev.md b/README.dev.md new file mode 100644 index 0000000..c22ffa4 --- /dev/null +++ b/README.dev.md @@ -0,0 +1,241 @@ +# Developing and testing `hyprland-headless-stream-plugin` + +The interface everything here asserts is [`docs/CONTRACT.md`](docs/CONTRACT.md). +That document is authoritative: if a test and the code disagree, the document +decides, and if the document is wrong it gets a version bump, not an edit. + +## Prerequisites + +- Nix with flakes enabled. +- `/dev/kvm` readable, for the two VM checks. They are `nixosTest`s; without + KVM they still run, just slowly enough that the timeouts below get tight. +- No GPU. Nothing in the test suite needs one — the compositor runs on + Hyprland's headless backend over software rendering. + +Nothing in the checks touches the host: no `/etc/nixos`, no network, and no +socket outside the build sandbox or the test VM. + +## The dev shell + +```bash +nix develop +``` + +Puts `python3`, `pytest`, `hyprland`, `grim`, `ffmpeg`, `jq` and `tmux` on PATH +and sets `PYTHONPATH=$PWD/src`, so `import hhstream` picks up the working tree +rather than an installed copy. + +## Build + +```bash +nix build .#hhstream # the package (server + client) +ls result/bin # hhstream-server, hhstream-client +``` + +Both entry points currently exit with `NotImplementedError` — see "Known gaps" +at the bottom. The package builds; it does not yet run. + +## The checks + +| Check | What it proves | Needs a VM | +|---|---|---| +| `unit` | the framing, the capture backend, the server's client handling, the client — in-process, against the `nix/package.nix` build | no | +| `vm-basic` | the NixOS module brings up both units and publishes a contract-conformant socket, with a **fake** capture backend | yes | +| `vm-hyprland` | the same, capturing the **real** headless compositor through `grim` | yes | + +All three at once: + +```bash +nix flake check -L +``` + +`nix flake check` stops at the first failing check, which makes it a poor way to +see the whole picture while the tree is still red. Prefer running them +individually — they are independent: + +```bash +nix build -L .#checks.x86_64-linux.unit +nix build -L .#checks.x86_64-linux.vm-basic +nix build -L .#checks.x86_64-linux.vm-hyprland +``` + +`-L` is not optional in practice: without it, Nix hides the build log, and the +build log *is* the test report. + +### `unit` + +pytest over `tests/unit/`, run in the build sandbox with `PYTHONPATH` pointing at +what `nix/package.nix` installed (`nix/package.nix` sets `doCheck = false` on +purpose, so this is the only place the suite runs in a build). + +The check fails if the suite is **empty**, not just if it is red. `pytest` exits +5 on "collected nothing", but it exits 0 when every test was skipped or +deselected, so `checks/lib/check_report.py` parses the JUnit report and refuses +a run with zero collected or zero executed tests. A green check that tested +nothing is worse than no check. On success the report is copied to +`result/junit.xml`. + +To run the same tests directly, without Nix: + +```bash +nix develop +pytest tests/unit -v # PYTHONPATH=$PWD/src comes from the shell +pytest tests/unit/test_protocol.py -v # one file +pytest tests/unit -k "hello or frame" -v # one theme +pytest tests/unit -x --ff # stop at the first failure, failures first +``` + +Note that the unit suite's *negative* tests use the `rejects` fixture from +`tests/unit/conftest.py` rather than `pytest.raises(Exception)`. `rejects` +treats `NotImplementedError` as a failure, because a plain `raises(Exception)` +is satisfied by an unimplemented stub — the test would then be green precisely +because nothing had been written. + +### `vm-basic` + +Boots one machine with `services.hyprland-headless-stream.enable = true` and +`captureCommand` pointed at `tests/fake_capture.py`, which writes one fixed, +deterministic PNG of the configured size to stdout. Because the image never +changes, "two frames arrived and they are the *same bytes*" is a real assertion +about the framing rather than a coincidence. + +It checks, in order: both units start; the socket exists and is +`hhstream:hhstream` mode `0660`; the Hello parses with `magic = "HHS1"` and +non-zero width/height matching the configured output; two frames arrive with +length > 0, identical bytes, and an IHDR matching the Hello; two clients are +served concurrently; and a client that dies mid-frame leaves the server serving. + +The reader is `tests/assert_stream.py`, a second, independent implementation of +the contract that shares no code with `hhstream.client` — so a framing bug +present in both the server's writer and the shipped client's reader still fails +the test. It validates the length prefix strictly (`0|[1-9][0-9]*`: no leading +zeros, no sign, no spaces), never reads past a newline, and prints the Hello, +every frame's size and SHA-256, and the parsed IHDR before it decides anything. + +### `vm-hyprland` + +The same machine with **no** `captureCommand` override, so the module's default +(`grim`) captures the real compositor. + +This is the check expected to be flaky, so it is deliberately impatient: every +wait has a bounded timeout, nothing is retried in a loop, and any failure dumps +diagnostics before it propagates. It asserts a frame is a valid PNG whose IHDR +dimensions match the configured output, and that it is more than a blank buffer: + +- the IDAT must decompress; +- the decompressed scanlines must contain a non-zero byte and at least two + distinct byte values — a zero-filled framebuffer fails both; +- the payload must be larger than a best-case blank PNG of the same geometry + (`blank_png_size()` in `tests/assert_stream.py`: an all-zero raster at zlib + level 9, i.e. the smallest a blank frame of that size can be encoded). + +It also runs `grim` directly against the session as a separate subtest, which +separates "grim cannot see the compositor" from "the server is wrong" — the two +failures that look identical from the socket. + +## When `vm-hyprland` fails: where the logs are + +Every failing subtest in both VM checks runs `hhs-hyprland-diagnostics` +(`tests/hyprland_diagnostics.sh`) in the guest first and prints its output to the +build log. So the answer to "where are the logs" is usually **in the build log +you already have**, above the traceback, between +`########## diagnostics: ... ##########` markers. Re-read it with: + +```bash +nix log .#checks.x86_64-linux.vm-hyprland +``` + +What the dump contains, and where it comes from in the guest: + +| Section | Source | +|---|---| +| `units` | `systemctl list-units --all 'hyprland*'` | +| per-unit `status` / `journalctl` | `hyprland-headless.service` and `hyprland-headless-stream.service`, 200 lines each | +| `socket and runtime directories` | `/run/hyprland-headless-stream`, `/run/user/*` | +| `hyprland logs` | every `hyprland*.log` found under `/run`, `/var/log`, `/tmp`, `/home` | +| `hyprctl monitors` | run once per instance found, with the right `XDG_RUNTIME_DIR` and `HYPRLAND_INSTANCE_SIGNATURE` | +| `processes` | `ps aux` filtered to `hypr`, `hhstream`, `grim` | + +Hyprland writes its log to +`$XDG_RUNTIME_DIR/hypr/$HYPRLAND_INSTANCE_SIGNATURE/hyprland.log`, and the +compositor runs on a *private* `XDG_RUNTIME_DIR`, so the path is not knowable in +advance — that is why the script searches for it rather than naming it. The +instance signature is the directory name under `/hypr`, which is also +what `hyprctl` needs in the environment: + +```bash +XDG_RUNTIME_DIR=/run/user/$(id -u hhstream) \ +HYPRLAND_INSTANCE_SIGNATURE=$(ls /run/user/$(id -u hhstream)/hypr | head -1) \ + hyprctl monitors +``` + +### Poking at a live VM + +The nicest way to debug a VM failure is to keep the machine: + +```bash +nix build .#checks.x86_64-linux.vm-hyprland.driverInteractive +./result/bin/nixos-test-driver +``` + +That drops you in a Python REPL with `machine` bound. `start_all()`, then +`machine.shell_interact()` for a root shell in the guest, or run the test script +a line at a time. `machine.succeed("hhs-hyprland-diagnostics")` works there too. + +## Layout + +``` +checks/ + default.nix wiring only: the three checks the flake exposes + unit.nix pytest in the sandbox, against the nix/package.nix build + vm-basic.nix nixosTest, fake capture backend + vm-hyprland.nix nixosTest, real compositor + grim + lib/test-tools.nix packages the tests/ helpers for the guest + lib/check_report.py JUnit report -> build verdict (and the empty-suite guard) +tests/ + fake_capture.py writes one fixed PNG to stdout: hhs-fake-capture + assert_stream.py independent contract reader/asserter: hhs-assert-stream + hyprland_diagnostics.sh the failure dump: hhs-hyprland-diagnostics + unit/ the pytest suite +``` + +`checks/lib/test-tools.nix` self-tests the two Python helpers at build time +(the fake emits a PNG of the size it was asked for; the asserter starts), because +a broken helper otherwise surfaces as a baffling failure three layers away. + +Adding a unit test: drop a `test_*.py` in `tests/unit/`. Nothing needs +registering. Adding a check: one file in `checks/`, one line in +`checks/default.nix`. + +## Known gaps in the interface (as of this branch) + +These are defects in code this branch does not own; they are recorded here +because the tests are written against the contract and therefore trip over them. + +1. **`nix/module.nix` has no `config` side yet.** Neither + `hyprland-headless.service` nor `hyprland-headless-stream.service` exists, so + both VM checks fail at their first wait with + `Unit hyprland-headless.service could not be found.` Nothing in `checks/` + works around this. +2. **`services.hyprland-headless-stream.package` is declared with no default.** + `docs/CONTRACT.md` says the default is "the flake's own build", but the option + has no `default`, so reading it fails with + + ``` + error: The option `services.hyprland-headless-stream.package' was accessed but + has no value defined. Try setting the option. + ``` + + Today that only bites code that reads `cfg.package`, which the unwritten + `config` side does not yet do; it will bite every consumer that sets only + `enable = true` the moment it does. Both VM checks set `package` explicitly. +3. **`src/hhstream/*.py` is stubs.** Every function raises + `NotImplementedError`, so 66 of the 70 unit tests fail. The 4 that pass are + `tests/unit/test_package.py`, which only reads constants — they are the + canary for the `unit` check's own wiring. +4. **The frame server's command line is unspecified.** `docs/CONTRACT.md` fixes + the module's option surface and the wire format but never says how the module + invokes `hhstream-server`. `tests/unit/test_server.py::test_build_parser_exposes_the_options_the_module_needs` + currently assumes `--socket PATH --fps N --format png -- `. + That is a guess; if the module ends up passing something else, that one test + is the thing to change, not the server. diff --git a/checks/default.nix b/checks/default.nix new file mode 100644 index 0000000..6b691c1 --- /dev/null +++ b/checks/default.nix @@ -0,0 +1,28 @@ +# The flake's `checks` output. One file per check; this is only the wiring. +# +# unit -- pytest over tests/unit/, against the nix/package.nix build +# vm-basic -- nixosTest with a fake capture backend: module + wire contract +# vm-hyprland -- nixosTest with the real compositor and grim +# +# Everything the guests need lives in checks/lib/test-tools.nix. +{ + pkgs, + self, + system, +}: +let + hhstream = self.packages.${system}.hhstream; + hhsModule = self.nixosModules.default; + testTools = pkgs.callPackage ./lib/test-tools.nix { }; +in +{ + unit = pkgs.callPackage ./unit.nix { inherit hhstream; }; + + vm-basic = pkgs.callPackage ./vm-basic.nix { + inherit hhstream hhsModule testTools; + }; + + vm-hyprland = pkgs.callPackage ./vm-hyprland.nix { + inherit hhstream hhsModule testTools; + }; +} diff --git a/checks/lib/check_report.py b/checks/lib/check_report.py new file mode 100644 index 0000000..d0a4772 --- /dev/null +++ b/checks/lib/check_report.py @@ -0,0 +1,90 @@ +#!/usr/bin/env python3 +"""Turn a pytest JUnit report into a build verdict. + +Two jobs pytest's exit code alone does not do well inside a Nix build: + +* say out loud how many tests ran, and which ones failed, in the build log; +* refuse a suite that collected nothing. An empty suite exits 5, but it also + exits 0 if every test was deselected or skipped away, and a check that passes + because it tested nothing is worse than no check at all. + +Exit codes: 0 all good, 1 the suite is empty or the report is unusable, +2 the suite ran and some tests failed. The caller distinguishes them so the +build log does not accuse an empty suite of being a failing one. +""" + +from __future__ import annotations + +import sys +import xml.etree.ElementTree as ET + + +def main(argv: list[str]) -> int: + if len(argv) != 2: + print("usage: check_report.py ", file=sys.stderr) + return 2 + path = argv[1] + + try: + root = ET.parse(path).getroot() + except FileNotFoundError: + print( + f"ERROR: pytest wrote no report at {path}: it collected nothing, or died " + "before it could run a single test.", + file=sys.stderr, + ) + return 1 + except ET.ParseError as exc: + print(f"ERROR: {path} is not parseable JUnit XML: {exc}", file=sys.stderr) + return 1 + + suites = root.findall("testsuite") or ([root] if root.tag == "testsuite" else []) + total = errors = failures = skipped = 0 + failing: list[str] = [] + for suite in suites: + total += int(suite.get("tests", 0)) + errors += int(suite.get("errors", 0)) + failures += int(suite.get("failures", 0)) + skipped += int(suite.get("skipped", 0)) + for case in suite.iter("testcase"): + for kind in ("failure", "error"): + for node in case.findall(kind): + name = f"{case.get('classname', '')}::{case.get('name', '')}" + message = (node.get("message") or "").splitlines() + head = message[0][:160] if message else node.get("type", kind) + failing.append(f" {kind.upper():7} {name}\n {head}") + + executed = total - skipped + print( + f"unit suite: {total} collected, {executed} executed, " + f"{failures} failed, {errors} errored, {skipped} skipped" + ) + + if total == 0: + print( + "ERROR: the unit suite collected no tests. An empty suite is not a " + "passing suite -- check that tests/unit/ is in the build and that the " + "files are named test_*.py.", + file=sys.stderr, + ) + return 1 + if executed == 0: + print( + f"ERROR: all {total} collected tests were skipped; nothing was actually " + "exercised.", + file=sys.stderr, + ) + return 1 + + if failing: + print(f"\n{len(failing)} failing test(s):") + for line in failing: + print(line) + return 2 + + print("unit suite: all executed tests passed") + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/checks/lib/test-tools.nix b/checks/lib/test-tools.nix new file mode 100644 index 0000000..e88ac5f --- /dev/null +++ b/checks/lib/test-tools.nix @@ -0,0 +1,32 @@ +# The helper programs the checks need inside the guest (and, for the fake +# capture, inside the guest's `captureCommand`). Plain python3/sh scripts from +# `tests/`, put on PATH with their shebangs pointing at the store. +{ + runCommandLocal, + python3, + coreutils, + gnugrep, + findutils, + systemd, +}: +runCommandLocal "hhstream-test-tools" +{ + nativeBuildInputs = [ python3 ]; + # hhs-hyprland-diagnostics runs on a failing guest, where PATH is whatever + # systemd left behind; give it the tools it needs explicitly. + diagPath = "${coreutils}/bin:${gnugrep}/bin:${findutils}/bin:${systemd}/bin"; + meta.description = "Fake capture backend and contract asserter for the hhstream checks"; +} '' + install -Dm755 ${../../tests/fake_capture.py} $out/bin/hhs-fake-capture + install -Dm755 ${../../tests/assert_stream.py} $out/bin/hhs-assert-stream + install -Dm755 ${../../tests/hyprland_diagnostics.sh} $out/bin/hhs-hyprland-diagnostics + patchShebangs $out/bin + substituteInPlace $out/bin/hhs-hyprland-diagnostics --replace-fail '@diagPath@' "$diagPath" + + # A broken helper shows up as a baffling test failure three layers away, so + # prove here that both python programs start and produce what they promise. + $out/bin/hhs-fake-capture --width 8 --height 4 > frame.png + python3 -c 'import struct,sys; d=open("frame.png","rb").read(); assert d[:8]==b"\x89PNG\r\n\x1a\n", d[:8]; assert struct.unpack(">II", d[16:24])==(8,4), d[16:24]' + $out/bin/hhs-assert-stream --help > /dev/null + $out/bin/hhs-hyprland-diagnostics --self-test > /dev/null +'' diff --git a/checks/unit.nix b/checks/unit.nix new file mode 100644 index 0000000..389f3fb --- /dev/null +++ b/checks/unit.nix @@ -0,0 +1,74 @@ +# checks..unit -- the Python unit suite from tests/unit/, run against the +# package built by nix/package.nix, inside the build sandbox. +# +# nix/package.nix sets doCheck = false deliberately, so this is the only place +# the suite runs in a build. +{ + lib, + runCommand, + python3, + hhstream, +}: +let + pytestEnv = python3.withPackages (ps: [ ps.pytest ]); + + # Only the suite itself, so an editor's droppings or a local .pytest_cache + # cannot change the derivation's hash. + testSrc = lib.cleanSourceWith { + name = "hhstream-unit-tests"; + src = ../tests/unit; + filter = + path: type: + let + base = baseNameOf (toString path); + in + !(type == "directory" && (base == "__pycache__" || base == ".pytest_cache")) + && !lib.hasSuffix ".pyc" base; + }; +in +runCommand "hhstream-unit" +{ + nativeBuildInputs = [ pytestEnv ]; + passthru.testSrc = testSrc; + meta.description = "hhstream unit tests (pytest)"; +} '' + cp -r ${testSrc} suite + chmod -R u+w suite + + # The package is an application, so its library is not on the default path: + # point PYTHONPATH at what nix/package.nix installed. Importing it here rather + # than letting pytest do it makes a packaging problem distinguishable from a + # test failure. + export PYTHONPATH=${hhstream}/${python3.sitePackages} + export PYTHONDONTWRITEBYTECODE=1 + export HOME=$PWD + python3 -c 'import hhstream; print("testing hhstream", hhstream.__version__, "from", hhstream.__file__)' + + set +e + pytest suite \ + --color=no -rf -q \ + --junit-xml=report.xml \ + -p no:cacheprovider \ + --strict-markers \ + --strict-config + pytest_rc=$? + set -e + + echo + set +e + python3 ${./lib/check_report.py} report.xml + report_rc=$? + set -e + + if [ "$report_rc" -eq 1 ]; then + echo "unit check: the suite is empty or its report is unusable -- failing." >&2 + exit 1 + fi + if [ "$report_rc" -ne 0 ] || [ "$pytest_rc" -ne 0 ]; then + echo "unit check: the suite ran but did not pass (pytest exit $pytest_rc)." >&2 + exit 1 + fi + + mkdir -p $out + cp report.xml $out/junit.xml +'' diff --git a/checks/vm-basic.nix b/checks/vm-basic.nix new file mode 100644 index 0000000..655dd42 --- /dev/null +++ b/checks/vm-basic.nix @@ -0,0 +1,145 @@ +# checks..vm-basic -- boot a real machine with the module enabled, point +# `captureCommand` at a fake capture program, and read the socket. +# +# This check is about the module's wiring and the wire contract, not about +# Hyprland: the fake capture emits one fixed PNG, so "two frames arrived and +# they are the same image" is a real assertion rather than a coincidence. +{ + testers, + hhstream, + testTools, + hhsModule, + width ? 1280, + height ? 720, + fps ? 30, + socketPath ? "/run/hyprland-headless-stream/stream.sock", +}: +testers.runNixOSTest { + name = "hhstream-vm-basic"; + + # KVM is available, so this is a minute or two; cap it so a hang is a failure + # rather than a wedged builder. + globalTimeout = 900; + + nodes.machine = { + imports = [ hhsModule ]; + + services.hyprland-headless-stream = { + enable = true; + # The module declares `package` with no default (see README.dev.md), so the + # test has to supply the flake's build. + package = hhstream; + inherit socketPath; + frameRate = fps; + format = "png"; + outputs = [ + { + name = "HEADLESS-1"; + inherit width height; + refresh = fps; + } + ]; + captureCommand = [ + "${testTools}/bin/hhs-fake-capture" + "--width" + (toString width) + "--height" + (toString height) + ]; + }; + + environment.systemPackages = [ + testTools + hhstream + ]; + + virtualisation = { + memorySize = 1024; + cores = 2; + diskSize = 3072; + graphics = false; + }; + }; + + testScript = '' + SOCKET = "${socketPath}" + ASSERT = "hhs-assert-stream --socket " + SOCKET + + + def dump(label): + print(f"########## diagnostics: {label} ##########") + print(machine.execute("hhs-hyprland-diagnostics")[1]) + + + start_all() + + with subtest("the module's two units come up"): + try: + machine.wait_for_unit("hyprland-headless.service", timeout=120) + machine.wait_for_unit("hyprland-headless-stream.service", timeout=120) + except Exception: + dump("units did not start") + raise + + with subtest("the socket exists, owned by the service user, mode 0660"): + try: + machine.wait_for_file(SOCKET, timeout=60) + print(machine.succeed(f"stat -c 'owner=%U group=%G mode=%a type=%F' {SOCKET}")) + machine.succeed(f"[ \"$(stat -c %a {SOCKET})\" = 660 ]") + machine.succeed(f"[ \"$(stat -c %U {SOCKET})\" = hhstream ]") + machine.succeed(f"[ \"$(stat -c %G {SOCKET})\" = hhstream ]") + machine.succeed(f"[ \"$(stat -c %F {SOCKET})\" = socket ]") + except Exception: + dump("socket missing or wrong ownership") + raise + + with subtest("the Hello parses, magic is HHS1, size is non-zero"): + try: + print(machine.succeed(f"{ASSERT} --frames 1 --timeout 30")) + except Exception: + dump("Hello did not parse") + raise + + with subtest("two frames arrive, non-empty, byte-identical, matching the Hello"): + try: + print( + machine.succeed( + f"{ASSERT} --frames 2 --timeout 60" + " --expect-width ${toString width} --expect-height ${toString height}" + " --expect-format image/png --expect-fps ${toString fps}" + " --require-png --identical-frames --min-bytes 1" + ) + ) + except Exception: + dump("frames did not satisfy the contract") + raise + + with subtest("two clients are served concurrently and independently"): + try: + print( + machine.succeed( + f"{ASSERT} --frames 3 --timeout 60 --require-png > /tmp/a.log 2>&1 & p1=$!; " + f"{ASSERT} --frames 3 --timeout 60 --require-png > /tmp/b.log 2>&1 & p2=$!; " + "wait $p1; r1=$?; wait $p2; r2=$?; " + "echo '--- client a ---'; cat /tmp/a.log; " + "echo '--- client b ---'; cat /tmp/b.log; " + "exit $((r1 + r2))" + ) + ) + except Exception: + dump("concurrent clients were not both served") + raise + + with subtest("a client dying mid-stream leaves the server serving"): + try: + machine.succeed( + "timeout 5 python3 -c 'import socket;" + f"s=socket.socket(socket.AF_UNIX);s.connect(\"{SOCKET}\");s.recv(1);s.close()'" + ) + print(machine.succeed(f"{ASSERT} --frames 2 --timeout 60 --require-png")) + machine.succeed("systemctl is-active hyprland-headless-stream.service") + except Exception: + dump("server did not survive a client dying mid-frame") + raise + ''; +} diff --git a/checks/vm-hyprland.nix b/checks/vm-hyprland.nix new file mode 100644 index 0000000..5b04a46 --- /dev/null +++ b/checks/vm-hyprland.nix @@ -0,0 +1,157 @@ +# checks..vm-hyprland -- the real one. Same machine as vm-basic, but no +# `captureCommand` override, so the server captures the actual headless +# compositor through grim. +# +# This is the check expected to be flaky, so it is deliberately impatient: every +# wait has a bounded timeout, nothing is retried in a loop, and any failure dumps +# the compositor's log and `hyprctl monitors` before it propagates. A flake you +# can read is worth more than a flake that eventually goes green. +{ + testers, + hhstream, + testTools, + hhsModule, + hyprland, + grim, + width ? 1280, + height ? 720, + fps ? 30, + socketPath ? "/run/hyprland-headless-stream/stream.sock", +}: +testers.runNixOSTest { + name = "hhstream-vm-hyprland"; + + globalTimeout = 1800; + + nodes.machine = { + imports = [ hhsModule ]; + + services.hyprland-headless-stream = { + enable = true; + package = hhstream; + hyprlandPackage = hyprland; + inherit socketPath; + frameRate = fps; + format = "png"; + outputs = [ + { + name = "HEADLESS-1"; + inherit width height; + refresh = fps; + } + ]; + # No captureCommand: the module's default (grim) is the point of this check. + }; + + # hyprctl and grim are here for the diagnostics dump and for the manual + # cross-check below, not for the code path under test. + environment.systemPackages = [ + testTools + hhstream + hyprland + grim + ]; + + # The compositor needs more room than the fake capture did, and its software + # renderer wants a CPU. There is still no GPU: Hyprland runs on its headless + # backend over llvmpipe. + virtualisation = { + memorySize = 2048; + cores = 2; + diskSize = 4096; + graphics = false; + }; + }; + + testScript = '' + SOCKET = "${socketPath}" + WIDTH, HEIGHT = ${toString width}, ${toString height} + + + def dump(label): + print(f"########## diagnostics: {label} ##########") + status, out = machine.execute("hhs-hyprland-diagnostics", timeout=120) + print(out) + print(f"########## end diagnostics (exit {status}) ##########") + + + start_all() + + with subtest("the compositor unit comes up"): + try: + machine.wait_for_unit("hyprland-headless.service", timeout=180) + except Exception: + dump("hyprland-headless.service did not start") + raise + + with subtest("the compositor reports the configured headless output"): + try: + # Proves the compositor is really running a 1280x720 HEADLESS-1 + # before we blame the capture for a wrong-sized frame. + out = machine.succeed("hhs-hyprland-diagnostics", timeout=120) + print(out) + assert "HEADLESS-1" in out, ( + "hyprctl never reported a HEADLESS-1 monitor; see the dump above" + ) + assert f"{WIDTH}x{HEIGHT}" in out, ( + f"hyprctl never reported a {WIDTH}x{HEIGHT} mode; see the dump above" + ) + except Exception: + dump("compositor output not as configured") + raise + + with subtest("the frame server unit comes up and publishes the socket"): + try: + machine.wait_for_unit("hyprland-headless-stream.service", timeout=180) + machine.wait_for_file(SOCKET, timeout=120) + except Exception: + dump("frame server did not publish the socket") + raise + + with subtest("grim can capture the session at all"): + # Separates "grim cannot see the compositor" from "the server is wrong", + # which are the two failures that look identical from the socket. + try: + print( + machine.succeed( + "systemd-run --pipe --wait --uid=hhstream" + " --setenv=XDG_RUNTIME_DIR=/run/user/$(id -u hhstream)" + " grim -t png /tmp/direct.png 2>&1 || true", + timeout=120, + ) + ) + print(machine.execute("stat -c '%s bytes' /tmp/direct.png")[1]) + except Exception: + dump("direct grim capture failed") + raise + + with subtest("a frame off the socket is a real PNG of the configured size"): + try: + print( + machine.succeed( + f"hhs-assert-stream --socket {SOCKET} --frames 1 --timeout 60" + f" --expect-width {WIDTH} --expect-height {HEIGHT}" + " --expect-format image/png --require-png --reject-blank", + timeout=180, + ) + ) + except Exception: + dump("frame off the socket failed the PNG assertions") + raise + + with subtest("the stream keeps producing frames"): + try: + print( + machine.succeed( + f"hhs-assert-stream --socket {SOCKET} --frames 3 --timeout 60" + f" --expect-width {WIDTH} --expect-height {HEIGHT}" + " --require-png --reject-blank", + timeout=240, + ) + ) + machine.succeed("systemctl is-active hyprland-headless-stream.service") + except Exception: + dump("the stream stopped after the first frame") + raise + ''; +} diff --git a/flake.nix b/flake.nix index 3a46af3..65e876e 100644 --- a/flake.nix +++ b/flake.nix @@ -8,6 +8,10 @@ outputs = { self, nixpkgs, ... }: let systems = [ "x86_64-linux" "aarch64-linux" ]; forAllSystems = f: nixpkgs.lib.genAttrs systems (system: f (import nixpkgs { inherit system; })); + # `checks` needs the system name as well as the package set, to reach + # self.packages.; forAllSystems only hands over pkgs. + forAllSystems' = f: nixpkgs.lib.genAttrs systems (system: + f { inherit system; pkgs = import nixpkgs { inherit system; }; }); in { packages = forAllSystems (pkgs: rec { hhstream = pkgs.callPackage ./nix/package.nix {}; @@ -19,8 +23,10 @@ default = hyprland-headless-stream; }; - # `checks` (unit tests + the VM integration test) is owned by the - # test-harness workstream -- see docs/CONTRACT.md for the interface. + # Unit tests + the two VM integration tests. See checks/default.nix and + # README.dev.md; the interface they assert is docs/CONTRACT.md. + checks = forAllSystems' ({ pkgs, system }: + import ./checks { inherit pkgs self system; }); devShells = forAllSystems (pkgs: { default = pkgs.mkShell { diff --git a/tests/assert_stream.py b/tests/assert_stream.py new file mode 100644 index 0000000..a892f99 --- /dev/null +++ b/tests/assert_stream.py @@ -0,0 +1,307 @@ +#!/usr/bin/env python3 +"""Read a hhstream socket and assert it obeys docs/CONTRACT.md. + +This is deliberately a *second, independent* implementation of the reader: it +shares no code with `hhstream.client`, so a bug that lives in both the server's +writer and the shipped client's reader still fails the test. Pure standard +library; it runs inside the test VM, where only python3 is guaranteed. + +Every run prints what it saw before it decides, so a failure in CI is readable +without reproducing it. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import re +import socket +import struct +import sys +import zlib + +MAGIC = "HHS1" +FORMATS = {"image/png", "image/jpeg"} +PNG_SIGNATURE = b"\x89PNG\r\n\x1a\n" +# Contract: plain ASCII digits, no leading zeros, no `+`, no spaces. +LENGTH_RE = re.compile(rb"\A(?:0|[1-9][0-9]*)\Z") + + +class ContractError(Exception): + """The stream violated docs/CONTRACT.md.""" + + +# -------------------------------------------------------------------------- +# wire reading +# -------------------------------------------------------------------------- + + +def read_line(sock: socket.socket, limit: int = 4096) -> bytes: + """Read up to and including the next 0x0A. Never over-reads past it.""" + out = bytearray() + while True: + b = sock.recv(1) + if not b: + raise ContractError( + f"socket closed after {len(out)} bytes while reading a line: {bytes(out)!r}" + ) + if b == b"\n": + return bytes(out) + out += b + if len(out) > limit: + raise ContractError(f"no newline within {limit} bytes: {bytes(out[:120])!r}...") + + +def read_exact(sock: socket.socket, n: int) -> bytes: + out = bytearray() + while len(out) < n: + part = sock.recv(min(65536, n - len(out))) + if not part: + raise ContractError( + f"truncated payload: wanted {n} bytes, socket closed after {len(out)}" + ) + out += part + return bytes(out) + + +def read_hello(sock: socket.socket) -> dict: + line = read_line(sock) + try: + obj = json.loads(line.decode("utf-8")) + except (UnicodeDecodeError, json.JSONDecodeError) as exc: + raise ContractError(f"Hello is not UTF-8 JSON ({exc}): {line!r}") from exc + if not isinstance(obj, dict): + raise ContractError(f"Hello is not a JSON object: {line!r}") + + if obj.get("magic") != MAGIC: + raise ContractError(f"Hello magic is {obj.get('magic')!r}, contract says {MAGIC!r}") + for key in ("width", "height", "fps"): + val = obj.get(key) + if not isinstance(val, int) or isinstance(val, bool): + raise ContractError(f"Hello {key} is {val!r}, expected an integer") + if val <= 0: + raise ContractError( + f"Hello {key} is {val}; the contract forbids a Hello with zero size" + ) + if obj.get("format") not in FORMATS: + raise ContractError( + f"Hello format is {obj.get('format')!r}, expected one of {sorted(FORMATS)}" + ) + return obj + + +def read_frame(sock: socket.socket) -> bytes: + prefix = read_line(sock) + if not LENGTH_RE.match(prefix): + raise ContractError( + f"malformed length prefix {prefix!r}: contract requires plain ASCII " + "digits with no leading zeros, sign or spaces" + ) + length = int(prefix) + if length == 0: + raise ContractError("frame length is 0; a frame must carry a complete image") + return read_exact(sock, length) + + +# -------------------------------------------------------------------------- +# PNG inspection +# -------------------------------------------------------------------------- + + +def png_chunks(data: bytes): + off = len(PNG_SIGNATURE) + while off + 8 <= len(data): + (size,) = struct.unpack(">I", data[off : off + 4]) + kind = data[off + 4 : off + 8] + body = data[off + 8 : off + 8 + size] + if len(body) != size: + raise ContractError(f"PNG chunk {kind!r} truncated: want {size}, got {len(body)}") + crc = data[off + 8 + size : off + 12 + size] + if len(crc) != 4: + raise ContractError(f"PNG chunk {kind!r} has no CRC") + want = struct.pack(">I", zlib.crc32(kind + body) & 0xFFFFFFFF) + if crc != want: + raise ContractError(f"PNG chunk {kind!r} CRC mismatch: {crc.hex()} != {want.hex()}") + yield kind, body + off += 12 + size + if kind == b"IEND": + return + raise ContractError("PNG ended without an IEND chunk") + + +def inspect_png(payload: bytes) -> dict: + """Parse enough of a PNG to know its size and whether it carries a picture.""" + if not payload.startswith(PNG_SIGNATURE): + raise ContractError( + f"payload is not a PNG: first 8 bytes are {payload[:8]!r}, " + f"expected {PNG_SIGNATURE!r}" + ) + ihdr = None + idat = bytearray() + for kind, body in png_chunks(payload): + if kind == b"IHDR": + if ihdr is not None: + raise ContractError("PNG has more than one IHDR") + if len(body) != 13: + raise ContractError(f"IHDR is {len(body)} bytes, expected 13") + w, h, depth, colour, comp, filt, interlace = struct.unpack(">IIBBBBB", body) + ihdr = dict( + width=w, height=h, bit_depth=depth, colour_type=colour, + compression=comp, filter=filt, interlace=interlace, + ) + elif kind == b"IDAT": + if ihdr is None: + raise ContractError("PNG has IDAT before IHDR") + idat += body + if ihdr is None: + raise ContractError("PNG has no IHDR chunk") + if not idat: + raise ContractError("PNG has no IDAT chunk: there is no image data") + + try: + raw = zlib.decompress(bytes(idat)) + except zlib.error as exc: + raise ContractError(f"PNG image data does not decompress: {exc}") from exc + + info = dict(ihdr) + info["payload_bytes"] = len(payload) + info["raw_bytes"] = len(raw) + info["distinct_raw_bytes"] = len(set(raw)) + info["nonzero_raw_bytes"] = len(raw) - raw.count(0) + return info + + +def blank_png_size(width: int, height: int, colour_type: int, bit_depth: int) -> int: + """Smallest plausible encoding of an all-zero image of the same geometry. + + Used as the floor for "bigger than a trivial blank image": a zero-filled + framebuffer, packed with the best compression available, cannot exceed this. + """ + channels = {0: 1, 2: 3, 3: 1, 4: 2, 6: 4}.get(colour_type, 3) + stride = 1 + (width * channels * bit_depth + 7) // 8 + raw = b"\x00" * (stride * height) + body = zlib.compress(raw, 9) + # signature + IHDR (25) + IDAT header/crc (12) + IEND (12) + return len(PNG_SIGNATURE) + 25 + 12 + len(body) + 12 + + +# -------------------------------------------------------------------------- +# driver +# -------------------------------------------------------------------------- + + +def build_parser() -> argparse.ArgumentParser: + p = argparse.ArgumentParser(description=__doc__) + p.add_argument("--socket", required=True, help="path of the hhstream socket") + p.add_argument("--frames", type=int, default=1, help="how many frames to read") + p.add_argument("--timeout", type=float, default=30.0, + help="per-recv socket timeout, seconds") + p.add_argument("--expect-width", type=int, help="assert Hello/IHDR width") + p.add_argument("--expect-height", type=int, help="assert Hello/IHDR height") + p.add_argument("--expect-fps", type=int, help="assert Hello fps") + p.add_argument("--expect-format", help="assert Hello format, e.g. image/png") + p.add_argument("--min-bytes", type=int, default=1, + help="every payload must be at least this many bytes") + p.add_argument("--require-png", action="store_true", + help="every payload must parse as a PNG whose IHDR matches the Hello") + p.add_argument("--identical-frames", action="store_true", + help="all payloads read must be byte-identical") + p.add_argument("--reject-blank", action="store_true", + help="reject a frame that decodes to an all-zero or single-valued " + "image, or that is no bigger than a best-case blank PNG") + return p + + +def run(args) -> None: + sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + sock.settimeout(args.timeout) + sock.connect(args.socket) + try: + hello = read_hello(sock) + print(f"hello: {json.dumps(hello, sort_keys=True)}", flush=True) + + for key, got in (("width", "expect_width"), ("height", "expect_height"), + ("fps", "expect_fps"), ("format", "expect_format")): + want = getattr(args, got) + if want is not None and hello[key] != want: + raise ContractError(f"Hello {key} is {hello[key]!r}, expected {want!r}") + + if args.require_png and hello["format"] != "image/png": + raise ContractError( + f"--require-png but Hello advertises format {hello['format']!r}" + ) + + payloads: list[bytes] = [] + for i in range(args.frames): + payload = read_frame(sock) + digest = hashlib.sha256(payload).hexdigest()[:16] + print(f"frame {i}: {len(payload)} bytes sha256:{digest}", flush=True) + if len(payload) < args.min_bytes: + raise ContractError( + f"frame {i} is {len(payload)} bytes, below the {args.min_bytes}-byte floor" + ) + payloads.append(payload) + + if args.require_png: + info = inspect_png(payload) + print(f"frame {i}: png {json.dumps(info, sort_keys=True)}", flush=True) + if (info["width"], info["height"]) != (hello["width"], hello["height"]): + raise ContractError( + f"frame {i} IHDR is {info['width']}x{info['height']} but the Hello " + f"announced {hello['width']}x{hello['height']}" + ) + if args.reject_blank: + floor = blank_png_size( + info["width"], info["height"], info["colour_type"], info["bit_depth"] + ) + print(f"frame {i}: blank-png floor is {floor} bytes", flush=True) + if info["nonzero_raw_bytes"] == 0: + raise ContractError( + f"frame {i} decodes to {info['raw_bytes']} zero bytes: " + "this is a zero-filled buffer, not a captured frame" + ) + if info["distinct_raw_bytes"] < 2: + raise ContractError( + f"frame {i} decodes to a single repeated byte value " + f"({info['raw_bytes']} bytes): a blank image, not a capture" + ) + if len(payload) <= floor: + raise ContractError( + f"frame {i} is {len(payload)} bytes, no bigger than a " + f"best-case blank {info['width']}x{info['height']} PNG " + f"({floor} bytes)" + ) + + if args.identical_frames and len(payloads) > 1: + first = payloads[0] + for i, payload in enumerate(payloads[1:], start=1): + if payload != first: + raise ContractError( + f"frame {i} differs from frame 0 ({len(payload)} vs {len(first)} " + "bytes) but the fake capture emits one fixed image" + ) + print(f"all {len(payloads)} frames byte-identical", flush=True) + finally: + sock.close() + + +def main(argv: list[str] | None = None) -> int: + args = build_parser().parse_args(argv) + if args.frames < 1: + print("assert-stream: --frames must be at least 1", file=sys.stderr) + return 2 + try: + run(args) + except ContractError as exc: + print(f"CONTRACT VIOLATION: {exc}", file=sys.stderr) + return 1 + except (OSError, socket.timeout) as exc: + print(f"SOCKET ERROR on {args.socket}: {exc!r}", file=sys.stderr) + return 1 + print("OK", flush=True) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/fake_capture.py b/tests/fake_capture.py new file mode 100644 index 0000000..e24c542 --- /dev/null +++ b/tests/fake_capture.py @@ -0,0 +1,89 @@ +#!/usr/bin/env python3 +"""A fake capture backend: writes one deterministic PNG to stdout and exits. + +Stands in for `grim` so `vm-basic` can exercise the socket, the Hello and the +framing without a compositor. Two properties matter to the test: + +* the bytes are a *valid* PNG of exactly the requested size, so a server that + derives the Hello's width/height from the captured image agrees with a server + that derives them from the configured output; +* the bytes are *byte-identical* on every invocation, so the test can assert + that two frames off the wire are the same image rather than the same length. + +Pure standard library (zlib + struct), because the test VM has nothing else. +""" + +from __future__ import annotations + +import argparse +import struct +import sys +import zlib + + +def chunk(kind: bytes, data: bytes) -> bytes: + return ( + struct.pack(">I", len(data)) + + kind + + data + + struct.pack(">I", zlib.crc32(kind + data) & 0xFFFFFFFF) + ) + + +def encode_png(width: int, height: int) -> bytes: + """A solid frame with one deterministic marker rectangle in the corner. + + The marker keeps the image from being uniform, which is what makes it a + believable stand-in for a real capture; being fixed keeps it reproducible. + """ + bg = (0x20, 0x30, 0x40) + fg = (0xE0, 0xC0, 0x10) + mark_w = min(64, width) + mark_h = min(32, height) + + plain = b"\x00" + bytes(bg) * width + marked = b"\x00" + bytes(fg) * mark_w + bytes(bg) * (width - mark_w) + raw = marked * mark_h + plain * (height - mark_h) + + ihdr = struct.pack(">IIBBBBB", width, height, 8, 2, 0, 0, 0) # 8-bit truecolour + return ( + b"\x89PNG\r\n\x1a\n" + + chunk(b"IHDR", ihdr) + + chunk(b"IDAT", zlib.compress(raw, 9)) + + chunk(b"IEND", b"") + ) + + +def build_parser() -> argparse.ArgumentParser: + p = argparse.ArgumentParser(description=__doc__) + p.add_argument("--width", type=int, default=1280) + p.add_argument("--height", type=int, default=720) + # Accepted and ignored so the fake can be dropped in where a grim-shaped + # argv is expected (`-t png -`). + p.add_argument("-t", "--type", default="png") + p.add_argument("output", nargs="?", default="-") + return p + + +def main(argv: list[str] | None = None) -> int: + args = build_parser().parse_args(argv) + if args.type != "png": + print(f"fake-capture: only png is supported, got {args.type!r}", file=sys.stderr) + return 2 + if args.width <= 0 or args.height <= 0: + print("fake-capture: width and height must be positive", file=sys.stderr) + return 2 + + data = encode_png(args.width, args.height) + out = sys.stdout.buffer if args.output == "-" else open(args.output, "wb") + try: + out.write(data) + out.flush() + finally: + if out is not sys.stdout.buffer: + out.close() + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/hyprland_diagnostics.sh b/tests/hyprland_diagnostics.sh new file mode 100755 index 0000000..b884382 --- /dev/null +++ b/tests/hyprland_diagnostics.sh @@ -0,0 +1,82 @@ +#!/bin/sh +# Dump everything worth knowing when `vm-hyprland` fails. +# +# The compositor's own log is the thing you actually want and the thing that is +# hardest to find: Hyprland writes it under +# $XDG_RUNTIME_DIR/hypr//hyprland.log, and the service runs +# on a private XDG_RUNTIME_DIR. So rather than guess the path, look for it. +# +# Never fails: a diagnostics dump that exits non-zero hides the real error. + +PATH="@diagPath@:$PATH" +export PATH + +if [ "$1" = "--self-test" ]; then + echo "hhs-hyprland-diagnostics: ok" + exit 0 +fi + +SERVICE_UNIT=hyprland-headless-stream.service +COMPOSITOR_UNIT=hyprland-headless.service + +say() { printf '\n----- %s -----\n' "$*"; } +run() { say "$*"; "$@" 2>&1 || echo "[exit $?]"; } + +say "units" +systemctl --no-pager --plain list-units --all 'hyprland*' 2>&1 || true + +for unit in "$COMPOSITOR_UNIT" "$SERVICE_UNIT"; do + run systemctl --no-pager --full status "$unit" + run journalctl --no-pager --boot --lines=200 --unit "$unit" +done + +run systemctl --no-pager --full list-dependencies "$SERVICE_UNIT" + +say "socket and runtime directories" +ls -la /run/hyprland-headless-stream 2>&1 || echo "[no /run/hyprland-headless-stream]" +ls -la /run/user 2>&1 || echo "[no /run/user]" +for d in /run/user/*; do + [ -d "$d" ] && ls -la "$d" 2>&1 +done + +say "hyprland logs" +# hyprland.log lives beside the compositor's IPC sockets; -xdev keeps this out +# of /nix/store and /proc. +found=0 +for root in /run /var/log /tmp /home; do + [ -d "$root" ] || continue + for log in $(find "$root" -xdev -maxdepth 6 -name 'hyprland*.log' 2>/dev/null); do + found=1 + say "tail -n 200 $log" + tail -n 200 "$log" 2>&1 + done +done +[ "$found" = 1 ] || echo "[no hyprland*.log anywhere under /run /var/log /tmp /home]" + +say "hyprctl monitors" +# Needs both XDG_RUNTIME_DIR and the instance signature, which is the directory +# name under /hypr. Try every instance we can find. +tried=0 +for hyprdir in /run/user/*/hypr/* /run/*/hypr/*; do + [ -d "$hyprdir" ] || continue + sig=$(basename "$hyprdir") + runtime=$(dirname "$(dirname "$hyprdir")") + owner=$(stat -c %U "$hyprdir" 2>/dev/null || echo root) + tried=1 + say "hyprctl monitors (instance $sig, XDG_RUNTIME_DIR=$runtime, as $owner)" + if command -v hyprctl > /dev/null 2>&1; then + XDG_RUNTIME_DIR="$runtime" HYPRLAND_INSTANCE_SIGNATURE="$sig" \ + hyprctl monitors 2>&1 || echo "[exit $?]" + XDG_RUNTIME_DIR="$runtime" HYPRLAND_INSTANCE_SIGNATURE="$sig" \ + hyprctl version 2>&1 || echo "[exit $?]" + else + echo "[hyprctl not on PATH; add the hyprlandPackage to environment.systemPackages]" + fi +done +[ "$tried" = 1 ] || echo "[no hypr instance directory found: the compositor never came up]" + +say "processes" +ps aux 2>&1 | grep -E 'hypr|hhstream|grim' | grep -v grep || echo "[no hyprland/hhstream/grim process]" + +say "end of diagnostics" +exit 0 diff --git a/tests/unit/test_package.py b/tests/unit/test_package.py new file mode 100644 index 0000000..bd6f99f --- /dev/null +++ b/tests/unit/test_package.py @@ -0,0 +1,38 @@ +"""The parts of the package that are plain constants, checked against the contract. + +These are the only tests here that do not depend on an implementation, which +makes them the canary for the `unit` check's own wiring: if they fail, the +package is not importable from the build sandbox and nothing else means anything. +""" + +from __future__ import annotations + +import hhstream +from hhstream import capture, client, protocol, server + +from conftest import CONTRACT_SOCKET + + +def test_magic_matches_the_contract(): + # docs/CONTRACT.md: {"magic":"HHS1", ...} + assert hhstream.MAGIC == "HHS1" + assert protocol.MAGIC == "HHS1" + + +def test_default_socket_path_matches_the_contract(): + assert server.DEFAULT_SOCKET == CONTRACT_SOCKET + assert client.DEFAULT_SOCKET == CONTRACT_SOCKET + + +def test_public_surface_is_present(): + for name in ("hello", "parse_hello", "frame", "read_frame", "read_hello"): + assert callable(getattr(protocol, name)), f"protocol.{name}" + assert callable(client.fetch) + assert callable(client.main) + assert callable(server.main) + assert isinstance(server.FrameServer, type) + assert isinstance(capture.CommandCapture, type) + + +def test_version_is_the_packaged_version(): + assert hhstream.__version__ == "0.1.0"