Everything below the CLI was merged and green against fakes; none of it was
reachable and none of it had ever met a compositor. This wires it up and then
puts the compositor in a position to disagree.
CLI (src/hhstream/cli.py, src/hhstream_session/**)
session container [ID], session reconcile [ID], and the seven pointer verbs
from PLAN §9. `start` still prints the bare session id and nothing else.
`list` gains WS / OUTPUT / POINTER, with the gap marker where a value is
genuinely unknown rather than a blank that reads as zero -- the POINTER
column needs a second, read-only `pointer.state` call because the `list`
reply carries no lease state.
The lease id is cached in `hhstream_session.lease` (under
$XDG_RUNTIME_DIR), because PLAN §9's gesture verbs take no --lease and the
wire requires a lease_id on every one of them; --lease and $HHSTREAM_LEASE
override it, and a `no_lease` refusal drops the stale entry.
Exit codes are the contract's: a gesture with no lease is 9 and never
reaches the broker, and a compositor failure is 10 with a message that says
the session is still running. `kill` now warns about
`session.container_survivors` -- windows the compositor still shows -- and
distinguishes `null` ("could not ask") from `[]`.
`shot`/`record --output NAME` capture one container through `grim -o`; the
frame socket publishes the main output only, so it cannot do it.
Module (nix/module.nix)
sessions.containers.{enable,max,mode,workspaceBase},
sessions.pointer.{enable,leaseTtl,leaseMax}, rendered into the generated
TOML with §10's key names. hhstream-pointer.service: the operator's account,
supplementary group hhstream, After=/BindsTo= the compositor,
Restart=on-failure, KillMode=process, and the command line §3.4 freezes.
tmpfiles for /run/hhstream plus RuntimeDirectoryPreserve on the broker,
without which restarting the broker deletes the driver's socket out from
under it. The runtime-directory length assertion is untouched.
And ExecStartPost on the compositor, opening its IPC directory to its own
group: Hyprland creates <runtime>/hypr with S_IRWXU, so the broker -- a
different account by design -- could not reach the request or event socket
at all, and degraded silently exactly as it is built to.
Checks (checks/**)
vm-ipc: both sockets at their frozen names, measured against sun_path's 108
bytes, and a socket2 subscriber that receives an openwindow for a window the
test creates. The subscriber is the shipped HyprlandIPC, not a second
implementation: this check exists to prove that this client hears this
compositor.
vm-containers: one hh-* output per session at the container mode, the window
on it, a second window of the same application beside it, an unmatched
window quarantined, grim -o isolating one container, teardown removing the
output and freeing the slot. PLAN experiment 3 is answered: `hyprctl output
create headless` works.
vm-pointer: a version-2 virtual pointer, a warp and a click whose
coordinates the *application* reports, BUSY naming the holder, an expired
lease swept with its held button lifted, and a broker restart that leaves
the lock free, the bookmark intact and the driver reachable.
checks/lib gains the guest the three share, their test-script helpers (no
assertions live there), an event-socket subscriber and a Wayland client that
says where it was clicked -- nothing in nixpkgs both honours
$HHSTREAM_APP_ID and reports pointer events.
Reported, not worked around: the broker reads none of the container or
pointer config (policy.py takes [defaults] and [apps] only), `start` has no
no_container field so --no-container is refused rather than silently ignored,
and nix/hyprland-conf.nix sets input:follow_mouse = 0, which sends pointer
events to the focused window only -- measured, with the cursor at the aimed
pixel and no client told. README.dev.md has the evidence for each.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WQy29x9AfHWyzNd1f7XPGK
152 lines
7.4 KiB
Markdown
152 lines
7.4 KiB
Markdown
# examples
|
||
|
||
## `agent-eyes` — the short commands
|
||
|
||
The everyday front door to the session: one subcommand each, no environment
|
||
juggling, and it re-execs itself under `sg hhstream` when the login predates the
|
||
group membership.
|
||
|
||
```bash
|
||
examples/agent-eyes status # services, socket, group access, liveness probe
|
||
examples/agent-eyes shot --out /tmp/s.jpg # one frame of the screen
|
||
examples/agent-eyes run -- chromium ... # launch a GUI app INTO the session
|
||
examples/agent-eyes record --seconds 10 --out clip.mp4
|
||
```
|
||
|
||
- `shot` defaults to `/var/lib/hermes/public/agent-eyes/latest.jpg`, which the
|
||
local file server already publishes, so the newest frame is always one
|
||
browser refresh away.
|
||
- `run` sets `XDG_RUNTIME_DIR`, `WAYLAND_DISPLAY` (it discovers whichever
|
||
`wayland-N` exists rather than assuming `wayland-0`) and `XDG_SESSION_TYPE`,
|
||
then `exec`s the command. Everything it launches runs as *your* user, not the
|
||
service user.
|
||
- `record` reads frames off the socket for the given wall-clock seconds and
|
||
encodes at the **measured** rate, so the clip plays at true speed. It strips
|
||
metadata for the media stores, same as the demo.
|
||
|
||
Both `shot` and `record` want nothing but `hhstream-client`, `python3` and
|
||
`ffmpeg`, which are all on the system PATH; only `chromium` needs `nix develop`.
|
||
|
||
## The session verbs these helpers do not wrap
|
||
|
||
`agent-eyes` is the *frame* side of the session: look at the screen, put an app
|
||
on it, film it. Everything that tracks an application, gives it its own screen
|
||
or drives a cursor at it lives on the `hhstream session` namespace and is not
|
||
wrapped here, because it needs the broker and `agent-eyes` deliberately does
|
||
not:
|
||
|
||
```bash
|
||
hhstream session start chromium -- https://example.com # prints the bare id
|
||
hhstream session list # WS, OUTPUT, POINTER
|
||
hhstream session container <id> # its output and workspace
|
||
hhstream shot --output hh-<id> --out /tmp/just-that.png # one container alone
|
||
|
||
hhstream session pointer acquire <id> # prints the lease id
|
||
hhstream session pointer move <id> --to 0.5,0.5 # 0..1, within ITS output
|
||
hhstream session pointer click <id>
|
||
hhstream session pointer release <id>
|
||
```
|
||
|
||
Two things worth knowing before using them:
|
||
|
||
- **Coordinates are output-normalised, 0..1**, inside that session's own
|
||
container — never window-local pixels. The broker's IPC client carries no
|
||
window geometry, so there is nothing honest to convert from.
|
||
- **No pointer verb counts as activity.** A session being driven is still a
|
||
session nobody has said they want kept; only `touch` resets the idle clock.
|
||
|
||
The lease id is cached under `$XDG_RUNTIME_DIR/hhstream`, which is why the
|
||
gesture verbs need no flag; `--lease` and `$HHSTREAM_LEASE` override it.
|
||
|
||
## `agent-eyes-demo.py`
|
||
|
||
Puts a real browser on the headless screen, drives it, and films the result by
|
||
reading frames off the published socket — the consumer side of
|
||
[`docs/CONTRACT.md`](../docs/CONTRACT.md).
|
||
|
||
```bash
|
||
nix develop
|
||
sg hhstream -c 'python3 examples/agent-eyes-demo.py --query "Ari Sosnovsky"'
|
||
```
|
||
|
||
Output is an MP4 (default `agent-eyes.mp4`): the Google homepage, the query
|
||
typed in a letter at a time, then the results page — captured from the
|
||
compositor, not from a screencast API.
|
||
|
||
Everything else is a flag: `--seconds`, `--capture-fps`, `--output`,
|
||
`--work-dir`, `--keep-work`, `--debug-port`, `--clear-screen`. `--help` is
|
||
accurate.
|
||
|
||
### What it checks rather than assumes
|
||
|
||
- the session is up and the socket is readable (a clear message if your user is
|
||
not in the stream's group);
|
||
- the browser actually reached a results page — it presses Enter, then waits for
|
||
the URL to change, and submits the form directly if Enter did nothing;
|
||
- it *fails* on Google's `/sorry/` bot wall instead of encoding a video of a
|
||
CAPTCHA, because a video that looks like a working demo and is not one is
|
||
worse than no video.
|
||
|
||
### Three things that will bite you
|
||
|
||
1. **Your shell needs the group.** The socket is `0660 hhstream:hhstream` inside
|
||
a `0750` directory; a browser also has to reach the session's Wayland socket
|
||
in the same directory. Add yourself to `hhstream` (this is what
|
||
`/etc/nixos/hyprland-stream.nix` does for `hermes`) and log in again — or use
|
||
`sg hhstream -c '...'` for the shell you already have open.
|
||
2. **Google rate-limits you.** The first search from a fresh profile usually
|
||
passes; the third in a minute gets `unusual traffic`. It is the IP, not the
|
||
browser. Wait a few minutes and re-run.
|
||
3. **The frame server was the bottleneck, and it was PNG.** Measured with `grim`
|
||
at 1280×720 on this software-rendered VM: PNG ~326 ms/frame (a ~3 fps
|
||
ceiling), PNG `-l 1` ~105 ms, **JPEG q80 46 ms**. The session therefore
|
||
streams JPEG and the demo records ~22 fps. The script never assumes the
|
||
format — it takes the extension from the Hello and hands it to ffmpeg — so
|
||
either setting works. Leave `--capture-fps` at its default of 30 so nothing
|
||
is thrown away: the encoder uses the *measured* rate, so the video plays at
|
||
real speed either way, and decimating only makes it choppier.
|
||
|
||
### It films a desktop, not a browser
|
||
|
||
The browser is **windowed by default** (1040×560, floating, centred), so the
|
||
session's wallpaper, gaps and window borders are all in shot — that is the
|
||
difference between "a screen recording of a browser" and "here is the machine I
|
||
look at". `--fullscreen` gives the old full-bleed kiosk window.
|
||
|
||
The desktop furniture itself lives in `/etc/nixos/hyprland-stream.nix` (wallpaper
|
||
via `swaybg`, `gaps_out = 36`, a catch-all float window rule), not in this repo:
|
||
it is this machine's taste, not part of the kit.
|
||
|
||
Two notes for anyone editing that file:
|
||
|
||
- Window rules are **v3** syntax here (`windowrule = match:class .*, float on`).
|
||
The v2 spelling (`float, class:^(chromium)$`) parses as an unknown field and
|
||
Hyprland draws the error *across the screen*, which lands in the recording.
|
||
Always check an edit with `Hyprland --verify-config --config <file>` — it does
|
||
not need a display, so it costs seconds instead of a rebuild.
|
||
- `hyprctl` **works** against this session; an older note here claimed the
|
||
compositor misnames its IPC socket, which was wrong. The real defect was path
|
||
length: `$XDG_RUNTIME_DIR` was `/run/hyprland-headless-stream` (29 chars) and
|
||
the event socket `hypr/<signature>/.socket2.sock` did not fit in `sun_path`, so
|
||
the compositor logged `ERR ]: Socket2 path is too long. (2) IPC will not work.`
|
||
while the request socket kept answering — which is why the wrong diagnosis
|
||
survived. The runtime directory is now `/run/hypr-stream`, and `nix/module.nix`
|
||
asserts it stays ≤ 25 characters.
|
||
|
||
### Posting the result to a chat
|
||
|
||
Xiph/Blossom media stores reject files that carry metadata. `encode()` in the
|
||
script already strips it with `-map_metadata -1 -fflags +bitexact -flags:v
|
||
+bitexact`; without those flags a Buzz relay answers
|
||
`422 media contains metadata or a non-canonical metadata channel` and the upload
|
||
fails. Uploading from the command line, with the same script-encoded file:
|
||
|
||
```bash
|
||
buzz upload file --file agent-eyes.mp4 # -> {"url": ".../media/<sha256>.mp4"}
|
||
buzz media get -o check.mp4 "<url>" # fetch it back and compare sha256
|
||
```
|
||
|
||
`buzz upload file` takes `--file <path>`, not a positional argument, and exits
|
||
**0** with a JSON error on stdout when the relay refuses the type — read the
|
||
output, never the exit status.
|