docs+examples: log the cursor marker as open, and film each desk on its own

The marker is now written up as the *open* item rather than a research note: it
is not implemented, the video went out without it, and everything else about the
pointer is exact and readable.

proof-video.py is rewritten for desks, one command, no arguments. It films the
session's own output with `record --output` instead of the published socket --
with desks, the socket carries every output stitched together, so a viewer would
watch two thirds wallpaper. The scrolling beats are real pointer axis events,
down twice and back up once, so the movement cannot be mistaken for a page that
merely loaded a different part of itself.
This commit is contained in:
Hermenix committed 2026-10-03 19:54:08 +00:00
1 parent cbf58f6443
commit 4b5cb5197a
2 files changed
+159 -45

No files matched your search

@@ -1,5 +1,14 @@
# The compositor draws no cursor for a virtual pointer
## What is open
**The overlay marker is not implemented.** The acceptance video was recorded
without it, so the frames show the pointer's *effects* — a page that scrolled,
a tab that opened — and never the pointer itself. Implement it by stamping
`hyprctl cursorpos` into each frame, mapped into the stitched layout, and label
it as an overlay wherever the frame is published. Nothing else about the pointer
is blocked: position, clicks and scrolling are all exact and all readable.
## The issue — answered, and it is not a capture setting
The pointer moves exactly where it is told, and **nothing is drawn**. Every
+150 -45
View File
@@ -1,35 +1,41 @@
#!/usr/bin/env python3
"""Drive the session's browser over CDP, then put a terminal next to it.
"""The acceptance video: an SKS search, real scrolling, tabs, and cowsay.
The acceptance proof for Agent Eyes: the agent searches Gunpost for an SKS, opens
a few listings in their own tabs, clears the browser away, and runs cowsay in a
terminal on the same screen. It is the demonstration that the agent can see and
work a real desktop, and it lives here rather than in a scratch directory because
it has been asked for more than once and a tool kept in scratch gets pruned.
One command, no arguments, and it cleans up after itself. It starts a browser
session (which gets a desk of its own — its own output and workspace), films
*that desk alone*, drives the search over the browser's own HTTP endpoints, and
scrolls the results page with the real pointer -- `hhstream session pointer
scroll`, which sends wl_pointer axis events rather than pressing a key.
Usage, with the browser session already started and its CDP port up:
Then it kills the browser, starts a terminal session on its own desk, and films
cowsay saying what it says. The two films are joined into one mp4.
hhstream session start chromium -- --ozone-platform=wayland ... \\
--remote-debugging-port=9334 --remote-debugging-address=127.0.0.1 about:blank
hhstream record --seconds 80 --out proof.mp4 &
./proof-video.py <chromium-session-id>
Why `--output` and not the published frame socket: with desks, the socket
carries every output stitched side by side, so a viewer would watch two thirds
wallpaper. `--output` runs grim per frame against the session's own desk and
costs more; the trade is a video of the thing being demonstrated.
Only the browser's own HTTP endpoints are used (/json/new, /json/activate) --
no websocket, no shell, no typing. The URLs are real: the search is the one the
gun watch uses (`key=`, singular, which is a bug that once cost every search its
filtering) and the listings are Gunpost pages the watch itself collected.
The cursor is **not** in the frames, and that is not a capture setting: the
virtual pointer has no cursor image for the compositor to draw. See
docs/TODO/cursor-visibility-in-captures.md. The scrolling is visible by what
moves, which is why the page is scrolled rather than merely pointed at.
./proof-video.py [--profile DIR] [--out FILE]
"""
from __future__ import annotations
import argparse
import json
import pathlib
import subprocess
import sys
import time
import urllib.parse
import urllib.request
CDP = "http://127.0.0.1:9334"
CDP_PORT = 9344
CDP = f"http://127.0.0.1:{CDP_PORT}"
SEARCH = "https://www.gunpost.ca/ads?key=SKS&sort=created&order=desc"
LISTINGS = [
"https://www.gunpost.ca/firearms/rifles/hamilton/sks-1951-russian-excellent-rifle-762x39-100000-obo",
@@ -41,6 +47,10 @@ MESSAGE = "hahaha I can see the world now"
#: profile in it: a bare `cowsay` fails inside foot with "No such file or directory".
COWSAY = "/nix/store/s4hlzmy3ll1bphdsy8vjdpcw447vbr93-cowsay-3.8.4/bin/cowsay"
#: How long each film runs. The beats below are timed to fit inside these.
BROWSER_SECONDS = 44
COW_SECONDS = 12
def call(path: str, method: str = "GET") -> dict:
req = urllib.request.Request(CDP + path, method=method)
@@ -50,8 +60,7 @@ def call(path: str, method: str = "GET") -> dict:
def open_tab(url: str) -> str:
tab = call("/json/new?" + urllib.parse.quote(url, safe=""), method="PUT")
return tab["id"]
return call("/json/new?" + urllib.parse.quote(url, safe=""), method="PUT")["id"]
def activate(tab_id: str) -> None:
@@ -63,45 +72,141 @@ def activate(tab_id: str) -> None:
print(f" (activate {tab_id[:8]} said: {exc})", flush=True)
def hhstream(*args: str) -> str:
out = subprocess.run(["hhstream", *args], capture_output=True, text=True, timeout=180)
def hhstream(*args: str, check: bool = False) -> str:
out = subprocess.run(["hhstream", *args], capture_output=True, text=True, timeout=300)
if out.returncode != 0:
print(f" (hhstream {' '.join(args[:2])} -> {out.returncode}: {out.stderr.strip()[:120]})")
print(f" (hhstream {' '.join(args[:3])} -> {out.returncode}: {out.stderr.strip()[:160]})")
if check:
raise SystemExit(f"hhstream {' '.join(args[:3])} failed")
return out.stdout.strip()
def main() -> int:
session = sys.argv[1] if len(sys.argv) > 1 else None
def desk_of(session: str) -> str:
"""The output name of a session's desk, as the broker recorded it."""
print("=== beat 1: the search ===", flush=True)
reply = json.loads(hhstream("session", "container", "--json") or "{}")
rows = reply.get("containers") or reply.get("sessions") or []
for row in rows:
if row.get("id") == session:
desk = row.get("output") or (row.get("container") or {}).get("output")
if desk:
return desk
raise SystemExit(f"{session} has no desk output: {json.dumps(rows)[:300]}")
def busy_work(session: str) -> None:
"""The pointer beats: scroll the results, then walk the listings.
Scrolling is the point of these beats. The pointer is aimed into this
session's own desk, so the scroll lands on its page and nowhere else -- and
because the cursor is not drawn, what a viewer sees is the page moving.
"""
lease = json.loads(hhstream("session", "pointer", "acquire", session, "--json") or "{}")
lease_id = lease.get("lease_id")
if not lease_id:
print(f" (no lease: {json.dumps(lease)[:200]})", flush=True)
return
# Down, down again, then back up: two directions, so it cannot be mistaken
# for a page that simply loaded a different part of itself.
hhstream("session", "pointer", "move", session, "--lease", lease_id, "--to", "0.5,0.55")
for amount, pause in ((6, 2.5), (6, 2.5), (-4, 2.0)):
print(f" scroll vertical {amount:+d}", flush=True)
hhstream(
"session", "pointer", "scroll", session,
"--lease", lease_id, "--axis", "vertical", "--amount", str(amount),
)
time.sleep(pause)
hhstream("session", "pointer", "release", session, "--lease", lease_id)
def film(session: str, seconds: int, out: pathlib.Path) -> None:
"""Record one desk until `seconds` have passed, then wait for the encoder."""
hhstream(
"record", "--output", desk_of(session), "--fps", "12",
"--seconds", str(seconds), "--out", str(out),
)
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--profile", default="/var/lib/hermes/agent-eyes/gunpost-profile")
parser.add_argument("--out", default="/var/lib/hermes/agent-eyes/desks-proof.mp4")
parser.add_argument("--work-dir", default="/var/lib/hermes/agent-eyes/takes")
args = parser.parse_args()
out = pathlib.Path(args.out)
work = pathlib.Path(args.work_dir)
work.mkdir(parents=True, exist_ok=True)
pathlib.Path(args.profile).mkdir(parents=True, exist_ok=True)
a, b = work / "desk-a.mp4", work / "desk-b.mp4"
print("=== starting the browser on its own desk ===", flush=True)
browser = hhstream(
"session", "start", "chromium", "--",
"--ozone-platform=wayland", f"--user-data-dir={args.profile}",
"--no-first-run", "--no-default-browser-check", "--window-size=1280,720",
f"--remote-debugging-port={CDP_PORT}", "--remote-debugging-address=127.0.0.1",
"--new-window", "about:blank",
check=True,
)
print(f" {browser} on {desk_of(browser)}", flush=True)
time.sleep(4) # let the window map and the debugging port come up
print("=== film 1: the search, scrolling, and three listings ===", flush=True)
recording = subprocess.Popen(
["hhstream", "record", "--output", desk_of(browser), "--fps", "12",
"--seconds", str(BROWSER_SECONDS), "--out", str(a)],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
)
time.sleep(2)
search_tab = open_tab(SEARCH)
activate(search_tab)
time.sleep(13) # long enough for Cloudflare's check to clear and results to paint
print("=== beats 2-4: three listings, each in its own tab ===", flush=True)
print(" the SKS search is loading (Cloudflare included)", flush=True)
time.sleep(13)
busy_work(browser)
for n, url in enumerate(LISTINGS, 1):
tab = open_tab(url)
print(f"listing {n}: {url}", flush=True)
activate(tab)
time.sleep(7)
print("=== beat 5: back to the results ===", flush=True)
activate(open_tab(url))
print(f" listing {n} in its own tab", flush=True)
time.sleep(4)
activate(search_tab)
time.sleep(5)
time.sleep(2)
recording.wait(timeout=300)
print(f" {a}: {a.stat().st_size} bytes", flush=True)
print("=== beat 6: clear the browser away ===", flush=True)
if session:
print(f" killed: {hhstream('session', 'kill', session)[:80]}", flush=True)
time.sleep(3)
print("=== clearing the browser away ===", flush=True)
print(f" {hhstream('session', 'kill', browser)[:90]}", flush=True)
time.sleep(2)
print("=== beat 7: a terminal, running cowsay ===", flush=True)
print(
" started: "
+ hhstream("session", "start", "foot", "--", "-H", "-f", "monospace:size=20",
COWSAY, MESSAGE)[:80],
flush=True,
print("=== film 2: a terminal on its own desk, running cowsay ===", flush=True)
terminal = hhstream(
"session", "start", "foot", "--",
"-H", "-f", "monospace:size=20", COWSAY, MESSAGE,
check=True,
)
time.sleep(20)
print(f" {terminal} on {desk_of(terminal)}", flush=True)
time.sleep(2)
film(terminal, COW_SECONDS, b)
print(f" {b}: {b.stat().st_size} bytes", flush=True)
print(f" {hhstream('session', 'kill', terminal)[:90]}", flush=True)
print("=== joining the two films ===", flush=True)
listing = work / "concat.txt"
listing.write_text(f"file '{a}'\nfile '{b}'\n")
joined = subprocess.run(
["ffmpeg", "-v", "error", "-f", "concat", "-safe", "0", "-i", str(listing),
"-c", "copy", "-y", str(out)],
capture_output=True, text=True,
)
if joined.returncode != 0:
raise SystemExit(f"join failed: {joined.stderr.strip()[:300]}")
probe = subprocess.run(
["ffprobe", "-v", "error", "-show_entries", "format=duration,size",
"-of", "default=noprint_wrappers=1", str(out)],
capture_output=True, text=True,
)
print(f" {out}\n {probe.stdout.strip().replace(chr(10), ' ')}", flush=True)
return 0