Files
hyprland-headless-stream-pl…/docs/plans/2026-10-03-keyboard-input.md
T

28 KiB
Raw Blame History

Keyboard input for sessions

Typing stops being a non-goal. docs/PLAN-workspaces-and-pointer.md §1 defers it, and §6.7 already specifies the shape: same seat, same serialisation, zwp_virtual_keyboard_manager_v1, and the existing pointer lease rather than a new lock. Stages 0–3 of that plan are what the tree implements today (broker lease in src/hhstreamd/pointer.py, driver in src/hhstream_pointer/, CLI in src/hhstream_session/__init__.py). This plan is §6.7, with one correction noted under lease semantics.

The Chrome path in examples/agent-eyes-demo.py (Input.insertText) is not this feature. It never touches the seat. This plan is for whatever window is focused on the compositor's one seat.


1. Protocol

Decision: zwp_virtual_keyboard_v1, spoken by the existing hhstream-pointer process. Not wp_virtual_keyboard_v1, not uinput/ydotool, not wtype.

Option Verdict
zwp_virtual_keyboard_v1 Use it. Unstable wayland-protocols interface (virtual-keyboard-unstable-v1.xml). Manager global zwp_virtual_keyboard_manager_v1. Requests are keymap(format, fd, size), key(time, keycode, state), modifiers(depressed, latched, locked, group). Version 1 only.
wp_virtual_keyboard_v1 Do not bind it. Nothing in this repo claims that global. A stable wp_ rename is not what §6.7 measured.
uinput / ydotool Reject. Needs /dev/uinput and a seat that actually reads that device. The headless compositor is driven as a Wayland client today (WaylandSink in src/hhstream_pointer/backend.py); the driver runs as hermes with group hhstream so it can open the Wayland socket (nix/module.nix). A kernel device is a second privilege and a second focus story, and it cannot be unit-tested as a recording sink.
wtype Reject as the mechanism. It is a short-lived client of the same zwp_ protocol: new virtual keyboard per invocation, its own keymap, no release_all across calls, no lease. Useful only as a reference for how text becomes keycodes.

What the running compositor exposes. Not re-checked in this session (no registry dump). The only evidence in-tree is §6.7's [RAN] claim, against the Hyprland it measured (plan header: tag v0.56.2): zwp_virtual_keyboard_manager_v1 is a live global. wp_virtual_keyboard_v1 is not mentioned. Treat that as unverified until Stage 0 dumps the globals on /run/hypr-stream. If the dump shows a different name, stop. Do not probe uinput as a fallback.

Tradeoffs, as they apply here:

  • Keycode, not keysym, on the wire to the compositor. key takes a Linux evdev code (the same value wl_keyboard.key carries; clients add 8 to reach an XKB keycode). Keysym names stay on our socket, the way button names stay on ours and evdev codes cross into zwlr_virtual_pointer_v1 (src/hhstream_pointer/buttons.py).
  • Keymap upload is mandatory. Until keymap is sent (format xkb_v1, memfd of an XKB text keymap), clients cannot decode a key to a character. Upload once, after connect and before listen, matching __main__.py's rule that a listening socket is a promise the device works. Do not upload again per string: a second keymap resets client state under held keys.
  • No unicode request. A codepoint is typeable only when the uploaded layout has a keysym for it. No Ctrl+Shift+u, no per-string custom keymap, no compose sequences in v1.
  • Modifiers are a separate request. The compositor does not derive the modifier mask from key. Shifted text needs modifiers as well as the letter key. Details in §4.
  • No per-output target. Unlike create_virtual_pointer_with_output (backend.py), a virtual keyboard is seat-global. Keys hit whichever surface has keyboard focus. There is no output argument to add.

Missing global is not fatal to the cursor. ManagerTooOld aborts process start because a version-1 pointer aims at the wrong output (backend.py). A missing keyboard global must not do that. Pointer ops keep working. Keyboard ops return {"ok": false, "error": ...} and the broker surfaces that as exit 10 COMPOSITOR. No silent no-op.


2. Surface

Decision: new CLI group session keyboard, new broker verbs keyboard.*, new ops on the existing broker↔driver socket. No new process, unit, binary, or socket.

CONTRACT-sessions.md §5 freezes the binary hhstream-pointer and the socket /run/hhstream/pointer.sock. Renaming the package to "input" is churn with no behavioural gain. server.py already frames newline-delimited JSON at MAX_LINE_BYTES (1 MiB) and is unchanged.

CLI

pointer acquire remains the only acquire. Keyboard verbs are gestures on that lease. They reuse _lease_of, --lease, and $HHSTREAM_LEASE (src/hhstream_session/lease.py). There is no keyboard acquire and no second cache file.

hhstream session pointer acquire ID          # unchanged; also authorises typing
hhstream session pointer release ID          # lifts buttons and keys
hhstream session pointer state               # JSON gains "keys"

hhstream session keyboard type ID TEXT       # one argument, raw UTF-8, no backslash escapes
hhstream session keyboard key  ID NAME       # tap: down then up
hhstream session keyboard down ID NAME
hhstream session keyboard up   ID NAME

NAME is an XKB keysym name (a, A, Return, BackSpace, Left, Control_L, Shift_L), not an evdev number. Help text on both groups must say keyboard commands refuse with exit 9 until pointer acquire. pointer --help stays about the cursor.

type does not interpret \n. A newline is a real newline in the argument. The shell quotes it. The wire value is that string.

Broker control socket

Extend src/hhstreamd/protocol.py (do not renumber ExitCode). Same codes as pointer: 8 BUSY only on a second acquire, 9 NO_LEASE when this session does not hold the lease, 2 USAGE for a bad keysym or an unmappable character, 10 COMPOSITOR when the driver has no keyboard global or the connection dies.

Verb Request beyond verb, request_id Response
keyboard.key id, lease_id, key, state (down/up) keys[]
keyboard.type id, lease_id, text keys[] (explicit holds still down; a type itself leaves none)
pointer.state — existing fields plus keys[]
pointer.release unchanged report lifted keys in the same place buttons are already reported (audit if that is where buttons_lifted lives today — do not invent a richer release body than buttons have)

keyboard.key and keyboard.type join the mutation set beside pointer.click (protocol.py): a replayed request_id returns the recorded answer and must not type twice. A new request_id types again.

Neither verb calls touch. Same rule as every pointer.* verb (CONTRACT-sessions.md §4, module docstring of src/hhstream_session/__init__.py, src/hhstreamd/manager.py).

Cap text at 4096 Unicode scalars in both broker and driver. Over the cap is invalid_argument, not a truncated type.

Audit: keyboard_key records the keysym name and state. keyboard_type records chars (an integer) and not the text. Typed strings will include passwords; the audit log is the wrong place for them.

Driver socket (CONTRACT-sessions.md §3.4)

The driver stays an executor with no lease (ops.py docstring, driver.py docstring). New ops:

{"op":"key","key":"Return","state":"down"}     -> {"ok":true}
{"op":"key","key":"a","state":"up"}            -> {"ok":true}
{"op":"type","text":"Hello"}                   -> {"ok":true}
{"op":"pressed"}                               -> {"ok":true,"buttons":["left"],"keys":["Control_L"]}
{"op":"release_all"}                           -> {"ok":true,"lifted":["left"],"lifted_keys":["Control_L"]}

There is no keyboard frame. zwp_virtual_keyboard_v1 has none. Do not reuse the pointer's end-of-gesture frame; a key flushed only on frame would never arrive. backend.py already flushes every pointer request; keyboard requests flush the same way.

pressed and release_all grow key fields. Do not add a parallel release_all_keys op. Expiry has one call site (SessionManager._expire_pointer).

Unknown keysym, non-finite or over-long text, and type while a non-modifier key is held are Rejected (answered, connection kept), same as a bad button.

File-by-file

File Change
src/hhstream_pointer/_protocol/virtual_keyboard_unstable_v1/ Vendor pywayland output, same method as _protocol/regenerate.py (seed the import map with core wl_*, do not use python -m pywayland.scanner alone). XML is wayland-protocols unstable/virtual-keyboard/virtual-keyboard-unstable-v1.xml, not the wlr pointer XML.
src/hhstream_pointer/_protocol/regenerate.py Second XML argument. State the path in the module docstring the way the pointer generator does.
src/hhstream_pointer/keymap.py New. xkbcommon only here. Builds one keymap from rules (evdev, model, layout, variant, options). Resolves a keysym name or a string to a list of actions (evdev_code, mods_depressed) or raises. Resolves the whole string before returning, so a bad character emits nothing.
src/hhstream_pointer/ops.py Extend Dispatcher. A small Keymap protocol (same idea as PointerSink) does the resolving, so this file still imports neither pywayland nor xkbcommon. New ops key and type. _pressed stays buttons. New insertion-ordered _keys. release_all lifts keys after buttons: key-up each held key in press order, then modifiers(0, 0, 0, group).
src/hhstream_pointer/backend.py Still the only pywayland importer. Bind the manager at version 1, create_virtual_keyboard(seat) once (not per output). keymap upload from a memfd. Sink methods key(code, state) and modifiers(dep, latch, lock, group) only — no names, no UTF-8.
src/hhstream_pointer/__main__.py Flags below. Build the keymap and upload it before listen. _lift_everything already calls release_all; once that op lifts keys, shutdown covers them. Update the comment that today only mentions buttons.
src/hhstream_pointer/server.py No change.
src/hhstreamd/driver.py key(), type_text() on PointerDriver. release_all / pressed read the new fields. Same socket, same "do not retry a request that was written" rule. A failed release_all is still DriverUnavailable: a dropped lift leaves a key down.
src/hhstreamd/pointer.py Not a new class. On _Held, a keys dict beside buttons. press_key / unpress_key / keys_pressed, mirroring press / unpress / pressed. expire() still returns LeaseInfo only — no keys. Callers read keys_pressed() first, as they must read pressed() first (CONTRACT-sessions.md §3.3).
src/hhstreamd/manager.py keyboard_key and keyboard_type go through _refresh (live session, holder, not expired, container output required). Driver first, then the lease mirror — the pointer_button order, so a refused key is not recorded as down. _expire_pointer reads keys before expire() and uses the one release_all. Same on pointer_release, kill, reap, and driver loss.
src/hhstreamd/protocol.py Verb allowlist plus the mutation set.
src/hhstreamd/policy.py [keyboard] next to [pointer].
src/hhstream_session/__init__.py _build_keyboard beside _build_pointer.
src/hhstream_session/format.py render_pointer_state shows held keys.
src/hhstream_session/lease.py No new store. pointer-leases.json remains the cache.
nix/module.nix No new unit. Extend hhstream-pointer.service argv. Emit [keyboard] into the generated TOML the way [pointer] is emitted. Options under sessions.keyboard: layout (default "us"), variant (default ""), options (default ""), model (default "pc105").
nix/package.nix, flake.nix devshell, pyproject.toml xkbcommon for the driver and for unit tests. No new console script.
nix/hyprland-conf.nix Keep input:follow_mouse = 1. Extend the comment: keyboard events have no output, so this warp is what moves keyboard focus, and the lease is what stops another session stealing it mid-string.
docs/CONTRACT-sessions.md Amend §3.3 (key mirror, expire() still carries no keys), §3.4 (ops above), §4 (verbs, idle rule).
README.dev.md item 3, checks/vm-pointer.nix header Stale relative to the conf file as read: they still say hyprland-conf.nix ships follow_mouse = 0 and that the keyboard task will choose. The conf file sets 1. Fix the prose when implementing; do not set the conf back to 0.

Flags the unit passes, defaults matching §5's style of frozen defaults:

hhstream-pointer --socket /run/hhstream/pointer.sock \
                 --compositor-runtime-dir /run/hypr-stream \
                 --xkb-layout us --xkb-model pc105 \
                 --xkb-variant '' --xkb-options ''

A layout xkbcommon rejects, before any Wayland connect, exits 2. Compositor failures stay 10.


3. Lease

Decision: one lease. Keyboard verbs require the pointer lease. Do not add a keyboard lease, a keyboard boolean, or a second sweeper.

§6.7 says the lease "gains a keyboard flag" and that no new mechanism is needed. The flag is the wrong reading of the code that shipped. _Held.buttons exists because release_all must know which buttons went down and in what order (src/hhstreamd/pointer.py, ops.py). A boolean cannot lift Control_L and not a. The key set is that pattern, not a new state machine.

Why sharing is required by the seat, not just by the plan: there is one wl_seat and one keyboard focus. follow_mouse = 1 (nix/hyprland-conf.nix) means another session's warp retargets the keys. Two leases would let session B move the pointer while session A believes it is typing into its own window. LeaseBusy on acquire is already the mutex (PointerLease.acquire refuses even the current holder a second id, so a drag — or a half-typed chord — cannot have its id silently replaced).

Rules that keyboard inherits unchanged:

  • Acquire / refresh / hard cap (ttl 30 s, max 300 s, refresh moves only the idle deadline).
  • expire() does not lift anything. The broker reads the mirror, drops the lease, then release_all on the driver (_expire_pointer).
  • Driver death force-releases (driver_gone). The driver is not asked to remember who held the lease.
  • Kill and reap of the holder drop the lease with the existing reasons (kill / crash).
  • CLI cache is re-checked by the broker; a stale id is exit 9 and then forgotten.
  • Gesture verbs do not mark activity.

TTL expiry with keys held down. Same as buttons (PLAN §6.2 "Never leave a button down", implemented in _expire_pointer and in _lift_everything):

  1. Sweeper (or the next verb's _refresh) sees the deadline.
  2. Broker reads keys_pressed() and pressed() before expire().
  3. expire() drops the lease. It is not kept open to finish the word.
  4. Driver release_all: button-ups, then key-ups in press order, then modifiers(0, 0, 0, group), then the pointer frames it already sends for buttons. No keyboard frame.
  5. Audit pointer_lease_timeout gains keys_lifted. The session's pointer bookmark is stamped released_by: timeout as today.
  6. The next acquire is free. Nothing is replayed. A held Shift does not leak into the next holder.

If release_all fails, do what the button path does: log it, still drop the lease, do not retry a request whose outcome is unknown (driver.py). The stuck-key case is a risk in §6, not a second expiry protocol.

type refreshes once per broker verb, inside _refresh, not once per character. Explicit down of a modifier stays in the lease mirror across later type calls. type itself must not leave letters down.

A warp while a key is held stays legal, same as a drag. Focus can follow that warp. That is another reason both devices share the lease: the holder is the only session allowed to move focus until release or TTL.


4. Keymap and text

Decision: one seat-wide XKB layout, chosen in config, default us / pc105 / empty variant / empty options. The virtual keyboard's keymap is the one we upload. Hyprland's input:kb_layout is not read and does not affect injected keys.

Do not query hyprctl getoption to fill a default. The broker's rule is that unverified fields stay null (PLAN §7); a layout guessed from the compositor would drift on reload. Operators who are not on us set sessions.keyboard.layout.

Decision: v1 is a single group. A comma-separated layout (us,ru) is rejected at config load. Group in every modifiers request is 0. No group-switch keysym.

Where it lives. [keyboard] in the broker TOML (policy.py), rendered by nix/module.nix, passed as the flags in §2. The driver compiles it at startup and uploads that byte string. The session record does not gain a layout field. Per-session layouts would mean re-uploading keymap on every acquire, which drops held state; not in v1.

How text is expressed.

  • keyboard type takes a Python str of Unicode scalars. The keymap maps each scalar to one keysym, then to one (evdev code, modifier mask).
  • Newline → keysym Return. Tab → Tab. Other C0 controls are rejected with the rest of the string, not skipped.
  • A is Shift+a via the mask, not via a typed Shift key, unless the user also has Shift explicitly held.
  • Explicit modifier holds are a baseline. type ORs each character's mask with that baseline, sends the letter, and restores the mask to the baseline afterwards. So down Control_L then type c is Ctrl+C, and Control is still down when type returns.
  • If any non-modifier key is already held, type is rejected and emits nothing.
  • Validation is complete before the first sink call. Unmappable character → the whole request fails, no prefix typed. This includes é on us.
  • Dead keys (dead_acute and the like) are a rejection even if the layout names them. v1 does not emit a two-event compose. us(intl) is therefore a poor choice until a later stage; default us does not need it.
  • No auto-repeat timer in our process. Whether Hyprland repeats a virtual key that we leave down is unverified (Stage 0).

Modifier decision: for a shifted character, send modifiers(shift_mask, 0, 0, 0) and the letter's key down/up, then restore the previous mask. Do not also synthesize Shift_L down/up for that character. For an explicit keyboard down Shift_L, send the Shift key and a matching modifiers update, and remember both in the held-key record so release_all undoes both. Sending both a synthetic Shift key and a mask for an ordinary capital is how toolkits apply Shift twice or not at all; the VM check in §5 is what proves the letter path against a real wl_keyboard client.

Named non-character keysyms used by key/down/up go through the same resolver, not a second table that can disagree with the layout. buttons.py can stay a hand-written evdev table because the protocol takes those codes directly; keysyms cannot.


5. Tests

Match the split already in the tree: the dispatcher is tested against a recording sink and is not tested for authority (tests/pointer/test_ops.py); the lease is pure with FakeClock and no sleep (tests/pointer/conftest.py); the broker is tested with a real UNIX socket and FakeDriver, Wayland substituted (tests/sessions/unit/test_pointer_verbs.py). No integration test grows a fake compositor. tests/sessions/integration/ does not gain Wayland.

Unit

Test Asserts
tests/pointer/test_keymap.py Real xkbcommon, layout us: a → evdev 30, mask 0; A → 30 plus Shift; Return and \n → KEY_ENTER; empty string → []; é and a C0 control other than \n/\t raise; the raise happens with zero sink calls when driven through the dispatcher. Dead keysym raises.
tests/pointer/test_ops.py Extend RecordingSink with key(code, state) and modifiers(...). key down then pressed lists the name. up of a key that was not down reports it not lifted (same as unpress). release_all lifts keys in press order and ends with a zeroed modifiers, even if the last warp was to another output (keys are not per output). type emits down/up pairs and leaves keys empty. type with Control_L held ORs the mask and leaves Control down. type with a held emits nothing and returns ok: false. A keymap that raises on the third character emits nothing.
tests/pointer/test_lease.py press_key / keys_pressed order. expire() still returns no key list; the test reads keys_pressed() first, then expire(), then sees an empty lease. Existing busy/stale/lease_id cases stay as they are.
tests/pointer/test_vendored_protocol.py Interface name zwp_virtual_keyboard_manager_v1 and requests keymap, key, modifiers. No display.
tests/sessions/unit/test_keyboard_verbs.py New, harness copied from test_pointer_verbs.py. No lease → 9. Other session holds the lease → 9 (not 8). Second pointer.acquire while typing → 8. type refreshes TTL: advance the ManualClock by less than ttl after a type and the lease survives; advance past it and release_all runs with the held key. Driver rejection does not stick the key in the lease. type does not move last_activity_boottime. pointer.release and kill both lift keys. pointer.state includes keys.
tests/sessions/unit/test_settings.py [keyboard] defaults to us; a non-table is PolicyError; us,ru is rejected.
CLI surface test (tests/sessions/unit/test_cli_surface.py) and broker verb list (test_broker_protocol.py) keyboard in help; help says the pointer lease is required; keyboard.type / keyboard.key are known verbs and are in the mutation set.

nix develop -c python -m pytest tests/pointer tests/sessions/unit -q is the gate. Not run for this plan.

Integration / VM

Decision: a new checks/vm-keyboard.nix, same shape as checks/vm-pointer.nix. Do not call the feature done on unit tests alone. docs/TODO/scroll-has-no-compositor-proof.md is the precedent: scroll shipped on the driver socket with no compositor proof. Keyboard's only purpose is that the client receives symbols.

  • Victim: a small Wayland client beside checks/lib/pointer_target.py (or an extension of it) that sets xdg_toplevel app id from $HHSTREAM_APP_ID and prints wl_keyboard.enter, each key (evdev code, state), each modifiers mask, and the keysym it decodes from the keymap the compositor forwarded. Read-back is hhstream session logs, as in vm-pointer.
  • Assertions, real time, no injected clock (the unit clock already covers the arithmetic; vm-pointer's expiry subtest states why):
    1. After pointer acquire on session A, A's log shows wl_keyboard.enter. This is the focus measurement. If it fails, the check is red and Stage 4 is in play — do not weaken the assertion.
    2. keyboard type A hello decodes to the characters hello on A and to nothing on session B.
    3. keyboard down A Shift_L, wait out a short --ttl, A's log shows the key-up and a cleared mask, and pointer state shows no holder.
    4. A second acquire during the hold exits 8.
  • Mutation, per PLAN §14: the character assertion must fail if keymap was never uploaded (keycodes without a decoded hello are a failure, not a pass).
  • globalTimeout on the order of vm-pointer (1800). This check is slow; that is not a reason to fold it into a unit test.

6. Risks, open questions, stages

Risks

  • Focus may not follow a virtual pointer. The conf comment says follow_mouse = 1 was set so pointer motion moves keyboard focus, after vm-pointer measured that follow_mouse = 0 delivers clicks to nobody when the aimed window is unfocused. That measurement was pointer delivery, not wl_keyboard.enter. Unverified for a zwlr_virtual_pointer_v1 warp. If Stage 0 shows no enter, the fallback is one existing IPC call — dispatch focuswindow on the session's window.stable_id — immediately before the first key of a lease. Do not build that fallback in Stages 1–2.
  • Hard death of the virtual keyboard can leave a key down inside Hyprland if destroy does not release keys. Unverified on this build. Our shutdown path sends key-ups first (_lift_everything). A killed -9 driver skips it. Do not add a retry loop the button path does not have.
  • Auto-repeat. If Hyprland repeats a virtual key we leave down, keyboard down a floods the window until TTL (default 30 s). Stage 0 records whether it repeats. If it does, the help text says so; v1 still has no repeat-rate control. Chords (Control_L held across a type) are the justified use of down.
  • Mask versus Shift keycode. §4's letter path can still be wrong for a toolkit that ignores modifiers and only shifts when it sees KEY_LEFTSHIFT. The VM victim must print both the mask and the decoded sym. Chromium is not the first victim (vm-pointer avoided it; it is expensive and the app-id rule in README.dev.md item 4 only binds clients that honour $HHSTREAM_APP_ID).
  • The lease is not a security boundary. PLAN §6.6: any client may bind the pointer global. The keyboard global is the same. Stage 4 sandboxing is unchanged and out of this work.
  • Concurrent typing is impossible on one compositor, for the same reason concurrent pointers are (PLAN §1). A second seat is Stage 5 and still CPU-gated. This work does not pretend otherwise.
  • xkbcommon in the devshell is a new dependency. Tests that import keymap.py cannot skip when it is absent; an optional dependency would let the unmappable-character path rot.

Open questions

  1. Stage 0 global name and version on the actual /run/hypr-stream socket. §6.7's [RAN] is the claim; this session did not repeat it.
  2. Does acquire's warp produce wl_keyboard.enter on the aimed window under the shipped follow_mouse = 1?
  3. On virtual-keyboard resource destroy, does Hyprland release keys that were down?
  4. Does Hyprland auto-repeat those keys?
  5. Operator: is default layout us acceptable for this machine, or must the NixOS option be set to something else before the VM check is written?
  6. A later stage, only if agents must type arbitrary UTF-8: a throwaway keymap that maps spare keycodes to 0x01000000 + codepoint for the duration of one type, then restores the layout keymap. Not v1. It reintroduces the "upload resets held modifiers" problem, so it has to refuse to run while any key is held.

Stages

Stage Work Rough size Gate
0 Read-only. Dump globals. One tiny client, or a few minutes in the existing VM, to record enter, repeat, and destroy-while-held. Write the results down next to the check. No broker changes. S, about half a day Named global is zwp_virtual_keyboard_manager_v1. Anything else stops the project.
1 Driver: vendor, keymap.py, ops, backend upload, __main__ flags, pointer unit tests. Pointer still runs if the global is missing. M, about 3–5 days pytest tests/pointer green. type of an unmappable character emits no events.
2 Broker mirror, verbs, CLI, policy, module argv, session unit tests, contract doc amendments, stale follow_mouse comments. M, about 2–4 days pytest tests/pointer tests/sessions/unit green. Expiry test shows release_all lifting a held key. No new unit file under nix/module.nix beyond argv and TOML.
3 vm-keyboard.nix and the keyboard target client. M, about 2–3 days Characters on A, silence on B, key-up after a real TTL, second acquire is 8. Red if the keymap was not uploaded.
4 Only if Stage 0 or Stage 3 shows no wl_keyboard.enter after warp. focuswindow via the broker's existing compositor IPC, then re-run Stage 3. S, about a day, contingent The Stage 3 enter assertion goes green without loosening it.

Rollback of Stages 1–2 is reverting the commit: old brokers never send key/type, and an old driver rejects unknown ops with ok: false (ops.py already answers unknown op instead of dropping the connection). No registry schema bump. Bookmarks stay as they are.