Files
hyprland-headless-stream-pl…/docs/plans/2026-10-03-keyboard-input.md
T
HermenixandClaude Opus 5 b1f2744cf0 Stage 0 review: correct what the log does not say, and edit the plan
The Stage 0 review (docs/reviews/2026-10-04-stage0-review.md, verdict
yes-with-fixes) found that the measurements were sound and the write-up
was not. All four of its high and medium defects, verified against the
committed transcript before fixing:

(1) 19e484b's README claimed a verbatim excerpt and printed serial=336
    while the run's (a2) enter was serial 371. It also claimed six
    reproductions with one transcript, and cited a `keys=[42, 0, 0, 0]`
    enter from a run whose log was never kept -- every enter in the
    committed file is `keys=[]`. The excerpt is now copied from the log
    with line numbers to check it against, the reproduction claim is
    gone, and the Shift observation is named as not-in-this-log.

(2) THE (a) CONCLUSION WAS WRONG, and it is the one that matters. The
    first idle warp onto the unfocused window gave wl_pointer.enter and
    NO keyboard enter; the leave/enter arrived on the LATER motion. So
    it is not "enter, one warp late" -- the first warp's delta is set to
    0, the threshold is 0.0, and the second motion's own non-zero delta
    is what focused. Two corollaries the old text got backwards: a
    second warp to the SAME point returns before mouseMoveUnified
    touches the delta and proves nothing, and waiting longer does not
    help. The probe now prints both targets and the measured gap
    (0.401s) instead of hardcoding the word "immediately", and its
    ANSWER string says the mechanism.

    Consequence, now stated everywhere it belongs: `pointer acquire` is
    one warp, so the first key of a lease can land on whatever was
    already focused. Stage 3's enter subtest must not be designed from
    acquire's single warp, and must not be made to pass by warping twice
    to the same bookmark.

(3) d07a1fd did not edit the plan -- `git diff 14b3f69 HEAD --
    docs/plans/2026-10-03-keyboard-input.md` was empty, so anyone told
    to follow the plan still had the contradicted text. All eight
    pre-code changes are now folded into that file, each marked
    [SUPERSEDED], plus a banner at the top. Changes 5 and 7 were the
    ones that had survived intact in four places: the [keyboard] TOML
    table (dropped -- the broker reads that file and the driver is a
    different unit, so it would have been a setting with no effect) and
    "if it repeats, the help text says so" (now unconditional). The four
    closed open questions and the stage table are updated with what
    Stage 0 measured, including that Stage 4 is no longer contingent.

(4) mask_cleared=no was reported as a held mask surviving destroy. The
    held key was a letter, so the mask was 0 throughout; the README now
    says destroy emitted no key-up and no modifiers event, and that the
    reason release_all still sends modifiers(0,0,0,0) is
    VirtualKeyboard.cpp's releasePressed, not this line. The real
    poison evidence is promoted: onKeyboardKey drops a release whose
    code is not in m_pressed, so the cleanup's `key code=30 state=0`
    arriving from a brand-new keyboard proves 30 was still in that set.

The transcript is a fresh run of the corrected probe, so the file and
the claims about it agree line for line.

19e484b's message carries defect (2)'s wrong causality. It is corrected
here rather than by rewriting that commit, so the hashes the reviewer
cites keep resolving.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GhNPM6rGcuzxocM1nATnuM
2026-10-04 03:46:53 +00:00

326 lines
39 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Keyboard input for sessions
> **Superseded in eight places. Read this first.**
>
> `docs/plans/2026-10-04-keyboard-verification.md` is an adversarial review of
> this file against the tree and against Hyprland v0.56.2. Its verdict is "safe
> to build on, not safe to implement as written", and the eight rows of its
> **"Changes required before code"** table **override this document wherever
> they differ**. Each one is also folded into the text below, marked
> `[SUPERSEDED]`, so nobody following a single paragraph gets the contradicted
> version:
>
> | # | What changed |
> | --- | --- |
> | 1 | Exit 2 vs `driver_refused` — §2's exit-code paragraph was wrong about what `_drive` does |
> | 2 | The keymap `size` includes the trailing NUL; `format = 1`; `time` is `_now_ms()` |
> | 3 | `release_all` no-ops its key half with no keyboard, and ends with `modifiers(0,0,0,0)` |
> | 4 | `input:virtualkeyboard:release_pressed_on_close = true` — destroy-while-held is no longer an open question |
> | 5 | The layout reaches the keymap **only** as `--xkb-layout` on the driver unit; no `[keyboard]` TOML table |
> | 6 | The modifier-keysym set is measured from the keymap; no Shift special case |
> | 7 | Stage 0's expectations, and `keyboard down` warns unconditionally |
> | 8 | Acquire's stderr mentions keyboard verbs; refusals carry a scalar index, never the string |
>
> **Stage 0 has run** (`tests/README-keyboard-probe.md`, with its transcript).
> The three open behavioural questions are answered, and one answer is worse
> than this document assumes: **a single idle warp does not move keyboard
> focus**, so `pointer acquire` — which is one warp — can leave the first key
> of a lease going to whatever was already focused. §6 risk 1 still holds and
> the fallback is still not built in Stages 1–2.
Typing stops being a non-goal. `docs/PLAN-workspaces-and-pointer.md` §1 defers it, and §6.7 already specifies the shape: same seat, same serialisation, `zwp_virtual_keyboard_manager_v1`, and the existing pointer lease rather than a new lock. Stages 0–3 of that plan are what the tree implements today (broker lease in `src/hhstreamd/pointer.py`, driver in `src/hhstream_pointer/`, CLI in `src/hhstream_session/__init__.py`). This plan is §6.7, with one correction noted under lease semantics.
The Chrome path in `examples/agent-eyes-demo.py` (`Input.insertText`) is not this feature. It never touches the seat. This plan is for whatever window is focused on the compositor's one seat.
---
## 1. Protocol
**Decision: `zwp_virtual_keyboard_v1`, spoken by the existing `hhstream-pointer` process. Not `wp_virtual_keyboard_v1`, not uinput/`ydotool`, not `wtype`.**
| Option | Verdict |
| --- | --- |
| `zwp_virtual_keyboard_v1` | Use it. Unstable wayland-protocols interface (`virtual-keyboard-unstable-v1.xml`). Manager global `zwp_virtual_keyboard_manager_v1`. Requests are `keymap(format, fd, size)`, `key(time, keycode, state)`, `modifiers(depressed, latched, locked, group)`. Version 1 only. |
| `wp_virtual_keyboard_v1` | Do not bind it. Nothing in this repo claims that global. A stable `wp_` rename is not what §6.7 measured. |
| uinput / `ydotool` | Reject. Needs `/dev/uinput` and a seat that actually reads that device. The headless compositor is driven as a Wayland client today (`WaylandSink` in `src/hhstream_pointer/backend.py`); the driver runs as `hermes` with group `hhstream` so it can open the Wayland socket (`nix/module.nix`). A kernel device is a second privilege and a second focus story, and it cannot be unit-tested as a recording sink. |
| `wtype` | Reject as the mechanism. It is a short-lived client of the same `zwp_` protocol: new virtual keyboard per invocation, its own keymap, no `release_all` across calls, no lease. Useful only as a reference for how text becomes keycodes. |
**What the running compositor exposes.** Not re-checked in this session (no registry dump). The only evidence in-tree is §6.7's **[RAN]** claim, against the Hyprland it measured (plan header: tag `v0.56.2`): `zwp_virtual_keyboard_manager_v1` is a live global. `wp_virtual_keyboard_v1` is not mentioned. Treat that as unverified until Stage 0 dumps the globals on `/run/hypr-stream`. If the dump shows a different name, stop. Do not probe uinput as a fallback.
**Tradeoffs, as they apply here:**
- **Keycode, not keysym, on the wire to the compositor.** `key` takes a Linux evdev code (the same value `wl_keyboard.key` carries; clients add 8 to reach an XKB keycode). Keysym names stay on our socket, the way button names stay on ours and evdev codes cross into `zwlr_virtual_pointer_v1` (`src/hhstream_pointer/buttons.py`).
- **Keymap upload is mandatory.** Until `keymap` is sent (format `xkb_v1`, memfd of an XKB text keymap), clients cannot decode a key to a character. Upload **once**, after connect and before `listen`, matching `__main__.py`'s rule that a listening socket is a promise the device works. Do not upload again per string: a second `keymap` resets client state under held keys.
**[SUPERSEDED — change 2.]** Three values this bullet left unstated, each of which fails silently:
- the `size` argument **includes the trailing NUL**. Hyprland `mmap`s exactly `size` bytes and hands the pointer to `xkb_keymap_new_from_string`, which wants a C string, so `size = len(text)` reads off the end of the mapping. The symptom is a keymap that will not compile, which from our side is indistinguishable from a compositor with no keyboard at all;
- the fd must outlive a **round trip**, not a flush: the compositor has to have received and mapped it before we close our end;
- `format` is the integer **1** (`XKB_KEYMAP_FORMAT_TEXT_V1`). Hyprland ignores it and parses as `TEXT_V2`, which accepts a V1 string; do not send 2;
- `time` on every `key` is `backend._now_ms()`, the same monotonic source the pointer requests use. The protocol requires one clock per object.
Also, a *second* upload is worse than "resets client state": Hyprland replaces its XKB state without walking the keys it believes are down, so re-uploading under a held modifier strands it.
- **No unicode request.** A codepoint is typeable only when the uploaded layout has a keysym for it. No `Ctrl+Shift+u`, no per-string custom keymap, no compose sequences in v1.
- **Modifiers are a separate request.** The compositor does not derive the modifier mask from `key`. Shifted text needs `modifiers` as well as the letter key. Details in §4.
- **No per-output target.** Unlike `create_virtual_pointer_with_output` (`backend.py`), a virtual keyboard is seat-global. Keys hit whichever surface has keyboard focus. There is no output argument to add.
**Missing global is not fatal to the cursor.** `ManagerTooOld` aborts process start because a version-1 pointer aims at the wrong output (`backend.py`). A missing keyboard global must not do that. Pointer ops keep working. Keyboard ops return `{"ok": false, "error": ...}` and the broker surfaces that as exit **10 COMPOSITOR**. No silent no-op.
---
## 2. Surface
**Decision: new CLI group `session keyboard`, new broker verbs `keyboard.*`, new ops on the existing broker↔driver socket. No new process, unit, binary, or socket.**
`CONTRACT-sessions.md` §5 freezes the binary `hhstream-pointer` and the socket `/run/hhstream/pointer.sock`. Renaming the package to "input" is churn with no behavioural gain. `server.py` already frames newline-delimited JSON at `MAX_LINE_BYTES` (1 MiB) and is unchanged.
### CLI
`pointer acquire` remains the only acquire. Keyboard verbs are gestures on that lease. They reuse `_lease_of`, `--lease`, and `$HHSTREAM_LEASE` (`src/hhstream_session/lease.py`). There is no `keyboard acquire` and no second cache file.
```
hhstream session pointer acquire ID # unchanged; also authorises typing
hhstream session pointer release ID # lifts buttons and keys
hhstream session pointer state # JSON gains "keys"
hhstream session keyboard type ID TEXT # one argument, raw UTF-8, no backslash escapes
hhstream session keyboard key ID NAME # tap: down then up
hhstream session keyboard down ID NAME
hhstream session keyboard up ID NAME
```
`NAME` is an XKB keysym name (`a`, `A`, `Return`, `BackSpace`, `Left`, `Control_L`, `Shift_L`), not an evdev number. Help text on both groups must say keyboard commands refuse with exit 9 until `pointer acquire`. `pointer --help` stays about the cursor.
`type` does not interpret `\n`. A newline is a real newline in the argument. The shell quotes it. The wire value is that string.
### Broker control socket
Extend `src/hhstreamd/protocol.py` (do not renumber `ExitCode`). Same codes as pointer: **8 BUSY** only on a second acquire, **9 NO_LEASE** when this session does not hold the lease, **2 USAGE** for a bad keysym or an unmappable character, **10 COMPOSITOR** when the driver has no keyboard global or the connection dies.
**[SUPERSEDED — change 1.]** "Exit 2 for a bad keysym" and "`Rejected`, same as a bad button" are not the same code, and this paragraph asserted both. `SessionManager._drive` turns *every* driver `ok: false` into `ManagerError("driver_refused")`, which is exit **10** — that is why `test_an_unknown_button_is_refused_by_the_driver_not_invented` asserts `driver_refused` and not usage. Exit 2 happens only for an `invalid_argument` the broker raises *before* `_drive`. So:
- the **broker** checks shape before it calls the driver: `text` must be a string, `state` must be `down`/`up`, `key` must be a non-empty string, and `text` must be within the 4096-scalar cap. Those are `invalid_argument`, exit 2;
- the **driver** tags a rejection it caused the caller to earn with a stable token, `kind: "invalid_argument"` (`hhstream_pointer.ops.KIND_INVALID`) — a bad keysym, an unmappable character, a dead key, a `type` under a held non-modifier. `_drive` maps that token to `invalid_argument`, exit 2;
- everything else keeps `kind: "refused"` and stays `driver_refused`, exit **10**: no keyboard global, no aimed output, a dead connection.
The token exists because the alternative is the broker telling "no such keysym on your layout" from "this compositor has no keyboard" by matching on English.
| Verb | Request beyond `verb`, `request_id` | Response |
| --- | --- | --- |
| `keyboard.key` | `id`, `lease_id`, `key`, `state` (`down`/`up`) | `keys[]` |
| `keyboard.type` | `id`, `lease_id`, `text` | `keys[]` (explicit holds still down; a `type` itself leaves none) |
| `pointer.state` | — | existing fields plus `keys[]` |
| `pointer.release` | unchanged | report lifted keys in the same place buttons are already reported (audit if that is where `buttons_lifted` lives today — do not invent a richer release body than buttons have) |
`keyboard.key` and `keyboard.type` join the mutation set beside `pointer.click` (`protocol.py`): a replayed `request_id` returns the recorded answer and must not type twice. A new `request_id` types again.
Neither verb calls `touch`. Same rule as every `pointer.*` verb (`CONTRACT-sessions.md` §4, module docstring of `src/hhstream_session/__init__.py`, `src/hhstreamd/manager.py`).
Cap `text` at **4096** Unicode scalars in both broker and driver. Over the cap is `invalid_argument`, not a truncated type.
**Audit:** `keyboard_key` records the keysym name and state. `keyboard_type` records `chars` (an integer) and **not the text**. Typed strings will include passwords; the audit log is the wrong place for them.
**[SUPERSEDED — change 8.]** `events.jsonl` is not the only place a string leaks, and the plan secured only that one. Two more:
- **Refusal messages and driver logs carry the scalar *index*, never the character and never the string.** "unmappable at scalar 3" is the whole of what a reader needs, and the alternative ends up in the journal. `AuditLog.emit` will happily write a `text` field if a caller passes one, so the rule is about every message, not only the audit event.
- **`pointer acquire`'s stderr is wrong once this lands.** It says "expires in … with no further **pointer** verb". A `keyboard.*` verb goes through `_refresh` and *does* move `expires_at`, so the sentence has to name both — otherwise an operator releases during a `type` that would have kept the lease, and the TTL test looks flaky against the help text.
(What the shell and `ps` show of the CLI argument is outside this rule: the text is an argv word by the time anything here sees it.)
### Driver socket (`CONTRACT-sessions.md` §3.4)
The driver stays an executor with no lease (`ops.py` docstring, `driver.py` docstring). New ops:
```json
{"op":"key","key":"Return","state":"down"} -> {"ok":true}
{"op":"key","key":"a","state":"up"} -> {"ok":true}
{"op":"type","text":"Hello"} -> {"ok":true}
{"op":"pressed"} -> {"ok":true,"buttons":["left"],"keys":["Control_L"]}
{"op":"release_all"} -> {"ok":true,"lifted":["left"],"lifted_keys":["Control_L"]}
```
There is no keyboard `frame`. `zwp_virtual_keyboard_v1` has none. Do not reuse the pointer's end-of-gesture `frame`; a key flushed only on `frame` would never arrive. `backend.py` already flushes every pointer request; keyboard requests flush the same way.
`pressed` and `release_all` grow key fields. Do not add a parallel `release_all_keys` op. Expiry has one call site (`SessionManager._expire_pointer`).
**[SUPERSEDED — change 3.]** Two things follow from there being exactly one call site, and this plan stated neither:
- `release_all` must **no-op its key half when no virtual keyboard exists**. If the key half raised, the one `release_all` would abandon the buttons it had just been asked to lift — and a lease that expired with a mouse button down is the failure the whole TTL exists to prevent.
- `release_all` ends with `modifiers(0, 0, 0, 0)` **whenever a mask was set**, because a virtual key-up clears no modifier state on this compositor. Without it a lease that ended under Shift hands the next holder a keyboard where every letter is a capital. One `modifiers(0, 0, 0, 0)` also goes out immediately after the initial keymap upload, so a previous driver's mask is not inherited.
Unknown keysym, non-finite or over-long text, and `type` while a non-modifier key is held are `Rejected` (answered, connection kept), same as a bad button.
### File-by-file
| File | Change |
| --- | --- |
| `src/hhstream_pointer/_protocol/virtual_keyboard_unstable_v1/` | Vendor pywayland output, same method as `_protocol/regenerate.py` (seed the import map with core `wl_*`, do not use `python -m pywayland.scanner` alone). XML is wayland-protocols `unstable/virtual-keyboard/virtual-keyboard-unstable-v1.xml`, not the wlr pointer XML. |
| `src/hhstream_pointer/_protocol/regenerate.py` | Second XML argument. State the path in the module docstring the way the pointer generator does. |
| `src/hhstream_pointer/keymap.py` | **New.** xkbcommon only here. Builds one keymap from rules (`evdev`, model, layout, variant, options). Resolves a keysym name or a string to a list of actions `(evdev_code, mods_depressed)` or raises. Resolves the **whole** string before returning, so a bad character emits nothing. |
| `src/hhstream_pointer/ops.py` | Extend `Dispatcher`. A small `Keymap` protocol (same idea as `PointerSink`) does the resolving, so this file still imports neither pywayland nor xkbcommon. New ops `key` and `type`. `_pressed` stays buttons. New insertion-ordered `_keys`. `release_all` lifts keys after buttons: key-up each held key in press order, then `modifiers(0, 0, 0, group)`. |
| `src/hhstream_pointer/backend.py` | Still the only pywayland importer. Bind the manager at version 1, `create_virtual_keyboard(seat)` **once** (not per output). `keymap` upload from a memfd. Sink methods `key(code, state)` and `modifiers(dep, latch, lock, group)` only — no names, no UTF-8. |
| `src/hhstream_pointer/__main__.py` | Flags below. Build the keymap and upload it before `listen`. `_lift_everything` already calls `release_all`; once that op lifts keys, shutdown covers them. Update the comment that today only mentions buttons. |
| `src/hhstream_pointer/server.py` | No change. |
| `src/hhstreamd/driver.py` | `key()`, `type_text()` on `PointerDriver`. `release_all` / `pressed` read the new fields. Same socket, same "do not retry a request that was written" rule. A failed `release_all` is still `DriverUnavailable`: a dropped lift leaves a key down. |
| `src/hhstreamd/pointer.py` | Not a new class. On `_Held`, a `keys` dict beside `buttons`. `press_key` / `unpress_key` / `keys_pressed`, mirroring `press` / `unpress` / `pressed`. `expire()` still returns `LeaseInfo` only — no keys. Callers read `keys_pressed()` first, as they must read `pressed()` first (`CONTRACT-sessions.md` §3.3). |
| `src/hhstreamd/manager.py` | `keyboard_key` and `keyboard_type` go through `_refresh` (live session, holder, not expired, container output required). Driver first, then the lease mirror — the `pointer_button` order, so a refused key is not recorded as down. `_expire_pointer` reads keys before `expire()` and uses the one `release_all`. Same on `pointer_release`, kill, reap, and driver loss. |
| `src/hhstreamd/protocol.py` | Verb allowlist plus the mutation set. |
| `src/hhstreamd/policy.py` | **[SUPERSEDED — change 5.] No change.** A `[keyboard]` table here would be read by nothing: `settings_from_dict` is read by `hhstreamd`, and the driver is a *different unit* that never opens that file. An operator could change the layout and every injected key would stay on `us`. The layout travels as `--xkb-layout` on the driver's argv, and nowhere else. |
| `src/hhstream_session/__init__.py` | `_build_keyboard` beside `_build_pointer`. |
| `src/hhstream_session/format.py` | `render_pointer_state` shows held keys. |
| `src/hhstream_session/lease.py` | No new store. `pointer-leases.json` remains the cache. |
| `nix/module.nix` | No new unit. Extend `hhstream-pointer.service` argv — **and nothing else**: **[SUPERSEDED — change 5]** the `[keyboard]` TOML emission is dropped, see `policy.py` above. Options under `sessions.keyboard`: `layout` (`lib.types.strMatching "[A-Za-z0-9_-]+"`, default `"us"` — the type is what refuses a comma-separated list), `variant` (default `""`), `options` (default `""`), `model` (default `"pc105"`). The driver also refuses a comma itself, before `xkb_keymap_new_from_names`, because the NixOS type cannot reach a binary run by hand. §3.4's "command line the module will pass, exactly" and the unit comment quoting it are amended in the same change. |
| `nix/package.nix`, `flake.nix` devshell, `pyproject.toml` | `xkbcommon` for the driver and for unit tests. No new console script. |
| `nix/hyprland-conf.nix` | Keep `input:follow_mouse = 1`. Extend the comment: keyboard events have no output, so the warp is what moves keyboard focus — but see §6 risk 1 and Stage 0, because **one** warp does not — and the lease is what stops another session stealing it mid-string. **[SUPERSEDED — change 4]** Also set `input:virtualkeyboard:release_pressed_on_close = true`, with the note that it releases keys and still does not clear the modifier mask. |
| `docs/CONTRACT-sessions.md` | Amend §3.3 (key mirror, `expire()` still carries no keys), §3.4 (ops above), §4 (verbs, idle rule). |
| `README.dev.md` item 3, `checks/vm-pointer.nix` header | Stale relative to the conf file as read: they still say `hyprland-conf.nix` ships `follow_mouse = 0` and that the keyboard task will choose. The conf file sets `1`. Fix the prose when implementing; do not set the conf back to `0`. |
Flags the unit passes, defaults matching §5's style of frozen defaults:
```
hhstream-pointer --socket /run/hhstream/pointer.sock \
--compositor-runtime-dir /run/hypr-stream \
--xkb-layout us --xkb-model pc105 \
--xkb-variant '' --xkb-options ''
```
A layout xkbcommon rejects, before any Wayland connect, exits **2**. Compositor failures stay **10**.
---
## 3. Lease
**Decision: one lease. Keyboard verbs require the pointer lease. Do not add a keyboard lease, a `keyboard` boolean, or a second sweeper.**
§6.7 says the lease "gains a `keyboard` flag" and that no new mechanism is needed. The flag is the wrong reading of the code that shipped. `_Held.buttons` exists because `release_all` must know **which** buttons went down and in what order (`src/hhstreamd/pointer.py`, `ops.py`). A boolean cannot lift `Control_L` and not `a`. The key set is that pattern, not a new state machine.
Why sharing is required by the seat, not just by the plan: there is one `wl_seat` and one keyboard focus. `follow_mouse = 1` (`nix/hyprland-conf.nix`) means another session's warp retargets the keys. Two leases would let session B move the pointer while session A believes it is typing into its own window. `LeaseBusy` on acquire is already the mutex (`PointerLease.acquire` refuses even the current holder a second id, so a drag — or a half-typed chord — cannot have its id silently replaced).
Rules that keyboard inherits unchanged:
- Acquire / refresh / hard cap (`ttl` 30 s, `max` 300 s, refresh moves only the idle deadline).
- `expire()` does not lift anything. The broker reads the mirror, drops the lease, then `release_all` on the driver (`_expire_pointer`).
- Driver death force-releases (`driver_gone`). The driver is not asked to remember who held the lease.
- Kill and reap of the holder drop the lease with the existing reasons (`kill` / `crash`).
- CLI cache is re-checked by the broker; a stale id is exit 9 and then forgotten.
- Gesture verbs do not mark activity.
**TTL expiry with keys held down.** Same as buttons (`PLAN` §6.2 "Never leave a button down", implemented in `_expire_pointer` and in `_lift_everything`):
1. Sweeper (or the next verb's `_refresh`) sees the deadline.
2. Broker reads `keys_pressed()` and `pressed()` **before** `expire()`.
3. `expire()` drops the lease. It is not kept open to finish the word.
4. Driver `release_all`: button-ups, then key-ups in press order, then `modifiers(0, 0, 0, group)`, then the pointer `frame`s it already sends for buttons. No keyboard frame.
5. Audit `pointer_lease_timeout` gains `keys_lifted`. The session's pointer bookmark is stamped `released_by: timeout` as today.
6. The next `acquire` is free. Nothing is replayed. A held Shift does not leak into the next holder.
If `release_all` fails, do what the button path does: log it, still drop the lease, do not retry a request whose outcome is unknown (`driver.py`). The stuck-key case is a risk in §6, not a second expiry protocol.
`type` refreshes **once per broker verb**, inside `_refresh`, not once per character. Explicit `down` of a modifier stays in the lease mirror across later `type` calls. `type` itself must not leave letters down.
A warp while a key is held stays legal, same as a drag. Focus can follow that warp. That is another reason both devices share the lease: the holder is the only session allowed to move focus until release or TTL.
---
## 4. Keymap and text
**Decision: one seat-wide XKB layout, chosen in config, default `us` / `pc105` / empty variant / empty options. The virtual keyboard's keymap is the one we upload. Hyprland's `input:kb_layout` is not read and does not affect injected keys.**
Do not query `hyprctl getoption` to fill a default. The broker's rule is that unverified fields stay null (`PLAN` §7); a layout guessed from the compositor would drift on reload. Operators who are not on `us` set `sessions.keyboard.layout`.
**Decision: v1 is a single group.** A comma-separated layout (`us,ru`) is rejected at config load. Group in every `modifiers` request is **0**. No group-switch keysym.
**Where it lives.** `[keyboard]` in the broker TOML (`policy.py`), rendered by `nix/module.nix`, passed as the flags in §2. The driver compiles it at startup and uploads that byte string. The session record does not gain a layout field. Per-session layouts would mean re-uploading `keymap` on every acquire, which drops held state; not in v1.
**How text is expressed.**
- `keyboard type` takes a Python `str` of Unicode scalars. The keymap maps each scalar to one keysym, then to one `(evdev code, modifier mask)`.
- Newline → keysym `Return`. Tab → `Tab`. Other C0 controls are rejected with the rest of the string, not skipped.
- `A` is Shift+`a` via the **mask**, not via a typed Shift key, unless the user also has Shift explicitly held.
- Explicit modifier holds are a baseline. `type` ORs each character's mask with that baseline, sends the letter, and restores the mask to the baseline afterwards. So `down Control_L` then `type c` is Ctrl+C, and Control is still down when `type` returns.
- If any **non-modifier** key is already held, `type` is rejected and emits nothing.
- Validation is complete before the first sink call. Unmappable character → the whole request fails, no prefix typed. This includes `é` on `us`.
- Dead keys (`dead_acute` and the like) are a rejection even if the layout names them. v1 does not emit a two-event compose. `us(intl)` is therefore a poor choice until a later stage; default `us` does not need it.
- No auto-repeat timer in our process. **[SUPERSEDED — change 7, and Stage 0 has measured it.]** Hyprland does not repeat on the wire: a key held 1.5 s produced exactly one `wl_keyboard.key`, and it publishes `repeat_info rate=25 delay=600` so the *client* repeats. That is not good news — a focused text field will insert characters from that one press until the key-up or the TTL — so the `down` help text warns **unconditionally** rather than only if a probe saw repeats, and v1 has no rate to offer because we do not generate them.
**Modifier decision:** for a shifted character, send `modifiers(shift_mask, 0, 0, 0)` and the letter's `key` down/up, then restore the previous mask. Do **not** also synthesize `Shift_L` down/up for that character. For an explicit `keyboard down Shift_L`, send the Shift **key** and a matching `modifiers` update, and remember both in the held-key record so `release_all` undoes both. Sending both a synthetic Shift key and a mask for an ordinary capital is how toolkits apply Shift twice or not at all; the VM check in §5 is what proves the letter path against a real `wl_keyboard` client.
Named non-character keysyms used by `key`/`down`/`up` go through the same resolver, not a second table that can disagree with the layout. `buttons.py` can stay a hand-written evdev table because the protocol takes those codes directly; keysyms cannot.
**[SUPERSEDED — change 6.]** This section never defined "modifier", and a Shift-only reading of it is wrong: `@` on `de` is a **level-three** symbol, so a resolver that only knew about Shift would type the wrong character. The definition is *measured from the keymap*, not listed — press each keycode in a throwaway XKB state and read the serialised masks back:
- a non-zero **depressed** mask and no locked mask makes it a **modifier**: on `us`/`pc105` that is exactly `Shift_L/R`, `Control_L/R`, `Alt_L/R`, `Super_L/R`, `ISO_Level3_Shift`, `ISO_Level5_Shift`. (`Meta_L/R` and `Hyper_L/R` are canonical modifier keysyms that **no keycode on this layout carries** — which is the argument for measuring rather than listing.)
- a non-zero **locked** mask makes it a **lock**: `Caps_Lock`, `Num_Lock`. A lock is not a modifier, and `type` refuses while one is held rather than producing a case nobody asked for.
The resolver returns `(evdev code, mask)` for a keysym at *any* level, so one code path serves `us` and `de`. And `key`/`type` never call `_aimed()`: a keyboard has no output, so demanding a prior warp would refuse a caller who only ever wanted to type.
---
## 5. Tests
Match the split already in the tree: the dispatcher is tested against a recording sink and is not tested for authority (`tests/pointer/test_ops.py`); the lease is pure with `FakeClock` and no sleep (`tests/pointer/conftest.py`); the broker is tested with a real UNIX socket and `FakeDriver`, Wayland substituted (`tests/sessions/unit/test_pointer_verbs.py`). No integration test grows a fake compositor. `tests/sessions/integration/` does not gain Wayland.
### Unit
| Test | Asserts |
| --- | --- |
| `tests/pointer/test_keymap.py` | Real xkbcommon, layout `us`: `a` → evdev 30, mask 0; `A` → 30 plus Shift; `Return` and `\n` → `KEY_ENTER`; empty string → `[]`; `é` and a C0 control other than `\n`/`\t` raise; the raise happens with zero sink calls when driven through the dispatcher. Dead keysym raises. |
| `tests/pointer/test_ops.py` | Extend `RecordingSink` with `key(code, state)` and `modifiers(...)`. `key down` then `pressed` lists the name. `up` of a key that was not down reports it not lifted (same as `unpress`). `release_all` lifts keys in press order and ends with a zeroed `modifiers`, even if the last `warp` was to another output (keys are not per output). `type` emits down/up pairs and leaves `keys` empty. `type` with `Control_L` held ORs the mask and leaves Control down. `type` with `a` held emits nothing and returns `ok: false`. A keymap that raises on the third character emits nothing. |
| `tests/pointer/test_lease.py` | `press_key` / `keys_pressed` order. `expire()` still returns no key list; the test reads `keys_pressed()` first, then `expire()`, then sees an empty lease. Existing busy/stale/`lease_id` cases stay as they are. |
| `tests/pointer/test_vendored_protocol.py` | Interface name `zwp_virtual_keyboard_manager_v1` and requests `keymap`, `key`, `modifiers`. No display. |
| `tests/sessions/unit/test_keyboard_verbs.py` | New, harness copied from `test_pointer_verbs.py`. No lease → 9. Other session holds the lease → 9 (not 8). Second `pointer.acquire` while typing → 8. `type` refreshes TTL: advance the `ManualClock` by less than `ttl` after a type and the lease survives; advance past it and `release_all` runs with the held key. Driver rejection does not stick the key in the lease. `type` does not move `last_activity_boottime`. `pointer.release` and kill both lift keys. `pointer.state` includes `keys`. |
| ~~`tests/sessions/unit/test_settings.py`~~ | **[SUPERSEDED — change 5.] Dropped.** A TOML default of `us` that nothing uploads is precisely the false pass that file's own docstring warns about. Replaced by `tests/pointer/test_driver_flags.py`: an omitted `--xkb-layout` is `us`, `de` is `de`, and `us,ru` exits 2 **with an injected sink proving the compositor was never touched**. |
| CLI surface test (`tests/sessions/unit/test_cli_surface.py`) and broker verb list (`test_broker_protocol.py`) | `keyboard` in help; help says the pointer lease is required; `keyboard.type` / `keyboard.key` are known verbs and are in the mutation set. |
`nix develop -c python -m pytest tests/pointer tests/sessions/unit -q` is the gate. Not run for this plan.
### Integration / VM
**Decision: a new `checks/vm-keyboard.nix`, same shape as `checks/vm-pointer.nix`. Do not call the feature done on unit tests alone.** `docs/TODO/scroll-has-no-compositor-proof.md` is the precedent: scroll shipped on the driver socket with no compositor proof. Keyboard's only purpose is that the client receives symbols.
- Victim: a small Wayland client beside `checks/lib/pointer_target.py` (or an extension of it) that sets `xdg_toplevel` app id from `$HHSTREAM_APP_ID` and prints `wl_keyboard.enter`, each `key` (evdev code, state), each `modifiers` mask, and the keysym it decodes from the keymap the compositor forwarded. Read-back is `hhstream session logs`, as in `vm-pointer`.
- Assertions, real time, no injected clock (the unit clock already covers the arithmetic; `vm-pointer`'s expiry subtest states why):
1. After `pointer acquire` on session A, A's log shows `wl_keyboard.enter`. This is the focus measurement. If it fails, the check is red and Stage 4 is in play — do not weaken the assertion.
2. `keyboard type A hello` decodes to the characters `hello` on A and to nothing on session B.
3. `keyboard down A Shift_L`, wait out a short `--ttl`, A's log shows the key-up and a cleared mask, and `pointer state` shows no holder.
4. A second acquire during the hold exits 8.
- Mutation, per `PLAN` §14: the character assertion must fail if `keymap` was never uploaded (keycodes without a decoded `hello` are a failure, not a pass).
- `globalTimeout` on the order of `vm-pointer` (1800). This check is slow; that is not a reason to fold it into a unit test.
---
## 6. Risks, open questions, stages
### Risks
- **Focus does not follow a single virtual-pointer warp. [MEASURED — Stage 0, and this is the headline risk of the feature.]** One `motion_absolute` onto an unfocused window, after the cursor had been still for 0.8 s, delivered `wl_pointer.enter` and **no** `wl_keyboard.enter`; the focus change arrived only after a *later* motion that changed the floored pixel. The mechanism is `InputManager::onMouseWarp` → `mouseMoveUnified(refocus=false)`: the delta is set to 0 when the previous movement was over 0.5 s ago, `mouseMoveUnified` returns early when the floored pixel did not change at all, and keyboard focus is that delta compared against `input:follow_mouse_threshold` (measured 0.0, unset). Transcript in `tests/README-keyboard-probe.md`.
**`pointer acquire` is one warp**, so the first key of a lease can land in whatever was already focused. The fallback is the one named below — `dispatch focuswindow` on the session's `window.stable_id`, immediately before the first key of a lease — and **it is still not built in Stages 1–2.** Two things that are *not* the fix: warping twice to the same bookmark (the same floored pixel returns early), and waiting longer after the warp (the first warp's settle window was 0.4 s and contained the pointer enter alone). Stage 3's enter subtest must not be written against acquire's single warp.
- **Destroy of the virtual keyboard leaves a key down inside Hyprland. [MEASURED — Stage 0.]** `destroy` with a key held, with a second virtual keyboard keeping the seat's capability up, delivered no key-up; `input:virtualkeyboard:release_pressed_on_close` is `0` and unset. Worse than "stuck": `InputManager::onKeyboardKey` drops a *press* whose code is already in the seat's `m_pressed`, so the next driver's press of that code is ignored and the key is **dead** until something sends a matching up. Our shutdown path sends key-ups first (`_lift_everything`); a `-9` skips it, which is why **[change 4]** `nix/hyprland-conf.nix` now sets `release_pressed_on_close = true`. That setting emits key-ups only and still never clears the modifier mask, so `release_all` keeps sending `modifiers(0, 0, 0, 0)` regardless. Still no retry loop the button path does not have.
- **Auto-repeat is the client's, which does not make it harmless. [MEASURED — Stage 0.]** `press_count=1` over a 1.5 s hold, with `repeat_info rate=25 delay=600` published at bind time. So the flood is real and it is foot/GTK/Qt producing it from our one press, for as long as the key is down — up to the TTL (default 30 s). **[SUPERSEDED — change 7]** the help text for `down` therefore warns unconditionally; the old "if it does, the help text says so" is the false pass the review named. v1 has no repeat-rate control, and `zwp_virtual_keyboard_v1` has no `repeated` state to ask the compositor about. Chords (`Control_L` held across a `type`) remain the justified use of `down`.
- **Mask versus Shift keycode.** §4's letter path can still be wrong for a toolkit that ignores `modifiers` and only shifts when it sees `KEY_LEFTSHIFT`. The VM victim must print both the mask and the decoded sym. Chromium is not the first victim (`vm-pointer` avoided it; it is expensive and the app-id rule in `README.dev.md` item 4 only binds clients that honour `$HHSTREAM_APP_ID`).
- **The lease is not a security boundary.** `PLAN` §6.6: any client may bind the pointer global. The keyboard global is the same. Stage 4 sandboxing is unchanged and out of this work.
- **Concurrent typing is impossible** on one compositor, for the same reason concurrent pointers are (`PLAN` §1). A second seat is Stage 5 and still CPU-gated. This work does not pretend otherwise.
- **`xkbcommon` in the devshell** is a new dependency. Tests that import `keymap.py` cannot skip when it is absent; an optional dependency would let the unmappable-character path rot.
### Open questions
Questions 1–4 are **closed by Stage 0** (`tests/README-keyboard-probe.md` and its transcript). Question 5 is answered by the operator.
1. ~~Stage 0 global name and version on the actual `/run/hypr-stream` socket.~~ **`zwp_virtual_keyboard_manager_v1` at version 1.** No `wp_virtual_keyboard_v1`. The gate holds.
2. ~~Does acquire's warp produce `wl_keyboard.enter`?~~ **No — not from a single idle warp.** See risk 1 above; this is the one answer worse than the plan assumed.
3. ~~On destroy, does Hyprland release keys that were down?~~ **No.** See risk 2.
4. ~~Does Hyprland auto-repeat those keys?~~ **Not on the wire; the client does.** See the auto-repeat risk.
5. **Answered: `us`, and configurable.** The operator's decision is a NixOS option in the `hhstream` namespace, string, default `"us"`, documented in `README.dev.md` — reaching the keymap as `--xkb-layout` on the driver unit, per change 5.
6. A later stage, only if agents must type arbitrary UTF-8: a throwaway keymap that maps spare keycodes to `0x01000000 + codepoint` for the duration of one `type`, then restores the layout keymap. Not v1. It reintroduces the "upload resets held modifiers" problem, so it has to refuse to run while any key is held.
### Stages
| Stage | Work | Rough size | Gate |
| --- | --- | --- | --- |
| 0 | **Done.** `tests/keyboard_probe.py`, its README and its transcript. Two windows, not one; read-only `hyprctl` for the boxes; `(a1)` and `(a2)` measured separately; no `hyprctl dispatch`. | S | **Passed.** Named global is `zwp_virtual_keyboard_manager_v1` v1. |
| 1 | Driver: vendor, `keymap.py`, ops, backend upload, `__main__` flags, pointer unit tests. Pointer still runs if the global is missing. | M, about 3–5 days | `pytest tests/pointer` green. `type` of an unmappable character emits no events. |
| 2 | Broker mirror, verbs, CLI, module argv, session unit tests, contract doc amendments, stale `follow_mouse` comments. **No `policy.py` and no TOML table** (change 5). | M, about 2–4 days | `pytest tests/pointer tests/sessions/unit` green. Expiry test shows `release_all` lifting a held key. No new unit under `nix/module.nix`, and nothing there beyond argv. |
| 3 | `vm-keyboard.nix` and the keyboard target client. **Do not design its enter subtest from acquire's single warp** — Stage 0 says that does not focus. | M, about 2–3 days | Characters on A, silence on B, key-up after a real TTL, second acquire is 8. Red if the keymap was not uploaded. |
| 4 | **No longer contingent: Stage 0 showed no `enter` after a single idle warp.** `focuswindow` via the broker's existing compositor IPC, immediately before a lease's first key, then re-run Stage 3. Still not to be built in Stages 1–2. | S, about a day | The Stage 3 enter assertion goes green without loosening it. |
Rollback of Stages 1–2 is reverting the commit: old brokers never send `key`/`type`, and an old driver rejects unknown ops with `ok: false` (`ops.py` already answers unknown `op` instead of dropping the connection). No registry schema bump. Bookmarks stay as they are.