T: review round 1 fix runner argv, filter cache, and the test builtin

This commit is contained in:
ari committed 2026-10-19 22:37:08 -04:00
1 parent 40cf1073f4
commit 6dade6239d
6 files changed
+562 -93

No files matched your search

+18 -6
View File
@@ -4,7 +4,7 @@ Part of [Phase 5: Consolidator](README.md). Contracts: [contracts.md](contracts.
| Field | Value |
|---|---|
| Status | T1 to T7 done: `nix/tests/peers.nix` and the `test` runner are written; `peers.nix` fails at the first peers assertion until WS-E lands (by design) |
| Status | T1 to T7 done; review round 1 addressed. `peers.nix` fails at the first peers assertion until WS-E lands (by design) |
| Owner | |
| Depends on | none (uses the module's freeform `settings` for `[peers]` and a systemd override for `LoadCredential`, so it needs no module changes). F1 later swaps this for the first-class module options |
| Blocks | nothing technically; it is the acceptance gate: goes green at the end of [WS-E](ws-e-telemetry.md) (with A, B, C, D landed), then F2 promotes it to `checks` |
@@ -35,14 +35,14 @@ Note on the written test: `nix/tests/peers.nix` also contains a third node `gamm
- **T6** no chaining and read-only: a request carrying `x-slinuxm-hop` is rejected; non-GET to a proxy path is 405; a TLS variant with a test CA via `ca_file`. **Done** (405 for POST/DELETE, hop header 4xx, `gamma` over HTTPS).
- **T7** **Done**: `dev/test.nu` with `unit`, `integration`, `lint`, `vm`, `all`, `ci`, `completions`, `self-check`; devenv `test` script plus zsh/bash completion hook; AGENTS.md and README.md command lists; `docs/consolidator.md` `## Testing`. A simple test runner CLI written in **Nushell**: `dev/test.nu` (devenv adds `pkgs.nushell`; also exposed as the devenv script `test`, so `devenv shell -- test unit` works from any shell). One entry point so contributors and agents do not memorise cargo/nix incantations. Structure it as a Nushell script with a `main` and `main <sub>` subcommands (`def "main unit"`, ...) so `--help` is generated and completion comes from the typed signatures:
- `test unit` runs `cargo test --lib --bins` (no sockets, no host access).
- `test integration [filter]` runs `cargo test --tests` (plain, tls, integration, binary_smoke, `peers` once it exists). `filter` is passed to cargo as a name filter; `--nocapture` flag supported.
- `test integration [filter]` runs `cargo test --test '*'` (integration targets only: plain, tls, integration, binary_smoke, `peers` once it exists), so `test all` does not run the unit tests twice. `filter` is passed to cargo as a name filter. `--nocapture` is a libtest flag and is always placed after `--`.
- `test lint` runs `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings` and the `cargo tree -i openssl` emptiness check.
- `test vm [name]` runs a NixOS VM test. `name` is one of `vm` (from `checks`) or `peers` (`nix build .#tests.x86_64-linux.peers -L`; moves to `checks` when green). Refuses without `/dev/kvm` and says why; prints that it is slow.
- `test vm [name]` runs a NixOS VM test. `name` is one of `vm` (from `checks`) or `peers` (`nix build .#tests.x86_64-linux.peers -L`; moves to `checks` when green). A missing `/dev/kvm` fails that step and says why, after earlier `ci` steps, and the summary table still prints. The step prints that the build is slow.
- `test all` = lint, unit, integration, stopping at the first failure; `test ci` = `all` plus `vm`. No args prints help.
- Each step prints a header and a final table (step, result, seconds); exit code non-zero on any failure (use `error make`/`exit 1`, never swallow cargo's exit status). Read-only against the host: no `/etc`, systemd or Docker changes (live-host checks stay manual, see Verification plan in the [README](README.md#verification-plan)).
- **Autocomplete (must work, and is a done-criterion):**
- Nushell: subcommands and flags complete natively once the script is loaded as a module. Dynamic values use custom completers (`def "nu-complete test vm" [] { ["vm" "peers"] }` and `name: string@"nu-complete test vm"`; integration filters complete from `cargo test --tests -- --list` test names, cached for the session). The devenv `enterShell` prints/loads `use dev/test.nu *` for Nushell users, and `docs/consolidator.md` shows the one-line `use` for a permanent setup.
- Other shells (the repo owner's login shell is zsh): `test completions zsh|bash` prints a small static completion script (subcommands, `vm` names, flags) generated from the same command table in the script, so zsh/bash tab completion works through `devenv shell` too (`source <(test completions zsh)` in `enterShell`).
- Nushell: subcommands and flags complete natively once the script is loaded as a module. Dynamic values use custom completers (`def "nu-complete test vm" [] { ["vm" "peers"] }` and `name: string@"nu-complete test vm"`; integration filters complete from `cargo test --test '*' -- --list`, cached in `target/skyg-test-filters.cache` and keyed on the newest mtime of the Rust sources and Cargo manifests). A completer cannot update the caller's environment, so an env var does not survive Tab. `docs/consolidator.md` shows the one-line `use dev/test.nu *` for a permanent setup.
- Other shells (the repo owner's login shell is zsh): `nu dev/test.nu completions zsh|bash` prints a completion script generated from the same command table. `test` is a shell builtin, so in an interactive bash or zsh `test unit` exits 0 without running the suite. The script defines a `test` function that sends known subcommands to the runner and otherwise calls `builtin test`, which keeps `test -f` working. `devenv shell -- test …` does not need the function: it finds the devenv script on PATH. Interactive `devenv shell` sources the bash script and prints a one-line note on stderr. zsh is not started by that hook; after `compinit`, zsh users add `source <(nu dev/test.nu completions zsh)` to `.zshrc` themselves. direnv does not carry the function.
- Verified by a self-test: `test --help` and `test vm --help` list every subcommand/flag, and a check in `dev/test.nu` (`test self-check`) fails if the completion table and the defined subcommands diverge.
- AGENTS.md and README replace their scattered command lists with `test ...`, keeping the underlying commands visible in `--help`.
@@ -65,4 +65,16 @@ The test builds under `nix build .#tests.x86_64-linux.peers` (or equivalent) and
- 2026-10-19: Split from the single phase plan 2026-10-19. `nix/tests/peers.nix` already written (T1 to T6) and failing at the first peers assertion by design; `flake.nix` exposes it as `tests.<system>.peers`. T7 not started. Ownership change from the single plan: `tests/peers.rs` moved to WS-F; WS-A/B in-process tests use their own files to avoid collisions.
- 2026-10-19 (WS-T): T7 done. `dev/test.nu` is a Nushell module: `use dev/test.nu *` gives `test unit|integration|lint|vm|all|ci` plus `completions` and `self-check`, and `nu dev/test.nu <sub>` dispatches through `main` for the devenv `test` script. Subcommands are declared as `test <sub>` (not `main <sub>`): with `use dev/test.nu *` Nushell binds the module's `main` to the file stem (`test`), and only commands literally named `test ...` complete and dispatch as `test unit`. Native subcommand help comes from the signatures; the entry help block lists the same table, and `test self-check` fails when the table, the defined commands, their flags or the help text diverge (mutation-tested: extra table row, extra command, extra flag). zsh/bash scripts are generated from the same table (`test completions zsh|bash`), sourced by `enterShell`. Verified: `test all` (fmt, clippy, openssl, unit, integration) green, `--ide-complete` returns subcommands, `vm`/filter completers, `bash -n`/`zsh -n` on the generated scripts. Not run here (no nix builds in this worktree): `nu dev/test.nu vm peers`; on a KVM host that is `nix build .#tests.x86_64-linux.peers -L`.
- Handoff 2026-10-19 (WS-T -> all): the runner is `dev/test.nu`; use `test lint`, `test unit`, `test integration [filter] [--nocapture]`, `test vm [vm|peers]`, `test all`, `test ci`. `nix/tests/peers.nix` is unchanged and still the acceptance gate. To WS-F: promote `tests.<system>.peers` to `checks.vm-peers` once `test vm peers` passes; `test integration` picks up `tests/peers*.rs` with no runner change. Caveat: per-subcommand `--help` is native in Nushell module mode; in script mode (`nu dev/test.nu vm --help`) Nushell prints the entry help, which lists every subcommand.
- Handoff 2026-10-19 (WS-T -> all): the runner is `dev/test.nu`; use `devenv shell -- test lint|unit|integration|vm|all|ci` or `nu dev/test.nu <subcommand>`. `nix/tests/peers.nix` is unchanged and still the acceptance gate. To WS-F: promote `tests.<system>.peers` to `checks.vm-peers` once `test vm peers` passes; `test integration` picks up `tests/peers*.rs` with no runner change. Caveat: per-subcommand `--help` is native in Nushell module mode; in script mode (`nu dev/test.nu vm --help`) the entry help lists every subcommand.
- 2026-10-19 (WS-T review round 1): Addressed every finding in `loop/T/review-1.md`. No contract change. Nothing outside the owned files. `TODO.md` is still unticked (not an owned file); the merger ticks T7.
1. `test integration --nocapture` with no filter now runs `cargo test --test '*' -- --nocapture`. Cargo rejects a bare `--nocapture`. `self-check` compares the step command, spawns the runner in a dry-run, and runs `cargo test --test '*' -- --nocapture --list`.
2. `test` stays the devenv script name (the plan's name, and `devenv shell -- test` already resolves it on PATH). Inside an interactive bash or zsh the builtin wins and `test unit` exits 0. The generated completion script defines a `test` function that dispatches known subcommands to this file and otherwise calls `builtin test`, so `test -f` still works. Interactive `devenv shell` sources the bash script. Docs tell zsh users to add `source <(nu dev/test.nu completions zsh)` themselves. `self-check` sources the bash and zsh scripts with a fake `nu` and checks both paths. Disagree with renaming: it would change the command every other stream already cites, and the PATH form the plan specified already works.
3. Filter completion is `target/skyg-test-filters.cache`, first line a mtime key of `src/**/*.rs`, `tests/**/*.rs`, `Cargo.toml` and `Cargo.lock`. The old `$env.SKYG_TEST_FILTERS` assignment died with the completer. `self-check` proves a second lookup does not re-list and a stale key does.
4. `enterShell` runs in devenv's bash, so the zsh branch never installed `compdef`. Removed. Docs say zsh users source the script from `.zshrc` after `compinit`. direnv will not carry the function.
5. The enterShell note goes to stderr and only when `$-` contains `i`, so `devenv shell -- test completions zsh` stays pipe-clean. Checked: a non-interactive `bash -c` does not take the `*i*` branch.
6. `test integration` is `cargo test --test '*'` (integration targets only). `test all` / `test ci` keep the separate `unit` step and no longer repeat it. Help text in `AGENTS.md` and `README.md` matches.
7. The openssl step passes only when stdout is empty and cargo exits 0, or when stderr contains `did not match any packages`. Any other cargo failure fails the step. `self-check` covers a clean miss, an empty success, a hit, and a lockfile-style error.
8. Missing `/dev/kvm` is an error inside the vm step body, not while the step list is built. `self-check` sets `SKYG_TEST_ASSUME_NO_KVM=1` and asserts `vm-steps` returns a step whose body reports `/dev/kvm`.
9. Script mode is `--wrapped`, so flags reach our parser. Unknown flags and extra positionals are errors. `--nocapture` is valid only on `integration`. `self-check` covers the parser and the CLI forms `unit --nocapture` and `integration a b`.
10. `TODO.md:28` T7 remains unticked on purpose.
- Handoff 2026-10-19 (WS-T review round 1 -> all): same subcommand names. Invoke as `devenv shell -- test <subcommand>` or `nu dev/test.nu <subcommand>`. Interactive `devenv shell` installs the bash wrapper; zsh needs the `.zshrc` line after `compinit`. `nu dev/test.nu self-check` is the runner's test. Not run here (no nix builds in this worktree): `nu dev/test.nu vm peers`, which on a KVM host is `nix build .#tests.x86_64-linux.peers -L`. `nix/tests/peers.nix` is unchanged.