diff --git a/AGENTS.md b/AGENTS.md index c78c141..3484fe3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -18,12 +18,12 @@ Canonical guidance for AI coding agents in this repo (no CLAUDE.md; this file is Enter the toolchain with `devenv shell` (Rust, pkg-config, libsystemd, nushell, otelcol). Without it you need a Rust toolchain, `pkg-config` and the libsystemd dev files. -All checks go through one runner, `dev/test.nu`, exposed as the `test` script inside `devenv shell`. `test --help` lists the underlying cargo and nix commands. +All checks go through one runner, `dev/test.nu`. `devenv shell -- test …` runs the devenv script on PATH. In an interactive bash or zsh, `test` is the shell builtin, so `test unit` exits 0 without running the suite; use `devenv shell -- test unit` or `nu dev/test.nu unit`. An interactive `devenv shell` installs a bash function that runs known subcommands and leaves `test -f` to the builtin. zsh users add `source <(nu dev/test.nu completions zsh)` to `.zshrc` after `compinit`. `nu dev/test.nu --help` lists the underlying cargo and nix commands. ```sh devenv shell -- test lint # cargo fmt --check, clippy -D warnings, cargo tree -i openssl (empty) devenv shell -- test unit # cargo test --lib --bins -devenv shell -- test integration [filter] [--nocapture] # cargo test --tests (plain, tls, integration, binary_smoke, peers) +devenv shell -- test integration [filter] [--nocapture] # cargo test --test '*' (plain, tls, integration, binary_smoke, peers) devenv shell -- test vm [vm|peers] # NixOS VM test (slow, needs /dev/kvm) devenv shell -- test all # lint, unit, integration; stops at the first failure devenv shell -- test ci # all plus the vm test @@ -32,7 +32,7 @@ cargo build --release # target/release/slinuxm nix build && nix flake check # package, module, VM test (slow; Nix changes only) ``` -`devenv up` runs a local OTLP collector (`dev/otelcol.yaml`). Do not run nix builds unless the task involves Nix files, and never while another agent is running the nix gate. When editing `dev/test.nu`, run `nu dev/test.nu self-check`; it fails if the command table, the subcommands and the help text drift apart. +`devenv up` runs a local OTLP collector (`dev/otelcol.yaml`). Do not run nix builds unless the task involves Nix files, and never while another agent is running the nix gate. When editing `dev/test.nu`, run `nu dev/test.nu self-check`; it fails if the command table, the subcommands, the help text, the cargo argv or the shell wrapper drift apart. ## Conventions diff --git a/README.md b/README.md index 4475e6a..7b7f48e 100644 --- a/README.md +++ b/README.md @@ -178,11 +178,11 @@ ldd /usr/local/bin/slinuxm # libc, libm, libgcc_s, libsystemd and its depende ## Testing -Checks run through one runner, `dev/test.nu`, which is also the `test` script inside `devenv shell`. `test --help` lists every subcommand and the cargo or nix command behind it. +Checks run through one runner, `dev/test.nu`. `devenv shell -- test …` runs it from any shell. In an interactive bash or zsh, `test` is the shell builtin, so `test unit` exits 0 without running the suite. `devenv shell -- test --help` lists every subcommand and the cargo or nix command behind it. ```sh devenv shell -- test unit # cargo test --lib --bins -devenv shell -- test integration [filter] [--nocapture] # cargo test --tests +devenv shell -- test integration [filter] [--nocapture] # cargo test --test '*' (integration targets only) devenv shell -- test lint # cargo fmt --check, clippy -D warnings, no openssl devenv shell -- test vm [vm|peers] # NixOS VM test (slow, needs /dev/kvm) devenv shell -- test all # lint, unit, integration; stops at the first failure @@ -198,4 +198,4 @@ The smoke test runs the real built `slinuxm` binary against a throwaway SQLite d devenv shell -- test all # fmt, clippy, unit and integration tests ``` -`test --help` shows the underlying commands (`cargo fmt --check`, `cargo clippy --all-targets -- -D warnings`, `cargo test`). In Nushell, `use dev/test.nu *` gives `test unit`, `test vm peers`, ... with tab completion; in zsh or bash, `source <(test completions zsh)` (or `bash`). The integration tests in `tests/` start the real router on an ephemeral port with fake sources. Design notes live in [ARCHITECTURE.md](ARCHITECTURE.md); work tracking in [TODO.md](TODO.md); extended docs in [docs/](docs/README.md). +`devenv shell -- test --help` shows the underlying commands (`cargo fmt --check`, `cargo clippy --all-targets -- -D warnings`, `cargo test --lib --bins`, `cargo test --test '*'`). Nushell: `use dev/test.nu *` gives `test unit`, `test vm peers`, ... with tab completion. zsh users add `source <(nu dev/test.nu completions zsh)` to `.zshrc` after `compinit`. An interactive `devenv shell` installs the bash function; `test -f` stays the builtin. The integration tests in `tests/` start the real router on an ephemeral port with fake sources. Design notes live in [ARCHITECTURE.md](ARCHITECTURE.md); work tracking in [TODO.md](TODO.md); extended docs in [docs/](docs/README.md). diff --git a/dev/test.nu b/dev/test.nu index 4de7e9c..08fe06a 100755 --- a/dev/test.nu +++ b/dev/test.nu @@ -5,7 +5,7 @@ # signatures below; the self-check keeps both in step. const command_table = [ { name: "unit", usage: "cargo test --lib --bins", flags: [] } - { name: "integration", usage: "cargo test --tests [filter] [--nocapture]", flags: ["--nocapture"] } + { name: "integration", usage: "cargo test --test '*' [filter] [--nocapture]", flags: ["--nocapture"] } { name: "lint", usage: "fmt, clippy and the openssl check", flags: [] } { name: "vm", usage: "nix build of a NixOS VM test: vm or peers", flags: [] } { name: "all", usage: "lint, unit, integration; stops at the first failure", flags: [] } @@ -14,11 +14,17 @@ const command_table = [ { name: "self-check", usage: "check that the table, commands and help agree", flags: [] } ] +# Baked into the generated shell function so completion works from any directory. +const SCRIPT_PATH = path self + +# Completers cannot update the caller's environment, so the filter list is a file. +const FILTER_CACHE = "target/skyg-test-filters.cache" + # One entry point for the repository's checks. # # Subcommands: # unit cargo test --lib --bins -# integration cargo test --tests [filter] [--nocapture] +# integration cargo test --test '*' [filter] [--nocapture] # lint cargo fmt --check, cargo clippy --all-targets -- -D warnings, cargo tree -i openssl (empty) # vm nix build of a NixOS VM test: vm (checks) or peers (tests..peers) # all lint, unit, integration; stops at the first failure @@ -30,30 +36,41 @@ const command_table = [ # command per subcommand, so Nushell completes names and flags natively. # # The checks are read-only against the host: no /etc, systemd or Docker changes. -export def main [ - ...args: string # subcommand and its arguments, as in `test integration peers` - --nocapture # forward to `test integration` -] { - if ($args | is-empty) { - print-help - return - } - let sub = ($args | first) - let rest = ($args | skip 1) - let positional = ($rest | where {|a| not ($a | str starts-with "-") }) - if $sub not-in ($command_table | get name) { - error make { msg: $"unknown subcommand '($sub)'; run 'test --help' for the list" } - } - match $sub { +# +# Script mode (`nu dev/test.nu`) is `--wrapped` so flags reach the parser below. +# Nushell would otherwise accept `--nocapture` on every subcommand and reject +# the rest before `main` can explain them. +export def --wrapped main [...args: string] { + let parsed = (parse-script-args $args) + match $parsed.cmd { + "help" => { print-help } "unit" => { test unit } - "integration" => { test integration ($positional | get 0?) --nocapture=$nocapture } + "integration" => { + if ($parsed.positional | is-empty) { + test integration --nocapture=$parsed.nocapture + } else { + test integration $parsed.positional --nocapture=$parsed.nocapture + } + } "lint" => { test lint } - "vm" => { test vm ($positional | get 0?) } + "vm" => { + if ($parsed.positional | is-empty) { + test vm + } else { + test vm $parsed.positional + } + } "all" => { test all } "ci" => { test ci } - "completions" => { test completions ($positional | get 0?) } + "completions" => { + if ($parsed.positional | is-empty) { + test completions + } else { + test completions $parsed.positional + } + } "self-check" => { test self-check } - _ => { error make { msg: $"subcommand '($sub)' has no dispatch arm" } } + _ => { error make { msg: $"subcommand '($parsed.cmd)' has no dispatch arm" } } } } @@ -110,13 +127,22 @@ export def "test completions" [ # Fail if the command table, the defined commands or the help text diverge. export def "test self-check" [] { - let problems = (check-consistency) + let problems = ( + (check-consistency) + ++ (check-integration-args) + ++ (check-dispatch) + ++ (check-filter-cache) + ++ (check-openssl-verdict) + ++ (check-kvm-step) + ++ (check-shell-wrapper) + ++ (check-docs) + ) if not ($problems | is-empty) { print -e "test self-check failed:" $problems | each {|p| print -e $" - ($p)" } | ignore exit 1 } - print "test self-check: command table, commands and help agree" + print "test self-check: ok" } def "nu-complete test vm" [] { @@ -127,26 +153,86 @@ def "nu-complete test shells" [] { ["zsh" "bash"] } -# Test names come from cargo; the first Tab compiles in debug mode and is slow, -# so the list is kept for the rest of the session. +# `cargo test --test '*'` matches integration targets only, same as the integration step. def "nu-complete test filters" [] { - if ($env.SKYG_TEST_FILTERS? | is-not-empty) { - return $env.SKYG_TEST_FILTERS + let key = (try { source-mtime-key } catch { "unknown" }) + try { + cached-lines (filter-cache-path) $key {|| list-test-names } + } catch { + [] } - let listed = (do -i { run-external "cargo" "test" "--tests" "--" "--list" | complete }) - let names = ($listed - | default {stdout: ""} - | get stdout - | lines - | where {|l| $l =~ ': (test|benchmark)$'} - | each {|l| $l | str replace -r ': (test|benchmark)$' '' | str replace -r '^.*: ' '' } - | uniq) +} + +def filter-cache-path [] { + let override = ($env.SKYG_TEST_FILTER_CACHE? | default "") + if ($override | is-empty) { $FILTER_CACHE } else { $override } +} + +def source-mtime-key [] { + let rows = (ls src/**/*.rs tests/**/*.rs Cargo.toml Cargo.lock) + if ($rows | is-empty) { + return "none" + } + $rows | get modified | math max | format date "%s.%f" +} + +# Null means a miss. An empty list is a hit that listed no tests. +def read-cached-lines [path: string, key: string] { + if not ($path | path exists) { + return null + } + let lines = (open --raw $path | lines | where {|l| not ($l | is-empty) }) + if ($lines | is-empty) or (($lines | first) != $key) { + return null + } + $lines | skip 1 +} + +def write-cached-lines [path: string, key: string, names: list] { + mkdir ($path | path dirname) + ([$key] | append $names | str join (char nl)) | save -f $path +} + +def cached-lines [path: string, key: string, produce: closure] { + let hit = (read-cached-lines $path $key) + if $hit != null { + return $hit + } + let names = (do $produce) if not ($names | is-empty) { - $env.SKYG_TEST_FILTERS = $names + write-cached-lines $path $key $names } $names } +def list-test-names [] { + # Self-check points this at a counter file to prove Tab does not re-list. + let stub = ($env.SKYG_TEST_LIST_STUB? | default "") + if not ($stub | is-empty) { + let n = (if ($stub | path exists) { + open --raw $stub | str trim | into int + } else { + 0 + }) + ($n + 1 | into string) | save -f $stub + return ["stub-test"] + } + let listed = (try { + run-external "cargo" "test" "--test" "*" "--" "--list" | complete + } catch { + {stdout: "", stderr: "", exit_code: 1} + }) + if $listed.exit_code != 0 { + return [] + } + $listed.stdout + | lines + | where {|l| $l =~ ': (test|benchmark)$' } + | each {|l| $l | str replace -r ': (test|benchmark)$' '' } + | uniq + | sort +} + def step-record [label: string, command: string, body: closure] { { label: $label, command: $command, body: $body } } @@ -157,35 +243,71 @@ def unit-steps [] { })] } -def require-kvm [] { - if not ("/dev/kvm" | path exists) { - error make { - msg: "/dev/kvm is missing: NixOS VM tests need hardware virtualisation. Run on a Linux host with KVM, or in CI with KVM exposed; there is no software fallback." - } +def kvm-available [] { + # Self-check sets this to prove a missing device fails inside the step. + if ($env.SKYG_TEST_ASSUME_NO_KVM? | default "") == "1" { + return false + } + "/dev/kvm" | path exists +} + +def kvm-refusal [] { + if (kvm-available) { + null + } else { + "/dev/kvm is missing: NixOS VM tests need hardware virtualisation. Run on a Linux host with KVM, or in CI with KVM exposed; there is no software fallback." } } def vm-steps [name: string] { - require-kvm + # The KVM check runs in the step body. Raising it here would abort `test ci` + # before lint, unit and integration, and the summary table would not print. let attr = (match $name { "peers" => "tests" _ => "checks" }) let system = $"($nu.os-info.arch)-($nu.os-info.name)" [(step-record $"vm ($name)" $"nix build .#($attr).($system).($name) -L" {|| + let why = (kvm-refusal) + if $why != null { + error make { msg: $why } + } print "NixOS VM tests are slow: expect several minutes." run-checked "nix" ["build" $".#($attr).($system).($name)" "-L"] })] } +# Libtest flags must follow `--`. A bare `--nocapture` is a cargo flag and cargo rejects it. +def integration-cargo-args [filter: any, nocapture: bool] { + mut tail = [] + if not ($filter | is-empty) { + $tail = ($tail | append ($filter | into string)) + } + if $nocapture { + $tail = ($tail | append "--nocapture") + } + let args = ["test" "--test" "*"] + if ($tail | is-empty) { + $args + } else { + $args | append "--" | append $tail + } +} + def integration-steps [filter: any, nocapture: bool] { - let filter_args = (if ($filter | is-empty) { [] } else { ["--", $filter] }) - let capture_args = (if $nocapture { ["--nocapture"] } else { [] }) - let args = (["test" "--tests"] | append $filter_args | append $capture_args) + let args = (integration-cargo-args $filter $nocapture) let label = (if ($filter | is-empty) { "integration" } else { $"integration ($filter)" }) [(step-record $label $"cargo ($args | str join ' ')" {|| run-checked "cargo" $args })] } +# Empty stdout is success only when cargo actually ran. A lockfile or registry +# error also leaves stdout empty and must fail the step. +def openssl-tree-ok [stdout: string, stderr: string, exit_code: int] { + let stdout_empty = (($stdout | str trim) | is-empty) + let no_match = ($stderr | str contains "did not match any packages") + ($exit_code == 0 and $stdout_empty) or $no_match +} + def lint-steps [] { [ (step-record "fmt" "cargo fmt --check" {|| @@ -194,12 +316,15 @@ def lint-steps [] { (step-record "clippy" "cargo clippy --all-targets -- -D warnings" {|| run-checked "cargo" ["clippy" "--all-targets" "--" "-D" "warnings"] }) - # An empty tree is the only acceptable answer; cargo itself exits non-zero - # when nothing matches, so the exit code is not the signal here. (step-record "openssl" "cargo tree -i openssl (must be empty)" {|| let found = (run-external "cargo" "tree" "-i" "openssl" | complete) - if not (($found.stdout | str trim) | is-empty) { - error make { msg: $"openssl is in the dependency tree:\n($found.stdout)" } + if not (openssl-tree-ok $found.stdout $found.stderr $found.exit_code) { + let detail = (if not (($found.stdout | str trim) | is-empty) { + $"openssl is in the dependency tree:\n($found.stdout)" + } else { + $"cargo tree failed (exit ($found.exit_code)): ($found.stderr | str trim)" + }) + error make { msg: $detail } } }) ] @@ -254,28 +379,46 @@ def run-step [s: record] { # Redirect straight to stdout so long builds and tests show progress; Nushell # raises on a non-zero exit, which the enclosing step records as a failure. def run-checked [cmd: string, args: list] { + # Self-check spawns the runner with this set to read the argv without the suite. + if ($env.SKYG_TEST_DRY_RUN? | default "") == "1" { + print $"DRY-ARGS: ([$cmd] | append $args | str join '|')" + return + } run-external $cmd ...$args o> /dev/stdout } +def help-footer [] { + [ + "In an interactive bash or zsh, `test` is the shell builtin, so `test unit` exits 0 without running the suite." + "Run `devenv shell -- test ` or `nu dev/test.nu `." + "An interactive `devenv shell` installs a bash function for known subcommands; `test -f` stays the builtin." + "zsh: after compinit, add `source <(nu dev/test.nu completions zsh)` to .zshrc. direnv cannot carry that function." + "Nushell: `use dev/test.nu *`." + ] | str join "\n" +} + def print-help [] { print "Run the repository's checks." print "" - print "Usage: test [args]" + print "Usage: devenv shell -- test [args]" + print " nu dev/test.nu [args]" print "" $command_table | each {|c| {subcommand: $c.name, flags: ($c.flags | str join " "), what: $c.usage} } | table -e | print print "" - print "In Nushell: use dev/test.nu * (then `test integration peers`). In zsh/bash: source <(test completions zsh|bash)." - print "Underlying commands: test --help, or `test --help`." + print (help-footer) } def completion-script [shell: string] { let subcommands = ($command_table | get name | str join " ") + let pattern = (($command_table | get name) | append ["--help" "-h"] | str join "|") let flags = ($command_table | get flags | flatten | uniq | str join " ") let replacements = { "__SUBCOMMANDS__": $subcommands + "__PATTERN__": $pattern "__FLAGS__": $flags "__VM__": (nu-complete test vm | str join " ") "__SHELLS__": (nu-complete test shells | str join " ") + "__SCRIPT__": $SCRIPT_PATH } let template = (match $shell { "zsh" => (zsh-template) @@ -288,31 +431,51 @@ def zsh-template [] { # A raw string cannot start with `#` in Nushell, so the template starts on the next line. r#' #compdef test -# Completion for the repository test runner: source <(test completions zsh) +# zsh completion for the repository test runner. +# `test` is a shell builtin: `test unit` exits 0 and runs nothing. This function +# sends known subcommands to the runner and leaves every other call to `builtin test`. +# Install after compinit (this hook cannot see zsh from devenv's bash, and direnv +# will not carry the function): +# source <(nu dev/test.nu completions zsh) _test_runner() { local -a subcommands subcommands=(__SUBCOMMANDS__) if (( CURRENT == 2 )); then - if [[ "${words[2]}" == -* ]]; then - compadd -- __FLAGS__ - else - _describe 'subcommand' subcommands - fi + _describe 'subcommand' subcommands return fi case "${words[2]}" in - vm) compadd -- __VM__ __FLAGS__ ;; + integration) compadd -- __FLAGS__ ;; + vm) compadd -- __VM__ ;; completions) compadd -- __SHELLS__ ;; - *) compadd -- __FLAGS__ ;; esac } -compdef _test_runner test +test() { + if [[ $# -eq 0 ]]; then + command nu "__SCRIPT__" + return + fi + case "$1" in + __PATTERN__) + command nu "__SCRIPT__" "$@" + ;; + *) + builtin test "$@" + ;; + esac +} +if (( $+functions[compdef] )); then + compdef _test_runner test +fi '# | str trim --left } def bash-template [] { r#' -# bash completion for the repository test runner: source <(test completions bash) +# bash completion for the repository test runner. +# `test` is a shell builtin: `test unit` exits 0 and runs nothing. This function +# sends known subcommands to the runner and leaves every other call to `builtin test`. +# source <(nu dev/test.nu completions bash) _test_runner() { local cur sub i cur="${COMP_WORDS[COMP_CWORD]}" @@ -328,16 +491,65 @@ _test_runner() { return 0 fi case "$sub" in - vm) COMPREPLY=( $(compgen -W "__VM__ __FLAGS__" -- "$cur") ) ;; + integration) COMPREPLY=( $(compgen -W "__FLAGS__" -- "$cur") ) ;; + vm) COMPREPLY=( $(compgen -W "__VM__" -- "$cur") ) ;; completions) COMPREPLY=( $(compgen -W "__SHELLS__" -- "$cur") ) ;; - *) COMPREPLY=( $(compgen -W "__FLAGS__" -- "$cur") ) ;; + *) COMPREPLY=() ;; esac return 0 } +test() { + if [ $# -eq 0 ]; then + command nu "__SCRIPT__" + return + fi + case "$1" in + __PATTERN__) + command nu "__SCRIPT__" "$@" + ;; + *) + builtin test "$@" + ;; + esac +} complete -F _test_runner test '# | str trim --left } +# Script-mode argv is one list. Swallowing a flag or a spare word made +# `test unit --nocapture` and `test integration a b` look successful. +def parse-script-args [args: list] { + if ($args | is-empty) or ($args | any {|a| ($a == "--help") or ($a == "-h") }) { + return {cmd: "help", positional: null, nocapture: false} + } + let sub = ($args | first) + if $sub not-in ($command_table | get name) { + error make { msg: $"unknown subcommand '($sub)'; run 'test --help' for the list" } + } + let rest = ($args | skip 1) + let allowed = ($command_table | where name == $sub | get 0.flags) + mut positionals = [] + mut nocapture = false + for a in $rest { + if ($a | str starts-with "-") { + if $a not-in $allowed { + error make { msg: $"unknown flag '($a)' for 'test ($sub)'" } + } + if $a == "--nocapture" { + $nocapture = true + } + } else { + $positionals = ($positionals | append $a) + } + } + let max_pos = (if $sub in ["integration" "vm" "completions"] { 1 } else { 0 }) + if ($positionals | length) > $max_pos { + let what = (if $max_pos == 0 { "no arguments" } else { "at most one argument" }) + error make { msg: $"test ($sub) takes ($what); got '($positionals | str join ' ')'" } + } + {cmd: $sub, positional: ($positionals | get 0?), nocapture: $nocapture} +} + # Compare the table with the commands Nushell actually parsed from this file and # with the help block above; every entry must agree in all three places. def check-consistency [] { @@ -380,5 +592,242 @@ def check-consistency [] { } } } + let footer = (help-footer) + if ($footer | str contains "source <(test completions") { + $problems = ($problems | append "help tells the shell to complete the builtin via 'test completions'") + } + if not ($footer | str contains "nu dev/test.nu completions") { + $problems = ($problems | append "help does not show 'nu dev/test.nu completions'") + } + $problems +} + +def check-integration-args [] { + mut problems = [] + let nocap = (integration-steps null true | first | get command) + if $nocap != "cargo test --test * -- --nocapture" { + $problems = ($problems | append $"integration --nocapture argv is '($nocap)'") + } + let both = (integration-steps "peers" true | first | get command) + if $both != "cargo test --test * -- peers --nocapture" { + $problems = ($problems | append $"integration filter+nocapture argv is '($both)'") + } + let plain = (integration-steps null false | first | get command) + if $plain != "cargo test --test *" { + $problems = ($problems | append $"integration argv is '($plain)'") + } + # The runner must emit that argv, not only build it in memory. + let child = (with-env { SKYG_TEST_DRY_RUN: "1" } { + run-external "nu" $SCRIPT_PATH "integration" "--nocapture" | complete + }) + let want = "DRY-ARGS: cargo|test|--test|*|--|--nocapture" + if $child.exit_code != 0 or not ($child.stdout | str contains $want) { + $problems = ($problems | append $"CLI integration --nocapture exit ($child.exit_code), wanted '($want)': ($child.stdout)\n($child.stderr)") + } + let filtered = (with-env { SKYG_TEST_DRY_RUN: "1" } { + run-external "nu" $SCRIPT_PATH "integration" "peers" "--nocapture" | complete + }) + let want_both = "DRY-ARGS: cargo|test|--test|*|--|peers|--nocapture" + if $filtered.exit_code != 0 or not ($filtered.stdout | str contains $want_both) { + $problems = ($problems | append $"CLI integration peers --nocapture exit ($filtered.exit_code), wanted '($want_both)'") + } + # Cargo itself must accept `--nocapture` with no filter. That is the bug. + let listed = (run-external "cargo" "test" "--test" "*" "--" "--nocapture" "--list" | complete) + if $listed.exit_code != 0 { + $problems = ($problems | append $"cargo rejected --nocapture after --: ($listed.stderr | str trim)") + } + $problems +} + +def check-dispatch [] { + mut problems = [] + let cases = [ + {args: ["unit" "--nocapture"], needle: "unknown flag"} + {args: ["integration" "a" "b"], needle: "at most one"} + {args: ["integration" "--bogus"], needle: "unknown flag"} + {args: ["lint" "-x"], needle: "unknown flag"} + {args: ["vm" "peers" "extra"], needle: "at most one"} + {args: ["all" "--nocapture"], needle: "unknown flag"} + {args: ["ci" "extra"], needle: "no arguments"} + ] + for c in $cases { + let got = (try { parse-script-args $c.args | ignore; "ok" } catch {|e| $e.msg }) + if not ($got | str contains $c.needle) { + $problems = ($problems | append $"parse ($c.args | str join ' ') -> '($got)', expected '($c.needle)'") + } + } + let ok = (parse-script-args ["integration" "peers" "--nocapture"]) + if $ok.cmd != "integration" or $ok.positional != "peers" or (not $ok.nocapture) { + $problems = ($problems | append $"parse integration peers --nocapture -> ($ok | to nuon)") + } + let bare = (parse-script-args ["integration" "--nocapture"]) + if (not $bare.nocapture) or (not ($bare.positional | is-empty)) { + $problems = ($problems | append $"parse integration --nocapture -> ($bare | to nuon)") + } + let cli = (run-external "nu" $SCRIPT_PATH "unit" "--nocapture" | complete) + if $cli.exit_code == 0 or not ($cli.stderr | str contains "unknown flag") { + $problems = ($problems | append $"CLI 'test unit --nocapture' exit ($cli.exit_code): ($cli.stderr | str trim) ($cli.stdout | str trim)") + } + let extra = (run-external "nu" $SCRIPT_PATH "integration" "a" "b" | complete) + if $extra.exit_code == 0 or not ($extra.stderr | str contains "at most one") { + $problems = ($problems | append $"CLI 'test integration a b' exit ($extra.exit_code): ($extra.stderr | str trim)") + } + $problems +} + +def check-filter-cache [] { + mut problems = [] + if not ($FILTER_CACHE | str starts-with "target/") { + $problems = ($problems | append "filter cache must live under target/") + } + let dir = ($nu.temp-dir | path join $"filter-cache-($nu.pid)") + rm -rf $dir + mkdir $dir + let counter = ($dir | path join "calls") + let cache = ($dir | path join "cache") + "0" | save -f $counter + let ran = (with-env { SKYG_TEST_LIST_STUB: $counter, SKYG_TEST_FILTER_CACHE: $cache } { + let first = (nu-complete test filters) + let second = (nu-complete test filters) + let n = (open --raw $counter | str trim | into int) + let key = (source-mtime-key) + let stored = (if ($cache | path exists) { open --raw $cache | lines | first } else { "" }) + # A stale key must list again. Rewriting the key is the mtime changing. + if ($cache | path exists) { + let names = (open --raw $cache | lines | skip 1) + (["stale"] | append $names | str join (char nl)) | save -f $cache + } + let third = (nu-complete test filters) + let n2 = (open --raw $counter | str trim | into int) + {first: $first, second: $second, third: $third, n: $n, n2: $n2, key: $key, stored: $stored} + }) + if $ran.n != 1 { + $problems = ($problems | append $"filter completer listed ($ran.n) times; a warm cache should list once") + } + if $ran.first != $ran.second or $ran.first != ["stub-test"] { + $problems = ($problems | append $"filter cache returned ($ran.first | to nuon) then ($ran.second | to nuon)") + } + if $ran.stored != $ran.key { + $problems = ($problems | append $"filter cache key is '($ran.stored)', source mtime key is '($ran.key)'") + } + if $ran.n2 != 2 { + $problems = ($problems | append $"stale filter cache listed ($ran.n2) times after a stale key") + } + rm -rf $dir + $problems +} + +def check-openssl-verdict [] { + mut problems = [] + let cases = [ + {out: "", err: "error: package ID specification `openssl` did not match any packages\n", code: 101, ok: true} + {out: "", err: "", code: 0, ok: true} + {out: "openssl v1.0.0\n", err: "", code: 0, ok: false} + {out: "", err: "error: failed to load source for a dependency", code: 101, ok: false} + ] + for c in $cases { + let got = (openssl-tree-ok $c.out $c.err $c.code) + if $got != $c.ok { + $problems = ($problems | append $"openssl-tree-ok exit ($c.code) = ($got), expected ($c.ok)") + } + } + $problems +} + +def check-kvm-step [] { + let built = (with-env { SKYG_TEST_ASSUME_NO_KVM: "1" } { + try { + let steps = (vm-steps "vm") + let msg = (try { + do ($steps | first | get body) + "no-error" + } catch {|e| $e.msg }) + {built: true, msg: $msg} + } catch {|e| + {built: false, msg: $e.msg} + } + }) + if not $built.built { + return [$"vm-steps aborted while building the step list: ($built.msg)"] + } + if not ($built.msg | str contains "/dev/kvm") { + return [$"missing kvm should fail the vm step, got: ($built.msg)"] + } + [] +} + +def check-shell-wrapper [] { + mut problems = [] + let dir = ($nu.temp-dir | path join $"runner-wrap-($nu.pid)") + rm -rf $dir + mkdir $dir + let fake = ($dir | path join "nu") + let nl = (char nl) + $"#!/bin/sh($nl)printf '%s\\n' \"\$*\"($nl)" | save -f $fake + run-external "chmod" "+x" $fake | complete | ignore + for shell in (nu-complete test shells) { + if (which $shell | is-empty) { + continue + } + let script = ($dir | path join $"completions.($shell)") + completion-script $shell | save -f $script + let probe = ($dir | path join $"probe.($shell)") + [ + $"source '($script)'" + "test unit" + $"test -f '($script)'" + "if [ $? -eq 0 ]; then echo POSIX:0; else echo POSIX:1; fi" + ] | str join (char nl) | save -f $probe + let out = (with-env { PATH: ([$dir] | append $env.PATH) } { + run-external $shell $probe | complete + }) + if $out.exit_code != 0 { + $problems = ($problems | append $"($shell) wrapper exited ($out.exit_code): ($out.stderr | str trim)") + continue + } + if not ($out.stdout | str contains $"($SCRIPT_PATH) unit") { + $problems = ($problems | append $"($shell) did not dispatch 'test unit': ($out.stdout)") + } + if ($out.stdout | str contains $"($SCRIPT_PATH) -f") { + $problems = ($problems | append $"($shell) sent POSIX test to the runner: ($out.stdout)") + } + if not ($out.stdout | str contains "POSIX:0") { + $problems = ($problems | append $"($shell) POSIX test -f failed: ($out.stdout) ($out.stderr)") + } + } + rm -rf $dir + $problems +} + +def check-docs [] { + mut problems = [] + for f in ["AGENTS.md" "README.md" "docs/consolidator.md" "devenv.nix"] { + let text = (open --raw $f) + if ($text | str contains "source <(test completions") { + $problems = ($problems | append $"($f) completes the shell builtin via 'test completions'") + } + } + for f in ["AGENTS.md" "README.md" "docs/consolidator.md"] { + let text = (open --raw $f) + if not ($text | str contains "nu dev/test.nu completions") { + $problems = ($problems | append $"($f) does not show 'nu dev/test.nu completions'") + } + if not ($text | str contains "builtin") { + $problems = ($problems | append $"($f) does not mention the test builtin") + } + } + let hook = (open --raw devenv.nix) + if not ($hook | str contains ">&2") { + $problems = ($problems | append "enterShell banner is not sent to stderr") + } + if not ($hook | str contains "completions bash") { + $problems = ($problems | append "enterShell does not install bash completion") + } + if ($hook | str contains "completions zsh") { + $problems = ($problems | append "enterShell tries to install zsh completion from bash") + } + if not (($hook | str contains "case $-") and ($hook | str contains "*i*")) { + $problems = ($problems | append "enterShell banner is not limited to interactive shells") + } $problems } diff --git a/devenv.nix b/devenv.nix index 819f0d4..8604b9e 100644 --- a/devenv.nix +++ b/devenv.nix @@ -22,22 +22,24 @@ scripts.run.exec = ''cargo run -- "$@"''; scripts.smoke.exec = "cargo test --test binary_smoke -- --nocapture"; scripts.token.exec = ''cargo run -- token "$@"''; - # `test unit`, `test integration`, `test lint`, `test vm`, `test all`, `test ci`. + # `devenv shell -- test unit|integration|lint|vm|all|ci`. scripts.test.exec = ''nu dev/test.nu "$@"''; enterShell = '' export CLAUDE_CONFIG_DIR="$(pwd)/.claude-account" - # Completion for the `test` runner in the shell that sourced this hook. - # Nushell loads the script itself and completes natively; zsh and bash get - # a generated script from the same command table. - if command -v nu >/dev/null 2>&1; then - if [ -n "''${ZSH_VERSION:-}" ]; then - source <(nu dev/test.nu completions zsh) - elif [ -n "''${BASH_VERSION:-}" ]; then - source <(nu dev/test.nu completions bash) - fi - echo "test runner: 'test --help'. Nushell: run 'use dev/test.nu *' for completion." - fi + # Interactive bash only. `devenv shell -- …` is non-interactive, so its + # stdout stays the command's own output, and the note goes to stderr. + # This hook cannot install zsh functions (direnv would not keep them + # either). zsh users source the completion script from .zshrc after + # compinit; see docs/consolidator.md. + case $- in + *i*) + if [ -n "''${BASH_VERSION:-}" ] && command -v nu >/dev/null 2>&1; then + source <(nu dev/test.nu completions bash) + echo "test runner: 'test ' in this shell ('test -f' stays the builtin). From outside: devenv shell -- test ." >&2 + fi + ;; + esac ''; } diff --git a/docs/consolidator.md b/docs/consolidator.md index e289a96..32c46f3 100644 --- a/docs/consolidator.md +++ b/docs/consolidator.md @@ -6,18 +6,24 @@ The peers acceptance test is a NixOS VM test. Three machines run the real `slinu devenv shell -- test vm peers # nix build .#tests.x86_64-linux.peers -L ``` -Against a tree without the consolidator the test stops at the first peers assertion; that is expected, the test is written ahead of the feature. `dev/test.nu` is the repository's test runner; `test --help` lists its subcommands and the command behind each. +Against a tree without the consolidator the test stops at the first peers assertion; that is expected, the test is written ahead of the feature. `dev/test.nu` is the repository's test runner. `devenv shell -- test --help` lists its subcommands and the command behind each. Inside an interactive bash or zsh, `test` is the shell builtin, so `test vm` exits 0 without running anything. Use `devenv shell -- test …` or `nu dev/test.nu …`. -In Nushell the runner is a module. Load it once, for example in `config.nu`, and `test unit`, `test vm peers`, ... complete natively: +In Nushell the runner is a module. Load it once, for example in `config.nu`, and `test unit`, `test vm peers`, ... complete natively. Integration-test name completion is cached in `target/skyg-test-filters.cache`, keyed on the newest mtime of the Rust sources and the Cargo manifests. ```nu use /home/you/src/skyg-linux-tracker/dev/test.nu * # absolute path to dev/test.nu ``` -zsh and bash get completion from the same command table; inside `devenv shell` this is already sourced for you: +zsh and bash completion comes from the same command table. The generated script defines a `test` function for known subcommands (`test unit`, `test vm peers`, ...) and leaves other invocations (`test -f`) to the builtin. An interactive `devenv shell` sources the bash script and prints a one-line note on stderr. That hook runs in bash, so it cannot install zsh functions, and direnv will not carry them either. zsh users add the following to `.zshrc` after `compinit`: ```sh -source <(test completions zsh) # or: source <(test completions bash) +source <(nu /home/you/src/skyg-linux-tracker/dev/test.nu completions zsh) ``` -`test self-check` fails if the command table, the defined subcommands and the help text drift apart. +bash, from the repo checkout: + +```sh +source <(nu dev/test.nu completions bash) +``` + +`nu dev/test.nu self-check` fails if the command table, the defined subcommands, the help text, the cargo argv, the filter cache or the shell wrapper drift apart. diff --git a/plan/phase-5-consolidator/ws-t-tests.md b/plan/phase-5-consolidator/ws-t-tests.md index 1087d72..c495021 100644 --- a/plan/phase-5-consolidator/ws-t-tests.md +++ b/plan/phase-5-consolidator/ws-t-tests.md @@ -4,7 +4,7 @@ Part of [Phase 5: Consolidator](README.md). Contracts: [contracts.md](contracts. | Field | Value | |---|---| -| Status | T1 to T7 done: `nix/tests/peers.nix` and the `test` runner are written; `peers.nix` fails at the first peers assertion until WS-E lands (by design) | +| Status | T1 to T7 done; review round 1 addressed. `peers.nix` fails at the first peers assertion until WS-E lands (by design) | | Owner | | | Depends on | none (uses the module's freeform `settings` for `[peers]` and a systemd override for `LoadCredential`, so it needs no module changes). F1 later swaps this for the first-class module options | | Blocks | nothing technically; it is the acceptance gate: goes green at the end of [WS-E](ws-e-telemetry.md) (with A, B, C, D landed), then F2 promotes it to `checks` | @@ -35,14 +35,14 @@ Note on the written test: `nix/tests/peers.nix` also contains a third node `gamm - **T6** no chaining and read-only: a request carrying `x-slinuxm-hop` is rejected; non-GET to a proxy path is 405; a TLS variant with a test CA via `ca_file`. **Done** (405 for POST/DELETE, hop header 4xx, `gamma` over HTTPS). - **T7** **Done**: `dev/test.nu` with `unit`, `integration`, `lint`, `vm`, `all`, `ci`, `completions`, `self-check`; devenv `test` script plus zsh/bash completion hook; AGENTS.md and README.md command lists; `docs/consolidator.md` `## Testing`. A simple test runner CLI written in **Nushell**: `dev/test.nu` (devenv adds `pkgs.nushell`; also exposed as the devenv script `test`, so `devenv shell -- test unit` works from any shell). One entry point so contributors and agents do not memorise cargo/nix incantations. Structure it as a Nushell script with a `main` and `main ` subcommands (`def "main unit"`, ...) so `--help` is generated and completion comes from the typed signatures: - `test unit` runs `cargo test --lib --bins` (no sockets, no host access). - - `test integration [filter]` runs `cargo test --tests` (plain, tls, integration, binary_smoke, `peers` once it exists). `filter` is passed to cargo as a name filter; `--nocapture` flag supported. + - `test integration [filter]` runs `cargo test --test '*'` (integration targets only: plain, tls, integration, binary_smoke, `peers` once it exists), so `test all` does not run the unit tests twice. `filter` is passed to cargo as a name filter. `--nocapture` is a libtest flag and is always placed after `--`. - `test lint` runs `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings` and the `cargo tree -i openssl` emptiness check. - - `test vm [name]` runs a NixOS VM test. `name` is one of `vm` (from `checks`) or `peers` (`nix build .#tests.x86_64-linux.peers -L`; moves to `checks` when green). Refuses without `/dev/kvm` and says why; prints that it is slow. + - `test vm [name]` runs a NixOS VM test. `name` is one of `vm` (from `checks`) or `peers` (`nix build .#tests.x86_64-linux.peers -L`; moves to `checks` when green). A missing `/dev/kvm` fails that step and says why, after earlier `ci` steps, and the summary table still prints. The step prints that the build is slow. - `test all` = lint, unit, integration, stopping at the first failure; `test ci` = `all` plus `vm`. No args prints help. - Each step prints a header and a final table (step, result, seconds); exit code non-zero on any failure (use `error make`/`exit 1`, never swallow cargo's exit status). Read-only against the host: no `/etc`, systemd or Docker changes (live-host checks stay manual, see Verification plan in the [README](README.md#verification-plan)). - **Autocomplete (must work, and is a done-criterion):** - - Nushell: subcommands and flags complete natively once the script is loaded as a module. Dynamic values use custom completers (`def "nu-complete test vm" [] { ["vm" "peers"] }` and `name: string@"nu-complete test vm"`; integration filters complete from `cargo test --tests -- --list` test names, cached for the session). The devenv `enterShell` prints/loads `use dev/test.nu *` for Nushell users, and `docs/consolidator.md` shows the one-line `use` for a permanent setup. - - Other shells (the repo owner's login shell is zsh): `test completions zsh|bash` prints a small static completion script (subcommands, `vm` names, flags) generated from the same command table in the script, so zsh/bash tab completion works through `devenv shell` too (`source <(test completions zsh)` in `enterShell`). + - Nushell: subcommands and flags complete natively once the script is loaded as a module. Dynamic values use custom completers (`def "nu-complete test vm" [] { ["vm" "peers"] }` and `name: string@"nu-complete test vm"`; integration filters complete from `cargo test --test '*' -- --list`, cached in `target/skyg-test-filters.cache` and keyed on the newest mtime of the Rust sources and Cargo manifests). A completer cannot update the caller's environment, so an env var does not survive Tab. `docs/consolidator.md` shows the one-line `use dev/test.nu *` for a permanent setup. + - Other shells (the repo owner's login shell is zsh): `nu dev/test.nu completions zsh|bash` prints a completion script generated from the same command table. `test` is a shell builtin, so in an interactive bash or zsh `test unit` exits 0 without running the suite. The script defines a `test` function that sends known subcommands to the runner and otherwise calls `builtin test`, which keeps `test -f` working. `devenv shell -- test …` does not need the function: it finds the devenv script on PATH. Interactive `devenv shell` sources the bash script and prints a one-line note on stderr. zsh is not started by that hook; after `compinit`, zsh users add `source <(nu dev/test.nu completions zsh)` to `.zshrc` themselves. direnv does not carry the function. - Verified by a self-test: `test --help` and `test vm --help` list every subcommand/flag, and a check in `dev/test.nu` (`test self-check`) fails if the completion table and the defined subcommands diverge. - AGENTS.md and README replace their scattered command lists with `test ...`, keeping the underlying commands visible in `--help`. @@ -65,4 +65,16 @@ The test builds under `nix build .#tests.x86_64-linux.peers` (or equivalent) and - 2026-10-19: Split from the single phase plan 2026-10-19. `nix/tests/peers.nix` already written (T1 to T6) and failing at the first peers assertion by design; `flake.nix` exposes it as `tests..peers`. T7 not started. Ownership change from the single plan: `tests/peers.rs` moved to WS-F; WS-A/B in-process tests use their own files to avoid collisions. - 2026-10-19 (WS-T): T7 done. `dev/test.nu` is a Nushell module: `use dev/test.nu *` gives `test unit|integration|lint|vm|all|ci` plus `completions` and `self-check`, and `nu dev/test.nu ` dispatches through `main` for the devenv `test` script. Subcommands are declared as `test ` (not `main `): with `use dev/test.nu *` Nushell binds the module's `main` to the file stem (`test`), and only commands literally named `test ...` complete and dispatch as `test unit`. Native subcommand help comes from the signatures; the entry help block lists the same table, and `test self-check` fails when the table, the defined commands, their flags or the help text diverge (mutation-tested: extra table row, extra command, extra flag). zsh/bash scripts are generated from the same table (`test completions zsh|bash`), sourced by `enterShell`. Verified: `test all` (fmt, clippy, openssl, unit, integration) green, `--ide-complete` returns subcommands, `vm`/filter completers, `bash -n`/`zsh -n` on the generated scripts. Not run here (no nix builds in this worktree): `nu dev/test.nu vm peers`; on a KVM host that is `nix build .#tests.x86_64-linux.peers -L`. -- Handoff 2026-10-19 (WS-T -> all): the runner is `dev/test.nu`; use `test lint`, `test unit`, `test integration [filter] [--nocapture]`, `test vm [vm|peers]`, `test all`, `test ci`. `nix/tests/peers.nix` is unchanged and still the acceptance gate. To WS-F: promote `tests..peers` to `checks.vm-peers` once `test vm peers` passes; `test integration` picks up `tests/peers*.rs` with no runner change. Caveat: per-subcommand `--help` is native in Nushell module mode; in script mode (`nu dev/test.nu vm --help`) Nushell prints the entry help, which lists every subcommand. +- Handoff 2026-10-19 (WS-T -> all): the runner is `dev/test.nu`; use `devenv shell -- test lint|unit|integration|vm|all|ci` or `nu dev/test.nu `. `nix/tests/peers.nix` is unchanged and still the acceptance gate. To WS-F: promote `tests..peers` to `checks.vm-peers` once `test vm peers` passes; `test integration` picks up `tests/peers*.rs` with no runner change. Caveat: per-subcommand `--help` is native in Nushell module mode; in script mode (`nu dev/test.nu vm --help`) the entry help lists every subcommand. +- 2026-10-19 (WS-T review round 1): Addressed every finding in `loop/T/review-1.md`. No contract change. Nothing outside the owned files. `TODO.md` is still unticked (not an owned file); the merger ticks T7. + 1. `test integration --nocapture` with no filter now runs `cargo test --test '*' -- --nocapture`. Cargo rejects a bare `--nocapture`. `self-check` compares the step command, spawns the runner in a dry-run, and runs `cargo test --test '*' -- --nocapture --list`. + 2. `test` stays the devenv script name (the plan's name, and `devenv shell -- test` already resolves it on PATH). Inside an interactive bash or zsh the builtin wins and `test unit` exits 0. The generated completion script defines a `test` function that dispatches known subcommands to this file and otherwise calls `builtin test`, so `test -f` still works. Interactive `devenv shell` sources the bash script. Docs tell zsh users to add `source <(nu dev/test.nu completions zsh)` themselves. `self-check` sources the bash and zsh scripts with a fake `nu` and checks both paths. Disagree with renaming: it would change the command every other stream already cites, and the PATH form the plan specified already works. + 3. Filter completion is `target/skyg-test-filters.cache`, first line a mtime key of `src/**/*.rs`, `tests/**/*.rs`, `Cargo.toml` and `Cargo.lock`. The old `$env.SKYG_TEST_FILTERS` assignment died with the completer. `self-check` proves a second lookup does not re-list and a stale key does. + 4. `enterShell` runs in devenv's bash, so the zsh branch never installed `compdef`. Removed. Docs say zsh users source the script from `.zshrc` after `compinit`. direnv will not carry the function. + 5. The enterShell note goes to stderr and only when `$-` contains `i`, so `devenv shell -- test completions zsh` stays pipe-clean. Checked: a non-interactive `bash -c` does not take the `*i*` branch. + 6. `test integration` is `cargo test --test '*'` (integration targets only). `test all` / `test ci` keep the separate `unit` step and no longer repeat it. Help text in `AGENTS.md` and `README.md` matches. + 7. The openssl step passes only when stdout is empty and cargo exits 0, or when stderr contains `did not match any packages`. Any other cargo failure fails the step. `self-check` covers a clean miss, an empty success, a hit, and a lockfile-style error. + 8. Missing `/dev/kvm` is an error inside the vm step body, not while the step list is built. `self-check` sets `SKYG_TEST_ASSUME_NO_KVM=1` and asserts `vm-steps` returns a step whose body reports `/dev/kvm`. + 9. Script mode is `--wrapped`, so flags reach our parser. Unknown flags and extra positionals are errors. `--nocapture` is valid only on `integration`. `self-check` covers the parser and the CLI forms `unit --nocapture` and `integration a b`. + 10. `TODO.md:28` T7 remains unticked on purpose. +- Handoff 2026-10-19 (WS-T review round 1 -> all): same subcommand names. Invoke as `devenv shell -- test ` or `nu dev/test.nu `. Interactive `devenv shell` installs the bash wrapper; zsh needs the `.zshrc` line after `compinit`. `nu dev/test.nu self-check` is the runner's test. Not run here (no nix builds in this worktree): `nu dev/test.nu vm peers`, which on a KVM host is `nix build .#tests.x86_64-linux.peers -L`. `nix/tests/peers.nix` is unchanged.