T: review round 1 fix runner argv, filter cache, and the test builtin

This commit is contained in:
ari committed 2026-10-19 22:37:08 -04:00
1 parent 40cf1073f4
commit 6dade6239d
6 files changed
+562 -93

No files matched your search

+3 -3
View File
@@ -18,12 +18,12 @@ Canonical guidance for AI coding agents in this repo (no CLAUDE.md; this file is
Enter the toolchain with `devenv shell` (Rust, pkg-config, libsystemd, nushell, otelcol). Without it you need a Rust toolchain, `pkg-config` and the libsystemd dev files.
All checks go through one runner, `dev/test.nu`, exposed as the `test` script inside `devenv shell`. `test --help` lists the underlying cargo and nix commands.
All checks go through one runner, `dev/test.nu`. `devenv shell -- test …` runs the devenv script on PATH. In an interactive bash or zsh, `test` is the shell builtin, so `test unit` exits 0 without running the suite; use `devenv shell -- test unit` or `nu dev/test.nu unit`. An interactive `devenv shell` installs a bash function that runs known subcommands and leaves `test -f` to the builtin. zsh users add `source <(nu dev/test.nu completions zsh)` to `.zshrc` after `compinit`. `nu dev/test.nu --help` lists the underlying cargo and nix commands.
```sh
devenv shell -- test lint # cargo fmt --check, clippy -D warnings, cargo tree -i openssl (empty)
devenv shell -- test unit # cargo test --lib --bins
devenv shell -- test integration [filter] [--nocapture] # cargo test --tests (plain, tls, integration, binary_smoke, peers)
devenv shell -- test integration [filter] [--nocapture] # cargo test --test '*' (plain, tls, integration, binary_smoke, peers)
devenv shell -- test vm [vm|peers] # NixOS VM test (slow, needs /dev/kvm)
devenv shell -- test all # lint, unit, integration; stops at the first failure
devenv shell -- test ci # all plus the vm test
@@ -32,7 +32,7 @@ cargo build --release # target/release/slinuxm
nix build && nix flake check # package, module, VM test (slow; Nix changes only)
```
`devenv up` runs a local OTLP collector (`dev/otelcol.yaml`). Do not run nix builds unless the task involves Nix files, and never while another agent is running the nix gate. When editing `dev/test.nu`, run `nu dev/test.nu self-check`; it fails if the command table, the subcommands and the help text drift apart.
`devenv up` runs a local OTLP collector (`dev/otelcol.yaml`). Do not run nix builds unless the task involves Nix files, and never while another agent is running the nix gate. When editing `dev/test.nu`, run `nu dev/test.nu self-check`; it fails if the command table, the subcommands, the help text, the cargo argv or the shell wrapper drift apart.
## Conventions
+3 -3
View File
@@ -178,11 +178,11 @@ ldd /usr/local/bin/slinuxm # libc, libm, libgcc_s, libsystemd and its depende
## Testing
Checks run through one runner, `dev/test.nu`, which is also the `test` script inside `devenv shell`. `test --help` lists every subcommand and the cargo or nix command behind it.
Checks run through one runner, `dev/test.nu`. `devenv shell -- test …` runs it from any shell. In an interactive bash or zsh, `test` is the shell builtin, so `test unit` exits 0 without running the suite. `devenv shell -- test --help` lists every subcommand and the cargo or nix command behind it.
```sh
devenv shell -- test unit # cargo test --lib --bins
devenv shell -- test integration [filter] [--nocapture] # cargo test --tests
devenv shell -- test integration [filter] [--nocapture] # cargo test --test '*' (integration targets only)
devenv shell -- test lint # cargo fmt --check, clippy -D warnings, no openssl
devenv shell -- test vm [vm|peers] # NixOS VM test (slow, needs /dev/kvm)
devenv shell -- test all # lint, unit, integration; stops at the first failure
@@ -198,4 +198,4 @@ The smoke test runs the real built `slinuxm` binary against a throwaway SQLite d
devenv shell -- test all # fmt, clippy, unit and integration tests
```
`test --help` shows the underlying commands (`cargo fmt --check`, `cargo clippy --all-targets -- -D warnings`, `cargo test`). In Nushell, `use dev/test.nu *` gives `test unit`, `test vm peers`, ... with tab completion; in zsh or bash, `source <(test completions zsh)` (or `bash`). The integration tests in `tests/` start the real router on an ephemeral port with fake sources. Design notes live in [ARCHITECTURE.md](ARCHITECTURE.md); work tracking in [TODO.md](TODO.md); extended docs in [docs/](docs/README.md).
`devenv shell -- test --help` shows the underlying commands (`cargo fmt --check`, `cargo clippy --all-targets -- -D warnings`, `cargo test --lib --bins`, `cargo test --test '*'`). Nushell: `use dev/test.nu *` gives `test unit`, `test vm peers`, ... with tab completion. zsh users add `source <(nu dev/test.nu completions zsh)` to `.zshrc` after `compinit`. An interactive `devenv shell` installs the bash function; `test -f` stays the builtin. The integration tests in `tests/` start the real router on an ephemeral port with fake sources. Design notes live in [ARCHITECTURE.md](ARCHITECTURE.md); work tracking in [TODO.md](TODO.md); extended docs in [docs/](docs/README.md).
+513 -64
View File
@@ -5,7 +5,7 @@
# signatures below; the self-check keeps both in step.
const command_table = [
{ name: "unit", usage: "cargo test --lib --bins", flags: [] }
{ name: "integration", usage: "cargo test --tests [filter] [--nocapture]", flags: ["--nocapture"] }
{ name: "integration", usage: "cargo test --test '*' [filter] [--nocapture]", flags: ["--nocapture"] }
{ name: "lint", usage: "fmt, clippy and the openssl check", flags: [] }
{ name: "vm", usage: "nix build of a NixOS VM test: vm or peers", flags: [] }
{ name: "all", usage: "lint, unit, integration; stops at the first failure", flags: [] }
@@ -14,11 +14,17 @@ const command_table = [
{ name: "self-check", usage: "check that the table, commands and help agree", flags: [] }
]
# Baked into the generated shell function so completion works from any directory.
const SCRIPT_PATH = path self
# Completers cannot update the caller's environment, so the filter list is a file.
const FILTER_CACHE = "target/skyg-test-filters.cache"
# One entry point for the repository's checks.
#
# Subcommands:
# unit cargo test --lib --bins
# integration cargo test --tests [filter] [--nocapture]
# integration cargo test --test '*' [filter] [--nocapture]
# lint cargo fmt --check, cargo clippy --all-targets -- -D warnings, cargo tree -i openssl (empty)
# vm nix build of a NixOS VM test: vm (checks) or peers (tests.<system>.peers)
# all lint, unit, integration; stops at the first failure
@@ -30,30 +36,41 @@ const command_table = [
# command per subcommand, so Nushell completes names and flags natively.
#
# The checks are read-only against the host: no /etc, systemd or Docker changes.
export def main [
...args: string # subcommand and its arguments, as in `test integration peers`
--nocapture # forward to `test integration`
] {
if ($args | is-empty) {
print-help
return
}
let sub = ($args | first)
let rest = ($args | skip 1)
let positional = ($rest | where {|a| not ($a | str starts-with "-") })
if $sub not-in ($command_table | get name) {
error make { msg: $"unknown subcommand '($sub)'; run 'test --help' for the list" }
}
match $sub {
#
# Script mode (`nu dev/test.nu`) is `--wrapped` so flags reach the parser below.
# Nushell would otherwise accept `--nocapture` on every subcommand and reject
# the rest before `main` can explain them.
export def --wrapped main [...args: string] {
let parsed = (parse-script-args $args)
match $parsed.cmd {
"help" => { print-help }
"unit" => { test unit }
"integration" => { test integration ($positional | get 0?) --nocapture=$nocapture }
"integration" => {
if ($parsed.positional | is-empty) {
test integration --nocapture=$parsed.nocapture
} else {
test integration $parsed.positional --nocapture=$parsed.nocapture
}
}
"lint" => { test lint }
"vm" => { test vm ($positional | get 0?) }
"vm" => {
if ($parsed.positional | is-empty) {
test vm
} else {
test vm $parsed.positional
}
}
"all" => { test all }
"ci" => { test ci }
"completions" => { test completions ($positional | get 0?) }
"completions" => {
if ($parsed.positional | is-empty) {
test completions
} else {
test completions $parsed.positional
}
}
"self-check" => { test self-check }
_ => { error make { msg: $"subcommand '($sub)' has no dispatch arm" } }
_ => { error make { msg: $"subcommand '($parsed.cmd)' has no dispatch arm" } }
}
}
@@ -110,13 +127,22 @@ export def "test completions" [
# Fail if the command table, the defined commands or the help text diverge.
export def "test self-check" [] {
let problems = (check-consistency)
let problems = (
(check-consistency)
++ (check-integration-args)
++ (check-dispatch)
++ (check-filter-cache)
++ (check-openssl-verdict)
++ (check-kvm-step)
++ (check-shell-wrapper)
++ (check-docs)
)
if not ($problems | is-empty) {
print -e "test self-check failed:"
$problems | each {|p| print -e $" - ($p)" } | ignore
exit 1
}
print "test self-check: command table, commands and help agree"
print "test self-check: ok"
}
def "nu-complete test vm" [] {
@@ -127,26 +153,86 @@ def "nu-complete test shells" [] {
["zsh" "bash"]
}
# Test names come from cargo; the first Tab compiles in debug mode and is slow,
# so the list is kept for the rest of the session.
# `cargo test --test '*'` matches integration targets only, same as the integration step.
def "nu-complete test filters" [] {
if ($env.SKYG_TEST_FILTERS? | is-not-empty) {
return $env.SKYG_TEST_FILTERS
let key = (try { source-mtime-key } catch { "unknown" })
try {
cached-lines (filter-cache-path) $key {|| list-test-names }
} catch {
[]
}
let listed = (do -i { run-external "cargo" "test" "--tests" "--" "--list" | complete })
let names = ($listed
| default {stdout: ""}
| get stdout
| lines
| where {|l| $l =~ ': (test|benchmark)$'}
| each {|l| $l | str replace -r ': (test|benchmark)$' '' | str replace -r '^.*: ' '' }
| uniq)
}
def filter-cache-path [] {
let override = ($env.SKYG_TEST_FILTER_CACHE? | default "")
if ($override | is-empty) { $FILTER_CACHE } else { $override }
}
def source-mtime-key [] {
let rows = (ls src/**/*.rs tests/**/*.rs Cargo.toml Cargo.lock)
if ($rows | is-empty) {
return "none"
}
$rows | get modified | math max | format date "%s.%f"
}
# Null means a miss. An empty list is a hit that listed no tests.
def read-cached-lines [path: string, key: string] {
if not ($path | path exists) {
return null
}
let lines = (open --raw $path | lines | where {|l| not ($l | is-empty) })
if ($lines | is-empty) or (($lines | first) != $key) {
return null
}
$lines | skip 1
}
def write-cached-lines [path: string, key: string, names: list<string>] {
mkdir ($path | path dirname)
([$key] | append $names | str join (char nl)) | save -f $path
}
def cached-lines [path: string, key: string, produce: closure] {
let hit = (read-cached-lines $path $key)
if $hit != null {
return $hit
}
let names = (do $produce)
if not ($names | is-empty) {
$env.SKYG_TEST_FILTERS = $names
write-cached-lines $path $key $names
}
$names
}
def list-test-names [] {
# Self-check points this at a counter file to prove Tab does not re-list.
let stub = ($env.SKYG_TEST_LIST_STUB? | default "")
if not ($stub | is-empty) {
let n = (if ($stub | path exists) {
open --raw $stub | str trim | into int
} else {
0
})
($n + 1 | into string) | save -f $stub
return ["stub-test"]
}
let listed = (try {
run-external "cargo" "test" "--test" "*" "--" "--list" | complete
} catch {
{stdout: "", stderr: "", exit_code: 1}
})
if $listed.exit_code != 0 {
return []
}
$listed.stdout
| lines
| where {|l| $l =~ ': (test|benchmark)$' }
| each {|l| $l | str replace -r ': (test|benchmark)$' '' }
| uniq
| sort
}
def step-record [label: string, command: string, body: closure] {
{ label: $label, command: $command, body: $body }
}
@@ -157,35 +243,71 @@ def unit-steps [] {
})]
}
def require-kvm [] {
if not ("/dev/kvm" | path exists) {
error make {
msg: "/dev/kvm is missing: NixOS VM tests need hardware virtualisation. Run on a Linux host with KVM, or in CI with KVM exposed; there is no software fallback."
}
def kvm-available [] {
# Self-check sets this to prove a missing device fails inside the step.
if ($env.SKYG_TEST_ASSUME_NO_KVM? | default "") == "1" {
return false
}
"/dev/kvm" | path exists
}
def kvm-refusal [] {
if (kvm-available) {
null
} else {
"/dev/kvm is missing: NixOS VM tests need hardware virtualisation. Run on a Linux host with KVM, or in CI with KVM exposed; there is no software fallback."
}
}
def vm-steps [name: string] {
require-kvm
# The KVM check runs in the step body. Raising it here would abort `test ci`
# before lint, unit and integration, and the summary table would not print.
let attr = (match $name {
"peers" => "tests"
_ => "checks"
})
let system = $"($nu.os-info.arch)-($nu.os-info.name)"
[(step-record $"vm ($name)" $"nix build .#($attr).($system).($name) -L" {||
let why = (kvm-refusal)
if $why != null {
error make { msg: $why }
}
print "NixOS VM tests are slow: expect several minutes."
run-checked "nix" ["build" $".#($attr).($system).($name)" "-L"]
})]
}
# Libtest flags must follow `--`. A bare `--nocapture` is a cargo flag and cargo rejects it.
def integration-cargo-args [filter: any, nocapture: bool] {
mut tail = []
if not ($filter | is-empty) {
$tail = ($tail | append ($filter | into string))
}
if $nocapture {
$tail = ($tail | append "--nocapture")
}
let args = ["test" "--test" "*"]
if ($tail | is-empty) {
$args
} else {
$args | append "--" | append $tail
}
}
def integration-steps [filter: any, nocapture: bool] {
let filter_args = (if ($filter | is-empty) { [] } else { ["--", $filter] })
let capture_args = (if $nocapture { ["--nocapture"] } else { [] })
let args = (["test" "--tests"] | append $filter_args | append $capture_args)
let args = (integration-cargo-args $filter $nocapture)
let label = (if ($filter | is-empty) { "integration" } else { $"integration ($filter)" })
[(step-record $label $"cargo ($args | str join ' ')" {|| run-checked "cargo" $args })]
}
# Empty stdout is success only when cargo actually ran. A lockfile or registry
# error also leaves stdout empty and must fail the step.
def openssl-tree-ok [stdout: string, stderr: string, exit_code: int] {
let stdout_empty = (($stdout | str trim) | is-empty)
let no_match = ($stderr | str contains "did not match any packages")
($exit_code == 0 and $stdout_empty) or $no_match
}
def lint-steps [] {
[
(step-record "fmt" "cargo fmt --check" {||
@@ -194,12 +316,15 @@ def lint-steps [] {
(step-record "clippy" "cargo clippy --all-targets -- -D warnings" {||
run-checked "cargo" ["clippy" "--all-targets" "--" "-D" "warnings"]
})
# An empty tree is the only acceptable answer; cargo itself exits non-zero
# when nothing matches, so the exit code is not the signal here.
(step-record "openssl" "cargo tree -i openssl (must be empty)" {||
let found = (run-external "cargo" "tree" "-i" "openssl" | complete)
if not (($found.stdout | str trim) | is-empty) {
error make { msg: $"openssl is in the dependency tree:\n($found.stdout)" }
if not (openssl-tree-ok $found.stdout $found.stderr $found.exit_code) {
let detail = (if not (($found.stdout | str trim) | is-empty) {
$"openssl is in the dependency tree:\n($found.stdout)"
} else {
$"cargo tree failed (exit ($found.exit_code)): ($found.stderr | str trim)"
})
error make { msg: $detail }
}
})
]
@@ -254,28 +379,46 @@ def run-step [s: record] {
# Redirect straight to stdout so long builds and tests show progress; Nushell
# raises on a non-zero exit, which the enclosing step records as a failure.
def run-checked [cmd: string, args: list<string>] {
# Self-check spawns the runner with this set to read the argv without the suite.
if ($env.SKYG_TEST_DRY_RUN? | default "") == "1" {
print $"DRY-ARGS: ([$cmd] | append $args | str join '|')"
return
}
run-external $cmd ...$args o> /dev/stdout
}
def help-footer [] {
[
"In an interactive bash or zsh, `test` is the shell builtin, so `test unit` exits 0 without running the suite."
"Run `devenv shell -- test <subcommand>` or `nu dev/test.nu <subcommand>`."
"An interactive `devenv shell` installs a bash function for known subcommands; `test -f` stays the builtin."
"zsh: after compinit, add `source <(nu dev/test.nu completions zsh)` to .zshrc. direnv cannot carry that function."
"Nushell: `use dev/test.nu *`."
] | str join "\n"
}
def print-help [] {
print "Run the repository's checks."
print ""
print "Usage: test <subcommand> [args]"
print "Usage: devenv shell -- test <subcommand> [args]"
print " nu dev/test.nu <subcommand> [args]"
print ""
$command_table | each {|c| {subcommand: $c.name, flags: ($c.flags | str join " "), what: $c.usage} } | table -e | print
print ""
print "In Nushell: use dev/test.nu * (then `test integration peers`). In zsh/bash: source <(test completions zsh|bash)."
print "Underlying commands: test --help, or `test <subcommand> --help`."
print (help-footer)
}
def completion-script [shell: string] {
let subcommands = ($command_table | get name | str join " ")
let pattern = (($command_table | get name) | append ["--help" "-h"] | str join "|")
let flags = ($command_table | get flags | flatten | uniq | str join " ")
let replacements = {
"__SUBCOMMANDS__": $subcommands
"__PATTERN__": $pattern
"__FLAGS__": $flags
"__VM__": (nu-complete test vm | str join " ")
"__SHELLS__": (nu-complete test shells | str join " ")
"__SCRIPT__": $SCRIPT_PATH
}
let template = (match $shell {
"zsh" => (zsh-template)
@@ -288,31 +431,51 @@ def zsh-template [] {
# A raw string cannot start with `#` in Nushell, so the template starts on the next line.
r#'
#compdef test
# Completion for the repository test runner: source <(test completions zsh)
# zsh completion for the repository test runner.
# `test` is a shell builtin: `test unit` exits 0 and runs nothing. This function
# sends known subcommands to the runner and leaves every other call to `builtin test`.
# Install after compinit (this hook cannot see zsh from devenv's bash, and direnv
# will not carry the function):
# source <(nu dev/test.nu completions zsh)
_test_runner() {
local -a subcommands
subcommands=(__SUBCOMMANDS__)
if (( CURRENT == 2 )); then
if [[ "${words[2]}" == -* ]]; then
compadd -- __FLAGS__
else
_describe 'subcommand' subcommands
fi
_describe 'subcommand' subcommands
return
fi
case "${words[2]}" in
vm) compadd -- __VM__ __FLAGS__ ;;
integration) compadd -- __FLAGS__ ;;
vm) compadd -- __VM__ ;;
completions) compadd -- __SHELLS__ ;;
*) compadd -- __FLAGS__ ;;
esac
}
compdef _test_runner test
test() {
if [[ $# -eq 0 ]]; then
command nu "__SCRIPT__"
return
fi
case "$1" in
__PATTERN__)
command nu "__SCRIPT__" "$@"
;;
*)
builtin test "$@"
;;
esac
}
if (( $+functions[compdef] )); then
compdef _test_runner test
fi
'# | str trim --left
}
def bash-template [] {
r#'
# bash completion for the repository test runner: source <(test completions bash)
# bash completion for the repository test runner.
# `test` is a shell builtin: `test unit` exits 0 and runs nothing. This function
# sends known subcommands to the runner and leaves every other call to `builtin test`.
# source <(nu dev/test.nu completions bash)
_test_runner() {
local cur sub i
cur="${COMP_WORDS[COMP_CWORD]}"
@@ -328,16 +491,65 @@ _test_runner() {
return 0
fi
case "$sub" in
vm) COMPREPLY=( $(compgen -W "__VM__ __FLAGS__" -- "$cur") ) ;;
integration) COMPREPLY=( $(compgen -W "__FLAGS__" -- "$cur") ) ;;
vm) COMPREPLY=( $(compgen -W "__VM__" -- "$cur") ) ;;
completions) COMPREPLY=( $(compgen -W "__SHELLS__" -- "$cur") ) ;;
*) COMPREPLY=( $(compgen -W "__FLAGS__" -- "$cur") ) ;;
*) COMPREPLY=() ;;
esac
return 0
}
test() {
if [ $# -eq 0 ]; then
command nu "__SCRIPT__"
return
fi
case "$1" in
__PATTERN__)
command nu "__SCRIPT__" "$@"
;;
*)
builtin test "$@"
;;
esac
}
complete -F _test_runner test
'# | str trim --left
}
# Script-mode argv is one list. Swallowing a flag or a spare word made
# `test unit --nocapture` and `test integration a b` look successful.
def parse-script-args [args: list<string>] {
if ($args | is-empty) or ($args | any {|a| ($a == "--help") or ($a == "-h") }) {
return {cmd: "help", positional: null, nocapture: false}
}
let sub = ($args | first)
if $sub not-in ($command_table | get name) {
error make { msg: $"unknown subcommand '($sub)'; run 'test --help' for the list" }
}
let rest = ($args | skip 1)
let allowed = ($command_table | where name == $sub | get 0.flags)
mut positionals = []
mut nocapture = false
for a in $rest {
if ($a | str starts-with "-") {
if $a not-in $allowed {
error make { msg: $"unknown flag '($a)' for 'test ($sub)'" }
}
if $a == "--nocapture" {
$nocapture = true
}
} else {
$positionals = ($positionals | append $a)
}
}
let max_pos = (if $sub in ["integration" "vm" "completions"] { 1 } else { 0 })
if ($positionals | length) > $max_pos {
let what = (if $max_pos == 0 { "no arguments" } else { "at most one argument" })
error make { msg: $"test ($sub) takes ($what); got '($positionals | str join ' ')'" }
}
{cmd: $sub, positional: ($positionals | get 0?), nocapture: $nocapture}
}
# Compare the table with the commands Nushell actually parsed from this file and
# with the help block above; every entry must agree in all three places.
def check-consistency [] {
@@ -380,5 +592,242 @@ def check-consistency [] {
}
}
}
let footer = (help-footer)
if ($footer | str contains "source <(test completions") {
$problems = ($problems | append "help tells the shell to complete the builtin via 'test completions'")
}
if not ($footer | str contains "nu dev/test.nu completions") {
$problems = ($problems | append "help does not show 'nu dev/test.nu completions'")
}
$problems
}
def check-integration-args [] {
mut problems = []
let nocap = (integration-steps null true | first | get command)
if $nocap != "cargo test --test * -- --nocapture" {
$problems = ($problems | append $"integration --nocapture argv is '($nocap)'")
}
let both = (integration-steps "peers" true | first | get command)
if $both != "cargo test --test * -- peers --nocapture" {
$problems = ($problems | append $"integration filter+nocapture argv is '($both)'")
}
let plain = (integration-steps null false | first | get command)
if $plain != "cargo test --test *" {
$problems = ($problems | append $"integration argv is '($plain)'")
}
# The runner must emit that argv, not only build it in memory.
let child = (with-env { SKYG_TEST_DRY_RUN: "1" } {
run-external "nu" $SCRIPT_PATH "integration" "--nocapture" | complete
})
let want = "DRY-ARGS: cargo|test|--test|*|--|--nocapture"
if $child.exit_code != 0 or not ($child.stdout | str contains $want) {
$problems = ($problems | append $"CLI integration --nocapture exit ($child.exit_code), wanted '($want)': ($child.stdout)\n($child.stderr)")
}
let filtered = (with-env { SKYG_TEST_DRY_RUN: "1" } {
run-external "nu" $SCRIPT_PATH "integration" "peers" "--nocapture" | complete
})
let want_both = "DRY-ARGS: cargo|test|--test|*|--|peers|--nocapture"
if $filtered.exit_code != 0 or not ($filtered.stdout | str contains $want_both) {
$problems = ($problems | append $"CLI integration peers --nocapture exit ($filtered.exit_code), wanted '($want_both)'")
}
# Cargo itself must accept `--nocapture` with no filter. That is the bug.
let listed = (run-external "cargo" "test" "--test" "*" "--" "--nocapture" "--list" | complete)
if $listed.exit_code != 0 {
$problems = ($problems | append $"cargo rejected --nocapture after --: ($listed.stderr | str trim)")
}
$problems
}
def check-dispatch [] {
mut problems = []
let cases = [
{args: ["unit" "--nocapture"], needle: "unknown flag"}
{args: ["integration" "a" "b"], needle: "at most one"}
{args: ["integration" "--bogus"], needle: "unknown flag"}
{args: ["lint" "-x"], needle: "unknown flag"}
{args: ["vm" "peers" "extra"], needle: "at most one"}
{args: ["all" "--nocapture"], needle: "unknown flag"}
{args: ["ci" "extra"], needle: "no arguments"}
]
for c in $cases {
let got = (try { parse-script-args $c.args | ignore; "ok" } catch {|e| $e.msg })
if not ($got | str contains $c.needle) {
$problems = ($problems | append $"parse ($c.args | str join ' ') -> '($got)', expected '($c.needle)'")
}
}
let ok = (parse-script-args ["integration" "peers" "--nocapture"])
if $ok.cmd != "integration" or $ok.positional != "peers" or (not $ok.nocapture) {
$problems = ($problems | append $"parse integration peers --nocapture -> ($ok | to nuon)")
}
let bare = (parse-script-args ["integration" "--nocapture"])
if (not $bare.nocapture) or (not ($bare.positional | is-empty)) {
$problems = ($problems | append $"parse integration --nocapture -> ($bare | to nuon)")
}
let cli = (run-external "nu" $SCRIPT_PATH "unit" "--nocapture" | complete)
if $cli.exit_code == 0 or not ($cli.stderr | str contains "unknown flag") {
$problems = ($problems | append $"CLI 'test unit --nocapture' exit ($cli.exit_code): ($cli.stderr | str trim) ($cli.stdout | str trim)")
}
let extra = (run-external "nu" $SCRIPT_PATH "integration" "a" "b" | complete)
if $extra.exit_code == 0 or not ($extra.stderr | str contains "at most one") {
$problems = ($problems | append $"CLI 'test integration a b' exit ($extra.exit_code): ($extra.stderr | str trim)")
}
$problems
}
def check-filter-cache [] {
mut problems = []
if not ($FILTER_CACHE | str starts-with "target/") {
$problems = ($problems | append "filter cache must live under target/")
}
let dir = ($nu.temp-dir | path join $"filter-cache-($nu.pid)")
rm -rf $dir
mkdir $dir
let counter = ($dir | path join "calls")
let cache = ($dir | path join "cache")
"0" | save -f $counter
let ran = (with-env { SKYG_TEST_LIST_STUB: $counter, SKYG_TEST_FILTER_CACHE: $cache } {
let first = (nu-complete test filters)
let second = (nu-complete test filters)
let n = (open --raw $counter | str trim | into int)
let key = (source-mtime-key)
let stored = (if ($cache | path exists) { open --raw $cache | lines | first } else { "" })
# A stale key must list again. Rewriting the key is the mtime changing.
if ($cache | path exists) {
let names = (open --raw $cache | lines | skip 1)
(["stale"] | append $names | str join (char nl)) | save -f $cache
}
let third = (nu-complete test filters)
let n2 = (open --raw $counter | str trim | into int)
{first: $first, second: $second, third: $third, n: $n, n2: $n2, key: $key, stored: $stored}
})
if $ran.n != 1 {
$problems = ($problems | append $"filter completer listed ($ran.n) times; a warm cache should list once")
}
if $ran.first != $ran.second or $ran.first != ["stub-test"] {
$problems = ($problems | append $"filter cache returned ($ran.first | to nuon) then ($ran.second | to nuon)")
}
if $ran.stored != $ran.key {
$problems = ($problems | append $"filter cache key is '($ran.stored)', source mtime key is '($ran.key)'")
}
if $ran.n2 != 2 {
$problems = ($problems | append $"stale filter cache listed ($ran.n2) times after a stale key")
}
rm -rf $dir
$problems
}
def check-openssl-verdict [] {
mut problems = []
let cases = [
{out: "", err: "error: package ID specification `openssl` did not match any packages\n", code: 101, ok: true}
{out: "", err: "", code: 0, ok: true}
{out: "openssl v1.0.0\n", err: "", code: 0, ok: false}
{out: "", err: "error: failed to load source for a dependency", code: 101, ok: false}
]
for c in $cases {
let got = (openssl-tree-ok $c.out $c.err $c.code)
if $got != $c.ok {
$problems = ($problems | append $"openssl-tree-ok exit ($c.code) = ($got), expected ($c.ok)")
}
}
$problems
}
def check-kvm-step [] {
let built = (with-env { SKYG_TEST_ASSUME_NO_KVM: "1" } {
try {
let steps = (vm-steps "vm")
let msg = (try {
do ($steps | first | get body)
"no-error"
} catch {|e| $e.msg })
{built: true, msg: $msg}
} catch {|e|
{built: false, msg: $e.msg}
}
})
if not $built.built {
return [$"vm-steps aborted while building the step list: ($built.msg)"]
}
if not ($built.msg | str contains "/dev/kvm") {
return [$"missing kvm should fail the vm step, got: ($built.msg)"]
}
[]
}
def check-shell-wrapper [] {
mut problems = []
let dir = ($nu.temp-dir | path join $"runner-wrap-($nu.pid)")
rm -rf $dir
mkdir $dir
let fake = ($dir | path join "nu")
let nl = (char nl)
$"#!/bin/sh($nl)printf '%s\\n' \"\$*\"($nl)" | save -f $fake
run-external "chmod" "+x" $fake | complete | ignore
for shell in (nu-complete test shells) {
if (which $shell | is-empty) {
continue
}
let script = ($dir | path join $"completions.($shell)")
completion-script $shell | save -f $script
let probe = ($dir | path join $"probe.($shell)")
[
$"source '($script)'"
"test unit"
$"test -f '($script)'"
"if [ $? -eq 0 ]; then echo POSIX:0; else echo POSIX:1; fi"
] | str join (char nl) | save -f $probe
let out = (with-env { PATH: ([$dir] | append $env.PATH) } {
run-external $shell $probe | complete
})
if $out.exit_code != 0 {
$problems = ($problems | append $"($shell) wrapper exited ($out.exit_code): ($out.stderr | str trim)")
continue
}
if not ($out.stdout | str contains $"($SCRIPT_PATH) unit") {
$problems = ($problems | append $"($shell) did not dispatch 'test unit': ($out.stdout)")
}
if ($out.stdout | str contains $"($SCRIPT_PATH) -f") {
$problems = ($problems | append $"($shell) sent POSIX test to the runner: ($out.stdout)")
}
if not ($out.stdout | str contains "POSIX:0") {
$problems = ($problems | append $"($shell) POSIX test -f failed: ($out.stdout) ($out.stderr)")
}
}
rm -rf $dir
$problems
}
def check-docs [] {
mut problems = []
for f in ["AGENTS.md" "README.md" "docs/consolidator.md" "devenv.nix"] {
let text = (open --raw $f)
if ($text | str contains "source <(test completions") {
$problems = ($problems | append $"($f) completes the shell builtin via 'test completions'")
}
}
for f in ["AGENTS.md" "README.md" "docs/consolidator.md"] {
let text = (open --raw $f)
if not ($text | str contains "nu dev/test.nu completions") {
$problems = ($problems | append $"($f) does not show 'nu dev/test.nu completions'")
}
if not ($text | str contains "builtin") {
$problems = ($problems | append $"($f) does not mention the test builtin")
}
}
let hook = (open --raw devenv.nix)
if not ($hook | str contains ">&2") {
$problems = ($problems | append "enterShell banner is not sent to stderr")
}
if not ($hook | str contains "completions bash") {
$problems = ($problems | append "enterShell does not install bash completion")
}
if ($hook | str contains "completions zsh") {
$problems = ($problems | append "enterShell tries to install zsh completion from bash")
}
if not (($hook | str contains "case $-") and ($hook | str contains "*i*")) {
$problems = ($problems | append "enterShell banner is not limited to interactive shells")
}
$problems
}
+14 -12
View File
@@ -22,22 +22,24 @@
scripts.run.exec = ''cargo run -- "$@"'';
scripts.smoke.exec = "cargo test --test binary_smoke -- --nocapture";
scripts.token.exec = ''cargo run -- token "$@"'';
# `test unit`, `test integration`, `test lint`, `test vm`, `test all`, `test ci`.
# `devenv shell -- test unit|integration|lint|vm|all|ci`.
scripts.test.exec = ''nu dev/test.nu "$@"'';
enterShell = ''
export CLAUDE_CONFIG_DIR="$(pwd)/.claude-account"
# Completion for the `test` runner in the shell that sourced this hook.
# Nushell loads the script itself and completes natively; zsh and bash get
# a generated script from the same command table.
if command -v nu >/dev/null 2>&1; then
if [ -n "''${ZSH_VERSION:-}" ]; then
source <(nu dev/test.nu completions zsh)
elif [ -n "''${BASH_VERSION:-}" ]; then
source <(nu dev/test.nu completions bash)
fi
echo "test runner: 'test --help'. Nushell: run 'use dev/test.nu *' for completion."
fi
# Interactive bash only. `devenv shell -- …` is non-interactive, so its
# stdout stays the command's own output, and the note goes to stderr.
# This hook cannot install zsh functions (direnv would not keep them
# either). zsh users source the completion script from .zshrc after
# compinit; see docs/consolidator.md.
case $- in
*i*)
if [ -n "''${BASH_VERSION:-}" ] && command -v nu >/dev/null 2>&1; then
source <(nu dev/test.nu completions bash)
echo "test runner: 'test <subcommand>' in this shell ('test -f' stays the builtin). From outside: devenv shell -- test <subcommand>." >&2
fi
;;
esac
'';
}
+11 -5
View File
@@ -6,18 +6,24 @@ The peers acceptance test is a NixOS VM test. Three machines run the real `slinu
devenv shell -- test vm peers # nix build .#tests.x86_64-linux.peers -L
```
Against a tree without the consolidator the test stops at the first peers assertion; that is expected, the test is written ahead of the feature. `dev/test.nu` is the repository's test runner; `test --help` lists its subcommands and the command behind each.
Against a tree without the consolidator the test stops at the first peers assertion; that is expected, the test is written ahead of the feature. `dev/test.nu` is the repository's test runner. `devenv shell -- test --help` lists its subcommands and the command behind each. Inside an interactive bash or zsh, `test` is the shell builtin, so `test vm` exits 0 without running anything. Use `devenv shell -- test …` or `nu dev/test.nu …`.
In Nushell the runner is a module. Load it once, for example in `config.nu`, and `test unit`, `test vm peers`, ... complete natively:
In Nushell the runner is a module. Load it once, for example in `config.nu`, and `test unit`, `test vm peers`, ... complete natively. Integration-test name completion is cached in `target/skyg-test-filters.cache`, keyed on the newest mtime of the Rust sources and the Cargo manifests.
```nu
use /home/you/src/skyg-linux-tracker/dev/test.nu * # absolute path to dev/test.nu
```
zsh and bash get completion from the same command table; inside `devenv shell` this is already sourced for you:
zsh and bash completion comes from the same command table. The generated script defines a `test` function for known subcommands (`test unit`, `test vm peers`, ...) and leaves other invocations (`test -f`) to the builtin. An interactive `devenv shell` sources the bash script and prints a one-line note on stderr. That hook runs in bash, so it cannot install zsh functions, and direnv will not carry them either. zsh users add the following to `.zshrc` after `compinit`:
```sh
source <(test completions zsh) # or: source <(test completions bash)
source <(nu /home/you/src/skyg-linux-tracker/dev/test.nu completions zsh)
```
`test self-check` fails if the command table, the defined subcommands and the help text drift apart.
bash, from the repo checkout:
```sh
source <(nu dev/test.nu completions bash)
```
`nu dev/test.nu self-check` fails if the command table, the defined subcommands, the help text, the cargo argv, the filter cache or the shell wrapper drift apart.
+18 -6
View File
@@ -4,7 +4,7 @@ Part of [Phase 5: Consolidator](README.md). Contracts: [contracts.md](contracts.
| Field | Value |
|---|---|
| Status | T1 to T7 done: `nix/tests/peers.nix` and the `test` runner are written; `peers.nix` fails at the first peers assertion until WS-E lands (by design) |
| Status | T1 to T7 done; review round 1 addressed. `peers.nix` fails at the first peers assertion until WS-E lands (by design) |
| Owner | |
| Depends on | none (uses the module's freeform `settings` for `[peers]` and a systemd override for `LoadCredential`, so it needs no module changes). F1 later swaps this for the first-class module options |
| Blocks | nothing technically; it is the acceptance gate: goes green at the end of [WS-E](ws-e-telemetry.md) (with A, B, C, D landed), then F2 promotes it to `checks` |
@@ -35,14 +35,14 @@ Note on the written test: `nix/tests/peers.nix` also contains a third node `gamm
- **T6** no chaining and read-only: a request carrying `x-slinuxm-hop` is rejected; non-GET to a proxy path is 405; a TLS variant with a test CA via `ca_file`. **Done** (405 for POST/DELETE, hop header 4xx, `gamma` over HTTPS).
- **T7** **Done**: `dev/test.nu` with `unit`, `integration`, `lint`, `vm`, `all`, `ci`, `completions`, `self-check`; devenv `test` script plus zsh/bash completion hook; AGENTS.md and README.md command lists; `docs/consolidator.md` `## Testing`. A simple test runner CLI written in **Nushell**: `dev/test.nu` (devenv adds `pkgs.nushell`; also exposed as the devenv script `test`, so `devenv shell -- test unit` works from any shell). One entry point so contributors and agents do not memorise cargo/nix incantations. Structure it as a Nushell script with a `main` and `main <sub>` subcommands (`def "main unit"`, ...) so `--help` is generated and completion comes from the typed signatures:
- `test unit` runs `cargo test --lib --bins` (no sockets, no host access).
- `test integration [filter]` runs `cargo test --tests` (plain, tls, integration, binary_smoke, `peers` once it exists). `filter` is passed to cargo as a name filter; `--nocapture` flag supported.
- `test integration [filter]` runs `cargo test --test '*'` (integration targets only: plain, tls, integration, binary_smoke, `peers` once it exists), so `test all` does not run the unit tests twice. `filter` is passed to cargo as a name filter. `--nocapture` is a libtest flag and is always placed after `--`.
- `test lint` runs `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings` and the `cargo tree -i openssl` emptiness check.
- `test vm [name]` runs a NixOS VM test. `name` is one of `vm` (from `checks`) or `peers` (`nix build .#tests.x86_64-linux.peers -L`; moves to `checks` when green). Refuses without `/dev/kvm` and says why; prints that it is slow.
- `test vm [name]` runs a NixOS VM test. `name` is one of `vm` (from `checks`) or `peers` (`nix build .#tests.x86_64-linux.peers -L`; moves to `checks` when green). A missing `/dev/kvm` fails that step and says why, after earlier `ci` steps, and the summary table still prints. The step prints that the build is slow.
- `test all` = lint, unit, integration, stopping at the first failure; `test ci` = `all` plus `vm`. No args prints help.
- Each step prints a header and a final table (step, result, seconds); exit code non-zero on any failure (use `error make`/`exit 1`, never swallow cargo's exit status). Read-only against the host: no `/etc`, systemd or Docker changes (live-host checks stay manual, see Verification plan in the [README](README.md#verification-plan)).
- **Autocomplete (must work, and is a done-criterion):**
- Nushell: subcommands and flags complete natively once the script is loaded as a module. Dynamic values use custom completers (`def "nu-complete test vm" [] { ["vm" "peers"] }` and `name: string@"nu-complete test vm"`; integration filters complete from `cargo test --tests -- --list` test names, cached for the session). The devenv `enterShell` prints/loads `use dev/test.nu *` for Nushell users, and `docs/consolidator.md` shows the one-line `use` for a permanent setup.
- Other shells (the repo owner's login shell is zsh): `test completions zsh|bash` prints a small static completion script (subcommands, `vm` names, flags) generated from the same command table in the script, so zsh/bash tab completion works through `devenv shell` too (`source <(test completions zsh)` in `enterShell`).
- Nushell: subcommands and flags complete natively once the script is loaded as a module. Dynamic values use custom completers (`def "nu-complete test vm" [] { ["vm" "peers"] }` and `name: string@"nu-complete test vm"`; integration filters complete from `cargo test --test '*' -- --list`, cached in `target/skyg-test-filters.cache` and keyed on the newest mtime of the Rust sources and Cargo manifests). A completer cannot update the caller's environment, so an env var does not survive Tab. `docs/consolidator.md` shows the one-line `use dev/test.nu *` for a permanent setup.
- Other shells (the repo owner's login shell is zsh): `nu dev/test.nu completions zsh|bash` prints a completion script generated from the same command table. `test` is a shell builtin, so in an interactive bash or zsh `test unit` exits 0 without running the suite. The script defines a `test` function that sends known subcommands to the runner and otherwise calls `builtin test`, which keeps `test -f` working. `devenv shell -- test …` does not need the function: it finds the devenv script on PATH. Interactive `devenv shell` sources the bash script and prints a one-line note on stderr. zsh is not started by that hook; after `compinit`, zsh users add `source <(nu dev/test.nu completions zsh)` to `.zshrc` themselves. direnv does not carry the function.
- Verified by a self-test: `test --help` and `test vm --help` list every subcommand/flag, and a check in `dev/test.nu` (`test self-check`) fails if the completion table and the defined subcommands diverge.
- AGENTS.md and README replace their scattered command lists with `test ...`, keeping the underlying commands visible in `--help`.
@@ -65,4 +65,16 @@ The test builds under `nix build .#tests.x86_64-linux.peers` (or equivalent) and
- 2026-10-19: Split from the single phase plan 2026-10-19. `nix/tests/peers.nix` already written (T1 to T6) and failing at the first peers assertion by design; `flake.nix` exposes it as `tests.<system>.peers`. T7 not started. Ownership change from the single plan: `tests/peers.rs` moved to WS-F; WS-A/B in-process tests use their own files to avoid collisions.
- 2026-10-19 (WS-T): T7 done. `dev/test.nu` is a Nushell module: `use dev/test.nu *` gives `test unit|integration|lint|vm|all|ci` plus `completions` and `self-check`, and `nu dev/test.nu <sub>` dispatches through `main` for the devenv `test` script. Subcommands are declared as `test <sub>` (not `main <sub>`): with `use dev/test.nu *` Nushell binds the module's `main` to the file stem (`test`), and only commands literally named `test ...` complete and dispatch as `test unit`. Native subcommand help comes from the signatures; the entry help block lists the same table, and `test self-check` fails when the table, the defined commands, their flags or the help text diverge (mutation-tested: extra table row, extra command, extra flag). zsh/bash scripts are generated from the same table (`test completions zsh|bash`), sourced by `enterShell`. Verified: `test all` (fmt, clippy, openssl, unit, integration) green, `--ide-complete` returns subcommands, `vm`/filter completers, `bash -n`/`zsh -n` on the generated scripts. Not run here (no nix builds in this worktree): `nu dev/test.nu vm peers`; on a KVM host that is `nix build .#tests.x86_64-linux.peers -L`.
- Handoff 2026-10-19 (WS-T -> all): the runner is `dev/test.nu`; use `test lint`, `test unit`, `test integration [filter] [--nocapture]`, `test vm [vm|peers]`, `test all`, `test ci`. `nix/tests/peers.nix` is unchanged and still the acceptance gate. To WS-F: promote `tests.<system>.peers` to `checks.vm-peers` once `test vm peers` passes; `test integration` picks up `tests/peers*.rs` with no runner change. Caveat: per-subcommand `--help` is native in Nushell module mode; in script mode (`nu dev/test.nu vm --help`) Nushell prints the entry help, which lists every subcommand.
- Handoff 2026-10-19 (WS-T -> all): the runner is `dev/test.nu`; use `devenv shell -- test lint|unit|integration|vm|all|ci` or `nu dev/test.nu <subcommand>`. `nix/tests/peers.nix` is unchanged and still the acceptance gate. To WS-F: promote `tests.<system>.peers` to `checks.vm-peers` once `test vm peers` passes; `test integration` picks up `tests/peers*.rs` with no runner change. Caveat: per-subcommand `--help` is native in Nushell module mode; in script mode (`nu dev/test.nu vm --help`) the entry help lists every subcommand.
- 2026-10-19 (WS-T review round 1): Addressed every finding in `loop/T/review-1.md`. No contract change. Nothing outside the owned files. `TODO.md` is still unticked (not an owned file); the merger ticks T7.
1. `test integration --nocapture` with no filter now runs `cargo test --test '*' -- --nocapture`. Cargo rejects a bare `--nocapture`. `self-check` compares the step command, spawns the runner in a dry-run, and runs `cargo test --test '*' -- --nocapture --list`.
2. `test` stays the devenv script name (the plan's name, and `devenv shell -- test` already resolves it on PATH). Inside an interactive bash or zsh the builtin wins and `test unit` exits 0. The generated completion script defines a `test` function that dispatches known subcommands to this file and otherwise calls `builtin test`, so `test -f` still works. Interactive `devenv shell` sources the bash script. Docs tell zsh users to add `source <(nu dev/test.nu completions zsh)` themselves. `self-check` sources the bash and zsh scripts with a fake `nu` and checks both paths. Disagree with renaming: it would change the command every other stream already cites, and the PATH form the plan specified already works.
3. Filter completion is `target/skyg-test-filters.cache`, first line a mtime key of `src/**/*.rs`, `tests/**/*.rs`, `Cargo.toml` and `Cargo.lock`. The old `$env.SKYG_TEST_FILTERS` assignment died with the completer. `self-check` proves a second lookup does not re-list and a stale key does.
4. `enterShell` runs in devenv's bash, so the zsh branch never installed `compdef`. Removed. Docs say zsh users source the script from `.zshrc` after `compinit`. direnv will not carry the function.
5. The enterShell note goes to stderr and only when `$-` contains `i`, so `devenv shell -- test completions zsh` stays pipe-clean. Checked: a non-interactive `bash -c` does not take the `*i*` branch.
6. `test integration` is `cargo test --test '*'` (integration targets only). `test all` / `test ci` keep the separate `unit` step and no longer repeat it. Help text in `AGENTS.md` and `README.md` matches.
7. The openssl step passes only when stdout is empty and cargo exits 0, or when stderr contains `did not match any packages`. Any other cargo failure fails the step. `self-check` covers a clean miss, an empty success, a hit, and a lockfile-style error.
8. Missing `/dev/kvm` is an error inside the vm step body, not while the step list is built. `self-check` sets `SKYG_TEST_ASSUME_NO_KVM=1` and asserts `vm-steps` returns a step whose body reports `/dev/kvm`.
9. Script mode is `--wrapped`, so flags reach our parser. Unknown flags and extra positionals are errors. `--nocapture` is valid only on `integration`. `self-check` covers the parser and the CLI forms `unit --nocapture` and `integration a b`.
10. `TODO.md:28` T7 remains unticked on purpose.
- Handoff 2026-10-19 (WS-T review round 1 -> all): same subcommand names. Invoke as `devenv shell -- test <subcommand>` or `nu dev/test.nu <subcommand>`. Interactive `devenv shell` installs the bash wrapper; zsh needs the `.zshrc` line after `compinit`. `nu dev/test.nu self-check` is the runner's test. Not run here (no nix builds in this worktree): `nu dev/test.nu vm peers`, which on a KVM host is `nix build .#tests.x86_64-linux.peers -L`. `nix/tests/peers.nix` is unchanged.