From 40cf1073f4c5bce3e2a5ba4edc4a5ba899707fbf Mon Sep 17 00:00:00 2001 From: ari Date: Mon, 19 Oct 2026 17:45:39 -0400 Subject: [PATCH] T: T7 nushell test runner, devenv test script, command lists, docs testing section --- AGENTS.md | 24 +- README.md | 16 +- dev/test.nu | 384 ++++++++++++++++++++++++ devenv.nix | 16 + docs/consolidator.md | 23 ++ plan/phase-5-consolidator/ws-t-tests.md | 6 +- 6 files changed, 452 insertions(+), 17 deletions(-) create mode 100755 dev/test.nu create mode 100644 docs/consolidator.md diff --git a/AGENTS.md b/AGENTS.md index 48a6ff1..c78c141 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -16,19 +16,23 @@ Canonical guidance for AI coding agents in this repo (no CLAUDE.md; this file is ## Commands -Enter the toolchain with `devenv shell` (Rust, pkg-config, libsystemd, otelcol). Without it you need a Rust toolchain, `pkg-config` and the libsystemd dev files. +Enter the toolchain with `devenv shell` (Rust, pkg-config, libsystemd, nushell, otelcol). Without it you need a Rust toolchain, `pkg-config` and the libsystemd dev files. + +All checks go through one runner, `dev/test.nu`, exposed as the `test` script inside `devenv shell`. `test --help` lists the underlying cargo and nix commands. ```sh -cargo build --release # target/release/slinuxm -cargo test # unit + integration tests (fake sources) -cargo clippy --all-targets -- -D warnings # must be clean -cargo fmt --check -devenv shell -- smoke # real binary smoke test (cargo test --test binary_smoke) -cargo tree -i openssl # must print nothing -nix build && nix flake check # package, module, VM test (slow; Nix changes only) +devenv shell -- test lint # cargo fmt --check, clippy -D warnings, cargo tree -i openssl (empty) +devenv shell -- test unit # cargo test --lib --bins +devenv shell -- test integration [filter] [--nocapture] # cargo test --tests (plain, tls, integration, binary_smoke, peers) +devenv shell -- test vm [vm|peers] # NixOS VM test (slow, needs /dev/kvm) +devenv shell -- test all # lint, unit, integration; stops at the first failure +devenv shell -- test ci # all plus the vm test +devenv shell -- smoke # real binary smoke test (cargo test --test binary_smoke -- --nocapture) +cargo build --release # target/release/slinuxm +nix build && nix flake check # package, module, VM test (slow; Nix changes only) ``` -`devenv up` runs a local OTLP collector (`dev/otelcol.yaml`). Do not run nix builds unless the task involves Nix files, and never while another agent is running the nix gate. +`devenv up` runs a local OTLP collector (`dev/otelcol.yaml`). Do not run nix builds unless the task involves Nix files, and never while another agent is running the nix gate. When editing `dev/test.nu`, run `nu dev/test.nu self-check`; it fails if the command table, the subcommands and the help text drift apart. ## Conventions @@ -59,7 +63,7 @@ nix build && nix flake check # package, module, VM test (slow; ## Definition of done -1. `cargo clippy --all-targets -- -D warnings`, `cargo test` and `cargo fmt --check` pass. +1. `devenv shell -- test all` (fmt, clippy, unit and integration tests) passes; that is `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings` and `cargo test`. 2. New behaviour has tests (fakes for handlers and tools; `tests/` for end to end). 3. Docs updated: `README.md`, `config.example.toml`, `ARCHITECTURE.md` as relevant. 4. `TODO.md` updated; any deferred work is in the Backlog with a reason. diff --git a/README.md b/README.md index 2d380ca..4475e6a 100644 --- a/README.md +++ b/README.md @@ -178,8 +178,16 @@ ldd /usr/local/bin/slinuxm # libc, libm, libgcc_s, libsystemd and its depende ## Testing +Checks run through one runner, `dev/test.nu`, which is also the `test` script inside `devenv shell`. `test --help` lists every subcommand and the cargo or nix command behind it. + ```sh -devenv shell -- smoke # same as: cargo test --test binary_smoke -- --nocapture +devenv shell -- test unit # cargo test --lib --bins +devenv shell -- test integration [filter] [--nocapture] # cargo test --tests +devenv shell -- test lint # cargo fmt --check, clippy -D warnings, no openssl +devenv shell -- test vm [vm|peers] # NixOS VM test (slow, needs /dev/kvm) +devenv shell -- test all # lint, unit, integration; stops at the first failure +devenv shell -- test ci # all plus the vm test +devenv shell -- smoke # same as: cargo test --test binary_smoke -- --nocapture ``` The smoke test runs the real built `slinuxm` binary against a throwaway SQLite database in a temp directory: it creates a token with the CLI, starts `serve` on a free loopback port, exercises `/healthz`, REST and MCP with and without the token, revokes the token, and checks a clean SIGTERM exit and a 0600 database. It needs no root and no network. Checks that depend on host data (systemd over D-Bus, the journal) are skipped with a printed reason when the source is unavailable; Docker is never required. @@ -187,9 +195,7 @@ The smoke test runs the real built `slinuxm` binary against a throwaway SQLite d ## Development ```sh -cargo test -cargo clippy --all-targets -- -D warnings -cargo fmt --check +devenv shell -- test all # fmt, clippy, unit and integration tests ``` -The integration tests in `tests/` start the real router on an ephemeral port with fake sources. Design notes live in [ARCHITECTURE.md](ARCHITECTURE.md); work tracking in [TODO.md](TODO.md); extended docs in [docs/](docs/README.md). +`test --help` shows the underlying commands (`cargo fmt --check`, `cargo clippy --all-targets -- -D warnings`, `cargo test`). In Nushell, `use dev/test.nu *` gives `test unit`, `test vm peers`, ... with tab completion; in zsh or bash, `source <(test completions zsh)` (or `bash`). The integration tests in `tests/` start the real router on an ephemeral port with fake sources. Design notes live in [ARCHITECTURE.md](ARCHITECTURE.md); work tracking in [TODO.md](TODO.md); extended docs in [docs/](docs/README.md). diff --git a/dev/test.nu b/dev/test.nu new file mode 100755 index 0000000..4de7e9c --- /dev/null +++ b/dev/test.nu @@ -0,0 +1,384 @@ +#!/usr/bin/env nu + +# The single source of truth for names and flags: drives dispatch, zsh/bash +# completion and the self-check. Nushell completion still comes from the +# signatures below; the self-check keeps both in step. +const command_table = [ + { name: "unit", usage: "cargo test --lib --bins", flags: [] } + { name: "integration", usage: "cargo test --tests [filter] [--nocapture]", flags: ["--nocapture"] } + { name: "lint", usage: "fmt, clippy and the openssl check", flags: [] } + { name: "vm", usage: "nix build of a NixOS VM test: vm or peers", flags: [] } + { name: "all", usage: "lint, unit, integration; stops at the first failure", flags: [] } + { name: "ci", usage: "all plus the vm test", flags: [] } + { name: "completions", usage: "print a zsh or bash completion script", flags: [] } + { name: "self-check", usage: "check that the table, commands and help agree", flags: [] } +] + +# One entry point for the repository's checks. +# +# Subcommands: +# unit cargo test --lib --bins +# integration cargo test --tests [filter] [--nocapture] +# lint cargo fmt --check, cargo clippy --all-targets -- -D warnings, cargo tree -i openssl (empty) +# vm nix build of a NixOS VM test: vm (checks) or peers (tests..peers) +# all lint, unit, integration; stops at the first failure +# ci all plus the vm test +# completions print a zsh or bash completion script +# self-check fail when the command table, the commands and the help text diverge +# +# Loaded as a module (`use dev/test.nu *`) this file provides `test` plus one +# command per subcommand, so Nushell completes names and flags natively. +# +# The checks are read-only against the host: no /etc, systemd or Docker changes. +export def main [ + ...args: string # subcommand and its arguments, as in `test integration peers` + --nocapture # forward to `test integration` +] { + if ($args | is-empty) { + print-help + return + } + let sub = ($args | first) + let rest = ($args | skip 1) + let positional = ($rest | where {|a| not ($a | str starts-with "-") }) + if $sub not-in ($command_table | get name) { + error make { msg: $"unknown subcommand '($sub)'; run 'test --help' for the list" } + } + match $sub { + "unit" => { test unit } + "integration" => { test integration ($positional | get 0?) --nocapture=$nocapture } + "lint" => { test lint } + "vm" => { test vm ($positional | get 0?) } + "all" => { test all } + "ci" => { test ci } + "completions" => { test completions ($positional | get 0?) } + "self-check" => { test self-check } + _ => { error make { msg: $"subcommand '($sub)' has no dispatch arm" } } + } +} + +# Run the unit tests: no sockets, no host access. +export def "test unit" [] { + run-steps (unit-steps) +} + +# Run the integration tests (plain, tls, integration, binary_smoke, peers). +export def "test integration" [ + filter?: string@"nu-complete test filters" # cargo name filter, e.g. a test name or file + --nocapture # do not capture test output +] { + run-steps (integration-steps $filter $nocapture) +} + +# Formatting, lints and the dependency check. +export def "test lint" [] { + run-steps (lint-steps) +} + +# Run a NixOS VM test. Slow: it builds and boots VMs. +export def "test vm" [ + name?: string@"nu-complete test vm" # vm (checks) or peers (tests..peers) +] { + let name = ($name | default "vm") + if $name not-in (nu-complete test vm) { + error make { msg: $"unknown VM test '($name)'; expected one of: (nu-complete test vm | str join ', ')" } + } + run-steps (vm-steps $name) +} + +# lint, unit and integration; stops at the first failure. +export def "test all" [] { + run-steps ((lint-steps) | append (unit-steps) | append (integration-steps null false)) +} + +# `test all` plus the VM test. +export def "test ci" [] { + run-steps ((lint-steps) | append (unit-steps) | append (integration-steps null false) | append (vm-steps "vm")) +} + +# Print a completion script for a shell that cannot load this module (zsh, bash). +export def "test completions" [ + shell?: string@"nu-complete test shells" # zsh or bash +] { + let shell = ($shell | default "zsh") + match $shell { + "zsh" => { print (completion-script zsh) } + "bash" => { print (completion-script bash) } + _ => { error make { msg: $"unknown shell '($shell)'; expected one of: (nu-complete test shells | str join ', ')" } } + } +} + +# Fail if the command table, the defined commands or the help text diverge. +export def "test self-check" [] { + let problems = (check-consistency) + if not ($problems | is-empty) { + print -e "test self-check failed:" + $problems | each {|p| print -e $" - ($p)" } | ignore + exit 1 + } + print "test self-check: command table, commands and help agree" +} + +def "nu-complete test vm" [] { + ["vm" "peers"] +} + +def "nu-complete test shells" [] { + ["zsh" "bash"] +} + +# Test names come from cargo; the first Tab compiles in debug mode and is slow, +# so the list is kept for the rest of the session. +def "nu-complete test filters" [] { + if ($env.SKYG_TEST_FILTERS? | is-not-empty) { + return $env.SKYG_TEST_FILTERS + } + let listed = (do -i { run-external "cargo" "test" "--tests" "--" "--list" | complete }) + let names = ($listed + | default {stdout: ""} + | get stdout + | lines + | where {|l| $l =~ ': (test|benchmark)$'} + | each {|l| $l | str replace -r ': (test|benchmark)$' '' | str replace -r '^.*: ' '' } + | uniq) + if not ($names | is-empty) { + $env.SKYG_TEST_FILTERS = $names + } + $names +} + +def step-record [label: string, command: string, body: closure] { + { label: $label, command: $command, body: $body } +} + +def unit-steps [] { + [(step-record "unit" "cargo test --lib --bins" {|| + run-checked "cargo" ["test" "--lib" "--bins"] + })] +} + +def require-kvm [] { + if not ("/dev/kvm" | path exists) { + error make { + msg: "/dev/kvm is missing: NixOS VM tests need hardware virtualisation. Run on a Linux host with KVM, or in CI with KVM exposed; there is no software fallback." + } + } +} + +def vm-steps [name: string] { + require-kvm + let attr = (match $name { + "peers" => "tests" + _ => "checks" + }) + let system = $"($nu.os-info.arch)-($nu.os-info.name)" + [(step-record $"vm ($name)" $"nix build .#($attr).($system).($name) -L" {|| + print "NixOS VM tests are slow: expect several minutes." + run-checked "nix" ["build" $".#($attr).($system).($name)" "-L"] + })] +} + +def integration-steps [filter: any, nocapture: bool] { + let filter_args = (if ($filter | is-empty) { [] } else { ["--", $filter] }) + let capture_args = (if $nocapture { ["--nocapture"] } else { [] }) + let args = (["test" "--tests"] | append $filter_args | append $capture_args) + let label = (if ($filter | is-empty) { "integration" } else { $"integration ($filter)" }) + [(step-record $label $"cargo ($args | str join ' ')" {|| run-checked "cargo" $args })] +} + +def lint-steps [] { + [ + (step-record "fmt" "cargo fmt --check" {|| + run-checked "cargo" ["fmt" "--check"] + }) + (step-record "clippy" "cargo clippy --all-targets -- -D warnings" {|| + run-checked "cargo" ["clippy" "--all-targets" "--" "-D" "warnings"] + }) + # An empty tree is the only acceptable answer; cargo itself exits non-zero + # when nothing matches, so the exit code is not the signal here. + (step-record "openssl" "cargo tree -i openssl (must be empty)" {|| + let found = (run-external "cargo" "tree" "-i" "openssl" | complete) + if not (($found.stdout | str trim) | is-empty) { + error make { msg: $"openssl is in the dependency tree:\n($found.stdout)" } + } + }) + ] +} + +# Run each step in order, stop at the first failure, always print the summary +# table and exit non-zero if anything failed. +def run-steps [steps: list] { + mut results = [] + for s in $steps { + let r = (run-step $s) + $results = ($results | append $r) + if $r.result != "ok" { break } + } + let width = ([26 ($results | each {|r| $r.step | str length } | math max)] | math max) + print "" + print "Summary:" + print ((pad "step" $width "left") + (pad "result" 8 "left") + (pad "seconds" 9 "right") + " command") + for r in $results { + let status = (pad $r.result 8 "left") + print ((pad $r.step $width "left") + (if $r.result == "ok" { $"(ansi green)($status)(ansi reset)" } else { $"(ansi red)($status)(ansi reset)" }) + (pad $"(($r.seconds))s" 9 "right") + $" ($r.command)") + } + print "" + if ($results | any {|r| $r.result != "ok" }) { + exit 1 + } +} + +def pad [text: string, width: int, alignment: string] { + $text | fill -a $alignment -w $width +} + +def run-step [s: record] { + print "" + print $"(ansi cyan)==> ($s.label)(ansi reset) ($s.command)" + let started = (date now) + let ok = (try { + do $s.body | ignore + true + } catch {|e| + print -e $"(ansi red)($s.label) failed: ($e.msg)(ansi reset)" + false + }) + { + step: $s.label + command: $s.command + result: (if $ok { "ok" } else { "FAILED" }) + seconds: ((((date now) - $started) / 1sec) | math round --precision 1) + } +} + +# Redirect straight to stdout so long builds and tests show progress; Nushell +# raises on a non-zero exit, which the enclosing step records as a failure. +def run-checked [cmd: string, args: list] { + run-external $cmd ...$args o> /dev/stdout +} + +def print-help [] { + print "Run the repository's checks." + print "" + print "Usage: test [args]" + print "" + $command_table | each {|c| {subcommand: $c.name, flags: ($c.flags | str join " "), what: $c.usage} } | table -e | print + print "" + print "In Nushell: use dev/test.nu * (then `test integration peers`). In zsh/bash: source <(test completions zsh|bash)." + print "Underlying commands: test --help, or `test --help`." +} + +def completion-script [shell: string] { + let subcommands = ($command_table | get name | str join " ") + let flags = ($command_table | get flags | flatten | uniq | str join " ") + let replacements = { + "__SUBCOMMANDS__": $subcommands + "__FLAGS__": $flags + "__VM__": (nu-complete test vm | str join " ") + "__SHELLS__": (nu-complete test shells | str join " ") + } + let template = (match $shell { + "zsh" => (zsh-template) + _ => (bash-template) + }) + $replacements | transpose key value | reduce --fold $template {|it, acc| $acc | str replace -a $it.key $it.value } +} + +def zsh-template [] { + # A raw string cannot start with `#` in Nushell, so the template starts on the next line. + r#' +#compdef test +# Completion for the repository test runner: source <(test completions zsh) +_test_runner() { + local -a subcommands + subcommands=(__SUBCOMMANDS__) + if (( CURRENT == 2 )); then + if [[ "${words[2]}" == -* ]]; then + compadd -- __FLAGS__ + else + _describe 'subcommand' subcommands + fi + return + fi + case "${words[2]}" in + vm) compadd -- __VM__ __FLAGS__ ;; + completions) compadd -- __SHELLS__ ;; + *) compadd -- __FLAGS__ ;; + esac +} +compdef _test_runner test +'# | str trim --left +} + +def bash-template [] { + r#' +# bash completion for the repository test runner: source <(test completions bash) +_test_runner() { + local cur sub i + cur="${COMP_WORDS[COMP_CWORD]}" + sub="" + for ((i=1; i/dev/null 2>&1; then + if [ -n "''${ZSH_VERSION:-}" ]; then + source <(nu dev/test.nu completions zsh) + elif [ -n "''${BASH_VERSION:-}" ]; then + source <(nu dev/test.nu completions bash) + fi + echo "test runner: 'test --help'. Nushell: run 'use dev/test.nu *' for completion." + fi ''; } diff --git a/docs/consolidator.md b/docs/consolidator.md new file mode 100644 index 0000000..e289a96 --- /dev/null +++ b/docs/consolidator.md @@ -0,0 +1,23 @@ +## Testing + +The peers acceptance test is a NixOS VM test. Three machines run the real `slinuxm`: `alpha` and `beta` consolidate each other over plain HTTP, and `alpha` also declares the TLS-only `gamma` with a private CA. It needs KVM and takes minutes. + +```sh +devenv shell -- test vm peers # nix build .#tests.x86_64-linux.peers -L +``` + +Against a tree without the consolidator the test stops at the first peers assertion; that is expected, the test is written ahead of the feature. `dev/test.nu` is the repository's test runner; `test --help` lists its subcommands and the command behind each. + +In Nushell the runner is a module. Load it once, for example in `config.nu`, and `test unit`, `test vm peers`, ... complete natively: + +```nu +use /home/you/src/skyg-linux-tracker/dev/test.nu * # absolute path to dev/test.nu +``` + +zsh and bash get completion from the same command table; inside `devenv shell` this is already sourced for you: + +```sh +source <(test completions zsh) # or: source <(test completions bash) +``` + +`test self-check` fails if the command table, the defined subcommands and the help text drift apart. diff --git a/plan/phase-5-consolidator/ws-t-tests.md b/plan/phase-5-consolidator/ws-t-tests.md index f160a5a..1087d72 100644 --- a/plan/phase-5-consolidator/ws-t-tests.md +++ b/plan/phase-5-consolidator/ws-t-tests.md @@ -4,7 +4,7 @@ Part of [Phase 5: Consolidator](README.md). Contracts: [contracts.md](contracts. | Field | Value | |---|---| -| Status | In progress: `nix/tests/peers.nix` is written; T7 runner and docs open | +| Status | T1 to T7 done: `nix/tests/peers.nix` and the `test` runner are written; `peers.nix` fails at the first peers assertion until WS-E lands (by design) | | Owner | | | Depends on | none (uses the module's freeform `settings` for `[peers]` and a systemd override for `LoadCredential`, so it needs no module changes). F1 later swaps this for the first-class module options | | Blocks | nothing technically; it is the acceptance gate: goes green at the end of [WS-E](ws-e-telemetry.md) (with A, B, C, D landed), then F2 promotes it to `checks` | @@ -33,7 +33,7 @@ Note on the written test: `nix/tests/peers.nix` also contains a third node `gamm - **T4** token rejected: revoke the token on `beta`; `alpha` reports server up, api up, `auth = rejected`, `peer.api.auth_ok = 0`, and proxy calls fail with 503 naming the peer, without becoming a 401 for the caller; restore with a new token file and no restart, status recovers. **Done** in the script except the `peer.api.auth_ok = 0` metric check (needs WS-E; add it to the script, or leave to F2 which may query the collector/metrics endpoint, note in Log). - **T5** outage: stop `beta`'s service: `alpha` flips server and api after `fail_threshold`, `alpha`'s own endpoints stay healthy, and recovery is detected; stop `beta` entirely (VM down) and check that `alpha`'s requests are bounded by timeouts. **Done** in the script. - **T6** no chaining and read-only: a request carrying `x-slinuxm-hop` is rejected; non-GET to a proxy path is 405; a TLS variant with a test CA via `ca_file`. **Done** (405 for POST/DELETE, hop header 4xx, `gamma` over HTTPS). -- **T7** a simple test runner CLI written in **Nushell**: `dev/test.nu` (devenv adds `pkgs.nushell`; also exposed as the devenv script `test`, so `devenv shell -- test unit` works from any shell). One entry point so contributors and agents do not memorise cargo/nix incantations. Structure it as a Nushell script with a `main` and `main ` subcommands (`def "main unit"`, ...) so `--help` is generated and completion comes from the typed signatures: +- **T7** **Done**: `dev/test.nu` with `unit`, `integration`, `lint`, `vm`, `all`, `ci`, `completions`, `self-check`; devenv `test` script plus zsh/bash completion hook; AGENTS.md and README.md command lists; `docs/consolidator.md` `## Testing`. A simple test runner CLI written in **Nushell**: `dev/test.nu` (devenv adds `pkgs.nushell`; also exposed as the devenv script `test`, so `devenv shell -- test unit` works from any shell). One entry point so contributors and agents do not memorise cargo/nix incantations. Structure it as a Nushell script with a `main` and `main ` subcommands (`def "main unit"`, ...) so `--help` is generated and completion comes from the typed signatures: - `test unit` runs `cargo test --lib --bins` (no sockets, no host access). - `test integration [filter]` runs `cargo test --tests` (plain, tls, integration, binary_smoke, `peers` once it exists). `filter` is passed to cargo as a name filter; `--nocapture` flag supported. - `test lint` runs `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings` and the `cargo tree -i openssl` emptiness check. @@ -64,3 +64,5 @@ The test builds under `nix build .#tests.x86_64-linux.peers` (or equivalent) and ## Log - 2026-10-19: Split from the single phase plan 2026-10-19. `nix/tests/peers.nix` already written (T1 to T6) and failing at the first peers assertion by design; `flake.nix` exposes it as `tests..peers`. T7 not started. Ownership change from the single plan: `tests/peers.rs` moved to WS-F; WS-A/B in-process tests use their own files to avoid collisions. +- 2026-10-19 (WS-T): T7 done. `dev/test.nu` is a Nushell module: `use dev/test.nu *` gives `test unit|integration|lint|vm|all|ci` plus `completions` and `self-check`, and `nu dev/test.nu ` dispatches through `main` for the devenv `test` script. Subcommands are declared as `test ` (not `main `): with `use dev/test.nu *` Nushell binds the module's `main` to the file stem (`test`), and only commands literally named `test ...` complete and dispatch as `test unit`. Native subcommand help comes from the signatures; the entry help block lists the same table, and `test self-check` fails when the table, the defined commands, their flags or the help text diverge (mutation-tested: extra table row, extra command, extra flag). zsh/bash scripts are generated from the same table (`test completions zsh|bash`), sourced by `enterShell`. Verified: `test all` (fmt, clippy, openssl, unit, integration) green, `--ide-complete` returns subcommands, `vm`/filter completers, `bash -n`/`zsh -n` on the generated scripts. Not run here (no nix builds in this worktree): `nu dev/test.nu vm peers`; on a KVM host that is `nix build .#tests.x86_64-linux.peers -L`. +- Handoff 2026-10-19 (WS-T -> all): the runner is `dev/test.nu`; use `test lint`, `test unit`, `test integration [filter] [--nocapture]`, `test vm [vm|peers]`, `test all`, `test ci`. `nix/tests/peers.nix` is unchanged and still the acceptance gate. To WS-F: promote `tests..peers` to `checks.vm-peers` once `test vm peers` passes; `test integration` picks up `tests/peers*.rs` with no runner change. Caveat: per-subcommand `--help` is native in Nushell module mode; in script mode (`nu dev/test.nu vm --help`) Nushell prints the entry help, which lists every subcommand.