Files
skyg-linux-tracker/plan/phase-5-consolidator/ws-t-tests.md
T

15 KiB

WS-T: Integration test suite (written first, user request)

Part of Phase 5: Consolidator. Contracts: contracts.md.

Field Value
Status T1 to T7 done; review round 1 addressed. peers.nix fails at the first peers assertion until WS-E lands (by design)
Owner
Depends on none (uses the module's freeform settings for [peers] and a systemd override for LoadCredential, so it needs no module changes). F1 later swaps this for the first-class module options
Blocks nothing technically; it is the acceptance gate: goes green at the end of WS-E (with A, B, C, D landed), then F2 promotes it to checks
Can start when day one
Owned files nix/tests/peers.nix (done), flake.nix (the tests.<system>.peers output only), dev/test.nu, devenv.nix (nushell package, test script, completion hook), AGENTS.md and README.md (command lists only, see seams), docs/consolidator.md (creates the file with only a ## Testing section, see seams)
Do not touch src/** (all streams), nix/module.nix, nix/tests/vm.nix, systemd/skyg-linux-tracker.service (F), config.example.toml (A/F), tests/peers.rs and tests/common/mod.rs (F; in-process variants by A/B live in their own test files, see ws-a, ws-b), docs/api.md (C)

Shared-file seams:

  • flake.nix: T owns it until F2. F2 later moves exactly one attribute, tests.${system}.peers, into checks (rename checks.vm-peers). Nobody else edits flake.nix; it already carries the uncommitted peers output.
  • devenv.nix: T only.
  • README.md: T edits only the command/test list near the existing commands; F4 owns the new peers section. Different headings, so no overlap. AGENTS.md: T replaces the command list; F4 adds only the MCP tool-count note.
  • docs/consolidator.md: T creates it with a single ## Testing section (the one-line use dev/test.nu * and the completion setup). F4 adds every other section and must not edit ## Testing.

Goal

Executable acceptance criteria before the implementation. Two NixOS VMs each run the real slinuxm binary and consolidate each other: VM alpha declares beta as a peer, VM beta declares alpha, and each proxies the other's data. Expected to fail until WS-E lands, which is the point.

Note on the written test: nix/tests/peers.nix also contains a third node gamma (TLS-only, no peers) that alpha declares with a private CA via ca_file, covering the TLS variant. It is already written and, by design, fails at the first peers assertion (phase 2, "declared but token is still a placeholder -> auth rejected") against the current binary; phase 1 (boot, healthz, token minting, marker units, token-file placeholders) already passes.

Tasks

  • T1 two machines alpha and beta with distinct hostnames, each with a distinct fake workload (a marker systemd unit marker-alpha.service / marker-beta.service, and a deliberately failing unit on one side) so proxied data is distinguishable from local data. Done in nix/tests/peers.nix (failing unit is broken-beta.service; gamma with marker-gamma.service added).
  • T2 each mints a read-only token via sudo -u skyg-linux-tracker slinuxm token create consolidator-<other>, writes the other's token to a root-only 0400 file (stand-in for an agenix path), and declares the other as a peer with that token_file. Done (files under /var/lib/peer-tokens, delivered via LoadCredential).
  • T3 assertions on both directions: /api/v1/peers shows the peer with server up, api up and auth ok; GET /api/v1/peers/<other>/systemd/units/marker-<other>.service returns the other machine's unit and not a local one; MCP list_peers and a list_systemd_units call with peer return the same data; response header x-slinuxm-peer. Done in the test script.
  • T4 token rejected: revoke the token on beta; alpha reports server up, api up, auth = rejected, peer.api.auth_ok = 0, and proxy calls fail with 503 naming the peer, without becoming a 401 for the caller; restore with a new token file and no restart, status recovers. Done in the script except the peer.api.auth_ok = 0 metric check (needs WS-E; add it to the script, or leave to F2 which may query the collector/metrics endpoint, note in Log).
  • T5 outage: stop beta's service: alpha flips server and api after fail_threshold, alpha's own endpoints stay healthy, and recovery is detected; stop beta entirely (VM down) and check that alpha's requests are bounded by timeouts. Done in the script.
  • T6 no chaining and read-only: a request carrying x-slinuxm-hop is rejected; non-GET to a proxy path is 405; a TLS variant with a test CA via ca_file. Done (405 for POST/DELETE, hop header 4xx, gamma over HTTPS).
  • T7 Done: dev/test.nu with unit, integration, lint, vm, all, ci, completions, self-check; devenv test script plus zsh/bash completion hook; AGENTS.md and README.md command lists; docs/consolidator.md ## Testing. A simple test runner CLI written in Nushell: dev/test.nu (devenv adds pkgs.nushell; also exposed as the devenv script test, so devenv shell -- test unit works from any shell). One entry point so contributors and agents do not memorise cargo/nix incantations. Structure it as a Nushell script with a main and main <sub> subcommands (def "main unit", ...) so --help is generated and completion comes from the typed signatures:
    • test unit runs cargo test --lib --bins (no sockets, no host access).
    • test integration [filter] runs cargo test --test '*' (integration targets only: plain, tls, integration, binary_smoke, peers once it exists), so test all does not run the unit tests twice. filter is passed to cargo as a name filter. --nocapture is a libtest flag and is always placed after --.
    • test lint runs cargo fmt --check, cargo clippy --all-targets -- -D warnings and the cargo tree -i openssl emptiness check.
    • test vm [name] runs a NixOS VM test. name is one of vm (from checks) or peers (nix build .#tests.x86_64-linux.peers -L; moves to checks when green). A missing /dev/kvm fails that step and says why, after earlier ci steps, and the summary table still prints. The step prints that the build is slow.
    • test all = lint, unit, integration, stopping at the first failure; test ci = all plus vm. No args prints help.
    • Each step prints a header and a final table (step, result, seconds); exit code non-zero on any failure (use error make/exit 1, never swallow cargo's exit status). Read-only against the host: no /etc, systemd or Docker changes (live-host checks stay manual, see Verification plan in the README).
    • Autocomplete (must work, and is a done-criterion):
      • Nushell: subcommands and flags complete natively once the script is loaded as a module. Dynamic values use custom completers (def "nu-complete test vm" [] { ["vm" "peers"] } and name: string@"nu-complete test vm"; integration filters complete from cargo test --test '*' -- --list, cached in target/skyg-test-filters.cache and keyed on the newest mtime of the Rust sources and Cargo manifests). A completer cannot update the caller's environment, so an env var does not survive Tab. docs/consolidator.md shows the one-line use dev/test.nu * for a permanent setup.
      • Other shells (the repo owner's login shell is zsh): nu dev/test.nu completions zsh|bash prints a completion script generated from the same command table. test is a shell builtin, so in an interactive bash or zsh test unit exits 0 without running the suite. The script defines a test function that sends known subcommands to the runner and otherwise calls builtin test, which keeps test -f working. devenv shell -- test … does not need the function: it finds the devenv script on PATH. Interactive devenv shell sources the bash script and prints a one-line note on stderr. zsh is not started by that hook; after compinit, zsh users add source <(nu dev/test.nu completions zsh) to .zshrc themselves. direnv does not carry the function.
      • Verified by a self-test: test --help and test vm --help list every subcommand/flag, and a check in dev/test.nu (test self-check) fails if the completion table and the defined subcommands diverge.
    • AGENTS.md and README replace their scattered command lists with test ..., keeping the underlying commands visible in --help.

Inputs

  • contracts.md section 10 (test config contract) and sections 5 and 7 (REST/MCP shapes the script asserts). No fakes: the test runs the real binary.
  • nix/tests/peers.nix is the source of truth for section 10; if the test must change, change it and update contracts.md in the same PR, then log in the affected streams.

Done when

The test builds under nix build .#tests.x86_64-linux.peers (or equivalent) and evaluates, fails cleanly at the first peers assertion against the current binary, and is documented in docs/consolidator.md. test unit|integration|lint|vm|all each run and report correctly on the current tree, and tab completion works in Nushell, zsh and bash (T7). It must pass at the end of WS-E, then move to checks.vm-peers (F2 does the move).

Handoff

  • To everyone: the test runner and its subcommand names (referenced by AGENTS.md/README and by every stream's "Done when").
  • To WS-F: a passing peers.nix (ready to promote) and any changes made to its assumptions, recorded in contracts.md section 10.
  • To WS-A/B: the test integration step picks up any tests/peers*.rs automatically.

Log

  • 2026-10-19: Split from the single phase plan 2026-10-19. nix/tests/peers.nix already written (T1 to T6) and failing at the first peers assertion by design; flake.nix exposes it as tests.<system>.peers. T7 not started. Ownership change from the single plan: tests/peers.rs moved to WS-F; WS-A/B in-process tests use their own files to avoid collisions.
  • 2026-10-19 (WS-T): T7 done. dev/test.nu is a Nushell module: use dev/test.nu * gives test unit|integration|lint|vm|all|ci plus completions and self-check, and nu dev/test.nu <sub> dispatches through main for the devenv test script. Subcommands are declared as test <sub> (not main <sub>): with use dev/test.nu * Nushell binds the module's main to the file stem (test), and only commands literally named test ... complete and dispatch as test unit. Native subcommand help comes from the signatures; the entry help block lists the same table, and test self-check fails when the table, the defined commands, their flags or the help text diverge (mutation-tested: extra table row, extra command, extra flag). zsh/bash scripts are generated from the same table (test completions zsh|bash), sourced by enterShell. Verified: test all (fmt, clippy, openssl, unit, integration) green, --ide-complete returns subcommands, vm/filter completers, bash -n/zsh -n on the generated scripts. Not run here (no nix builds in this worktree): nu dev/test.nu vm peers; on a KVM host that is nix build .#tests.x86_64-linux.peers -L.
  • Handoff 2026-10-19 (WS-T -> all): the runner is dev/test.nu; use devenv shell -- test lint|unit|integration|vm|all|ci or nu dev/test.nu <subcommand>. nix/tests/peers.nix is unchanged and still the acceptance gate. To WS-F: promote tests.<system>.peers to checks.vm-peers once test vm peers passes; test integration picks up tests/peers*.rs with no runner change. Caveat: per-subcommand --help is native in Nushell module mode; in script mode (nu dev/test.nu vm --help) the entry help lists every subcommand.
  • 2026-10-19 (WS-T review round 1): Addressed every finding in loop/T/review-1.md. No contract change. Nothing outside the owned files. TODO.md is still unticked (not an owned file); the merger ticks T7.
    1. test integration --nocapture with no filter now runs cargo test --test '*' -- --nocapture. Cargo rejects a bare --nocapture. self-check compares the step command, spawns the runner in a dry-run, and runs cargo test --test '*' -- --nocapture --list.
    2. test stays the devenv script name (the plan's name, and devenv shell -- test already resolves it on PATH). Inside an interactive bash or zsh the builtin wins and test unit exits 0. The generated completion script defines a test function that dispatches known subcommands to this file and otherwise calls builtin test, so test -f still works. Interactive devenv shell sources the bash script. Docs tell zsh users to add source <(nu dev/test.nu completions zsh) themselves. self-check sources the bash and zsh scripts with a fake nu and checks both paths. Disagree with renaming: it would change the command every other stream already cites, and the PATH form the plan specified already works.
    3. Filter completion is target/skyg-test-filters.cache, first line a mtime key of src/**/*.rs, tests/**/*.rs, Cargo.toml and Cargo.lock. The old $env.SKYG_TEST_FILTERS assignment died with the completer. self-check proves a second lookup does not re-list and a stale key does.
    4. enterShell runs in devenv's bash, so the zsh branch never installed compdef. Removed. Docs say zsh users source the script from .zshrc after compinit. direnv will not carry the function.
    5. The enterShell note goes to stderr and only when $- contains i, so devenv shell -- test completions zsh stays pipe-clean. Checked: a non-interactive bash -c does not take the *i* branch.
    6. test integration is cargo test --test '*' (integration targets only). test all / test ci keep the separate unit step and no longer repeat it. Help text in AGENTS.md and README.md matches.
    7. The openssl step passes only when stdout is empty and cargo exits 0, or when stderr contains did not match any packages. Any other cargo failure fails the step. self-check covers a clean miss, an empty success, a hit, and a lockfile-style error.
    8. Missing /dev/kvm is an error inside the vm step body, not while the step list is built. self-check sets SKYG_TEST_ASSUME_NO_KVM=1 and asserts vm-steps returns a step whose body reports /dev/kvm.
    9. Script mode is --wrapped, so flags reach our parser. Unknown flags and extra positionals are errors. --nocapture is valid only on integration. self-check covers the parser and the CLI forms unit --nocapture and integration a b.
    10. TODO.md:28 T7 remains unticked on purpose.
  • Handoff 2026-10-19 (WS-T review round 1 -> all): same subcommand names. Invoke as devenv shell -- test <subcommand> or nu dev/test.nu <subcommand>. Interactive devenv shell installs the bash wrapper; zsh needs the .zshrc line after compinit. nu dev/test.nu self-check is the runner's test. Not run here (no nix builds in this worktree): nu dev/test.nu vm peers, which on a KVM host is nix build .#tests.x86_64-linux.peers -L. nix/tests/peers.nix is unchanged.