Files
skyg-linux-tracker/plan/phase-5-consolidator/ws-t-tests.md
T
2026-10-19 22:37:08 -04:00

18 KiB

WS-T: Integration test suite (written first, user request)

Part of Phase 5: Consolidator. Contracts: contracts.md.

Field Value
Status T1 to T7 done; review round 1 addressed; runner renamed to dev-test (dev/dev-test.nu). peers.nix fails at the first peers assertion until WS-E lands (by design)
Owner
Depends on none (uses the module's freeform settings for [peers] and a systemd override for LoadCredential, so it needs no module changes). F1 later swaps this for the first-class module options
Blocks nothing technically; it is the acceptance gate: goes green at the end of WS-E (with A, B, C, D landed), then F2 promotes it to checks
Can start when day one
Owned files nix/tests/peers.nix (done), flake.nix (the tests.<system>.peers output only), dev/dev-test.nu, devenv.nix (nushell package, dev-test script, completion hook), AGENTS.md and README.md (command lists only, see seams), docs/consolidator.md (creates the file with only a ## Testing section, see seams)
Do not touch src/** (all streams), nix/module.nix, nix/tests/vm.nix, systemd/skyg-linux-tracker.service (F), config.example.toml (A/F), tests/peers.rs and tests/common/mod.rs (F; in-process variants by A/B live in their own test files, see ws-a, ws-b), docs/api.md (C)

Shared-file seams:

  • flake.nix: T owns it until F2. F2 later moves exactly one attribute, tests.${system}.peers, into checks (rename checks.vm-peers). Nobody else edits flake.nix; it already carries the uncommitted peers output.
  • devenv.nix: T only.
  • README.md: T edits only the command/test list near the existing commands; F4 owns the new peers section. Different headings, so no overlap. AGENTS.md: T replaces the command list; F4 adds only the MCP tool-count note.
  • docs/consolidator.md: T creates it with a single ## Testing section (the one-line use dev/dev-test.nu * and the completion setup). F4 adds every other section and must not edit ## Testing.

Goal

Executable acceptance criteria before the implementation. Two NixOS VMs each run the real slinuxm binary and consolidate each other: VM alpha declares beta as a peer, VM beta declares alpha, and each proxies the other's data. Expected to fail until WS-E lands, which is the point.

Note on the written test: nix/tests/peers.nix also contains a third node gamma (TLS-only, no peers) that alpha declares with a private CA via ca_file, covering the TLS variant. It is already written and, by design, fails at the first peers assertion (phase 2, "declared but token is still a placeholder -> auth rejected") against the current binary; phase 1 (boot, healthz, token minting, marker units, token-file placeholders) already passes.

Tasks

  • T1 two machines alpha and beta with distinct hostnames, each with a distinct fake workload (a marker systemd unit marker-alpha.service / marker-beta.service, and a deliberately failing unit on one side) so proxied data is distinguishable from local data. Done in nix/tests/peers.nix (failing unit is broken-beta.service; gamma with marker-gamma.service added).
  • T2 each mints a read-only token via sudo -u skyg-linux-tracker slinuxm token create consolidator-<other>, writes the other's token to a root-only 0400 file (stand-in for an agenix path), and declares the other as a peer with that token_file. Done (files under /var/lib/peer-tokens, delivered via LoadCredential).
  • T3 assertions on both directions: /api/v1/peers shows the peer with server up, api up and auth ok; GET /api/v1/peers/<other>/systemd/units/marker-<other>.service returns the other machine's unit and not a local one; MCP list_peers and a list_systemd_units call with peer return the same data; response header x-slinuxm-peer. Done in the test script.
  • T4 token rejected: revoke the token on beta; alpha reports server up, api up, auth = rejected, peer.api.auth_ok = 0, and proxy calls fail with 503 naming the peer, without becoming a 401 for the caller; restore with a new token file and no restart, status recovers. Done in the script except the peer.api.auth_ok = 0 metric check (needs WS-E; add it to the script, or leave to F2 which may query the collector/metrics endpoint, note in Log).
  • T5 outage: stop beta's service: alpha flips server and api after fail_threshold, alpha's own endpoints stay healthy, and recovery is detected; stop beta entirely (VM down) and check that alpha's requests are bounded by timeouts. Done in the script.
  • T6 no chaining and read-only: a request carrying x-slinuxm-hop is rejected; non-GET to a proxy path is 405; a TLS variant with a test CA via ca_file. Done (405 for POST/DELETE, hop header 4xx, gamma over HTTPS).
  • T7 Done: dev/dev-test.nu with unit, integration, lint, vm, all, ci, completions, self-check; devenv dev-test script plus zsh/bash completion hook; AGENTS.md and README.md command lists; docs/consolidator.md ## Testing. A simple test runner CLI written in Nushell: dev/dev-test.nu (devenv adds pkgs.nushell; also exposed as the devenv script dev-test, so devenv shell -- dev-test unit works from any shell, including an interactive one). One entry point so contributors and agents do not memorise cargo/nix incantations. The command was renamed from test to dev-test (user decision, see Log) because test is a shell builtin. Structure it as a Nushell script with a main and subcommands literally named dev-test <sub> (export def "dev-test unit", ...) so --help is generated and completion comes from the typed signatures. Nushell binds main to the file stem, so only commands named dev-test ... complete as dev-test unit:
    • dev-test unit runs cargo test --lib --bins (no sockets, no host access).
    • dev-test integration [filter] runs cargo test --test '*' (integration targets only: plain, tls, integration, binary_smoke, peers once it exists), so dev-test all does not run the unit tests twice. filter is passed to cargo as a name filter. --nocapture is a libtest flag and is always placed after --.
    • dev-test lint runs cargo fmt --check, cargo clippy --all-targets -- -D warnings and the cargo tree -i openssl emptiness check.
    • dev-test vm [name] runs a NixOS VM test. name is one of vm (from checks) or peers (nix build .#tests.x86_64-linux.peers -L; moves to checks when green). A missing /dev/kvm fails that step and says why, after earlier ci steps, and the summary table still prints. The step prints that the build is slow.
    • dev-test all = lint, unit, integration, stopping at the first failure; dev-test ci = all plus vm. No args prints help.
    • Each step prints a header and a final table (step, result, seconds); exit code non-zero on any failure (use error make/exit 1, never swallow cargo's exit status). Read-only against the host: no /etc, systemd or Docker changes (live-host checks stay manual, see Verification plan in the README).
    • Autocomplete (must work, and is a done-criterion):
      • Nushell: subcommands and flags complete natively once the script is loaded as a module. Dynamic values use custom completers (def "nu-complete dev-test vm" [] { ["vm" "peers"] } and name: string@"nu-complete dev-test vm"; integration filters complete from cargo test --test '*' -- --list, cached in target/skyg-test-filters.cache and keyed on the newest mtime of the Rust sources and Cargo manifests). A completer cannot update the caller's environment, so an env var does not survive Tab. docs/consolidator.md shows the one-line use dev/dev-test.nu * for a permanent setup.
      • Other shells (the repo owner's login shell is zsh): nu dev/dev-test.nu completions zsh|bash prints a completion script generated from the same command table. dev-test is not a builtin, so the script only registers completion (compadd in zsh, compgen in bash) and does not define a function. Interactive dev-test … runs the devenv script on PATH. self-check invokes those completion functions and fails if the candidate list is empty. Interactive devenv shell sources the bash script and prints nothing, so devenv shell -- dev-test completions zsh stays pipe-clean. zsh is not started by that hook; after compinit, zsh users add source <(nu dev/dev-test.nu completions zsh) to .zshrc themselves. direnv does not source it for them.
      • Verified by a self-test: dev-test --help and dev-test vm --help list every subcommand/flag, and a check in dev/dev-test.nu (dev-test self-check) fails if the completion table and the defined subcommands diverge.
    • AGENTS.md and README replace their scattered command lists with dev-test ..., keeping the underlying commands visible in --help.

Inputs

  • contracts.md section 10 (test config contract) and sections 5 and 7 (REST/MCP shapes the script asserts). No fakes: the test runs the real binary.
  • nix/tests/peers.nix is the source of truth for section 10; if the test must change, change it and update contracts.md in the same PR, then log in the affected streams.

Done when

The test builds under nix build .#tests.x86_64-linux.peers (or equivalent) and evaluates, fails cleanly at the first peers assertion against the current binary, and is documented in docs/consolidator.md. dev-test unit|integration|lint|vm|all each run and report correctly on the current tree, and tab completion works in Nushell, zsh and bash (T7). It must pass at the end of WS-E, then move to checks.vm-peers (F2 does the move).

Handoff

  • To everyone: the dev-test runner and its subcommand names (referenced by AGENTS.md/README and by every stream's "Done when").
  • To WS-F: a passing peers.nix (ready to promote) and any changes made to its assumptions, recorded in contracts.md section 10.
  • To WS-A/B: the dev-test integration step picks up any tests/peers*.rs automatically.

Log

  • 2026-10-19: Split from the single phase plan 2026-10-19. nix/tests/peers.nix already written (T1 to T6) and failing at the first peers assertion by design; flake.nix exposes it as tests.<system>.peers. T7 not started. Ownership change from the single plan: tests/peers.rs moved to WS-F; WS-A/B in-process tests use their own files to avoid collisions.
  • 2026-10-19 (WS-T): T7 done. dev/test.nu is a Nushell module: use dev/test.nu * gives test unit|integration|lint|vm|all|ci plus completions and self-check, and nu dev/test.nu <sub> dispatches through main for the devenv test script. Subcommands are declared as test <sub> (not main <sub>): with use dev/test.nu * Nushell binds the module's main to the file stem (test), and only commands literally named test ... complete and dispatch as test unit. Native subcommand help comes from the signatures; the entry help block lists the same table, and test self-check fails when the table, the defined commands, their flags or the help text diverge (mutation-tested: extra table row, extra command, extra flag). zsh/bash scripts are generated from the same table (test completions zsh|bash), sourced by enterShell. Verified: test all (fmt, clippy, openssl, unit, integration) green, --ide-complete returns subcommands, vm/filter completers, bash -n/zsh -n on the generated scripts. Not run here (no nix builds in this worktree): nu dev/test.nu vm peers; on a KVM host that is nix build .#tests.x86_64-linux.peers -L.
  • Handoff 2026-10-19 (WS-T -> all): the runner is dev/test.nu; use devenv shell -- test lint|unit|integration|vm|all|ci or nu dev/test.nu <subcommand>. nix/tests/peers.nix is unchanged and still the acceptance gate. To WS-F: promote tests.<system>.peers to checks.vm-peers once test vm peers passes; test integration picks up tests/peers*.rs with no runner change. Caveat: per-subcommand --help is native in Nushell module mode; in script mode (nu dev/test.nu vm --help) the entry help lists every subcommand.
  • 2026-10-19 (WS-T review round 1): Addressed every finding in loop/T/review-1.md. No contract change. Nothing outside the owned files. TODO.md is still unticked (not an owned file); the merger ticks T7.
    1. test integration --nocapture with no filter now runs cargo test --test '*' -- --nocapture. Cargo rejects a bare --nocapture. self-check compares the step command, spawns the runner in a dry-run, and runs cargo test --test '*' -- --nocapture --list.
    2. test stays the devenv script name (the plan's name, and devenv shell -- test already resolves it on PATH). Inside an interactive bash or zsh the builtin wins and test unit exits 0. The generated completion script defines a test function that dispatches known subcommands to this file and otherwise calls builtin test, so test -f still works. Interactive devenv shell sources the bash script. Docs tell zsh users to add source <(nu dev/test.nu completions zsh) themselves. self-check sources the bash and zsh scripts with a fake nu and checks both paths. Disagree with renaming: it would change the command every other stream already cites, and the PATH form the plan specified already works.
    3. Filter completion is target/skyg-test-filters.cache, first line a mtime key of src/**/*.rs, tests/**/*.rs, Cargo.toml and Cargo.lock. The old $env.SKYG_TEST_FILTERS assignment died with the completer. self-check proves a second lookup does not re-list and a stale key does.
    4. enterShell runs in devenv's bash, so the zsh branch never installed compdef. Removed. Docs say zsh users source the script from .zshrc after compinit. direnv will not carry the function.
    5. The enterShell note goes to stderr and only when $- contains i, so devenv shell -- test completions zsh stays pipe-clean. Checked: a non-interactive bash -c does not take the *i* branch.
    6. test integration is cargo test --test '*' (integration targets only). test all / test ci keep the separate unit step and no longer repeat it. Help text in AGENTS.md and README.md matches.
    7. The openssl step passes only when stdout is empty and cargo exits 0, or when stderr contains did not match any packages. Any other cargo failure fails the step. self-check covers a clean miss, an empty success, a hit, and a lockfile-style error.
    8. Missing /dev/kvm is an error inside the vm step body, not while the step list is built. self-check sets SKYG_TEST_ASSUME_NO_KVM=1 and asserts vm-steps returns a step whose body reports /dev/kvm.
    9. Script mode is --wrapped, so flags reach our parser. Unknown flags and extra positionals are errors. --nocapture is valid only on integration. self-check covers the parser and the CLI forms unit --nocapture and integration a b.
    10. TODO.md:28 T7 remains unticked on purpose.
  • Handoff 2026-10-19 (WS-T review round 1 -> all): same subcommand names. Invoke as devenv shell -- test <subcommand> or nu dev/test.nu <subcommand>. Interactive devenv shell installs the bash wrapper; zsh needs the .zshrc line after compinit. nu dev/test.nu self-check is the runner's test. Not run here (no nix builds in this worktree): nu dev/test.nu vm peers, which on a KVM host is nix build .#tests.x86_64-linux.peers -L. nix/tests/peers.nix is unchanged.
  • 2026-10-19 (WS-T, user decision): Renamed the runner from test to dev-test and the module from dev/test.nu to dev/dev-test.nu. Supersedes the review-round-1 note that kept the name test. test is a shell builtin, so an interactive test unit never reached the runner and the completion script had to define a test function that fell through to builtin test. dev-test is an external command: devenv shell -- dev-test unit|integration|lint|vm|all|ci and an interactive dev-test … both run the script. Completion scripts only register completion (compadd / compgen); they do not define a function. _describe emitted no candidates outside a completion widget, so zsh now uses compadd and self-check fails if the bash or zsh candidate list is empty. Nushell commands are named dev-test <sub> because the file stem is dev-test. argv/--nocapture and the filter cache are unchanged. nix/tests/peers.nix is unchanged. Also fixed a latent help bug: the <(...) example sat in a double-quoted string, so Nushell ran the completions command and --help inlined the whole zsh script; the line is concatenated now and self-check rejects an embedded script. Earlier dated Log entries keep the old name because they record what happened then; the Status line, T7 text and Handoff use the current name. Stale dev/test.nu names left for the merger (other streams' files, not edited here):
    • plan/phase-5-consolidator/README.md workstream index
    • plan/phase-5-consolidator/ws-a-config.md Do not touch
    • plan/phase-5-consolidator/ws-f-nixos-docs.md Do not touch
    • TODO.md T7 line (unticked on purpose; the merger ticks it)
  • Handoff 2026-10-19 (WS-T rename -> all): invoke the runner as devenv shell -- dev-test <subcommand> or nu dev/dev-test.nu <subcommand>. Interactive bash completion is installed by devenv shell; zsh, after compinit: source <(nu dev/dev-test.nu completions zsh). Nushell: use dev/dev-test.nu *. nu dev/dev-test.nu self-check is the runner's test. Verified here: nu dev/dev-test.nu self-check and nu dev/dev-test.nu all (fmt, clippy, openssl, unit, integration) green, plus devenv shell -- dev-test --help. Not run here (no nix builds in this worktree): nu dev/dev-test.nu vm peers, which on a KVM host is nix build .#tests.x86_64-linux.peers -L. nix/tests/peers.nix is unchanged.