Files
2026-10-20 16:35:11 -04:00

23 KiB

WS-T: Integration test suite (written first, user request)

Part of Phase 5: Consolidator. Contracts: contracts.md.

Field Value
Status Merged
Owner
Depends on none (uses the module's freeform settings for [peers] and a systemd override for LoadCredential, so it needs no module changes). F1 later swaps this for the first-class module options
Blocks nothing technically; it is the acceptance gate: goes green at the end of WS-E (with A, B, C, D landed), then F2 promotes it to checks
Can start when day one
Owned files nix/tests/peers.nix (done), flake.nix (the tests.<system>.peers output only), dev/dev-test.nu, devenv.nix (nushell package, dev-test script, completion hook), AGENTS.md and README.md (command lists only, see seams), docs/consolidator.md (creates the file with only a ## Testing section, see seams)
Do not touch src/** (all streams), nix/module.nix, nix/tests/vm.nix, systemd/skyg-linux-tracker.service (F), config.example.toml (A/F), tests/peers.rs and tests/common/mod.rs (F; in-process variants by A/B live in their own test files, see ws-a, ws-b), docs/api.md (C)

Shared-file seams:

  • flake.nix: T owns it until F2. F2 later moves exactly one attribute, tests.${system}.peers, into checks (rename checks.vm-peers). Nobody else edits flake.nix; it already carries the uncommitted peers output.
  • devenv.nix: T only.
  • README.md: T edits only the command/test list near the existing commands; F4 owns the new peers section. Different headings, so no overlap. AGENTS.md: T replaces the command list; F4 adds only the MCP tool-count note.
  • docs/consolidator.md: T creates it with a single ## Testing section (the one-line use dev/dev-test.nu * and the completion setup). F4 adds every other section and must not edit ## Testing.

Goal

Executable acceptance criteria before the implementation. Two NixOS VMs each run the real slinuxm binary and consolidate each other: VM alpha declares beta as a peer, VM beta declares alpha, and each proxies the other's data. Expected to fail until WS-E lands, which is the point.

Note on the written test: nix/tests/peers.nix also contains a third node gamma (TLS-only, no peers) that alpha declares with a private CA via ca_file, covering the TLS variant. It is already written and, by design, fails at the first peers assertion (phase 2, "declared but token is still a placeholder -> auth rejected") against the current binary; phase 1 (boot, healthz, token minting, marker units, token-file placeholders) already passes.

Tasks

  • T1 two machines alpha and beta with distinct hostnames, each with a distinct fake workload (a marker systemd unit marker-alpha.service / marker-beta.service, and a deliberately failing unit on one side) so proxied data is distinguishable from local data. Done in nix/tests/peers.nix (failing unit is broken-beta.service; gamma with marker-gamma.service added).
  • T2 each mints a read-only token via sudo -u skyg-linux-tracker slinuxm token create consolidator-<other>, writes the other's token to a root-only 0400 file (stand-in for an agenix path), and declares the other as a peer with that token_file. Done (files under /var/lib/peer-tokens, delivered via LoadCredential).
  • T3 assertions on both directions: /api/v1/peers shows the peer with server up, api up and auth ok; GET /api/v1/peers/<other>/systemd/units/marker-<other>.service returns the other machine's unit and not a local one; MCP list_peers and a list_systemd_units call with peer return the same data; response header x-slinuxm-peer. Done in the test script.
  • T4 token rejected: revoke the token on beta; alpha reports server up, api up, auth = rejected, peer.api.auth_ok = 0, and proxy calls fail with 503 naming the peer, without becoming a 401 for the caller; restore with a new token file and no restart, status recovers. Done in the script except the peer.api.auth_ok = 0 metric check (needs WS-E; add it to the script, or leave to F2 which may query the collector/metrics endpoint, note in Log).
  • T5 outage: stop beta's service: alpha flips server and api after fail_threshold, alpha's own endpoints stay healthy, and recovery is detected; stop beta entirely (VM down) and check that alpha's requests are bounded by timeouts. Done in the script.
  • T6 no chaining and read-only: a request carrying x-slinuxm-hop is rejected; non-GET to a proxy path is 405; a TLS variant with a test CA via ca_file. Done (405 for POST/DELETE, hop header 4xx, gamma over HTTPS).
  • T7 Done: dev/dev-test.nu with unit, integration, lint, vm, all, ci, completions, self-check; devenv dev-test script plus zsh/bash completion hook; AGENTS.md and README.md command lists; docs/consolidator.md ## Testing. A simple test runner CLI written in Nushell: dev/dev-test.nu (devenv adds pkgs.nushell; also exposed as the devenv script dev-test, so devenv shell -- dev-test unit works from any shell, including an interactive one). One entry point so contributors and agents do not memorise cargo/nix incantations. The command was renamed from test to dev-test (user decision, see Log) because test is a shell builtin. Structure it as a Nushell script with a main and subcommands literally named dev-test <sub> (export def "dev-test unit", ...) so --help is generated and completion comes from the typed signatures. Nushell binds main to the file stem, so only commands named dev-test ... complete as dev-test unit:
    • dev-test unit runs cargo test --lib --bins (no sockets, no host access).
    • dev-test integration [filter] runs cargo test --test '*' (integration targets only: plain, tls, integration, binary_smoke, peers once it exists), so dev-test all does not run the unit tests twice. filter is passed to cargo as a name filter. --nocapture is a libtest flag and is always placed after --.
    • dev-test lint runs cargo fmt --check, cargo clippy --all-targets -- -D warnings and the cargo tree -i openssl emptiness check.
    • dev-test vm [name] runs a NixOS VM test. name is one of vm (from checks) or peers (nix build .#tests.x86_64-linux.peers -L; moves to checks when green). A missing /dev/kvm fails that step and says why, after earlier ci steps, and the summary table still prints. The step prints that the build is slow.
    • dev-test all = lint, unit, integration, stopping at the first failure; dev-test ci = all plus vm. No args prints help.
    • Each step prints a header and a final table (step, result, seconds); exit code non-zero on any failure (use error make/exit 1, never swallow cargo's exit status). Read-only against the host: no /etc, systemd or Docker changes (live-host checks stay manual, see Verification plan in the README).
    • Autocomplete (must work, and is a done-criterion):
      • Nushell: subcommands and flags complete natively once the script is loaded as a module. Dynamic values use custom completers (def "nu-complete dev-test vm" [] { ["vm" "peers"] } and name: string@"nu-complete dev-test vm"; integration filters complete from cargo test --test '*' -- --list, cached in target/skyg-test-filters.cache and keyed on the newest mtime of the Rust sources and Cargo manifests). A completer cannot update the caller's environment, so an env var does not survive Tab. docs/consolidator.md shows the one-line use dev/dev-test.nu * for a permanent setup.
      • Other shells (the repo owner's login shell is zsh): nu dev/dev-test.nu completions zsh|bash prints a completion script generated from the same command table. dev-test is not a builtin, so the script only registers completion (compadd in zsh, compgen in bash) and does not define a function. Interactive dev-test … runs the devenv script on PATH. self-check invokes those completion functions and fails if the candidate list is empty. Interactive devenv shell sources the bash script and prints nothing, so devenv shell -- dev-test completions zsh stays pipe-clean. zsh is not started by that hook; after compinit, zsh users add source <(nu dev/dev-test.nu completions zsh) to .zshrc themselves. direnv does not source it for them.
      • Verified by a self-test: dev-test --help and dev-test vm --help list every subcommand/flag, and a check in dev/dev-test.nu (dev-test self-check) fails if the completion table and the defined subcommands diverge.
    • AGENTS.md and README replace their scattered command lists with dev-test ..., keeping the underlying commands visible in --help.

Inputs

  • contracts.md section 10 (test config contract) and sections 5 and 7 (REST/MCP shapes the script asserts). No fakes: the test runs the real binary.
  • nix/tests/peers.nix is the source of truth for section 10; if the test must change, change it and update contracts.md in the same PR, then log in the affected streams.

Done when

The test builds under nix build .#tests.x86_64-linux.peers (or equivalent) and evaluates, fails cleanly at the first peers assertion against the current binary, and is documented in docs/consolidator.md. dev-test unit|integration|lint|vm|all each run and report correctly on the current tree, and tab completion works in Nushell, zsh and bash (T7). It must pass at the end of WS-E, then move to checks.vm-peers (F2 does the move).

Handoff

  • To everyone: the dev-test runner and its subcommand names (referenced by AGENTS.md/README and by every stream's "Done when").
  • To WS-F: a passing peers.nix (ready to promote) and any changes made to its assumptions, recorded in contracts.md section 10.
  • To WS-A/B: the dev-test integration step picks up any tests/peers*.rs automatically.

Log

  • 2026-10-19: Split from the single phase plan 2026-10-19. nix/tests/peers.nix already written (T1 to T6) and failing at the first peers assertion by design; flake.nix exposes it as tests.<system>.peers. T7 not started. Ownership change from the single plan: tests/peers.rs moved to WS-F; WS-A/B in-process tests use their own files to avoid collisions.
  • 2026-10-19 (WS-T): T7 done. dev/test.nu is a Nushell module: use dev/test.nu * gives test unit|integration|lint|vm|all|ci plus completions and self-check, and nu dev/test.nu <sub> dispatches through main for the devenv test script. Subcommands are declared as test <sub> (not main <sub>): with use dev/test.nu * Nushell binds the module's main to the file stem (test), and only commands literally named test ... complete and dispatch as test unit. Native subcommand help comes from the signatures; the entry help block lists the same table, and test self-check fails when the table, the defined commands, their flags or the help text diverge (mutation-tested: extra table row, extra command, extra flag). zsh/bash scripts are generated from the same table (test completions zsh|bash), sourced by enterShell. Verified: test all (fmt, clippy, openssl, unit, integration) green, --ide-complete returns subcommands, vm/filter completers, bash -n/zsh -n on the generated scripts. Not run here (no nix builds in this worktree): nu dev/test.nu vm peers; on a KVM host that is nix build .#tests.x86_64-linux.peers -L.
  • Handoff 2026-10-19 (WS-T -> all): the runner is dev/test.nu; use devenv shell -- test lint|unit|integration|vm|all|ci or nu dev/test.nu <subcommand>. nix/tests/peers.nix is unchanged and still the acceptance gate. To WS-F: promote tests.<system>.peers to checks.vm-peers once test vm peers passes; test integration picks up tests/peers*.rs with no runner change. Caveat: per-subcommand --help is native in Nushell module mode; in script mode (nu dev/test.nu vm --help) the entry help lists every subcommand.
  • 2026-10-19 (WS-T review round 1): Addressed every finding in loop/T/review-1.md. No contract change. Nothing outside the owned files. TODO.md is still unticked (not an owned file); the merger ticks T7.
    1. test integration --nocapture with no filter now runs cargo test --test '*' -- --nocapture. Cargo rejects a bare --nocapture. self-check compares the step command, spawns the runner in a dry-run, and runs cargo test --test '*' -- --nocapture --list.
    2. test stays the devenv script name (the plan's name, and devenv shell -- test already resolves it on PATH). Inside an interactive bash or zsh the builtin wins and test unit exits 0. The generated completion script defines a test function that dispatches known subcommands to this file and otherwise calls builtin test, so test -f still works. Interactive devenv shell sources the bash script. Docs tell zsh users to add source <(nu dev/test.nu completions zsh) themselves. self-check sources the bash and zsh scripts with a fake nu and checks both paths. Disagree with renaming: it would change the command every other stream already cites, and the PATH form the plan specified already works.
    3. Filter completion is target/skyg-test-filters.cache, first line a mtime key of src/**/*.rs, tests/**/*.rs, Cargo.toml and Cargo.lock. The old $env.SKYG_TEST_FILTERS assignment died with the completer. self-check proves a second lookup does not re-list and a stale key does.
    4. enterShell runs in devenv's bash, so the zsh branch never installed compdef. Removed. Docs say zsh users source the script from .zshrc after compinit. direnv will not carry the function.
    5. The enterShell note goes to stderr and only when $- contains i, so devenv shell -- test completions zsh stays pipe-clean. Checked: a non-interactive bash -c does not take the *i* branch.
    6. test integration is cargo test --test '*' (integration targets only). test all / test ci keep the separate unit step and no longer repeat it. Help text in AGENTS.md and README.md matches.
    7. The openssl step passes only when stdout is empty and cargo exits 0, or when stderr contains did not match any packages. Any other cargo failure fails the step. self-check covers a clean miss, an empty success, a hit, and a lockfile-style error.
    8. Missing /dev/kvm is an error inside the vm step body, not while the step list is built. self-check sets SKYG_TEST_ASSUME_NO_KVM=1 and asserts vm-steps returns a step whose body reports /dev/kvm.
    9. Script mode is --wrapped, so flags reach our parser. Unknown flags and extra positionals are errors. --nocapture is valid only on integration. self-check covers the parser and the CLI forms unit --nocapture and integration a b.
    10. TODO.md:28 T7 remains unticked on purpose.
  • Handoff 2026-10-19 (WS-T review round 1 -> all): same subcommand names. Invoke as devenv shell -- test <subcommand> or nu dev/test.nu <subcommand>. Interactive devenv shell installs the bash wrapper; zsh needs the .zshrc line after compinit. nu dev/test.nu self-check is the runner's test. Not run here (no nix builds in this worktree): nu dev/test.nu vm peers, which on a KVM host is nix build .#tests.x86_64-linux.peers -L. nix/tests/peers.nix is unchanged.
  • 2026-10-19 (WS-T, user decision): Renamed the runner from test to dev-test and the module from dev/test.nu to dev/dev-test.nu. Supersedes the review-round-1 note that kept the name test. test is a shell builtin, so an interactive test unit never reached the runner and the completion script had to define a test function that fell through to builtin test. dev-test is an external command: devenv shell -- dev-test unit|integration|lint|vm|all|ci and an interactive dev-test … both run the script. Completion scripts only register completion (compadd / compgen); they do not define a function. _describe emitted no candidates outside a completion widget, so zsh now uses compadd and self-check fails if the bash or zsh candidate list is empty. Nushell commands are named dev-test <sub> because the file stem is dev-test. argv/--nocapture and the filter cache are unchanged. nix/tests/peers.nix is unchanged. Also fixed a latent help bug: the <(...) example sat in a double-quoted string, so Nushell ran the completions command and --help inlined the whole zsh script; the line is concatenated now and self-check rejects an embedded script. Earlier dated Log entries keep the old name because they record what happened then; the Status line, T7 text and Handoff use the current name. Stale dev/test.nu names left for the merger (other streams' files, not edited here):
    • plan/phase-5-consolidator/README.md workstream index
    • plan/phase-5-consolidator/ws-a-config.md Do not touch
    • plan/phase-5-consolidator/ws-f-nixos-docs.md Do not touch
    • TODO.md T7 line (unticked on purpose; the merger ticks it)
  • Handoff 2026-10-19 (WS-T rename -> all): invoke the runner as devenv shell -- dev-test <subcommand> or nu dev/dev-test.nu <subcommand>. Interactive bash completion is installed by devenv shell; zsh, after compinit: source <(nu dev/dev-test.nu completions zsh). Nushell: use dev/dev-test.nu *. nu dev/dev-test.nu self-check is the runner's test. Verified here: nu dev/dev-test.nu self-check and nu dev/dev-test.nu all (fmt, clippy, openssl, unit, integration) green, plus devenv shell -- dev-test --help. Not run here (no nix builds in this worktree): nu dev/dev-test.nu vm peers, which on a KVM host is nix build .#tests.x86_64-linux.peers -L. nix/tests/peers.nix is unchanged.
  • 2026-10-19 (WS-T review round 2): Addressed loop/T/review-r.md. No contract change, no file outside the owned list, no new dependency (the self-check reuses the nu and shell probes already there).
    1. Blocking, module mode closed the caller's shell. run-steps and dev-test self-check ended with exit 1, so with use dev/dev-test.nu * in config.nu any red dev-test unit|all|ci ended the Nushell session after the summary (exit cannot be caught). Both now raise with error make; main is the only place that exits, after printing the reason to stderr, so script mode keeps its non-zero status. check-module-mode guards it three ways: exit outside main is a problem, (exit ...) interpolated into a double-quoted string is a problem, and a real failing step (SKYG_TEST_ASSUME_NO_KVM=1) must be caught in module mode while nu dev/dev-test.nu vm still exits non-zero. Mutation-checked: restoring exit 1 in run-steps fails the self-check with both the line number and the uncaught step.
      • Found on the way: the openssl step's message was $"cargo tree failed (exit ($found.exit_code)): ...". In Nushell (...) inside a double-quoted string is a subexpression, so that message would have called exit with cargo's status instead of raising a step failure (same class of bug, and it kills a module-mode session). Reworded to "failed with status (...)"; the new interpolation scan rejects a reoccurrence.
    2. Filter-cache key relative to the cwd. source-mtime-key globbed src/**/*.rs … from the cwd and FILTER_CACHE was the relative path. From src/ the globs found nothing, the key became "unknown" and the result was written to src/target/... under a key that never goes stale. The globs and the cache path are now anchored on repo-root ($SCRIPT_PATH | path dirname | path dirname); they go through glob because ls only expands patterns written literally in the source, and an unusable key (unknown or none) is neither read nor written (cached-lines returns a fresh listing). check-filter-cache now asserts the key from src/ equals the key from the root and is usable, that the cache path is absolute and <repo root>/target/skyg-test-filters.cache, and that an unknown key writes no file. Mutation-checked: relative globs and a removed unknown guard each fail the self-check. The 33-file mtime scan costs ~25 ms including Nushell start.
    3. Stale dev/test.nu in other streams' plan files. Not edited here, on purpose: the review agrees the merger owns it, and those rows are other streams' ownership guards. Still to change: plan/phase-5-consolidator/README.md (workstream index), ws-a-config.md and ws-f-nixos-docs.md (Do not touch rows).
    4. TODO.md T7. Not edited here (outside WS-T's owned files, and the phase gate rejects changes outside them). Still says dev/test.nu and is unticked; the merger updates the name and ticks it.
    5. AGENTS.md wording. "the same command works in an interactive shell" now reads "inside an interactive devenv shell". docs/consolidator.md says the cache sits at the repo root and that a failing check raises rather than closing the session.
  • Handoff 2026-10-19 (WS-T review round 2 -> all): unchanged commands and subcommand names. nu dev/dev-test.nu self-check now also fails if any command exits (outside main), if a message interpolates a call to exit, if a failing step is not catchable in module mode, or if the filter cache is not anchored on the repo root. Merger: items 3 and 4 above.
  • 2026-10-19 (WS-T review round 3): Addressed loop/T/review-n2.md (one blocking item). No contract change, no file outside the owned list, no new dependency.
    1. Blocking, main still exited on a dispatch error. export def --wrapped main wrapped dispatch-script-args in try/catch and called exit 1. With use dev/dev-test.nu * the bare name dev-test is bound to main, so every unmatched call reaches it: dev-test bogus or a typo like dev-test unti ended the caller's Nushell session, and the comment claiming "only script mode reaches this entry point" was wrong. The try/exit 1 is gone; the error propagates, which keeps module mode catchable (try { dev-test bogus } catch { print caught }; print after prints both lines) and still makes script mode exit 1 (nu dev/dev-test.nu bogus exits 1 with the unknown-subcommand message; Nushell gives an uncaught error make in a script status 1). The comment now says both modes land there and why exit must not be used. This supersedes the round-2 sentence "main is the only place that exits".
      • check-module-mode no longer allows a window for main: any exit is a problem, including an inline one such as catch { …; exit 1 } (the scan drops comment lines, then matches exit at the start of a line or after ;/{). The entry-point slicing (entry/next) is gone.
      • New probes: module mode try { dev-test bogus } catch { print 'caught' }; print 'after' must print both lines with exit 0, next to the existing vm probe; script mode nu dev/dev-test.nu bogus must exit non-zero with the unknown-subcommand reason, so the "script mode still fails" guarantee is tested rather than assumed. run-steps' doc comment now says it raises (the process exits non-zero).
      • Mutation-checked: restoring try { … } catch { exit 1 } in main fails the self-check on the exit line, and the module-mode bogus probe fails on its own.
  • Handoff 2026-10-19 (WS-T review round 3 -> all): unchanged commands and subcommand names. nu dev/dev-test.nu self-check now fails if any line calls exit at all (including in main), if a message interpolates a call to exit, if a failing step or an unknown subcommand is not catchable in module mode, if nu dev/dev-test.nu bogus does not exit non-zero, or if the filter cache is not anchored on the repo root. Merger items from round 2 (stale dev/test.nu in other streams' plan files, TODO.md T7) stand.