23 KiB
WS-T: Integration test suite (written first, user request)
Part of Phase 5: Consolidator. Contracts: contracts.md.
| Field | Value |
|---|---|
| Status | Merged |
| Owner | |
| Depends on | none (uses the module's freeform settings for [peers] and a systemd override for LoadCredential, so it needs no module changes). F1 later swaps this for the first-class module options |
| Blocks | nothing technically; it is the acceptance gate: goes green at the end of WS-E (with A, B, C, D landed), then F2 promotes it to checks |
| Can start when | day one |
| Owned files | nix/tests/peers.nix (done), flake.nix (the tests.<system>.peers output only), dev/dev-test.nu, devenv.nix (nushell package, dev-test script, completion hook), AGENTS.md and README.md (command lists only, see seams), docs/consolidator.md (creates the file with only a ## Testing section, see seams) |
| Do not touch | src/** (all streams), nix/module.nix, nix/tests/vm.nix, systemd/skyg-linux-tracker.service (F), config.example.toml (A/F), tests/peers.rs and tests/common/mod.rs (F; in-process variants by A/B live in their own test files, see ws-a, ws-b), docs/api.md (C) |
Shared-file seams:
flake.nix: T owns it until F2. F2 later moves exactly one attribute,tests.${system}.peers, intochecks(renamechecks.vm-peers). Nobody else editsflake.nix; it already carries the uncommitted peers output.devenv.nix: T only.README.md: T edits only the command/test list near the existing commands; F4 owns the new peers section. Different headings, so no overlap.AGENTS.md: T replaces the command list; F4 adds only the MCP tool-count note.docs/consolidator.md: T creates it with a single## Testingsection (the one-lineuse dev/dev-test.nu *and the completion setup). F4 adds every other section and must not edit## Testing.
Goal
Executable acceptance criteria before the implementation. Two NixOS VMs each run the real slinuxm binary and consolidate each other: VM alpha declares beta as a peer, VM beta declares alpha, and each proxies the other's data. Expected to fail until WS-E lands, which is the point.
Note on the written test: nix/tests/peers.nix also contains a third node gamma (TLS-only, no peers) that alpha declares with a private CA via ca_file, covering the TLS variant. It is already written and, by design, fails at the first peers assertion (phase 2, "declared but token is still a placeholder -> auth rejected") against the current binary; phase 1 (boot, healthz, token minting, marker units, token-file placeholders) already passes.
Tasks
- T1 two machines
alphaandbetawith distinct hostnames, each with a distinct fake workload (a marker systemd unitmarker-alpha.service/marker-beta.service, and a deliberately failing unit on one side) so proxied data is distinguishable from local data. Done innix/tests/peers.nix(failing unit isbroken-beta.service;gammawithmarker-gamma.serviceadded). - T2 each mints a read-only token via
sudo -u skyg-linux-tracker slinuxm token create consolidator-<other>, writes the other's token to a root-only 0400 file (stand-in for an agenix path), and declares the other as a peer with thattoken_file. Done (files under/var/lib/peer-tokens, delivered viaLoadCredential). - T3 assertions on both directions:
/api/v1/peersshows the peer with server up, api up and auth ok;GET /api/v1/peers/<other>/systemd/units/marker-<other>.servicereturns the other machine's unit and not a local one; MCPlist_peersand alist_systemd_unitscall withpeerreturn the same data; response headerx-slinuxm-peer. Done in the test script. - T4 token rejected: revoke the token on
beta;alphareports server up, api up,auth = rejected,peer.api.auth_ok = 0, and proxy calls fail with 503 naming the peer, without becoming a 401 for the caller; restore with a new token file and no restart, status recovers. Done in the script except thepeer.api.auth_ok = 0metric check (needs WS-E; add it to the script, or leave to F2 which may query the collector/metrics endpoint, note in Log). - T5 outage: stop
beta's service:alphaflips server and api afterfail_threshold,alpha's own endpoints stay healthy, and recovery is detected; stopbetaentirely (VM down) and check thatalpha's requests are bounded by timeouts. Done in the script. - T6 no chaining and read-only: a request carrying
x-slinuxm-hopis rejected; non-GET to a proxy path is 405; a TLS variant with a test CA viaca_file. Done (405 for POST/DELETE, hop header 4xx,gammaover HTTPS). - T7 Done:
dev/dev-test.nuwithunit,integration,lint,vm,all,ci,completions,self-check; devenvdev-testscript plus zsh/bash completion hook; AGENTS.md and README.md command lists;docs/consolidator.md## Testing. A simple test runner CLI written in Nushell:dev/dev-test.nu(devenv addspkgs.nushell; also exposed as the devenv scriptdev-test, sodevenv shell -- dev-test unitworks from any shell, including an interactive one). One entry point so contributors and agents do not memorise cargo/nix incantations. The command was renamed fromtesttodev-test(user decision, see Log) becausetestis a shell builtin. Structure it as a Nushell script with amainand subcommands literally nameddev-test <sub>(export def "dev-test unit", ...) so--helpis generated and completion comes from the typed signatures. Nushell bindsmainto the file stem, so only commands nameddev-test ...complete asdev-test unit:dev-test unitrunscargo test --lib --bins(no sockets, no host access).dev-test integration [filter]runscargo test --test '*'(integration targets only: plain, tls, integration, binary_smoke,peersonce it exists), sodev-test alldoes not run the unit tests twice.filteris passed to cargo as a name filter.--nocaptureis a libtest flag and is always placed after--.dev-test lintrunscargo fmt --check,cargo clippy --all-targets -- -D warningsand thecargo tree -i opensslemptiness check.dev-test vm [name]runs a NixOS VM test.nameis one ofvm(fromchecks) orpeers(nix build .#tests.x86_64-linux.peers -L; moves tocheckswhen green). A missing/dev/kvmfails that step and says why, after earliercisteps, and the summary table still prints. The step prints that the build is slow.dev-test all= lint, unit, integration, stopping at the first failure;dev-test ci=allplusvm. No args prints help.- Each step prints a header and a final table (step, result, seconds); exit code non-zero on any failure (use
error make/exit 1, never swallow cargo's exit status). Read-only against the host: no/etc, systemd or Docker changes (live-host checks stay manual, see Verification plan in the README). - Autocomplete (must work, and is a done-criterion):
- Nushell: subcommands and flags complete natively once the script is loaded as a module. Dynamic values use custom completers (
def "nu-complete dev-test vm" [] { ["vm" "peers"] }andname: string@"nu-complete dev-test vm"; integration filters complete fromcargo test --test '*' -- --list, cached intarget/skyg-test-filters.cacheand keyed on the newest mtime of the Rust sources and Cargo manifests). A completer cannot update the caller's environment, so an env var does not survive Tab.docs/consolidator.mdshows the one-lineuse dev/dev-test.nu *for a permanent setup. - Other shells (the repo owner's login shell is zsh):
nu dev/dev-test.nu completions zsh|bashprints a completion script generated from the same command table.dev-testis not a builtin, so the script only registers completion (compaddin zsh,compgenin bash) and does not define a function. Interactivedev-test …runs the devenv script on PATH.self-checkinvokes those completion functions and fails if the candidate list is empty. Interactivedevenv shellsources the bash script and prints nothing, sodevenv shell -- dev-test completions zshstays pipe-clean. zsh is not started by that hook; aftercompinit, zsh users addsource <(nu dev/dev-test.nu completions zsh)to.zshrcthemselves. direnv does not source it for them. - Verified by a self-test:
dev-test --helpanddev-test vm --helplist every subcommand/flag, and a check indev/dev-test.nu(dev-test self-check) fails if the completion table and the defined subcommands diverge.
- Nushell: subcommands and flags complete natively once the script is loaded as a module. Dynamic values use custom completers (
- AGENTS.md and README replace their scattered command lists with
dev-test ..., keeping the underlying commands visible in--help.
Inputs
- contracts.md section 10 (test config contract) and sections 5 and 7 (REST/MCP shapes the script asserts). No fakes: the test runs the real binary.
nix/tests/peers.nixis the source of truth for section 10; if the test must change, change it and update contracts.md in the same PR, then log in the affected streams.
Done when
The test builds under nix build .#tests.x86_64-linux.peers (or equivalent) and evaluates, fails cleanly at the first peers assertion against the current binary, and is documented in docs/consolidator.md. dev-test unit|integration|lint|vm|all each run and report correctly on the current tree, and tab completion works in Nushell, zsh and bash (T7). It must pass at the end of WS-E, then move to checks.vm-peers (F2 does the move).
Handoff
- To everyone: the
dev-testrunner and its subcommand names (referenced by AGENTS.md/README and by every stream's "Done when"). - To WS-F: a passing
peers.nix(ready to promote) and any changes made to its assumptions, recorded in contracts.md section 10. - To WS-A/B: the
dev-test integrationstep picks up anytests/peers*.rsautomatically.
Log
- 2026-10-19: Split from the single phase plan 2026-10-19.
nix/tests/peers.nixalready written (T1 to T6) and failing at the first peers assertion by design;flake.nixexposes it astests.<system>.peers. T7 not started. Ownership change from the single plan:tests/peers.rsmoved to WS-F; WS-A/B in-process tests use their own files to avoid collisions. - 2026-10-19 (WS-T): T7 done.
dev/test.nuis a Nushell module:use dev/test.nu *givestest unit|integration|lint|vm|all|cipluscompletionsandself-check, andnu dev/test.nu <sub>dispatches throughmainfor the devenvtestscript. Subcommands are declared astest <sub>(notmain <sub>): withuse dev/test.nu *Nushell binds the module'smainto the file stem (test), and only commands literally namedtest ...complete and dispatch astest unit. Native subcommand help comes from the signatures; the entry help block lists the same table, andtest self-checkfails when the table, the defined commands, their flags or the help text diverge (mutation-tested: extra table row, extra command, extra flag). zsh/bash scripts are generated from the same table (test completions zsh|bash), sourced byenterShell. Verified:test all(fmt, clippy, openssl, unit, integration) green,--ide-completereturns subcommands,vm/filter completers,bash -n/zsh -non the generated scripts. Not run here (no nix builds in this worktree):nu dev/test.nu vm peers; on a KVM host that isnix build .#tests.x86_64-linux.peers -L. - Handoff 2026-10-19 (WS-T -> all): the runner is
dev/test.nu; usedevenv shell -- test lint|unit|integration|vm|all|ciornu dev/test.nu <subcommand>.nix/tests/peers.nixis unchanged and still the acceptance gate. To WS-F: promotetests.<system>.peerstochecks.vm-peersoncetest vm peerspasses;test integrationpicks uptests/peers*.rswith no runner change. Caveat: per-subcommand--helpis native in Nushell module mode; in script mode (nu dev/test.nu vm --help) the entry help lists every subcommand. - 2026-10-19 (WS-T review round 1): Addressed every finding in
loop/T/review-1.md. No contract change. Nothing outside the owned files.TODO.mdis still unticked (not an owned file); the merger ticks T7.test integration --nocapturewith no filter now runscargo test --test '*' -- --nocapture. Cargo rejects a bare--nocapture.self-checkcompares the step command, spawns the runner in a dry-run, and runscargo test --test '*' -- --nocapture --list.teststays the devenv script name (the plan's name, anddevenv shell -- testalready resolves it on PATH). Inside an interactive bash or zsh the builtin wins andtest unitexits 0. The generated completion script defines atestfunction that dispatches known subcommands to this file and otherwise callsbuiltin test, sotest -fstill works. Interactivedevenv shellsources the bash script. Docs tell zsh users to addsource <(nu dev/test.nu completions zsh)themselves.self-checksources the bash and zsh scripts with a fakenuand checks both paths. Disagree with renaming: it would change the command every other stream already cites, and the PATH form the plan specified already works.- Filter completion is
target/skyg-test-filters.cache, first line a mtime key ofsrc/**/*.rs,tests/**/*.rs,Cargo.tomlandCargo.lock. The old$env.SKYG_TEST_FILTERSassignment died with the completer.self-checkproves a second lookup does not re-list and a stale key does. enterShellruns in devenv's bash, so the zsh branch never installedcompdef. Removed. Docs say zsh users source the script from.zshrcaftercompinit. direnv will not carry the function.- The enterShell note goes to stderr and only when
$-containsi, sodevenv shell -- test completions zshstays pipe-clean. Checked: a non-interactivebash -cdoes not take the*i*branch. test integrationiscargo test --test '*'(integration targets only).test all/test cikeep the separateunitstep and no longer repeat it. Help text inAGENTS.mdandREADME.mdmatches.- The openssl step passes only when stdout is empty and cargo exits 0, or when stderr contains
did not match any packages. Any other cargo failure fails the step.self-checkcovers a clean miss, an empty success, a hit, and a lockfile-style error. - Missing
/dev/kvmis an error inside the vm step body, not while the step list is built.self-checksetsSKYG_TEST_ASSUME_NO_KVM=1and assertsvm-stepsreturns a step whose body reports/dev/kvm. - Script mode is
--wrapped, so flags reach our parser. Unknown flags and extra positionals are errors.--nocaptureis valid only onintegration.self-checkcovers the parser and the CLI formsunit --nocaptureandintegration a b. TODO.md:28T7 remains unticked on purpose.
- Handoff 2026-10-19 (WS-T review round 1 -> all): same subcommand names. Invoke as
devenv shell -- test <subcommand>ornu dev/test.nu <subcommand>. Interactivedevenv shellinstalls the bash wrapper; zsh needs the.zshrcline aftercompinit.nu dev/test.nu self-checkis the runner's test. Not run here (no nix builds in this worktree):nu dev/test.nu vm peers, which on a KVM host isnix build .#tests.x86_64-linux.peers -L.nix/tests/peers.nixis unchanged. - 2026-10-19 (WS-T, user decision): Renamed the runner from
testtodev-testand the module fromdev/test.nutodev/dev-test.nu. Supersedes the review-round-1 note that kept the nametest.testis a shell builtin, so an interactivetest unitnever reached the runner and the completion script had to define atestfunction that fell through tobuiltin test.dev-testis an external command:devenv shell -- dev-test unit|integration|lint|vm|all|ciand an interactivedev-test …both run the script. Completion scripts only register completion (compadd/compgen); they do not define a function._describeemitted no candidates outside a completion widget, so zsh now usescompaddandself-checkfails if the bash or zsh candidate list is empty. Nushell commands are nameddev-test <sub>because the file stem isdev-test. argv/--nocaptureand the filter cache are unchanged.nix/tests/peers.nixis unchanged. Also fixed a latent help bug: the<(...)example sat in a double-quoted string, so Nushell ran the completions command and--helpinlined the whole zsh script; the line is concatenated now andself-checkrejects an embedded script. Earlier dated Log entries keep the old name because they record what happened then; the Status line, T7 text and Handoff use the current name. Staledev/test.nunames left for the merger (other streams' files, not edited here):plan/phase-5-consolidator/README.mdworkstream indexplan/phase-5-consolidator/ws-a-config.mdDo not touchplan/phase-5-consolidator/ws-f-nixos-docs.mdDo not touchTODO.mdT7 line (unticked on purpose; the merger ticks it)
- Handoff 2026-10-19 (WS-T rename -> all): invoke the runner as
devenv shell -- dev-test <subcommand>ornu dev/dev-test.nu <subcommand>. Interactive bash completion is installed bydevenv shell; zsh, aftercompinit:source <(nu dev/dev-test.nu completions zsh). Nushell:use dev/dev-test.nu *.nu dev/dev-test.nu self-checkis the runner's test. Verified here:nu dev/dev-test.nu self-checkandnu dev/dev-test.nu all(fmt, clippy, openssl, unit, integration) green, plusdevenv shell -- dev-test --help. Not run here (no nix builds in this worktree):nu dev/dev-test.nu vm peers, which on a KVM host isnix build .#tests.x86_64-linux.peers -L.nix/tests/peers.nixis unchanged. - 2026-10-19 (WS-T review round 2): Addressed
loop/T/review-r.md. No contract change, no file outside the owned list, no new dependency (the self-check reuses thenuand shell probes already there).- Blocking, module mode closed the caller's shell.
run-stepsanddev-test self-checkended withexit 1, so withuse dev/dev-test.nu *inconfig.nuany reddev-test unit|all|ciended the Nushell session after the summary (exitcannot be caught). Both now raise witherror make;mainis the only place that exits, after printing the reason to stderr, so script mode keeps its non-zero status.check-module-modeguards it three ways:exitoutsidemainis a problem,(exit ...)interpolated into a double-quoted string is a problem, and a real failing step (SKYG_TEST_ASSUME_NO_KVM=1) must be caught in module mode whilenu dev/dev-test.nu vmstill exits non-zero. Mutation-checked: restoringexit 1inrun-stepsfails the self-check with both the line number and the uncaught step.- Found on the way: the openssl step's message was
$"cargo tree failed (exit ($found.exit_code)): ...". In Nushell(...)inside a double-quoted string is a subexpression, so that message would have calledexitwith cargo's status instead of raising a step failure (same class of bug, and it kills a module-mode session). Reworded to "failed with status (...)"; the new interpolation scan rejects a reoccurrence.
- Found on the way: the openssl step's message was
- Filter-cache key relative to the cwd.
source-mtime-keyglobbedsrc/**/*.rs …from the cwd andFILTER_CACHEwas the relative path. Fromsrc/the globs found nothing, the key became "unknown" and the result was written tosrc/target/...under a key that never goes stale. The globs and the cache path are now anchored onrepo-root($SCRIPT_PATH | path dirname | path dirname); they go throughglobbecauselsonly expands patterns written literally in the source, and an unusable key (unknownornone) is neither read nor written (cached-linesreturns a fresh listing).check-filter-cachenow asserts the key fromsrc/equals the key from the root and is usable, that the cache path is absolute and<repo root>/target/skyg-test-filters.cache, and that an unknown key writes no file. Mutation-checked: relative globs and a removedunknownguard each fail the self-check. The 33-file mtime scan costs ~25 ms including Nushell start. - Stale
dev/test.nuin other streams' plan files. Not edited here, on purpose: the review agrees the merger owns it, and those rows are other streams' ownership guards. Still to change:plan/phase-5-consolidator/README.md(workstream index),ws-a-config.mdandws-f-nixos-docs.md(Do not touch rows). TODO.mdT7. Not edited here (outside WS-T's owned files, and the phase gate rejects changes outside them). Still saysdev/test.nuand is unticked; the merger updates the name and ticks it.AGENTS.mdwording. "the same command works in an interactive shell" now reads "inside an interactivedevenv shell".docs/consolidator.mdsays the cache sits at the repo root and that a failing check raises rather than closing the session.
- Blocking, module mode closed the caller's shell.
- Handoff 2026-10-19 (WS-T review round 2 -> all): unchanged commands and subcommand names.
nu dev/dev-test.nu self-checknow also fails if any command exits (outsidemain), if a message interpolates a call toexit, if a failing step is not catchable in module mode, or if the filter cache is not anchored on the repo root. Merger: items 3 and 4 above. - 2026-10-19 (WS-T review round 3): Addressed
loop/T/review-n2.md(one blocking item). No contract change, no file outside the owned list, no new dependency.- Blocking,
mainstill exited on a dispatch error.export def --wrapped mainwrappeddispatch-script-argsintry/catchand calledexit 1. Withuse dev/dev-test.nu *the bare namedev-testis bound tomain, so every unmatched call reaches it:dev-test bogusor a typo likedev-test untiended the caller's Nushell session, and the comment claiming "only script mode reaches this entry point" was wrong. Thetry/exit 1is gone; the error propagates, which keeps module mode catchable (try { dev-test bogus } catch { print caught }; print afterprints both lines) and still makes script mode exit 1 (nu dev/dev-test.nu bogusexits 1 with the unknown-subcommand message; Nushell gives an uncaughterror makein a script status 1). The comment now says both modes land there and whyexitmust not be used. This supersedes the round-2 sentence "mainis the only place that exits".check-module-modeno longer allows a window formain: anyexitis a problem, including an inline one such ascatch { …; exit 1 }(the scan drops comment lines, then matchesexitat the start of a line or after;/{). The entry-point slicing (entry/next) is gone.- New probes: module mode
try { dev-test bogus } catch { print 'caught' }; print 'after'must print both lines with exit 0, next to the existingvmprobe; script modenu dev/dev-test.nu bogusmust exit non-zero with the unknown-subcommand reason, so the "script mode still fails" guarantee is tested rather than assumed.run-steps' doc comment now says it raises (the process exits non-zero). - Mutation-checked: restoring
try { … } catch { exit 1 }inmainfails the self-check on theexitline, and the module-modebogusprobe fails on its own.
- Blocking,
- Handoff 2026-10-19 (WS-T review round 3 -> all): unchanged commands and subcommand names.
nu dev/dev-test.nu self-checknow fails if any line callsexitat all (including inmain), if a message interpolates a call toexit, if a failing step or an unknown subcommand is not catchable in module mode, ifnu dev/dev-test.nu bogusdoes not exit non-zero, or if the filter cache is not anchored on the repo root. Merger items from round 2 (staledev/test.nuin other streams' plan files,TODO.mdT7) stand.