# WS-T: Integration test suite (written first, user request) Part of [Phase 5: Consolidator](README.md). Contracts: [contracts.md](contracts.md). | Field | Value | |---|---| | Status | T1 to T7 done; review round 1 addressed; runner renamed to `dev-test` (`dev/dev-test.nu`). `peers.nix` fails at the first peers assertion until WS-E lands (by design) | | Owner | | | Depends on | none (uses the module's freeform `settings` for `[peers]` and a systemd override for `LoadCredential`, so it needs no module changes). F1 later swaps this for the first-class module options | | Blocks | nothing technically; it is the acceptance gate: goes green at the end of [WS-E](ws-e-telemetry.md) (with A, B, C, D landed), then F2 promotes it to `checks` | | Can start when | day one | | Owned files | `nix/tests/peers.nix` (done), `flake.nix` (the `tests..peers` output only), `dev/dev-test.nu`, `devenv.nix` (nushell package, `dev-test` script, completion hook), `AGENTS.md` and `README.md` (command lists only, see seams), `docs/consolidator.md` (creates the file with only a `## Testing` section, see seams) | | Do not touch | `src/**` (all streams), `nix/module.nix`, `nix/tests/vm.nix`, `systemd/skyg-linux-tracker.service` (F), `config.example.toml` (A/F), `tests/peers.rs` and `tests/common/mod.rs` (F; in-process variants by A/B live in their own test files, see [ws-a](ws-a-config.md), [ws-b](ws-b-registry.md)), `docs/api.md` (C) | Shared-file seams: - `flake.nix`: T owns it until F2. F2 later moves exactly one attribute, `tests.${system}.peers`, into `checks` (rename `checks.vm-peers`). Nobody else edits `flake.nix`; it already carries the uncommitted peers output. - `devenv.nix`: T only. - `README.md`: T edits only the command/test list near the existing commands; F4 owns the new peers section. Different headings, so no overlap. `AGENTS.md`: T replaces the command list; F4 adds only the MCP tool-count note. - `docs/consolidator.md`: T creates it with a single `## Testing` section (the one-line `use dev/dev-test.nu *` and the completion setup). F4 adds every other section and must not edit `## Testing`. ## Goal Executable acceptance criteria before the implementation. Two NixOS VMs each run the real `slinuxm` binary and consolidate each other: VM `alpha` declares `beta` as a peer, VM `beta` declares `alpha`, and each proxies the other's data. Expected to fail until WS-E lands, which is the point. Note on the written test: `nix/tests/peers.nix` also contains a third node `gamma` (TLS-only, no peers) that `alpha` declares with a private CA via `ca_file`, covering the TLS variant. It is already written and, by design, fails at the first peers assertion (phase 2, "declared but token is still a placeholder -> auth rejected") against the current binary; phase 1 (boot, healthz, token minting, marker units, token-file placeholders) already passes. ## Tasks - **T1** two machines `alpha` and `beta` with distinct hostnames, each with a distinct fake workload (a marker systemd unit `marker-alpha.service` / `marker-beta.service`, and a deliberately failing unit on one side) so proxied data is distinguishable from local data. **Done** in `nix/tests/peers.nix` (failing unit is `broken-beta.service`; `gamma` with `marker-gamma.service` added). - **T2** each mints a read-only token via `sudo -u skyg-linux-tracker slinuxm token create consolidator-`, writes the other's token to a root-only 0400 file (stand-in for an agenix path), and declares the other as a peer with that `token_file`. **Done** (files under `/var/lib/peer-tokens`, delivered via `LoadCredential`). - **T3** assertions on both directions: `/api/v1/peers` shows the peer with server up, api up and auth ok; `GET /api/v1/peers//systemd/units/marker-.service` returns the other machine's unit and not a local one; MCP `list_peers` and a `list_systemd_units` call with `peer` return the same data; response header `x-slinuxm-peer`. **Done** in the test script. - **T4** token rejected: revoke the token on `beta`; `alpha` reports server up, api up, `auth = rejected`, `peer.api.auth_ok = 0`, and proxy calls fail with 503 naming the peer, without becoming a 401 for the caller; restore with a new token file and no restart, status recovers. **Done** in the script except the `peer.api.auth_ok = 0` metric check (needs WS-E; add it to the script, or leave to F2 which may query the collector/metrics endpoint, note in Log). - **T5** outage: stop `beta`'s service: `alpha` flips server and api after `fail_threshold`, `alpha`'s own endpoints stay healthy, and recovery is detected; stop `beta` entirely (VM down) and check that `alpha`'s requests are bounded by timeouts. **Done** in the script. - **T6** no chaining and read-only: a request carrying `x-slinuxm-hop` is rejected; non-GET to a proxy path is 405; a TLS variant with a test CA via `ca_file`. **Done** (405 for POST/DELETE, hop header 4xx, `gamma` over HTTPS). - **T7** **Done**: `dev/dev-test.nu` with `unit`, `integration`, `lint`, `vm`, `all`, `ci`, `completions`, `self-check`; devenv `dev-test` script plus zsh/bash completion hook; AGENTS.md and README.md command lists; `docs/consolidator.md` `## Testing`. A simple test runner CLI written in **Nushell**: `dev/dev-test.nu` (devenv adds `pkgs.nushell`; also exposed as the devenv script `dev-test`, so `devenv shell -- dev-test unit` works from any shell, including an interactive one). One entry point so contributors and agents do not memorise cargo/nix incantations. The command was renamed from `test` to `dev-test` (user decision, see Log) because `test` is a shell builtin. Structure it as a Nushell script with a `main` and subcommands literally named `dev-test ` (`export def "dev-test unit"`, ...) so `--help` is generated and completion comes from the typed signatures. Nushell binds `main` to the file stem, so only commands named `dev-test ...` complete as `dev-test unit`: - `dev-test unit` runs `cargo test --lib --bins` (no sockets, no host access). - `dev-test integration [filter]` runs `cargo test --test '*'` (integration targets only: plain, tls, integration, binary_smoke, `peers` once it exists), so `dev-test all` does not run the unit tests twice. `filter` is passed to cargo as a name filter. `--nocapture` is a libtest flag and is always placed after `--`. - `dev-test lint` runs `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings` and the `cargo tree -i openssl` emptiness check. - `dev-test vm [name]` runs a NixOS VM test. `name` is one of `vm` (from `checks`) or `peers` (`nix build .#tests.x86_64-linux.peers -L`; moves to `checks` when green). A missing `/dev/kvm` fails that step and says why, after earlier `ci` steps, and the summary table still prints. The step prints that the build is slow. - `dev-test all` = lint, unit, integration, stopping at the first failure; `dev-test ci` = `all` plus `vm`. No args prints help. - Each step prints a header and a final table (step, result, seconds); exit code non-zero on any failure (use `error make`/`exit 1`, never swallow cargo's exit status). Read-only against the host: no `/etc`, systemd or Docker changes (live-host checks stay manual, see Verification plan in the [README](README.md#verification-plan)). - **Autocomplete (must work, and is a done-criterion):** - Nushell: subcommands and flags complete natively once the script is loaded as a module. Dynamic values use custom completers (`def "nu-complete dev-test vm" [] { ["vm" "peers"] }` and `name: string@"nu-complete dev-test vm"`; integration filters complete from `cargo test --test '*' -- --list`, cached in `target/skyg-test-filters.cache` and keyed on the newest mtime of the Rust sources and Cargo manifests). A completer cannot update the caller's environment, so an env var does not survive Tab. `docs/consolidator.md` shows the one-line `use dev/dev-test.nu *` for a permanent setup. - Other shells (the repo owner's login shell is zsh): `nu dev/dev-test.nu completions zsh|bash` prints a completion script generated from the same command table. `dev-test` is not a builtin, so the script only registers completion (`compadd` in zsh, `compgen` in bash) and does not define a function. Interactive `dev-test …` runs the devenv script on PATH. `self-check` invokes those completion functions and fails if the candidate list is empty. Interactive `devenv shell` sources the bash script and prints nothing, so `devenv shell -- dev-test completions zsh` stays pipe-clean. zsh is not started by that hook; after `compinit`, zsh users add `source <(nu dev/dev-test.nu completions zsh)` to `.zshrc` themselves. direnv does not source it for them. - Verified by a self-test: `dev-test --help` and `dev-test vm --help` list every subcommand/flag, and a check in `dev/dev-test.nu` (`dev-test self-check`) fails if the completion table and the defined subcommands diverge. - AGENTS.md and README replace their scattered command lists with `dev-test ...`, keeping the underlying commands visible in `--help`. ## Inputs - [contracts.md](contracts.md) section 10 (test config contract) and sections 5 and 7 (REST/MCP shapes the script asserts). No fakes: the test runs the real binary. - `nix/tests/peers.nix` is the source of truth for section 10; if the test must change, change it and update contracts.md in the same PR, then log in the affected streams. ## Done when The test builds under `nix build .#tests.x86_64-linux.peers` (or equivalent) and evaluates, fails cleanly at the first peers assertion against the current binary, and is documented in `docs/consolidator.md`. `dev-test unit|integration|lint|vm|all` each run and report correctly on the current tree, and tab completion works in Nushell, zsh and bash (T7). It must pass at the end of WS-E, then move to `checks.vm-peers` (F2 does the move). ## Handoff - To everyone: the `dev-test` runner and its subcommand names (referenced by AGENTS.md/README and by every stream's "Done when"). - To WS-F: a passing `peers.nix` (ready to promote) and any changes made to its assumptions, recorded in contracts.md section 10. - To WS-A/B: the `dev-test integration` step picks up any `tests/peers*.rs` automatically. ## Log - 2026-10-19: Split from the single phase plan 2026-10-19. `nix/tests/peers.nix` already written (T1 to T6) and failing at the first peers assertion by design; `flake.nix` exposes it as `tests..peers`. T7 not started. Ownership change from the single plan: `tests/peers.rs` moved to WS-F; WS-A/B in-process tests use their own files to avoid collisions. - 2026-10-19 (WS-T): T7 done. `dev/test.nu` is a Nushell module: `use dev/test.nu *` gives `test unit|integration|lint|vm|all|ci` plus `completions` and `self-check`, and `nu dev/test.nu ` dispatches through `main` for the devenv `test` script. Subcommands are declared as `test ` (not `main `): with `use dev/test.nu *` Nushell binds the module's `main` to the file stem (`test`), and only commands literally named `test ...` complete and dispatch as `test unit`. Native subcommand help comes from the signatures; the entry help block lists the same table, and `test self-check` fails when the table, the defined commands, their flags or the help text diverge (mutation-tested: extra table row, extra command, extra flag). zsh/bash scripts are generated from the same table (`test completions zsh|bash`), sourced by `enterShell`. Verified: `test all` (fmt, clippy, openssl, unit, integration) green, `--ide-complete` returns subcommands, `vm`/filter completers, `bash -n`/`zsh -n` on the generated scripts. Not run here (no nix builds in this worktree): `nu dev/test.nu vm peers`; on a KVM host that is `nix build .#tests.x86_64-linux.peers -L`. - Handoff 2026-10-19 (WS-T -> all): the runner is `dev/test.nu`; use `devenv shell -- test lint|unit|integration|vm|all|ci` or `nu dev/test.nu `. `nix/tests/peers.nix` is unchanged and still the acceptance gate. To WS-F: promote `tests..peers` to `checks.vm-peers` once `test vm peers` passes; `test integration` picks up `tests/peers*.rs` with no runner change. Caveat: per-subcommand `--help` is native in Nushell module mode; in script mode (`nu dev/test.nu vm --help`) the entry help lists every subcommand. - 2026-10-19 (WS-T review round 1): Addressed every finding in `loop/T/review-1.md`. No contract change. Nothing outside the owned files. `TODO.md` is still unticked (not an owned file); the merger ticks T7. 1. `test integration --nocapture` with no filter now runs `cargo test --test '*' -- --nocapture`. Cargo rejects a bare `--nocapture`. `self-check` compares the step command, spawns the runner in a dry-run, and runs `cargo test --test '*' -- --nocapture --list`. 2. `test` stays the devenv script name (the plan's name, and `devenv shell -- test` already resolves it on PATH). Inside an interactive bash or zsh the builtin wins and `test unit` exits 0. The generated completion script defines a `test` function that dispatches known subcommands to this file and otherwise calls `builtin test`, so `test -f` still works. Interactive `devenv shell` sources the bash script. Docs tell zsh users to add `source <(nu dev/test.nu completions zsh)` themselves. `self-check` sources the bash and zsh scripts with a fake `nu` and checks both paths. Disagree with renaming: it would change the command every other stream already cites, and the PATH form the plan specified already works. 3. Filter completion is `target/skyg-test-filters.cache`, first line a mtime key of `src/**/*.rs`, `tests/**/*.rs`, `Cargo.toml` and `Cargo.lock`. The old `$env.SKYG_TEST_FILTERS` assignment died with the completer. `self-check` proves a second lookup does not re-list and a stale key does. 4. `enterShell` runs in devenv's bash, so the zsh branch never installed `compdef`. Removed. Docs say zsh users source the script from `.zshrc` after `compinit`. direnv will not carry the function. 5. The enterShell note goes to stderr and only when `$-` contains `i`, so `devenv shell -- test completions zsh` stays pipe-clean. Checked: a non-interactive `bash -c` does not take the `*i*` branch. 6. `test integration` is `cargo test --test '*'` (integration targets only). `test all` / `test ci` keep the separate `unit` step and no longer repeat it. Help text in `AGENTS.md` and `README.md` matches. 7. The openssl step passes only when stdout is empty and cargo exits 0, or when stderr contains `did not match any packages`. Any other cargo failure fails the step. `self-check` covers a clean miss, an empty success, a hit, and a lockfile-style error. 8. Missing `/dev/kvm` is an error inside the vm step body, not while the step list is built. `self-check` sets `SKYG_TEST_ASSUME_NO_KVM=1` and asserts `vm-steps` returns a step whose body reports `/dev/kvm`. 9. Script mode is `--wrapped`, so flags reach our parser. Unknown flags and extra positionals are errors. `--nocapture` is valid only on `integration`. `self-check` covers the parser and the CLI forms `unit --nocapture` and `integration a b`. 10. `TODO.md:28` T7 remains unticked on purpose. - Handoff 2026-10-19 (WS-T review round 1 -> all): same subcommand names. Invoke as `devenv shell -- test ` or `nu dev/test.nu `. Interactive `devenv shell` installs the bash wrapper; zsh needs the `.zshrc` line after `compinit`. `nu dev/test.nu self-check` is the runner's test. Not run here (no nix builds in this worktree): `nu dev/test.nu vm peers`, which on a KVM host is `nix build .#tests.x86_64-linux.peers -L`. `nix/tests/peers.nix` is unchanged. - 2026-10-19 (WS-T, user decision): Renamed the runner from `test` to `dev-test` and the module from `dev/test.nu` to `dev/dev-test.nu`. Supersedes the review-round-1 note that kept the name `test`. `test` is a shell builtin, so an interactive `test unit` never reached the runner and the completion script had to define a `test` function that fell through to `builtin test`. `dev-test` is an external command: `devenv shell -- dev-test unit|integration|lint|vm|all|ci` and an interactive `dev-test …` both run the script. Completion scripts only register completion (`compadd` / `compgen`); they do not define a function. `_describe` emitted no candidates outside a completion widget, so zsh now uses `compadd` and `self-check` fails if the bash or zsh candidate list is empty. Nushell commands are named `dev-test ` because the file stem is `dev-test`. argv/`--nocapture` and the filter cache are unchanged. `nix/tests/peers.nix` is unchanged. Also fixed a latent help bug: the `<(...)` example sat in a double-quoted string, so Nushell ran the completions command and `--help` inlined the whole zsh script; the line is concatenated now and `self-check` rejects an embedded script. Earlier dated Log entries keep the old name because they record what happened then; the Status line, T7 text and Handoff use the current name. Stale `dev/test.nu` names left for the merger (other streams' files, not edited here): - `plan/phase-5-consolidator/README.md` workstream index - `plan/phase-5-consolidator/ws-a-config.md` Do not touch - `plan/phase-5-consolidator/ws-f-nixos-docs.md` Do not touch - `TODO.md` T7 line (unticked on purpose; the merger ticks it) - Handoff 2026-10-19 (WS-T rename -> all): invoke the runner as `devenv shell -- dev-test ` or `nu dev/dev-test.nu `. Interactive bash completion is installed by `devenv shell`; zsh, after `compinit`: `source <(nu dev/dev-test.nu completions zsh)`. Nushell: `use dev/dev-test.nu *`. `nu dev/dev-test.nu self-check` is the runner's test. Verified here: `nu dev/dev-test.nu self-check` and `nu dev/dev-test.nu all` (fmt, clippy, openssl, unit, integration) green, plus `devenv shell -- dev-test --help`. Not run here (no nix builds in this worktree): `nu dev/dev-test.nu vm peers`, which on a KVM host is `nix build .#tests.x86_64-linux.peers -L`. `nix/tests/peers.nix` is unchanged.