Files
2026-10-20 16:35:11 -04:00

102 lines
23 KiB
Markdown

# WS-T: Integration test suite (written first, user request)
Part of [Phase 5: Consolidator](README.md). Contracts: [contracts.md](contracts.md).
| Field | Value |
|---|---|
| Status | Merged |
| Owner | |
| Depends on | none (uses the module's freeform `settings` for `[peers]` and a systemd override for `LoadCredential`, so it needs no module changes). F1 later swaps this for the first-class module options |
| Blocks | nothing technically; it is the acceptance gate: goes green at the end of [WS-E](ws-e-telemetry.md) (with A, B, C, D landed), then F2 promotes it to `checks` |
| Can start when | day one |
| Owned files | `nix/tests/peers.nix` (done), `flake.nix` (the `tests.<system>.peers` output only), `dev/dev-test.nu`, `devenv.nix` (nushell package, `dev-test` script, completion hook), `AGENTS.md` and `README.md` (command lists only, see seams), `docs/consolidator.md` (creates the file with only a `## Testing` section, see seams) |
| Do not touch | `src/**` (all streams), `nix/module.nix`, `nix/tests/vm.nix`, `systemd/skyg-linux-tracker.service` (F), `config.example.toml` (A/F), `tests/peers.rs` and `tests/common/mod.rs` (F; in-process variants by A/B live in their own test files, see [ws-a](ws-a-config.md), [ws-b](ws-b-registry.md)), `docs/api.md` (C) |
Shared-file seams:
- `flake.nix`: T owns it until F2. F2 later moves exactly one attribute, `tests.${system}.peers`, into `checks` (rename `checks.vm-peers`). Nobody else edits `flake.nix`; it already carries the uncommitted peers output.
- `devenv.nix`: T only.
- `README.md`: T edits only the command/test list near the existing commands; F4 owns the new peers section. Different headings, so no overlap. `AGENTS.md`: T replaces the command list; F4 adds only the MCP tool-count note.
- `docs/consolidator.md`: T creates it with a single `## Testing` section (the one-line `use dev/dev-test.nu *` and the completion setup). F4 adds every other section and must not edit `## Testing`.
## Goal
Executable acceptance criteria before the implementation. Two NixOS VMs each run the real `slinuxm` binary and consolidate each other: VM `alpha` declares `beta` as a peer, VM `beta` declares `alpha`, and each proxies the other's data. Expected to fail until WS-E lands, which is the point.
Note on the written test: `nix/tests/peers.nix` also contains a third node `gamma` (TLS-only, no peers) that `alpha` declares with a private CA via `ca_file`, covering the TLS variant. It is already written and, by design, fails at the first peers assertion (phase 2, "declared but token is still a placeholder -> auth rejected") against the current binary; phase 1 (boot, healthz, token minting, marker units, token-file placeholders) already passes.
## Tasks
- **T1** two machines `alpha` and `beta` with distinct hostnames, each with a distinct fake workload (a marker systemd unit `marker-alpha.service` / `marker-beta.service`, and a deliberately failing unit on one side) so proxied data is distinguishable from local data. **Done** in `nix/tests/peers.nix` (failing unit is `broken-beta.service`; `gamma` with `marker-gamma.service` added).
- **T2** each mints a read-only token via `sudo -u skyg-linux-tracker slinuxm token create consolidator-<other>`, writes the other's token to a root-only 0400 file (stand-in for an agenix path), and declares the other as a peer with that `token_file`. **Done** (files under `/var/lib/peer-tokens`, delivered via `LoadCredential`).
- **T3** assertions on both directions: `/api/v1/peers` shows the peer with server up, api up and auth ok; `GET /api/v1/peers/<other>/systemd/units/marker-<other>.service` returns the other machine's unit and not a local one; MCP `list_peers` and a `list_systemd_units` call with `peer` return the same data; response header `x-slinuxm-peer`. **Done** in the test script.
- **T4** token rejected: revoke the token on `beta`; `alpha` reports server up, api up, `auth = rejected`, `peer.api.auth_ok = 0`, and proxy calls fail with 503 naming the peer, without becoming a 401 for the caller; restore with a new token file and no restart, status recovers. **Done** in the script except the `peer.api.auth_ok = 0` metric check (needs WS-E; add it to the script, or leave to F2 which may query the collector/metrics endpoint, note in Log).
- **T5** outage: stop `beta`'s service: `alpha` flips server and api after `fail_threshold`, `alpha`'s own endpoints stay healthy, and recovery is detected; stop `beta` entirely (VM down) and check that `alpha`'s requests are bounded by timeouts. **Done** in the script.
- **T6** no chaining and read-only: a request carrying `x-slinuxm-hop` is rejected; non-GET to a proxy path is 405; a TLS variant with a test CA via `ca_file`. **Done** (405 for POST/DELETE, hop header 4xx, `gamma` over HTTPS).
- **T7** **Done**: `dev/dev-test.nu` with `unit`, `integration`, `lint`, `vm`, `all`, `ci`, `completions`, `self-check`; devenv `dev-test` script plus zsh/bash completion hook; AGENTS.md and README.md command lists; `docs/consolidator.md` `## Testing`. A simple test runner CLI written in **Nushell**: `dev/dev-test.nu` (devenv adds `pkgs.nushell`; also exposed as the devenv script `dev-test`, so `devenv shell -- dev-test unit` works from any shell, including an interactive one). One entry point so contributors and agents do not memorise cargo/nix incantations. The command was renamed from `test` to `dev-test` (user decision, see Log) because `test` is a shell builtin. Structure it as a Nushell script with a `main` and subcommands literally named `dev-test <sub>` (`export def "dev-test unit"`, ...) so `--help` is generated and completion comes from the typed signatures. Nushell binds `main` to the file stem, so only commands named `dev-test ...` complete as `dev-test unit`:
- `dev-test unit` runs `cargo test --lib --bins` (no sockets, no host access).
- `dev-test integration [filter]` runs `cargo test --test '*'` (integration targets only: plain, tls, integration, binary_smoke, `peers` once it exists), so `dev-test all` does not run the unit tests twice. `filter` is passed to cargo as a name filter. `--nocapture` is a libtest flag and is always placed after `--`.
- `dev-test lint` runs `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings` and the `cargo tree -i openssl` emptiness check.
- `dev-test vm [name]` runs a NixOS VM test. `name` is one of `vm` (from `checks`) or `peers` (`nix build .#tests.x86_64-linux.peers -L`; moves to `checks` when green). A missing `/dev/kvm` fails that step and says why, after earlier `ci` steps, and the summary table still prints. The step prints that the build is slow.
- `dev-test all` = lint, unit, integration, stopping at the first failure; `dev-test ci` = `all` plus `vm`. No args prints help.
- Each step prints a header and a final table (step, result, seconds); exit code non-zero on any failure (use `error make`/`exit 1`, never swallow cargo's exit status). Read-only against the host: no `/etc`, systemd or Docker changes (live-host checks stay manual, see Verification plan in the [README](README.md#verification-plan)).
- **Autocomplete (must work, and is a done-criterion):**
- Nushell: subcommands and flags complete natively once the script is loaded as a module. Dynamic values use custom completers (`def "nu-complete dev-test vm" [] { ["vm" "peers"] }` and `name: string@"nu-complete dev-test vm"`; integration filters complete from `cargo test --test '*' -- --list`, cached in `target/skyg-test-filters.cache` and keyed on the newest mtime of the Rust sources and Cargo manifests). A completer cannot update the caller's environment, so an env var does not survive Tab. `docs/consolidator.md` shows the one-line `use dev/dev-test.nu *` for a permanent setup.
- Other shells (the repo owner's login shell is zsh): `nu dev/dev-test.nu completions zsh|bash` prints a completion script generated from the same command table. `dev-test` is not a builtin, so the script only registers completion (`compadd` in zsh, `compgen` in bash) and does not define a function. Interactive `dev-test …` runs the devenv script on PATH. `self-check` invokes those completion functions and fails if the candidate list is empty. Interactive `devenv shell` sources the bash script and prints nothing, so `devenv shell -- dev-test completions zsh` stays pipe-clean. zsh is not started by that hook; after `compinit`, zsh users add `source <(nu dev/dev-test.nu completions zsh)` to `.zshrc` themselves. direnv does not source it for them.
- Verified by a self-test: `dev-test --help` and `dev-test vm --help` list every subcommand/flag, and a check in `dev/dev-test.nu` (`dev-test self-check`) fails if the completion table and the defined subcommands diverge.
- AGENTS.md and README replace their scattered command lists with `dev-test ...`, keeping the underlying commands visible in `--help`.
## Inputs
- [contracts.md](contracts.md) section 10 (test config contract) and sections 5 and 7 (REST/MCP shapes the script asserts). No fakes: the test runs the real binary.
- `nix/tests/peers.nix` is the source of truth for section 10; if the test must change, change it and update contracts.md in the same PR, then log in the affected streams.
## Done when
The test builds under `nix build .#tests.x86_64-linux.peers` (or equivalent) and evaluates, fails cleanly at the first peers assertion against the current binary, and is documented in `docs/consolidator.md`. `dev-test unit|integration|lint|vm|all` each run and report correctly on the current tree, and tab completion works in Nushell, zsh and bash (T7). It must pass at the end of WS-E, then move to `checks.vm-peers` (F2 does the move).
## Handoff
- To everyone: the `dev-test` runner and its subcommand names (referenced by AGENTS.md/README and by every stream's "Done when").
- To WS-F: a passing `peers.nix` (ready to promote) and any changes made to its assumptions, recorded in contracts.md section 10.
- To WS-A/B: the `dev-test integration` step picks up any `tests/peers*.rs` automatically.
## Log
- 2026-10-19: Split from the single phase plan 2026-10-19. `nix/tests/peers.nix` already written (T1 to T6) and failing at the first peers assertion by design; `flake.nix` exposes it as `tests.<system>.peers`. T7 not started. Ownership change from the single plan: `tests/peers.rs` moved to WS-F; WS-A/B in-process tests use their own files to avoid collisions.
- 2026-10-19 (WS-T): T7 done. `dev/test.nu` is a Nushell module: `use dev/test.nu *` gives `test unit|integration|lint|vm|all|ci` plus `completions` and `self-check`, and `nu dev/test.nu <sub>` dispatches through `main` for the devenv `test` script. Subcommands are declared as `test <sub>` (not `main <sub>`): with `use dev/test.nu *` Nushell binds the module's `main` to the file stem (`test`), and only commands literally named `test ...` complete and dispatch as `test unit`. Native subcommand help comes from the signatures; the entry help block lists the same table, and `test self-check` fails when the table, the defined commands, their flags or the help text diverge (mutation-tested: extra table row, extra command, extra flag). zsh/bash scripts are generated from the same table (`test completions zsh|bash`), sourced by `enterShell`. Verified: `test all` (fmt, clippy, openssl, unit, integration) green, `--ide-complete` returns subcommands, `vm`/filter completers, `bash -n`/`zsh -n` on the generated scripts. Not run here (no nix builds in this worktree): `nu dev/test.nu vm peers`; on a KVM host that is `nix build .#tests.x86_64-linux.peers -L`.
- Handoff 2026-10-19 (WS-T -> all): the runner is `dev/test.nu`; use `devenv shell -- test lint|unit|integration|vm|all|ci` or `nu dev/test.nu <subcommand>`. `nix/tests/peers.nix` is unchanged and still the acceptance gate. To WS-F: promote `tests.<system>.peers` to `checks.vm-peers` once `test vm peers` passes; `test integration` picks up `tests/peers*.rs` with no runner change. Caveat: per-subcommand `--help` is native in Nushell module mode; in script mode (`nu dev/test.nu vm --help`) the entry help lists every subcommand.
- 2026-10-19 (WS-T review round 1): Addressed every finding in `loop/T/review-1.md`. No contract change. Nothing outside the owned files. `TODO.md` is still unticked (not an owned file); the merger ticks T7.
1. `test integration --nocapture` with no filter now runs `cargo test --test '*' -- --nocapture`. Cargo rejects a bare `--nocapture`. `self-check` compares the step command, spawns the runner in a dry-run, and runs `cargo test --test '*' -- --nocapture --list`.
2. `test` stays the devenv script name (the plan's name, and `devenv shell -- test` already resolves it on PATH). Inside an interactive bash or zsh the builtin wins and `test unit` exits 0. The generated completion script defines a `test` function that dispatches known subcommands to this file and otherwise calls `builtin test`, so `test -f` still works. Interactive `devenv shell` sources the bash script. Docs tell zsh users to add `source <(nu dev/test.nu completions zsh)` themselves. `self-check` sources the bash and zsh scripts with a fake `nu` and checks both paths. Disagree with renaming: it would change the command every other stream already cites, and the PATH form the plan specified already works.
3. Filter completion is `target/skyg-test-filters.cache`, first line a mtime key of `src/**/*.rs`, `tests/**/*.rs`, `Cargo.toml` and `Cargo.lock`. The old `$env.SKYG_TEST_FILTERS` assignment died with the completer. `self-check` proves a second lookup does not re-list and a stale key does.
4. `enterShell` runs in devenv's bash, so the zsh branch never installed `compdef`. Removed. Docs say zsh users source the script from `.zshrc` after `compinit`. direnv will not carry the function.
5. The enterShell note goes to stderr and only when `$-` contains `i`, so `devenv shell -- test completions zsh` stays pipe-clean. Checked: a non-interactive `bash -c` does not take the `*i*` branch.
6. `test integration` is `cargo test --test '*'` (integration targets only). `test all` / `test ci` keep the separate `unit` step and no longer repeat it. Help text in `AGENTS.md` and `README.md` matches.
7. The openssl step passes only when stdout is empty and cargo exits 0, or when stderr contains `did not match any packages`. Any other cargo failure fails the step. `self-check` covers a clean miss, an empty success, a hit, and a lockfile-style error.
8. Missing `/dev/kvm` is an error inside the vm step body, not while the step list is built. `self-check` sets `SKYG_TEST_ASSUME_NO_KVM=1` and asserts `vm-steps` returns a step whose body reports `/dev/kvm`.
9. Script mode is `--wrapped`, so flags reach our parser. Unknown flags and extra positionals are errors. `--nocapture` is valid only on `integration`. `self-check` covers the parser and the CLI forms `unit --nocapture` and `integration a b`.
10. `TODO.md:28` T7 remains unticked on purpose.
- Handoff 2026-10-19 (WS-T review round 1 -> all): same subcommand names. Invoke as `devenv shell -- test <subcommand>` or `nu dev/test.nu <subcommand>`. Interactive `devenv shell` installs the bash wrapper; zsh needs the `.zshrc` line after `compinit`. `nu dev/test.nu self-check` is the runner's test. Not run here (no nix builds in this worktree): `nu dev/test.nu vm peers`, which on a KVM host is `nix build .#tests.x86_64-linux.peers -L`. `nix/tests/peers.nix` is unchanged.
- 2026-10-19 (WS-T, user decision): Renamed the runner from `test` to `dev-test` and the module from `dev/test.nu` to `dev/dev-test.nu`. Supersedes the review-round-1 note that kept the name `test`. `test` is a shell builtin, so an interactive `test unit` never reached the runner and the completion script had to define a `test` function that fell through to `builtin test`. `dev-test` is an external command: `devenv shell -- dev-test unit|integration|lint|vm|all|ci` and an interactive `dev-test …` both run the script. Completion scripts only register completion (`compadd` / `compgen`); they do not define a function. `_describe` emitted no candidates outside a completion widget, so zsh now uses `compadd` and `self-check` fails if the bash or zsh candidate list is empty. Nushell commands are named `dev-test <sub>` because the file stem is `dev-test`. argv/`--nocapture` and the filter cache are unchanged. `nix/tests/peers.nix` is unchanged. Also fixed a latent help bug: the `<(...)` example sat in a double-quoted string, so Nushell ran the completions command and `--help` inlined the whole zsh script; the line is concatenated now and `self-check` rejects an embedded script. Earlier dated Log entries keep the old name because they record what happened then; the Status line, T7 text and Handoff use the current name.
Stale `dev/test.nu` names left for the merger (other streams' files, not edited here):
- `plan/phase-5-consolidator/README.md` workstream index
- `plan/phase-5-consolidator/ws-a-config.md` Do not touch
- `plan/phase-5-consolidator/ws-f-nixos-docs.md` Do not touch
- `TODO.md` T7 line (unticked on purpose; the merger ticks it)
- Handoff 2026-10-19 (WS-T rename -> all): invoke the runner as `devenv shell -- dev-test <subcommand>` or `nu dev/dev-test.nu <subcommand>`. Interactive bash completion is installed by `devenv shell`; zsh, after `compinit`: `source <(nu dev/dev-test.nu completions zsh)`. Nushell: `use dev/dev-test.nu *`. `nu dev/dev-test.nu self-check` is the runner's test. Verified here: `nu dev/dev-test.nu self-check` and `nu dev/dev-test.nu all` (fmt, clippy, openssl, unit, integration) green, plus `devenv shell -- dev-test --help`. Not run here (no nix builds in this worktree): `nu dev/dev-test.nu vm peers`, which on a KVM host is `nix build .#tests.x86_64-linux.peers -L`. `nix/tests/peers.nix` is unchanged.
- 2026-10-19 (WS-T review round 2): Addressed `loop/T/review-r.md`. No contract change, no file outside the owned list, no new dependency (the self-check reuses the `nu` and shell probes already there).
1. **Blocking, module mode closed the caller's shell.** `run-steps` and `dev-test self-check` ended with `exit 1`, so with `use dev/dev-test.nu *` in `config.nu` any red `dev-test unit|all|ci` ended the Nushell session after the summary (`exit` cannot be caught). Both now raise with `error make`; `main` is the only place that exits, after printing the reason to stderr, so script mode keeps its non-zero status. `check-module-mode` guards it three ways: `exit` outside `main` is a problem, `(exit ...)` interpolated into a double-quoted string is a problem, and a real failing step (`SKYG_TEST_ASSUME_NO_KVM=1`) must be caught in module mode while `nu dev/dev-test.nu vm` still exits non-zero. Mutation-checked: restoring `exit 1` in `run-steps` fails the self-check with both the line number and the uncaught step.
- Found on the way: the openssl step's message was `$"cargo tree failed (exit ($found.exit_code)): ..."`. In Nushell `(...)` inside a double-quoted string is a subexpression, so that message would have called `exit` with cargo's status instead of raising a step failure (same class of bug, and it kills a module-mode session). Reworded to "failed with status (...)"; the new interpolation scan rejects a reoccurrence.
2. **Filter-cache key relative to the cwd.** `source-mtime-key` globbed `src/**/*.rs …` from the cwd and `FILTER_CACHE` was the relative path. From `src/` the globs found nothing, the key became "unknown" and the result was written to `src/target/...` under a key that never goes stale. The globs and the cache path are now anchored on `repo-root` (`$SCRIPT_PATH | path dirname | path dirname`); they go through `glob` because `ls` only expands patterns written literally in the source, and an unusable key (`unknown` or `none`) is neither read nor written (`cached-lines` returns a fresh listing). `check-filter-cache` now asserts the key from `src/` equals the key from the root and is usable, that the cache path is absolute and `<repo root>/target/skyg-test-filters.cache`, and that an unknown key writes no file. Mutation-checked: relative globs and a removed `unknown` guard each fail the self-check. The 33-file mtime scan costs ~25 ms including Nushell start.
3. **Stale `dev/test.nu` in other streams' plan files.** Not edited here, on purpose: the review agrees the merger owns it, and those rows are other streams' ownership guards. Still to change: `plan/phase-5-consolidator/README.md` (workstream index), `ws-a-config.md` and `ws-f-nixos-docs.md` (Do not touch rows).
4. **`TODO.md` T7.** Not edited here (outside WS-T's owned files, and the phase gate rejects changes outside them). Still says `dev/test.nu` and is unticked; the merger updates the name and ticks it.
5. **`AGENTS.md` wording.** "the same command works in an interactive shell" now reads "inside an interactive `devenv shell`". `docs/consolidator.md` says the cache sits at the repo root and that a failing check raises rather than closing the session.
- Handoff 2026-10-19 (WS-T review round 2 -> all): unchanged commands and subcommand names. `nu dev/dev-test.nu self-check` now also fails if any command exits (outside `main`), if a message interpolates a call to `exit`, if a failing step is not catchable in module mode, or if the filter cache is not anchored on the repo root. Merger: items 3 and 4 above.
- 2026-10-19 (WS-T review round 3): Addressed `loop/T/review-n2.md` (one blocking item). No contract change, no file outside the owned list, no new dependency.
1. **Blocking, `main` still exited on a dispatch error.** `export def --wrapped main` wrapped `dispatch-script-args` in `try`/`catch` and called `exit 1`. With `use dev/dev-test.nu *` the bare name `dev-test` is bound to `main`, so every unmatched call reaches it: `dev-test bogus` or a typo like `dev-test unti` ended the caller's Nushell session, and the comment claiming "only script mode reaches this entry point" was wrong. The `try`/`exit 1` is gone; the error propagates, which keeps module mode catchable (`try { dev-test bogus } catch { print caught }; print after` prints both lines) and still makes script mode exit 1 (`nu dev/dev-test.nu bogus` exits 1 with the unknown-subcommand message; Nushell gives an uncaught `error make` in a script status 1). The comment now says both modes land there and why `exit` must not be used. This supersedes the round-2 sentence "`main` is the only place that exits".
- `check-module-mode` no longer allows a window for `main`: any `exit` is a problem, including an inline one such as `catch { …; exit 1 }` (the scan drops comment lines, then matches `exit` at the start of a line or after `;`/`{`). The entry-point slicing (`entry`/`next`) is gone.
- New probes: module mode `try { dev-test bogus } catch { print 'caught' }; print 'after'` must print both lines with exit 0, next to the existing `vm` probe; script mode `nu dev/dev-test.nu bogus` must exit non-zero with the unknown-subcommand reason, so the "script mode still fails" guarantee is tested rather than assumed. `run-steps`' doc comment now says it raises (the process exits non-zero).
- Mutation-checked: restoring `try { … } catch { exit 1 }` in `main` fails the self-check on the `exit` line, and the module-mode `bogus` probe fails on its own.
- Handoff 2026-10-19 (WS-T review round 3 -> all): unchanged commands and subcommand names. `nu dev/dev-test.nu self-check` now fails if any line calls `exit` at all (including in `main`), if a message interpolates a call to `exit`, if a failing step or an unknown subcommand is not catchable in module mode, if `nu dev/dev-test.nu bogus` does not exit non-zero, or if the filter cache is not anchored on the repo root. Merger items from round 2 (stale `dev/test.nu` in other streams' plan files, `TODO.md` T7) stand.