Files
skyg-linux-tracker/plan/phase-5-consolidator/ws-f-nixos-docs.md
T

12 KiB

WS-F: NixOS, docs, end-to-end tests

Part of Phase 5: Consolidator. Contracts: contracts.md sections 5 to 10.

Field Value
Status In progress (F1 done; F2 to F4 pending)
Owner
Depends on C, D, E (original plan). F1 (module) and F4 (docs drafts) can begin once A1 settles the config keys, since they depend only on contracts sections 6 and 9; F2/F3 need the other streams
Blocks phase exit
Can start when F1 and doc drafts: after A0/A1 land. F2: after C, D, E and WS-T are green. F3: after B and C
Owned files nix/module.nix, nix/tests/module-peers.nix (eval tests for the module, new in F1), nix/tests/vm.nix, systemd/skyg-linux-tracker.service, README.md (peers section), docs/consolidator.md (new; except the ## Testing section owned by WS-T), tests/peers.rs, tests/common/mod.rs (shared helpers, append-only with B), config.example.toml (merge with A1: comments and example peers, not the keys), flake.nix (only the move of tests.<system>.peers into checks.vm-peers plus checks.module-peers, in F2), AGENTS.md (MCP tool count note)
Do not touch src/** (all code streams; report bugs to the owning stream's Log), nix/tests/peers.nix except to adapt it to the first-class options in F2 (coordinate with WS-T, update contracts section 10), dev/test.nu, devenv.nix (T), docs/api.md (C), src/config.rs (A)

Shared-file seams:

  • flake.nix: WS-T owns it until F2; F2 then moves exactly one attribute into checks.
  • README.md / AGENTS.md: WS-T owns the command lists; F4 owns the peers section and the MCP tool count note (tool count becomes "8, or 10 with peers enabled").
  • config.example.toml: A1 owns the keys and defaults; F reviews that it is in sync with docs/consolidator.md and adds only commentary/examples.
  • tests/common/mod.rs: append-only; B may have added helpers already.

Goal

Shippable, documented, verified.

Tasks

  • F1 module: peers.enable; peers.peers.<name> = { url; tokenFile; caFile ? null; enable ? true; allowPlainHttp ? false; }; peers.settings for intervals/caps. The module renders [[peers.peer]] entries into the TOML with the paths only and adds one LoadCredential=peer-<name>:<tokenFile> per peer, pointing each token_file at /run/credentials/skyg-linux-tracker.service/peer-<name> so the service user needs no read access to /run/agenix/* (same mechanism as TLS, DEC-12). tokenFile is typed as a string (config.age.secrets.<x>.path), never a path literal, so secrets cannot enter the Nix store; an eval assertion rejects a store path. Config change reloads via ExecReload (SIGHUP) with reloadIfChanged, else restarts (implemented with reloadTriggers plus a restart trigger derived from the settings SIGHUP cannot apply; see the Log). Assertions: at least one enabled peer when peers.enable; names valid and unique. Also add RefreshOnReload=credentials as nix/tests/peers.nix does (contracts section 10) if the systemd version supports it.
  • F2 promote WS-T to checks, switched to the first-class module options (peers.peers.<name>.tokenFile); add nix/tests/module-peers.nix as checks.module-peers so nix flake check and dev-test ci run the eval cases (it is eval-only, no VM); plus a three-machine variant (consolidator, peer1, peer2) for fan-in. Mint tokens on peers, write them to root-owned 0400 files on the consolidator (stand-ins for agenix paths) and declare the peers through the module, assert status via REST and MCP, stop peer2's service and assert server/api flip, assert proxied units match a direct call, then rotate a token file and revoke the token on the peer: auth flips to rejected, then back after rewriting the file, with no restart. Remove a peer from the config, SIGHUP, assert the series is gone.
  • F3 tests/peers.rs end-to-end (plain, TLS with an rcgen CA via --ca-file, wrong CA rejected, redirect not followed, oversize body, slow-loris peer, 64-peer cap, cancel on shutdown).
  • F4 README section, docs/consolidator.md (threat model, token handling, firewall note: peers must allow the consolidator), a worked agenix example (secrets.nix entry, age.secrets.peer-minipc1.file, peers.peers.minipc1.tokenFile = config.age.secrets.peer-minipc1.path), AGENTS.md MCP tool count note.

Inputs

  • Contracts section 9 (Nix options), 6 (TOML), 10 (test contract), 5 and 7 (what the tests assert). Fakes: none for F2 (real binary); F3 uses in-process servers, not the fake transport.
  • Constraint: nix builds only when the nix gate is free (AGENTS.md).

Done when

nix flake check passes including the new VM test; docs and example config in sync.

Handoff

  • Phase exit evidence: nix flake check output, docs, example config, the Verification plan results in the README.
  • To the orchestrator: ARCHITECTURE.md decision rows (DEC-13 and the DEC-11 clarification), TODO.md checklists, and moving plan/phase-5-consolidator/ to docs/workstreams/.

Log

  • 2026-10-19: Split from the single phase plan 2026-10-19. tests/peers.rs ownership moved here from WS-T; docs/consolidator.md shares one ## Testing section with WS-T.
  • 2026-10-19: F1 done (commit F: F1 NixOS peers options with LoadCredential token delivery).
    • Options under services.skyg-linux-tracker.peers: enable (off by default), settings (freeform [peers] tuning, snake_case, no secrets) and peers.<name> = { url; tokenFile; caFile ? null; enable ? true; allowPlainHttp ? false; }.
    • Rendering: one [[peers.peer]] per peer with enable = true, with name, url, allow_plain_http, ca_file (only when set) and token_file = /run/credentials/skyg-linux-tracker.service/peer-<name>. Peers with enable = false are left out of both the TOML and LoadCredential. A peer-less config gains no [peers] section, so existing deployments keep the same generated file.
    • Credentials: LoadCredential=peer-<name>:<tokenFile> per enabled peer, merged with the TLS cert/key entries; RefreshOnReload=credentials now applies when TLS or peers are on; ExecReload stays SIGHUP.
    • Reload vs restart: the unit always sets reloadTriggers = [ configFile ] and restartTriggers = [ restartTrigger ], where restartTrigger is the JSON of settings with [peers] reduced to the keys SIGHUP cannot apply (enabled, deny_cidrs, max_response_bytes, max_concurrent_per_peer, max_concurrent_total). switch-to-configuration reloads only when the reload trigger is the sole difference in the unit file and restarts otherwise, so a peer entry or interval edit reloads (and the credential copy is refreshed) while a port, bind, auth, sources, otel or limit edit restarts. That keeps peers from turning unrelated config edits into silent no-ops. This is deliberate instead of reloadIfChanged: that flag would also reload on unrelated unit changes such as a package upgrade, keeping the old binary running. The trigger is a plain string (the JSON), so evaluating the unit needs no build.
    • Assertions: declaring peers.<name> requires peers.enable; peers.enable requires at least one enabled peer (or the legacy freeform list); names match ^[a-z0-9][a-z0-9_-]{0,31}$ (uniqueness is by attribute name); at most 64 peers; tokenFile is absolute, outside the store, and free of :/whitespace (it would corrupt the LoadCredential line; a store path here has already been copied into the local store, so the assertion blocks the deployment, not the copy, and the secret must be rotated); caFile is absolute when set; enabled and peer may not be set in peers.settings or settings.peers while the first-class options are in use. peersSection puts the freeform keys first and the first-class enabled/peer last, and settings re-applies that section, so a stale freeform value cannot switch off a feature the module believes is on. The pre-F1 freeform style stays valid, so nix/tests/peers.nix evaluates unchanged (verified: alpha keeps its two peers, gamma none).
    • Tests: nix/tests/module-peers.nix (new file, eval only, no builds) renders the config and the unit and checks 40 cases: peer-less default, TLS-only unchanged, TLS+peers credential list, rendering of enabled/disabled peers, token path never in the config, the reload/restart split (peer url and interval changes leave the restart trigger alone and move the reload trigger; port, auth, deny_cidrs and max_concurrent_total changes move the restart trigger), freeform compatibility and every assertion. Not wired into flake.nix (F2 owns that file and adds checks.module-peers); run it with: nix eval --impure --json --expr 'let f = builtins.getFlake (toString ./.); in import ./nix/tests/module-peers.nix { pkgs = f.inputs.nixpkgs.legacyPackages.x86_64-linux; module = f.nixosModules.default; }'
    • Not run (no nix builds in this worktree): nix build .#checks.x86_64-linux.vm -L, nix build .#tests.x86_64-linux.peers -L, nix flake check. These must pass before F1 is merged (the nix gate was busy for this round): vm covers the unchanged peer-less/TLS path, peers is the only place the mkMerge of LoadCredential is proven together with the test's own LoadCredential/RefreshOnReload override on the legacy freeform path. The eval test above was run: 40/40 pass, and the legacy nix/tests/peers.nix still evaluates to the same TOML, credentials and trigger shape.
    • Handoff: F1 is complete pending the VM runs above; F2 can switch nix/tests/peers.nix to peers.enable/peers.peers.<name>.tokenFile (the TOML it produced is identical), move the test into checks.vm-peers, add checks.module-peers and the three-machine variant. F4 still owes the README peers section, docs/consolidator.md and the DEC-14 note (a consolidator token reaches every peer's data). systemd/skyg-linux-tracker.service is unchanged: the example is a manual install without peers.
  • 2026-10-19: F1 review round 1 (loop/F/review-n4.md) addressed.
    • Blocking: with peers on, the config file no longer moves wholesale to reloadTriggers. reloadTriggers = [ configFile ] stays and restartTriggers = [ restartTrigger ], where the trigger is the JSON of settings with [peers] reduced to enabled, deny_cidrs, max_response_bytes, max_concurrent_per_peer and max_concurrent_total (the keys PeerRegistry::reload refuses). A peer entry, an interval or a timeout change now reloads; a port, bind, allowed hosts, auth, otel/metrics or sources change restarts, so switch never leaves the service on stale settings. enabled is in the restart set because the router mounts the peer routes once, at startup.
    • Precedence: peersSection is freeform keys first and the first-class enabled/peer last, and settings re-applies it after recursiveUpdate, so peers.settings.enabled and settings.peers.enabled/peer can no longer override or bypass the first-class options; an assertion rejects that mix while the first-class options are in use.
    • Assertions added: peers.enable now requires at least one enabled peer; at most 64 peers (the binary's MAX_PEERS); tokenFile may not contain :, a space or a tab (it would corrupt LoadCredential=peer-<name>:<path>). The store-path comment and the tokenFile description no longer claim the secret never enters the store: the assertion blocks deployment, after the interpolation has already copied the file into the local store, so such a secret must be rotated. The caFile description now says the service user reads it directly, unlike the token, so a root-only file fails at startup.
    • Eval cases: 40, the new ones proving a peer url change and an api_interval_secs change move only reloadTriggers while port, auth, deny_cidrs and max_concurrent_total changes move restartTriggers, plus the new assertions and both enabled-conflict directions.
    • Recorded for F2: nix/tests/module-peers.nix is now in the Owned files row, and F2 must add it as checks.module-peers so nix flake check and dev-test ci run it.
    • Still open: the two VM tests, see the "Not run" note above.