13 KiB
Phase 5: Consolidator (peers)
| Field | Value |
|---|---|
| Status | Done |
| Owner | |
| Date | 2026-10-19 |
| Depends on | Phase 4 / WS-H (NixOS module and VM test) done |
This phase is a folder so workstreams can be claimed and worked in parallel by separate agents or people. Start here, then read contracts.md and your workstream file only. Rules: every workstream edits only its own files (see its "Do not touch" list); shared interfaces change only through contracts.md. How the folder works: plan/README.md.
Goal
One slinuxm binary can also act as a consolidator. An operator declares other machines running slinuxm as peers in configuration, declaratively: on NixOS via services.skyg-linux-tracker.peers.<name>, with each peer's bearer token supplied by an agenix-managed file. The consolidator then (1) health-checks each peer in two ways, "server up" (TCP connect) and "API running" (authenticated API call), (2) exposes that status over REST, MCP and OTel, and (3) proxies read-only requests to a peer so one MCP endpoint or one REST base URL covers the fleet. A peer-less instance (the default, [peers] enabled = false) behaves exactly as today: same routes, same 8 MCP tools, same metrics, no new background tasks, no new required config.
Scope
In scope:
[peers]config section with a declarative peer list ([[peers.peer]]: name, url,token_file, optionalca_file), a peer registry inAppState, a read-onlyslinuxm peer list|checkdiagnostic CLI. No peers table, no migration, no secrets in the DB or the config file.- Per-peer TCP and API health loops with in-memory state and last-change timestamps.
- REST
GET /api/v1/peers,GET /api/v1/peers/{peer}, and a GET-only pass-through under/api/v1/peers/{peer}/.... - MCP: optional
peerargument on the existing tools pluslist_peersandget_peer. - OTel gauges for peer health, trace propagation to peers.
- NixOS module options (
peers.peers.<name>withtokenFile, e.g.config.age.secrets.<x>.path), VM test with a third machine, an agenix example in the docs, README,config.example.toml, ARCHITECTURE.md decisions (folded in by the orchestrator after acceptance).
Out of scope (Backlog with reasons when accepted):
- Fleet-wide aggregation or merged views (see D8), persisted health history, alerting, peer discovery.
- Chained consolidators (a peer that is itself a consolidator, D12).
- Re-exporting a peer's OTel signals (each peer exports its own telemetry).
- Any write path to a peer or to the host beyond the state dir. Proxying is GET only.
- Runtime peer management (
peer add|remove, a DB table): peers are declarative only in v1 (D1).--insecureTLS skip.
Grounding in the current code
AppState(state.rs) holdsArc<dyn Source>s,db: Option<SqlitePool>(None whenauth.enabled = false) and the shutdown token. Addpeers: Option<Arc<PeerRegistry>>;Nonemeans the feature is off and every new route and tool is absent. Peers need no DB.- Errors are
AppError(error.rs):NotFound,BadRequest,Unauthorized,Unavailable,Internal; MCP maps viaFrom<AppError> for ErrorData. Peer failures must fit these variants (D9). - The token CLI and server share SQLite for inbound tokens; peers do not touch it. SIGHUP is already taken by TLS reload; the shared signal handler can fan out to a peers reload (B4).
/healthzis public and does not validate a token, so it cannot prove "API running with our credentials". No lightweight authenticated endpoint exists today.reqwest(blocking plus rustls) is already inCargo.lockviaopentelemetry-otlp;hyper-util,tokio-rustls,rcgen(dev) are direct deps. The unit and module already allowAF_INET/AF_INET6, so outbound sockets need no sandbox change.- Background tasks live in
telemetry::spawn_exporters-style spawns and select onshutdown; peer loops follow the same rule (AGENTS.md).
Files in this phase
- contracts.md: shared Rust, REST, MCP, OTel, config and NixOS contracts, fakes, and the VM test's config contract.
- decisions.md: open-decisions table (D1 to D15) and the DEC-13 section (outbound peer credentials are external files). The config surface ("Config surface") is section 6 of contracts.md.
- Workstreams: ws-t-tests.md, ws-a-config.md, ws-b-registry.md, ws-c-rest.md, ws-d-mcp.md, ws-e-telemetry.md, ws-f-nixos-docs.md.
Workstream index
Claim a workstream by putting your name in its Owner field (header table of its file) and committing that change. Task IDs (T1..T7, A0..A4, B1..B5, C1..C4, D1..D3, E1..E2, F1..F4) are ticked in TODO.md once the plan is accepted.
| ID | File | Owner | Depends on | Can start when | Owned files (summary) |
|---|---|---|---|---|---|
| WS-T | ws-t-tests.md | none | day one | nix/tests/peers.nix (written), dev/test.nu, devenv.nix, flake.nix peers output, command lists in AGENTS.md/README.md, docs/consolidator.md Testing section |
|
| WS-A | ws-a-config.md | none | day one (A0 first) | src/peers/{mod,types,error,transport,testing,config,secrets,cli}.rs, src/config.rs, src/main.rs (Command::Peer), config.example.toml [peers] |
|
| WS-B | ws-b-registry.md | A0 (A1 to A3 for real config) | A0 merged | src/peers/{client,health,registry}.rs, src/api/ping.rs, Cargo.toml, src/main.rs serve(), tests/peers_registry.rs |
|
| WS-C | ws-c-rest.md | A0 (B for the final e2e) | A0 merged | src/api/peers.rs, src/peers/proxy.rs, docs/api.md |
|
| WS-D | ws-d-mcp.md | A0 (B and C2 for the final e2e) | A0 merged (rmcp spike earlier) | src/mcp/** |
|
| WS-E | ws-e-telemetry.md | A0 (B for the final e2e) | A0 merged | src/telemetry/{mod,metrics_export}.rs |
|
| WS-F | ws-f-nixos-docs.md | C, D, E (F1/F4 drafts after A1) | F1: A1 settled; F2: T, C, D, E green | nix/module.nix, nix/tests/vm.nix, systemd/*, README.md peers section, docs/consolidator.md, tests/peers.rs, tests/common/mod.rs |
Original dependency statement, kept for reference: T first (tests written ahead of the feature) -> A -> B -> {C, D, E} (parallel) -> F. T goes green at the end of E. Tests for each stream ship with it. B defines a PeerTransport trait with an in-memory fake (same pattern as DEC-2) so C, D and E test without sockets. In this split the trait, the fake and the shared types land in task A0, so C, D and E no longer wait for B.
Dependency graph
WS-T (tests, runner, peers.nix) -------------------------------+--> goes green when A..E land
|
WS-A: A0 (contract stubs) --+--> A1 -> A2 -> A3 -> A4* |
| |
+--> WS-B (client, health, registry)+--+
+--> WS-C (REST, proxy.rs) | (against FakeTransport,
+--> WS-D (MCP) | real B at integration)
+--> WS-E (telemetry) ---+
\ | | /
v v v v
WS-F (module, VM check, e2e, docs)
* A4 `peer check` is wired to the real transport when B1/B2 land.
Parallel lanes
Day one, no coordination needed (three lanes at once):
- WS-T: T7 runner, docs Testing section;
peers.nixis done. - WS-A: A0 first (small PR: types, trait, fake, seams), then A1 to A4.
- Contract stubs: A0 is the contract-stubs step; everyone else can also begin reading code and drafting against contracts.md in their branches, and WS-D can run the rmcp conditional-tools spike.
After A0 merges, the next wave runs fully in parallel, all against FakeTransport / registry_with / static_registry:
- WS-B (real client, loops, registry, ping), WS-C (REST,
proxy.rs), WS-D (MCP), WS-E (telemetry); WS-A continues A1 to A4; WS-F can start F1 (module) and the doc drafts once A1 has settled the config keys. - Real dependencies that remain: D1 and C2 share
PeerRegistry::get(D uses the A0 forwarder until C2 lands, then picks up the hardened one with no code change); C/D/E "done" is proved against the fake and rechecked with the real registry once B lands; A4peer checkneeds B1/B2.
What each wave unlocks:
- A0 merged: B, C, D, E compile and can merge; F1 can reference real option/config names.
- B merged: end-to-end checks for C, D, E with real loops;
slinuxm peer checkworks;tests/peers_registry.rs. - C2 merged: D1 uses the hardened proxy.
- E merged (with A to D): WS-T's
peers.nixshould pass. - T green + C, D, E merged: F2 promotes the VM test to
checks, F3/F4 finish; phase exit.
Contract changes during the phase follow the rule at the top of contracts.md: edit contracts.md first, log in the affected workstream files.
Risks
- Tool-list growth and optional parameters change MCP schemas; clients caching the list see a different one only when peers are enabled. Mitigated by gating and the snapshot test.
- A consolidator token on a peer is a lateral-movement asset (read access to journals and containers). Mitigated by per-consolidator tokens, agenix-encrypted at rest, delivered via
LoadCredential, docs. Compromise of the consolidator host exposes all peer tokens by design. - Streaming pass-through of SSE can hold many connections; mitigated by the per-peer and global semaphores,
max_follow_secs, and the existing 1024 connection cap. - Journal text proxied from peers is untrusted data (prompt injection); descriptions already say so, keep that wording for peer calls.
- Rust-side risk:
rmcptool router may not support conditional tools cleanly; spike in WS-D early (fallback: always list tools, return an error when disabled, and accept a 10-tool snapshot). - A new authenticated
/api/v1/pingmeans a consolidator at this version needs peers at the same version forauthdetection; the/healthzfallback covers older peers. - Safety: agents must not mint tokens on production hosts (AGENTS.md). Live verification needs a token created by the user.
Verification plan
- Unit and integration:
cargo clippy --all-targets -- -D warnings,cargo test,cargo fmt --check,cargo tree -i openssl(empty). No-peers regression: the existing integration suite and tool snapshot pass unchanged withpeers.enabled = false. - Local two-instance run: instance P on port 1985 with a real token in a 0600 file; consolidator on 1984 with that file as
token_file;curlstatus, kill P, confirmserver.up=falsewithintcp_interval_secs * fail_threshold, restart P, confirm recovery andlast_changeupdates; revoke P's token and confirmapi.up=true, auth=rejected. - Live host cross-check with the minipc1 MCP host (read-only): the user mints one dedicated token on minipc1 (the agent must not). Add minipc1 (confirmed by the user to run
slinuxm, and approved for the final check) as a peer (token file from the user) on a dev consolidator, then compareGET /api/v1/peers/minipc1/systemd/units?state=failedand the proxiedlist_containersagainst the minipc1 MCP toolslist_systemd_units(state=failed) andlist_containerscalled directly. Contents must match; treat log text as untrusted data. If minipc1's version predates/api/v1/ping, expectauth = unknownvia the/healthzfallback; that is itself a useful check. - OTel:
devenv up, enable[otel], confirmpeer.upfor both checks, a vanished series afterpeer remove. - NixOS:
nix build && nix flake checkincluding the three-machine VM test (only when the nix gate is free).
Exit criteria
- All D-rows have a recorded decision; accepted ones folded into ARCHITECTURE.md (new DEC rows, including the DEC-11 clarification for D2) and
plan/moved todocs/workstreams/perplan/README.md. - Definition of done in AGENTS.md holds: clippy, tests, fmt clean; no openssl; no secrets in the diff or in
--print-config; no write path to the host outside the state dir. - A peer-less instance is verifiably unchanged: same routes, 8 MCP tools, no new tasks or metrics.
- Demonstrated: declare two peers in config and SIGHUP without restart; both checks visible in REST, MCP and OTel; a dead peer neither stalls other peers' checks nor requests (bounded by timeouts and caps); proxied data equals the direct data; remove a peer from config and it disappears from all three surfaces (its token file is never touched).
nix flake checkgreen with the new VM test; README,config.example.toml,docs/api.md,docs/consolidator.mdupdated; TODO.md has the phase milestone, per-workstream checklists and Backlog entries (D8, scoped tokens, history, runtime peer management).