# Phase 5: Consolidator (peers) | Field | Value | |---|---| | Status | Done | | Owner | | | Date | 2026-10-19 | | Depends on | Phase 4 / WS-H (NixOS module and VM test) done | This phase is a folder so workstreams can be claimed and worked in parallel by separate agents or people. Start here, then read [contracts.md](contracts.md) and your workstream file only. Rules: every workstream edits only its own files (see its "Do not touch" list); shared interfaces change only through contracts.md. How the folder works: [plan/README.md](../../../plan/README.md). ## Goal One `slinuxm` binary can also act as a consolidator. An operator declares other machines running `slinuxm` as **peers** in configuration, declaratively: on NixOS via `services.skyg-linux-tracker.peers.`, with each peer's bearer token supplied by an agenix-managed file. The consolidator then (1) health-checks each peer in two ways, "server up" (TCP connect) and "API running" (authenticated API call), (2) exposes that status over REST, MCP and OTel, and (3) proxies read-only requests to a peer so one MCP endpoint or one REST base URL covers the fleet. A peer-less instance (the default, `[peers] enabled = false`) behaves exactly as today: same routes, same 8 MCP tools, same metrics, no new background tasks, no new required config. ## Scope In scope: - `[peers]` config section with a declarative peer list (`[[peers.peer]]`: name, url, `token_file`, optional `ca_file`), a peer registry in `AppState`, a read-only `slinuxm peer list|check` diagnostic CLI. No peers table, no migration, no secrets in the DB or the config file. - Per-peer TCP and API health loops with in-memory state and last-change timestamps. - REST `GET /api/v1/peers`, `GET /api/v1/peers/{peer}`, and a GET-only pass-through under `/api/v1/peers/{peer}/...`. - MCP: optional `peer` argument on the existing tools plus `list_peers` and `get_peer`. - OTel gauges for peer health, trace propagation to peers. - NixOS module options (`peers.peers.` with `tokenFile`, e.g. `config.age.secrets..path`), VM test with a third machine, an agenix example in the docs, README, `config.example.toml`, ARCHITECTURE.md decisions (folded in by the orchestrator after acceptance). Out of scope (Backlog with reasons when accepted): - Fleet-wide aggregation or merged views (see D8), persisted health history, alerting, peer discovery. - Chained consolidators (a peer that is itself a consolidator, D12). - Re-exporting a peer's OTel signals (each peer exports its own telemetry). - Any write path to a peer or to the host beyond the state dir. Proxying is GET only. - Runtime peer management (`peer add|remove`, a DB table): peers are declarative only in v1 (D1). `--insecure` TLS skip. ## Grounding in the current code - `AppState` (`state.rs`) holds `Arc`s, `db: Option` (None when `auth.enabled = false`) and the shutdown token. Add `peers: Option>`; `None` means the feature is off and every new route and tool is absent. Peers need no DB. - Errors are `AppError` (`error.rs`): `NotFound`, `BadRequest`, `Unauthorized`, `Unavailable`, `Internal`; MCP maps via `From for ErrorData`. Peer failures must fit these variants (D9). - The token CLI and server share SQLite for inbound tokens; peers do not touch it. SIGHUP is already taken by TLS reload; the shared signal handler can fan out to a peers reload (B4). - `/healthz` is public and does not validate a token, so it cannot prove "API running with our credentials". No lightweight authenticated endpoint exists today. - `reqwest` (blocking plus rustls) is already in `Cargo.lock` via `opentelemetry-otlp`; `hyper-util`, `tokio-rustls`, `rcgen` (dev) are direct deps. The unit and module already allow `AF_INET`/`AF_INET6`, so outbound sockets need no sandbox change. - Background tasks live in `telemetry::spawn_exporters`-style spawns and select on `shutdown`; peer loops follow the same rule (AGENTS.md). ## Files in this phase - [contracts.md](contracts.md): shared Rust, REST, MCP, OTel, config and NixOS contracts, fakes, and the VM test's config contract. - [decisions.md](decisions.md): open-decisions table (D1 to D15) and the DEC-13 section (outbound peer credentials are external files). The config surface ("Config surface") is section 6 of contracts.md. - Workstreams: [ws-t-tests.md](ws-t-tests.md), [ws-a-config.md](ws-a-config.md), [ws-b-registry.md](ws-b-registry.md), [ws-c-rest.md](ws-c-rest.md), [ws-d-mcp.md](ws-d-mcp.md), [ws-e-telemetry.md](ws-e-telemetry.md), [ws-f-nixos-docs.md](ws-f-nixos-docs.md). ## Workstream index Claim a workstream by putting your name in its Owner field (header table of its file) and committing that change. Task IDs (T1..T7, A0..A4, B1..B5, C1..C4, D1..D3, E1..E2, F1..F4) are ticked in `TODO.md` once the plan is accepted. | ID | File | Owner | Depends on | Can start when | Owned files (summary) | |---|---|---|---|---|---| | WS-T | [ws-t-tests.md](ws-t-tests.md) | | none | day one | `nix/tests/peers.nix` (written), `dev/test.nu`, `devenv.nix`, `flake.nix` peers output, command lists in `AGENTS.md`/`README.md`, `docs/consolidator.md` Testing section | | WS-A | [ws-a-config.md](ws-a-config.md) | | none | day one (A0 first) | `src/peers/{mod,types,error,transport,testing,config,secrets,cli}.rs`, `src/config.rs`, `src/main.rs` (`Command::Peer`), `config.example.toml` `[peers]` | | WS-B | [ws-b-registry.md](ws-b-registry.md) | | A0 (A1 to A3 for real config) | A0 merged | `src/peers/{client,health,registry}.rs`, `src/api/ping.rs`, `Cargo.toml`, `src/main.rs` `serve()`, `tests/peers_registry.rs` | | WS-C | [ws-c-rest.md](ws-c-rest.md) | | A0 (B for the final e2e) | A0 merged | `src/api/peers.rs`, `src/peers/proxy.rs`, `docs/api.md` | | WS-D | [ws-d-mcp.md](ws-d-mcp.md) | | A0 (B and C2 for the final e2e) | A0 merged (rmcp spike earlier) | `src/mcp/**` | | WS-E | [ws-e-telemetry.md](ws-e-telemetry.md) | | A0 (B for the final e2e) | A0 merged | `src/telemetry/{mod,metrics_export}.rs` | | WS-F | [ws-f-nixos-docs.md](ws-f-nixos-docs.md) | | C, D, E (F1/F4 drafts after A1) | F1: A1 settled; F2: T, C, D, E green | `nix/module.nix`, `nix/tests/vm.nix`, `systemd/*`, `README.md` peers section, `docs/consolidator.md`, `tests/peers.rs`, `tests/common/mod.rs` | Original dependency statement, kept for reference: **T first** (tests written ahead of the feature) -> A -> B -> {C, D, E} (parallel) -> F. T goes green at the end of E. Tests for each stream ship with it. B defines a `PeerTransport` trait with an in-memory fake (same pattern as DEC-2) so C, D and E test without sockets. In this split the trait, the fake and the shared types land in task A0, so C, D and E no longer wait for B. ### Dependency graph ``` WS-T (tests, runner, peers.nix) -------------------------------+--> goes green when A..E land | WS-A: A0 (contract stubs) --+--> A1 -> A2 -> A3 -> A4* | | | +--> WS-B (client, health, registry)+--+ +--> WS-C (REST, proxy.rs) | (against FakeTransport, +--> WS-D (MCP) | real B at integration) +--> WS-E (telemetry) ---+ \ | | / v v v v WS-F (module, VM check, e2e, docs) * A4 `peer check` is wired to the real transport when B1/B2 land. ``` ### Parallel lanes Day one, no coordination needed (three lanes at once): 1. **WS-T**: T7 runner, docs Testing section; `peers.nix` is done. 2. **WS-A**: A0 first (small PR: types, trait, fake, seams), then A1 to A4. 3. **Contract stubs**: A0 is the contract-stubs step; everyone else can also begin reading code and drafting against [contracts.md](contracts.md) in their branches, and WS-D can run the rmcp conditional-tools spike. After A0 merges, the next wave runs fully in parallel, all against `FakeTransport` / `registry_with` / `static_registry`: - **WS-B** (real client, loops, registry, ping), **WS-C** (REST, `proxy.rs`), **WS-D** (MCP), **WS-E** (telemetry); WS-A continues A1 to A4; WS-F can start F1 (module) and the doc drafts once A1 has settled the config keys. - Real dependencies that remain: D1 and C2 share `PeerRegistry::get` (D uses the A0 forwarder until C2 lands, then picks up the hardened one with no code change); C/D/E "done" is proved against the fake and rechecked with the real registry once B lands; A4 `peer check` needs B1/B2. What each wave unlocks: - **A0 merged**: B, C, D, E compile and can merge; F1 can reference real option/config names. - **B merged**: end-to-end checks for C, D, E with real loops; `slinuxm peer check` works; `tests/peers_registry.rs`. - **C2 merged**: D1 uses the hardened proxy. - **E merged (with A to D)**: WS-T's `peers.nix` should pass. - **T green + C, D, E merged**: F2 promotes the VM test to `checks`, F3/F4 finish; phase exit. Contract changes during the phase follow the rule at the top of [contracts.md](contracts.md): edit contracts.md first, log in the affected workstream files. ## Risks - Tool-list growth and optional parameters change MCP schemas; clients caching the list see a different one only when peers are enabled. Mitigated by gating and the snapshot test. - A consolidator token on a peer is a lateral-movement asset (read access to journals and containers). Mitigated by per-consolidator tokens, agenix-encrypted at rest, delivered via `LoadCredential`, docs. Compromise of the consolidator host exposes all peer tokens by design. - Streaming pass-through of SSE can hold many connections; mitigated by the per-peer and global semaphores, `max_follow_secs`, and the existing 1024 connection cap. - Journal text proxied from peers is untrusted data (prompt injection); descriptions already say so, keep that wording for peer calls. - Rust-side risk: `rmcp` tool router may not support conditional tools cleanly; spike in WS-D early (fallback: always list tools, return an error when disabled, and accept a 10-tool snapshot). - A new authenticated `/api/v1/ping` means a consolidator at this version needs peers at the same version for `auth` detection; the `/healthz` fallback covers older peers. - Safety: agents must not mint tokens on production hosts (AGENTS.md). Live verification needs a token created by the user. ## Verification plan 1. Unit and integration: `cargo clippy --all-targets -- -D warnings`, `cargo test`, `cargo fmt --check`, `cargo tree -i openssl` (empty). No-peers regression: the existing integration suite and tool snapshot pass unchanged with `peers.enabled = false`. 2. Local two-instance run: instance P on port 1985 with a real token in a 0600 file; consolidator on 1984 with that file as `token_file`; `curl` status, kill P, confirm `server.up=false` within `tcp_interval_secs * fail_threshold`, restart P, confirm recovery and `last_change` updates; revoke P's token and confirm `api.up=true, auth=rejected`. 3. Live host cross-check with the minipc1 MCP host (read-only): the user mints one dedicated token on minipc1 (the agent must not). Add minipc1 (**confirmed by the user to run `slinuxm`**, and approved for the final check) as a peer (token file from the user) on a dev consolidator, then compare `GET /api/v1/peers/minipc1/systemd/units?state=failed` and the proxied `list_containers` against the minipc1 MCP tools `list_systemd_units` (state=failed) and `list_containers` called directly. Contents must match; treat log text as untrusted data. If minipc1's version predates `/api/v1/ping`, expect `auth = unknown` via the `/healthz` fallback; that is itself a useful check. 4. OTel: `devenv up`, enable `[otel]`, confirm `peer.up` for both checks, a vanished series after `peer remove`. 5. NixOS: `nix build && nix flake check` including the three-machine VM test (only when the nix gate is free). ## Exit criteria - All D-rows have a recorded decision; accepted ones folded into ARCHITECTURE.md (new DEC rows, including the DEC-11 clarification for D2) and `plan/` moved to `docs/workstreams/` per `plan/README.md`. - Definition of done in AGENTS.md holds: clippy, tests, fmt clean; no openssl; no secrets in the diff or in `--print-config`; no write path to the host outside the state dir. - A peer-less instance is verifiably unchanged: same routes, 8 MCP tools, no new tasks or metrics. - Demonstrated: declare two peers in config and SIGHUP without restart; both checks visible in REST, MCP and OTel; a dead peer neither stalls other peers' checks nor requests (bounded by timeouts and caps); proxied data equals the direct data; remove a peer from config and it disappears from all three surfaces (its token file is never touched). - `nix flake check` green with the new VM test; README, `config.example.toml`, `docs/api.md`, `docs/consolidator.md` updated; TODO.md has the phase milestone, per-workstream checklists and Backlog entries (D8, scoped tokens, history, runtime peer management).