Files
2026-10-20 16:35:11 -04:00

13 KiB

Phase 5: Consolidator (peers)

Field Value
Status Done
Owner
Date 2026-10-19
Depends on Phase 4 / WS-H (NixOS module and VM test) done

This phase is a folder so workstreams can be claimed and worked in parallel by separate agents or people. Start here, then read contracts.md and your workstream file only. Rules: every workstream edits only its own files (see its "Do not touch" list); shared interfaces change only through contracts.md. How the folder works: plan/README.md.

Goal

One slinuxm binary can also act as a consolidator. An operator declares other machines running slinuxm as peers in configuration, declaratively: on NixOS via services.skyg-linux-tracker.peers.<name>, with each peer's bearer token supplied by an agenix-managed file. The consolidator then (1) health-checks each peer in two ways, "server up" (TCP connect) and "API running" (authenticated API call), (2) exposes that status over REST, MCP and OTel, and (3) proxies read-only requests to a peer so one MCP endpoint or one REST base URL covers the fleet. A peer-less instance (the default, [peers] enabled = false) behaves exactly as today: same routes, same 8 MCP tools, same metrics, no new background tasks, no new required config.

Scope

In scope:

  • [peers] config section with a declarative peer list ([[peers.peer]]: name, url, token_file, optional ca_file), a peer registry in AppState, a read-only slinuxm peer list|check diagnostic CLI. No peers table, no migration, no secrets in the DB or the config file.
  • Per-peer TCP and API health loops with in-memory state and last-change timestamps.
  • REST GET /api/v1/peers, GET /api/v1/peers/{peer}, and a GET-only pass-through under /api/v1/peers/{peer}/....
  • MCP: optional peer argument on the existing tools plus list_peers and get_peer.
  • OTel gauges for peer health, trace propagation to peers.
  • NixOS module options (peers.peers.<name> with tokenFile, e.g. config.age.secrets.<x>.path), VM test with a third machine, an agenix example in the docs, README, config.example.toml, ARCHITECTURE.md decisions (folded in by the orchestrator after acceptance).

Out of scope (Backlog with reasons when accepted):

  • Fleet-wide aggregation or merged views (see D8), persisted health history, alerting, peer discovery.
  • Chained consolidators (a peer that is itself a consolidator, D12).
  • Re-exporting a peer's OTel signals (each peer exports its own telemetry).
  • Any write path to a peer or to the host beyond the state dir. Proxying is GET only.
  • Runtime peer management (peer add|remove, a DB table): peers are declarative only in v1 (D1). --insecure TLS skip.

Grounding in the current code

  • AppState (state.rs) holds Arc<dyn Source>s, db: Option<SqlitePool> (None when auth.enabled = false) and the shutdown token. Add peers: Option<Arc<PeerRegistry>>; None means the feature is off and every new route and tool is absent. Peers need no DB.
  • Errors are AppError (error.rs): NotFound, BadRequest, Unauthorized, Unavailable, Internal; MCP maps via From<AppError> for ErrorData. Peer failures must fit these variants (D9).
  • The token CLI and server share SQLite for inbound tokens; peers do not touch it. SIGHUP is already taken by TLS reload; the shared signal handler can fan out to a peers reload (B4).
  • /healthz is public and does not validate a token, so it cannot prove "API running with our credentials". No lightweight authenticated endpoint exists today.
  • reqwest (blocking plus rustls) is already in Cargo.lock via opentelemetry-otlp; hyper-util, tokio-rustls, rcgen (dev) are direct deps. The unit and module already allow AF_INET/AF_INET6, so outbound sockets need no sandbox change.
  • Background tasks live in telemetry::spawn_exporters-style spawns and select on shutdown; peer loops follow the same rule (AGENTS.md).

Files in this phase

Workstream index

Claim a workstream by putting your name in its Owner field (header table of its file) and committing that change. Task IDs (T1..T7, A0..A4, B1..B5, C1..C4, D1..D3, E1..E2, F1..F4) are ticked in TODO.md once the plan is accepted.

ID File Owner Depends on Can start when Owned files (summary)
WS-T ws-t-tests.md none day one nix/tests/peers.nix (written), dev/test.nu, devenv.nix, flake.nix peers output, command lists in AGENTS.md/README.md, docs/consolidator.md Testing section
WS-A ws-a-config.md none day one (A0 first) src/peers/{mod,types,error,transport,testing,config,secrets,cli}.rs, src/config.rs, src/main.rs (Command::Peer), config.example.toml [peers]
WS-B ws-b-registry.md A0 (A1 to A3 for real config) A0 merged src/peers/{client,health,registry}.rs, src/api/ping.rs, Cargo.toml, src/main.rs serve(), tests/peers_registry.rs
WS-C ws-c-rest.md A0 (B for the final e2e) A0 merged src/api/peers.rs, src/peers/proxy.rs, docs/api.md
WS-D ws-d-mcp.md A0 (B and C2 for the final e2e) A0 merged (rmcp spike earlier) src/mcp/**
WS-E ws-e-telemetry.md A0 (B for the final e2e) A0 merged src/telemetry/{mod,metrics_export}.rs
WS-F ws-f-nixos-docs.md C, D, E (F1/F4 drafts after A1) F1: A1 settled; F2: T, C, D, E green nix/module.nix, nix/tests/vm.nix, systemd/*, README.md peers section, docs/consolidator.md, tests/peers.rs, tests/common/mod.rs

Original dependency statement, kept for reference: T first (tests written ahead of the feature) -> A -> B -> {C, D, E} (parallel) -> F. T goes green at the end of E. Tests for each stream ship with it. B defines a PeerTransport trait with an in-memory fake (same pattern as DEC-2) so C, D and E test without sockets. In this split the trait, the fake and the shared types land in task A0, so C, D and E no longer wait for B.

Dependency graph

 WS-T  (tests, runner, peers.nix)  -------------------------------+--> goes green when A..E land
                                                                  |
 WS-A:  A0 (contract stubs) --+--> A1 -> A2 -> A3 -> A4*         |
                              |                                   |
                              +--> WS-B (client, health, registry)+--+
                              +--> WS-C (REST, proxy.rs)           |  (against FakeTransport,
                              +--> WS-D (MCP)                      |   real B at integration)
                              +--> WS-E (telemetry)             ---+
                                          \  |  |  /
                                           v v v v
                                            WS-F (module, VM check, e2e, docs)

 * A4 `peer check` is wired to the real transport when B1/B2 land.

Parallel lanes

Day one, no coordination needed (three lanes at once):

  1. WS-T: T7 runner, docs Testing section; peers.nix is done.
  2. WS-A: A0 first (small PR: types, trait, fake, seams), then A1 to A4.
  3. Contract stubs: A0 is the contract-stubs step; everyone else can also begin reading code and drafting against contracts.md in their branches, and WS-D can run the rmcp conditional-tools spike.

After A0 merges, the next wave runs fully in parallel, all against FakeTransport / registry_with / static_registry:

  • WS-B (real client, loops, registry, ping), WS-C (REST, proxy.rs), WS-D (MCP), WS-E (telemetry); WS-A continues A1 to A4; WS-F can start F1 (module) and the doc drafts once A1 has settled the config keys.
  • Real dependencies that remain: D1 and C2 share PeerRegistry::get (D uses the A0 forwarder until C2 lands, then picks up the hardened one with no code change); C/D/E "done" is proved against the fake and rechecked with the real registry once B lands; A4 peer check needs B1/B2.

What each wave unlocks:

  • A0 merged: B, C, D, E compile and can merge; F1 can reference real option/config names.
  • B merged: end-to-end checks for C, D, E with real loops; slinuxm peer check works; tests/peers_registry.rs.
  • C2 merged: D1 uses the hardened proxy.
  • E merged (with A to D): WS-T's peers.nix should pass.
  • T green + C, D, E merged: F2 promotes the VM test to checks, F3/F4 finish; phase exit.

Contract changes during the phase follow the rule at the top of contracts.md: edit contracts.md first, log in the affected workstream files.

Risks

  • Tool-list growth and optional parameters change MCP schemas; clients caching the list see a different one only when peers are enabled. Mitigated by gating and the snapshot test.
  • A consolidator token on a peer is a lateral-movement asset (read access to journals and containers). Mitigated by per-consolidator tokens, agenix-encrypted at rest, delivered via LoadCredential, docs. Compromise of the consolidator host exposes all peer tokens by design.
  • Streaming pass-through of SSE can hold many connections; mitigated by the per-peer and global semaphores, max_follow_secs, and the existing 1024 connection cap.
  • Journal text proxied from peers is untrusted data (prompt injection); descriptions already say so, keep that wording for peer calls.
  • Rust-side risk: rmcp tool router may not support conditional tools cleanly; spike in WS-D early (fallback: always list tools, return an error when disabled, and accept a 10-tool snapshot).
  • A new authenticated /api/v1/ping means a consolidator at this version needs peers at the same version for auth detection; the /healthz fallback covers older peers.
  • Safety: agents must not mint tokens on production hosts (AGENTS.md). Live verification needs a token created by the user.

Verification plan

  1. Unit and integration: cargo clippy --all-targets -- -D warnings, cargo test, cargo fmt --check, cargo tree -i openssl (empty). No-peers regression: the existing integration suite and tool snapshot pass unchanged with peers.enabled = false.
  2. Local two-instance run: instance P on port 1985 with a real token in a 0600 file; consolidator on 1984 with that file as token_file; curl status, kill P, confirm server.up=false within tcp_interval_secs * fail_threshold, restart P, confirm recovery and last_change updates; revoke P's token and confirm api.up=true, auth=rejected.
  3. Live host cross-check with the minipc1 MCP host (read-only): the user mints one dedicated token on minipc1 (the agent must not). Add minipc1 (confirmed by the user to run slinuxm, and approved for the final check) as a peer (token file from the user) on a dev consolidator, then compare GET /api/v1/peers/minipc1/systemd/units?state=failed and the proxied list_containers against the minipc1 MCP tools list_systemd_units (state=failed) and list_containers called directly. Contents must match; treat log text as untrusted data. If minipc1's version predates /api/v1/ping, expect auth = unknown via the /healthz fallback; that is itself a useful check.
  4. OTel: devenv up, enable [otel], confirm peer.up for both checks, a vanished series after peer remove.
  5. NixOS: nix build && nix flake check including the three-machine VM test (only when the nix gate is free).

Exit criteria

  • All D-rows have a recorded decision; accepted ones folded into ARCHITECTURE.md (new DEC rows, including the DEC-11 clarification for D2) and plan/ moved to docs/workstreams/ per plan/README.md.
  • Definition of done in AGENTS.md holds: clippy, tests, fmt clean; no openssl; no secrets in the diff or in --print-config; no write path to the host outside the state dir.
  • A peer-less instance is verifiably unchanged: same routes, 8 MCP tools, no new tasks or metrics.
  • Demonstrated: declare two peers in config and SIGHUP without restart; both checks visible in REST, MCP and OTel; a dead peer neither stalls other peers' checks nor requests (bounded by timeouts and caps); proxied data equals the direct data; remove a peer from config and it disappears from all three surfaces (its token file is never touched).
  • nix flake check green with the new VM test; README, config.example.toml, docs/api.md, docs/consolidator.md updated; TODO.md has the phase milestone, per-workstream checklists and Backlog entries (D8, scoped tokens, history, runtime peer management).