Files
skyg-linux-tracker/docs/workstreams/P5-consolidator/README.md
T
2026-10-20 16:35:11 -04:00

134 lines
13 KiB
Markdown

# Phase 5: Consolidator (peers)
| Field | Value |
|---|---|
| Status | Done |
| Owner | |
| Date | 2026-10-19 |
| Depends on | Phase 4 / WS-H (NixOS module and VM test) done |
This phase is a folder so workstreams can be claimed and worked in parallel by separate agents or people. Start here, then read [contracts.md](contracts.md) and your workstream file only. Rules: every workstream edits only its own files (see its "Do not touch" list); shared interfaces change only through contracts.md. How the folder works: [plan/README.md](../../../plan/README.md).
## Goal
One `slinuxm` binary can also act as a consolidator. An operator declares other machines running `slinuxm` as **peers** in configuration, declaratively: on NixOS via `services.skyg-linux-tracker.peers.<name>`, with each peer's bearer token supplied by an agenix-managed file. The consolidator then (1) health-checks each peer in two ways, "server up" (TCP connect) and "API running" (authenticated API call), (2) exposes that status over REST, MCP and OTel, and (3) proxies read-only requests to a peer so one MCP endpoint or one REST base URL covers the fleet. A peer-less instance (the default, `[peers] enabled = false`) behaves exactly as today: same routes, same 8 MCP tools, same metrics, no new background tasks, no new required config.
## Scope
In scope:
- `[peers]` config section with a declarative peer list (`[[peers.peer]]`: name, url, `token_file`, optional `ca_file`), a peer registry in `AppState`, a read-only `slinuxm peer list|check` diagnostic CLI. No peers table, no migration, no secrets in the DB or the config file.
- Per-peer TCP and API health loops with in-memory state and last-change timestamps.
- REST `GET /api/v1/peers`, `GET /api/v1/peers/{peer}`, and a GET-only pass-through under `/api/v1/peers/{peer}/...`.
- MCP: optional `peer` argument on the existing tools plus `list_peers` and `get_peer`.
- OTel gauges for peer health, trace propagation to peers.
- NixOS module options (`peers.peers.<name>` with `tokenFile`, e.g. `config.age.secrets.<x>.path`), VM test with a third machine, an agenix example in the docs, README, `config.example.toml`, ARCHITECTURE.md decisions (folded in by the orchestrator after acceptance).
Out of scope (Backlog with reasons when accepted):
- Fleet-wide aggregation or merged views (see D8), persisted health history, alerting, peer discovery.
- Chained consolidators (a peer that is itself a consolidator, D12).
- Re-exporting a peer's OTel signals (each peer exports its own telemetry).
- Any write path to a peer or to the host beyond the state dir. Proxying is GET only.
- Runtime peer management (`peer add|remove`, a DB table): peers are declarative only in v1 (D1). `--insecure` TLS skip.
## Grounding in the current code
- `AppState` (`state.rs`) holds `Arc<dyn Source>`s, `db: Option<SqlitePool>` (None when `auth.enabled = false`) and the shutdown token. Add `peers: Option<Arc<PeerRegistry>>`; `None` means the feature is off and every new route and tool is absent. Peers need no DB.
- Errors are `AppError` (`error.rs`): `NotFound`, `BadRequest`, `Unauthorized`, `Unavailable`, `Internal`; MCP maps via `From<AppError> for ErrorData`. Peer failures must fit these variants (D9).
- The token CLI and server share SQLite for inbound tokens; peers do not touch it. SIGHUP is already taken by TLS reload; the shared signal handler can fan out to a peers reload (B4).
- `/healthz` is public and does not validate a token, so it cannot prove "API running with our credentials". No lightweight authenticated endpoint exists today.
- `reqwest` (blocking plus rustls) is already in `Cargo.lock` via `opentelemetry-otlp`; `hyper-util`, `tokio-rustls`, `rcgen` (dev) are direct deps. The unit and module already allow `AF_INET`/`AF_INET6`, so outbound sockets need no sandbox change.
- Background tasks live in `telemetry::spawn_exporters`-style spawns and select on `shutdown`; peer loops follow the same rule (AGENTS.md).
## Files in this phase
- [contracts.md](contracts.md): shared Rust, REST, MCP, OTel, config and NixOS contracts, fakes, and the VM test's config contract.
- [decisions.md](decisions.md): open-decisions table (D1 to D15) and the DEC-13 section (outbound peer credentials are external files). The config surface ("Config surface") is section 6 of contracts.md.
- Workstreams: [ws-t-tests.md](ws-t-tests.md), [ws-a-config.md](ws-a-config.md), [ws-b-registry.md](ws-b-registry.md), [ws-c-rest.md](ws-c-rest.md), [ws-d-mcp.md](ws-d-mcp.md), [ws-e-telemetry.md](ws-e-telemetry.md), [ws-f-nixos-docs.md](ws-f-nixos-docs.md).
## Workstream index
Claim a workstream by putting your name in its Owner field (header table of its file) and committing that change. Task IDs (T1..T7, A0..A4, B1..B5, C1..C4, D1..D3, E1..E2, F1..F4) are ticked in `TODO.md` once the plan is accepted.
| ID | File | Owner | Depends on | Can start when | Owned files (summary) |
|---|---|---|---|---|---|
| WS-T | [ws-t-tests.md](ws-t-tests.md) | | none | day one | `nix/tests/peers.nix` (written), `dev/test.nu`, `devenv.nix`, `flake.nix` peers output, command lists in `AGENTS.md`/`README.md`, `docs/consolidator.md` Testing section |
| WS-A | [ws-a-config.md](ws-a-config.md) | | none | day one (A0 first) | `src/peers/{mod,types,error,transport,testing,config,secrets,cli}.rs`, `src/config.rs`, `src/main.rs` (`Command::Peer`), `config.example.toml` `[peers]` |
| WS-B | [ws-b-registry.md](ws-b-registry.md) | | A0 (A1 to A3 for real config) | A0 merged | `src/peers/{client,health,registry}.rs`, `src/api/ping.rs`, `Cargo.toml`, `src/main.rs` `serve()`, `tests/peers_registry.rs` |
| WS-C | [ws-c-rest.md](ws-c-rest.md) | | A0 (B for the final e2e) | A0 merged | `src/api/peers.rs`, `src/peers/proxy.rs`, `docs/api.md` |
| WS-D | [ws-d-mcp.md](ws-d-mcp.md) | | A0 (B and C2 for the final e2e) | A0 merged (rmcp spike earlier) | `src/mcp/**` |
| WS-E | [ws-e-telemetry.md](ws-e-telemetry.md) | | A0 (B for the final e2e) | A0 merged | `src/telemetry/{mod,metrics_export}.rs` |
| WS-F | [ws-f-nixos-docs.md](ws-f-nixos-docs.md) | | C, D, E (F1/F4 drafts after A1) | F1: A1 settled; F2: T, C, D, E green | `nix/module.nix`, `nix/tests/vm.nix`, `systemd/*`, `README.md` peers section, `docs/consolidator.md`, `tests/peers.rs`, `tests/common/mod.rs` |
Original dependency statement, kept for reference: **T first** (tests written ahead of the feature) -> A -> B -> {C, D, E} (parallel) -> F. T goes green at the end of E. Tests for each stream ship with it. B defines a `PeerTransport` trait with an in-memory fake (same pattern as DEC-2) so C, D and E test without sockets. In this split the trait, the fake and the shared types land in task A0, so C, D and E no longer wait for B.
### Dependency graph
```
WS-T (tests, runner, peers.nix) -------------------------------+--> goes green when A..E land
|
WS-A: A0 (contract stubs) --+--> A1 -> A2 -> A3 -> A4* |
| |
+--> WS-B (client, health, registry)+--+
+--> WS-C (REST, proxy.rs) | (against FakeTransport,
+--> WS-D (MCP) | real B at integration)
+--> WS-E (telemetry) ---+
\ | | /
v v v v
WS-F (module, VM check, e2e, docs)
* A4 `peer check` is wired to the real transport when B1/B2 land.
```
### Parallel lanes
Day one, no coordination needed (three lanes at once):
1. **WS-T**: T7 runner, docs Testing section; `peers.nix` is done.
2. **WS-A**: A0 first (small PR: types, trait, fake, seams), then A1 to A4.
3. **Contract stubs**: A0 is the contract-stubs step; everyone else can also begin reading code and drafting against [contracts.md](contracts.md) in their branches, and WS-D can run the rmcp conditional-tools spike.
After A0 merges, the next wave runs fully in parallel, all against `FakeTransport` / `registry_with` / `static_registry`:
- **WS-B** (real client, loops, registry, ping), **WS-C** (REST, `proxy.rs`), **WS-D** (MCP), **WS-E** (telemetry); WS-A continues A1 to A4; WS-F can start F1 (module) and the doc drafts once A1 has settled the config keys.
- Real dependencies that remain: D1 and C2 share `PeerRegistry::get` (D uses the A0 forwarder until C2 lands, then picks up the hardened one with no code change); C/D/E "done" is proved against the fake and rechecked with the real registry once B lands; A4 `peer check` needs B1/B2.
What each wave unlocks:
- **A0 merged**: B, C, D, E compile and can merge; F1 can reference real option/config names.
- **B merged**: end-to-end checks for C, D, E with real loops; `slinuxm peer check` works; `tests/peers_registry.rs`.
- **C2 merged**: D1 uses the hardened proxy.
- **E merged (with A to D)**: WS-T's `peers.nix` should pass.
- **T green + C, D, E merged**: F2 promotes the VM test to `checks`, F3/F4 finish; phase exit.
Contract changes during the phase follow the rule at the top of [contracts.md](contracts.md): edit contracts.md first, log in the affected workstream files.
## Risks
- Tool-list growth and optional parameters change MCP schemas; clients caching the list see a different one only when peers are enabled. Mitigated by gating and the snapshot test.
- A consolidator token on a peer is a lateral-movement asset (read access to journals and containers). Mitigated by per-consolidator tokens, agenix-encrypted at rest, delivered via `LoadCredential`, docs. Compromise of the consolidator host exposes all peer tokens by design.
- Streaming pass-through of SSE can hold many connections; mitigated by the per-peer and global semaphores, `max_follow_secs`, and the existing 1024 connection cap.
- Journal text proxied from peers is untrusted data (prompt injection); descriptions already say so, keep that wording for peer calls.
- Rust-side risk: `rmcp` tool router may not support conditional tools cleanly; spike in WS-D early (fallback: always list tools, return an error when disabled, and accept a 10-tool snapshot).
- A new authenticated `/api/v1/ping` means a consolidator at this version needs peers at the same version for `auth` detection; the `/healthz` fallback covers older peers.
- Safety: agents must not mint tokens on production hosts (AGENTS.md). Live verification needs a token created by the user.
## Verification plan
1. Unit and integration: `cargo clippy --all-targets -- -D warnings`, `cargo test`, `cargo fmt --check`, `cargo tree -i openssl` (empty). No-peers regression: the existing integration suite and tool snapshot pass unchanged with `peers.enabled = false`.
2. Local two-instance run: instance P on port 1985 with a real token in a 0600 file; consolidator on 1984 with that file as `token_file`; `curl` status, kill P, confirm `server.up=false` within `tcp_interval_secs * fail_threshold`, restart P, confirm recovery and `last_change` updates; revoke P's token and confirm `api.up=true, auth=rejected`.
3. Live host cross-check with the minipc1 MCP host (read-only): the user mints one dedicated token on minipc1 (the agent must not). Add minipc1 (**confirmed by the user to run `slinuxm`**, and approved for the final check) as a peer (token file from the user) on a dev consolidator, then compare `GET /api/v1/peers/minipc1/systemd/units?state=failed` and the proxied `list_containers` against the minipc1 MCP tools `list_systemd_units` (state=failed) and `list_containers` called directly. Contents must match; treat log text as untrusted data. If minipc1's version predates `/api/v1/ping`, expect `auth = unknown` via the `/healthz` fallback; that is itself a useful check.
4. OTel: `devenv up`, enable `[otel]`, confirm `peer.up` for both checks, a vanished series after `peer remove`.
5. NixOS: `nix build && nix flake check` including the three-machine VM test (only when the nix gate is free).
## Exit criteria
- All D-rows have a recorded decision; accepted ones folded into ARCHITECTURE.md (new DEC rows, including the DEC-11 clarification for D2) and `plan/` moved to `docs/workstreams/` per `plan/README.md`.
- Definition of done in AGENTS.md holds: clippy, tests, fmt clean; no openssl; no secrets in the diff or in `--print-config`; no write path to the host outside the state dir.
- A peer-less instance is verifiably unchanged: same routes, 8 MCP tools, no new tasks or metrics.
- Demonstrated: declare two peers in config and SIGHUP without restart; both checks visible in REST, MCP and OTel; a dead peer neither stalls other peers' checks nor requests (bounded by timeouts and caps); proxied data equals the direct data; remove a peer from config and it disappears from all three surfaces (its token file is never touched).
- `nix flake check` green with the new VM test; README, `config.example.toml`, `docs/api.md`, `docs/consolidator.md` updated; TODO.md has the phase milestone, per-workstream checklists and Backlog entries (D8, scoped tokens, history, runtime peer management).