Files
skyg-linux-tracker/docs/workstreams/P5-consolidator/ws-b-registry.md
T
2026-10-20 16:35:11 -04:00

88 lines
15 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# WS-B: Client, health, registry
Part of [Phase 5: Consolidator](README.md). Contracts: [contracts.md](contracts.md) sections 1 to 4 and 8.
| Field | Value |
|---|---|
| Status | Merged |
| Owner | goose |
| Depends on | A0 (stubs and seams) and A1 to A3 for real config/secrets; can start on B1 as soon as A0 lands |
| Blocks | integration of C, D, E (they are developed against the fake registry and finish for real once B lands); F |
| Can start when | A0 is merged (design and client code can begin earlier in a branch against contracts.md) |
| Owned files | `src/peers/client.rs`, `src/peers/health.rs`, `src/peers/registry.rs` (bodies; signatures per contracts), `src/api/ping.rs` (new authenticated `GET /api/v1/ping`, see D4), `Cargo.toml`/`Cargo.lock`, `src/main.rs` (`serve()` only), `src/tls.rs` (SIGHUP fan-out line only, if the handler stays there), `tests/peers_registry.rs`, `tests/common/mod.rs` (append new helpers only) |
| Do not touch | `src/peers/types.rs`, `transport.rs`, `error.rs`, `testing.rs` (change via contracts.md only); `src/peers/config.rs`, `secrets.rs`, `cli.rs`, `src/config.rs` (WS-A); `src/peers/proxy.rs`, `src/api/peers.rs`, `src/api/mod.rs`, `src/error.rs` (WS-C / A0); `src/mcp/**` (D); `src/telemetry/**` (E); `nix/**`, `docs/**`, `README.md` (F, T) |
Shared-file seams:
- `src/state.rs`: do not edit; A0 added `AppState.peers`. B assigns it in `serve()` in `src/main.rs` (state is built there by `AppState::live`, then `state.peers = peers::start(&state)` before `server::build_router`). One edit region, `serve()`; WS-A edits the `Command` enum elsewhere in the file.
- SIGHUP: `serve()` installs the handler and `tls::spawn_sighup_reload` consumes it. B adds the fan-out so the same signal also calls `PeerRegistry::reload` (and the plain-TLS-off branch no longer just logs when peers are enabled). Keep the TLS reload behaviour unchanged.
- `src/api/mod.rs`: untouched; `ping.rs` is already merged by A0. B only fills `ping::routes()`.
- `Cargo.toml`/`Cargo.lock`: B owns dependency edits (reqwest direct dep per D15, features, any CIDR crate not already added by A). Regenerate `Cargo.lock`, never hand-merge.
- `tests/common/mod.rs`: append-only helpers; WS-F appends too.
## Goal
A `PeerRegistry` built from config that keeps live status; a bounded HTTP client.
## Tasks
- **B1** client: connect/request timeouts, redirects disabled, GET only, response byte cap enforced while streaming (not just `Content-Length`), per-peer `Semaphore` plus a global one, custom CA per peer, rustls with the ring provider passed explicitly; `cargo tree -i openssl` stays empty. Implements `PeerTransport` for real (`ReqwestTransport`) and records the D15 outcome in the Log.
- **B2** health loops: one task per peer with its own child `CancellationToken` and start jitter; TCP check = `TcpStream::connect(host:port)` under timeout; API check = authenticated `GET /api/v1/ping`; classification `ok | unauthorized | http_error | timeout | unreachable | tls_error`. Provides `health::check_once` (shared with `slinuxm peer check`, A4) and the `/api/v1/ping` handler in `ping.rs` returning `{status, version}`; fallback for peers without `/ping` is `/healthz` plus `auth = unknown` (D4).
- **B3** status model (in memory only): `{server: {up, since, last_checked, last_error}, api: {up, since, last_checked, last_error, latency_ms, auth: ok|rejected|unknown, peer_version}}`; flips need `fail_threshold` consecutive failures, recovery is immediate; `since` is the last change time and resets on restart.
- **B4** reload: on startup build the registry; on SIGHUP (shared handler with TLS) re-read only the `[peers]` block and token files, start loops for new peers, cancel removed ones, restart changed ones; a bad reload keeps the old registry (same rule as TLS T3). Token files are re-read on every API check, so rotating an agenix secret needs no restart. Shutdown awaits all loops (bounded).
- **B5** `PeerTransport` trait and fake; SSRF guard (D11) applied at config load and again at connect time on the resolved address. The trait and fake already exist from A0 (contracts section 2); B5 is now the SSRF guard on the real transport plus any fake extensions the loops need (changed through contracts.md).
## Inputs
- [contracts.md](contracts.md) sections 1 to 4 (types, transport, registry API, error mapping), 6 (config), 10 (the test's timings and rotation flow).
- Fake: `FakeTransport` and `registry_with` for loop/classification unit tests with paused time; real in-process servers for the integration test.
## Done when
Integration test starts a second in-process server (fake sources) as a peer and observes up, down (port closed), api-down-but-server-up (listener that accepts and hangs), unauthorized (wrong token) and recovery; one hung peer does not delay another peer's check beyond its own timeout; clippy, fmt, `cargo tree -i openssl` clean.
## Handoff
- Real `PeerRegistry` with the contract API (`snapshot`, `status`, `get`, `reload`, `shutdown`) assigned to `AppState.peers` when `peers.enabled`.
- `GET /api/v1/ping`; `ReqwestTransport`; `health::check_once` for A4.
- Post in C, D, E, F Logs when the registry is wired, so they swap their fakes for real end-to-end checks.
## Log
- 2026-10-19: Split from the single phase plan 2026-10-19. The trait and fake moved to A0 so C/D/E are not blocked on this stream; B keeps the implementations.
- 2026-10-19 B1 (D15): chose reqwest (option a). Direct dependency `reqwest` 0.13.5 with `default-features = false` and `rustls-no-provider` only (no `native-tls`, no `stream` — that feature cannot resolve `wasm-streams` — and no `rustls` feature that selects aws-lc). Response bodies are capped by polling `chunk()`. Each client is a rustls `ClientConfig` built with `builder_with_provider(ring)` and webpki roots, plus an optional per-peer CA; `install_default` is not called. `cargo tree -i openssl` is empty. `aws-lc-rs` remains in the tree only through `opentelemetry-otlp`'s own reqwest and is not used by this client. `src/peers/mod.rs` is not owned here, so `client.rs` is mounted as `peers::registry::client` (`ReqwestTransport` is re-exported from `peers::registry`).
- 2026-10-19 B2: one task per peer (both probes in one `tokio::select!`) with a deterministic name-hash jitter capped at a quarter of the shorter interval. Probes are `health::probe_server` / `health::probe_api`; `check_once` joins them for A4. A 401/403 is `api.up = true, auth = rejected`; a 404 falls back to public `/healthz` with `auth = unknown` (older peers, D4). The loop reuses `FakeTransport` for the classification/threshold tests and a second in-process `slinuxm` (fake sources) for up/down/recovery, api-down-but-server-up (listener accepts, never answers) and rejected-token cases in `tests/peers_registry.rs`.
- 2026-10-19 B3: `ServerTracker`/`ApiTracker` keep `since` and consecutive failures in loop-local state, so a surviving peer keeps its timestamps across a reload and a restart resets them. `fail_threshold` consecutive failures flip a check down; a success recovers in one check. `last_error` is always the fixed class string (never peer text or a URL). The first check records `since` whether it is up or down.
- 2026-10-19 B4: `PeerRegistry::start` builds the real transport and loops; `src/main.rs::serve()` assigns `state.peers`. SIGHUP now reloads TLS (when enabled) and peers from one handler (`tls::spawn_sighup_reload`); a config error is logged and the old registry stays. `reload` keeps statuses of surviving peers, cancels removed/disabled/changed loops and starts new ones. Token files are read on every API check, so agenix rotation needs no restart. `peers` concurrency and size caps are fixed for the process lifetime; only the peer set, intervals, timeouts and threshold follow a reload. `PeerRegistry::shutdown` cancels and joins loops with a 5 s bound. A reload with `[peers] enabled = false` empties the snapshot (routes stay mounted until restart). `main()` passes the config path and CLI overrides into `serve()` because a SIGHUP reload must re-read the same file; that is the only edit outside `serve()`.
- 2026-10-19 B5: connect-time SSRF guard resolves the peer host, rejects any address in `deny_cidrs` (parsed with `ipnet` at `ReqwestTransport::new`), and pins the allowed address into the HTTP client; plain HTTP to a non-loopback host requires `allow_plain_http`. `tcp_check` uses the same resolution and guard. DNS failures map to `dns`, all-denied to `unreachable`.
## Handoff (2026-10-19)
Complete; B1 to B5 and the authenticated `GET /api/v1/ping` route are in. D15 outcome is recorded above (reqwest, rustls-no-provider with ring passed explicitly, body cap enforced while streaming; `cargo tree -i openssl` is empty). The registry API matches contracts section 3 and is already used by the A0 seams (`snapshot`, `status`, `get`, `reload`, `shutdown`); C/D/E can now switch their fake registries to `PeerRegistry::start`.
Verification run: `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings`, `cargo test` (unit plus the in-process two-server cases in `tests/peers_registry.rs`), `cargo tree -i openssl` (empty). No Nix files were touched, so no VM test is involved; when the Nix gate is free, WS-T covers the module/VM path. Nothing in this workstream mints tokens or writes outside the state dir.
## Round 1 review (2026-10-19)
Responses to `loop/B/review-r.md`; every Blocking and Major item is addressed, Minors where cheap.
- **Blocking 1 (stream cut off after `request_timeout_secs`) — fixed.** `get` no longer sets reqwest's total request timeout. One `within()` deadline covers preparation (validation, DNS, token and CA reads, a permit, building the client, sending and response headers); a buffered body is read under the same deadline. A stream escapes that deadline and is bounded by an idle timeout instead (`STREAM_IDLE_TIMEOUT`, 120 s): peers send SSE keep-alives and end their own follow streams at `limits.max_follow_secs`, so this only catches a dead connection. Test `a_stream_outlives_the_request_timeout` sends chunks across ~750 ms with a 150 ms timeout and fails on the old code.
- **Blocking 2 (token sent to disabled peers) — fixed.** `PeerRegistry::get` looks the peer up with `enabled` and `config.enabled` applied. A disabled peer (or a disabled feature) returns `NotFound` (`peer <name>: disabled`, or `unknown peer` when the feature is off) and the transport is never called. Test `a_disabled_peer_is_never_proxied` asserts the fake records no call before and after both kinds of reload.
- **Blocking 3 (DNS/token/CA outside the timeout) — fixed.** `tcp_check` and `get` run every phase inside a single `within(deadline, ...)`. Test `one_deadline_covers_every_phase` drives the wrapper with a pending future on a paused clock (plus error and success pass-through). The health claim in `health.rs` now holds for the transport too.
- **Blocking 4 (IPv6 URLs) — fixed.** Targets come from `url::Host` (`Ipv4`/`Ipv6`/`Domain`); only a domain is resolved and pinned, and `url_is_loopback` uses the parsed host. Tests `loopback_hosts_as_the_url_parser_reports_them`, `ipv6_plain_http_to_loopback_needs_no_opt_in` and `ipv6_literal_peers_resolve_and_connect` (skips when the environment has no `::1`).
- **Major 5 (probes compete with proxy permits) — fixed.** `/api/v1/ping` and `/healthz` skip the per-peer and global semaphores; the proxy allowlist excludes both, so no caller can use the exemption for data traffic. Test `probes_bypass_the_concurrency_limits` holds a follow stream with `max_concurrent_per_peer = 1` and still gets a ping through, while a data call is `Saturated`.
- **Major 6 (reload only changes limits on paper) — fixed by rejecting such reloads.** `deny_cidrs`, `max_response_bytes`, `max_concurrent_per_peer` and `max_concurrent_total` cannot change without a restart; a reload that changes them fails and the old registry stays, so `config()` never reports a value that is not enforced. The deny list is now parsed by one shared `deny_networks()` used at startup and on reload, so an invalid CIDR on SIGHUP is rejected instead of accepted and ignored. Tests `reload_rejects_changes_the_transport_cannot_apply`.
- **Major 7 (token loader belongs to WS-A) — recorded disagreement.** The rules are A3's, but `src/peers/secrets.rs` does not exist in this branch and WS-A owns it. `load_token_in` here is private to the transport and documented as the single loader until A3 lands; when it does, `ReqwestTransport` should call `peers::secrets` and this function should be deleted. Likewise `#[path = "client.rs"] pub mod client;` in `registry.rs` is the workaround for not owning `peers/mod.rs`; at merge the module should move to `mod.rs` and `ReqwestTransport` be re-exported from `peers`.
- **Major 8 (test gaps).** Added: `sighup_reloads_peers_when_tls_is_off` (TLS off, peers on; TLS off/peers off is still covered by `tests/tls.rs::sighup_with_tls_disabled_keeps_serving`), `a_rotated_token_recovers_without_a_restart` in `tests/peers_registry.rs` (rejected -> rotate file -> `auth = ok`, no restart or SIGHUP), the threshold test now observes both checks flip down, and `shutdown_waits_for_loops_retired_by_a_reload`. The loop tests that do not need a real socket now run on paused time.
- **Minor 9–14 — fixed.** `request_url` calls `validate_target` instead of repeating its rules; probes share `probe_request`; `map_io` lost the dead DNS string matches; `error_is_tls` matches structured error kinds (rustls downcast, or an `io::Error` whose kind is not a socket failure) instead of text; the Authorization `HeaderValue` is `set_sensitive(true)`; a peer whose URL changes on reload starts from a clean status; loops retired by a reload are kept and awaited by `shutdown`; the `/healthz` fallback reports its own latency and `ApiTracker` keeps the last known `peer_version` while the API stays up. Tests were added for each.
- **Minor 15 (scope).** `src/tls.rs` changes the SIGHUP handler's signature and imports `PeersReloader`; that is the minimum for one handler to reload TLS and peers with independent failures. `main()` changes only the arguments to `serve()`, as logged under B4. The handoff to the C/D/E/F logs still has to be posted by the orchestrator; this worktree does not own those files.
- 2026-10-19 verification: `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings`, `cargo test` and `cargo tree -i openssl` (empty) are clean after the round-1 fixes.
## Round 2 note from WS-A (2026-10-19)
- **`real_transport()` seam (for `slinuxm peer check`) — consumed by WS-A.** B1/B2 being merged was the condition for wiring A4, and `src/peers/cli.rs` is A's file, so WS-A now builds `ReqwestTransport::new(&config.peers)` there (limits and deny list included) instead of bailing. No change is needed in B's files; the binary test now probes a real loopback server (`tests/peers_config.rs`, `peer_check_binary_probes_a_real_peer_and_prints_latency`).
- **Token reads are blocking (A3 loader).** `peers::secrets::load_token` is synchronous file I/O. Today the only per-check caller is B's transport, which reads inside its request deadline, so a slow read is bounded but still occupies an async worker. If that path gets hotter (many peers, short intervals), move the read to `spawn_blocking` or a size-bounded `tokio::fs` read; the loader stays A's contract.
- **Test fixture fix in this workstream's file.** On the rebase with WS-A, `reloader_rereads_the_peers_block_and_keeps_the_old_set_on_error` failed because its inline `[peers]` used `api_interval_secs = 2` with the default `api_timeout_secs = 5`, which A2's "timeout below interval" rule rejects. WS-A added `api_timeout_secs = 1` to that fixture; the SIGHUP test needed no change once `Config::load` stopped opening token files.