88 lines
15 KiB
Markdown
88 lines
15 KiB
Markdown
# WS-B: Client, health, registry
|
||
|
||
Part of [Phase 5: Consolidator](README.md). Contracts: [contracts.md](contracts.md) sections 1 to 4 and 8.
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Status | Merged |
|
||
| Owner | goose |
|
||
| Depends on | A0 (stubs and seams) and A1 to A3 for real config/secrets; can start on B1 as soon as A0 lands |
|
||
| Blocks | integration of C, D, E (they are developed against the fake registry and finish for real once B lands); F |
|
||
| Can start when | A0 is merged (design and client code can begin earlier in a branch against contracts.md) |
|
||
| Owned files | `src/peers/client.rs`, `src/peers/health.rs`, `src/peers/registry.rs` (bodies; signatures per contracts), `src/api/ping.rs` (new authenticated `GET /api/v1/ping`, see D4), `Cargo.toml`/`Cargo.lock`, `src/main.rs` (`serve()` only), `src/tls.rs` (SIGHUP fan-out line only, if the handler stays there), `tests/peers_registry.rs`, `tests/common/mod.rs` (append new helpers only) |
|
||
| Do not touch | `src/peers/types.rs`, `transport.rs`, `error.rs`, `testing.rs` (change via contracts.md only); `src/peers/config.rs`, `secrets.rs`, `cli.rs`, `src/config.rs` (WS-A); `src/peers/proxy.rs`, `src/api/peers.rs`, `src/api/mod.rs`, `src/error.rs` (WS-C / A0); `src/mcp/**` (D); `src/telemetry/**` (E); `nix/**`, `docs/**`, `README.md` (F, T) |
|
||
|
||
Shared-file seams:
|
||
|
||
- `src/state.rs`: do not edit; A0 added `AppState.peers`. B assigns it in `serve()` in `src/main.rs` (state is built there by `AppState::live`, then `state.peers = peers::start(&state)` before `server::build_router`). One edit region, `serve()`; WS-A edits the `Command` enum elsewhere in the file.
|
||
- SIGHUP: `serve()` installs the handler and `tls::spawn_sighup_reload` consumes it. B adds the fan-out so the same signal also calls `PeerRegistry::reload` (and the plain-TLS-off branch no longer just logs when peers are enabled). Keep the TLS reload behaviour unchanged.
|
||
- `src/api/mod.rs`: untouched; `ping.rs` is already merged by A0. B only fills `ping::routes()`.
|
||
- `Cargo.toml`/`Cargo.lock`: B owns dependency edits (reqwest direct dep per D15, features, any CIDR crate not already added by A). Regenerate `Cargo.lock`, never hand-merge.
|
||
- `tests/common/mod.rs`: append-only helpers; WS-F appends too.
|
||
|
||
## Goal
|
||
|
||
A `PeerRegistry` built from config that keeps live status; a bounded HTTP client.
|
||
|
||
## Tasks
|
||
|
||
- **B1** client: connect/request timeouts, redirects disabled, GET only, response byte cap enforced while streaming (not just `Content-Length`), per-peer `Semaphore` plus a global one, custom CA per peer, rustls with the ring provider passed explicitly; `cargo tree -i openssl` stays empty. Implements `PeerTransport` for real (`ReqwestTransport`) and records the D15 outcome in the Log.
|
||
- **B2** health loops: one task per peer with its own child `CancellationToken` and start jitter; TCP check = `TcpStream::connect(host:port)` under timeout; API check = authenticated `GET /api/v1/ping`; classification `ok | unauthorized | http_error | timeout | unreachable | tls_error`. Provides `health::check_once` (shared with `slinuxm peer check`, A4) and the `/api/v1/ping` handler in `ping.rs` returning `{status, version}`; fallback for peers without `/ping` is `/healthz` plus `auth = unknown` (D4).
|
||
- **B3** status model (in memory only): `{server: {up, since, last_checked, last_error}, api: {up, since, last_checked, last_error, latency_ms, auth: ok|rejected|unknown, peer_version}}`; flips need `fail_threshold` consecutive failures, recovery is immediate; `since` is the last change time and resets on restart.
|
||
- **B4** reload: on startup build the registry; on SIGHUP (shared handler with TLS) re-read only the `[peers]` block and token files, start loops for new peers, cancel removed ones, restart changed ones; a bad reload keeps the old registry (same rule as TLS T3). Token files are re-read on every API check, so rotating an agenix secret needs no restart. Shutdown awaits all loops (bounded).
|
||
- **B5** `PeerTransport` trait and fake; SSRF guard (D11) applied at config load and again at connect time on the resolved address. The trait and fake already exist from A0 (contracts section 2); B5 is now the SSRF guard on the real transport plus any fake extensions the loops need (changed through contracts.md).
|
||
|
||
## Inputs
|
||
|
||
- [contracts.md](contracts.md) sections 1 to 4 (types, transport, registry API, error mapping), 6 (config), 10 (the test's timings and rotation flow).
|
||
- Fake: `FakeTransport` and `registry_with` for loop/classification unit tests with paused time; real in-process servers for the integration test.
|
||
|
||
## Done when
|
||
|
||
Integration test starts a second in-process server (fake sources) as a peer and observes up, down (port closed), api-down-but-server-up (listener that accepts and hangs), unauthorized (wrong token) and recovery; one hung peer does not delay another peer's check beyond its own timeout; clippy, fmt, `cargo tree -i openssl` clean.
|
||
|
||
## Handoff
|
||
|
||
- Real `PeerRegistry` with the contract API (`snapshot`, `status`, `get`, `reload`, `shutdown`) assigned to `AppState.peers` when `peers.enabled`.
|
||
- `GET /api/v1/ping`; `ReqwestTransport`; `health::check_once` for A4.
|
||
- Post in C, D, E, F Logs when the registry is wired, so they swap their fakes for real end-to-end checks.
|
||
|
||
## Log
|
||
|
||
- 2026-10-19: Split from the single phase plan 2026-10-19. The trait and fake moved to A0 so C/D/E are not blocked on this stream; B keeps the implementations.
|
||
- 2026-10-19 B1 (D15): chose reqwest (option a). Direct dependency `reqwest` 0.13.5 with `default-features = false` and `rustls-no-provider` only (no `native-tls`, no `stream` — that feature cannot resolve `wasm-streams` — and no `rustls` feature that selects aws-lc). Response bodies are capped by polling `chunk()`. Each client is a rustls `ClientConfig` built with `builder_with_provider(ring)` and webpki roots, plus an optional per-peer CA; `install_default` is not called. `cargo tree -i openssl` is empty. `aws-lc-rs` remains in the tree only through `opentelemetry-otlp`'s own reqwest and is not used by this client. `src/peers/mod.rs` is not owned here, so `client.rs` is mounted as `peers::registry::client` (`ReqwestTransport` is re-exported from `peers::registry`).
|
||
|
||
- 2026-10-19 B2: one task per peer (both probes in one `tokio::select!`) with a deterministic name-hash jitter capped at a quarter of the shorter interval. Probes are `health::probe_server` / `health::probe_api`; `check_once` joins them for A4. A 401/403 is `api.up = true, auth = rejected`; a 404 falls back to public `/healthz` with `auth = unknown` (older peers, D4). The loop reuses `FakeTransport` for the classification/threshold tests and a second in-process `slinuxm` (fake sources) for up/down/recovery, api-down-but-server-up (listener accepts, never answers) and rejected-token cases in `tests/peers_registry.rs`.
|
||
- 2026-10-19 B3: `ServerTracker`/`ApiTracker` keep `since` and consecutive failures in loop-local state, so a surviving peer keeps its timestamps across a reload and a restart resets them. `fail_threshold` consecutive failures flip a check down; a success recovers in one check. `last_error` is always the fixed class string (never peer text or a URL). The first check records `since` whether it is up or down.
|
||
- 2026-10-19 B4: `PeerRegistry::start` builds the real transport and loops; `src/main.rs::serve()` assigns `state.peers`. SIGHUP now reloads TLS (when enabled) and peers from one handler (`tls::spawn_sighup_reload`); a config error is logged and the old registry stays. `reload` keeps statuses of surviving peers, cancels removed/disabled/changed loops and starts new ones. Token files are read on every API check, so agenix rotation needs no restart. `peers` concurrency and size caps are fixed for the process lifetime; only the peer set, intervals, timeouts and threshold follow a reload. `PeerRegistry::shutdown` cancels and joins loops with a 5 s bound. A reload with `[peers] enabled = false` empties the snapshot (routes stay mounted until restart). `main()` passes the config path and CLI overrides into `serve()` because a SIGHUP reload must re-read the same file; that is the only edit outside `serve()`.
|
||
- 2026-10-19 B5: connect-time SSRF guard resolves the peer host, rejects any address in `deny_cidrs` (parsed with `ipnet` at `ReqwestTransport::new`), and pins the allowed address into the HTTP client; plain HTTP to a non-loopback host requires `allow_plain_http`. `tcp_check` uses the same resolution and guard. DNS failures map to `dns`, all-denied to `unreachable`.
|
||
|
||
## Handoff (2026-10-19)
|
||
|
||
Complete; B1 to B5 and the authenticated `GET /api/v1/ping` route are in. D15 outcome is recorded above (reqwest, rustls-no-provider with ring passed explicitly, body cap enforced while streaming; `cargo tree -i openssl` is empty). The registry API matches contracts section 3 and is already used by the A0 seams (`snapshot`, `status`, `get`, `reload`, `shutdown`); C/D/E can now switch their fake registries to `PeerRegistry::start`.
|
||
|
||
Verification run: `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings`, `cargo test` (unit plus the in-process two-server cases in `tests/peers_registry.rs`), `cargo tree -i openssl` (empty). No Nix files were touched, so no VM test is involved; when the Nix gate is free, WS-T covers the module/VM path. Nothing in this workstream mints tokens or writes outside the state dir.
|
||
|
||
## Round 1 review (2026-10-19)
|
||
|
||
Responses to `loop/B/review-r.md`; every Blocking and Major item is addressed, Minors where cheap.
|
||
|
||
- **Blocking 1 (stream cut off after `request_timeout_secs`) — fixed.** `get` no longer sets reqwest's total request timeout. One `within()` deadline covers preparation (validation, DNS, token and CA reads, a permit, building the client, sending and response headers); a buffered body is read under the same deadline. A stream escapes that deadline and is bounded by an idle timeout instead (`STREAM_IDLE_TIMEOUT`, 120 s): peers send SSE keep-alives and end their own follow streams at `limits.max_follow_secs`, so this only catches a dead connection. Test `a_stream_outlives_the_request_timeout` sends chunks across ~750 ms with a 150 ms timeout and fails on the old code.
|
||
- **Blocking 2 (token sent to disabled peers) — fixed.** `PeerRegistry::get` looks the peer up with `enabled` and `config.enabled` applied. A disabled peer (or a disabled feature) returns `NotFound` (`peer <name>: disabled`, or `unknown peer` when the feature is off) and the transport is never called. Test `a_disabled_peer_is_never_proxied` asserts the fake records no call before and after both kinds of reload.
|
||
- **Blocking 3 (DNS/token/CA outside the timeout) — fixed.** `tcp_check` and `get` run every phase inside a single `within(deadline, ...)`. Test `one_deadline_covers_every_phase` drives the wrapper with a pending future on a paused clock (plus error and success pass-through). The health claim in `health.rs` now holds for the transport too.
|
||
- **Blocking 4 (IPv6 URLs) — fixed.** Targets come from `url::Host` (`Ipv4`/`Ipv6`/`Domain`); only a domain is resolved and pinned, and `url_is_loopback` uses the parsed host. Tests `loopback_hosts_as_the_url_parser_reports_them`, `ipv6_plain_http_to_loopback_needs_no_opt_in` and `ipv6_literal_peers_resolve_and_connect` (skips when the environment has no `::1`).
|
||
- **Major 5 (probes compete with proxy permits) — fixed.** `/api/v1/ping` and `/healthz` skip the per-peer and global semaphores; the proxy allowlist excludes both, so no caller can use the exemption for data traffic. Test `probes_bypass_the_concurrency_limits` holds a follow stream with `max_concurrent_per_peer = 1` and still gets a ping through, while a data call is `Saturated`.
|
||
- **Major 6 (reload only changes limits on paper) — fixed by rejecting such reloads.** `deny_cidrs`, `max_response_bytes`, `max_concurrent_per_peer` and `max_concurrent_total` cannot change without a restart; a reload that changes them fails and the old registry stays, so `config()` never reports a value that is not enforced. The deny list is now parsed by one shared `deny_networks()` used at startup and on reload, so an invalid CIDR on SIGHUP is rejected instead of accepted and ignored. Tests `reload_rejects_changes_the_transport_cannot_apply`.
|
||
- **Major 7 (token loader belongs to WS-A) — recorded disagreement.** The rules are A3's, but `src/peers/secrets.rs` does not exist in this branch and WS-A owns it. `load_token_in` here is private to the transport and documented as the single loader until A3 lands; when it does, `ReqwestTransport` should call `peers::secrets` and this function should be deleted. Likewise `#[path = "client.rs"] pub mod client;` in `registry.rs` is the workaround for not owning `peers/mod.rs`; at merge the module should move to `mod.rs` and `ReqwestTransport` be re-exported from `peers`.
|
||
- **Major 8 (test gaps).** Added: `sighup_reloads_peers_when_tls_is_off` (TLS off, peers on; TLS off/peers off is still covered by `tests/tls.rs::sighup_with_tls_disabled_keeps_serving`), `a_rotated_token_recovers_without_a_restart` in `tests/peers_registry.rs` (rejected -> rotate file -> `auth = ok`, no restart or SIGHUP), the threshold test now observes both checks flip down, and `shutdown_waits_for_loops_retired_by_a_reload`. The loop tests that do not need a real socket now run on paused time.
|
||
- **Minor 9–14 — fixed.** `request_url` calls `validate_target` instead of repeating its rules; probes share `probe_request`; `map_io` lost the dead DNS string matches; `error_is_tls` matches structured error kinds (rustls downcast, or an `io::Error` whose kind is not a socket failure) instead of text; the Authorization `HeaderValue` is `set_sensitive(true)`; a peer whose URL changes on reload starts from a clean status; loops retired by a reload are kept and awaited by `shutdown`; the `/healthz` fallback reports its own latency and `ApiTracker` keeps the last known `peer_version` while the API stays up. Tests were added for each.
|
||
- **Minor 15 (scope).** `src/tls.rs` changes the SIGHUP handler's signature and imports `PeersReloader`; that is the minimum for one handler to reload TLS and peers with independent failures. `main()` changes only the arguments to `serve()`, as logged under B4. The handoff to the C/D/E/F logs still has to be posted by the orchestrator; this worktree does not own those files.
|
||
|
||
- 2026-10-19 verification: `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings`, `cargo test` and `cargo tree -i openssl` (empty) are clean after the round-1 fixes.
|
||
|
||
## Round 2 note from WS-A (2026-10-19)
|
||
|
||
- **`real_transport()` seam (for `slinuxm peer check`) — consumed by WS-A.** B1/B2 being merged was the condition for wiring A4, and `src/peers/cli.rs` is A's file, so WS-A now builds `ReqwestTransport::new(&config.peers)` there (limits and deny list included) instead of bailing. No change is needed in B's files; the binary test now probes a real loopback server (`tests/peers_config.rs`, `peer_check_binary_probes_a_real_peer_and_prints_latency`).
|
||
- **Token reads are blocking (A3 loader).** `peers::secrets::load_token` is synchronous file I/O. Today the only per-check caller is B's transport, which reads inside its request deadline, so a slow read is bounded but still occupies an async worker. If that path gets hotter (many peers, short intervals), move the read to `spawn_blocking` or a size-bounded `tokio::fs` read; the loader stays A's contract.
|
||
- **Test fixture fix in this workstream's file.** On the rebase with WS-A, `reloader_rereads_the_peers_block_and_keeps_the_old_set_on_error` failed because its inline `[peers]` used `api_interval_secs = 2` with the default `api_timeout_secs = 5`, which A2's "timeout below interval" rule rejects. WS-A added `api_timeout_secs = 1` to that fixture; the SIGHUP test needed no change once `Config::load` stopped opening token files.
|