Files
skyg-linux-tracker/plan/phase-5-consolidator/ws-b-registry.md
T

15 KiB
Raw Blame History

WS-B: Client, health, registry

Part of Phase 5: Consolidator. Contracts: contracts.md sections 1 to 4 and 8.

Field Value
Status Merged
Owner goose
Depends on A0 (stubs and seams) and A1 to A3 for real config/secrets; can start on B1 as soon as A0 lands
Blocks integration of C, D, E (they are developed against the fake registry and finish for real once B lands); F
Can start when A0 is merged (design and client code can begin earlier in a branch against contracts.md)
Owned files src/peers/client.rs, src/peers/health.rs, src/peers/registry.rs (bodies; signatures per contracts), src/api/ping.rs (new authenticated GET /api/v1/ping, see D4), Cargo.toml/Cargo.lock, src/main.rs (serve() only), src/tls.rs (SIGHUP fan-out line only, if the handler stays there), tests/peers_registry.rs, tests/common/mod.rs (append new helpers only)
Do not touch src/peers/types.rs, transport.rs, error.rs, testing.rs (change via contracts.md only); src/peers/config.rs, secrets.rs, cli.rs, src/config.rs (WS-A); src/peers/proxy.rs, src/api/peers.rs, src/api/mod.rs, src/error.rs (WS-C / A0); src/mcp/** (D); src/telemetry/** (E); nix/**, docs/**, README.md (F, T)

Shared-file seams:

  • src/state.rs: do not edit; A0 added AppState.peers. B assigns it in serve() in src/main.rs (state is built there by AppState::live, then state.peers = peers::start(&state) before server::build_router). One edit region, serve(); WS-A edits the Command enum elsewhere in the file.
  • SIGHUP: serve() installs the handler and tls::spawn_sighup_reload consumes it. B adds the fan-out so the same signal also calls PeerRegistry::reload (and the plain-TLS-off branch no longer just logs when peers are enabled). Keep the TLS reload behaviour unchanged.
  • src/api/mod.rs: untouched; ping.rs is already merged by A0. B only fills ping::routes().
  • Cargo.toml/Cargo.lock: B owns dependency edits (reqwest direct dep per D15, features, any CIDR crate not already added by A). Regenerate Cargo.lock, never hand-merge.
  • tests/common/mod.rs: append-only helpers; WS-F appends too.

Goal

A PeerRegistry built from config that keeps live status; a bounded HTTP client.

Tasks

  • B1 client: connect/request timeouts, redirects disabled, GET only, response byte cap enforced while streaming (not just Content-Length), per-peer Semaphore plus a global one, custom CA per peer, rustls with the ring provider passed explicitly; cargo tree -i openssl stays empty. Implements PeerTransport for real (ReqwestTransport) and records the D15 outcome in the Log.
  • B2 health loops: one task per peer with its own child CancellationToken and start jitter; TCP check = TcpStream::connect(host:port) under timeout; API check = authenticated GET /api/v1/ping; classification ok | unauthorized | http_error | timeout | unreachable | tls_error. Provides health::check_once (shared with slinuxm peer check, A4) and the /api/v1/ping handler in ping.rs returning {status, version}; fallback for peers without /ping is /healthz plus auth = unknown (D4).
  • B3 status model (in memory only): {server: {up, since, last_checked, last_error}, api: {up, since, last_checked, last_error, latency_ms, auth: ok|rejected|unknown, peer_version}}; flips need fail_threshold consecutive failures, recovery is immediate; since is the last change time and resets on restart.
  • B4 reload: on startup build the registry; on SIGHUP (shared handler with TLS) re-read only the [peers] block and token files, start loops for new peers, cancel removed ones, restart changed ones; a bad reload keeps the old registry (same rule as TLS T3). Token files are re-read on every API check, so rotating an agenix secret needs no restart. Shutdown awaits all loops (bounded).
  • B5 PeerTransport trait and fake; SSRF guard (D11) applied at config load and again at connect time on the resolved address. The trait and fake already exist from A0 (contracts section 2); B5 is now the SSRF guard on the real transport plus any fake extensions the loops need (changed through contracts.md).

Inputs

  • contracts.md sections 1 to 4 (types, transport, registry API, error mapping), 6 (config), 10 (the test's timings and rotation flow).
  • Fake: FakeTransport and registry_with for loop/classification unit tests with paused time; real in-process servers for the integration test.

Done when

Integration test starts a second in-process server (fake sources) as a peer and observes up, down (port closed), api-down-but-server-up (listener that accepts and hangs), unauthorized (wrong token) and recovery; one hung peer does not delay another peer's check beyond its own timeout; clippy, fmt, cargo tree -i openssl clean.

Handoff

  • Real PeerRegistry with the contract API (snapshot, status, get, reload, shutdown) assigned to AppState.peers when peers.enabled.
  • GET /api/v1/ping; ReqwestTransport; health::check_once for A4.
  • Post in C, D, E, F Logs when the registry is wired, so they swap their fakes for real end-to-end checks.

Log

  • 2026-10-19: Split from the single phase plan 2026-10-19. The trait and fake moved to A0 so C/D/E are not blocked on this stream; B keeps the implementations.

  • 2026-10-19 B1 (D15): chose reqwest (option a). Direct dependency reqwest 0.13.5 with default-features = false and rustls-no-provider only (no native-tls, no stream — that feature cannot resolve wasm-streams — and no rustls feature that selects aws-lc). Response bodies are capped by polling chunk(). Each client is a rustls ClientConfig built with builder_with_provider(ring) and webpki roots, plus an optional per-peer CA; install_default is not called. cargo tree -i openssl is empty. aws-lc-rs remains in the tree only through opentelemetry-otlp's own reqwest and is not used by this client. src/peers/mod.rs is not owned here, so client.rs is mounted as peers::registry::client (ReqwestTransport is re-exported from peers::registry).

  • 2026-10-19 B2: one task per peer (both probes in one tokio::select!) with a deterministic name-hash jitter capped at a quarter of the shorter interval. Probes are health::probe_server / health::probe_api; check_once joins them for A4. A 401/403 is api.up = true, auth = rejected; a 404 falls back to public /healthz with auth = unknown (older peers, D4). The loop reuses FakeTransport for the classification/threshold tests and a second in-process slinuxm (fake sources) for up/down/recovery, api-down-but-server-up (listener accepts, never answers) and rejected-token cases in tests/peers_registry.rs.

  • 2026-10-19 B3: ServerTracker/ApiTracker keep since and consecutive failures in loop-local state, so a surviving peer keeps its timestamps across a reload and a restart resets them. fail_threshold consecutive failures flip a check down; a success recovers in one check. last_error is always the fixed class string (never peer text or a URL). The first check records since whether it is up or down.

  • 2026-10-19 B4: PeerRegistry::start builds the real transport and loops; src/main.rs::serve() assigns state.peers. SIGHUP now reloads TLS (when enabled) and peers from one handler (tls::spawn_sighup_reload); a config error is logged and the old registry stays. reload keeps statuses of surviving peers, cancels removed/disabled/changed loops and starts new ones. Token files are read on every API check, so agenix rotation needs no restart. peers concurrency and size caps are fixed for the process lifetime; only the peer set, intervals, timeouts and threshold follow a reload. PeerRegistry::shutdown cancels and joins loops with a 5 s bound. A reload with [peers] enabled = false empties the snapshot (routes stay mounted until restart). main() passes the config path and CLI overrides into serve() because a SIGHUP reload must re-read the same file; that is the only edit outside serve().

  • 2026-10-19 B5: connect-time SSRF guard resolves the peer host, rejects any address in deny_cidrs (parsed with ipnet at ReqwestTransport::new), and pins the allowed address into the HTTP client; plain HTTP to a non-loopback host requires allow_plain_http. tcp_check uses the same resolution and guard. DNS failures map to dns, all-denied to unreachable.

Handoff (2026-10-19)

Complete; B1 to B5 and the authenticated GET /api/v1/ping route are in. D15 outcome is recorded above (reqwest, rustls-no-provider with ring passed explicitly, body cap enforced while streaming; cargo tree -i openssl is empty). The registry API matches contracts section 3 and is already used by the A0 seams (snapshot, status, get, reload, shutdown); C/D/E can now switch their fake registries to PeerRegistry::start.

Verification run: cargo fmt --check, cargo clippy --all-targets -- -D warnings, cargo test (unit plus the in-process two-server cases in tests/peers_registry.rs), cargo tree -i openssl (empty). No Nix files were touched, so no VM test is involved; when the Nix gate is free, WS-T covers the module/VM path. Nothing in this workstream mints tokens or writes outside the state dir.

Round 1 review (2026-10-19)

Responses to loop/B/review-r.md; every Blocking and Major item is addressed, Minors where cheap.

  • Blocking 1 (stream cut off after request_timeout_secs) — fixed. get no longer sets reqwest's total request timeout. One within() deadline covers preparation (validation, DNS, token and CA reads, a permit, building the client, sending and response headers); a buffered body is read under the same deadline. A stream escapes that deadline and is bounded by an idle timeout instead (STREAM_IDLE_TIMEOUT, 120 s): peers send SSE keep-alives and end their own follow streams at limits.max_follow_secs, so this only catches a dead connection. Test a_stream_outlives_the_request_timeout sends chunks across ~750 ms with a 150 ms timeout and fails on the old code.

  • Blocking 2 (token sent to disabled peers) — fixed. PeerRegistry::get looks the peer up with enabled and config.enabled applied. A disabled peer (or a disabled feature) returns NotFound (peer <name>: disabled, or unknown peer when the feature is off) and the transport is never called. Test a_disabled_peer_is_never_proxied asserts the fake records no call before and after both kinds of reload.

  • Blocking 3 (DNS/token/CA outside the timeout) — fixed. tcp_check and get run every phase inside a single within(deadline, ...). Test one_deadline_covers_every_phase drives the wrapper with a pending future on a paused clock (plus error and success pass-through). The health claim in health.rs now holds for the transport too.

  • Blocking 4 (IPv6 URLs) — fixed. Targets come from url::Host (Ipv4/Ipv6/Domain); only a domain is resolved and pinned, and url_is_loopback uses the parsed host. Tests loopback_hosts_as_the_url_parser_reports_them, ipv6_plain_http_to_loopback_needs_no_opt_in and ipv6_literal_peers_resolve_and_connect (skips when the environment has no ::1).

  • Major 5 (probes compete with proxy permits) — fixed. /api/v1/ping and /healthz skip the per-peer and global semaphores; the proxy allowlist excludes both, so no caller can use the exemption for data traffic. Test probes_bypass_the_concurrency_limits holds a follow stream with max_concurrent_per_peer = 1 and still gets a ping through, while a data call is Saturated.

  • Major 6 (reload only changes limits on paper) — fixed by rejecting such reloads. deny_cidrs, max_response_bytes, max_concurrent_per_peer and max_concurrent_total cannot change without a restart; a reload that changes them fails and the old registry stays, so config() never reports a value that is not enforced. The deny list is now parsed by one shared deny_networks() used at startup and on reload, so an invalid CIDR on SIGHUP is rejected instead of accepted and ignored. Tests reload_rejects_changes_the_transport_cannot_apply.

  • Major 7 (token loader belongs to WS-A) — recorded disagreement. The rules are A3's, but src/peers/secrets.rs does not exist in this branch and WS-A owns it. load_token_in here is private to the transport and documented as the single loader until A3 lands; when it does, ReqwestTransport should call peers::secrets and this function should be deleted. Likewise #[path = "client.rs"] pub mod client; in registry.rs is the workaround for not owning peers/mod.rs; at merge the module should move to mod.rs and ReqwestTransport be re-exported from peers.

  • Major 8 (test gaps). Added: sighup_reloads_peers_when_tls_is_off (TLS off, peers on; TLS off/peers off is still covered by tests/tls.rs::sighup_with_tls_disabled_keeps_serving), a_rotated_token_recovers_without_a_restart in tests/peers_registry.rs (rejected -> rotate file -> auth = ok, no restart or SIGHUP), the threshold test now observes both checks flip down, and shutdown_waits_for_loops_retired_by_a_reload. The loop tests that do not need a real socket now run on paused time.

  • Minor 9–14 — fixed. request_url calls validate_target instead of repeating its rules; probes share probe_request; map_io lost the dead DNS string matches; error_is_tls matches structured error kinds (rustls downcast, or an io::Error whose kind is not a socket failure) instead of text; the Authorization HeaderValue is set_sensitive(true); a peer whose URL changes on reload starts from a clean status; loops retired by a reload are kept and awaited by shutdown; the /healthz fallback reports its own latency and ApiTracker keeps the last known peer_version while the API stays up. Tests were added for each.

  • Minor 15 (scope). src/tls.rs changes the SIGHUP handler's signature and imports PeersReloader; that is the minimum for one handler to reload TLS and peers with independent failures. main() changes only the arguments to serve(), as logged under B4. The handoff to the C/D/E/F logs still has to be posted by the orchestrator; this worktree does not own those files.

  • 2026-10-19 verification: cargo fmt --check, cargo clippy --all-targets -- -D warnings, cargo test and cargo tree -i openssl (empty) are clean after the round-1 fixes.

Round 2 note from WS-A (2026-10-19)

  • real_transport() seam (for slinuxm peer check) — consumed by WS-A. B1/B2 being merged was the condition for wiring A4, and src/peers/cli.rs is A's file, so WS-A now builds ReqwestTransport::new(&config.peers) there (limits and deny list included) instead of bailing. No change is needed in B's files; the binary test now probes a real loopback server (tests/peers_config.rs, peer_check_binary_probes_a_real_peer_and_prints_latency).
  • Token reads are blocking (A3 loader). peers::secrets::load_token is synchronous file I/O. Today the only per-check caller is B's transport, which reads inside its request deadline, so a slow read is bounded but still occupies an async worker. If that path gets hotter (many peers, short intervals), move the read to spawn_blocking or a size-bounded tokio::fs read; the loader stays A's contract.
  • Test fixture fix in this workstream's file. On the rebase with WS-A, reloader_rereads_the_peers_block_and_keeps_the_old_set_on_error failed because its inline [peers] used api_interval_secs = 2 with the default api_timeout_secs = 5, which A2's "timeout below interval" rule rejects. WS-A added api_timeout_secs = 1 to that fixture; the SIGHUP test needed no change once Config::load stopped opening token files.