15 KiB
WS-B: Client, health, registry
Part of Phase 5: Consolidator. Contracts: contracts.md sections 1 to 4 and 8.
| Field | Value |
|---|---|
| Status | Merged |
| Owner | goose |
| Depends on | A0 (stubs and seams) and A1 to A3 for real config/secrets; can start on B1 as soon as A0 lands |
| Blocks | integration of C, D, E (they are developed against the fake registry and finish for real once B lands); F |
| Can start when | A0 is merged (design and client code can begin earlier in a branch against contracts.md) |
| Owned files | src/peers/client.rs, src/peers/health.rs, src/peers/registry.rs (bodies; signatures per contracts), src/api/ping.rs (new authenticated GET /api/v1/ping, see D4), Cargo.toml/Cargo.lock, src/main.rs (serve() only), src/tls.rs (SIGHUP fan-out line only, if the handler stays there), tests/peers_registry.rs, tests/common/mod.rs (append new helpers only) |
| Do not touch | src/peers/types.rs, transport.rs, error.rs, testing.rs (change via contracts.md only); src/peers/config.rs, secrets.rs, cli.rs, src/config.rs (WS-A); src/peers/proxy.rs, src/api/peers.rs, src/api/mod.rs, src/error.rs (WS-C / A0); src/mcp/** (D); src/telemetry/** (E); nix/**, docs/**, README.md (F, T) |
Shared-file seams:
src/state.rs: do not edit; A0 addedAppState.peers. B assigns it inserve()insrc/main.rs(state is built there byAppState::live, thenstate.peers = peers::start(&state)beforeserver::build_router). One edit region,serve(); WS-A edits theCommandenum elsewhere in the file.- SIGHUP:
serve()installs the handler andtls::spawn_sighup_reloadconsumes it. B adds the fan-out so the same signal also callsPeerRegistry::reload(and the plain-TLS-off branch no longer just logs when peers are enabled). Keep the TLS reload behaviour unchanged. src/api/mod.rs: untouched;ping.rsis already merged by A0. B only fillsping::routes().Cargo.toml/Cargo.lock: B owns dependency edits (reqwest direct dep per D15, features, any CIDR crate not already added by A). RegenerateCargo.lock, never hand-merge.tests/common/mod.rs: append-only helpers; WS-F appends too.
Goal
A PeerRegistry built from config that keeps live status; a bounded HTTP client.
Tasks
- B1 client: connect/request timeouts, redirects disabled, GET only, response byte cap enforced while streaming (not just
Content-Length), per-peerSemaphoreplus a global one, custom CA per peer, rustls with the ring provider passed explicitly;cargo tree -i opensslstays empty. ImplementsPeerTransportfor real (ReqwestTransport) and records the D15 outcome in the Log. - B2 health loops: one task per peer with its own child
CancellationTokenand start jitter; TCP check =TcpStream::connect(host:port)under timeout; API check = authenticatedGET /api/v1/ping; classificationok | unauthorized | http_error | timeout | unreachable | tls_error. Provideshealth::check_once(shared withslinuxm peer check, A4) and the/api/v1/pinghandler inping.rsreturning{status, version}; fallback for peers without/pingis/healthzplusauth = unknown(D4). - B3 status model (in memory only):
{server: {up, since, last_checked, last_error}, api: {up, since, last_checked, last_error, latency_ms, auth: ok|rejected|unknown, peer_version}}; flips needfail_thresholdconsecutive failures, recovery is immediate;sinceis the last change time and resets on restart. - B4 reload: on startup build the registry; on SIGHUP (shared handler with TLS) re-read only the
[peers]block and token files, start loops for new peers, cancel removed ones, restart changed ones; a bad reload keeps the old registry (same rule as TLS T3). Token files are re-read on every API check, so rotating an agenix secret needs no restart. Shutdown awaits all loops (bounded). - B5
PeerTransporttrait and fake; SSRF guard (D11) applied at config load and again at connect time on the resolved address. The trait and fake already exist from A0 (contracts section 2); B5 is now the SSRF guard on the real transport plus any fake extensions the loops need (changed through contracts.md).
Inputs
- contracts.md sections 1 to 4 (types, transport, registry API, error mapping), 6 (config), 10 (the test's timings and rotation flow).
- Fake:
FakeTransportandregistry_withfor loop/classification unit tests with paused time; real in-process servers for the integration test.
Done when
Integration test starts a second in-process server (fake sources) as a peer and observes up, down (port closed), api-down-but-server-up (listener that accepts and hangs), unauthorized (wrong token) and recovery; one hung peer does not delay another peer's check beyond its own timeout; clippy, fmt, cargo tree -i openssl clean.
Handoff
- Real
PeerRegistrywith the contract API (snapshot,status,get,reload,shutdown) assigned toAppState.peerswhenpeers.enabled. GET /api/v1/ping;ReqwestTransport;health::check_oncefor A4.- Post in C, D, E, F Logs when the registry is wired, so they swap their fakes for real end-to-end checks.
Log
-
2026-10-19: Split from the single phase plan 2026-10-19. The trait and fake moved to A0 so C/D/E are not blocked on this stream; B keeps the implementations.
-
2026-10-19 B1 (D15): chose reqwest (option a). Direct dependency
reqwest0.13.5 withdefault-features = falseandrustls-no-provideronly (nonative-tls, nostream— that feature cannot resolvewasm-streams— and norustlsfeature that selects aws-lc). Response bodies are capped by pollingchunk(). Each client is a rustlsClientConfigbuilt withbuilder_with_provider(ring)and webpki roots, plus an optional per-peer CA;install_defaultis not called.cargo tree -i opensslis empty.aws-lc-rsremains in the tree only throughopentelemetry-otlp's own reqwest and is not used by this client.src/peers/mod.rsis not owned here, soclient.rsis mounted aspeers::registry::client(ReqwestTransportis re-exported frompeers::registry). -
2026-10-19 B2: one task per peer (both probes in one
tokio::select!) with a deterministic name-hash jitter capped at a quarter of the shorter interval. Probes arehealth::probe_server/health::probe_api;check_oncejoins them for A4. A 401/403 isapi.up = true, auth = rejected; a 404 falls back to public/healthzwithauth = unknown(older peers, D4). The loop reusesFakeTransportfor the classification/threshold tests and a second in-processslinuxm(fake sources) for up/down/recovery, api-down-but-server-up (listener accepts, never answers) and rejected-token cases intests/peers_registry.rs. -
2026-10-19 B3:
ServerTracker/ApiTrackerkeepsinceand consecutive failures in loop-local state, so a surviving peer keeps its timestamps across a reload and a restart resets them.fail_thresholdconsecutive failures flip a check down; a success recovers in one check.last_erroris always the fixed class string (never peer text or a URL). The first check recordssincewhether it is up or down. -
2026-10-19 B4:
PeerRegistry::startbuilds the real transport and loops;src/main.rs::serve()assignsstate.peers. SIGHUP now reloads TLS (when enabled) and peers from one handler (tls::spawn_sighup_reload); a config error is logged and the old registry stays.reloadkeeps statuses of surviving peers, cancels removed/disabled/changed loops and starts new ones. Token files are read on every API check, so agenix rotation needs no restart.peersconcurrency and size caps are fixed for the process lifetime; only the peer set, intervals, timeouts and threshold follow a reload.PeerRegistry::shutdowncancels and joins loops with a 5 s bound. A reload with[peers] enabled = falseempties the snapshot (routes stay mounted until restart).main()passes the config path and CLI overrides intoserve()because a SIGHUP reload must re-read the same file; that is the only edit outsideserve(). -
2026-10-19 B5: connect-time SSRF guard resolves the peer host, rejects any address in
deny_cidrs(parsed withipnetatReqwestTransport::new), and pins the allowed address into the HTTP client; plain HTTP to a non-loopback host requiresallow_plain_http.tcp_checkuses the same resolution and guard. DNS failures map todns, all-denied tounreachable.
Handoff (2026-10-19)
Complete; B1 to B5 and the authenticated GET /api/v1/ping route are in. D15 outcome is recorded above (reqwest, rustls-no-provider with ring passed explicitly, body cap enforced while streaming; cargo tree -i openssl is empty). The registry API matches contracts section 3 and is already used by the A0 seams (snapshot, status, get, reload, shutdown); C/D/E can now switch their fake registries to PeerRegistry::start.
Verification run: cargo fmt --check, cargo clippy --all-targets -- -D warnings, cargo test (unit plus the in-process two-server cases in tests/peers_registry.rs), cargo tree -i openssl (empty). No Nix files were touched, so no VM test is involved; when the Nix gate is free, WS-T covers the module/VM path. Nothing in this workstream mints tokens or writes outside the state dir.
Round 1 review (2026-10-19)
Responses to loop/B/review-r.md; every Blocking and Major item is addressed, Minors where cheap.
-
Blocking 1 (stream cut off after
request_timeout_secs) — fixed.getno longer sets reqwest's total request timeout. Onewithin()deadline covers preparation (validation, DNS, token and CA reads, a permit, building the client, sending and response headers); a buffered body is read under the same deadline. A stream escapes that deadline and is bounded by an idle timeout instead (STREAM_IDLE_TIMEOUT, 120 s): peers send SSE keep-alives and end their own follow streams atlimits.max_follow_secs, so this only catches a dead connection. Testa_stream_outlives_the_request_timeoutsends chunks across ~750 ms with a 150 ms timeout and fails on the old code. -
Blocking 2 (token sent to disabled peers) — fixed.
PeerRegistry::getlooks the peer up withenabledandconfig.enabledapplied. A disabled peer (or a disabled feature) returnsNotFound(peer <name>: disabled, orunknown peerwhen the feature is off) and the transport is never called. Testa_disabled_peer_is_never_proxiedasserts the fake records no call before and after both kinds of reload. -
Blocking 3 (DNS/token/CA outside the timeout) — fixed.
tcp_checkandgetrun every phase inside a singlewithin(deadline, ...). Testone_deadline_covers_every_phasedrives the wrapper with a pending future on a paused clock (plus error and success pass-through). The health claim inhealth.rsnow holds for the transport too. -
Blocking 4 (IPv6 URLs) — fixed. Targets come from
url::Host(Ipv4/Ipv6/Domain); only a domain is resolved and pinned, andurl_is_loopbackuses the parsed host. Testsloopback_hosts_as_the_url_parser_reports_them,ipv6_plain_http_to_loopback_needs_no_opt_inandipv6_literal_peers_resolve_and_connect(skips when the environment has no::1). -
Major 5 (probes compete with proxy permits) — fixed.
/api/v1/pingand/healthzskip the per-peer and global semaphores; the proxy allowlist excludes both, so no caller can use the exemption for data traffic. Testprobes_bypass_the_concurrency_limitsholds a follow stream withmax_concurrent_per_peer = 1and still gets a ping through, while a data call isSaturated. -
Major 6 (reload only changes limits on paper) — fixed by rejecting such reloads.
deny_cidrs,max_response_bytes,max_concurrent_per_peerandmax_concurrent_totalcannot change without a restart; a reload that changes them fails and the old registry stays, soconfig()never reports a value that is not enforced. The deny list is now parsed by one shareddeny_networks()used at startup and on reload, so an invalid CIDR on SIGHUP is rejected instead of accepted and ignored. Testsreload_rejects_changes_the_transport_cannot_apply. -
Major 7 (token loader belongs to WS-A) — recorded disagreement. The rules are A3's, but
src/peers/secrets.rsdoes not exist in this branch and WS-A owns it.load_token_inhere is private to the transport and documented as the single loader until A3 lands; when it does,ReqwestTransportshould callpeers::secretsand this function should be deleted. Likewise#[path = "client.rs"] pub mod client;inregistry.rsis the workaround for not owningpeers/mod.rs; at merge the module should move tomod.rsandReqwestTransportbe re-exported frompeers. -
Major 8 (test gaps). Added:
sighup_reloads_peers_when_tls_is_off(TLS off, peers on; TLS off/peers off is still covered bytests/tls.rs::sighup_with_tls_disabled_keeps_serving),a_rotated_token_recovers_without_a_restartintests/peers_registry.rs(rejected -> rotate file ->auth = ok, no restart or SIGHUP), the threshold test now observes both checks flip down, andshutdown_waits_for_loops_retired_by_a_reload. The loop tests that do not need a real socket now run on paused time. -
Minor 9–14 — fixed.
request_urlcallsvalidate_targetinstead of repeating its rules; probes shareprobe_request;map_iolost the dead DNS string matches;error_is_tlsmatches structured error kinds (rustls downcast, or anio::Errorwhose kind is not a socket failure) instead of text; the AuthorizationHeaderValueisset_sensitive(true); a peer whose URL changes on reload starts from a clean status; loops retired by a reload are kept and awaited byshutdown; the/healthzfallback reports its own latency andApiTrackerkeeps the last knownpeer_versionwhile the API stays up. Tests were added for each. -
Minor 15 (scope).
src/tls.rschanges the SIGHUP handler's signature and importsPeersReloader; that is the minimum for one handler to reload TLS and peers with independent failures.main()changes only the arguments toserve(), as logged under B4. The handoff to the C/D/E/F logs still has to be posted by the orchestrator; this worktree does not own those files. -
2026-10-19 verification:
cargo fmt --check,cargo clippy --all-targets -- -D warnings,cargo testandcargo tree -i openssl(empty) are clean after the round-1 fixes.
Round 2 note from WS-A (2026-10-19)
real_transport()seam (forslinuxm peer check) — consumed by WS-A. B1/B2 being merged was the condition for wiring A4, andsrc/peers/cli.rsis A's file, so WS-A now buildsReqwestTransport::new(&config.peers)there (limits and deny list included) instead of bailing. No change is needed in B's files; the binary test now probes a real loopback server (tests/peers_config.rs,peer_check_binary_probes_a_real_peer_and_prints_latency).- Token reads are blocking (A3 loader).
peers::secrets::load_tokenis synchronous file I/O. Today the only per-check caller is B's transport, which reads inside its request deadline, so a slow read is bounded but still occupies an async worker. If that path gets hotter (many peers, short intervals), move the read tospawn_blockingor a size-boundedtokio::fsread; the loader stays A's contract. - Test fixture fix in this workstream's file. On the rebase with WS-A,
reloader_rereads_the_peers_block_and_keeps_the_old_set_on_errorfailed because its inline[peers]usedapi_interval_secs = 2with the defaultapi_timeout_secs = 5, which A2's "timeout below interval" rule rejects. WS-A addedapi_timeout_secs = 1to that fixture; the SIGHUP test needed no change onceConfig::loadstopped opening token files.