7.2 KiB
WS-E: Telemetry
Part of Phase 5: Consolidator. Contracts: contracts.md sections 3 and 8.
| Field | Value |
|---|---|
| Status | Merged |
| Owner | goose |
| Depends on | B (registry). In practice: A0 for compiling; read the snapshot from static_registry / registry_with until B lands |
| Blocks | F; WS-T goes green at the end of E |
| Can start when | A0 is merged |
| Owned files | src/telemetry/metrics_export.rs, src/telemetry/mod.rs |
| Do not touch | src/peers/** (A, B, C), src/api/**, src/mcp/**, src/state.rs, src/server.rs, src/main.rs, Cargo.toml (B; if a new telemetry dependency is truly needed, ask B and log it), nix/**, dev/otelcol.yaml (E may add peer series examples only if needed, note in Log), docs/** |
Shared-file seams: telemetry reads state.peers (the field exists from A0, set by B in serve()); E does not edit state.rs/main.rs. spawn_exporters(state) already receives AppState, so no signature change is expected. inject_traceparent() (E2) is exported from telemetry/mod.rs and called by src/peers/proxy.rs/client via ProxyOpts.traceparent (contracts section 8); until E2 lands ProxyOpts.traceparent is simply None.
Goal
Peer health in OTel, peer calls in traces.
Tasks
- E1 observable gauges (f64) fed from the registry snapshot, so removed peers vanish (G11); no polling of peers by the exporter, it reads the registry. Metric table in D10 (names and attributes frozen in contracts section 8).
- E2 outbound client spans with
peer.name; injecttraceparentinto peer requests.
Inputs
- Contracts section 8 (metric names, attributes, units) and 3 (
snapshot); decisions D10. - Fakes:
static_registry(Vec<PeerStatus>)to feed the gauges;FakeTransportcall records to asserttraceparentinjection.
Done when
Metrics test with the fake registry; with otel.enabled and no peers nothing new is exported; devenv up collector shows peer.up series for two peers and drops one after peer remove. (Peers are declarative, so "peer remove" means removing the entry from config and sending SIGHUP; see D1.)
Handoff
- Working gauges and
inject_traceparentfor WS-B/C wiring and WS-F's docs. - Signal to WS-T and WS-F: with E, B, C, D merged the VM test should pass; F2 then promotes it. E also unlocks adding the
peer.api.auth_ok = 0check from T4 if wanted.
Log
-
2026-10-19: Split from the single phase plan 2026-10-19. Note added that "peer remove" in the collector check means removal from config plus SIGHUP (D1).
-
2026-10-19 (A0): contract stubs landed.
PeerRegistry::snapshot()returnsPeerStatussorted by name (jiff timestamps,since: Noneuntil a check),AppState.peersisNonewhen the feature is off, andtesting::static_registryrenders fixed snapshots for metric tests. -
2026-10-19 (E1): observable peer gauges in
src/telemetry/metrics_export.rs.PEER_METRICSholds the contract names and units;peer_observations(&[PeerStatus])maps the registry snapshot (both checks,auth_okseparate, latency in seconds,sinceas unix seconds, no URL or version attributes).spawnregisters them only whenstate.peersisSome; the callback readsPeerRegistry::snapshot()on every collection, so the exporter never polls a peer and a peer removed by config + SIGHUP leaves the next export. No new background task, no new dependency, no feature flag. One interpretation recorded: disabled peers are omitted from the gauges (the contract is silent; a0would read as "down" and page for a peer the operator switched off deliberately — no name or attribute changed). -
2026-10-19 (E1 tests): contract names/units, mapping per check,
auth_okforok/rejected/unknown, milliseconds-to-seconds conversion, unixsince, absence of URL/version attributes, no point for a missing latency orsince, disabled peer omitted, series vanish after the snapshot shrinks, and collection makes noFakeTransportcall. Two SDK-level tests drive the realspawnthrough aPeriodicReaderwith a capturingPushMetricExporter(both public and already in the tree: no feature such asexperimental_metrics_custom_reader, no new dependency): one asserts the exported points and that a removed peer's series disappear, the other that a peerless state exports nopeer.*series. -
2026-10-19 (E2):
telemetry::peer_request_span(peer)createspeer.requestwith the single attributepeer.name;telemetry::inject_traceparent()returns the W3C header for the current span (ids and flags from the span context) andNonewithout a valid OpenTelemetry context, so a peer-less or OTel-disabled instance is unchanged. Tests: span name and field, header shape with ids equal to the span context,Noneoutside a span and without the OTel layer, and propagation end to end throughPeerRegistry::get+FakeTransport(header present, noAuthorization, the outbound span is the parent).ProxyOpts.traceparentstaysNoneuntil WS-C/WS-D build their requests inside that span; see the Handoff for the exact call pattern. -
2026-10-19 (Handoff): E1 and E2 are implemented and tested against
registry_with,static_registryandFakeTransport; notests/file was added because this workstream owns notests/path and the shared fakes arepub, so the unit tests use them directly.Wiring required outside E's files:
- B (
serve()):spawn_exporters(state)registers the peer gauges at call time, sostate.peersmust be assigned beforetelemetry::spawn_exporters(state.clone()). Assigning it only just beforeserver::build_routerleaves the peer metrics empty until a restart. SIGHUP reload must keep mutating the sameArc<PeerRegistry>(PeerRegistry::reload); swappingAppState.peerswould leave the gauges reading the old registry. E cannot enforce either from its own files. - C/D: wrap the proxy call in the outbound span and inject inside it, e.g.
let span = telemetry::peer_request_span(name);then inside the span (or via.instrument(span)) buildProxyOpts { traceparent: telemetry::inject_traceparent(), accept_sse }and callPeerRegistry::get. Nothing else is needed; the proxy layer already copiestraceparentinto the peer request.
Verification commands (not run in this worktree; no nix builds, and the collector check needs a live instance):
- Collector:
devenv up(OTLP on127.0.0.1:4317, debug exporter) with[otel] enabled = true;dev/otelcol.yamlneeds no change because the debug exporter prints every metric. Expectpeer.upwithpeer.check=server|api,peer.api.auth_ok,peer.check.duration_secondsandpeer.last_change_timestamp_seconds; remove the[[peers.peer]]entry, send SIGHUP, and the series stop after oneotel.metrics_interval_secs. - VM test (WS-T):
nix build .#tests.x86_64-linux.peers -L; once promoted in F2,nix build .#checks.x86_64-linux.peers -L. - Docs (F/orchestrator, since E owns no docs): the README metrics paragraph and the ARCHITECTURE.md metric table need the four
peer.*gauges, thepeer.name/peer.checkattributes, and the note that they are registered only with[peers] enabled.docs/consolidator.mdshould state that removing a peer from config and sending SIGHUP stops its series (no history is kept).
- B (