Files
2026-10-20 16:35:11 -04:00

7.2 KiB

WS-E: Telemetry

Part of Phase 5: Consolidator. Contracts: contracts.md sections 3 and 8.

Field Value
Status Merged
Owner goose
Depends on B (registry). In practice: A0 for compiling; read the snapshot from static_registry / registry_with until B lands
Blocks F; WS-T goes green at the end of E
Can start when A0 is merged
Owned files src/telemetry/metrics_export.rs, src/telemetry/mod.rs
Do not touch src/peers/** (A, B, C), src/api/**, src/mcp/**, src/state.rs, src/server.rs, src/main.rs, Cargo.toml (B; if a new telemetry dependency is truly needed, ask B and log it), nix/**, dev/otelcol.yaml (E may add peer series examples only if needed, note in Log), docs/**

Shared-file seams: telemetry reads state.peers (the field exists from A0, set by B in serve()); E does not edit state.rs/main.rs. spawn_exporters(state) already receives AppState, so no signature change is expected. inject_traceparent() (E2) is exported from telemetry/mod.rs and called by src/peers/proxy.rs/client via ProxyOpts.traceparent (contracts section 8); until E2 lands ProxyOpts.traceparent is simply None.

Goal

Peer health in OTel, peer calls in traces.

Tasks

  • E1 observable gauges (f64) fed from the registry snapshot, so removed peers vanish (G11); no polling of peers by the exporter, it reads the registry. Metric table in D10 (names and attributes frozen in contracts section 8).
  • E2 outbound client spans with peer.name; inject traceparent into peer requests.

Inputs

  • Contracts section 8 (metric names, attributes, units) and 3 (snapshot); decisions D10.
  • Fakes: static_registry(Vec<PeerStatus>) to feed the gauges; FakeTransport call records to assert traceparent injection.

Done when

Metrics test with the fake registry; with otel.enabled and no peers nothing new is exported; devenv up collector shows peer.up series for two peers and drops one after peer remove. (Peers are declarative, so "peer remove" means removing the entry from config and sending SIGHUP; see D1.)

Handoff

  • Working gauges and inject_traceparent for WS-B/C wiring and WS-F's docs.
  • Signal to WS-T and WS-F: with E, B, C, D merged the VM test should pass; F2 then promotes it. E also unlocks adding the peer.api.auth_ok = 0 check from T4 if wanted.

Log

  • 2026-10-19: Split from the single phase plan 2026-10-19. Note added that "peer remove" in the collector check means removal from config plus SIGHUP (D1).

  • 2026-10-19 (A0): contract stubs landed. PeerRegistry::snapshot() returns PeerStatus sorted by name (jiff timestamps, since: None until a check), AppState.peers is None when the feature is off, and testing::static_registry renders fixed snapshots for metric tests.

  • 2026-10-19 (E1): observable peer gauges in src/telemetry/metrics_export.rs. PEER_METRICS holds the contract names and units; peer_observations(&[PeerStatus]) maps the registry snapshot (both checks, auth_ok separate, latency in seconds, since as unix seconds, no URL or version attributes). spawn registers them only when state.peers is Some; the callback reads PeerRegistry::snapshot() on every collection, so the exporter never polls a peer and a peer removed by config + SIGHUP leaves the next export. No new background task, no new dependency, no feature flag. One interpretation recorded: disabled peers are omitted from the gauges (the contract is silent; a 0 would read as "down" and page for a peer the operator switched off deliberately — no name or attribute changed).

  • 2026-10-19 (E1 tests): contract names/units, mapping per check, auth_ok for ok/rejected/unknown, milliseconds-to-seconds conversion, unix since, absence of URL/version attributes, no point for a missing latency or since, disabled peer omitted, series vanish after the snapshot shrinks, and collection makes no FakeTransport call. Two SDK-level tests drive the real spawn through a PeriodicReader with a capturing PushMetricExporter (both public and already in the tree: no feature such as experimental_metrics_custom_reader, no new dependency): one asserts the exported points and that a removed peer's series disappear, the other that a peerless state exports no peer.* series.

  • 2026-10-19 (E2): telemetry::peer_request_span(peer) creates peer.request with the single attribute peer.name; telemetry::inject_traceparent() returns the W3C header for the current span (ids and flags from the span context) and None without a valid OpenTelemetry context, so a peer-less or OTel-disabled instance is unchanged. Tests: span name and field, header shape with ids equal to the span context, None outside a span and without the OTel layer, and propagation end to end through PeerRegistry::get + FakeTransport (header present, no Authorization, the outbound span is the parent). ProxyOpts.traceparent stays None until WS-C/WS-D build their requests inside that span; see the Handoff for the exact call pattern.

  • 2026-10-19 (Handoff): E1 and E2 are implemented and tested against registry_with, static_registry and FakeTransport; no tests/ file was added because this workstream owns no tests/ path and the shared fakes are pub, so the unit tests use them directly.

    Wiring required outside E's files:

    • B (serve()): spawn_exporters(state) registers the peer gauges at call time, so state.peers must be assigned before telemetry::spawn_exporters(state.clone()). Assigning it only just before server::build_router leaves the peer metrics empty until a restart. SIGHUP reload must keep mutating the same Arc<PeerRegistry> (PeerRegistry::reload); swapping AppState.peers would leave the gauges reading the old registry. E cannot enforce either from its own files.
    • C/D: wrap the proxy call in the outbound span and inject inside it, e.g. let span = telemetry::peer_request_span(name); then inside the span (or via .instrument(span)) build ProxyOpts { traceparent: telemetry::inject_traceparent(), accept_sse } and call PeerRegistry::get. Nothing else is needed; the proxy layer already copies traceparent into the peer request.

    Verification commands (not run in this worktree; no nix builds, and the collector check needs a live instance):

    • Collector: devenv up (OTLP on 127.0.0.1:4317, debug exporter) with [otel] enabled = true; dev/otelcol.yaml needs no change because the debug exporter prints every metric. Expect peer.up with peer.check=server|api, peer.api.auth_ok, peer.check.duration_seconds and peer.last_change_timestamp_seconds; remove the [[peers.peer]] entry, send SIGHUP, and the series stop after one otel.metrics_interval_secs.
    • VM test (WS-T): nix build .#tests.x86_64-linux.peers -L; once promoted in F2, nix build .#checks.x86_64-linux.peers -L.
    • Docs (F/orchestrator, since E owns no docs): the README metrics paragraph and the ARCHITECTURE.md metric table need the four peer.* gauges, the peer.name/peer.check attributes, and the note that they are registered only with [peers] enabled. docs/consolidator.md should state that removing a peer from config and sending SIGHUP stops its series (no history is kept).