From 7fe498ed669276b1ef33eb5e28f1bb91ca3adbda Mon Sep 17 00:00:00 2001 From: ari Date: Mon, 19 Oct 2026 20:51:47 -0400 Subject: [PATCH] E: record E1 and E2 results, seams and handoff in the workstream log --- plan/phase-5-consolidator/ws-e-telemetry.md | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/plan/phase-5-consolidator/ws-e-telemetry.md b/plan/phase-5-consolidator/ws-e-telemetry.md index 3418007..66d6aca 100644 --- a/plan/phase-5-consolidator/ws-e-telemetry.md +++ b/plan/phase-5-consolidator/ws-e-telemetry.md @@ -4,8 +4,8 @@ Part of [Phase 5: Consolidator](README.md). Contracts: [contracts.md](contracts. | Field | Value | |---|---| -| Status | Not started | -| Owner | | +| Status | E1, E2 implemented; end-to-end with C/D wiring pending | +| Owner | goose | | Depends on | B (registry). In practice: A0 for compiling; read the snapshot from `static_registry` / `registry_with` until B lands | | Blocks | F; [WS-T](ws-t-tests.md) goes green at the end of E | | Can start when | A0 is merged | @@ -41,3 +41,18 @@ Metrics test with the fake registry; with `otel.enabled` and no peers nothing ne - 2026-10-19: Split from the single phase plan 2026-10-19. Note added that "peer remove" in the collector check means removal from config plus SIGHUP (D1). - 2026-10-19 (A0): contract stubs landed. `PeerRegistry::snapshot()` returns `PeerStatus` sorted by name (jiff timestamps, `since: None` until a check), `AppState.peers` is `None` when the feature is off, and `testing::static_registry` renders fixed snapshots for metric tests. +- 2026-10-19 (E1): observable peer gauges in `src/telemetry/metrics_export.rs`. `PEER_METRICS` holds the contract names and units; `peer_observations(&[PeerStatus])` maps the registry snapshot (both checks, `auth_ok` separate, latency in seconds, `since` as unix seconds, no URL or version attributes). `spawn` registers them only when `state.peers` is `Some`; the callback reads `PeerRegistry::snapshot()` on every collection, so the exporter never polls a peer and a peer removed by config + SIGHUP leaves the next export. No new background task, no new dependency, no feature flag. One interpretation recorded: disabled peers are omitted from the gauges (the contract is silent; a `0` would read as "down" and page for a peer the operator switched off deliberately — no name or attribute changed). +- 2026-10-19 (E1 tests): contract names/units, mapping per check, `auth_ok` for `ok`/`rejected`/`unknown`, milliseconds-to-seconds conversion, unix `since`, absence of URL/version attributes, no point for a missing latency or `since`, disabled peer omitted, series vanish after the snapshot shrinks, and collection makes no `FakeTransport` call. Two SDK-level tests drive the real `spawn` through a `PeriodicReader` with a capturing `PushMetricExporter` (both public and already in the tree: no feature such as `experimental_metrics_custom_reader`, no new dependency): one asserts the exported points and that a removed peer's series disappear, the other that a peerless state exports no `peer.*` series. +- 2026-10-19 (E2): `telemetry::peer_request_span(peer)` creates `peer.request` with the single attribute `peer.name`; `telemetry::inject_traceparent()` returns the W3C header for the current span (ids and flags from the span context) and `None` without a valid OpenTelemetry context, so a peer-less or OTel-disabled instance is unchanged. Tests: span name and field, header shape with ids equal to the span context, `None` outside a span and without the OTel layer, and propagation end to end through `PeerRegistry::get` + `FakeTransport` (header present, no `Authorization`, the outbound span is the parent). `ProxyOpts.traceparent` stays `None` until WS-C/WS-D build their requests inside that span; see the Handoff for the exact call pattern. +- 2026-10-19 (Handoff): E1 and E2 are implemented and tested against `registry_with`, `static_registry` and `FakeTransport`; no `tests/` file was added because this workstream owns no `tests/` path and the shared fakes are `pub`, so the unit tests use them directly. + + Wiring required outside E's files: + + - **B (`serve()`)**: `spawn_exporters(state)` registers the peer gauges at call time, so `state.peers` must be assigned *before* `telemetry::spawn_exporters(state.clone())`. Assigning it only just before `server::build_router` leaves the peer metrics empty until a restart. SIGHUP reload must keep mutating the same `Arc` (`PeerRegistry::reload`); swapping `AppState.peers` would leave the gauges reading the old registry. E cannot enforce either from its own files. + - **C/D**: wrap the proxy call in the outbound span and inject inside it, e.g. `let span = telemetry::peer_request_span(name);` then inside the span (or via `.instrument(span)`) build `ProxyOpts { traceparent: telemetry::inject_traceparent(), accept_sse }` and call `PeerRegistry::get`. Nothing else is needed; the proxy layer already copies `traceparent` into the peer request. + + Verification commands (not run in this worktree; no nix builds, and the collector check needs a live instance): + + - Collector: `devenv up` (OTLP on `127.0.0.1:4317`, debug exporter) with `[otel] enabled = true`; `dev/otelcol.yaml` needs no change because the debug exporter prints every metric. Expect `peer.up` with `peer.check=server|api`, `peer.api.auth_ok`, `peer.check.duration_seconds` and `peer.last_change_timestamp_seconds`; remove the `[[peers.peer]]` entry, send SIGHUP, and the series stop after one `otel.metrics_interval_secs`. + - VM test (WS-T): `nix build .#tests.x86_64-linux.peers -L`; once promoted in F2, `nix build .#checks.x86_64-linux.peers -L`. + - Docs (F/orchestrator, since E owns no docs): the README metrics paragraph and the ARCHITECTURE.md metric table need the four `peer.*` gauges, the `peer.name`/`peer.check` attributes, and the note that they are registered only with `[peers] enabled`. `docs/consolidator.md` should state that removing a peer from config and sending SIGHUP stops its series (no history is kept).