59 lines
7.2 KiB
Markdown
59 lines
7.2 KiB
Markdown
# WS-E: Telemetry
|
|
|
|
Part of [Phase 5: Consolidator](README.md). Contracts: [contracts.md](contracts.md) sections 3 and 8.
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Status | Merged |
|
|
| Owner | goose |
|
|
| Depends on | B (registry). In practice: A0 for compiling; read the snapshot from `static_registry` / `registry_with` until B lands |
|
|
| Blocks | F; [WS-T](ws-t-tests.md) goes green at the end of E |
|
|
| Can start when | A0 is merged |
|
|
| Owned files | `src/telemetry/metrics_export.rs`, `src/telemetry/mod.rs` |
|
|
| Do not touch | `src/peers/**` (A, B, C), `src/api/**`, `src/mcp/**`, `src/state.rs`, `src/server.rs`, `src/main.rs`, `Cargo.toml` (B; if a new telemetry dependency is truly needed, ask B and log it), `nix/**`, `dev/otelcol.yaml` (E may add peer series examples only if needed, note in Log), `docs/**` |
|
|
|
|
Shared-file seams: telemetry reads `state.peers` (the field exists from A0, set by B in `serve()`); E does not edit `state.rs`/`main.rs`. `spawn_exporters(state)` already receives `AppState`, so no signature change is expected. `inject_traceparent()` (E2) is exported from `telemetry/mod.rs` and called by `src/peers/proxy.rs`/client via `ProxyOpts.traceparent` (contracts section 8); until E2 lands `ProxyOpts.traceparent` is simply `None`.
|
|
|
|
## Goal
|
|
|
|
Peer health in OTel, peer calls in traces.
|
|
|
|
## Tasks
|
|
|
|
- **E1** observable gauges (f64) fed from the registry snapshot, so removed peers vanish (G11); no polling of peers by the exporter, it reads the registry. Metric table in D10 (names and attributes frozen in contracts section 8).
|
|
- **E2** outbound client spans with `peer.name`; inject `traceparent` into peer requests.
|
|
|
|
## Inputs
|
|
|
|
- Contracts section 8 (metric names, attributes, units) and 3 (`snapshot`); decisions D10.
|
|
- Fakes: `static_registry(Vec<PeerStatus>)` to feed the gauges; `FakeTransport` call records to assert `traceparent` injection.
|
|
|
|
## Done when
|
|
|
|
Metrics test with the fake registry; with `otel.enabled` and no peers nothing new is exported; `devenv up` collector shows `peer.up` series for two peers and drops one after `peer remove`. (Peers are declarative, so "peer remove" means removing the entry from config and sending SIGHUP; see D1.)
|
|
|
|
## Handoff
|
|
|
|
- Working gauges and `inject_traceparent` for WS-B/C wiring and WS-F's docs.
|
|
- Signal to WS-T and WS-F: with E, B, C, D merged the VM test should pass; F2 then promotes it. E also unlocks adding the `peer.api.auth_ok = 0` check from T4 if wanted.
|
|
|
|
## Log
|
|
|
|
- 2026-10-19: Split from the single phase plan 2026-10-19. Note added that "peer remove" in the collector check means removal from config plus SIGHUP (D1).
|
|
- 2026-10-19 (A0): contract stubs landed. `PeerRegistry::snapshot()` returns `PeerStatus` sorted by name (jiff timestamps, `since: None` until a check), `AppState.peers` is `None` when the feature is off, and `testing::static_registry` renders fixed snapshots for metric tests.
|
|
- 2026-10-19 (E1): observable peer gauges in `src/telemetry/metrics_export.rs`. `PEER_METRICS` holds the contract names and units; `peer_observations(&[PeerStatus])` maps the registry snapshot (both checks, `auth_ok` separate, latency in seconds, `since` as unix seconds, no URL or version attributes). `spawn` registers them only when `state.peers` is `Some`; the callback reads `PeerRegistry::snapshot()` on every collection, so the exporter never polls a peer and a peer removed by config + SIGHUP leaves the next export. No new background task, no new dependency, no feature flag. One interpretation recorded: disabled peers are omitted from the gauges (the contract is silent; a `0` would read as "down" and page for a peer the operator switched off deliberately — no name or attribute changed).
|
|
- 2026-10-19 (E1 tests): contract names/units, mapping per check, `auth_ok` for `ok`/`rejected`/`unknown`, milliseconds-to-seconds conversion, unix `since`, absence of URL/version attributes, no point for a missing latency or `since`, disabled peer omitted, series vanish after the snapshot shrinks, and collection makes no `FakeTransport` call. Two SDK-level tests drive the real `spawn` through a `PeriodicReader` with a capturing `PushMetricExporter` (both public and already in the tree: no feature such as `experimental_metrics_custom_reader`, no new dependency): one asserts the exported points and that a removed peer's series disappear, the other that a peerless state exports no `peer.*` series.
|
|
- 2026-10-19 (E2): `telemetry::peer_request_span(peer)` creates `peer.request` with the single attribute `peer.name`; `telemetry::inject_traceparent()` returns the W3C header for the current span (ids and flags from the span context) and `None` without a valid OpenTelemetry context, so a peer-less or OTel-disabled instance is unchanged. Tests: span name and field, header shape with ids equal to the span context, `None` outside a span and without the OTel layer, and propagation end to end through `PeerRegistry::get` + `FakeTransport` (header present, no `Authorization`, the outbound span is the parent). `ProxyOpts.traceparent` stays `None` until WS-C/WS-D build their requests inside that span; see the Handoff for the exact call pattern.
|
|
- 2026-10-19 (Handoff): E1 and E2 are implemented and tested against `registry_with`, `static_registry` and `FakeTransport`; no `tests/` file was added because this workstream owns no `tests/` path and the shared fakes are `pub`, so the unit tests use them directly.
|
|
|
|
Wiring required outside E's files:
|
|
|
|
- **B (`serve()`)**: `spawn_exporters(state)` registers the peer gauges at call time, so `state.peers` must be assigned *before* `telemetry::spawn_exporters(state.clone())`. Assigning it only just before `server::build_router` leaves the peer metrics empty until a restart. SIGHUP reload must keep mutating the same `Arc<PeerRegistry>` (`PeerRegistry::reload`); swapping `AppState.peers` would leave the gauges reading the old registry. E cannot enforce either from its own files.
|
|
- **C/D**: wrap the proxy call in the outbound span and inject inside it, e.g. `let span = telemetry::peer_request_span(name);` then inside the span (or via `.instrument(span)`) build `ProxyOpts { traceparent: telemetry::inject_traceparent(), accept_sse }` and call `PeerRegistry::get`. Nothing else is needed; the proxy layer already copies `traceparent` into the peer request.
|
|
|
|
Verification commands (not run in this worktree; no nix builds, and the collector check needs a live instance):
|
|
|
|
- Collector: `devenv up` (OTLP on `127.0.0.1:4317`, debug exporter) with `[otel] enabled = true`; `dev/otelcol.yaml` needs no change because the debug exporter prints every metric. Expect `peer.up` with `peer.check=server|api`, `peer.api.auth_ok`, `peer.check.duration_seconds` and `peer.last_change_timestamp_seconds`; remove the `[[peers.peer]]` entry, send SIGHUP, and the series stop after one `otel.metrics_interval_secs`.
|
|
- VM test (WS-T): `nix build .#tests.x86_64-linux.peers -L`; once promoted in F2, `nix build .#checks.x86_64-linux.peers -L`.
|
|
- Docs (F/orchestrator, since E owns no docs): the README metrics paragraph and the ARCHITECTURE.md metric table need the four `peer.*` gauges, the `peer.name`/`peer.check` attributes, and the note that they are registered only with `[peers] enabled`. `docs/consolidator.md` should state that removing a peer from config and sending SIGHUP stops its series (no history is kept).
|