Files
skyg-linux-tracker/plan/workstreams/WS-G-otel.md
T
2026-10-15 18:43:21 -04:00

59 lines
5.2 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# WS-G — OpenTelemetry export
| Field | Value |
|---|---|
| Status | Review |
| Owner | — |
| Phase | 2 (parallel) |
| Depends on | P0 to start (fakes, spike findings). A, B, C to finish. |
| Blocks | P3 |
| Owned files | `src/telemetry/mod.rs`, `src/telemetry/metrics_export.rs`, `src/telemetry/logs_export.rs`, `dev/otelcol.yaml` |
## Goal
Export traces, metrics and logs over a single OTLP exporter config. This is the only push path in the system. Metrics come from a poll loop, and logs fan out from the journal and Docker follow streams.
## Tasks
Tick these in [`TODO.md`](../../TODO.md).
- **G1** `telemetry::init(&Config) -> TelemetryGuard`:
- Build tracer, meter and logger providers from one exporter config: `otel.endpoint`, `otel.protocol` (`grpc` or `http`) and `otel.headers`.
- Resource attributes: `service.name`, `service.version`, `host.name`.
- When `otel.enabled = false`, return a no-op guard.
- **G2** Tracing subscriber: `fmt` to stderr (journald captures it under the unit) plus `tracing-opentelemetry` when enabled. Do **not** route the app's own logs into the OTel logs pipeline. That pipeline is for host logs.
- **G3** Metrics (review G11):
- A background task runs every `metrics_interval_secs`, calls `list_units`, `list_timers` and `ps`, and stores the result in an `ArcSwap` or `RwLock` snapshot.
- Observable gauges read the latest snapshot in their callbacks, so units or containers that disappear stop being reported.
- **G4** Metric set from `plan.md`:
- `systemd.unit.active`, `systemd.unit.failed`, `systemd.unit.restarts`, `systemd.timer.next_elapse_seconds`
- `docker.container.running`, `docker.container.healthy`, `docker.container.uptime_seconds`, `docker.container.restarts`
- Attribute names: `systemd.unit`, `systemd.sub_state`, `systemd.load_state`; `container.name`, `container.id`, `container.image.name` (semconv).
- Scope: apply `otel.metrics.units` / `otel.metrics.containers` globs. With an empty unit list, use the `*.service` + `*.timer` default (DEC-7).
- **G5** Logs (`otel.export_logs = true`, review G4):
- Journal: one `follow()` per configured unit, or one unfiltered follow when `otel.logs.journal_units` is empty. Map priority 0–7 to OTel severity (emerg/alert/crit to FATAL, err to ERROR, warning to WARN, notice/info to INFO, debug to DEBUG).
- Docker: a reconciler subscribes to Docker `events` (`start`, `die`, `destroy`). It starts and stops per-container `logs(follow)` tasks for allowlisted containers and does an initial list at startup. stderr maps to WARN and stdout to INFO.
- Every task restarts with exponential backoff on error. If Docker is unavailable, log once and retry slowly.
- Attributes: `systemd.unit` or `container.name`, plus `host.name` from the resource.
- **G6** On shutdown: cancel all tasks via `state.shutdown`, then `force_flush` and `shutdown` each provider, with a bounded timeout.
- **G7** `dev/otelcol.yaml` with an OTLP receiver (grpc 4317, http 4318) and a `debug` exporter at `verbosity: detailed`. Add a devenv `processes.otelcol` entry, coordinating the `devenv.nix` edit with the P0 owner.
## Done when
- Verification #5 passes. With `[otel] enabled = true`, the collector shows the gauges every interval and log records with correct severity and attributes.
- `logger -p user.err test` appears as an ERROR record.
- Starting a new container shows its logs without restarting the service.
- SIGTERM flushes pending records before exit.
## Notes
- opentelemetry-rust marks metrics and logs Stable and traces Beta. Pin versions together, because the `opentelemetry*` crates must match each other.
- Cardinality: one series per unit or container per attribute set. With the DEC-7 default, expect a few hundred series on a typical host.
## Log
- 2026-10-15: Card created.
- 2026-10-15: Review fixes F1-F8. Restart polling uses buffer_unordered(8) with a 15s budget; failed sections keep the previous snapshot; series carry identity attributes only; timer metric renamed `systemd.timer.next_elapse_timestamp_seconds`; units added; observations computed once per poll. New containers start from StartedAt, docker lines deduped on reconnect, own unit/identifier excluded from the journal follow (by name; cursor not used). `TelemetryGuard::shutdown().await` added; Drop is best-effort.
- P3 notes: (1) call `guard.shutdown().await` after awaiting the exporter handles. (2) devenv: add `processes.otelcol.exec = "otelcol-contrib --config dev/otelcol.yaml"` to devenv.nix (not done here, P0 owner). (3) Cargo.toml needs `reqwest-blocking-client` on opentelemetry-otlp for http/protobuf; the exporter-build test is strict and fails without it. (4) RUST_LOG/EnvFilter deferred.
- 2026-10-15: Implemented G1-G7 on `ws-g/otel`. Notes: no glob crate, so a small `*`/`?` matcher lives in `telemetry/mod.rs`. `DockerSource` has no events stream, so the log reconciler polls `ps(false)` every 5s (30s while Docker is unavailable) instead of subscribing to `events`. No `EnvFilter` (feature not enabled in Cargo.toml): level is fixed at INFO. `systemd.timer.next_elapse_seconds` exports absolute unix seconds. `systemd.unit.restarts` costs one `unit_status` per in-scope loaded service per interval. HTTP endpoints get `/v1/<signal>` appended unless already present. Live otelcol smoke test not run (no collector binary on PATH).