# WS-G — OpenTelemetry export | Field | Value | |---|---| | Status | Review | | Owner | — | | Phase | 2 (parallel) | | Depends on | P0 to start (fakes, spike findings). A, B, C to finish. | | Blocks | P3 | | Owned files | `src/telemetry/mod.rs`, `src/telemetry/metrics_export.rs`, `src/telemetry/logs_export.rs`, `dev/otelcol.yaml` | ## Goal Export traces, metrics and logs over a single OTLP exporter config. This is the only push path in the system. Metrics come from a poll loop, and logs fan out from the journal and Docker follow streams. ## Tasks Tick these in [`TODO.md`](../../TODO.md). - **G1** `telemetry::init(&Config) -> TelemetryGuard`: - Build tracer, meter and logger providers from one exporter config: `otel.endpoint`, `otel.protocol` (`grpc` or `http`) and `otel.headers`. - Resource attributes: `service.name`, `service.version`, `host.name`. - When `otel.enabled = false`, return a no-op guard. - **G2** Tracing subscriber: `fmt` to stderr (journald captures it under the unit) plus `tracing-opentelemetry` when enabled. Do **not** route the app's own logs into the OTel logs pipeline. That pipeline is for host logs. - **G3** Metrics (review G11): - A background task runs every `metrics_interval_secs`, calls `list_units`, `list_timers` and `ps`, and stores the result in an `ArcSwap` or `RwLock` snapshot. - Observable gauges read the latest snapshot in their callbacks, so units or containers that disappear stop being reported. - **G4** Metric set from `plan.md`: - `systemd.unit.active`, `systemd.unit.failed`, `systemd.unit.restarts`, `systemd.timer.next_elapse_seconds` - `docker.container.running`, `docker.container.healthy`, `docker.container.uptime_seconds`, `docker.container.restarts` - Attribute names: `systemd.unit`, `systemd.sub_state`, `systemd.load_state`; `container.name`, `container.id`, `container.image.name` (semconv). - Scope: apply `otel.metrics.units` / `otel.metrics.containers` globs. With an empty unit list, use the `*.service` + `*.timer` default (DEC-7). - **G5** Logs (`otel.export_logs = true`, review G4): - Journal: one `follow()` per configured unit, or one unfiltered follow when `otel.logs.journal_units` is empty. Map priority 0–7 to OTel severity (emerg/alert/crit to FATAL, err to ERROR, warning to WARN, notice/info to INFO, debug to DEBUG). - Docker: a reconciler subscribes to Docker `events` (`start`, `die`, `destroy`). It starts and stops per-container `logs(follow)` tasks for allowlisted containers and does an initial list at startup. stderr maps to WARN and stdout to INFO. - Every task restarts with exponential backoff on error. If Docker is unavailable, log once and retry slowly. - Attributes: `systemd.unit` or `container.name`, plus `host.name` from the resource. - **G6** On shutdown: cancel all tasks via `state.shutdown`, then `force_flush` and `shutdown` each provider, with a bounded timeout. - **G7** `dev/otelcol.yaml` with an OTLP receiver (grpc 4317, http 4318) and a `debug` exporter at `verbosity: detailed`. Add a devenv `processes.otelcol` entry, coordinating the `devenv.nix` edit with the P0 owner. ## Done when - Verification #5 passes. With `[otel] enabled = true`, the collector shows the gauges every interval and log records with correct severity and attributes. - `logger -p user.err test` appears as an ERROR record. - Starting a new container shows its logs without restarting the service. - SIGTERM flushes pending records before exit. ## Notes - opentelemetry-rust marks metrics and logs Stable and traces Beta. Pin versions together, because the `opentelemetry*` crates must match each other. - Cardinality: one series per unit or container per attribute set. With the DEC-7 default, expect a few hundred series on a typical host. ## Log - 2026-10-15: Card created. - 2026-10-15: Review fixes F1-F8. Restart polling uses buffer_unordered(8) with a 15s budget; failed sections keep the previous snapshot; series carry identity attributes only; timer metric renamed `systemd.timer.next_elapse_timestamp_seconds`; units added; observations computed once per poll. New containers start from StartedAt, docker lines deduped on reconnect, own unit/identifier excluded from the journal follow (by name; cursor not used). `TelemetryGuard::shutdown().await` added; Drop is best-effort. - P3 notes: (1) call `guard.shutdown().await` after awaiting the exporter handles. (2) devenv: add `processes.otelcol.exec = "otelcol-contrib --config dev/otelcol.yaml"` to devenv.nix (not done here, P0 owner). (3) Cargo.toml needs `reqwest-blocking-client` on opentelemetry-otlp for http/protobuf; the exporter-build test is strict and fails without it. (4) RUST_LOG/EnvFilter deferred. - 2026-10-15: Implemented G1-G7 on `ws-g/otel`. Notes: no glob crate, so a small `*`/`?` matcher lives in `telemetry/mod.rs`. `DockerSource` has no events stream, so the log reconciler polls `ps(false)` every 5s (30s while Docker is unavailable) instead of subscribing to `events`. No `EnvFilter` (feature not enabled in Cargo.toml): level is fixed at INFO. `systemd.timer.next_elapse_seconds` exports absolute unix seconds. `systemd.unit.restarts` costs one `unit_status` per in-scope loaded service per interval. HTTP endpoints get `/v1/` appended unless already present. Live otelcol smoke test not run (no collector binary on PATH).