59 lines
5.3 KiB
Markdown
59 lines
5.3 KiB
Markdown
# WS-G — OpenTelemetry export
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Status | Done |
|
||
| Owner | — |
|
||
| Phase | 2 (parallel) |
|
||
| Depends on | P0 to start (fakes, spike findings). A, B, C to finish. |
|
||
| Blocks | P3 |
|
||
| Owned files | `src/telemetry/mod.rs`, `src/telemetry/metrics_export.rs`, `src/telemetry/logs_export.rs`, `dev/otelcol.yaml` |
|
||
|
||
## Goal
|
||
|
||
Export traces, metrics and logs over a single OTLP exporter config. This is the only push path in the system. Metrics come from a poll loop, and logs fan out from the journal and Docker follow streams.
|
||
|
||
## Tasks
|
||
|
||
Tick these in [`TODO.md`](../../TODO.md).
|
||
|
||
- **G1** `telemetry::init(&Config) -> TelemetryGuard`:
|
||
- Build tracer, meter and logger providers from one exporter config: `otel.endpoint`, `otel.protocol` (`grpc` or `http`) and `otel.headers`.
|
||
- Resource attributes: `service.name`, `service.version`, `host.name`.
|
||
- When `otel.enabled = false`, return a no-op guard.
|
||
- **G2** Tracing subscriber: `fmt` to stderr (journald captures it under the unit) plus `tracing-opentelemetry` when enabled. Do **not** route the app's own logs into the OTel logs pipeline. That pipeline is for host logs.
|
||
- **G3** Metrics (review G11):
|
||
- A background task runs every `metrics_interval_secs`, calls `list_units`, `list_timers` and `ps`, and stores the result in an `ArcSwap` or `RwLock` snapshot.
|
||
- Observable gauges read the latest snapshot in their callbacks, so units or containers that disappear stop being reported.
|
||
- **G4** Metric set from [ARCHITECTURE.md](../../ARCHITECTURE.md#opentelemetry-telemetry):
|
||
- `systemd.unit.active`, `systemd.unit.failed`, `systemd.unit.restarts`, `systemd.timer.next_elapse_seconds`
|
||
- `docker.container.running`, `docker.container.healthy`, `docker.container.uptime_seconds`, `docker.container.restarts`
|
||
- Attribute names: `systemd.unit`, `systemd.sub_state`, `systemd.load_state`; `container.name`, `container.id`, `container.image.name` (semconv).
|
||
- Scope: apply `otel.metrics.units` / `otel.metrics.containers` globs. With an empty unit list, use the `*.service` + `*.timer` default (DEC-7).
|
||
- **G5** Logs (`otel.export_logs = true`, review G4):
|
||
- Journal: one `follow()` per configured unit, or one unfiltered follow when `otel.logs.journal_units` is empty. Map priority 0–7 to OTel severity (emerg/alert/crit to FATAL, err to ERROR, warning to WARN, notice/info to INFO, debug to DEBUG).
|
||
- Docker: a reconciler subscribes to Docker `events` (`start`, `die`, `destroy`). It starts and stops per-container `logs(follow)` tasks for allowlisted containers and does an initial list at startup. stderr maps to WARN and stdout to INFO.
|
||
- Every task restarts with exponential backoff on error. If Docker is unavailable, log once and retry slowly.
|
||
- Attributes: `systemd.unit` or `container.name`, plus `host.name` from the resource.
|
||
- **G6** On shutdown: cancel all tasks via `state.shutdown`, then `force_flush` and `shutdown` each provider, with a bounded timeout.
|
||
- **G7** `dev/otelcol.yaml` with an OTLP receiver (grpc 4317, http 4318) and a `debug` exporter at `verbosity: detailed`. Add a devenv `processes.otelcol` entry, coordinating the `devenv.nix` edit with the P0 owner.
|
||
|
||
## Done when
|
||
|
||
- Verification #5 passes. With `[otel] enabled = true`, the collector shows the gauges every interval and log records with correct severity and attributes.
|
||
- `logger -p user.err test` appears as an ERROR record.
|
||
- Starting a new container shows its logs without restarting the service.
|
||
- SIGTERM flushes pending records before exit.
|
||
|
||
## Notes
|
||
|
||
- opentelemetry-rust marks metrics and logs Stable and traces Beta. Pin versions together, because the `opentelemetry*` crates must match each other.
|
||
- Cardinality: one series per unit or container per attribute set. With the DEC-7 default, expect a few hundred series on a typical host.
|
||
|
||
## Log
|
||
|
||
- 2026-10-15: Card created.
|
||
- 2026-10-15: Review fixes F1-F8. Restart polling uses buffer_unordered(8) with a 15s budget; failed sections keep the previous snapshot; series carry identity attributes only; timer metric renamed `systemd.timer.next_elapse_timestamp_seconds`; units added; observations computed once per poll. New containers start from StartedAt, docker lines deduped on reconnect, own unit/identifier excluded from the journal follow (by name; cursor not used). `TelemetryGuard::shutdown().await` added; Drop is best-effort.
|
||
- P3 notes: (1) call `guard.shutdown().await` after awaiting the exporter handles. (2) devenv: add `processes.otelcol.exec = "otelcol-contrib --config dev/otelcol.yaml"` to devenv.nix (not done here, P0 owner). (3) Cargo.toml needs `reqwest-blocking-client` on opentelemetry-otlp for http/protobuf; the exporter-build test is strict and fails without it. (4) RUST_LOG/EnvFilter deferred.
|
||
- 2026-10-15: Implemented G1-G7 on `ws-g/otel`. Notes: no glob crate, so a small `*`/`?` matcher lives in `telemetry/mod.rs`. `DockerSource` has no events stream, so the log reconciler polls `ps(false)` every 5s (30s while Docker is unavailable) instead of subscribing to `events`. No `EnvFilter` (feature not enabled in Cargo.toml): level is fixed at INFO. `systemd.timer.next_elapse_seconds` exports absolute unix seconds. `systemd.unit.restarts` costs one `unit_status` per in-scope loaded service per interval. HTTP endpoints get `/v1/<signal>` appended unless already present. Live otelcol smoke test not run (no collector binary on PATH).
|