3.7 KiB
3.7 KiB
WS-G — OpenTelemetry export
| Field | Value |
|---|---|
| Status | Backlog |
| Owner | — |
| Phase | 2 (parallel) |
| Depends on | P0 to start (fakes, spike findings). A, B, C to finish. |
| Blocks | P3 |
| Owned files | src/telemetry/mod.rs, src/telemetry/metrics_export.rs, src/telemetry/logs_export.rs, dev/otelcol.yaml |
Goal
Export traces, metrics and logs over a single OTLP exporter config. This is the only push path in the system. Metrics come from a poll loop, and logs fan out from the journal and Docker follow streams.
Tasks
Tick these in TODO.md.
- G1
telemetry::init(&Config) -> TelemetryGuard:- Build tracer, meter and logger providers from one exporter config:
otel.endpoint,otel.protocol(grpcorhttp) andotel.headers. - Resource attributes:
service.name,service.version,host.name. - When
otel.enabled = false, return a no-op guard.
- Build tracer, meter and logger providers from one exporter config:
- G2 Tracing subscriber:
fmtto stderr (journald captures it under the unit) plustracing-opentelemetrywhen enabled. Do not route the app's own logs into the OTel logs pipeline. That pipeline is for host logs. - G3 Metrics (review G11):
- A background task runs every
metrics_interval_secs, callslist_units,list_timersandps, and stores the result in anArcSwaporRwLocksnapshot. - Observable gauges read the latest snapshot in their callbacks, so units or containers that disappear stop being reported.
- A background task runs every
- G4 Metric set from
plan.md:systemd.unit.active,systemd.unit.failed,systemd.unit.restarts,systemd.timer.next_elapse_secondsdocker.container.running,docker.container.healthy,docker.container.uptime_seconds,docker.container.restarts- Attribute names:
systemd.unit,systemd.sub_state,systemd.load_state;container.name,container.id,container.image.name(semconv). - Scope: apply
otel.metrics.units/otel.metrics.containersglobs. With an empty unit list, use the*.service+*.timerdefault (DEC-7).
- G5 Logs (
otel.export_logs = true, review G4):- Journal: one
follow()per configured unit, or one unfiltered follow whenotel.logs.journal_unitsis empty. Map priority 0–7 to OTel severity (emerg/alert/crit to FATAL, err to ERROR, warning to WARN, notice/info to INFO, debug to DEBUG). - Docker: a reconciler subscribes to Docker
events(start,die,destroy). It starts and stops per-containerlogs(follow)tasks for allowlisted containers and does an initial list at startup. stderr maps to WARN and stdout to INFO. - Every task restarts with exponential backoff on error. If Docker is unavailable, log once and retry slowly.
- Attributes:
systemd.unitorcontainer.name, plushost.namefrom the resource.
- Journal: one
- G6 On shutdown: cancel all tasks via
state.shutdown, thenforce_flushandshutdowneach provider, with a bounded timeout. - G7
dev/otelcol.yamlwith an OTLP receiver (grpc 4317, http 4318) and adebugexporter atverbosity: detailed. Add a devenvprocesses.otelcolentry, coordinating thedevenv.nixedit with the P0 owner.
Done when
- Verification #5 passes. With
[otel] enabled = true, the collector shows the gauges every interval and log records with correct severity and attributes. logger -p user.err testappears as an ERROR record.- Starting a new container shows its logs without restarting the service.
- SIGTERM flushes pending records before exit.
Notes
- opentelemetry-rust marks metrics and logs Stable and traces Beta. Pin versions together, because the
opentelemetry*crates must match each other. - Cardinality: one series per unit or container per attribute set. With the DEC-7 default, expect a few hundred series on a typical host.
Log
- 2026-10-15: Card created.