5.3 KiB
5.3 KiB
WS-G — OpenTelemetry export
| Field | Value |
|---|---|
| Status | Done |
| Owner | — |
| Phase | 2 (parallel) |
| Depends on | P0 to start (fakes, spike findings). A, B, C to finish. |
| Blocks | P3 |
| Owned files | src/telemetry/mod.rs, src/telemetry/metrics_export.rs, src/telemetry/logs_export.rs, dev/otelcol.yaml |
Goal
Export traces, metrics and logs over a single OTLP exporter config. This is the only push path in the system. Metrics come from a poll loop, and logs fan out from the journal and Docker follow streams.
Tasks
Tick these in TODO.md.
- G1
telemetry::init(&Config) -> TelemetryGuard:- Build tracer, meter and logger providers from one exporter config:
otel.endpoint,otel.protocol(grpcorhttp) andotel.headers. - Resource attributes:
service.name,service.version,host.name. - When
otel.enabled = false, return a no-op guard.
- Build tracer, meter and logger providers from one exporter config:
- G2 Tracing subscriber:
fmtto stderr (journald captures it under the unit) plustracing-opentelemetrywhen enabled. Do not route the app's own logs into the OTel logs pipeline. That pipeline is for host logs. - G3 Metrics (review G11):
- A background task runs every
metrics_interval_secs, callslist_units,list_timersandps, and stores the result in anArcSwaporRwLocksnapshot. - Observable gauges read the latest snapshot in their callbacks, so units or containers that disappear stop being reported.
- A background task runs every
- G4 Metric set from ARCHITECTURE.md:
systemd.unit.active,systemd.unit.failed,systemd.unit.restarts,systemd.timer.next_elapse_secondsdocker.container.running,docker.container.healthy,docker.container.uptime_seconds,docker.container.restarts- Attribute names:
systemd.unit,systemd.sub_state,systemd.load_state;container.name,container.id,container.image.name(semconv). - Scope: apply
otel.metrics.units/otel.metrics.containersglobs. With an empty unit list, use the*.service+*.timerdefault (DEC-7).
- G5 Logs (
otel.export_logs = true, review G4):- Journal: one
follow()per configured unit, or one unfiltered follow whenotel.logs.journal_unitsis empty. Map priority 0–7 to OTel severity (emerg/alert/crit to FATAL, err to ERROR, warning to WARN, notice/info to INFO, debug to DEBUG). - Docker: a reconciler subscribes to Docker
events(start,die,destroy). It starts and stops per-containerlogs(follow)tasks for allowlisted containers and does an initial list at startup. stderr maps to WARN and stdout to INFO. - Every task restarts with exponential backoff on error. If Docker is unavailable, log once and retry slowly.
- Attributes:
systemd.unitorcontainer.name, plushost.namefrom the resource.
- Journal: one
- G6 On shutdown: cancel all tasks via
state.shutdown, thenforce_flushandshutdowneach provider, with a bounded timeout. - G7
dev/otelcol.yamlwith an OTLP receiver (grpc 4317, http 4318) and adebugexporter atverbosity: detailed. Add a devenvprocesses.otelcolentry, coordinating thedevenv.nixedit with the P0 owner.
Done when
- Verification #5 passes. With
[otel] enabled = true, the collector shows the gauges every interval and log records with correct severity and attributes. logger -p user.err testappears as an ERROR record.- Starting a new container shows its logs without restarting the service.
- SIGTERM flushes pending records before exit.
Notes
- opentelemetry-rust marks metrics and logs Stable and traces Beta. Pin versions together, because the
opentelemetry*crates must match each other. - Cardinality: one series per unit or container per attribute set. With the DEC-7 default, expect a few hundred series on a typical host.
Log
- 2026-10-15: Card created.
- 2026-10-15: Review fixes F1-F8. Restart polling uses buffer_unordered(8) with a 15s budget; failed sections keep the previous snapshot; series carry identity attributes only; timer metric renamed
systemd.timer.next_elapse_timestamp_seconds; units added; observations computed once per poll. New containers start from StartedAt, docker lines deduped on reconnect, own unit/identifier excluded from the journal follow (by name; cursor not used).TelemetryGuard::shutdown().awaitadded; Drop is best-effort. - P3 notes: (1) call
guard.shutdown().awaitafter awaiting the exporter handles. (2) devenv: addprocesses.otelcol.exec = "otelcol-contrib --config dev/otelcol.yaml"to devenv.nix (not done here, P0 owner). (3) Cargo.toml needsreqwest-blocking-clienton opentelemetry-otlp for http/protobuf; the exporter-build test is strict and fails without it. (4) RUST_LOG/EnvFilter deferred. - 2026-10-15: Implemented G1-G7 on
ws-g/otel. Notes: no glob crate, so a small*/?matcher lives intelemetry/mod.rs.DockerSourcehas no events stream, so the log reconciler pollsps(false)every 5s (30s while Docker is unavailable) instead of subscribing toevents. NoEnvFilter(feature not enabled in Cargo.toml): level is fixed at INFO.systemd.timer.next_elapse_secondsexports absolute unix seconds.systemd.unit.restartscosts oneunit_statusper in-scope loaded service per interval. HTTP endpoints get/v1/<signal>appended unless already present. Live otelcol smoke test not run (no collector binary on PATH).