Files
2026-10-19 14:14:29 -04:00

5.3 KiB
Raw Permalink Blame History

WS-G — OpenTelemetry export

Field Value
Status Done
Owner —
Phase 2 (parallel)
Depends on P0 to start (fakes, spike findings). A, B, C to finish.
Blocks P3
Owned files src/telemetry/mod.rs, src/telemetry/metrics_export.rs, src/telemetry/logs_export.rs, dev/otelcol.yaml

Goal

Export traces, metrics and logs over a single OTLP exporter config. This is the only push path in the system. Metrics come from a poll loop, and logs fan out from the journal and Docker follow streams.

Tasks

Tick these in TODO.md.

  • G1 telemetry::init(&Config) -> TelemetryGuard:
    • Build tracer, meter and logger providers from one exporter config: otel.endpoint, otel.protocol (grpc or http) and otel.headers.
    • Resource attributes: service.name, service.version, host.name.
    • When otel.enabled = false, return a no-op guard.
  • G2 Tracing subscriber: fmt to stderr (journald captures it under the unit) plus tracing-opentelemetry when enabled. Do not route the app's own logs into the OTel logs pipeline. That pipeline is for host logs.
  • G3 Metrics (review G11):
    • A background task runs every metrics_interval_secs, calls list_units, list_timers and ps, and stores the result in an ArcSwap or RwLock snapshot.
    • Observable gauges read the latest snapshot in their callbacks, so units or containers that disappear stop being reported.
  • G4 Metric set from ARCHITECTURE.md:
    • systemd.unit.active, systemd.unit.failed, systemd.unit.restarts, systemd.timer.next_elapse_seconds
    • docker.container.running, docker.container.healthy, docker.container.uptime_seconds, docker.container.restarts
    • Attribute names: systemd.unit, systemd.sub_state, systemd.load_state; container.name, container.id, container.image.name (semconv).
    • Scope: apply otel.metrics.units / otel.metrics.containers globs. With an empty unit list, use the *.service + *.timer default (DEC-7).
  • G5 Logs (otel.export_logs = true, review G4):
    • Journal: one follow() per configured unit, or one unfiltered follow when otel.logs.journal_units is empty. Map priority 0–7 to OTel severity (emerg/alert/crit to FATAL, err to ERROR, warning to WARN, notice/info to INFO, debug to DEBUG).
    • Docker: a reconciler subscribes to Docker events (start, die, destroy). It starts and stops per-container logs(follow) tasks for allowlisted containers and does an initial list at startup. stderr maps to WARN and stdout to INFO.
    • Every task restarts with exponential backoff on error. If Docker is unavailable, log once and retry slowly.
    • Attributes: systemd.unit or container.name, plus host.name from the resource.
  • G6 On shutdown: cancel all tasks via state.shutdown, then force_flush and shutdown each provider, with a bounded timeout.
  • G7 dev/otelcol.yaml with an OTLP receiver (grpc 4317, http 4318) and a debug exporter at verbosity: detailed. Add a devenv processes.otelcol entry, coordinating the devenv.nix edit with the P0 owner.

Done when

  • Verification #5 passes. With [otel] enabled = true, the collector shows the gauges every interval and log records with correct severity and attributes.
  • logger -p user.err test appears as an ERROR record.
  • Starting a new container shows its logs without restarting the service.
  • SIGTERM flushes pending records before exit.

Notes

  • opentelemetry-rust marks metrics and logs Stable and traces Beta. Pin versions together, because the opentelemetry* crates must match each other.
  • Cardinality: one series per unit or container per attribute set. With the DEC-7 default, expect a few hundred series on a typical host.

Log

  • 2026-10-15: Card created.
  • 2026-10-15: Review fixes F1-F8. Restart polling uses buffer_unordered(8) with a 15s budget; failed sections keep the previous snapshot; series carry identity attributes only; timer metric renamed systemd.timer.next_elapse_timestamp_seconds; units added; observations computed once per poll. New containers start from StartedAt, docker lines deduped on reconnect, own unit/identifier excluded from the journal follow (by name; cursor not used). TelemetryGuard::shutdown().await added; Drop is best-effort.
  • P3 notes: (1) call guard.shutdown().await after awaiting the exporter handles. (2) devenv: add processes.otelcol.exec = "otelcol-contrib --config dev/otelcol.yaml" to devenv.nix (not done here, P0 owner). (3) Cargo.toml needs reqwest-blocking-client on opentelemetry-otlp for http/protobuf; the exporter-build test is strict and fails without it. (4) RUST_LOG/EnvFilter deferred.
  • 2026-10-15: Implemented G1-G7 on ws-g/otel. Notes: no glob crate, so a small */? matcher lives in telemetry/mod.rs. DockerSource has no events stream, so the log reconciler polls ps(false) every 5s (30s while Docker is unavailable) instead of subscribing to events. No EnvFilter (feature not enabled in Cargo.toml): level is fixed at INFO. systemd.timer.next_elapse_seconds exports absolute unix seconds. systemd.unit.restarts costs one unit_status per in-scope loaded service per interval. HTTP endpoints get /v1/<signal> appended unless already present. Live otelcol smoke test not run (no collector binary on PATH).