Files
skyg-linux-tracker/plan.md
T
2026-10-15 17:07:03 -04:00

16 KiB

skyg-linux-tracker — Design Doc

A read-only Linux system monitor in Rust, distributed as a single binary. Exposes systemd state, journald logs, and Docker container state/logs over three surfaces that share one core: a REST API, an MCP server (Streamable HTTP), and OpenTelemetry export. All data is fetched live from the source on every request — no metric/log collection or buffering, except a small SQLite database that stores API access tokens.

Confirmed requirements

  • Single Rust binary, read-only against the host.
  • MCP over Streamable HTTP with auth (mounts on the same axum server as the REST API).
  • Config via TOML file + CLI flags + env vars (precedence: CLI > env > file > defaults).
  • OTel: systemd unit / container status exported as metrics (gauges); journald / docker logs exported as OTel logs. Traces for the API itself are included since they come nearly free.
  • MCP and REST expose the same data; MCP adds agent-facing context (tool descriptions, instructions, curated resources/prompts).
  • State: /var/lib/skyg-linux-tracker/db.sqlite (SQLite via sqlx), access tokens only.

Tech choices

Area Crate Why
HTTP axum + tokio Industry standard; rmcp's HTTP transport is a Tower service that mounts directly onto an axum router
MCP rmcp (official modelcontextprotocol/rust-sdk), features server, macros, schemars, transport-streamable-http-server Official SDK; StreamableHttpService mounts at /mcp on the same axum router; tools via #[tool] macros
systemd zbus_systemd (feature systemd1, tokio executor) Pure-Rust generated bindings for org.freedesktop.systemd1 — ListUnits, ListTimers, unit properties. No libsystemd C dependency
journald systemd crate, journal feature (sd-journal FFI) Only maintained option supporting live tail (seek_tail + next + wait). Links libsystemd at build time (present on any systemd host; the devenv shell provides it)
Docker bollard (rustls, no openssl) Async Docker/Podman API over the unix socket; list_containers, inspect_container (health), logs(follow=true) streams
OTel opentelemetry, opentelemetry_sdk, opentelemetry-otlp (metrics + logs both Stable in current opentelemetry-rust; traces Beta) One OTLP exporter config (endpoint, protocol, headers) drives all three signals
Auth DB sqlx + sqlite (bundled) Async, compile-time-checked queries, pure-Rust bundled SQLite — keeps the binary self-contained
Config figment (TOML + env) + clap Layered config with clear precedence

Repo layout

linux-system-monitor/
├── Cargo.toml
├── devenv.nix                 # + languages.rust, pkg-config/openssl/libsystemd build deps
├── devenv.yaml
├── config.example.toml
├── systemd/
│   └── skyg-linux-tracker.service   # example unit file (hardened, read-only)
└── src/
    ├── main.rs                # entry: config load → telemetry init → DB → server
    ├── config.rs              # figment layering: defaults < TOML < env(SKYG_) < CLI
    ├── error.rs               # thiserror AppError → axum + MCP error mappings
    ├── auth/
    │   ├── mod.rs             # bearer middleware + token verify (sha256 hash lookup)
    │   └── tokens.rs          # token CRUD against SQLite (`slinuxm token create/revoke/list` CLI)
    ├── db.rs                  # sqlx pool, embedded migrations, open-or-create
    ├── sources/               # === the core: one trait per domain, live reads only ===
    │   ├── mod.rs
    │   ├── systemd.rs         # list_units(), list_timers(), unit_status(name)
    │   ├── journal.rs         # query(filter) snapshot, follow(filter) → mpsc stream
    │   └── docker.rs          # ps(), container_status(name), logs(name, follow)
    ├── api/                   # REST layer (thin): DTOs + handlers calling sources/
    │   ├── mod.rs             # router assembly, auth middleware layer
    │   ├── systemd.rs         # GET /api/v1/systemd/{units,timers,units/:name}
    │   ├── journal.rs         # GET /api/v1/journal (snapshot), GET .../journal/follow (SSE)
    │   └── docker.rs          # GET /api/v1/docker/{ps,containers/:name,containers/:name/logs}
    ├── mcp/
    │   ├── mod.rs             # TrackerService: #[tool_router] tools wrapping sources/
    │   └── prompts.rs         # agent-facing prompts ("diagnose failing units", etc.)
    └── telemetry/
        ├── mod.rs             # OTel providers init (traces/metrics/logs), shutdown
        ├── metrics_export.rs  # background loop → observable gauges
        └── logs_export.rs     # journal/docker follow → OTel log records fan-out

The key structural decision: sources/ is the only place that talks to D-Bus/journald/Docker. api/, mcp/, and telemetry/ are three thin projections over it, which is what makes "MCP and API expose the same data" true by construction.

Implementation detail

Config (config.rs, config.example.toml)

[server]
bind = "127.0.0.1"        # set 0.0.0.0 only with auth enabled
port = 9109

[auth]
enabled = true            # hard-required when bind != localhost
db_path = "/var/lib/skyg-linux-tracker/db.sqlite"

[otel]
enabled = false
endpoint = "http://localhost:4317"   # OTLP gRPC; http://…:4318 for HTTP/proto
protocol = "grpc"
service_name = "skyg-linux-tracker"  # OTel resource attr service.name
export_unit_metrics = true   # poll loop, interval below
export_logs = false          # journal/docker log fan-out to OTLP
metrics_interval_secs = 30

[otel.logs]
journal_units = []           # empty = all units
docker_containers = []       # empty = all containers

[sources]
docker_socket = "/var/run/docker.sock"
journal_namespace = ""       # default journal

Precedence: built-in defaults < --config /etc/skyg-linux-tracker.toml < SKYG_* env vars < CLI flags. CLI subcommands: serve (default), token create|revoke|list.

Sources (live reads only)

  • systemd (sources/systemd.rs): one shared zbus::Connection to the system bus. list_units() → ManagerProxy::list_units(); list_timers() → list_timers() (next-elapse realtime converted to RFC3339); unit_status(name) → load_unit + properties (ActiveState, SubState, ExecMainStatus, NRestarts, ActiveEnterTimestamp). No caching — every call hits D-Bus.
  • journal (sources/journal.rs): Journal::open(), match_add("_SYSTEMD_UNIT", …), seek_tail(). Snapshot queries seek backward N entries. follow() wraps Journal::wait() in a tokio::task::spawn_blocking bridge (sd-journal is blocking) feeding a tokio::sync::mpsc — consumers (SSE endpoint, MCP tool, OTel log exporter) all read from channels.
  • docker (sources/docker.rs): shared bollard::Docker over the unix socket. ps() → list_containers + inspect_container per container for State.Health, StartedAt (→ uptime), restart count. logs(name, follow) → bollard log stream (stdout, stderr, timestamps, since) as a Stream<Item = LogOutput>.

REST API (api/)

All under /api/v1, all behind the auth middleware when enabled, all read-only (GET only):

Endpoint Source
GET /api/v1/systemd/units?state=failed list_units()
GET /api/v1/systemd/units/:name unit_status()
GET /api/v1/systemd/timers list_timers()
GET /api/v1/journal?unit=X&priority=err&lines=100 journal snapshot
GET /api/v1/journal/follow?unit=X (SSE) journal follow()
GET /api/v1/docker/ps ps()
GET /api/v1/docker/containers/:name container_status()
GET /api/v1/docker/containers/:name/logs?follow=1 (SSE) logs(follow)
GET /healthz unauthenticated liveness

Streaming endpoints use SSE (axum::response::Sse) over the mpsc channels from sources/.

MCP server (mcp/)

rmcp's StreamableHttpService is a Tower service mounted on the same axum router:

let mcp_service = StreamableHttpService::new(
    || Ok(TrackerService::new(app_state.clone())),  // per-session handler, shared state
    LocalSessionManager::default().into(),
    StreamableHttpServerConfig::default().with_json_response(false),
);
let app = Router::new()
    .nest("/api/v1", api_router)
    .nest_service("/mcp", mcp_service)
    .layer(auth_middleware);   // same bearer auth covers both

Tools (each a thin wrapper over sources/, params derive schemars::JsonSchema):

  • list_systemd_units { state? } / get_systemd_unit { name } / list_systemd_timers {}
  • get_journal_logs { unit?, priority?, lines } / tail_journal { unit, seconds } (bounded follow — collects for N seconds, returns; no infinite MCP streams)
  • list_containers {} / get_container { name } / get_container_logs { name, tail }

The "bit more context for the agent" lives in:

  • ServerHandler instructions: how the tools relate ("check list_systemd_units state=failed first, then pull journal logs for that unit").
  • Rich #[tool(description = …)] strings explaining systemd states, health statuses, priorities.
  • Prompts (mcp/prompts.rs): diagnose_unit { name }, container_health_report {} — canned investigation workflows.
  • Resources: systemd://units/failed, docker://containers as subscribable snapshot resources.

OpenTelemetry (telemetry/)

One OTLP exporter config → three signals:

  1. Traces (Beta in opentelemetry-rust): tracing-opentelemetry layer + tower-http TraceLayer — spans per HTTP/MCP request.
  2. Metrics (Stable): metrics_export.rs runs a poll loop every metrics_interval_secs:
    • systemd.unit.active (u64 gauge 0/1, attrs: unit, sub_state, load_state)
    • systemd.unit.failed (u64 0/1), systemd.unit.restarts (u64 counter-ish gauge)
    • systemd.timer.next_elapse_seconds (f64, attr timer)
    • docker.container.running (0/1, attrs container, image), docker.container.healthy (0/1, health_status attr), docker.container.uptime_seconds, docker.container.restarts
    • Configurable unit/container allowlists so cardinality stays bounded.
  3. Logs (Stable): when otel.export_logs = true, logs_export.rs spawns journal follow() + per-container docker logs(follow=true) tasks, converts each entry to an OTel LogRecord (severity from journal priority / docker stderr-vs-stdout, attrs: unit, container, host) and emits through the SDK LoggerProvider.

"Same information via OTel" is satisfied by reusing the identical sources/ functions; the OTel path is the only one that does periodic push (metrics poll) and continuous fan-out (logs), because OTLP is a push model — everything else stays pull/live.

Auth & database (auth/, db.rs)

  • SQLite at /var/lib/skyg-linux-tracker/db.sqlite (dir created 0700 on first run; sqlx embedded migrations).
  • Table tokens(id, name, token_hash TEXT UNIQUE, created_at, revoked_at) — only sha256 hashes stored, plaintext shown once at slinuxm token create.
  • Middleware: constant-time compare of Authorization: Bearer <token> hash; 401 JSON body on REST, JSON-RPC error on MCP.
  • auth.enabled = false is rejected at startup if bind isn't loopback.

Build & packaging

  • devenv.nix: enable languages.rust.enable = true; add pkg-config, systemd (libsystemd headers for the journal FFI) to packages. Add scripts: build, run, token wrappers.
  • Static-ish single binary: everything pure-Rust except libsystemd (acceptable — target hosts run systemd by definition) and bundled SQLite. rustls everywhere (no openssl).
  • Example systemd/skyg-linux-tracker.service: hardening flags (ProtectSystem=strict, NoNewPrivileges, ReadWritePaths=/var/lib/skyg-linux-tracker), supplementary group systemd-journal + optional docker group for socket access.

Phases & workstreams

Work is split so that after a small shared foundation, four tracks run in parallel, then three surfaces run in parallel. Each workstream lists its dependencies, the files it owns (to avoid merge conflicts), and its done-criteria.

Phase 0 — Foundation (blocking, single track)

Everything depends on this; keep it deliberately small.

Task Files
Cargo.toml + devenv.nix setup (rust toolchain, libsystemd build dep, all deps pinned) Cargo.toml, devenv.nix
Config layering (defaults < TOML < SKYG_* env < CLI), config.example.toml, clap CLI skeleton with serve + token subcommands src/config.rs, src/main.rs, config.example.toml
Shared DTOs + error types: the response shapes (UnitStatus, TimerInfo, JournalEntry, ContainerSummary, …) every surface returns src/sources/mod.rs (DTO definitions), src/error.rs

Gate: cargo run -- serve starts, loads layered config, prints effective config. No sources or surfaces yet.

Phase 1 — Sources + auth (4 parallel tracks)

Once DTOs exist, each track is fully independent.

WS-A: systemd source

  • Files: src/sources/systemd.rs
  • zbus connection + list_units / list_timers / unit_status returning shared DTOs.
  • Done: an example binary calls each fn against the live bus and prints JSON.

WS-B: journal source

  • Files: src/sources/journal.rs
  • query() snapshot + follow() → mpsc (spawn_blocking bridge).
  • Done: example tails the journal for a unit and prints entries; snapshot respects lines/priority filters.

WS-C: docker source

  • Files: src/sources/docker.rs
  • ps() / container_status() / logs(follow).
  • Done: example lists containers with health/uptime; log follow streams to stdout.

WS-D: auth + DB

  • Files: src/db.rs, src/auth/
  • sqlx pool + migration, token CRUD, slinuxm token create/revoke/list CLI, bearer middleware (axum-compatible).
  • Done: token round-trips CLI → DB → middleware check; startup rejects auth.enabled=false + non-loopback bind.

Phase 2 — Surfaces (3 parallel tracks, after Phase 1)

Each surface consumes sources/ + the auth middleware; they never touch each other's files.

WS-E: REST API (src/api/)

  • All endpoints from the table above, SSE streaming, /healthz.
  • Depends on: A, B, C, D.
  • Done: curl checks pass (Verification #3).

WS-F: MCP server (src/mcp/)

  • StreamableHttpService at /mcp, 9 tools, prompts, resources, server instructions.
  • Depends on: A, B, C, D.
  • Done: MCP inspector checks pass (Verification #4).

WS-G: OTel export (src/telemetry/)

  • Provider init (traces/metrics/logs), metrics poll loop, log fan-out tasks, graceful shutdown.
  • Depends on: A, B, C.
  • Done: otelcol debug exporter shows gauges + log records (Verification #5).

Phase 3 — Integration & hardening (single track, after Phase 2)

  • Wire all surfaces into one axum server in main.rs (the only place surfaces meet).
  • End-to-end negative tests (Verification #6), example systemd unit file, README/config docs, binary size + ldd check for the single-binary claim.

Dependency graph

Phase 0 (config + DTOs + skeleton)
   ├── WS-A systemd ──┐
   ├── WS-B journal ──┼── WS-E REST ──┐
   ├── WS-C docker ── │── WS-F MCP ── ┼── Phase 3 integration
   └── WS-D auth ─────┘── WS-G OTel ──┘

Solo, do it sequentially: 0 → A → B → C → D → E → F → G → 3. With multiple contributors (or parallel agents), Phase 1's four tracks and Phase 2's three tracks are fully parallel — disjoint files, DTO contract fixed in Phase 0.

Verification

  1. cargo build inside devenv shell; unit tests for config precedence and DTO mapping.
  2. Run locally: cargo run -- serve --bind 127.0.0.1; create a token via cargo run -- token create.
  3. REST: curl -H "Authorization: Bearer …" localhost:9109/api/v1/systemd/units?state=failed, SSE follow endpoints via curl -N.
  4. MCP: point a client at http://localhost:9109/mcp (e.g. npx @modelcontextprotocol/inspector) — verify tool list, call each tool, check auth rejection without the header.
  5. OTel: run otelcol with a debug exporter, enable [otel], confirm unit/container gauges appear on the poll interval and journal/docker log records flow with correct severity/attributes.
  6. Negative tests: no token → 401; auth.enabled=false + bind=0.0.0.0 → startup refuses; docker socket absent → clean error on docker endpoints, rest of API unaffected.