# skyg-linux-tracker — Design Doc A read-only Linux system monitor in Rust, distributed as a single binary. Exposes systemd state, journald logs, and Docker container state/logs over three surfaces that share one core: a **REST API**, an **MCP server** (Streamable HTTP), and **OpenTelemetry export**. All data is fetched live from the source on every request — no metric/log collection or buffering, except a small SQLite database that stores API access tokens. ## Confirmed requirements - **Single Rust binary**, read-only against the host. - **MCP over Streamable HTTP with auth** (mounts on the same axum server as the REST API). - **Config via TOML file + CLI flags + env vars** (precedence: CLI > env > file > defaults). - **OTel**: systemd unit / container status exported as **metrics** (gauges); journald / docker logs exported as **OTel logs**. Traces for the API itself are included since they come nearly free. - **MCP and REST expose the same data**; MCP adds agent-facing context (tool descriptions, instructions, curated resources/prompts). - **State**: `/var/lib/skyg-linux-tracker/db.sqlite` (SQLite via `sqlx`), access tokens only. ## Tech choices | Area | Crate | Why | |---|---|---| | HTTP | `axum` + `tokio` | Industry standard; rmcp's HTTP transport is a Tower service that mounts directly onto an axum router | | MCP | `rmcp` (official `modelcontextprotocol/rust-sdk`), features `server`, `macros`, `schemars`, `transport-streamable-http-server` | Official SDK; `StreamableHttpService` mounts at `/mcp` on the same axum router; tools via `#[tool]` macros | | systemd | `zbus_systemd` (feature `systemd1`, tokio executor) | Pure-Rust generated bindings for `org.freedesktop.systemd1` — `ListUnits`, `ListTimers`, unit properties. No libsystemd C dependency | | journald | `systemd` crate, `journal` feature (sd-journal FFI) | Only maintained option supporting live tail (`seek_tail` + `next` + `wait`). Links `libsystemd` at build time (present on any systemd host; the devenv shell provides it) | | Docker | `bollard` (rustls, no openssl) | Async Docker/Podman API over the unix socket; `list_containers`, `inspect_container` (health), `logs(follow=true)` streams | | OTel | `opentelemetry`, `opentelemetry_sdk`, `opentelemetry-otlp` (metrics + logs both **Stable** in current opentelemetry-rust; traces Beta) | One OTLP exporter config (endpoint, protocol, headers) drives all three signals | | Auth DB | `sqlx` + `sqlite` (bundled) | Async, compile-time-checked queries, pure-Rust bundled SQLite — keeps the binary self-contained | | Config | `figment` (TOML + env) + `clap` | Layered config with clear precedence | ## Repo layout ``` linux-system-monitor/ ├── Cargo.toml ├── devenv.nix # + languages.rust, pkg-config/openssl/libsystemd build deps ├── devenv.yaml ├── config.example.toml ├── systemd/ │ └── skyg-linux-tracker.service # example unit file (hardened, read-only) └── src/ ├── main.rs # entry: config load → telemetry init → DB → server ├── config.rs # figment layering: defaults < TOML < env(SKYG_) < CLI ├── error.rs # thiserror AppError → axum + MCP error mappings ├── auth/ │ ├── mod.rs # bearer middleware + token verify (sha256 hash lookup) │ └── tokens.rs # token CRUD against SQLite (`slinuxm token create/revoke/list` CLI) ├── db.rs # sqlx pool, embedded migrations, open-or-create ├── sources/ # === the core: one trait per domain, live reads only === │ ├── mod.rs │ ├── systemd.rs # list_units(), list_timers(), unit_status(name) │ ├── journal.rs # query(filter) snapshot, follow(filter) → mpsc stream │ └── docker.rs # ps(), container_status(name), logs(name, follow) ├── api/ # REST layer (thin): DTOs + handlers calling sources/ │ ├── mod.rs # router assembly, auth middleware layer │ ├── systemd.rs # GET /api/v1/systemd/{units,timers,units/:name} │ ├── journal.rs # GET /api/v1/journal (snapshot), GET .../journal/follow (SSE) │ └── docker.rs # GET /api/v1/docker/{ps,containers/:name,containers/:name/logs} ├── mcp/ │ ├── mod.rs # TrackerService: #[tool_router] tools wrapping sources/ │ └── prompts.rs # agent-facing prompts ("diagnose failing units", etc.) └── telemetry/ ├── mod.rs # OTel providers init (traces/metrics/logs), shutdown ├── metrics_export.rs # background loop → observable gauges └── logs_export.rs # journal/docker follow → OTel log records fan-out ``` The key structural decision: **`sources/` is the only place that talks to D-Bus/journald/Docker.** `api/`, `mcp/`, and `telemetry/` are three thin projections over it, which is what makes "MCP and API expose the same data" true by construction. ## Implementation detail ### Config (`config.rs`, `config.example.toml`) ```toml [server] bind = "127.0.0.1" # set 0.0.0.0 only with auth enabled port = 9109 [auth] enabled = true # hard-required when bind != localhost db_path = "/var/lib/skyg-linux-tracker/db.sqlite" [otel] enabled = false endpoint = "http://localhost:4317" # OTLP gRPC; http://…:4318 for HTTP/proto protocol = "grpc" service_name = "skyg-linux-tracker" # OTel resource attr service.name export_unit_metrics = true # poll loop, interval below export_logs = false # journal/docker log fan-out to OTLP metrics_interval_secs = 30 [otel.logs] journal_units = [] # empty = all units docker_containers = [] # empty = all containers [sources] docker_socket = "/var/run/docker.sock" journal_namespace = "" # default journal ``` Precedence: built-in defaults < `--config /etc/skyg-linux-tracker.toml` < `SKYG_*` env vars < CLI flags. CLI subcommands: `serve` (default), `token create|revoke|list`. ### Sources (live reads only) - **systemd** (`sources/systemd.rs`): one shared `zbus::Connection` to the system bus. `list_units()` → `ManagerProxy::list_units()`; `list_timers()` → `list_timers()` (next-elapse realtime converted to RFC3339); `unit_status(name)` → `get_unit` (read-only, never `load_unit`) + properties (`ActiveState`, `SubState`, `ExecMainStatus`, `NRestarts`, `ActiveEnterTimestamp`). No caching — every call hits D-Bus. - **journal** (`sources/journal.rs`): `Journal::open()`, `match_add("_SYSTEMD_UNIT", …)`, `seek_tail()`. Snapshot queries seek backward N entries. `follow()` wraps `Journal::wait()` in a `tokio::task::spawn_blocking` bridge (sd-journal is blocking) feeding a `tokio::sync::mpsc` — consumers (SSE endpoint, MCP tool, OTel log exporter) all read from channels. - **docker** (`sources/docker.rs`): shared `bollard::Docker` over the unix socket. `ps()` → `list_containers` + `inspect_container` per container for `State.Health`, `StartedAt` (→ uptime), restart count. `logs(name, follow)` → `bollard` log stream (`stdout`, `stderr`, `timestamps`, `since`) as a `Stream`. ### REST API (`api/`) All under `/api/v1`, all behind the auth middleware when enabled, all read-only (GET only): | Endpoint | Source | |---|---| | `GET /api/v1/systemd/units?state=failed` | `list_units()` | | `GET /api/v1/systemd/units/:name` | `unit_status()` | | `GET /api/v1/systemd/timers` | `list_timers()` | | `GET /api/v1/journal?unit=X&priority=err&lines=100` | journal snapshot | | `GET /api/v1/journal/follow?unit=X` (SSE) | journal `follow()` | | `GET /api/v1/docker/ps` | `ps()` | | `GET /api/v1/docker/containers/:name` | `container_status()` | | `GET /api/v1/docker/containers/:name/logs?follow=1` (SSE) | `logs(follow)` | | `GET /healthz` | unauthenticated liveness | Streaming endpoints use SSE (`axum::response::Sse`) over the mpsc channels from `sources/`. ### MCP server (`mcp/`) `rmcp`'s `StreamableHttpService` is a Tower service mounted on the same axum router: ```rust let mcp_service = StreamableHttpService::new( || Ok(TrackerService::new(app_state.clone())), // per-session handler, shared state LocalSessionManager::default().into(), StreamableHttpServerConfig::default().with_json_response(false), ); let app = Router::new() .nest("/api/v1", api_router) .nest_service("/mcp", mcp_service) .layer(auth_middleware); // same bearer auth covers both ``` Tools (each a thin wrapper over `sources/`, params derive `schemars::JsonSchema`): - `list_systemd_units { state? }` / `get_systemd_unit { name }` / `list_systemd_timers {}` - `get_journal_logs { unit?, priority?, lines }` / `tail_journal { unit, seconds }` (bounded follow — collects for N seconds, returns; no infinite MCP streams) - `list_containers {}` / `get_container { name }` / `get_container_logs { name, tail }` The "bit more context for the agent" lives in: - `ServerHandler` **instructions**: how the tools relate ("check `list_systemd_units state=failed` first, then pull journal logs for that unit"). - Rich `#[tool(description = …)]` strings explaining systemd states, health statuses, priorities. - **Prompts** (`mcp/prompts.rs`): `diagnose_unit { name }`, `container_health_report {}` — canned investigation workflows. - **Resources**: `systemd://units/failed`, `docker://containers` as subscribable snapshot resources. ### OpenTelemetry (`telemetry/`) One OTLP exporter config → three signals: 1. **Traces** (Beta in opentelemetry-rust): `tracing-opentelemetry` layer + `tower-http` `TraceLayer` — spans per HTTP/MCP request. 2. **Metrics** (Stable): `metrics_export.rs` runs a poll loop every `metrics_interval_secs`: - `systemd.unit.active` (u64 gauge 0/1, attrs: `unit`, `sub_state`, `load_state`) - `systemd.unit.failed` (u64 0/1), `systemd.unit.restarts` (u64 counter-ish gauge) - `systemd.timer.next_elapse_seconds` (f64, attr `timer`) - `docker.container.running` (0/1, attrs `container`, `image`), `docker.container.healthy` (0/1, `health_status` attr), `docker.container.uptime_seconds`, `docker.container.restarts` - Configurable unit/container allowlists so cardinality stays bounded. 3. **Logs** (Stable): when `otel.export_logs = true`, `logs_export.rs` spawns journal `follow()` + per-container docker `logs(follow=true)` tasks, converts each entry to an OTel `LogRecord` (severity from journal priority / docker stderr-vs-stdout, attrs: `unit`, `container`, `host`) and emits through the SDK `LoggerProvider`. "Same information via OTel" is satisfied by reusing the identical `sources/` functions; the OTel path is the only one that does periodic push (metrics poll) and continuous fan-out (logs), because OTLP is a push model — everything else stays pull/live. ### Auth & database (`auth/`, `db.rs`) - SQLite at `/var/lib/skyg-linux-tracker/db.sqlite` (dir created `0700` on first run; `sqlx` embedded migrations). - Table `tokens(id, name, token_hash TEXT UNIQUE, created_at, revoked_at)` — only **sha256 hashes** stored, plaintext shown once at `slinuxm token create`. - Middleware: constant-time compare of `Authorization: Bearer ` hash; 401 JSON body on REST, JSON-RPC error on MCP. - `auth.enabled = false` is rejected at startup if `bind` isn't loopback. ## Build & packaging - **devenv.nix**: enable `languages.rust.enable = true`; add `pkg-config`, `systemd` (libsystemd headers for the journal FFI) to `packages`. Add scripts: `build`, `run`, `token` wrappers. - Static-ish single binary: everything pure-Rust except libsystemd (acceptable — target hosts run systemd by definition) and bundled SQLite. rustls everywhere (no openssl). - Example `systemd/skyg-linux-tracker.service`: hardening flags (`ProtectSystem=strict`, `NoNewPrivileges`, `ReadWritePaths=/var/lib/skyg-linux-tracker`), supplementary group `systemd-journal` + optional `docker` group for socket access. ## Phases & workstreams Work is split so that after a small shared foundation, four tracks run in parallel, then three surfaces run in parallel. Each workstream lists its dependencies, the files it owns (to avoid merge conflicts), and its done-criteria. ### Phase 0 — Foundation (blocking, single track) Everything depends on this; keep it deliberately small. | Task | Files | |---|---| | Cargo.toml + devenv.nix setup (rust toolchain, libsystemd build dep, all deps pinned) | `Cargo.toml`, `devenv.nix` | | Config layering (defaults < TOML < `SKYG_*` env < CLI), `config.example.toml`, clap CLI skeleton with `serve` + `token` subcommands | `src/config.rs`, `src/main.rs`, `config.example.toml` | | Shared DTOs + error types: the response shapes (`UnitStatus`, `TimerInfo`, `JournalEntry`, `ContainerSummary`, …) every surface returns | `src/sources/mod.rs` (DTO definitions), `src/error.rs` | **Gate:** `cargo run -- serve` starts, loads layered config, prints effective config. No sources or surfaces yet. ### Phase 1 — Sources + auth (4 parallel tracks) Once DTOs exist, each track is fully independent. **WS-A: systemd source** - Files: `src/sources/systemd.rs` - zbus connection + `list_units` / `list_timers` / `unit_status` returning shared DTOs. - Done: an example binary calls each fn against the live bus and prints JSON. **WS-B: journal source** - Files: `src/sources/journal.rs` - `query()` snapshot + `follow()` → mpsc (spawn_blocking bridge). - Done: example tails the journal for a unit and prints entries; snapshot respects `lines`/`priority` filters. **WS-C: docker source** - Files: `src/sources/docker.rs` - `ps()` / `container_status()` / `logs(follow)`. - Done: example lists containers with health/uptime; log follow streams to stdout. **WS-D: auth + DB** - Files: `src/db.rs`, `src/auth/` - sqlx pool + migration, token CRUD, `slinuxm token create/revoke/list` CLI, bearer middleware (axum-compatible). - Done: token round-trips CLI → DB → middleware check; startup rejects `auth.enabled=false` + non-loopback bind. ### Phase 2 — Surfaces (3 parallel tracks, after Phase 1) Each surface consumes `sources/` + the auth middleware; they never touch each other's files. **WS-E: REST API** (`src/api/`) - All endpoints from the table above, SSE streaming, `/healthz`. - Depends on: A, B, C, D. - Done: curl checks pass (Verification #3). **WS-F: MCP server** (`src/mcp/`) - `StreamableHttpService` at `/mcp`, 9 tools, prompts, resources, server instructions. - Depends on: A, B, C, D. - Done: MCP inspector checks pass (Verification #4). **WS-G: OTel export** (`src/telemetry/`) - Provider init (traces/metrics/logs), metrics poll loop, log fan-out tasks, graceful shutdown. - Depends on: A, B, C. - Done: otelcol debug exporter shows gauges + log records (Verification #5). ### Phase 3 — Integration & hardening (single track, after Phase 2) - Wire all surfaces into one axum server in `main.rs` (the only place surfaces meet). - End-to-end negative tests (Verification #6), example systemd unit file, README/config docs, binary size + `ldd` check for the single-binary claim. ### Dependency graph ``` Phase 0 (config + DTOs + skeleton) ├── WS-A systemd ──┐ ├── WS-B journal ──┼── WS-E REST ──┐ ├── WS-C docker ── │── WS-F MCP ── ┼── Phase 3 integration └── WS-D auth ─────┘── WS-G OTel ──┘ ``` Solo, do it sequentially: 0 → A → B → C → D → E → F → G → 3. With multiple contributors (or parallel agents), Phase 1's four tracks and Phase 2's three tracks are fully parallel — disjoint files, DTO contract fixed in Phase 0. ## Verification 1. `cargo build` inside `devenv shell`; unit tests for config precedence and DTO mapping. 2. Run locally: `cargo run -- serve --bind 127.0.0.1`; create a token via `cargo run -- token create`. 3. REST: `curl -H "Authorization: Bearer …" localhost:9109/api/v1/systemd/units?state=failed`, SSE follow endpoints via `curl -N`. 4. MCP: point a client at `http://localhost:9109/mcp` (e.g. `npx @modelcontextprotocol/inspector`) — verify tool list, call each tool, check auth rejection without the header. 5. OTel: run `otelcol` with a debug exporter, enable `[otel]`, confirm unit/container gauges appear on the poll interval and journal/docker log records flow with correct severity/attributes. 6. Negative tests: no token → 401; `auth.enabled=false` + `bind=0.0.0.0` → startup refuses; docker socket absent → clean error on docker endpoints, rest of API unaffected.