Files
skyg-linux-tracker/plan.md
T

263 lines
16 KiB
Markdown

# skyg-linux-tracker — Design Doc
A read-only Linux system monitor in Rust, distributed as a single binary. Exposes systemd state, journald logs, and Docker container state/logs over three surfaces that share one core: a **REST API**, an **MCP server** (Streamable HTTP), and **OpenTelemetry export**. All data is fetched live from the source on every request — no metric/log collection or buffering, except a small SQLite database that stores API access tokens.
## Confirmed requirements
- **Single Rust binary**, read-only against the host.
- **MCP over Streamable HTTP with auth** (mounts on the same axum server as the REST API).
- **Config via TOML file + CLI flags + env vars** (precedence: CLI > env > file > defaults).
- **OTel**: systemd unit / container status exported as **metrics** (gauges); journald / docker logs exported as **OTel logs**. Traces for the API itself are included since they come nearly free.
- **MCP and REST expose the same data**; MCP adds agent-facing context (tool descriptions, instructions, curated resources/prompts).
- **State**: `/var/lib/skyg-linux-tracker/db.sqlite` (SQLite via `sqlx`), access tokens only.
## Tech choices
| Area | Crate | Why |
|---|---|---|
| HTTP | `axum` + `tokio` | Industry standard; rmcp's HTTP transport is a Tower service that mounts directly onto an axum router |
| MCP | `rmcp` (official `modelcontextprotocol/rust-sdk`), features `server`, `macros`, `schemars`, `transport-streamable-http-server` | Official SDK; `StreamableHttpService` mounts at `/mcp` on the same axum router; tools via `#[tool]` macros |
| systemd | `zbus_systemd` (feature `systemd1`, tokio executor) | Pure-Rust generated bindings for `org.freedesktop.systemd1` — `ListUnits`, `ListTimers`, unit properties. No libsystemd C dependency |
| journald | `systemd` crate, `journal` feature (sd-journal FFI) | Only maintained option supporting live tail (`seek_tail` + `next` + `wait`). Links `libsystemd` at build time (present on any systemd host; the devenv shell provides it) |
| Docker | `bollard` (rustls, no openssl) | Async Docker/Podman API over the unix socket; `list_containers`, `inspect_container` (health), `logs(follow=true)` streams |
| OTel | `opentelemetry`, `opentelemetry_sdk`, `opentelemetry-otlp` (metrics + logs both **Stable** in current opentelemetry-rust; traces Beta) | One OTLP exporter config (endpoint, protocol, headers) drives all three signals |
| Auth DB | `sqlx` + `sqlite` (bundled) | Async, compile-time-checked queries, pure-Rust bundled SQLite — keeps the binary self-contained |
| Config | `figment` (TOML + env) + `clap` | Layered config with clear precedence |
## Repo layout
```
linux-system-monitor/
├── Cargo.toml
├── devenv.nix # + languages.rust, pkg-config/openssl/libsystemd build deps
├── devenv.yaml
├── config.example.toml
├── systemd/
│ └── skyg-linux-tracker.service # example unit file (hardened, read-only)
└── src/
├── main.rs # entry: config load → telemetry init → DB → server
├── config.rs # figment layering: defaults < TOML < env(SKYG_) < CLI
├── error.rs # thiserror AppError → axum + MCP error mappings
├── auth/
│ ├── mod.rs # bearer middleware + token verify (sha256 hash lookup)
│ └── tokens.rs # token CRUD against SQLite (`slinuxm token create/revoke/list` CLI)
├── db.rs # sqlx pool, embedded migrations, open-or-create
├── sources/ # === the core: one trait per domain, live reads only ===
│ ├── mod.rs
│ ├── systemd.rs # list_units(), list_timers(), unit_status(name)
│ ├── journal.rs # query(filter) snapshot, follow(filter) → mpsc stream
│ └── docker.rs # ps(), container_status(name), logs(name, follow)
├── api/ # REST layer (thin): DTOs + handlers calling sources/
│ ├── mod.rs # router assembly, auth middleware layer
│ ├── systemd.rs # GET /api/v1/systemd/{units,timers,units/:name}
│ ├── journal.rs # GET /api/v1/journal (snapshot), GET .../journal/follow (SSE)
│ └── docker.rs # GET /api/v1/docker/{ps,containers/:name,containers/:name/logs}
├── mcp/
│ ├── mod.rs # TrackerService: #[tool_router] tools wrapping sources/
│ └── prompts.rs # agent-facing prompts ("diagnose failing units", etc.)
└── telemetry/
├── mod.rs # OTel providers init (traces/metrics/logs), shutdown
├── metrics_export.rs # background loop → observable gauges
└── logs_export.rs # journal/docker follow → OTel log records fan-out
```
The key structural decision: **`sources/` is the only place that talks to D-Bus/journald/Docker.** `api/`, `mcp/`, and `telemetry/` are three thin projections over it, which is what makes "MCP and API expose the same data" true by construction.
## Implementation detail
### Config (`config.rs`, `config.example.toml`)
```toml
[server]
bind = "127.0.0.1" # set 0.0.0.0 only with auth enabled
port = 1984
[auth]
enabled = true # hard-required when bind != localhost
db_path = "/var/lib/skyg-linux-tracker/db.sqlite"
[otel]
enabled = false
endpoint = "http://localhost:4317" # OTLP gRPC; http://…:4318 for HTTP/proto
protocol = "grpc"
service_name = "skyg-linux-tracker" # OTel resource attr service.name
export_unit_metrics = true # poll loop, interval below
export_logs = false # journal/docker log fan-out to OTLP
metrics_interval_secs = 30
[otel.logs]
journal_units = [] # empty = all units
docker_containers = [] # empty = all containers
[sources]
docker_socket = "/var/run/docker.sock"
journal_namespace = "" # default journal
```
Precedence: built-in defaults < `--config /etc/skyg-linux-tracker.toml` < `SKYG_*` env vars < CLI flags. CLI subcommands: `serve` (default), `token create|revoke|list`.
### Sources (live reads only)
- **systemd** (`sources/systemd.rs`): one shared `zbus::Connection` to the system bus. `list_units()` → `ManagerProxy::list_units()`; `list_timers()` → `list_timers()` (next-elapse realtime converted to RFC3339); `unit_status(name)` → `get_unit` (read-only, never `load_unit`) + properties (`ActiveState`, `SubState`, `ExecMainStatus`, `NRestarts`, `ActiveEnterTimestamp`). No caching — every call hits D-Bus.
- **journal** (`sources/journal.rs`): `Journal::open()`, `match_add("_SYSTEMD_UNIT", …)`, `seek_tail()`. Snapshot queries seek backward N entries. `follow()` wraps `Journal::wait()` in a `tokio::task::spawn_blocking` bridge (sd-journal is blocking) feeding a `tokio::sync::mpsc` — consumers (SSE endpoint, MCP tool, OTel log exporter) all read from channels.
- **docker** (`sources/docker.rs`): shared `bollard::Docker` over the unix socket. `ps()` → `list_containers` + `inspect_container` per container for `State.Health`, `StartedAt` (→ uptime), restart count. `logs(name, follow)` → `bollard` log stream (`stdout`, `stderr`, `timestamps`, `since`) as a `Stream<Item = LogOutput>`.
### REST API (`api/`)
All under `/api/v1`, all behind the auth middleware when enabled, all read-only (GET only):
| Endpoint | Source |
|---|---|
| `GET /api/v1/systemd/units?state=failed` | `list_units()` |
| `GET /api/v1/systemd/units/:name` | `unit_status()` |
| `GET /api/v1/systemd/timers` | `list_timers()` |
| `GET /api/v1/journal?unit=X&priority=err&lines=100` | journal snapshot |
| `GET /api/v1/journal/follow?unit=X` (SSE) | journal `follow()` |
| `GET /api/v1/docker/ps` | `ps()` |
| `GET /api/v1/docker/containers/:name` | `container_status()` |
| `GET /api/v1/docker/containers/:name/logs?follow=1` (SSE) | `logs(follow)` |
| `GET /healthz` | unauthenticated liveness |
Streaming endpoints use SSE (`axum::response::Sse`) over the mpsc channels from `sources/`.
### MCP server (`mcp/`)
`rmcp`'s `StreamableHttpService` is a Tower service mounted on the same axum router:
```rust
let mcp_service = StreamableHttpService::new(
|| Ok(TrackerService::new(app_state.clone())), // per-session handler, shared state
LocalSessionManager::default().into(),
StreamableHttpServerConfig::default().with_json_response(false),
);
let app = Router::new()
.nest("/api/v1", api_router)
.nest_service("/mcp", mcp_service)
.layer(auth_middleware); // same bearer auth covers both
```
Tools (each a thin wrapper over `sources/`, params derive `schemars::JsonSchema`):
- `list_systemd_units { state? }` / `get_systemd_unit { name }` / `list_systemd_timers {}`
- `get_journal_logs { unit?, priority?, lines }` / `tail_journal { unit, seconds }` (bounded follow — collects for N seconds, returns; no infinite MCP streams)
- `list_containers {}` / `get_container { name }` / `get_container_logs { name, tail }`
The "bit more context for the agent" lives in:
- `ServerHandler` **instructions**: how the tools relate ("check `list_systemd_units state=failed` first, then pull journal logs for that unit").
- Rich `#[tool(description = …)]` strings explaining systemd states, health statuses, priorities.
- **Prompts** (`mcp/prompts.rs`): `diagnose_unit { name }`, `container_health_report {}` — canned investigation workflows.
- **Resources**: `systemd://units/failed`, `docker://containers` as subscribable snapshot resources.
### OpenTelemetry (`telemetry/`)
One OTLP exporter config → three signals:
1. **Traces** (Beta in opentelemetry-rust): `tracing-opentelemetry` layer + `tower-http` `TraceLayer` — spans per HTTP/MCP request.
2. **Metrics** (Stable): `metrics_export.rs` runs a poll loop every `metrics_interval_secs`:
- `systemd.unit.active` (u64 gauge 0/1, attrs: `unit`, `sub_state`, `load_state`)
- `systemd.unit.failed` (u64 0/1), `systemd.unit.restarts` (u64 counter-ish gauge)
- `systemd.timer.next_elapse_seconds` (f64, attr `timer`)
- `docker.container.running` (0/1, attrs `container`, `image`), `docker.container.healthy` (0/1, `health_status` attr), `docker.container.uptime_seconds`, `docker.container.restarts`
- Configurable unit/container allowlists so cardinality stays bounded.
3. **Logs** (Stable): when `otel.export_logs = true`, `logs_export.rs` spawns journal `follow()` + per-container docker `logs(follow=true)` tasks, converts each entry to an OTel `LogRecord` (severity from journal priority / docker stderr-vs-stdout, attrs: `unit`, `container`, `host`) and emits through the SDK `LoggerProvider`.
"Same information via OTel" is satisfied by reusing the identical `sources/` functions; the OTel path is the only one that does periodic push (metrics poll) and continuous fan-out (logs), because OTLP is a push model — everything else stays pull/live.
### Auth & database (`auth/`, `db.rs`)
- SQLite at `/var/lib/skyg-linux-tracker/db.sqlite` (dir created `0700` on first run; `sqlx` embedded migrations).
- Table `tokens(id, name, token_hash TEXT UNIQUE, created_at, revoked_at)` — only **sha256 hashes** stored, plaintext shown once at `slinuxm token create`.
- Middleware: constant-time compare of `Authorization: Bearer <token>` hash; 401 JSON body on REST, JSON-RPC error on MCP.
- `auth.enabled = false` is rejected at startup if `bind` isn't loopback.
## Build & packaging
- **devenv.nix**: enable `languages.rust.enable = true`; add `pkg-config`, `systemd` (libsystemd headers for the journal FFI) to `packages`. Add scripts: `build`, `run`, `token` wrappers.
- Static-ish single binary: everything pure-Rust except libsystemd (acceptable — target hosts run systemd by definition) and bundled SQLite. rustls everywhere (no openssl).
- Example `systemd/skyg-linux-tracker.service`: hardening flags (`ProtectSystem=strict`, `NoNewPrivileges`, `ReadWritePaths=/var/lib/skyg-linux-tracker`), supplementary group `systemd-journal` + optional `docker` group for socket access.
## Phases & workstreams
Work is split so that after a small shared foundation, four tracks run in parallel, then three surfaces run in parallel. Each workstream lists its dependencies, the files it owns (to avoid merge conflicts), and its done-criteria.
### Phase 0 — Foundation (blocking, single track)
Everything depends on this; keep it deliberately small.
| Task | Files |
|---|---|
| Cargo.toml + devenv.nix setup (rust toolchain, libsystemd build dep, all deps pinned) | `Cargo.toml`, `devenv.nix` |
| Config layering (defaults < TOML < `SKYG_*` env < CLI), `config.example.toml`, clap CLI skeleton with `serve` + `token` subcommands | `src/config.rs`, `src/main.rs`, `config.example.toml` |
| Shared DTOs + error types: the response shapes (`UnitStatus`, `TimerInfo`, `JournalEntry`, `ContainerSummary`, …) every surface returns | `src/sources/mod.rs` (DTO definitions), `src/error.rs` |
**Gate:** `cargo run -- serve` starts, loads layered config, prints effective config. No sources or surfaces yet.
### Phase 1 — Sources + auth (4 parallel tracks)
Once DTOs exist, each track is fully independent.
**WS-A: systemd source**
- Files: `src/sources/systemd.rs`
- zbus connection + `list_units` / `list_timers` / `unit_status` returning shared DTOs.
- Done: an example binary calls each fn against the live bus and prints JSON.
**WS-B: journal source**
- Files: `src/sources/journal.rs`
- `query()` snapshot + `follow()` → mpsc (spawn_blocking bridge).
- Done: example tails the journal for a unit and prints entries; snapshot respects `lines`/`priority` filters.
**WS-C: docker source**
- Files: `src/sources/docker.rs`
- `ps()` / `container_status()` / `logs(follow)`.
- Done: example lists containers with health/uptime; log follow streams to stdout.
**WS-D: auth + DB**
- Files: `src/db.rs`, `src/auth/`
- sqlx pool + migration, token CRUD, `slinuxm token create/revoke/list` CLI, bearer middleware (axum-compatible).
- Done: token round-trips CLI → DB → middleware check; startup rejects `auth.enabled=false` + non-loopback bind.
### Phase 2 — Surfaces (3 parallel tracks, after Phase 1)
Each surface consumes `sources/` + the auth middleware; they never touch each other's files.
**WS-E: REST API** (`src/api/`)
- All endpoints from the table above, SSE streaming, `/healthz`.
- Depends on: A, B, C, D.
- Done: curl checks pass (Verification #3).
**WS-F: MCP server** (`src/mcp/`)
- `StreamableHttpService` at `/mcp`, 9 tools, prompts, resources, server instructions.
- Depends on: A, B, C, D.
- Done: MCP inspector checks pass (Verification #4).
**WS-G: OTel export** (`src/telemetry/`)
- Provider init (traces/metrics/logs), metrics poll loop, log fan-out tasks, graceful shutdown.
- Depends on: A, B, C.
- Done: otelcol debug exporter shows gauges + log records (Verification #5).
### Phase 3 — Integration & hardening (single track, after Phase 2)
- Wire all surfaces into one axum server in `main.rs` (the only place surfaces meet).
- End-to-end negative tests (Verification #6), example systemd unit file, README/config docs, binary size + `ldd` check for the single-binary claim.
### Dependency graph
```
Phase 0 (config + DTOs + skeleton)
├── WS-A systemd ──┐
├── WS-B journal ──┼── WS-E REST ──┐
├── WS-C docker ── │── WS-F MCP ── ┼── Phase 3 integration
└── WS-D auth ─────┘── WS-G OTel ──┘
```
Solo, do it sequentially: 0 → A → B → C → D → E → F → G → 3. With multiple contributors (or parallel agents), Phase 1's four tracks and Phase 2's three tracks are fully parallel — disjoint files, DTO contract fixed in Phase 0.
## Verification
1. `cargo build` inside `devenv shell`; unit tests for config precedence and DTO mapping.
2. Run locally: `cargo run -- serve --bind 127.0.0.1`; create a token via `cargo run -- token create`.
3. REST: `curl -H "Authorization: Bearer …" localhost:1984/api/v1/systemd/units?state=failed`, SSE follow endpoints via `curl -N`.
4. MCP: point a client at `http://localhost:1984/mcp` (e.g. `npx @modelcontextprotocol/inspector`) — verify tool list, call each tool, check auth rejection without the header.
5. OTel: run `otelcol` with a debug exporter, enable `[otel]`, confirm unit/container gauges appear on the poll interval and journal/docker log records flow with correct severity/attributes.
6. Negative tests: no token → 401; `auth.enabled=false` + `bind=0.0.0.0` → startup refuses; docker socket absent → clean error on docker endpoints, rest of API unaffected.