263 lines
16 KiB
Markdown
263 lines
16 KiB
Markdown
# skyg-linux-tracker — Design Doc
|
|
|
|
A read-only Linux system monitor in Rust, distributed as a single binary. Exposes systemd state, journald logs, and Docker container state/logs over three surfaces that share one core: a **REST API**, an **MCP server** (Streamable HTTP), and **OpenTelemetry export**. All data is fetched live from the source on every request — no metric/log collection or buffering, except a small SQLite database that stores API access tokens.
|
|
|
|
## Confirmed requirements
|
|
|
|
- **Single Rust binary**, read-only against the host.
|
|
- **MCP over Streamable HTTP with auth** (mounts on the same axum server as the REST API).
|
|
- **Config via TOML file + CLI flags + env vars** (precedence: CLI > env > file > defaults).
|
|
- **OTel**: systemd unit / container status exported as **metrics** (gauges); journald / docker logs exported as **OTel logs**. Traces for the API itself are included since they come nearly free.
|
|
- **MCP and REST expose the same data**; MCP adds agent-facing context (tool descriptions, instructions, curated resources/prompts).
|
|
- **State**: `/var/lib/skyg-linux-tracker/db.sqlite` (SQLite via `sqlx`), access tokens only.
|
|
|
|
## Tech choices
|
|
|
|
| Area | Crate | Why |
|
|
|---|---|---|
|
|
| HTTP | `axum` + `tokio` | Industry standard; rmcp's HTTP transport is a Tower service that mounts directly onto an axum router |
|
|
| MCP | `rmcp` (official `modelcontextprotocol/rust-sdk`), features `server`, `macros`, `schemars`, `transport-streamable-http-server` | Official SDK; `StreamableHttpService` mounts at `/mcp` on the same axum router; tools via `#[tool]` macros |
|
|
| systemd | `zbus_systemd` (feature `systemd1`, tokio executor) | Pure-Rust generated bindings for `org.freedesktop.systemd1` — `ListUnits`, `ListTimers`, unit properties. No libsystemd C dependency |
|
|
| journald | `systemd` crate, `journal` feature (sd-journal FFI) | Only maintained option supporting live tail (`seek_tail` + `next` + `wait`). Links `libsystemd` at build time (present on any systemd host; the devenv shell provides it) |
|
|
| Docker | `bollard` (rustls, no openssl) | Async Docker/Podman API over the unix socket; `list_containers`, `inspect_container` (health), `logs(follow=true)` streams |
|
|
| OTel | `opentelemetry`, `opentelemetry_sdk`, `opentelemetry-otlp` (metrics + logs both **Stable** in current opentelemetry-rust; traces Beta) | One OTLP exporter config (endpoint, protocol, headers) drives all three signals |
|
|
| Auth DB | `sqlx` + `sqlite` (bundled) | Async, compile-time-checked queries, pure-Rust bundled SQLite — keeps the binary self-contained |
|
|
| Config | `figment` (TOML + env) + `clap` | Layered config with clear precedence |
|
|
|
|
## Repo layout
|
|
|
|
```
|
|
linux-system-monitor/
|
|
├── Cargo.toml
|
|
├── devenv.nix # + languages.rust, pkg-config/openssl/libsystemd build deps
|
|
├── devenv.yaml
|
|
├── config.example.toml
|
|
├── systemd/
|
|
│ └── skyg-linux-tracker.service # example unit file (hardened, read-only)
|
|
└── src/
|
|
├── main.rs # entry: config load → telemetry init → DB → server
|
|
├── config.rs # figment layering: defaults < TOML < env(SKYG_) < CLI
|
|
├── error.rs # thiserror AppError → axum + MCP error mappings
|
|
├── auth/
|
|
│ ├── mod.rs # bearer middleware + token verify (sha256 hash lookup)
|
|
│ └── tokens.rs # token CRUD against SQLite (`slinuxm token create/revoke/list` CLI)
|
|
├── db.rs # sqlx pool, embedded migrations, open-or-create
|
|
├── sources/ # === the core: one trait per domain, live reads only ===
|
|
│ ├── mod.rs
|
|
│ ├── systemd.rs # list_units(), list_timers(), unit_status(name)
|
|
│ ├── journal.rs # query(filter) snapshot, follow(filter) → mpsc stream
|
|
│ └── docker.rs # ps(), container_status(name), logs(name, follow)
|
|
├── api/ # REST layer (thin): DTOs + handlers calling sources/
|
|
│ ├── mod.rs # router assembly, auth middleware layer
|
|
│ ├── systemd.rs # GET /api/v1/systemd/{units,timers,units/:name}
|
|
│ ├── journal.rs # GET /api/v1/journal (snapshot), GET .../journal/follow (SSE)
|
|
│ └── docker.rs # GET /api/v1/docker/{ps,containers/:name,containers/:name/logs}
|
|
├── mcp/
|
|
│ ├── mod.rs # TrackerService: #[tool_router] tools wrapping sources/
|
|
│ └── prompts.rs # agent-facing prompts ("diagnose failing units", etc.)
|
|
└── telemetry/
|
|
├── mod.rs # OTel providers init (traces/metrics/logs), shutdown
|
|
├── metrics_export.rs # background loop → observable gauges
|
|
└── logs_export.rs # journal/docker follow → OTel log records fan-out
|
|
```
|
|
|
|
The key structural decision: **`sources/` is the only place that talks to D-Bus/journald/Docker.** `api/`, `mcp/`, and `telemetry/` are three thin projections over it, which is what makes "MCP and API expose the same data" true by construction.
|
|
|
|
## Implementation detail
|
|
|
|
### Config (`config.rs`, `config.example.toml`)
|
|
|
|
```toml
|
|
[server]
|
|
bind = "127.0.0.1" # set 0.0.0.0 only with auth enabled
|
|
port = 9109
|
|
|
|
[auth]
|
|
enabled = true # hard-required when bind != localhost
|
|
db_path = "/var/lib/skyg-linux-tracker/db.sqlite"
|
|
|
|
[otel]
|
|
enabled = false
|
|
endpoint = "http://localhost:4317" # OTLP gRPC; http://…:4318 for HTTP/proto
|
|
protocol = "grpc"
|
|
service_name = "skyg-linux-tracker" # OTel resource attr service.name
|
|
export_unit_metrics = true # poll loop, interval below
|
|
export_logs = false # journal/docker log fan-out to OTLP
|
|
metrics_interval_secs = 30
|
|
|
|
[otel.logs]
|
|
journal_units = [] # empty = all units
|
|
docker_containers = [] # empty = all containers
|
|
|
|
[sources]
|
|
docker_socket = "/var/run/docker.sock"
|
|
journal_namespace = "" # default journal
|
|
```
|
|
|
|
Precedence: built-in defaults < `--config /etc/skyg-linux-tracker.toml` < `SKYG_*` env vars < CLI flags. CLI subcommands: `serve` (default), `token create|revoke|list`.
|
|
|
|
### Sources (live reads only)
|
|
|
|
- **systemd** (`sources/systemd.rs`): one shared `zbus::Connection` to the system bus. `list_units()` → `ManagerProxy::list_units()`; `list_timers()` → `list_timers()` (next-elapse realtime converted to RFC3339); `unit_status(name)` → `load_unit` + properties (`ActiveState`, `SubState`, `ExecMainStatus`, `NRestarts`, `ActiveEnterTimestamp`). No caching — every call hits D-Bus.
|
|
- **journal** (`sources/journal.rs`): `Journal::open()`, `match_add("_SYSTEMD_UNIT", …)`, `seek_tail()`. Snapshot queries seek backward N entries. `follow()` wraps `Journal::wait()` in a `tokio::task::spawn_blocking` bridge (sd-journal is blocking) feeding a `tokio::sync::mpsc` — consumers (SSE endpoint, MCP tool, OTel log exporter) all read from channels.
|
|
- **docker** (`sources/docker.rs`): shared `bollard::Docker` over the unix socket. `ps()` → `list_containers` + `inspect_container` per container for `State.Health`, `StartedAt` (→ uptime), restart count. `logs(name, follow)` → `bollard` log stream (`stdout`, `stderr`, `timestamps`, `since`) as a `Stream<Item = LogOutput>`.
|
|
|
|
### REST API (`api/`)
|
|
|
|
All under `/api/v1`, all behind the auth middleware when enabled, all read-only (GET only):
|
|
|
|
| Endpoint | Source |
|
|
|---|---|
|
|
| `GET /api/v1/systemd/units?state=failed` | `list_units()` |
|
|
| `GET /api/v1/systemd/units/:name` | `unit_status()` |
|
|
| `GET /api/v1/systemd/timers` | `list_timers()` |
|
|
| `GET /api/v1/journal?unit=X&priority=err&lines=100` | journal snapshot |
|
|
| `GET /api/v1/journal/follow?unit=X` (SSE) | journal `follow()` |
|
|
| `GET /api/v1/docker/ps` | `ps()` |
|
|
| `GET /api/v1/docker/containers/:name` | `container_status()` |
|
|
| `GET /api/v1/docker/containers/:name/logs?follow=1` (SSE) | `logs(follow)` |
|
|
| `GET /healthz` | unauthenticated liveness |
|
|
|
|
Streaming endpoints use SSE (`axum::response::Sse`) over the mpsc channels from `sources/`.
|
|
|
|
### MCP server (`mcp/`)
|
|
|
|
`rmcp`'s `StreamableHttpService` is a Tower service mounted on the same axum router:
|
|
|
|
```rust
|
|
let mcp_service = StreamableHttpService::new(
|
|
|| Ok(TrackerService::new(app_state.clone())), // per-session handler, shared state
|
|
LocalSessionManager::default().into(),
|
|
StreamableHttpServerConfig::default().with_json_response(false),
|
|
);
|
|
let app = Router::new()
|
|
.nest("/api/v1", api_router)
|
|
.nest_service("/mcp", mcp_service)
|
|
.layer(auth_middleware); // same bearer auth covers both
|
|
```
|
|
|
|
Tools (each a thin wrapper over `sources/`, params derive `schemars::JsonSchema`):
|
|
|
|
- `list_systemd_units { state? }` / `get_systemd_unit { name }` / `list_systemd_timers {}`
|
|
- `get_journal_logs { unit?, priority?, lines }` / `tail_journal { unit, seconds }` (bounded follow — collects for N seconds, returns; no infinite MCP streams)
|
|
- `list_containers {}` / `get_container { name }` / `get_container_logs { name, tail }`
|
|
|
|
The "bit more context for the agent" lives in:
|
|
- `ServerHandler` **instructions**: how the tools relate ("check `list_systemd_units state=failed` first, then pull journal logs for that unit").
|
|
- Rich `#[tool(description = …)]` strings explaining systemd states, health statuses, priorities.
|
|
- **Prompts** (`mcp/prompts.rs`): `diagnose_unit { name }`, `container_health_report {}` — canned investigation workflows.
|
|
- **Resources**: `systemd://units/failed`, `docker://containers` as subscribable snapshot resources.
|
|
|
|
### OpenTelemetry (`telemetry/`)
|
|
|
|
One OTLP exporter config → three signals:
|
|
|
|
1. **Traces** (Beta in opentelemetry-rust): `tracing-opentelemetry` layer + `tower-http` `TraceLayer` — spans per HTTP/MCP request.
|
|
2. **Metrics** (Stable): `metrics_export.rs` runs a poll loop every `metrics_interval_secs`:
|
|
- `systemd.unit.active` (u64 gauge 0/1, attrs: `unit`, `sub_state`, `load_state`)
|
|
- `systemd.unit.failed` (u64 0/1), `systemd.unit.restarts` (u64 counter-ish gauge)
|
|
- `systemd.timer.next_elapse_seconds` (f64, attr `timer`)
|
|
- `docker.container.running` (0/1, attrs `container`, `image`), `docker.container.healthy` (0/1, `health_status` attr), `docker.container.uptime_seconds`, `docker.container.restarts`
|
|
- Configurable unit/container allowlists so cardinality stays bounded.
|
|
3. **Logs** (Stable): when `otel.export_logs = true`, `logs_export.rs` spawns journal `follow()` + per-container docker `logs(follow=true)` tasks, converts each entry to an OTel `LogRecord` (severity from journal priority / docker stderr-vs-stdout, attrs: `unit`, `container`, `host`) and emits through the SDK `LoggerProvider`.
|
|
|
|
"Same information via OTel" is satisfied by reusing the identical `sources/` functions; the OTel path is the only one that does periodic push (metrics poll) and continuous fan-out (logs), because OTLP is a push model — everything else stays pull/live.
|
|
|
|
### Auth & database (`auth/`, `db.rs`)
|
|
|
|
- SQLite at `/var/lib/skyg-linux-tracker/db.sqlite` (dir created `0700` on first run; `sqlx` embedded migrations).
|
|
- Table `tokens(id, name, token_hash TEXT UNIQUE, created_at, revoked_at)` — only **sha256 hashes** stored, plaintext shown once at `slinuxm token create`.
|
|
- Middleware: constant-time compare of `Authorization: Bearer <token>` hash; 401 JSON body on REST, JSON-RPC error on MCP.
|
|
- `auth.enabled = false` is rejected at startup if `bind` isn't loopback.
|
|
|
|
## Build & packaging
|
|
|
|
- **devenv.nix**: enable `languages.rust.enable = true`; add `pkg-config`, `systemd` (libsystemd headers for the journal FFI) to `packages`. Add scripts: `build`, `run`, `token` wrappers.
|
|
- Static-ish single binary: everything pure-Rust except libsystemd (acceptable — target hosts run systemd by definition) and bundled SQLite. rustls everywhere (no openssl).
|
|
- Example `systemd/skyg-linux-tracker.service`: hardening flags (`ProtectSystem=strict`, `NoNewPrivileges`, `ReadWritePaths=/var/lib/skyg-linux-tracker`), supplementary group `systemd-journal` + optional `docker` group for socket access.
|
|
|
|
## Phases & workstreams
|
|
|
|
Work is split so that after a small shared foundation, four tracks run in parallel, then three surfaces run in parallel. Each workstream lists its dependencies, the files it owns (to avoid merge conflicts), and its done-criteria.
|
|
|
|
### Phase 0 — Foundation (blocking, single track)
|
|
|
|
Everything depends on this; keep it deliberately small.
|
|
|
|
| Task | Files |
|
|
|---|---|
|
|
| Cargo.toml + devenv.nix setup (rust toolchain, libsystemd build dep, all deps pinned) | `Cargo.toml`, `devenv.nix` |
|
|
| Config layering (defaults < TOML < `SKYG_*` env < CLI), `config.example.toml`, clap CLI skeleton with `serve` + `token` subcommands | `src/config.rs`, `src/main.rs`, `config.example.toml` |
|
|
| Shared DTOs + error types: the response shapes (`UnitStatus`, `TimerInfo`, `JournalEntry`, `ContainerSummary`, …) every surface returns | `src/sources/mod.rs` (DTO definitions), `src/error.rs` |
|
|
|
|
**Gate:** `cargo run -- serve` starts, loads layered config, prints effective config. No sources or surfaces yet.
|
|
|
|
### Phase 1 — Sources + auth (4 parallel tracks)
|
|
|
|
Once DTOs exist, each track is fully independent.
|
|
|
|
**WS-A: systemd source**
|
|
- Files: `src/sources/systemd.rs`
|
|
- zbus connection + `list_units` / `list_timers` / `unit_status` returning shared DTOs.
|
|
- Done: an example binary calls each fn against the live bus and prints JSON.
|
|
|
|
**WS-B: journal source**
|
|
- Files: `src/sources/journal.rs`
|
|
- `query()` snapshot + `follow()` → mpsc (spawn_blocking bridge).
|
|
- Done: example tails the journal for a unit and prints entries; snapshot respects `lines`/`priority` filters.
|
|
|
|
**WS-C: docker source**
|
|
- Files: `src/sources/docker.rs`
|
|
- `ps()` / `container_status()` / `logs(follow)`.
|
|
- Done: example lists containers with health/uptime; log follow streams to stdout.
|
|
|
|
**WS-D: auth + DB**
|
|
- Files: `src/db.rs`, `src/auth/`
|
|
- sqlx pool + migration, token CRUD, `slinuxm token create/revoke/list` CLI, bearer middleware (axum-compatible).
|
|
- Done: token round-trips CLI → DB → middleware check; startup rejects `auth.enabled=false` + non-loopback bind.
|
|
|
|
### Phase 2 — Surfaces (3 parallel tracks, after Phase 1)
|
|
|
|
Each surface consumes `sources/` + the auth middleware; they never touch each other's files.
|
|
|
|
**WS-E: REST API** (`src/api/`)
|
|
- All endpoints from the table above, SSE streaming, `/healthz`.
|
|
- Depends on: A, B, C, D.
|
|
- Done: curl checks pass (Verification #3).
|
|
|
|
**WS-F: MCP server** (`src/mcp/`)
|
|
- `StreamableHttpService` at `/mcp`, 9 tools, prompts, resources, server instructions.
|
|
- Depends on: A, B, C, D.
|
|
- Done: MCP inspector checks pass (Verification #4).
|
|
|
|
**WS-G: OTel export** (`src/telemetry/`)
|
|
- Provider init (traces/metrics/logs), metrics poll loop, log fan-out tasks, graceful shutdown.
|
|
- Depends on: A, B, C.
|
|
- Done: otelcol debug exporter shows gauges + log records (Verification #5).
|
|
|
|
### Phase 3 — Integration & hardening (single track, after Phase 2)
|
|
|
|
- Wire all surfaces into one axum server in `main.rs` (the only place surfaces meet).
|
|
- End-to-end negative tests (Verification #6), example systemd unit file, README/config docs, binary size + `ldd` check for the single-binary claim.
|
|
|
|
### Dependency graph
|
|
|
|
```
|
|
Phase 0 (config + DTOs + skeleton)
|
|
├── WS-A systemd ──┐
|
|
├── WS-B journal ──┼── WS-E REST ──┐
|
|
├── WS-C docker ── │── WS-F MCP ── ┼── Phase 3 integration
|
|
└── WS-D auth ─────┘── WS-G OTel ──┘
|
|
```
|
|
|
|
Solo, do it sequentially: 0 → A → B → C → D → E → F → G → 3. With multiple contributors (or parallel agents), Phase 1's four tracks and Phase 2's three tracks are fully parallel — disjoint files, DTO contract fixed in Phase 0.
|
|
|
|
## Verification
|
|
|
|
1. `cargo build` inside `devenv shell`; unit tests for config precedence and DTO mapping.
|
|
2. Run locally: `cargo run -- serve --bind 127.0.0.1`; create a token via `cargo run -- token create`.
|
|
3. REST: `curl -H "Authorization: Bearer …" localhost:9109/api/v1/systemd/units?state=failed`, SSE follow endpoints via `curl -N`.
|
|
4. MCP: point a client at `http://localhost:9109/mcp` (e.g. `npx @modelcontextprotocol/inspector`) — verify tool list, call each tool, check auth rejection without the header.
|
|
5. OTel: run `otelcol` with a debug exporter, enable `[otel]`, confirm unit/container gauges appear on the poll interval and journal/docker log records flow with correct severity/attributes.
|
|
6. Negative tests: no token → 401; `auth.enabled=false` + `bind=0.0.0.0` → startup refuses; docker socket absent → clean error on docker endpoints, rest of API unaffected.
|