16 KiB
skyg-linux-tracker — Design Doc
A read-only Linux system monitor in Rust, distributed as a single binary. Exposes systemd state, journald logs, and Docker container state/logs over three surfaces that share one core: a REST API, an MCP server (Streamable HTTP), and OpenTelemetry export. All data is fetched live from the source on every request — no metric/log collection or buffering, except a small SQLite database that stores API access tokens.
Confirmed requirements
- Single Rust binary, read-only against the host.
- MCP over Streamable HTTP with auth (mounts on the same axum server as the REST API).
- Config via TOML file + CLI flags + env vars (precedence: CLI > env > file > defaults).
- OTel: systemd unit / container status exported as metrics (gauges); journald / docker logs exported as OTel logs. Traces for the API itself are included since they come nearly free.
- MCP and REST expose the same data; MCP adds agent-facing context (tool descriptions, instructions, curated resources/prompts).
- State:
/var/lib/skyg-linux-tracker/db.sqlite(SQLite viasqlx), access tokens only.
Tech choices
| Area | Crate | Why |
|---|---|---|
| HTTP | axum + tokio |
Industry standard; rmcp's HTTP transport is a Tower service that mounts directly onto an axum router |
| MCP | rmcp (official modelcontextprotocol/rust-sdk), features server, macros, schemars, transport-streamable-http-server |
Official SDK; StreamableHttpService mounts at /mcp on the same axum router; tools via #[tool] macros |
| systemd | zbus_systemd (feature systemd1, tokio executor) |
Pure-Rust generated bindings for org.freedesktop.systemd1 — ListUnits, ListTimers, unit properties. No libsystemd C dependency |
| journald | systemd crate, journal feature (sd-journal FFI) |
Only maintained option supporting live tail (seek_tail + next + wait). Links libsystemd at build time (present on any systemd host; the devenv shell provides it) |
| Docker | bollard (rustls, no openssl) |
Async Docker/Podman API over the unix socket; list_containers, inspect_container (health), logs(follow=true) streams |
| OTel | opentelemetry, opentelemetry_sdk, opentelemetry-otlp (metrics + logs both Stable in current opentelemetry-rust; traces Beta) |
One OTLP exporter config (endpoint, protocol, headers) drives all three signals |
| Auth DB | sqlx + sqlite (bundled) |
Async, compile-time-checked queries, pure-Rust bundled SQLite — keeps the binary self-contained |
| Config | figment (TOML + env) + clap |
Layered config with clear precedence |
Repo layout
linux-system-monitor/
├── Cargo.toml
├── devenv.nix # + languages.rust, pkg-config/openssl/libsystemd build deps
├── devenv.yaml
├── config.example.toml
├── systemd/
│ └── skyg-linux-tracker.service # example unit file (hardened, read-only)
└── src/
├── main.rs # entry: config load → telemetry init → DB → server
├── config.rs # figment layering: defaults < TOML < env(SKYG_) < CLI
├── error.rs # thiserror AppError → axum + MCP error mappings
├── auth/
│ ├── mod.rs # bearer middleware + token verify (sha256 hash lookup)
│ └── tokens.rs # token CRUD against SQLite (`slinuxm token create/revoke/list` CLI)
├── db.rs # sqlx pool, embedded migrations, open-or-create
├── sources/ # === the core: one trait per domain, live reads only ===
│ ├── mod.rs
│ ├── systemd.rs # list_units(), list_timers(), unit_status(name)
│ ├── journal.rs # query(filter) snapshot, follow(filter) → mpsc stream
│ └── docker.rs # ps(), container_status(name), logs(name, follow)
├── api/ # REST layer (thin): DTOs + handlers calling sources/
│ ├── mod.rs # router assembly, auth middleware layer
│ ├── systemd.rs # GET /api/v1/systemd/{units,timers,units/:name}
│ ├── journal.rs # GET /api/v1/journal (snapshot), GET .../journal/follow (SSE)
│ └── docker.rs # GET /api/v1/docker/{ps,containers/:name,containers/:name/logs}
├── mcp/
│ ├── mod.rs # TrackerService: #[tool_router] tools wrapping sources/
│ └── prompts.rs # agent-facing prompts ("diagnose failing units", etc.)
└── telemetry/
├── mod.rs # OTel providers init (traces/metrics/logs), shutdown
├── metrics_export.rs # background loop → observable gauges
└── logs_export.rs # journal/docker follow → OTel log records fan-out
The key structural decision: sources/ is the only place that talks to D-Bus/journald/Docker. api/, mcp/, and telemetry/ are three thin projections over it, which is what makes "MCP and API expose the same data" true by construction.
Implementation detail
Config (config.rs, config.example.toml)
[server]
bind = "127.0.0.1" # set 0.0.0.0 only with auth enabled
port = 9109
[auth]
enabled = true # hard-required when bind != localhost
db_path = "/var/lib/skyg-linux-tracker/db.sqlite"
[otel]
enabled = false
endpoint = "http://localhost:4317" # OTLP gRPC; http://…:4318 for HTTP/proto
protocol = "grpc"
service_name = "skyg-linux-tracker" # OTel resource attr service.name
export_unit_metrics = true # poll loop, interval below
export_logs = false # journal/docker log fan-out to OTLP
metrics_interval_secs = 30
[otel.logs]
journal_units = [] # empty = all units
docker_containers = [] # empty = all containers
[sources]
docker_socket = "/var/run/docker.sock"
journal_namespace = "" # default journal
Precedence: built-in defaults < --config /etc/skyg-linux-tracker.toml < SKYG_* env vars < CLI flags. CLI subcommands: serve (default), token create|revoke|list.
Sources (live reads only)
- systemd (
sources/systemd.rs): one sharedzbus::Connectionto the system bus.list_units()→ManagerProxy::list_units();list_timers()→list_timers()(next-elapse realtime converted to RFC3339);unit_status(name)→load_unit+ properties (ActiveState,SubState,ExecMainStatus,NRestarts,ActiveEnterTimestamp). No caching — every call hits D-Bus. - journal (
sources/journal.rs):Journal::open(),match_add("_SYSTEMD_UNIT", …),seek_tail(). Snapshot queries seek backward N entries.follow()wrapsJournal::wait()in atokio::task::spawn_blockingbridge (sd-journal is blocking) feeding atokio::sync::mpsc— consumers (SSE endpoint, MCP tool, OTel log exporter) all read from channels. - docker (
sources/docker.rs): sharedbollard::Dockerover the unix socket.ps()→list_containers+inspect_containerper container forState.Health,StartedAt(→ uptime), restart count.logs(name, follow)→bollardlog stream (stdout,stderr,timestamps,since) as aStream<Item = LogOutput>.
REST API (api/)
All under /api/v1, all behind the auth middleware when enabled, all read-only (GET only):
| Endpoint | Source |
|---|---|
GET /api/v1/systemd/units?state=failed |
list_units() |
GET /api/v1/systemd/units/:name |
unit_status() |
GET /api/v1/systemd/timers |
list_timers() |
GET /api/v1/journal?unit=X&priority=err&lines=100 |
journal snapshot |
GET /api/v1/journal/follow?unit=X (SSE) |
journal follow() |
GET /api/v1/docker/ps |
ps() |
GET /api/v1/docker/containers/:name |
container_status() |
GET /api/v1/docker/containers/:name/logs?follow=1 (SSE) |
logs(follow) |
GET /healthz |
unauthenticated liveness |
Streaming endpoints use SSE (axum::response::Sse) over the mpsc channels from sources/.
MCP server (mcp/)
rmcp's StreamableHttpService is a Tower service mounted on the same axum router:
let mcp_service = StreamableHttpService::new(
|| Ok(TrackerService::new(app_state.clone())), // per-session handler, shared state
LocalSessionManager::default().into(),
StreamableHttpServerConfig::default().with_json_response(false),
);
let app = Router::new()
.nest("/api/v1", api_router)
.nest_service("/mcp", mcp_service)
.layer(auth_middleware); // same bearer auth covers both
Tools (each a thin wrapper over sources/, params derive schemars::JsonSchema):
list_systemd_units { state? }/get_systemd_unit { name }/list_systemd_timers {}get_journal_logs { unit?, priority?, lines }/tail_journal { unit, seconds }(bounded follow — collects for N seconds, returns; no infinite MCP streams)list_containers {}/get_container { name }/get_container_logs { name, tail }
The "bit more context for the agent" lives in:
ServerHandlerinstructions: how the tools relate ("checklist_systemd_units state=failedfirst, then pull journal logs for that unit").- Rich
#[tool(description = …)]strings explaining systemd states, health statuses, priorities. - Prompts (
mcp/prompts.rs):diagnose_unit { name },container_health_report {}— canned investigation workflows. - Resources:
systemd://units/failed,docker://containersas subscribable snapshot resources.
OpenTelemetry (telemetry/)
One OTLP exporter config → three signals:
- Traces (Beta in opentelemetry-rust):
tracing-opentelemetrylayer +tower-httpTraceLayer— spans per HTTP/MCP request. - Metrics (Stable):
metrics_export.rsruns a poll loop everymetrics_interval_secs:systemd.unit.active(u64 gauge 0/1, attrs:unit,sub_state,load_state)systemd.unit.failed(u64 0/1),systemd.unit.restarts(u64 counter-ish gauge)systemd.timer.next_elapse_seconds(f64, attrtimer)docker.container.running(0/1, attrscontainer,image),docker.container.healthy(0/1,health_statusattr),docker.container.uptime_seconds,docker.container.restarts- Configurable unit/container allowlists so cardinality stays bounded.
- Logs (Stable): when
otel.export_logs = true,logs_export.rsspawns journalfollow()+ per-container dockerlogs(follow=true)tasks, converts each entry to an OTelLogRecord(severity from journal priority / docker stderr-vs-stdout, attrs:unit,container,host) and emits through the SDKLoggerProvider.
"Same information via OTel" is satisfied by reusing the identical sources/ functions; the OTel path is the only one that does periodic push (metrics poll) and continuous fan-out (logs), because OTLP is a push model — everything else stays pull/live.
Auth & database (auth/, db.rs)
- SQLite at
/var/lib/skyg-linux-tracker/db.sqlite(dir created0700on first run;sqlxembedded migrations). - Table
tokens(id, name, token_hash TEXT UNIQUE, created_at, revoked_at)— only sha256 hashes stored, plaintext shown once atslinuxm token create. - Middleware: constant-time compare of
Authorization: Bearer <token>hash; 401 JSON body on REST, JSON-RPC error on MCP. auth.enabled = falseis rejected at startup ifbindisn't loopback.
Build & packaging
- devenv.nix: enable
languages.rust.enable = true; addpkg-config,systemd(libsystemd headers for the journal FFI) topackages. Add scripts:build,run,tokenwrappers. - Static-ish single binary: everything pure-Rust except libsystemd (acceptable — target hosts run systemd by definition) and bundled SQLite. rustls everywhere (no openssl).
- Example
systemd/skyg-linux-tracker.service: hardening flags (ProtectSystem=strict,NoNewPrivileges,ReadWritePaths=/var/lib/skyg-linux-tracker), supplementary groupsystemd-journal+ optionaldockergroup for socket access.
Phases & workstreams
Work is split so that after a small shared foundation, four tracks run in parallel, then three surfaces run in parallel. Each workstream lists its dependencies, the files it owns (to avoid merge conflicts), and its done-criteria.
Phase 0 — Foundation (blocking, single track)
Everything depends on this; keep it deliberately small.
| Task | Files |
|---|---|
| Cargo.toml + devenv.nix setup (rust toolchain, libsystemd build dep, all deps pinned) | Cargo.toml, devenv.nix |
Config layering (defaults < TOML < SKYG_* env < CLI), config.example.toml, clap CLI skeleton with serve + token subcommands |
src/config.rs, src/main.rs, config.example.toml |
Shared DTOs + error types: the response shapes (UnitStatus, TimerInfo, JournalEntry, ContainerSummary, …) every surface returns |
src/sources/mod.rs (DTO definitions), src/error.rs |
Gate: cargo run -- serve starts, loads layered config, prints effective config. No sources or surfaces yet.
Phase 1 — Sources + auth (4 parallel tracks)
Once DTOs exist, each track is fully independent.
WS-A: systemd source
- Files:
src/sources/systemd.rs - zbus connection +
list_units/list_timers/unit_statusreturning shared DTOs. - Done: an example binary calls each fn against the live bus and prints JSON.
WS-B: journal source
- Files:
src/sources/journal.rs query()snapshot +follow()→ mpsc (spawn_blocking bridge).- Done: example tails the journal for a unit and prints entries; snapshot respects
lines/priorityfilters.
WS-C: docker source
- Files:
src/sources/docker.rs ps()/container_status()/logs(follow).- Done: example lists containers with health/uptime; log follow streams to stdout.
WS-D: auth + DB
- Files:
src/db.rs,src/auth/ - sqlx pool + migration, token CRUD,
slinuxm token create/revoke/listCLI, bearer middleware (axum-compatible). - Done: token round-trips CLI → DB → middleware check; startup rejects
auth.enabled=false+ non-loopback bind.
Phase 2 — Surfaces (3 parallel tracks, after Phase 1)
Each surface consumes sources/ + the auth middleware; they never touch each other's files.
WS-E: REST API (src/api/)
- All endpoints from the table above, SSE streaming,
/healthz. - Depends on: A, B, C, D.
- Done: curl checks pass (Verification #3).
WS-F: MCP server (src/mcp/)
StreamableHttpServiceat/mcp, 9 tools, prompts, resources, server instructions.- Depends on: A, B, C, D.
- Done: MCP inspector checks pass (Verification #4).
WS-G: OTel export (src/telemetry/)
- Provider init (traces/metrics/logs), metrics poll loop, log fan-out tasks, graceful shutdown.
- Depends on: A, B, C.
- Done: otelcol debug exporter shows gauges + log records (Verification #5).
Phase 3 — Integration & hardening (single track, after Phase 2)
- Wire all surfaces into one axum server in
main.rs(the only place surfaces meet). - End-to-end negative tests (Verification #6), example systemd unit file, README/config docs, binary size +
lddcheck for the single-binary claim.
Dependency graph
Phase 0 (config + DTOs + skeleton)
├── WS-A systemd ──┐
├── WS-B journal ──┼── WS-E REST ──┐
├── WS-C docker ── │── WS-F MCP ── ┼── Phase 3 integration
└── WS-D auth ─────┘── WS-G OTel ──┘
Solo, do it sequentially: 0 → A → B → C → D → E → F → G → 3. With multiple contributors (or parallel agents), Phase 1's four tracks and Phase 2's three tracks are fully parallel — disjoint files, DTO contract fixed in Phase 0.
Verification
cargo buildinsidedevenv shell; unit tests for config precedence and DTO mapping.- Run locally:
cargo run -- serve --bind 127.0.0.1; create a token viacargo run -- token create. - REST:
curl -H "Authorization: Bearer …" localhost:9109/api/v1/systemd/units?state=failed, SSE follow endpoints viacurl -N. - MCP: point a client at
http://localhost:9109/mcp(e.g.npx @modelcontextprotocol/inspector) — verify tool list, call each tool, check auth rejection without the header. - OTel: run
otelcolwith a debug exporter, enable[otel], confirm unit/container gauges appear on the poll interval and journal/docker log records flow with correct severity/attributes. - Negative tests: no token → 401;
auth.enabled=false+bind=0.0.0.0→ startup refuses; docker socket absent → clean error on docker endpoints, rest of API unaffected.