Files
skyg-linux-tracker/ARCHITECTURE.md
T
2026-10-20 16:35:11 -04:00

31 KiB

Architecture

skyg-linux-tracker (binary and CLI: slinuxm) is a read-only Linux system monitor in Rust, shipped as a single binary. It exposes systemd state, journald logs and Docker container state/logs over three surfaces that share one core: a REST API, an MCP server (Streamable HTTP) and OpenTelemetry export. Everything is read live from the source on every request. Nothing is collected or buffered, except a small SQLite database holding hashed API tokens.

User-facing usage is in README.md. Work tracking is in TODO.md. Per-workstream design logs from the original build and Phase 5 are archived in docs/workstreams/.

Requirements

  • Single Rust binary, read-only against the host.
  • MCP over Streamable HTTP with auth, mounted on the same axum server as REST.
  • Config from a TOML file, SKYG_* env vars and CLI flags. Precedence: CLI > env > file > defaults.
  • OTel: unit and container status as metrics (gauges), journal and docker logs as OTel logs, plus traces for the API itself.
  • REST and MCP expose the same data. MCP adds agent-facing context (instructions, tool descriptions, prompts, resources).
  • State: SQLite (/var/lib/skyg-linux-tracker/db.sqlite), access tokens only.
  • Optional in-process TLS (rustls). No openssl anywhere in the dependency tree.

Tech choices

Area Crate Why
HTTP axum 0.8 + tokio rmcp's HTTP transport is a Tower service that mounts onto an axum router
TLS rustls (ring provider, pinned explicitly) + tokio-rustls + hyper-util In-process HTTPS with SIGHUP reload; no openssl
MCP rmcp 3.5 (official modelcontextprotocol/rust-sdk), features server, macros, schemars, transport-streamable-http-server StreamableHttpService mounts at /mcp; tools via #[tool] macros; built-in Host/Origin validation
systemd zbus_systemd (feature systemd1, tokio executor) Pure-Rust generated bindings for org.freedesktop.systemd1. No C dependency
journald systemd crate, journal + systemd_v245 features (sd-journal FFI) Only maintained option with live tail (seek_tail, next, wait) and namespaces. Links libsystemd (present on any systemd host)
Docker bollard (unix socket only) Async Docker/Podman API; list_containers, inspect_container (health), log streams
OTel opentelemetry, opentelemetry_sdk, opentelemetry-otlp 0.33, tracing-opentelemetry One OTLP config (endpoint, protocol, headers) drives all three signals. Metrics and logs are Stable, traces Beta
Auth DB sqlx (sqlite, bundled) with runtime queries Async, pure-Rust bundled SQLite, no build-time database (DEC-3)
Peer client reqwest 0.13 (rustls-no-provider, ring passed in) + webpki-roots Outbound calls to peers; no openssl, nothing installed process-wide (DEC-14)
OpenAPI utoipa 6 + schemars Generated REST and MCP description (DEC-15)
Config figment (TOML + env) + clap Layered config with clear precedence

Rust edition 2021; release profile uses LTO and strip. Cargo.lock is committed. The only dynamic dependencies are libc, libm, libgcc_s and libsystemd (check with ldd).

Repo layout

Cargo.toml, Cargo.lock       # bin `slinuxm`, lib `skyg_linux_tracker`
devenv.nix                   # rust toolchain, pkg-config, libsystemd, otelcol; scripts build/run/smoke/token
flake.nix, nix/              # package.nix, module.nix (NixOS), tests/vm.nix (VM test)
config.example.toml          # every key with its default and env name
migrations/0001_tokens.sql   # embedded sqlx migration
systemd/skyg-linux-tracker.service   # hardened example unit
dev/otelcol.yaml             # local collector with debug exporter (`devenv up`)
examples/                    # systemd_dump, journal_tail, docker_ps: run one source against the live host
docs/openapi.json            # generated OpenAPI 3.1 doc (with peers), guarded by a drift test
tests/                       # integration.rs, tls.rs, plain.rs, binary_smoke.rs, peers.rs, peers_config.rs, peers_registry.rs, mcp_peers.rs, openapi.rs, common/
src/
  main.rs         # clap CLI: serve | token create|list|revoke | peer list|check | openapi [--peers], --print-config
  lib.rs
  cli.rs          # `slinuxm openapi`
  openapi.rs      # builds the OpenAPI document (route and CLI share it)
  config.rs       # figment layering + validation
  error.rs        # AppError -> HTTP status + JSON body, and -> MCP ErrorData
  state.rs        # AppState: sources (Arc<dyn ...>), optional DB pool, config, CancellationToken
  server.rs       # router assembly, accept loop, graceful shutdown
  tls.rs          # rustls config, cert/key load and SIGHUP reload
  db.rs           # sqlx pool, WAL + busy_timeout, migrations, open-or-create (dir 0700, file 0600)
  auth/           # bearer middleware (mod.rs), token CRUD (tokens.rs)
  sources/        # the core: one trait per domain, live reads only
    mod.rs        # traits + shared DTOs (UnitSummary, TimerInfo, JournalEntry, ContainerSummary, ...)
    systemd.rs, journal.rs, docker.rs   # real implementations
    fake.rs       # in-memory fakes used by handler and tool tests
  peers/          # consolidator: config.rs, secrets.rs (token loader), registry.rs, health.rs, client.rs (reqwest transport), proxy.rs, transport.rs (trait), types.rs, error.rs, cli.rs, testing.rs (fakes)
  api/            # REST layer: systemd.rs, journal.rs, docker.rs, ping.rs, peers.rs, mod.rs
  mcp/            # TrackerService (mod.rs), prompts.rs, resources.rs
  telemetry/      # mod.rs (providers, guard, peer request span), metrics_export.rs, logs_export.rs

The key structural decision: sources/ is the only place that talks to D-Bus, journald or Docker. api/, mcp/ and telemetry/ are thin projections over it, which makes "MCP and REST expose the same data" true by construction. peers/ is the one outbound HTTP client; the registry exists only when [peers] enabled = true, and REST and MCP share its proxy functions. Sources are traits held as Arc<dyn ...> in AppState (DEC-2), so every handler and tool test runs against fakes without systemd or Docker. server.rs is the only place the surfaces meet.

Configuration

See config.example.toml; every value shown is the default. Sections: [server] (bind 127.0.0.1, port 1984, allowed_hosts, allowed_origins, [server.tls]), [auth] (enabled, db_path), [otel] (enabled, endpoint, protocol, service_name, export_unit_metrics, export_logs, metrics_interval_secs, [otel.headers], [otel.logs], [otel.metrics] allowlists), [sources] (docker_socket, journal_namespace) and [limits] (max_lines, max_tail, max_follow_secs) and [peers] (off by default; see Consolidator).

Precedence: defaults < config file (--config, default /etc/skyg-linux-tracker.toml) < SKYG_* env (nesting with __, e.g. SKYG_SERVER__PORT) < CLI flags. slinuxm --print-config prints the effective config with secrets redacted. Validation rejects auth.enabled = false with a non-loopback bind, port = 0, and incomplete TLS settings.

Sources (live reads only)

  • systemd (sources/systemd.rs): one shared system-bus zbus::Connection. list_units(state) uses ListUnits. list_timers() lists *.timer units (ListUnitsByPatterns) and reads the Timer interface on each (NextElapseUSecRealtime, LastTriggerUSec, Unit); 0 or u64::MAX means not scheduled. unit_status(name) uses GetUnit only, never LoadUnit, so an unloaded unit is NotFound. Unit properties are read for every unit, Service properties (ExecMainStatus, NRestarts) only for .service units, so those DTO fields are Option. Unit names are validated. No caching.
  • journal (sources/journal.rs): all sd-journal access runs inside spawn_blocking (a Journal is neither Send nor Sync and is opened and kept in the blocking closure). query(filter) seeks backward N entries (unit, priority <= N, lines capped by limits.max_lines, returned oldest first). follow(filter) is a blocking wait() loop with a short timeout that feeds an mpsc channel and exits when the receiver is dropped or the shutdown token fires. Binary MESSAGE fields are mapped lossily. sources.journal_namespace opens a journal namespace.
  • docker (sources/docker.rs): lazily constructed bollard::Docker on sources.docker_socket. A missing or denied socket never fails startup; each call maps it to AppError::Unavailable (HTTP 503 on docker routes only). ps() lists containers and inspects them concurrently (bounded) for health, uptime, restart count and image. logs(name, follow, tail, since) is a stream that stops on drop or shutdown; tail is capped by limits.max_tail.

REST API

All under /api/v1, GET only, behind the auth middleware when enabled. Routes use axum 0.8 {name} syntax. Errors are {"error":{"code","message"}}.

Endpoint Source
GET /api/v1/systemd/units?state=failed list_units()
GET /api/v1/systemd/units/{name} unit_status()
GET /api/v1/systemd/timers list_timers()
GET /api/v1/journal?unit=&priority=&lines= journal snapshot
GET /api/v1/journal/follow?unit=&priority= (SSE, event journal) journal follow()
GET /api/v1/docker/ps ps()
GET /api/v1/docker/containers/{name} container_status()
GET /api/v1/docker/containers/{name}/logs?tail=&since=&follow=0|1 (JSON, or SSE events stdout/stderr) logs()
GET /api/v1/ping authenticated liveness, used by a consolidator's API check
GET /api/v1/openapi.json generated OpenAPI 3.1 document; peer paths and tools only when peers are enabled
GET /api/v1/peers, GET /api/v1/peers/{peer} peer status (only with peers.enabled)
GET /api/v1/peers/{peer}/{*path} allowlisted GET pass-through to the peer; the response carries x-slinuxm-peer (only with peers.enabled)
GET /healthz public liveness, {"status":"ok","version":...}

SSE streams (axum::response::Sse with keep-alive) read from the mpsc channels of sources/ and end on client disconnect or shutdown.

MCP server

rmcp's StreamableHttpService is a Tower service nested at /mcp on the protected router (mcp::service(state)). One per-session TrackerService shares AppState. Auth failures are HTTP 401 with WWW-Authenticate: Bearer (DEC-4), the same middleware as REST.

Eight tools (ten with peers.enabled, below), each a thin wrapper over sources/ with schemars parameter schemas: list_systemd_units { state? }, get_systemd_unit { name }, list_systemd_timers, get_journal_logs { unit?, priority?, lines }, tail_journal { unit, seconds }, list_containers, get_container { name }, get_container_logs { name, tail }.

  • With peers.enabled there are ten: list_peers and get_peer { name } are added, each of the eight takes an optional peer argument that reads from that peer through this instance, and the instructions gain a peers paragraph. The tool-list snapshot test guards both tables.
  • tail_journal is a bounded follow: it clamps seconds to limits.max_follow_secs, stops at an entry cap and returns what it collected. There are no infinite MCP streams.
  • Log tools return {entries, truncated, note} capped at 256 KiB. Source errors come back as CallToolResult errors with an actionable message. Log content is documented as untrusted data.
  • Agent context: ServerHandler instructions (start with failed units or unhealthy containers, then pull logs, use timers to explain jobs), rich tool descriptions (systemd states, health states, journal priorities), prompts diagnose_unit { name } and container_health_report, resources systemd://units/failed and docker://containers plus templates systemd://unit/{name} and docker://container/{name}. Resources are list/read, resolved live, not subscribable (DEC-5).
  • Host/Origin validation uses rmcp's own with_allowed_hosts / with_allowed_origins (DEC-8). Requests with no Origin pass, so the bearer token is the real gate.

Server wiring (server.rs, main.rs)

let protected = Router::new()
    .nest("/api/v1", api::router(state))
    .nest_service("/mcp", mcp::service(state))
    .layer(auth::layer(state));            // bearer auth for API + MCP only
let app = public_router_with_healthz       // /healthz stays unauthenticated
    .merge(protected)
    .layer(TraceLayer::new_for_http());

Startup order: config, telemetry init, DB open (when auth is on), sources and AppState, exporter tasks, server. On SIGINT/SIGTERM the shared CancellationToken ends SSE streams and background tasks, graceful serve drains (bounded 10 s), exporters are awaited (bounded 10 s), the telemetry guard flushes OTel and the pool closes. The unit uses TimeoutStopSec=30. With TLS and plain HTTP alike the listener is HTTP/1.1, shares one accept loop, caps concurrent connections at 1024 and closes connections that send no request header within 30 s. With auth.enabled = false (loopback only) REST requests get a Host allowlist check (403 otherwise).

TLS

Optional and off by default (DEC-12). With server.tls.enabled the server serves REST and /mcp over HTTPS only, using a PEM chain and key. Both files are validated at startup. SIGHUP re-reads them; a bad or mismatched pair keeps the old certificate and logs an error, established connections are never dropped. The default MCP Origin allowlist switches to https://. A non-loopback bind without TLS logs a startup warning but is not refused. Operator-facing details: docs/tls.md.

OpenTelemetry (telemetry/)

One OTLP exporter config (grpc or http/protobuf, optional headers) drives three signals. The resource carries service.name, service.version and host.name. OTLP is push-based, so this is the one place that polls on a timer (metrics) or fans out continuously (logs); it reuses the same sources/ functions as the other surfaces.

  1. Traces: a tracing subscriber (fmt to stderr) plus the tracing-opentelemetry layer when enabled, and tower-http TraceLayer for spans per HTTP/MCP request.

  2. Metrics: metrics_export.rs polls every metrics_interval_secs into a snapshot. All instruments are observable gauges (f64) reading the latest snapshot, so a vanished unit or container stops being reported instead of keeping a stale value. A failed or timed-out section keeps its previous values.

    Metric Unit Attributes
    systemd.unit.active (0/1) 1 systemd.unit, systemd.sub_state, systemd.load_state
    systemd.unit.failed (0/1) 1 systemd.unit
    systemd.unit.restarts (services only) {restart} systemd.unit
    systemd.timer.next_elapse_timestamp_seconds (absolute unix time; unscheduled timers omitted) s systemd.unit
    docker.container.running (0/1) 1 container.name, container.image.name
    docker.container.healthy (0/1, only when a healthcheck exists) 1 container.name
    docker.container.uptime_seconds s container.name
    docker.container.restarts {restart} container.name

    Scope: an empty otel.metrics.units exports *.service and *.timer only (DEC-7); a non-empty list is a glob allowlist. An empty otel.metrics.containers exports all containers.

  3. Peer gauges (only with peers.enabled): peer.up (peer.name, peer.check), peer.check.duration_seconds, peer.last_change_timestamp_seconds and peer.api.auth_ok (peer.name). They read the in-memory registry snapshot at each collection, so peers added or removed by a SIGHUP reload appear without re-registering; disabled peers are left out. Outbound peer requests run in a peer.request span exported as a client span (otel.kind = client) and carry traceparent. Metrics need otel.enabled and otel.export_unit_metrics.

  4. Logs: when otel.export_logs = true, logs_export.rs runs a journal follow plus per-container docker follow tasks, reconciled against the Docker events stream (start/die/destroy) with backoff, so containers that come and go are tracked. Journal priority maps to OTel severity (emerg/alert/crit to FATAL, err ERROR, warning WARN, notice/info INFO, debug DEBUG); docker stderr is WARN, stdout INFO. Attributes: systemd.unit, log.syslog.identifier, process.pid, container.name, log.iostream. otel.logs.journal_units / docker_containers narrow the scope (empty = all).

Shutdown cancels the tasks and flushes all providers.

Consolidator (peers/)

Opt-in ([peers] enabled = true): an instance reads other slinuxm instances declared in config. Without it there is no registry, no background task, no peer route or gauge, and 8 MCP tools. The registry keeps peer status in memory; two checks per peer (TCP connect, authenticated GET /api/v1/ping) run in background loops and SIGHUP reloads the peer list in place. Operator docs: docs/consolidator.md. Decisions: DEC-13, DEC-14.

OpenAPI (openapi.rs)

openapi::document(peers) is the single builder behind GET /api/v1/openapi.json and slinuxm openapi [--peers]. docs/openapi.json is the --peers output. See DEC-15.

Auth and state (auth/, db.rs)

  • SQLite at auth.db_path (default /var/lib/skyg-linux-tracker/db.sqlite). A server-created directory is 0700 and the file 0600; an existing directory is never chmod'd (a group/world-accessible one only logs a warning). WAL mode plus busy_timeout, because the token CLI and the server share the DB. Under systemd, StateDirectory= creates the directory with the right owner.
  • Table tokens(id, name UNIQUE, token_hash UNIQUE, created_at, revoked_at, last_used_at). Tokens are 32 random bytes with the slinuxm_ prefix; only the sha256 hash is stored and the plaintext is printed once. No bootstrap or declarative tokens (DEC-11).
  • Middleware: sha256 of the presented bearer token, indexed lookup of non-revoked tokens (no separate constant-time compare is needed for a high-entropy token's hash). Failure is 401 with WWW-Authenticate: Bearer and the JSON error body; unmatched paths also return 401 so route existence is not disclosed. Revocation is immediate.
  • The token CLI refuses to run when the effective uid (root included) does not own the DB file or, if absent, its state directory, and prints the matching sudo -u <owner> slinuxm token ... command (DEC-10). Token commands do not initialise telemetry.

Build, packaging and deployment

  • devenv.nix: Rust toolchain, pkg-config, systemd (libsystemd headers for the journal FFI), opentelemetry-collector-contrib; scripts build, run, smoke, token; devenv up runs the local collector.
  • Single binary: pure Rust except libsystemd and bundled SQLite. rustls only; cargo tree -i openssl must be empty. The release binary was 19 MB when measured at P3.
  • systemd unit (systemd/skyg-linux-tracker.service): static User=skyg (not DynamicUser, so sudo -u skyg slinuxm token ... reaches the same DB), StateDirectory= mode 0700, SupplementaryGroups=systemd-journal, docker group commented out (DEC-6), plus ProtectSystem=strict, NoNewPrivileges, syscall filter and related hardening. systemd-analyze security --offline scores 1.5 OK.
  • Nix (flake.nix, nix/): packages.default (buildRustPackage from Cargo.lock; cmake is needed for aws-lc-sys, a transitive dependency; the pinned crypto provider for our own TLS is ring) and nixosModules.default under services.skyg-linux-tracker (user, StateDirectory, journal group, the same hardening as the unit, /etc/skyg-linux-tracker.toml, system-wide slinuxm, firewall options, LoadCredential TLS, eval assertion that a non-loopback bind needs auth). nix flake check runs a two-machine VM test (plain and TLS).

Decisions

All accepted. Rationale is kept; the build history is in docs/workstreams/.

ID Decision Status
DEC-1 Binary and CLI are slinuxm Decided 2026-10-15
DEC-2 Source traits plus in-memory fakes from the start Decided 2026-10-15
DEC-3 sqlx runtime queries, no macros Decided 2026-10-15
DEC-4 MCP auth failure is HTTP 401 Decided 2026-10-15
DEC-5 MCP resources not subscribable in v1 Decided 2026-10-15
DEC-6 Docker socket access model Decided 2026-10-15
DEC-7 Default metrics scope Decided 2026-10-15
DEC-8 Origin validation for /mcp via rmcp Decided 2026-10-15
DEC-9 Port 1984 and loopback bind by default Decided 2026-10-16
DEC-10 No CLI wrapper; the app guards the root mistake Decided 2026-10-16
DEC-11 No declarative or bootstrap tokens in v1 Decided 2026-10-16
DEC-12 TLS terminated in-process with rustls Decided 2026-10-16
DEC-13 Outbound peer credentials are external files Decided 2026-10-19
DEC-14 Consolidator (peers) design Decided 2026-10-19, implemented 2026-10-20
DEC-15 Generated OpenAPI document Decided 2026-10-20
  • DEC-1 name. The plan used three names (repo linux-system-monitor, service skyg-linux-tracker, CLI skyg). The binary, CLI and [[bin]] are slinuxm; tokens use the slinuxm_ prefix. Unchanged: product name skyg-linux-tracker, unit skyg-linux-tracker.service, state dir /var/lib/skyg-linux-tracker, config /etc/skyg-linux-tracker.toml, env prefix SKYG_.

  • DEC-2 traits and fakes. SystemdSource, JournalSource, DockerSource live in src/sources/mod.rs, fakes in src/sources/fake.rs, AppState holds Arc<dyn ...> (via async-trait). Surfaces can be built and tested without systemd or Docker, at the cost of about 100 extra lines.

  • DEC-3 runtime queries. sqlx::query! macros need DATABASE_URL at build time or a committed .sqlx/ cache. With one table and about five queries, runtime query/query_as plus tests gives the same safety with no build-time DB.

  • DEC-4 HTTP 401. The MCP authorization spec puts auth at the HTTP layer and clients expect 401 with WWW-Authenticate: Bearer. One middleware gives the same response for /api/v1 and /mcp; AppError::Unauthorized renders it.

  • DEC-5 no subscriptions. Subscriptions need change detection, i.e. background polling, which conflicts with "live reads only". Resources are resolved live on each read.

  • DEC-6 Docker socket. docker group membership is root-equivalent. The unit ships with SupplementaryGroups=docker commented out; the hardened setup points sources.docker_socket at a read-only socket proxy (see the README). No code change is needed either way. The NixOS module's docker.enable adds the group explicitly.

  • DEC-7 metrics scope. "Empty allowlist = all units" would export every loaded unit (often 300+ including devices and mounts), so empty means *.service and *.timer.

  • DEC-8 Origin validation. The P0 spike showed rmcp 3.5 validates both headers itself: with_allowed_hosts (default loopback names) and with_allowed_origins (default empty means off; requests without Origin always pass). No custom tower layer. Hosts: loopback names, server.bind, server.allowed_hosts. Origins: http(s)://localhost:*, 127.0.0.1:*, [::1]:*, <bind>:* plus server.allowed_origins.

  • DEC-9 defaults. Binary and NixOS module both default to port 1984 and bind = "127.0.0.1" (the binary default was 9109 until 2026-10-16). LAN exposure is an install-time choice: set bind, openFirewall or an interface-scoped rule, and allowedHosts.

  • DEC-10 no wrapper. The plain slinuxm binary is on PATH; mint tokens with sudo -u skyg-linux-tracker slinuxm token create <name>. Running as root can create a root-owned state dir and WAL files that lock the service out, so the token commands refuse (see Auth and state).

  • DEC-11 tokens. Inbound tokens only (outbound: DEC-13). Printed once, never stored in plaintext; no bootstrapTokens option and no import of pre-hashed tokens.

  • DEC-12 TLS. Bearer tokens over plain HTTP are readable on a LAN, which is the intended deployment, so TLS is native (server.tls.*, SIGHUP reload so ACME/sops rotation needs no restart). A reverse proxy still works. Off by default; nothing changes when it is off.

  • DEC-13 outbound credentials. To call a peer the consolidator must replay a plaintext bearer token, so DEC-11's hash-only storage cannot apply. slinuxm never persists a peer token: config holds only a token_file path (on NixOS an agenix path delivered via LoadCredential), read into memory with redacted Debug and re-read on each check so rotation needs no restart. It does not protect against root on the consolidator; mitigate with one read-only token per consolidator. Full text: docs/workstreams/P5-consolidator/decisions.md. There is one token loader, peers::secrets::load_token, used by startup validation, peer check and every request: it accepts a regular file only, requires owner-only permissions (with an exemption for $CREDENTIALS_DIRECTORY) and rejects CR, LF and NUL so a token cannot split a header.

  • DEC-14 consolidator design. Peers are declarative (config / NixOS module, SIGHUP reload, no DB table). Two checks per peer: TCP connect and an authenticated GET /api/v1/ping (401 reports api.up=true, auth=rejected; older peers fall back to /healthz, auth=unknown). TCP 10s/3s, API 30s/5s, two failures to flip down, state in memory. REST: status plus a GET-only allowlisted pass-through under /api/v1/peers/{peer}/.... MCP: optional peer argument plus list_peers/get_peer (10 tools only when peers.enabled). No aggregation and no chaining in v1. Peer failures reuse Unavailable; a peer's 401/403 is never surfaced as Unauthorized. Four OTel gauges. SSRF: config-only URLs, no redirects, deny_cidrs at connect time, webpki roots plus optional ca_file, no skip-verify, plain http to non-loopback needs allow_plain_http. Any valid inbound token may read peer data (no scopes).

    • HTTP client outcome. reqwest 0.13 with rustls-no-provider; the ring provider and webpki roots are passed to the client explicitly, so nothing is installed process-wide and cargo tree -i openssl stays empty. Redirects are off and the system proxy is ignored.
    • SSRF at connect time. The host is resolved once, every address is checked against deny_cidrs after to_canonical() (so ::ffff:169.254.169.254 matches the IPv4 range), one denied address refuses the whole name, and the checked set is pinned with resolve_to_addrs, which closes DNS rebinding. The same deny_networks / denied code runs at config validation and at connect time. The default deny_cidrs covers both families (169.254.0.0/16, 0.0.0.0/8, fe80::/10, ::/128, fd00:ec2::254/128); private LAN ranges stay allowed.
  • DEC-15 generated OpenAPI. The document is generated so it cannot drift from the code. REST paths are declared on the handlers with #[utoipa::path] (utoipa 6, OpenAPI 3.1). DTO schemas are reflected from the existing schemars::JsonSchema derives through a bridge in openapi.rs, so one derive serves MCP and OpenAPI; only payloads that exist as serde_json literals (error body, health) are written by hand and compared with real output in tests. MCP is not REST, so the tools go in the x-mcp-tools extension, built from the tool router's own schemas. Peer paths and the two peer tools appear only when peers are enabled. tests/openapi.rs is the drift guard: every router route must be documented, x-mcp-tools must match tools/list, and docs/openapi.json must equal slinuxm openapi --peers (regenerate with cargo run -- openapi --peers > docs/openapi.json). The route is authenticated like the rest of /api/v1.

Design notes and gotchas

Corrections found in the original design review and still worth knowing when changing the code.

  • systemd has no ListTimers (R1). Timers are *.timer units plus the Timer interface properties (see Sources).
  • ExecMainStatus and NRestarts are Service properties (R2), absent for timers, sockets and mounts, hence Option DTO fields.
  • /healthz must stay public (R3). The auth layer sits on the protected router only, merged with a public router.
  • axum 0.8 routes use /{name} (R4). The old :name syntax panics at startup.
  • There are 8 MCP tools, or 10 with peers.enabled (R5). Changing the count is a deliberate change; the tool-list snapshot test guards both tables.
  • Metric allowlists and otel.headers are config keys (R6) and documented in config.example.toml.
  • No openssl (R7). Gate: cargo tree -i openssl prints nothing.
  • Runtime sqlx queries (R8, DEC-3).
  • Blocking sd-journal work, snapshot or follow, always runs in spawn_blocking and never holds a Journal across .await (G1).
  • Follow loops use wait() with a timeout and exit when the receiver closes or shutdown fires, otherwise threads leak when an SSE client disconnects (G2). Every stream and background task selects on the one CancellationToken, otherwise graceful shutdown waits forever on open SSE (G3).
  • Docker containers come and go, so log fan-out reconciles from the events stream (G4). A missing Docker socket is a per-request 503, never a startup failure (G9).
  • Request bounds are config-driven (limits.*) and unit/container names are validated (G7). Observable gauges fed from a snapshot avoid stale series (G11).
  • unit_status uses GetUnit rather than LoadUnit so a read-only tool has no side effects.
  • rmcp accepts requests without an Origin header, so the bearer token is the real gate. With bind = 0.0.0.0, LAN clients need their Host names in server.allowed_hosts (a startup warning is logged otherwise).
  • The Docker group is root-equivalent even though our code only reads; prefer a read-only socket proxy.

Verification

A release is verified by hand on a real host with these six checks (last full run: minipc1, 2026-10-19; #5 is still open as I6-5 in TODO.md). With peers enabled, also run two instances against each other, cross-check a real peer, and confirm the peer gauges in the collector; the last two are still open (M6 in TODO.md).

  1. cargo build inside devenv shell; unit tests for config precedence and DTO mapping.
  2. Run locally: cargo run -- serve; create a token with cargo run -- token create <name>.
  3. REST: curl -H "Authorization: Bearer ..." localhost:1984/api/v1/systemd/units?state=failed; SSE follow endpoints via curl -N.
  4. MCP: point a client at http://localhost:1984/mcp (for example npx @modelcontextprotocol/inspector), verify the tool list, call each tool, check rejection without the header.
  5. OTel: run otelcol with a debug exporter (devenv up), enable [otel], confirm unit/container gauges appear on the poll interval and journal/docker log records flow with correct severity and attributes.
  6. Negative tests: no token gives 401; auth.enabled = false with bind = 0.0.0.0 makes startup refuse; a missing docker socket gives a clean 503 on docker endpoints with the rest of the API unaffected; SIGTERM with an open SSE client exits promptly. Automated in tests/integration.rs.

For NixOS, the gate is nix build and nix flake check including the VM test.