Container details now carry the image id and the HostConfig resource limits
(CPU cores and shares, memory and swap bytes, pids, device requests) plus sandbox flags
(privileged, read-only rootfs, network mode, restart policy), with null meaning unlimited. A new
GET /api/v1/docker/containers/{name}/stats returns one live sample: CPU percent
of one core, memory working set, processes, block IO and network bytes. Peers
proxy the new route.
Env, labels, command lines and mounts are deliberately excluded from the payload.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
266 lines
41 KiB
Markdown
266 lines
41 KiB
Markdown
# Architecture
|
|
|
|
skyg-linux-tracker (binary and CLI: `slinuxm`) is a read-only Linux system monitor in Rust, shipped as a single binary. It exposes systemd state, journald logs and Docker container state/logs over three surfaces that share one core: a **REST API**, an **MCP server** (Streamable HTTP) and **OpenTelemetry export**. Everything is read live from the source on every request. Nothing is collected or buffered, except a small SQLite database holding hashed API tokens.
|
|
|
|
User-facing docs start at [`README.md`](README.md) and the guides in [`docs/`](docs/README.md). Work tracking is in [`TODO.md`](TODO.md). Per-workstream design logs from the original build and Phase 5 are archived in [`docs/workstreams/`](docs/workstreams/).
|
|
|
|
## Requirements
|
|
|
|
- Single Rust binary, read-only against the host.
|
|
- MCP over Streamable HTTP with auth, mounted on the same axum server as REST.
|
|
- Config from a TOML file, `SKYG_*` env vars and CLI flags. Precedence: CLI > env > file > defaults.
|
|
- OTel: unit and container status as **metrics** (gauges), journal and docker logs as **OTel logs**, plus traces for the API itself.
|
|
- REST and MCP expose the same data. MCP adds agent-facing context (instructions, tool descriptions, prompts, resources).
|
|
- State: SQLite (`/var/lib/skyg-linux-tracker/db.sqlite`), access tokens only.
|
|
- Optional in-process TLS (rustls). No openssl anywhere in the dependency tree.
|
|
|
|
## Tech choices
|
|
|
|
| Area | Crate | Why |
|
|
| ----------- | ---------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
|
|
| HTTP | `axum` 0.8 + `tokio` | rmcp's HTTP transport is a Tower service that mounts onto an axum router |
|
|
| TLS | `rustls` (ring provider, pinned explicitly) + `tokio-rustls` + `hyper-util` | In-process HTTPS with SIGHUP reload; no openssl |
|
|
| MCP | `rmcp` 3.5 (official `modelcontextprotocol/rust-sdk`), features `server`, `macros`, `schemars`, `transport-streamable-http-server` | `StreamableHttpService` mounts at `/mcp`; tools via `#[tool]` macros; built-in Host/Origin validation |
|
|
| systemd | `zbus_systemd` (feature `systemd1`, tokio executor) | Pure-Rust generated bindings for `org.freedesktop.systemd1`. No C dependency |
|
|
| journald | `systemd` crate, `journal` + `systemd_v245` features (sd-journal FFI) | Only maintained option with live tail (`seek_tail`, `next`, `wait`) and namespaces. Links `libsystemd` (present on any systemd host) |
|
|
| Docker | `bollard` (unix socket only) | Async Docker/Podman API; `list_containers`, `inspect_container` (health), log streams |
|
|
| OTel | `opentelemetry`, `opentelemetry_sdk`, `opentelemetry-otlp` 0.33, `tracing-opentelemetry` | One OTLP config (endpoint, protocol, headers) drives all three signals. Metrics and logs are Stable, traces Beta |
|
|
| Auth DB | `sqlx` (sqlite, bundled) with **runtime** queries | Async, pure-Rust bundled SQLite, no build-time database (DEC-3) |
|
|
| Peer client | `reqwest` 0.13 (`rustls-no-provider`, ring passed in) + `webpki-roots` | Outbound calls to peers; no openssl, nothing installed process-wide (DEC-14) |
|
|
| OpenAPI | `utoipa` 6 + `schemars` | Generated REST and MCP description (DEC-15) |
|
|
| Config | `figment` (TOML + env) + `clap` | Layered config with clear precedence |
|
|
|
|
Rust edition 2021; release profile uses thin LTO and `strip`. Cargo.lock is committed. Local cargo builds link with lld via `.cargo/config.toml` (devenv provides it); the Nix package build excludes that file and keeps the default linker. The only dynamic dependencies are libc, libm, libgcc_s and libsystemd (check with `ldd`).
|
|
|
|
## Repo layout
|
|
|
|
```
|
|
Cargo.toml, Cargo.lock # bin `slinuxm`, lib `skyg_linux_tracker`
|
|
build.rs # stamps SLINUXM_VERSION (YYYY.MM.DD-<branch>-<sha>); logic in src/version.rs
|
|
devenv.nix # rust toolchain, pkg-config, libsystemd, otelcol; scripts build/run/smoke/token
|
|
flake.nix, nix/ # package.nix, module.nix (NixOS), tests/vm.nix (VM test)
|
|
config.example.toml # every key with its default and env name
|
|
migrations/0001_tokens.sql # embedded sqlx migration
|
|
systemd/skyg-linux-tracker.service # hardened example unit
|
|
dev/otelcol.yaml # local collector with debug exporter (`devenv up`)
|
|
examples/ # systemd_dump, journal_tail, docker_ps: run one source against the live host
|
|
docs/openapi.json # generated OpenAPI 3.1 doc (with peers), guarded by a drift test
|
|
tests/ # integration.rs, tls.rs, plain.rs, binary_smoke.rs, peers.rs, peers_config.rs, peers_registry.rs, mcp_peers.rs, openapi.rs, common/
|
|
src/
|
|
main.rs # clap CLI: serve | token create|list|revoke | peer list|check | openapi [--peers], --print-config
|
|
lib.rs
|
|
cli.rs # `slinuxm openapi`
|
|
openapi.rs # builds the OpenAPI document (route and CLI share it)
|
|
docs.rs # Redoc viewer at /docs; assets (page, loader, vendored Redoc 2.5.4 and its licenses) in src/docs/
|
|
config.rs # figment layering + validation
|
|
error.rs # AppError -> HTTP status + JSON body, and -> MCP ErrorData
|
|
state.rs # AppState: sources (Arc<dyn ...>), optional DB pool, config, CancellationToken
|
|
server.rs # router assembly, accept loop, graceful shutdown
|
|
tls.rs # rustls config, cert/key load and SIGHUP reload
|
|
db.rs # sqlx pool, WAL + busy_timeout, migrations, open-or-create (dir 0700, file 0600)
|
|
auth/ # bearer middleware (mod.rs), token CRUD (tokens.rs)
|
|
sources/ # the core: one trait per domain, live reads only
|
|
mod.rs # traits + shared DTOs (UnitSummary, TimerInfo, JournalEntry, ContainerSummary, ...)
|
|
systemd.rs, journal.rs, docker.rs # real implementations
|
|
fake.rs # in-memory fakes used by handler and tool tests
|
|
peers/ # consolidator: config.rs, secrets.rs (token loader), registry.rs, health.rs, client.rs (reqwest transport), proxy.rs, transport.rs (trait), types.rs, error.rs, cli.rs, testing.rs (fakes)
|
|
api/ # REST layer: systemd.rs, journal.rs, docker.rs, ping.rs, peers.rs, mod.rs
|
|
mcp/ # TrackerService (mod.rs), prompts.rs, resources.rs
|
|
telemetry/ # mod.rs (providers, guard, peer request span), metrics_export.rs, logs_export.rs
|
|
```
|
|
|
|
The key structural decision: **`sources/` is the only place that talks to D-Bus, journald or Docker.** `api/`, `mcp/` and `telemetry/` are thin projections over it, which makes "MCP and REST expose the same data" true by construction. `peers/` is the one outbound HTTP client; the registry exists only when `[peers] enabled = true`, and REST and MCP share its proxy functions. Sources are traits held as `Arc<dyn ...>` in `AppState` (DEC-2), so every handler and tool test runs against fakes without systemd or Docker. `server.rs` is the only place the surfaces meet.
|
|
|
|
## Configuration
|
|
|
|
See [`config.example.toml`](config.example.toml); every value shown is the default. Sections: `[server]` (`bind` 127.0.0.1, `port` 1984, `allowed_hosts`, `allowed_origins`, `[server.tls]`), `[auth]` (`enabled`, `db_path`), `[otel]` (`enabled`, `endpoint`, `protocol`, `service_name`, `export_unit_metrics`, `export_logs`, `metrics_interval_secs`, `[otel.headers]`, `[otel.logs]`, `[otel.metrics]` allowlists), `[sources]` (`docker_socket`, `journal_namespace`) and `[limits]` (`max_lines`, `max_tail`, `max_follow_secs`) and `[peers]` (off by default; see Consolidator).
|
|
|
|
Precedence: defaults < config file (`--config`, default `/etc/skyg-linux-tracker.toml`) < `SKYG_*` env (nesting with `__`, e.g. `SKYG_SERVER__PORT`) < CLI flags. `slinuxm --print-config` prints the effective config with secrets redacted. Validation rejects `auth.enabled = false` with a non-loopback `bind`, `port = 0`, and incomplete TLS settings.
|
|
|
|
## Sources (live reads only)
|
|
|
|
- **systemd** (`sources/systemd.rs`): one shared system-bus `zbus::Connection`. `list_units(state)` uses `ListUnits`. `list_timers()` lists `*.timer` units (`ListUnitsByPatterns`) and reads the `Timer` interface on each (`NextElapseUSecRealtime`, `LastTriggerUSec`, `Unit`); `0` or `u64::MAX` means not scheduled. `unit_status(name)` uses `GetUnit` only, never `LoadUnit`, so an unloaded unit is NotFound. `Unit` properties are read for every unit, `Service` properties (`ExecMainStatus`, `NRestarts`) only for `.service` units, so those DTO fields are `Option`. Unit names are validated. No caching.
|
|
- **journal** (`sources/journal.rs`): all sd-journal access runs inside `spawn_blocking` (a `Journal` is neither `Send` nor `Sync` and is opened and kept in the blocking closure). `query(filter)` seeks backward N entries (unit, priority <= N, `lines` capped by `limits.max_lines`, returned oldest first). `follow(filter)` is a blocking `wait()` loop with a short timeout that feeds an mpsc channel and exits when the receiver is dropped or the shutdown token fires. Binary `MESSAGE` fields are mapped lossily. `sources.journal_namespace` opens a journal namespace.
|
|
- **docker** (`sources/docker.rs`): lazily constructed `bollard::Docker` on `sources.docker_socket`. A missing or denied socket never fails startup; each call maps it to `AppError::Unavailable` (HTTP 503 on docker routes only). `ps()` lists containers and inspects them concurrently (bounded) for health, uptime, restart count and image. `container_status()` adds the sandbox facts a frozen box otherwise hides: `image_id`, and `resources` (CPU cores, memory and swap limits, pids limit, privileged, read-only rootfs, restart policy, network mode, device requests such as a CDI GPU). Those come from `HostConfig` only — `Env`, `Labels`, `Cmd`, `Args` and mounts are never copied, so no credential can reach the API. `stats(name)` takes one `/stats` sample (`stream=false`, `one-shot=true`) and derives CPU percent from the container/system deltas against `online_cpus`, a working set (usage minus page cache) and IO counters. `logs(name, follow, tail, since)` is a stream that stops on drop or shutdown; `tail` is capped by `limits.max_tail`.
|
|
|
|
## REST API
|
|
|
|
All under `/api/v1`, GET only, behind the auth middleware when enabled. Routes use axum 0.8 `{name}` syntax. Errors are `{"error":{"code","message"}}`.
|
|
|
|
| Endpoint | Source |
|
|
| ------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------- |
|
|
| `GET /api/v1/systemd/units?state=failed` | `list_units()` |
|
|
| `GET /api/v1/systemd/units/{name}` | `unit_status()` |
|
|
| `GET /api/v1/systemd/timers` | `list_timers()` |
|
|
| `GET /api/v1/journal?unit=&priority=&lines=` | journal snapshot |
|
|
| `GET /api/v1/journal/follow?unit=&priority=` (SSE, event `journal`) | journal `follow()` |
|
|
| `GET /api/v1/docker/ps` | `ps()` |
|
|
| `GET /api/v1/docker/containers/{name}` | `container_status()` |
|
|
| `GET /api/v1/docker/containers/{name}/stats` | `stats()` — one live sample (CPU, memory, pids, block IO, network) |
|
|
| `GET /api/v1/docker/containers/{name}/logs?tail=&since=&follow=0\|1` (JSON, or SSE events `stdout`/`stderr`) | `logs()` |
|
|
| `GET /api/v1/ping` | authenticated liveness, used by a consolidator's API check |
|
|
| `GET /api/v1/openapi.json` | generated OpenAPI 3.1 document; peer paths and tools only when peers are enabled |
|
|
| `GET /api/v1/peers`, `GET /api/v1/peers/{peer}` | peer status (only with `peers.enabled`) |
|
|
| `GET /api/v1/peers/{peer}/{*path}` | allowlisted GET pass-through to the peer; the response carries `x-slinuxm-peer` (only with `peers.enabled`) |
|
|
| `GET /healthz` | public liveness, `{"status":"ok","version":...}` |
|
|
| `GET /docs`, `GET /docs/redoc.standalone.js` | public static Redoc viewer of the OpenAPI document; the script is embedded in the binary (DEC-16) |
|
|
|
|
SSE streams (`axum::response::Sse` with keep-alive) read from the mpsc channels of `sources/` and end on client disconnect or shutdown.
|
|
|
|
## MCP server
|
|
|
|
`rmcp`'s `StreamableHttpService` is a Tower service nested at `/mcp` on the protected router (`mcp::service(state)`). One per-session `TrackerService` shares `AppState`. Auth failures are HTTP 401 with `WWW-Authenticate: Bearer` (DEC-4), the same middleware as REST.
|
|
|
|
Eight tools (ten with `peers.enabled`, below), each a thin wrapper over `sources/` with `schemars` parameter schemas: `list_systemd_units { state? }`, `get_systemd_unit { name }`, `list_systemd_timers`, `get_journal_logs { unit?, priority?, lines }`, `tail_journal { unit, seconds }`, `list_containers`, `get_container { name }`, `get_container_logs { name, tail }`.
|
|
|
|
- With `peers.enabled` there are ten: `list_peers` and `get_peer { name }` are added, each of the eight takes an optional `peer` argument that reads from that peer through this instance, and the instructions gain a peers paragraph. The tool-list snapshot test guards both tables.
|
|
- `tail_journal` is a bounded follow: it clamps `seconds` to `limits.max_follow_secs`, stops at an entry cap and returns what it collected. There are no infinite MCP streams.
|
|
- Log tools return `{entries, truncated, note}` capped at 256 KiB. Source errors come back as `CallToolResult` errors with an actionable message. Log content is documented as untrusted data.
|
|
- Agent context: `ServerHandler` **instructions** (start with failed units or unhealthy containers, then pull logs, use timers to explain jobs), rich tool descriptions (systemd states, health states, journal priorities), **prompts** `diagnose_unit { name }` and `container_health_report`, **resources** `systemd://units/failed` and `docker://containers` plus templates `systemd://unit/{name}` and `docker://container/{name}`. Resources are list/read, resolved live, not subscribable (DEC-5).
|
|
- Host/Origin validation uses rmcp's own `with_allowed_hosts` / `with_allowed_origins` (DEC-8). Requests with no `Origin` pass, so the bearer token is the real gate.
|
|
|
|
## Server wiring (`server.rs`, `main.rs`)
|
|
|
|
```rust
|
|
let protected = Router::new()
|
|
.nest("/api/v1", api::router(state))
|
|
.nest_service("/mcp", mcp::service(state))
|
|
.layer(auth::layer(state)); // bearer auth for API + MCP only
|
|
let app = public_router_with_healthz // /healthz and the static /docs viewer stay unauthenticated
|
|
.merge(protected)
|
|
.layer(TraceLayer::new_for_http());
|
|
```
|
|
|
|
Startup order: config, telemetry init, DB open (when auth is on), sources and `AppState`, exporter tasks, server. On SIGINT/SIGTERM the shared `CancellationToken` ends SSE streams and background tasks, graceful serve drains (bounded 10 s), exporters are awaited (bounded 10 s), the telemetry guard flushes OTel and the pool closes. The unit uses `TimeoutStopSec=30`. With TLS and plain HTTP alike the listener is HTTP/1.1, shares one accept loop, caps concurrent connections at 1024 and closes connections that send no request header within 30 s. With `auth.enabled = false` (loopback only) REST requests get a Host allowlist check (403 otherwise).
|
|
|
|
## TLS
|
|
|
|
Optional and off by default (DEC-12). With `server.tls.enabled` the server serves REST and `/mcp` over HTTPS only, using a PEM chain and key. Both files are validated at startup. SIGHUP re-reads them; a bad or mismatched pair keeps the old certificate and logs an error, established connections are never dropped. The default MCP Origin allowlist switches to `https://`. A non-loopback bind without TLS logs a startup warning but is not refused. Operator-facing details: [`docs/tls.md`](docs/tls.md).
|
|
|
|
## OpenTelemetry (`telemetry/`)
|
|
|
|
One OTLP exporter config (grpc or http/protobuf, optional headers) drives three signals. The resource carries `service.name`, `service.version` and `host.name`. OTLP is push-based, so this is the one place that polls on a timer (metrics) or fans out continuously (logs); it reuses the same `sources/` functions as the other surfaces.
|
|
|
|
1. **Traces**: a `tracing` subscriber (fmt to stderr) plus the `tracing-opentelemetry` layer when enabled, and `tower-http` `TraceLayer` for spans per HTTP/MCP request.
|
|
2. **Metrics**: `metrics_export.rs` polls every `metrics_interval_secs` into a snapshot. All instruments are **observable gauges (f64)** reading the latest snapshot, so a vanished unit or container stops being reported instead of keeping a stale value. A failed or timed-out section keeps its previous values.
|
|
|
|
| Metric | Unit | Attributes |
|
|
| ---------------------------------------------------------------------------------------------- | ----------- | --------------------------------------------------------- |
|
|
| `systemd.unit.active` (0/1) | `1` | `systemd.unit`, `systemd.sub_state`, `systemd.load_state` |
|
|
| `systemd.unit.failed` (0/1) | `1` | `systemd.unit` |
|
|
| `systemd.unit.restarts` (services only) | `{restart}` | `systemd.unit` |
|
|
| `systemd.timer.next_elapse_timestamp_seconds` (absolute unix time; unscheduled timers omitted) | `s` | `systemd.unit` |
|
|
| `docker.container.running` (0/1) | `1` | `container.name`, `container.image.name` |
|
|
| `docker.container.healthy` (0/1, only when a healthcheck exists) | `1` | `container.name` |
|
|
| `docker.container.uptime_seconds` | `s` | `container.name` |
|
|
| `docker.container.restarts` | `{restart}` | `container.name` |
|
|
|
|
Scope: an empty `otel.metrics.units` exports `*.service` and `*.timer` only (DEC-7); a non-empty list is a glob allowlist. An empty `otel.metrics.containers` exports all containers.
|
|
|
|
3. **Peer gauges** (only with `peers.enabled`): `peer.up` (`peer.name`, `peer.check`), `peer.check.duration_seconds`, `peer.last_change_timestamp_seconds` and `peer.api.auth_ok` (`peer.name`). They read the in-memory registry snapshot at each collection, so peers added or removed by a SIGHUP reload appear without re-registering; disabled peers are left out. Outbound peer requests run in a `peer.request` span exported as a client span (`otel.kind = client`) and carry `traceparent`. Metrics need `otel.enabled` and `otel.export_unit_metrics`.
|
|
4. **Logs**: when `otel.export_logs = true`, `logs_export.rs` runs a journal follow plus per-container docker follow tasks, reconciled against the Docker events stream (start/die/destroy) with backoff, so containers that come and go are tracked. Journal priority maps to OTel severity (emerg/alert/crit to FATAL, err ERROR, warning WARN, notice/info INFO, debug DEBUG); docker stderr is WARN, stdout INFO. Attributes: `systemd.unit`, `log.syslog.identifier`, `process.pid`, `container.name`, `log.iostream`. `otel.logs.journal_units` / `docker_containers` narrow the scope (empty = all).
|
|
|
|
Shutdown cancels the tasks and flushes all providers.
|
|
|
|
## Consolidator (`peers/`)
|
|
|
|
Opt-in (`[peers] enabled = true`): an instance reads other `slinuxm` instances declared in config. Without it there is no registry, no background task, no peer route or gauge, and 8 MCP tools. The registry keeps peer status in memory; two checks per peer (TCP connect, authenticated `GET /api/v1/ping`) run in background loops and SIGHUP reloads the peer list in place. Operator docs: [`docs/consolidator.md`](docs/consolidator.md). Decisions: DEC-13, DEC-14.
|
|
|
|
## OpenAPI (`openapi.rs`)
|
|
|
|
`openapi::document(peers)` is the single builder behind `GET /api/v1/openapi.json` and `slinuxm openapi [--peers]`. `docs/openapi.json` is the `--peers` output. See DEC-15.
|
|
|
|
`docs.rs` serves the Redoc viewer (`/docs`, `/docs/redoc.standalone.js`) from assets under `src/docs/` embedded with `include_str!`/`include_bytes!`. See DEC-16.
|
|
|
|
## Auth and state (`auth/`, `db.rs`)
|
|
|
|
- SQLite at `auth.db_path` (default `/var/lib/skyg-linux-tracker/db.sqlite`). A server-created directory is 0700 and the file 0600; an existing directory is never chmod'd (a group/world-accessible one only logs a warning). WAL mode plus `busy_timeout`, because the token CLI and the server share the DB. Under systemd, `StateDirectory=` creates the directory with the right owner.
|
|
- Table `tokens(id, name UNIQUE, token_hash UNIQUE, created_at, revoked_at, last_used_at)`. Tokens are 32 random bytes with the `slinuxm_` prefix; only the **sha256 hash** is stored and the plaintext is printed once. No bootstrap or declarative tokens (DEC-11).
|
|
- Middleware: sha256 of the presented bearer token, indexed lookup of non-revoked tokens (no separate constant-time compare is needed for a high-entropy token's hash). Failure is 401 with `WWW-Authenticate: Bearer` and the JSON error body; unmatched paths also return 401 so route existence is not disclosed. Revocation is immediate.
|
|
- The token CLI refuses to run when the effective uid (root included) does not own the DB file or, if absent, its state directory, and prints the matching `sudo -u <owner> slinuxm token ...` command (DEC-10). Token commands do not initialise telemetry.
|
|
|
|
## Build, packaging and deployment
|
|
|
|
- **devenv.nix**: Rust toolchain, `pkg-config`, `systemd` (libsystemd headers for the journal FFI), `opentelemetry-collector-contrib`; scripts `build`, `run`, `smoke`, `token`; `devenv up` runs the local collector.
|
|
- **Single binary**: pure Rust except libsystemd and bundled SQLite. rustls only; `cargo tree -i openssl` must be empty. The release binary was 19 MB when measured at P3.
|
|
- **systemd unit** (`systemd/skyg-linux-tracker.service`): static `User=skyg` (not `DynamicUser`, so `sudo -u skyg slinuxm token ...` reaches the same DB), `StateDirectory=` mode 0700, `SupplementaryGroups=systemd-journal`, `docker` group commented out (DEC-6), plus `ProtectSystem=strict`, `NoNewPrivileges`, syscall filter and related hardening. `systemd-analyze security --offline` scores 1.5 OK.
|
|
- **Nix** (`flake.nix`, `nix/`): `packages.default` (`buildRustPackage` from `Cargo.lock`; `cmake` is needed for `aws-lc-sys`, a transitive dependency; the pinned crypto provider for our own TLS is ring) and `nixosModules.default` under `services.skyg-linux-tracker` (user, `StateDirectory`, journal group, the same hardening as the unit, `/etc/skyg-linux-tracker.toml`, system-wide `slinuxm`, firewall options, `LoadCredential` TLS, eval assertion that a non-loopback bind needs auth). `nix flake check` runs a two-machine VM test (plain and TLS).
|
|
|
|
## Decisions
|
|
|
|
All accepted. Rationale is kept; the build history is in [`docs/workstreams/`](docs/workstreams/).
|
|
|
|
| ID | Decision | Status |
|
|
| ------ | ------------------------------------------------- | ------------------------------------------ |
|
|
| DEC-1 | Binary and CLI are `slinuxm` | Decided 2026-10-15 |
|
|
| DEC-2 | Source traits plus in-memory fakes from the start | Decided 2026-10-15 |
|
|
| DEC-3 | sqlx runtime queries, no macros | Decided 2026-10-15 |
|
|
| DEC-4 | MCP auth failure is HTTP 401 | Decided 2026-10-15 |
|
|
| DEC-5 | MCP resources not subscribable in v1 | Decided 2026-10-15 |
|
|
| DEC-6 | Docker socket access model | Decided 2026-10-15 |
|
|
| DEC-7 | Default metrics scope | Decided 2026-10-15 |
|
|
| DEC-8 | Origin validation for `/mcp` via rmcp | Decided 2026-10-15 |
|
|
| DEC-9 | Port 1984 and loopback bind by default | Decided 2026-10-16 |
|
|
| DEC-10 | No CLI wrapper; the app guards the root mistake | Decided 2026-10-16 |
|
|
| DEC-11 | No declarative or bootstrap tokens in v1 | Decided 2026-10-16 |
|
|
| DEC-12 | TLS terminated in-process with rustls | Decided 2026-10-16 |
|
|
| DEC-13 | Outbound peer credentials are external files | Decided 2026-10-19 |
|
|
| DEC-14 | Consolidator (peers) design | Decided 2026-10-19, implemented 2026-10-20 |
|
|
| DEC-15 | Generated OpenAPI document | Decided 2026-10-20 |
|
|
| DEC-16 | Redoc viewer served by the binary | Decided 2026-10-20 |
|
|
| DEC-17 | Build-stamped version | Decided 2026-10-21 |
|
|
|
|
- **DEC-1 name.** The plan used three names (repo `linux-system-monitor`, service `skyg-linux-tracker`, CLI `skyg`). The binary, CLI and `[[bin]]` are `slinuxm`; tokens use the `slinuxm_` prefix. Unchanged: product name `skyg-linux-tracker`, unit `skyg-linux-tracker.service`, state dir `/var/lib/skyg-linux-tracker`, config `/etc/skyg-linux-tracker.toml`, env prefix `SKYG_`.
|
|
- **DEC-2 traits and fakes.** `SystemdSource`, `JournalSource`, `DockerSource` live in `src/sources/mod.rs`, fakes in `src/sources/fake.rs`, `AppState` holds `Arc<dyn ...>` (via `async-trait`). Surfaces can be built and tested without systemd or Docker, at the cost of about 100 extra lines.
|
|
- **DEC-3 runtime queries.** `sqlx::query!` macros need `DATABASE_URL` at build time or a committed `.sqlx/` cache. With one table and about five queries, runtime `query`/`query_as` plus tests gives the same safety with no build-time DB.
|
|
- **DEC-4 HTTP 401.** The MCP authorization spec puts auth at the HTTP layer and clients expect 401 with `WWW-Authenticate: Bearer`. One middleware gives the same response for `/api/v1` and `/mcp`; `AppError::Unauthorized` renders it.
|
|
- **DEC-5 no subscriptions.** Subscriptions need change detection, i.e. background polling, which conflicts with "live reads only". Resources are resolved live on each read.
|
|
- **DEC-6 Docker socket.** `docker` group membership is root-equivalent. The unit ships with `SupplementaryGroups=docker` commented out; the hardened setup points `sources.docker_socket` at a read-only socket proxy (see [docs/operations.md](docs/operations.md#docker-access)). No code change is needed either way. The NixOS module's `docker.enable` adds the group explicitly.
|
|
- **DEC-7 metrics scope.** "Empty allowlist = all units" would export every loaded unit (often 300+ including devices and mounts), so empty means `*.service` and `*.timer`.
|
|
- **DEC-8 Origin validation.** The P0 spike showed rmcp 3.5 validates both headers itself: `with_allowed_hosts` (default loopback names) and `with_allowed_origins` (default empty means off; requests without `Origin` always pass). No custom tower layer. Hosts: loopback names, `server.bind`, `server.allowed_hosts`. Origins: `http(s)://localhost:*`, `127.0.0.1:*`, `[::1]:*`, `<bind>:*` plus `server.allowed_origins`.
|
|
- **DEC-9 defaults.** Binary and NixOS module both default to port 1984 and `bind = "127.0.0.1"` (the binary default was 9109 until 2026-10-16). LAN exposure is an install-time choice: set `bind`, `openFirewall` or an interface-scoped rule, and `allowedHosts`.
|
|
- **DEC-10 no wrapper.** The plain `slinuxm` binary is on `PATH`; mint tokens with `sudo -u skyg-linux-tracker slinuxm token create <name>`. Running as root can create a root-owned state dir and WAL files that lock the service out, so the token commands refuse (see Auth and state).
|
|
- **DEC-11 tokens.** Inbound tokens only (outbound: DEC-13). Printed once, never stored in plaintext; no `bootstrapTokens` option and no import of pre-hashed tokens.
|
|
- **DEC-12 TLS.** Bearer tokens over plain HTTP are readable on a LAN, which is the intended deployment, so TLS is native (`server.tls.*`, SIGHUP reload so ACME/sops rotation needs no restart). A reverse proxy still works. Off by default; nothing changes when it is off.
|
|
|
|
- **DEC-13 outbound credentials.** To call a peer the consolidator must replay a plaintext bearer token, so DEC-11's hash-only storage cannot apply. `slinuxm` never persists a peer token: config holds only a `token_file` path (on NixOS an agenix path delivered via `LoadCredential`), read into memory with redacted `Debug` and re-read on each check so rotation needs no restart. It does not protect against root on the consolidator; mitigate with one read-only token per consolidator. Full text: [`docs/workstreams/P5-consolidator/decisions.md`](docs/workstreams/P5-consolidator/decisions.md). There is one token loader, `peers::secrets::load_token`, used by startup validation, `peer check` and every request: it accepts a regular file only, requires owner-only permissions (with an exemption for `$CREDENTIALS_DIRECTORY`) and rejects CR, LF and NUL so a token cannot split a header.
|
|
- **DEC-14 consolidator design.** Peers are declarative (config / NixOS module, SIGHUP reload, no DB table). Two checks per peer: TCP connect and an authenticated `GET /api/v1/ping` (401 reports `api.up=true, auth=rejected`; older peers fall back to `/healthz`, `auth=unknown`). TCP 10s/3s, API 30s/5s, two failures to flip down, state in memory. REST: status plus a GET-only allowlisted pass-through under `/api/v1/peers/{peer}/...`. MCP: optional `peer` argument plus `list_peers`/`get_peer` (10 tools only when `peers.enabled`). No aggregation and no chaining in v1. Peer failures reuse `Unavailable`; a peer's 401/403 is never surfaced as `Unauthorized`. Four OTel gauges. SSRF: config-only URLs, no redirects, `deny_cidrs` at connect time, webpki roots plus optional `ca_file`, no skip-verify, plain http to non-loopback needs `allow_plain_http`. Any valid inbound token may read peer data (no scopes).
|
|
- **HTTP client outcome.** `reqwest` 0.13 with `rustls-no-provider`; the ring provider and webpki roots are passed to the client explicitly, so nothing is installed process-wide and `cargo tree -i openssl` stays empty. Redirects are off and the system proxy is ignored.
|
|
- **SSRF at connect time.** The host is resolved once, every address is checked against `deny_cidrs` after `to_canonical()` (so `::ffff:169.254.169.254` matches the IPv4 range), one denied address refuses the whole name, and the checked set is pinned with `resolve_to_addrs`, which closes DNS rebinding. The same `deny_networks` / `denied` code runs at config validation and at connect time. The default `deny_cidrs` covers both families (`169.254.0.0/16`, `0.0.0.0/8`, `fe80::/10`, `::/128`, `fd00:ec2::254/128`); private LAN ranges stay allowed.
|
|
- **DEC-15 generated OpenAPI.** The document is generated so it cannot drift from the code. REST paths are declared on the handlers with `#[utoipa::path]` (utoipa 6, OpenAPI 3.1). DTO schemas are reflected from the existing `schemars::JsonSchema` derives through a bridge in `openapi.rs`, so one derive serves MCP and OpenAPI; only payloads that exist as `serde_json` literals (error body, health) are written by hand and compared with real output in tests. MCP is not REST, so the tools go in the `x-mcp-tools` extension, built from the tool router's own schemas. Peer paths and the two peer tools appear only when peers are enabled. `tests/openapi.rs` is the drift guard: every router route must be documented, `x-mcp-tools` must match `tools/list`, and `docs/openapi.json` must equal `slinuxm openapi --peers` (regenerate with `cargo run -- openapi --peers > docs/openapi.json`). The route is authenticated like the rest of `/api/v1`.
|
|
- **DEC-16 Redoc viewer.** `GET /docs` renders the generated document with Redoc, served by the binary. The Redoc 2.5.4 standalone bundle is vendored under `src/docs/` (with its licenses) and embedded, so the page makes no CDN or other outbound request. It is stored gzipped (`redoc.standalone.js.gz`, about 320 KiB, under the 512 KiB hygiene limit) and sent as is with `Content-Encoding: gzip`; the refresh recipe is in the `docs.rs` module doc. 2.x was chosen over 3.x because 3.x is an ES module with a telemetry option; the 2.x bundle is a plain script. The viewer routes are public because a browser navigation cannot carry a bearer header, but they are static and hold no host data: the page asks for a token, keeps it in `sessionStorage` and sends it only to `GET /api/v1/openapi.json`, which stays authenticated. The page's one inline script is admitted by a CSP `sha256` computed from the same constant, so the policy cannot drift (`default-src 'none'`, `connect-src 'self'`, `form-action 'none'`, `frame-ancestors 'none'`; `style-src 'unsafe-inline'` and `worker-src blob:` are needed by Redoc itself). Both routes are in the OpenAPI document, so the drift test covers them. Redoc does not render the `x-mcp-tools` extension; the tools are in the JSON only.
|
|
- **DEC-17 build-stamped version.** The runtime version is `YYYY.MM.DD-<branch>-<sha>`, computed by `build.rs` and exposed as `skyg_linux_tracker::VERSION` (`SLINUXM_VERSION`); `--version`, `/healthz`, `/api/v1/ping`, MCP server info and OTel `service.version` use it. `Cargo.toml` stays semver because zero-padded CalVer parts (`2026.03.05`) are not valid semver, and `docs/openapi.json` keeps `info.version` from `CARGO_PKG_VERSION`: a stamp that changes per commit would fail the byte-for-byte drift test on every commit. The string logic (`src/version.rs`) is std-only and compiled into both `build.rs` and the library, so one implementation is unit-tested. Every part can come from `SLINUXM_GIT_SHA`, `SLINUXM_GIT_BRANCH` and `SLINUXM_BUILD_DATE`, and `SLINUXM_VERSION` overrides the whole string; the Nix sandbox has no `.git`, so `flake.nix` passes the commit date (`lastModifiedDate`, which keeps the build reproducible) and short rev, with the placeholder branch `nix` because a flake does not expose its branch. `SOURCE_DATE_EPOCH` is ignored because Nix dev shells set it to 1980. A dirty tree appends `-dirty` to the sha, a small extension of the pattern. Because naming any `rerun-if-changed` path turns off cargo's default package-wide watch, `build.rs` also lists `src`, `Cargo.toml`, `Cargo.lock` and `build.rs`, and watches HEAD through `git rev-parse --git-path` so linked worktrees work. A peer's `peer_version` is an opaque string, so a consolidator shows each peer's own stamp.
|
|
|
|
## Design notes and gotchas
|
|
|
|
Corrections found in the original design review and still worth knowing when changing the code.
|
|
|
|
- **systemd has no `ListTimers`** (R1). Timers are `*.timer` units plus the `Timer` interface properties (see Sources).
|
|
- **`ExecMainStatus` and `NRestarts` are `Service` properties** (R2), absent for timers, sockets and mounts, hence `Option` DTO fields.
|
|
- **`/healthz` must stay public** (R3). The auth layer sits on the protected router only, merged with a public router.
|
|
- **axum 0.8 routes use `/{name}`** (R4). The old `:name` syntax panics at startup.
|
|
- **There are 8 MCP tools, or 10 with `peers.enabled`** (R5). Changing the count is a deliberate change; the tool-list snapshot test guards both tables.
|
|
- **Metric allowlists and `otel.headers` are config keys** (R6) and documented in `config.example.toml`.
|
|
- **No openssl** (R7). Gate: `cargo tree -i openssl` prints nothing.
|
|
- **Runtime sqlx queries** (R8, DEC-3).
|
|
- Blocking sd-journal work, snapshot or follow, always runs in `spawn_blocking` and never holds a `Journal` across `.await` (G1).
|
|
- Follow loops use `wait()` with a timeout and exit when the receiver closes or shutdown fires, otherwise threads leak when an SSE client disconnects (G2). Every stream and background task selects on the one `CancellationToken`, otherwise graceful shutdown waits forever on open SSE (G3).
|
|
- Docker containers come and go, so log fan-out reconciles from the events stream (G4). A missing Docker socket is a per-request 503, never a startup failure (G9).
|
|
- Request bounds are config-driven (`limits.*`) and unit/container names are validated (G7). Observable gauges fed from a snapshot avoid stale series (G11).
|
|
- `unit_status` uses `GetUnit` rather than `LoadUnit` so a read-only tool has no side effects.
|
|
- `rmcp` accepts requests without an `Origin` header, so the bearer token is the real gate. With `bind = 0.0.0.0`, LAN clients need their Host names in `server.allowed_hosts` (a startup warning is logged otherwise).
|
|
- The Docker group is root-equivalent even though our code only reads; prefer a read-only socket proxy.
|
|
|
|
## Verification
|
|
|
|
A release is verified by hand on a real host with these six checks (last full run: minipc1, 2026-10-19; #5 is still open as I6-5 in [`TODO.md`](TODO.md)). With peers enabled, also run two instances against each other, cross-check a real peer, and confirm the peer gauges in the collector; the last two are still open (M6 in `TODO.md`).
|
|
|
|
1. `cargo build` inside `devenv shell`; unit tests for config precedence and DTO mapping.
|
|
2. Run locally: `cargo run -- serve`; create a token with `cargo run -- token create <name>`.
|
|
3. REST: `curl -H "Authorization: Bearer ..." localhost:1984/api/v1/systemd/units?state=failed`; SSE follow endpoints via `curl -N`.
|
|
4. MCP: point a client at `http://localhost:1984/mcp` (for example `npx @modelcontextprotocol/inspector`), verify the tool list, call each tool, check rejection without the header.
|
|
5. OTel: run `otelcol` with a debug exporter (`devenv up`), enable `[otel]`, confirm unit/container gauges appear on the poll interval and journal/docker log records flow with correct severity and attributes.
|
|
6. Negative tests: no token gives 401; `auth.enabled = false` with `bind = 0.0.0.0` makes startup refuse; a missing docker socket gives a clean 503 on docker endpoints with the rest of the API unaffected; SIGTERM with an open SSE client exits promptly. Automated in `tests/integration.rs`.
|
|
|
|
For NixOS, the gate is `nix build` and `nix flake check` including the VM test.
|