Files
2026-10-21 00:12:33 -04:00

265 lines
41 KiB
Markdown

# Architecture
skyg-linux-tracker (binary and CLI: `slinuxm`) is a read-only Linux system monitor in Rust, shipped as a single binary. It exposes systemd state, journald logs and Docker container state/logs over three surfaces that share one core: a **REST API**, an **MCP server** (Streamable HTTP) and **OpenTelemetry export**. Everything is read live from the source on every request. Nothing is collected or buffered, except a small SQLite database holding hashed API tokens.
User-facing docs start at [`README.md`](README.md) and the guides in [`docs/`](docs/README.md). Work tracking is in [`TODO.md`](TODO.md). Per-workstream design logs from the original build and Phase 5 are archived in [`docs/workstreams/`](docs/workstreams/).
## Requirements
- Single Rust binary, read-only against the host.
- MCP over Streamable HTTP with auth, mounted on the same axum server as REST.
- Config from a TOML file, `SKYG_*` env vars and CLI flags. Precedence: CLI > env > file > defaults.
- OTel: unit and container status as **metrics** (gauges), journal and docker logs as **OTel logs**, plus traces for the API itself.
- REST and MCP expose the same data. MCP adds agent-facing context (instructions, tool descriptions, prompts, resources).
- State: SQLite (`/var/lib/skyg-linux-tracker/db.sqlite`), access tokens only.
- Optional in-process TLS (rustls). No openssl anywhere in the dependency tree.
## Tech choices
| Area | Crate | Why |
| ----------- | ---------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| HTTP | `axum` 0.8 + `tokio` | rmcp's HTTP transport is a Tower service that mounts onto an axum router |
| TLS | `rustls` (ring provider, pinned explicitly) + `tokio-rustls` + `hyper-util` | In-process HTTPS with SIGHUP reload; no openssl |
| MCP | `rmcp` 3.5 (official `modelcontextprotocol/rust-sdk`), features `server`, `macros`, `schemars`, `transport-streamable-http-server` | `StreamableHttpService` mounts at `/mcp`; tools via `#[tool]` macros; built-in Host/Origin validation |
| systemd | `zbus_systemd` (feature `systemd1`, tokio executor) | Pure-Rust generated bindings for `org.freedesktop.systemd1`. No C dependency |
| journald | `systemd` crate, `journal` + `systemd_v245` features (sd-journal FFI) | Only maintained option with live tail (`seek_tail`, `next`, `wait`) and namespaces. Links `libsystemd` (present on any systemd host) |
| Docker | `bollard` (unix socket only) | Async Docker/Podman API; `list_containers`, `inspect_container` (health), log streams |
| OTel | `opentelemetry`, `opentelemetry_sdk`, `opentelemetry-otlp` 0.33, `tracing-opentelemetry` | One OTLP config (endpoint, protocol, headers) drives all three signals. Metrics and logs are Stable, traces Beta |
| Auth DB | `sqlx` (sqlite, bundled) with **runtime** queries | Async, pure-Rust bundled SQLite, no build-time database (DEC-3) |
| Peer client | `reqwest` 0.13 (`rustls-no-provider`, ring passed in) + `webpki-roots` | Outbound calls to peers; no openssl, nothing installed process-wide (DEC-14) |
| OpenAPI | `utoipa` 6 + `schemars` | Generated REST and MCP description (DEC-15) |
| Config | `figment` (TOML + env) + `clap` | Layered config with clear precedence |
Rust edition 2021; release profile uses thin LTO and `strip`. Cargo.lock is committed. Local cargo builds link with lld via `.cargo/config.toml` (devenv provides it); the Nix package build excludes that file and keeps the default linker. The only dynamic dependencies are libc, libm, libgcc_s and libsystemd (check with `ldd`).
## Repo layout
```
Cargo.toml, Cargo.lock # bin `slinuxm`, lib `skyg_linux_tracker`
build.rs # stamps SLINUXM_VERSION (YYYY.MM.DD-<branch>-<sha>); logic in src/version.rs
devenv.nix # rust toolchain, pkg-config, libsystemd, otelcol; scripts build/run/smoke/token
flake.nix, nix/ # package.nix, module.nix (NixOS), tests/vm.nix (VM test)
config.example.toml # every key with its default and env name
migrations/0001_tokens.sql # embedded sqlx migration
systemd/skyg-linux-tracker.service # hardened example unit
dev/otelcol.yaml # local collector with debug exporter (`devenv up`)
examples/ # systemd_dump, journal_tail, docker_ps: run one source against the live host
docs/openapi.json # generated OpenAPI 3.1 doc (with peers), guarded by a drift test
tests/ # integration.rs, tls.rs, plain.rs, binary_smoke.rs, peers.rs, peers_config.rs, peers_registry.rs, mcp_peers.rs, openapi.rs, common/
src/
main.rs # clap CLI: serve | token create|list|revoke | peer list|check | openapi [--peers], --print-config
lib.rs
cli.rs # `slinuxm openapi`
openapi.rs # builds the OpenAPI document (route and CLI share it)
docs.rs # Redoc viewer at /docs; assets (page, loader, vendored Redoc 2.5.4 and its licenses) in src/docs/
config.rs # figment layering + validation
error.rs # AppError -> HTTP status + JSON body, and -> MCP ErrorData
state.rs # AppState: sources (Arc<dyn ...>), optional DB pool, config, CancellationToken
server.rs # router assembly, accept loop, graceful shutdown
tls.rs # rustls config, cert/key load and SIGHUP reload
db.rs # sqlx pool, WAL + busy_timeout, migrations, open-or-create (dir 0700, file 0600)
auth/ # bearer middleware (mod.rs), token CRUD (tokens.rs)
sources/ # the core: one trait per domain, live reads only
mod.rs # traits + shared DTOs (UnitSummary, TimerInfo, JournalEntry, ContainerSummary, ...)
systemd.rs, journal.rs, docker.rs # real implementations
fake.rs # in-memory fakes used by handler and tool tests
peers/ # consolidator: config.rs, secrets.rs (token loader), registry.rs, health.rs, client.rs (reqwest transport), proxy.rs, transport.rs (trait), types.rs, error.rs, cli.rs, testing.rs (fakes)
api/ # REST layer: systemd.rs, journal.rs, docker.rs, ping.rs, peers.rs, mod.rs
mcp/ # TrackerService (mod.rs), prompts.rs, resources.rs
telemetry/ # mod.rs (providers, guard, peer request span), metrics_export.rs, logs_export.rs
```
The key structural decision: **`sources/` is the only place that talks to D-Bus, journald or Docker.** `api/`, `mcp/` and `telemetry/` are thin projections over it, which makes "MCP and REST expose the same data" true by construction. `peers/` is the one outbound HTTP client; the registry exists only when `[peers] enabled = true`, and REST and MCP share its proxy functions. Sources are traits held as `Arc<dyn ...>` in `AppState` (DEC-2), so every handler and tool test runs against fakes without systemd or Docker. `server.rs` is the only place the surfaces meet.
## Configuration
See [`config.example.toml`](config.example.toml); every value shown is the default. Sections: `[server]` (`bind` 127.0.0.1, `port` 1984, `allowed_hosts`, `allowed_origins`, `[server.tls]`), `[auth]` (`enabled`, `db_path`), `[otel]` (`enabled`, `endpoint`, `protocol`, `service_name`, `export_unit_metrics`, `export_logs`, `metrics_interval_secs`, `[otel.headers]`, `[otel.logs]`, `[otel.metrics]` allowlists), `[sources]` (`docker_socket`, `journal_namespace`) and `[limits]` (`max_lines`, `max_tail`, `max_follow_secs`) and `[peers]` (off by default; see Consolidator).
Precedence: defaults < config file (`--config`, default `/etc/skyg-linux-tracker.toml`) < `SKYG_*` env (nesting with `__`, e.g. `SKYG_SERVER__PORT`) < CLI flags. `slinuxm --print-config` prints the effective config with secrets redacted. Validation rejects `auth.enabled = false` with a non-loopback `bind`, `port = 0`, and incomplete TLS settings.
## Sources (live reads only)
- **systemd** (`sources/systemd.rs`): one shared system-bus `zbus::Connection`. `list_units(state)` uses `ListUnits`. `list_timers()` lists `*.timer` units (`ListUnitsByPatterns`) and reads the `Timer` interface on each (`NextElapseUSecRealtime`, `LastTriggerUSec`, `Unit`); `0` or `u64::MAX` means not scheduled. `unit_status(name)` uses `GetUnit` only, never `LoadUnit`, so an unloaded unit is NotFound. `Unit` properties are read for every unit, `Service` properties (`ExecMainStatus`, `NRestarts`) only for `.service` units, so those DTO fields are `Option`. Unit names are validated. No caching.
- **journal** (`sources/journal.rs`): all sd-journal access runs inside `spawn_blocking` (a `Journal` is neither `Send` nor `Sync` and is opened and kept in the blocking closure). `query(filter)` seeks backward N entries (unit, priority <= N, `lines` capped by `limits.max_lines`, returned oldest first). `follow(filter)` is a blocking `wait()` loop with a short timeout that feeds an mpsc channel and exits when the receiver is dropped or the shutdown token fires. Binary `MESSAGE` fields are mapped lossily. `sources.journal_namespace` opens a journal namespace.
- **docker** (`sources/docker.rs`): lazily constructed `bollard::Docker` on `sources.docker_socket`. A missing or denied socket never fails startup; each call maps it to `AppError::Unavailable` (HTTP 503 on docker routes only). `ps()` lists containers and inspects them concurrently (bounded) for health, uptime, restart count and image. `logs(name, follow, tail, since)` is a stream that stops on drop or shutdown; `tail` is capped by `limits.max_tail`.
## REST API
All under `/api/v1`, GET only, behind the auth middleware when enabled. Routes use axum 0.8 `{name}` syntax. Errors are `{"error":{"code","message"}}`.
| Endpoint | Source |
| ------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------- |
| `GET /api/v1/systemd/units?state=failed` | `list_units()` |
| `GET /api/v1/systemd/units/{name}` | `unit_status()` |
| `GET /api/v1/systemd/timers` | `list_timers()` |
| `GET /api/v1/journal?unit=&priority=&lines=` | journal snapshot |
| `GET /api/v1/journal/follow?unit=&priority=` (SSE, event `journal`) | journal `follow()` |
| `GET /api/v1/docker/ps` | `ps()` |
| `GET /api/v1/docker/containers/{name}` | `container_status()` |
| `GET /api/v1/docker/containers/{name}/logs?tail=&since=&follow=0\|1` (JSON, or SSE events `stdout`/`stderr`) | `logs()` |
| `GET /api/v1/ping` | authenticated liveness, used by a consolidator's API check |
| `GET /api/v1/openapi.json` | generated OpenAPI 3.1 document; peer paths and tools only when peers are enabled |
| `GET /api/v1/peers`, `GET /api/v1/peers/{peer}` | peer status (only with `peers.enabled`) |
| `GET /api/v1/peers/{peer}/{*path}` | allowlisted GET pass-through to the peer; the response carries `x-slinuxm-peer` (only with `peers.enabled`) |
| `GET /healthz` | public liveness, `{"status":"ok","version":...}` |
| `GET /docs`, `GET /docs/redoc.standalone.js` | public static Redoc viewer of the OpenAPI document; the script is embedded in the binary (DEC-16) |
SSE streams (`axum::response::Sse` with keep-alive) read from the mpsc channels of `sources/` and end on client disconnect or shutdown.
## MCP server
`rmcp`'s `StreamableHttpService` is a Tower service nested at `/mcp` on the protected router (`mcp::service(state)`). One per-session `TrackerService` shares `AppState`. Auth failures are HTTP 401 with `WWW-Authenticate: Bearer` (DEC-4), the same middleware as REST.
Eight tools (ten with `peers.enabled`, below), each a thin wrapper over `sources/` with `schemars` parameter schemas: `list_systemd_units { state? }`, `get_systemd_unit { name }`, `list_systemd_timers`, `get_journal_logs { unit?, priority?, lines }`, `tail_journal { unit, seconds }`, `list_containers`, `get_container { name }`, `get_container_logs { name, tail }`.
- With `peers.enabled` there are ten: `list_peers` and `get_peer { name }` are added, each of the eight takes an optional `peer` argument that reads from that peer through this instance, and the instructions gain a peers paragraph. The tool-list snapshot test guards both tables.
- `tail_journal` is a bounded follow: it clamps `seconds` to `limits.max_follow_secs`, stops at an entry cap and returns what it collected. There are no infinite MCP streams.
- Log tools return `{entries, truncated, note}` capped at 256 KiB. Source errors come back as `CallToolResult` errors with an actionable message. Log content is documented as untrusted data.
- Agent context: `ServerHandler` **instructions** (start with failed units or unhealthy containers, then pull logs, use timers to explain jobs), rich tool descriptions (systemd states, health states, journal priorities), **prompts** `diagnose_unit { name }` and `container_health_report`, **resources** `systemd://units/failed` and `docker://containers` plus templates `systemd://unit/{name}` and `docker://container/{name}`. Resources are list/read, resolved live, not subscribable (DEC-5).
- Host/Origin validation uses rmcp's own `with_allowed_hosts` / `with_allowed_origins` (DEC-8). Requests with no `Origin` pass, so the bearer token is the real gate.
## Server wiring (`server.rs`, `main.rs`)
```rust
let protected = Router::new()
.nest("/api/v1", api::router(state))
.nest_service("/mcp", mcp::service(state))
.layer(auth::layer(state)); // bearer auth for API + MCP only
let app = public_router_with_healthz // /healthz and the static /docs viewer stay unauthenticated
.merge(protected)
.layer(TraceLayer::new_for_http());
```
Startup order: config, telemetry init, DB open (when auth is on), sources and `AppState`, exporter tasks, server. On SIGINT/SIGTERM the shared `CancellationToken` ends SSE streams and background tasks, graceful serve drains (bounded 10 s), exporters are awaited (bounded 10 s), the telemetry guard flushes OTel and the pool closes. The unit uses `TimeoutStopSec=30`. With TLS and plain HTTP alike the listener is HTTP/1.1, shares one accept loop, caps concurrent connections at 1024 and closes connections that send no request header within 30 s. With `auth.enabled = false` (loopback only) REST requests get a Host allowlist check (403 otherwise).
## TLS
Optional and off by default (DEC-12). With `server.tls.enabled` the server serves REST and `/mcp` over HTTPS only, using a PEM chain and key. Both files are validated at startup. SIGHUP re-reads them; a bad or mismatched pair keeps the old certificate and logs an error, established connections are never dropped. The default MCP Origin allowlist switches to `https://`. A non-loopback bind without TLS logs a startup warning but is not refused. Operator-facing details: [`docs/tls.md`](docs/tls.md).
## OpenTelemetry (`telemetry/`)
One OTLP exporter config (grpc or http/protobuf, optional headers) drives three signals. The resource carries `service.name`, `service.version` and `host.name`. OTLP is push-based, so this is the one place that polls on a timer (metrics) or fans out continuously (logs); it reuses the same `sources/` functions as the other surfaces.
1. **Traces**: a `tracing` subscriber (fmt to stderr) plus the `tracing-opentelemetry` layer when enabled, and `tower-http` `TraceLayer` for spans per HTTP/MCP request.
2. **Metrics**: `metrics_export.rs` polls every `metrics_interval_secs` into a snapshot. All instruments are **observable gauges (f64)** reading the latest snapshot, so a vanished unit or container stops being reported instead of keeping a stale value. A failed or timed-out section keeps its previous values.
| Metric | Unit | Attributes |
| ---------------------------------------------------------------------------------------------- | ----------- | --------------------------------------------------------- |
| `systemd.unit.active` (0/1) | `1` | `systemd.unit`, `systemd.sub_state`, `systemd.load_state` |
| `systemd.unit.failed` (0/1) | `1` | `systemd.unit` |
| `systemd.unit.restarts` (services only) | `{restart}` | `systemd.unit` |
| `systemd.timer.next_elapse_timestamp_seconds` (absolute unix time; unscheduled timers omitted) | `s` | `systemd.unit` |
| `docker.container.running` (0/1) | `1` | `container.name`, `container.image.name` |
| `docker.container.healthy` (0/1, only when a healthcheck exists) | `1` | `container.name` |
| `docker.container.uptime_seconds` | `s` | `container.name` |
| `docker.container.restarts` | `{restart}` | `container.name` |
Scope: an empty `otel.metrics.units` exports `*.service` and `*.timer` only (DEC-7); a non-empty list is a glob allowlist. An empty `otel.metrics.containers` exports all containers.
3. **Peer gauges** (only with `peers.enabled`): `peer.up` (`peer.name`, `peer.check`), `peer.check.duration_seconds`, `peer.last_change_timestamp_seconds` and `peer.api.auth_ok` (`peer.name`). They read the in-memory registry snapshot at each collection, so peers added or removed by a SIGHUP reload appear without re-registering; disabled peers are left out. Outbound peer requests run in a `peer.request` span exported as a client span (`otel.kind = client`) and carry `traceparent`. Metrics need `otel.enabled` and `otel.export_unit_metrics`.
4. **Logs**: when `otel.export_logs = true`, `logs_export.rs` runs a journal follow plus per-container docker follow tasks, reconciled against the Docker events stream (start/die/destroy) with backoff, so containers that come and go are tracked. Journal priority maps to OTel severity (emerg/alert/crit to FATAL, err ERROR, warning WARN, notice/info INFO, debug DEBUG); docker stderr is WARN, stdout INFO. Attributes: `systemd.unit`, `log.syslog.identifier`, `process.pid`, `container.name`, `log.iostream`. `otel.logs.journal_units` / `docker_containers` narrow the scope (empty = all).
Shutdown cancels the tasks and flushes all providers.
## Consolidator (`peers/`)
Opt-in (`[peers] enabled = true`): an instance reads other `slinuxm` instances declared in config. Without it there is no registry, no background task, no peer route or gauge, and 8 MCP tools. The registry keeps peer status in memory; two checks per peer (TCP connect, authenticated `GET /api/v1/ping`) run in background loops and SIGHUP reloads the peer list in place. Operator docs: [`docs/consolidator.md`](docs/consolidator.md). Decisions: DEC-13, DEC-14.
## OpenAPI (`openapi.rs`)
`openapi::document(peers)` is the single builder behind `GET /api/v1/openapi.json` and `slinuxm openapi [--peers]`. `docs/openapi.json` is the `--peers` output. See DEC-15.
`docs.rs` serves the Redoc viewer (`/docs`, `/docs/redoc.standalone.js`) from assets under `src/docs/` embedded with `include_str!`/`include_bytes!`. See DEC-16.
## Auth and state (`auth/`, `db.rs`)
- SQLite at `auth.db_path` (default `/var/lib/skyg-linux-tracker/db.sqlite`). A server-created directory is 0700 and the file 0600; an existing directory is never chmod'd (a group/world-accessible one only logs a warning). WAL mode plus `busy_timeout`, because the token CLI and the server share the DB. Under systemd, `StateDirectory=` creates the directory with the right owner.
- Table `tokens(id, name UNIQUE, token_hash UNIQUE, created_at, revoked_at, last_used_at)`. Tokens are 32 random bytes with the `slinuxm_` prefix; only the **sha256 hash** is stored and the plaintext is printed once. No bootstrap or declarative tokens (DEC-11).
- Middleware: sha256 of the presented bearer token, indexed lookup of non-revoked tokens (no separate constant-time compare is needed for a high-entropy token's hash). Failure is 401 with `WWW-Authenticate: Bearer` and the JSON error body; unmatched paths also return 401 so route existence is not disclosed. Revocation is immediate.
- The token CLI refuses to run when the effective uid (root included) does not own the DB file or, if absent, its state directory, and prints the matching `sudo -u <owner> slinuxm token ...` command (DEC-10). Token commands do not initialise telemetry.
## Build, packaging and deployment
- **devenv.nix**: Rust toolchain, `pkg-config`, `systemd` (libsystemd headers for the journal FFI), `opentelemetry-collector-contrib`; scripts `build`, `run`, `smoke`, `token`; `devenv up` runs the local collector.
- **Single binary**: pure Rust except libsystemd and bundled SQLite. rustls only; `cargo tree -i openssl` must be empty. The release binary was 19 MB when measured at P3.
- **systemd unit** (`systemd/skyg-linux-tracker.service`): static `User=skyg` (not `DynamicUser`, so `sudo -u skyg slinuxm token ...` reaches the same DB), `StateDirectory=` mode 0700, `SupplementaryGroups=systemd-journal`, `docker` group commented out (DEC-6), plus `ProtectSystem=strict`, `NoNewPrivileges`, syscall filter and related hardening. `systemd-analyze security --offline` scores 1.5 OK.
- **Nix** (`flake.nix`, `nix/`): `packages.default` (`buildRustPackage` from `Cargo.lock`; `cmake` is needed for `aws-lc-sys`, a transitive dependency; the pinned crypto provider for our own TLS is ring) and `nixosModules.default` under `services.skyg-linux-tracker` (user, `StateDirectory`, journal group, the same hardening as the unit, `/etc/skyg-linux-tracker.toml`, system-wide `slinuxm`, firewall options, `LoadCredential` TLS, eval assertion that a non-loopback bind needs auth). `nix flake check` runs a two-machine VM test (plain and TLS).
## Decisions
All accepted. Rationale is kept; the build history is in [`docs/workstreams/`](docs/workstreams/).
| ID | Decision | Status |
| ------ | ------------------------------------------------- | ------------------------------------------ |
| DEC-1 | Binary and CLI are `slinuxm` | Decided 2026-10-15 |
| DEC-2 | Source traits plus in-memory fakes from the start | Decided 2026-10-15 |
| DEC-3 | sqlx runtime queries, no macros | Decided 2026-10-15 |
| DEC-4 | MCP auth failure is HTTP 401 | Decided 2026-10-15 |
| DEC-5 | MCP resources not subscribable in v1 | Decided 2026-10-15 |
| DEC-6 | Docker socket access model | Decided 2026-10-15 |
| DEC-7 | Default metrics scope | Decided 2026-10-15 |
| DEC-8 | Origin validation for `/mcp` via rmcp | Decided 2026-10-15 |
| DEC-9 | Port 1984 and loopback bind by default | Decided 2026-10-16 |
| DEC-10 | No CLI wrapper; the app guards the root mistake | Decided 2026-10-16 |
| DEC-11 | No declarative or bootstrap tokens in v1 | Decided 2026-10-16 |
| DEC-12 | TLS terminated in-process with rustls | Decided 2026-10-16 |
| DEC-13 | Outbound peer credentials are external files | Decided 2026-10-19 |
| DEC-14 | Consolidator (peers) design | Decided 2026-10-19, implemented 2026-10-20 |
| DEC-15 | Generated OpenAPI document | Decided 2026-10-20 |
| DEC-16 | Redoc viewer served by the binary | Decided 2026-10-20 |
| DEC-17 | Build-stamped version | Decided 2026-10-21 |
- **DEC-1 name.** The plan used three names (repo `linux-system-monitor`, service `skyg-linux-tracker`, CLI `skyg`). The binary, CLI and `[[bin]]` are `slinuxm`; tokens use the `slinuxm_` prefix. Unchanged: product name `skyg-linux-tracker`, unit `skyg-linux-tracker.service`, state dir `/var/lib/skyg-linux-tracker`, config `/etc/skyg-linux-tracker.toml`, env prefix `SKYG_`.
- **DEC-2 traits and fakes.** `SystemdSource`, `JournalSource`, `DockerSource` live in `src/sources/mod.rs`, fakes in `src/sources/fake.rs`, `AppState` holds `Arc<dyn ...>` (via `async-trait`). Surfaces can be built and tested without systemd or Docker, at the cost of about 100 extra lines.
- **DEC-3 runtime queries.** `sqlx::query!` macros need `DATABASE_URL` at build time or a committed `.sqlx/` cache. With one table and about five queries, runtime `query`/`query_as` plus tests gives the same safety with no build-time DB.
- **DEC-4 HTTP 401.** The MCP authorization spec puts auth at the HTTP layer and clients expect 401 with `WWW-Authenticate: Bearer`. One middleware gives the same response for `/api/v1` and `/mcp`; `AppError::Unauthorized` renders it.
- **DEC-5 no subscriptions.** Subscriptions need change detection, i.e. background polling, which conflicts with "live reads only". Resources are resolved live on each read.
- **DEC-6 Docker socket.** `docker` group membership is root-equivalent. The unit ships with `SupplementaryGroups=docker` commented out; the hardened setup points `sources.docker_socket` at a read-only socket proxy (see [docs/operations.md](docs/operations.md#docker-access)). No code change is needed either way. The NixOS module's `docker.enable` adds the group explicitly.
- **DEC-7 metrics scope.** "Empty allowlist = all units" would export every loaded unit (often 300+ including devices and mounts), so empty means `*.service` and `*.timer`.
- **DEC-8 Origin validation.** The P0 spike showed rmcp 3.5 validates both headers itself: `with_allowed_hosts` (default loopback names) and `with_allowed_origins` (default empty means off; requests without `Origin` always pass). No custom tower layer. Hosts: loopback names, `server.bind`, `server.allowed_hosts`. Origins: `http(s)://localhost:*`, `127.0.0.1:*`, `[::1]:*`, `<bind>:*` plus `server.allowed_origins`.
- **DEC-9 defaults.** Binary and NixOS module both default to port 1984 and `bind = "127.0.0.1"` (the binary default was 9109 until 2026-10-16). LAN exposure is an install-time choice: set `bind`, `openFirewall` or an interface-scoped rule, and `allowedHosts`.
- **DEC-10 no wrapper.** The plain `slinuxm` binary is on `PATH`; mint tokens with `sudo -u skyg-linux-tracker slinuxm token create <name>`. Running as root can create a root-owned state dir and WAL files that lock the service out, so the token commands refuse (see Auth and state).
- **DEC-11 tokens.** Inbound tokens only (outbound: DEC-13). Printed once, never stored in plaintext; no `bootstrapTokens` option and no import of pre-hashed tokens.
- **DEC-12 TLS.** Bearer tokens over plain HTTP are readable on a LAN, which is the intended deployment, so TLS is native (`server.tls.*`, SIGHUP reload so ACME/sops rotation needs no restart). A reverse proxy still works. Off by default; nothing changes when it is off.
- **DEC-13 outbound credentials.** To call a peer the consolidator must replay a plaintext bearer token, so DEC-11's hash-only storage cannot apply. `slinuxm` never persists a peer token: config holds only a `token_file` path (on NixOS an agenix path delivered via `LoadCredential`), read into memory with redacted `Debug` and re-read on each check so rotation needs no restart. It does not protect against root on the consolidator; mitigate with one read-only token per consolidator. Full text: [`docs/workstreams/P5-consolidator/decisions.md`](docs/workstreams/P5-consolidator/decisions.md). There is one token loader, `peers::secrets::load_token`, used by startup validation, `peer check` and every request: it accepts a regular file only, requires owner-only permissions (with an exemption for `$CREDENTIALS_DIRECTORY`) and rejects CR, LF and NUL so a token cannot split a header.
- **DEC-14 consolidator design.** Peers are declarative (config / NixOS module, SIGHUP reload, no DB table). Two checks per peer: TCP connect and an authenticated `GET /api/v1/ping` (401 reports `api.up=true, auth=rejected`; older peers fall back to `/healthz`, `auth=unknown`). TCP 10s/3s, API 30s/5s, two failures to flip down, state in memory. REST: status plus a GET-only allowlisted pass-through under `/api/v1/peers/{peer}/...`. MCP: optional `peer` argument plus `list_peers`/`get_peer` (10 tools only when `peers.enabled`). No aggregation and no chaining in v1. Peer failures reuse `Unavailable`; a peer's 401/403 is never surfaced as `Unauthorized`. Four OTel gauges. SSRF: config-only URLs, no redirects, `deny_cidrs` at connect time, webpki roots plus optional `ca_file`, no skip-verify, plain http to non-loopback needs `allow_plain_http`. Any valid inbound token may read peer data (no scopes).
- **HTTP client outcome.** `reqwest` 0.13 with `rustls-no-provider`; the ring provider and webpki roots are passed to the client explicitly, so nothing is installed process-wide and `cargo tree -i openssl` stays empty. Redirects are off and the system proxy is ignored.
- **SSRF at connect time.** The host is resolved once, every address is checked against `deny_cidrs` after `to_canonical()` (so `::ffff:169.254.169.254` matches the IPv4 range), one denied address refuses the whole name, and the checked set is pinned with `resolve_to_addrs`, which closes DNS rebinding. The same `deny_networks` / `denied` code runs at config validation and at connect time. The default `deny_cidrs` covers both families (`169.254.0.0/16`, `0.0.0.0/8`, `fe80::/10`, `::/128`, `fd00:ec2::254/128`); private LAN ranges stay allowed.
- **DEC-15 generated OpenAPI.** The document is generated so it cannot drift from the code. REST paths are declared on the handlers with `#[utoipa::path]` (utoipa 6, OpenAPI 3.1). DTO schemas are reflected from the existing `schemars::JsonSchema` derives through a bridge in `openapi.rs`, so one derive serves MCP and OpenAPI; only payloads that exist as `serde_json` literals (error body, health) are written by hand and compared with real output in tests. MCP is not REST, so the tools go in the `x-mcp-tools` extension, built from the tool router's own schemas. Peer paths and the two peer tools appear only when peers are enabled. `tests/openapi.rs` is the drift guard: every router route must be documented, `x-mcp-tools` must match `tools/list`, and `docs/openapi.json` must equal `slinuxm openapi --peers` (regenerate with `cargo run -- openapi --peers > docs/openapi.json`). The route is authenticated like the rest of `/api/v1`.
- **DEC-16 Redoc viewer.** `GET /docs` renders the generated document with Redoc, served by the binary. The Redoc 2.5.4 standalone bundle is vendored under `src/docs/` (with its licenses) and embedded, so the page makes no CDN or other outbound request. It is stored gzipped (`redoc.standalone.js.gz`, about 320 KiB, under the 512 KiB hygiene limit) and sent as is with `Content-Encoding: gzip`; the refresh recipe is in the `docs.rs` module doc. 2.x was chosen over 3.x because 3.x is an ES module with a telemetry option; the 2.x bundle is a plain script. The viewer routes are public because a browser navigation cannot carry a bearer header, but they are static and hold no host data: the page asks for a token, keeps it in `sessionStorage` and sends it only to `GET /api/v1/openapi.json`, which stays authenticated. The page's one inline script is admitted by a CSP `sha256` computed from the same constant, so the policy cannot drift (`default-src 'none'`, `connect-src 'self'`, `form-action 'none'`, `frame-ancestors 'none'`; `style-src 'unsafe-inline'` and `worker-src blob:` are needed by Redoc itself). Both routes are in the OpenAPI document, so the drift test covers them. Redoc does not render the `x-mcp-tools` extension; the tools are in the JSON only.
- **DEC-17 build-stamped version.** The runtime version is `YYYY.MM.DD-<branch>-<sha>`, computed by `build.rs` and exposed as `skyg_linux_tracker::VERSION` (`SLINUXM_VERSION`); `--version`, `/healthz`, `/api/v1/ping`, MCP server info and OTel `service.version` use it. `Cargo.toml` stays semver because zero-padded CalVer parts (`2026.03.05`) are not valid semver, and `docs/openapi.json` keeps `info.version` from `CARGO_PKG_VERSION`: a stamp that changes per commit would fail the byte-for-byte drift test on every commit. The string logic (`src/version.rs`) is std-only and compiled into both `build.rs` and the library, so one implementation is unit-tested. Every part can come from `SLINUXM_GIT_SHA`, `SLINUXM_GIT_BRANCH` and `SLINUXM_BUILD_DATE`, and `SLINUXM_VERSION` overrides the whole string; the Nix sandbox has no `.git`, so `flake.nix` passes the commit date (`lastModifiedDate`, which keeps the build reproducible) and short rev, with the placeholder branch `nix` because a flake does not expose its branch. `SOURCE_DATE_EPOCH` is ignored because Nix dev shells set it to 1980. A dirty tree appends `-dirty` to the sha, a small extension of the pattern. Because naming any `rerun-if-changed` path turns off cargo's default package-wide watch, `build.rs` also lists `src`, `Cargo.toml`, `Cargo.lock` and `build.rs`, and watches HEAD through `git rev-parse --git-path` so linked worktrees work. A peer's `peer_version` is an opaque string, so a consolidator shows each peer's own stamp.
## Design notes and gotchas
Corrections found in the original design review and still worth knowing when changing the code.
- **systemd has no `ListTimers`** (R1). Timers are `*.timer` units plus the `Timer` interface properties (see Sources).
- **`ExecMainStatus` and `NRestarts` are `Service` properties** (R2), absent for timers, sockets and mounts, hence `Option` DTO fields.
- **`/healthz` must stay public** (R3). The auth layer sits on the protected router only, merged with a public router.
- **axum 0.8 routes use `/{name}`** (R4). The old `:name` syntax panics at startup.
- **There are 8 MCP tools, or 10 with `peers.enabled`** (R5). Changing the count is a deliberate change; the tool-list snapshot test guards both tables.
- **Metric allowlists and `otel.headers` are config keys** (R6) and documented in `config.example.toml`.
- **No openssl** (R7). Gate: `cargo tree -i openssl` prints nothing.
- **Runtime sqlx queries** (R8, DEC-3).
- Blocking sd-journal work, snapshot or follow, always runs in `spawn_blocking` and never holds a `Journal` across `.await` (G1).
- Follow loops use `wait()` with a timeout and exit when the receiver closes or shutdown fires, otherwise threads leak when an SSE client disconnects (G2). Every stream and background task selects on the one `CancellationToken`, otherwise graceful shutdown waits forever on open SSE (G3).
- Docker containers come and go, so log fan-out reconciles from the events stream (G4). A missing Docker socket is a per-request 503, never a startup failure (G9).
- Request bounds are config-driven (`limits.*`) and unit/container names are validated (G7). Observable gauges fed from a snapshot avoid stale series (G11).
- `unit_status` uses `GetUnit` rather than `LoadUnit` so a read-only tool has no side effects.
- `rmcp` accepts requests without an `Origin` header, so the bearer token is the real gate. With `bind = 0.0.0.0`, LAN clients need their Host names in `server.allowed_hosts` (a startup warning is logged otherwise).
- The Docker group is root-equivalent even though our code only reads; prefer a read-only socket proxy.
## Verification
A release is verified by hand on a real host with these six checks (last full run: minipc1, 2026-10-19; #5 is still open as I6-5 in [`TODO.md`](TODO.md)). With peers enabled, also run two instances against each other, cross-check a real peer, and confirm the peer gauges in the collector; the last two are still open (M6 in `TODO.md`).
1. `cargo build` inside `devenv shell`; unit tests for config precedence and DTO mapping.
2. Run locally: `cargo run -- serve`; create a token with `cargo run -- token create <name>`.
3. REST: `curl -H "Authorization: Bearer ..." localhost:1984/api/v1/systemd/units?state=failed`; SSE follow endpoints via `curl -N`.
4. MCP: point a client at `http://localhost:1984/mcp` (for example `npx @modelcontextprotocol/inspector`), verify the tool list, call each tool, check rejection without the header.
5. OTel: run `otelcol` with a debug exporter (`devenv up`), enable `[otel]`, confirm unit/container gauges appear on the poll interval and journal/docker log records flow with correct severity and attributes.
6. Negative tests: no token gives 401; `auth.enabled = false` with `bind = 0.0.0.0` makes startup refuse; a missing docker socket gives a clean 503 on docker endpoints with the rest of the API unaffected; SIGTERM with an open SSE client exits promptly. Automated in `tests/integration.rs`.
For NixOS, the gate is `nix build` and `nix flake check` including the VM test.