Fold Phase 5 into docs
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
6be774e98b
commit
f5e5f752e3
17 files changed
+71
-43
No files matched your search
+38
-14
@@ -2,7 +2,7 @@
|
||||
|
||||
skyg-linux-tracker (binary and CLI: `slinuxm`) is a read-only Linux system monitor in Rust, shipped as a single binary. It exposes systemd state, journald logs and Docker container state/logs over three surfaces that share one core: a **REST API**, an **MCP server** (Streamable HTTP) and **OpenTelemetry export**. Everything is read live from the source on every request. Nothing is collected or buffered, except a small SQLite database holding hashed API tokens.
|
||||
|
||||
User-facing usage is in [`README.md`](README.md). Work tracking is in [`TODO.md`](TODO.md). Per-workstream design logs from the original build are archived in [`docs/workstreams/`](docs/workstreams/).
|
||||
User-facing usage is in [`README.md`](README.md). Work tracking is in [`TODO.md`](TODO.md). Per-workstream design logs from the original build and Phase 5 are archived in [`docs/workstreams/`](docs/workstreams/).
|
||||
|
||||
## Requirements
|
||||
|
||||
@@ -26,6 +26,8 @@ User-facing usage is in [`README.md`](README.md). Work tracking is in [`TODO.md`
|
||||
| Docker | `bollard` (unix socket only) | Async Docker/Podman API; `list_containers`, `inspect_container` (health), log streams |
|
||||
| OTel | `opentelemetry`, `opentelemetry_sdk`, `opentelemetry-otlp` 0.33, `tracing-opentelemetry` | One OTLP config (endpoint, protocol, headers) drives all three signals. Metrics and logs are Stable, traces Beta |
|
||||
| Auth DB | `sqlx` (sqlite, bundled) with **runtime** queries | Async, pure-Rust bundled SQLite, no build-time database (DEC-3) |
|
||||
| Peer client | `reqwest` 0.13 (`rustls-no-provider`, ring passed in) + `webpki-roots` | Outbound calls to peers; no openssl, nothing installed process-wide (DEC-14) |
|
||||
| OpenAPI | `utoipa` 6 + `schemars` | Generated REST and MCP description (DEC-15) |
|
||||
| Config | `figment` (TOML + env) + `clap` | Layered config with clear precedence |
|
||||
|
||||
Rust edition 2021; release profile uses LTO and `strip`. Cargo.lock is committed. The only dynamic dependencies are libc, libm, libgcc_s and libsystemd (check with `ldd`).
|
||||
@@ -41,10 +43,13 @@ migrations/0001_tokens.sql # embedded sqlx migration
|
||||
systemd/skyg-linux-tracker.service # hardened example unit
|
||||
dev/otelcol.yaml # local collector with debug exporter (`devenv up`)
|
||||
examples/ # systemd_dump, journal_tail, docker_ps: run one source against the live host
|
||||
tests/ # integration.rs, tls.rs, plain.rs, binary_smoke.rs, common/
|
||||
docs/openapi.json # generated OpenAPI 3.1 doc (with peers), guarded by a drift test
|
||||
tests/ # integration.rs, tls.rs, plain.rs, binary_smoke.rs, peers.rs, peers_config.rs, peers_registry.rs, mcp_peers.rs, openapi.rs, common/
|
||||
src/
|
||||
main.rs # clap CLI: serve | token create|list|revoke, --print-config
|
||||
main.rs # clap CLI: serve | token create|list|revoke | peer list|check | openapi [--peers], --print-config
|
||||
lib.rs
|
||||
cli.rs # `slinuxm openapi`
|
||||
openapi.rs # builds the OpenAPI document (route and CLI share it)
|
||||
config.rs # figment layering + validation
|
||||
error.rs # AppError -> HTTP status + JSON body, and -> MCP ErrorData
|
||||
state.rs # AppState: sources (Arc<dyn ...>), optional DB pool, config, CancellationToken
|
||||
@@ -56,16 +61,17 @@ src/
|
||||
mod.rs # traits + shared DTOs (UnitSummary, TimerInfo, JournalEntry, ContainerSummary, ...)
|
||||
systemd.rs, journal.rs, docker.rs # real implementations
|
||||
fake.rs # in-memory fakes used by handler and tool tests
|
||||
api/ # REST layer: systemd.rs, journal.rs, docker.rs, mod.rs
|
||||
peers/ # consolidator: config.rs, secrets.rs (token loader), registry.rs, health.rs, client.rs (reqwest transport), proxy.rs, transport.rs (trait), types.rs, error.rs, cli.rs, testing.rs (fakes)
|
||||
api/ # REST layer: systemd.rs, journal.rs, docker.rs, ping.rs, peers.rs, mod.rs
|
||||
mcp/ # TrackerService (mod.rs), prompts.rs, resources.rs
|
||||
telemetry/ # mod.rs (providers, guard), metrics_export.rs, logs_export.rs
|
||||
telemetry/ # mod.rs (providers, guard, peer request span), metrics_export.rs, logs_export.rs
|
||||
```
|
||||
|
||||
The key structural decision: **`sources/` is the only place that talks to D-Bus, journald or Docker.** `api/`, `mcp/` and `telemetry/` are thin projections over it, which makes "MCP and REST expose the same data" true by construction. Sources are traits held as `Arc<dyn ...>` in `AppState` (DEC-2), so every handler and tool test runs against fakes without systemd or Docker. `server.rs` is the only place the surfaces meet.
|
||||
The key structural decision: **`sources/` is the only place that talks to D-Bus, journald or Docker.** `api/`, `mcp/` and `telemetry/` are thin projections over it, which makes "MCP and REST expose the same data" true by construction. `peers/` is the one outbound HTTP client; the registry exists only when `[peers] enabled = true`, and REST and MCP share its proxy functions. Sources are traits held as `Arc<dyn ...>` in `AppState` (DEC-2), so every handler and tool test runs against fakes without systemd or Docker. `server.rs` is the only place the surfaces meet.
|
||||
|
||||
## Configuration
|
||||
|
||||
See [`config.example.toml`](config.example.toml); every value shown is the default. Sections: `[server]` (`bind` 127.0.0.1, `port` 1984, `allowed_hosts`, `allowed_origins`, `[server.tls]`), `[auth]` (`enabled`, `db_path`), `[otel]` (`enabled`, `endpoint`, `protocol`, `service_name`, `export_unit_metrics`, `export_logs`, `metrics_interval_secs`, `[otel.headers]`, `[otel.logs]`, `[otel.metrics]` allowlists), `[sources]` (`docker_socket`, `journal_namespace`) and `[limits]` (`max_lines`, `max_tail`, `max_follow_secs`).
|
||||
See [`config.example.toml`](config.example.toml); every value shown is the default. Sections: `[server]` (`bind` 127.0.0.1, `port` 1984, `allowed_hosts`, `allowed_origins`, `[server.tls]`), `[auth]` (`enabled`, `db_path`), `[otel]` (`enabled`, `endpoint`, `protocol`, `service_name`, `export_unit_metrics`, `export_logs`, `metrics_interval_secs`, `[otel.headers]`, `[otel.logs]`, `[otel.metrics]` allowlists), `[sources]` (`docker_socket`, `journal_namespace`) and `[limits]` (`max_lines`, `max_tail`, `max_follow_secs`) and `[peers]` (off by default; see Consolidator).
|
||||
|
||||
Precedence: defaults < config file (`--config`, default `/etc/skyg-linux-tracker.toml`) < `SKYG_*` env (nesting with `__`, e.g. `SKYG_SERVER__PORT`) < CLI flags. `slinuxm --print-config` prints the effective config with secrets redacted. Validation rejects `auth.enabled = false` with a non-loopback `bind`, `port = 0`, and incomplete TLS settings.
|
||||
|
||||
@@ -89,6 +95,10 @@ All under `/api/v1`, GET only, behind the auth middleware when enabled. Routes u
|
||||
| `GET /api/v1/docker/ps` | `ps()` |
|
||||
| `GET /api/v1/docker/containers/{name}` | `container_status()` |
|
||||
| `GET /api/v1/docker/containers/{name}/logs?tail=&since=&follow=0\|1` (JSON, or SSE events `stdout`/`stderr`) | `logs()` |
|
||||
| `GET /api/v1/ping` | authenticated liveness, used by a consolidator's API check |
|
||||
| `GET /api/v1/openapi.json` | generated OpenAPI 3.1 document; peer paths and tools only when peers are enabled |
|
||||
| `GET /api/v1/peers`, `GET /api/v1/peers/{peer}` | peer status (only with `peers.enabled`) |
|
||||
| `GET /api/v1/peers/{peer}/{*path}` | allowlisted GET pass-through to the peer; the response carries `x-slinuxm-peer` (only with `peers.enabled`) |
|
||||
| `GET /healthz` | public liveness, `{"status":"ok","version":...}` |
|
||||
|
||||
SSE streams (`axum::response::Sse` with keep-alive) read from the mpsc channels of `sources/` and end on client disconnect or shutdown.
|
||||
@@ -97,8 +107,9 @@ SSE streams (`axum::response::Sse` with keep-alive) read from the mpsc channels
|
||||
|
||||
`rmcp`'s `StreamableHttpService` is a Tower service nested at `/mcp` on the protected router (`mcp::service(state)`). One per-session `TrackerService` shares `AppState`. Auth failures are HTTP 401 with `WWW-Authenticate: Bearer` (DEC-4), the same middleware as REST.
|
||||
|
||||
Eight tools, each a thin wrapper over `sources/` with `schemars` parameter schemas: `list_systemd_units { state? }`, `get_systemd_unit { name }`, `list_systemd_timers`, `get_journal_logs { unit?, priority?, lines }`, `tail_journal { unit, seconds }`, `list_containers`, `get_container { name }`, `get_container_logs { name, tail }`.
|
||||
Eight tools (ten with `peers.enabled`, below), each a thin wrapper over `sources/` with `schemars` parameter schemas: `list_systemd_units { state? }`, `get_systemd_unit { name }`, `list_systemd_timers`, `get_journal_logs { unit?, priority?, lines }`, `tail_journal { unit, seconds }`, `list_containers`, `get_container { name }`, `get_container_logs { name, tail }`.
|
||||
|
||||
- With `peers.enabled` there are ten: `list_peers` and `get_peer { name }` are added, each of the eight takes an optional `peer` argument that reads from that peer through this instance, and the instructions gain a peers paragraph. The tool-list snapshot test guards both tables.
|
||||
- `tail_journal` is a bounded follow: it clamps `seconds` to `limits.max_follow_secs`, stops at an entry cap and returns what it collected. There are no infinite MCP streams.
|
||||
- Log tools return `{entries, truncated, note}` capped at 256 KiB. Source errors come back as `CallToolResult` errors with an actionable message. Log content is documented as untrusted data.
|
||||
- Agent context: `ServerHandler` **instructions** (start with failed units or unhealthy containers, then pull logs, use timers to explain jobs), rich tool descriptions (systemd states, health states, journal priorities), **prompts** `diagnose_unit { name }` and `container_health_report`, **resources** `systemd://units/failed` and `docker://containers` plus templates `systemd://unit/{name}` and `docker://container/{name}`. Resources are list/read, resolved live, not subscribable (DEC-5).
|
||||
@@ -141,10 +152,19 @@ One OTLP exporter config (grpc or http/protobuf, optional headers) drives three
|
||||
| `docker.container.restarts` | `{restart}` | `container.name` |
|
||||
|
||||
Scope: an empty `otel.metrics.units` exports `*.service` and `*.timer` only (DEC-7); a non-empty list is a glob allowlist. An empty `otel.metrics.containers` exports all containers.
|
||||
3. **Logs**: when `otel.export_logs = true`, `logs_export.rs` runs a journal follow plus per-container docker follow tasks, reconciled against the Docker events stream (start/die/destroy) with backoff, so containers that come and go are tracked. Journal priority maps to OTel severity (emerg/alert/crit to FATAL, err ERROR, warning WARN, notice/info INFO, debug DEBUG); docker stderr is WARN, stdout INFO. Attributes: `systemd.unit`, `log.syslog.identifier`, `process.pid`, `container.name`, `log.iostream`. `otel.logs.journal_units` / `docker_containers` narrow the scope (empty = all).
|
||||
3. **Peer gauges** (only with `peers.enabled`): `peer.up` (`peer.name`, `peer.check`), `peer.check.duration_seconds`, `peer.last_change_timestamp_seconds` and `peer.api.auth_ok` (`peer.name`). They read the in-memory registry snapshot at each collection, so peers added or removed by a SIGHUP reload appear without re-registering; disabled peers are left out. Outbound peer requests run in a `peer.request` span exported as a client span (`otel.kind = client`) and carry `traceparent`. Metrics need `otel.enabled` and `otel.export_unit_metrics`.
|
||||
4. **Logs**: when `otel.export_logs = true`, `logs_export.rs` runs a journal follow plus per-container docker follow tasks, reconciled against the Docker events stream (start/die/destroy) with backoff, so containers that come and go are tracked. Journal priority maps to OTel severity (emerg/alert/crit to FATAL, err ERROR, warning WARN, notice/info INFO, debug DEBUG); docker stderr is WARN, stdout INFO. Attributes: `systemd.unit`, `log.syslog.identifier`, `process.pid`, `container.name`, `log.iostream`. `otel.logs.journal_units` / `docker_containers` narrow the scope (empty = all).
|
||||
|
||||
Shutdown cancels the tasks and flushes all providers.
|
||||
|
||||
## Consolidator (`peers/`)
|
||||
|
||||
Opt-in (`[peers] enabled = true`): an instance reads other `slinuxm` instances declared in config. Without it there is no registry, no background task, no peer route or gauge, and 8 MCP tools. The registry keeps peer status in memory; two checks per peer (TCP connect, authenticated `GET /api/v1/ping`) run in background loops and SIGHUP reloads the peer list in place. Operator docs: [`docs/consolidator.md`](docs/consolidator.md). Decisions: DEC-13, DEC-14.
|
||||
|
||||
## OpenAPI (`openapi.rs`)
|
||||
|
||||
`openapi::document(peers)` is the single builder behind `GET /api/v1/openapi.json` and `slinuxm openapi [--peers]`. `docs/openapi.json` is the `--peers` output. See DEC-15.
|
||||
|
||||
## Auth and state (`auth/`, `db.rs`)
|
||||
|
||||
- SQLite at `auth.db_path` (default `/var/lib/skyg-linux-tracker/db.sqlite`). A server-created directory is 0700 and the file 0600; an existing directory is never chmod'd (a group/world-accessible one only logs a warning). WAL mode plus `busy_timeout`, because the token CLI and the server share the DB. Under systemd, `StateDirectory=` creates the directory with the right owner.
|
||||
@@ -178,7 +198,8 @@ All accepted. Rationale is kept; the build history is in [`docs/workstreams/`](d
|
||||
| DEC-11 | No declarative or bootstrap tokens in v1 | Decided 2026-10-16 |
|
||||
| DEC-12 | TLS terminated in-process with rustls | Decided 2026-10-16 |
|
||||
| DEC-13 | Outbound peer credentials are external files | Decided 2026-10-19 |
|
||||
| DEC-14 | Consolidator (peers) design | Decided 2026-10-19 |
|
||||
| DEC-14 | Consolidator (peers) design | Decided 2026-10-19, implemented 2026-10-20 |
|
||||
| DEC-15 | Generated OpenAPI document | Decided 2026-10-20 |
|
||||
|
||||
- **DEC-1 name.** The plan used three names (repo `linux-system-monitor`, service `skyg-linux-tracker`, CLI `skyg`). The binary, CLI and `[[bin]]` are `slinuxm`; tokens use the `slinuxm_` prefix. Unchanged: product name `skyg-linux-tracker`, unit `skyg-linux-tracker.service`, state dir `/var/lib/skyg-linux-tracker`, config `/etc/skyg-linux-tracker.toml`, env prefix `SKYG_`.
|
||||
- **DEC-2 traits and fakes.** `SystemdSource`, `JournalSource`, `DockerSource` live in `src/sources/mod.rs`, fakes in `src/sources/fake.rs`, `AppState` holds `Arc<dyn ...>` (via `async-trait`). Surfaces can be built and tested without systemd or Docker, at the cost of about 100 extra lines.
|
||||
@@ -193,8 +214,11 @@ All accepted. Rationale is kept; the build history is in [`docs/workstreams/`](d
|
||||
- **DEC-11 tokens.** Inbound tokens only (outbound: DEC-13). Printed once, never stored in plaintext; no `bootstrapTokens` option and no import of pre-hashed tokens.
|
||||
- **DEC-12 TLS.** Bearer tokens over plain HTTP are readable on a LAN, which is the intended deployment, so TLS is native (`server.tls.*`, SIGHUP reload so ACME/sops rotation needs no restart). A reverse proxy still works. Off by default; nothing changes when it is off.
|
||||
|
||||
- **DEC-13 outbound credentials.** To call a peer the consolidator must replay a plaintext bearer token, so DEC-11's hash-only storage cannot apply. `slinuxm` never persists a peer token: config holds only a `token_file` path (on NixOS an agenix path delivered via `LoadCredential`), read into memory with redacted `Debug` and re-read on each check so rotation needs no restart. It does not protect against root on the consolidator; mitigate with one read-only token per consolidator. Full text: `plan/phase-5-consolidator/decisions.md`.
|
||||
- **DEC-14 consolidator design.** Peers are declarative (config / NixOS module, SIGHUP reload, no DB table). Two checks per peer: TCP connect and an authenticated `GET /api/v1/ping` (401 reports `api.up=true, auth=rejected`; older peers fall back to `/healthz`, `auth=unknown`). TCP 10s/3s, API 30s/5s, two failures to flip down, state in memory. REST: status plus a GET-only allowlisted pass-through under `/api/v1/peers/{peer}/...`. MCP: optional `peer` argument plus `list_peers`/`get_peer` (10 tools only when `peers.enabled`). No aggregation and no chaining in v1. Peer failures reuse `Unavailable`; a peer's 401/403 is never surfaced as `Unauthorized`. Four OTel gauges. SSRF: config-only URLs, no redirects, `deny_cidrs` at connect time, webpki roots plus optional `ca_file`, no skip-verify, plain http to non-loopback needs `allow_plain_http`. Any valid inbound token may read peer data (no scopes). HTTP client is reqwest with rustls/ring unless openssl appears (decided in B1).
|
||||
- **DEC-13 outbound credentials.** To call a peer the consolidator must replay a plaintext bearer token, so DEC-11's hash-only storage cannot apply. `slinuxm` never persists a peer token: config holds only a `token_file` path (on NixOS an agenix path delivered via `LoadCredential`), read into memory with redacted `Debug` and re-read on each check so rotation needs no restart. It does not protect against root on the consolidator; mitigate with one read-only token per consolidator. Full text: [`docs/workstreams/P5-consolidator/decisions.md`](docs/workstreams/P5-consolidator/decisions.md). There is one token loader, `peers::secrets::load_token`, used by startup validation, `peer check` and every request: it accepts a regular file only, requires owner-only permissions (with an exemption for `$CREDENTIALS_DIRECTORY`) and rejects CR, LF and NUL so a token cannot split a header.
|
||||
- **DEC-14 consolidator design.** Peers are declarative (config / NixOS module, SIGHUP reload, no DB table). Two checks per peer: TCP connect and an authenticated `GET /api/v1/ping` (401 reports `api.up=true, auth=rejected`; older peers fall back to `/healthz`, `auth=unknown`). TCP 10s/3s, API 30s/5s, two failures to flip down, state in memory. REST: status plus a GET-only allowlisted pass-through under `/api/v1/peers/{peer}/...`. MCP: optional `peer` argument plus `list_peers`/`get_peer` (10 tools only when `peers.enabled`). No aggregation and no chaining in v1. Peer failures reuse `Unavailable`; a peer's 401/403 is never surfaced as `Unauthorized`. Four OTel gauges. SSRF: config-only URLs, no redirects, `deny_cidrs` at connect time, webpki roots plus optional `ca_file`, no skip-verify, plain http to non-loopback needs `allow_plain_http`. Any valid inbound token may read peer data (no scopes).
|
||||
- **HTTP client outcome.** `reqwest` 0.13 with `rustls-no-provider`; the ring provider and webpki roots are passed to the client explicitly, so nothing is installed process-wide and `cargo tree -i openssl` stays empty. Redirects are off and the system proxy is ignored.
|
||||
- **SSRF at connect time.** The host is resolved once, every address is checked against `deny_cidrs` after `to_canonical()` (so `::ffff:169.254.169.254` matches the IPv4 range), one denied address refuses the whole name, and the checked set is pinned with `resolve_to_addrs`, which closes DNS rebinding. The same `deny_networks` / `denied` code runs at config validation and at connect time. The default `deny_cidrs` covers both families (`169.254.0.0/16`, `0.0.0.0/8`, `fe80::/10`, `::/128`, `fd00:ec2::254/128`); private LAN ranges stay allowed.
|
||||
- **DEC-15 generated OpenAPI.** The document is generated so it cannot drift from the code. REST paths are declared on the handlers with `#[utoipa::path]` (utoipa 6, OpenAPI 3.1). DTO schemas are reflected from the existing `schemars::JsonSchema` derives through a bridge in `openapi.rs`, so one derive serves MCP and OpenAPI; only payloads that exist as `serde_json` literals (error body, health) are written by hand and compared with real output in tests. MCP is not REST, so the tools go in the `x-mcp-tools` extension, built from the tool router's own schemas. Peer paths and the two peer tools appear only when peers are enabled. `tests/openapi.rs` is the drift guard: every router route must be documented, `x-mcp-tools` must match `tools/list`, and `docs/openapi.json` must equal `slinuxm openapi --peers` (regenerate with `cargo run -- openapi --peers > docs/openapi.json`). The route is authenticated like the rest of `/api/v1`.
|
||||
|
||||
## Design notes and gotchas
|
||||
|
||||
@@ -204,7 +228,7 @@ Corrections found in the original design review and still worth knowing when cha
|
||||
- **`ExecMainStatus` and `NRestarts` are `Service` properties** (R2), absent for timers, sockets and mounts, hence `Option` DTO fields.
|
||||
- **`/healthz` must stay public** (R3). The auth layer sits on the protected router only, merged with a public router.
|
||||
- **axum 0.8 routes use `/{name}`** (R4). The old `:name` syntax panics at startup.
|
||||
- **There are 8 MCP tools** (R5). Adding a ninth is a deliberate change; the tool-list snapshot test catches accidental ones.
|
||||
- **There are 8 MCP tools, or 10 with `peers.enabled`** (R5). Changing the count is a deliberate change; the tool-list snapshot test guards both tables.
|
||||
- **Metric allowlists and `otel.headers` are config keys** (R6) and documented in `config.example.toml`.
|
||||
- **No openssl** (R7). Gate: `cargo tree -i openssl` prints nothing.
|
||||
- **Runtime sqlx queries** (R8, DEC-3).
|
||||
@@ -218,7 +242,7 @@ Corrections found in the original design review and still worth knowing when cha
|
||||
|
||||
## Verification
|
||||
|
||||
A release is verified by hand on a real host with these six checks (last full run: minipc1, 2026-10-19; #5 is still open as I6-5 in [`TODO.md`](TODO.md)).
|
||||
A release is verified by hand on a real host with these six checks (last full run: minipc1, 2026-10-19; #5 is still open as I6-5 in [`TODO.md`](TODO.md)). With peers enabled, also run two instances against each other, cross-check a real peer, and confirm the peer gauges in the collector; the last two are still open (M6 in `TODO.md`).
|
||||
|
||||
1. `cargo build` inside `devenv shell`; unit tests for config precedence and DTO mapping.
|
||||
2. Run locally: `cargo run -- serve`; create a token with `cargo run -- token create <name>`.
|
||||
|
||||
Reference in new issue
Block a user