doc updates

This commit is contained in:
ari committed 2026-10-19 14:14:29 -04:00
1 parent 7e1bd4a9a4
commit a511ff532e
26 files changed
+573 -770

No files matched your search

+30
View File
@@ -0,0 +1,30 @@
# Documentation index
Root files: [`README.md`](../README.md) (install and usage), [`ARCHITECTURE.md`](../ARCHITECTURE.md) (design, decisions, verification), [`TODO.md`](../TODO.md) (work tracking), [`AGENTS.md`](../AGENTS.md) (guidance for AI coding agents). Plans for the next big phase go in [`plan/`](../plan/README.md).
## Reference
| Doc | Contents |
|---|---|
| [`api.md`](api.md) | Full REST route table, MCP tools/prompts/resources, client setup, Origin and Host checks |
| [`tls.md`](tls.md) | Native TLS: configuration, self-signed example, SIGHUP rotation, ACME and sops paths |
## Workstream records (archived)
> **Archived.** The files in [`workstreams/`](workstreams/) are historical records of how the project was built (scope, owned files, design notes, logs). They are not maintained. Current design is in [`ARCHITECTURE.md`](../ARCHITECTURE.md) and open work is in [`TODO.md`](../TODO.md).
| Card | Phase | Depends on |
|---|---|---|
| [P0 Foundation](workstreams/P0-foundation.md) | 0 | none |
| [WS-A systemd](workstreams/WS-A-systemd.md) | 1 | P0 |
| [WS-B journal](workstreams/WS-B-journal.md) | 1 | P0 |
| [WS-C docker](workstreams/WS-C-docker.md) | 1 | P0 |
| [WS-D auth + DB](workstreams/WS-D-auth.md) | 1 | P0 |
| [WS-E REST](workstreams/WS-E-rest.md) | 2 | A, B, C, D |
| [WS-F MCP](workstreams/WS-F-mcp.md) | 2 | A, B, C, D |
| [WS-G OTel](workstreams/WS-G-otel.md) | 2 | A, B, C |
| [P3 Integration](workstreams/P3-integration.md) | 3 | E, F, G |
| [WS-I TLS](workstreams/WS-I-tls.md) | 4 | P3 |
| [WS-H NixOS](workstreams/WS-H-nixos.md) | 4 | P3, WS-I (gate result pending) |
Dependency order: P0, then A to D in parallel, then E to G in parallel, then P3, then WS-I, then WS-H.
+59
View File
@@ -0,0 +1,59 @@
# REST and MCP reference
Full endpoint and MCP client reference. The short version is in the [README](../README.md); the design is in [ARCHITECTURE.md](../ARCHITECTURE.md#rest-api).
## REST
Base: `http://127.0.0.1:1984`. Errors are `{"error":{"code","message"}}`.
| Route | Notes |
|---|---|
| `GET /healthz` | public, `{"status":"ok","version":...}` |
| `GET /api/v1/systemd/units?state=failed` | `state` is an active state (`failed`, `active`, ...) |
| `GET /api/v1/systemd/units/{name}` | one unit with details; read-only, so only units systemd has loaded (unloaded or unknown is 404) |
| `GET /api/v1/systemd/timers` | |
| `GET /api/v1/journal?unit=&priority=err&lines=100` | priority by name or number |
| `GET /api/v1/journal/follow?unit=&priority=` | SSE, event `journal` |
| `GET /api/v1/docker/ps` | `503` when Docker is unavailable |
| `GET /api/v1/docker/containers/{name}` | |
| `GET /api/v1/docker/containers/{name}/logs?tail=&since=&follow=0\|1` | JSON, or SSE with events `stdout`/`stderr` |
```sh
TOKEN=slinuxm_...
curl -H "Authorization: Bearer $TOKEN" 'http://127.0.0.1:1984/api/v1/systemd/units?state=failed'
curl -N -H "Authorization: Bearer $TOKEN" 'http://127.0.0.1:1984/api/v1/journal/follow?priority=err'
```
## MCP
Endpoint: `http://127.0.0.1:1984/mcp` (Streamable HTTP). Eight tools (`list_systemd_units`, `get_systemd_unit`, `list_systemd_timers`, `get_journal_logs`, `tail_journal`, `list_containers`, `get_container`, `get_container_logs`), two prompts (`diagnose_unit`, `container_health_report`) two resources (`systemd://units/failed`, `docker://containers`) and two resource templates (`systemd://unit/{name}`, `docker://container/{name}`, same JSON as `get_systemd_unit` and `get_container`; percent-encode unusual characters in the name). Log content is untrusted data and is labelled as such in tool results.
Generic client config (header-capable clients):
```json
{
"mcpServers": {
"linux-tracker": {
"url": "http://127.0.0.1:1984/mcp",
"headers": { "Authorization": "Bearer slinuxm_..." }
}
}
}
```
Claude Code: `claude mcp add --transport http linux-tracker http://127.0.0.1:1984/mcp --header "Authorization: Bearer $TOKEN"`.
Inspector: `npx @modelcontextprotocol/inspector`, transport Streamable HTTP, add the bearer header.
Raw handshake with curl:
```sh
curl -si -X POST http://127.0.0.1:1984/mcp \
-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
-H 'Accept: application/json, text/event-stream' \
-d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"curl","version":"0"}}}'
```
### Origin and Host checks
`/mcp` validates `Origin` and `Host` to defend against DNS rebinding from browsers. Loopback names, `server.bind` and the configured port are accepted by default. Requests without an `Origin` header (non-browser clients) pass this check, so the bearer token remains the real gate. If you bind to `0.0.0.0` or serve behind a name, add the names clients use to `server.allowed_hosts` (for example `["monitor.lan:1984"]`) and any browser origins to `server.allowed_origins`. With `auth.enabled = false` (loopback only), REST requests are checked against a Host allowlist (loopback names, `server.bind` and `server.allowed_hosts`) and get `403` otherwise, to blunt DNS rebinding. There is no Origin check on REST and no token in that mode, so keep auth enabled on any multi-user host. If the host is reachable beyond a trusted network, enable native TLS (see [tls.md](tls.md)) or terminate TLS in a reverse proxy.
+27
View File
@@ -0,0 +1,27 @@
# TLS
Reference for native TLS. Summary in the [README](../README.md#tls); design in [ARCHITECTURE.md](../ARCHITECTURE.md#tls).
TLS is optional and off by default. With `server.tls.enabled = false` (the default) the server speaks plain HTTP exactly as before, the certificate paths are ignored and never read, and no TLS code runs. A non-loopback `bind` without TLS logs a startup warning (it is not refused).
Enable it in the config file or the environment (`SKYG_SERVER__TLS__ENABLED=true`, `SKYG_SERVER__TLS__CERT_FILE`, `SKYG_SERVER__TLS__KEY_FILE`):
```toml
[server.tls]
enabled = true
cert_file = "/etc/skyg/tls/cert.pem" # PEM chain, leaf first
key_file = "/etc/skyg/tls/key.pem" # PEM, PKCS#8, PKCS#1 or SEC1
```
Both files are required and checked at startup; a missing, unparsable or mismatched pair stops the server with an error naming the file (key contents are never logged). REST and `/mcp` are then served over HTTPS only (plain HTTP on that port fails), and the default MCP `Origin` allowlist uses `https://` instead of `http://`. Add your public names to `server.allowed_hosts` and `server.allowed_origins` as before. TLS is rustls with the ring provider; openssl is not used.
Self-signed example:
```sh
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 -nodes -days 365 \
-keyout key.pem -out cert.pem -subj '/CN=monitor.lan' \
-addext 'subjectAltName=DNS:monitor.lan,DNS:localhost,IP:127.0.0.1'
curl --cacert cert.pem -H "Authorization: Bearer $TOKEN" https://localhost:1984/api/v1/systemd/units
```
Rotation: replace the files and send `SIGHUP` (`kill -HUP $(pidof slinuxm)`, or `systemctl kill -s HUP <unit>`). New connections get the new certificate; established ones are not dropped. The certificate and key are read as two separate files, so rotation tooling must send SIGHUP only after BOTH are in place; if the pair is unreadable or mismatched (for example the cert was replaced but the key not yet), the old certificate stays, an error is logged, and you send SIGHUP again once the files are consistent. Any intermediate certificates must be in the `cert_file` chain. Both the TLS and the plain listener are HTTP/1.1 only, share one accept loop, cap concurrent connections at 1024 and close connections that send no request header within 30 seconds. SIGHUP is harmless when TLS is disabled (it is logged and ignored). For ACME, point `cert_file` and `key_file` at the client's output (for example certbot's `live/<name>/fullchain.pem` and `privkey.pem`, or the NixOS `security.acme` directory) and send SIGHUP from the renewal hook. For sops-nix or systemd `LoadCredential`, point at the decrypted or credential path (`/run/credentials/<unit>/...`) so the service user needs no access to the original key file.
+120
View File
@@ -0,0 +1,120 @@
# P0 — Foundation
| Field | Value |
|---|---|
| Status | Done |
| Owner | — |
| Phase | 0 (blocking, single track) |
| Depends on | — |
| Blocks | Every other card |
| Owned files | `Cargo.toml`, `devenv.nix`, `config.example.toml`, `src/main.rs`, `src/config.rs`, `src/error.rs`, `src/state.rs`, `src/sources/mod.rs`, `src/sources/fake.rs`, initial stubs of every other module |
## Goal
Fix every contract that parallel work depends on: dependencies, config shape, DTOs, source traits, error type, `AppState`, and module signatures. After P0, no other card should need to touch `Cargo.toml`, `main.rs` or `sources/mod.rs`, except by agreed exception.
Keep it small. Stubs contain `todo!()` and no logic.
## Tasks
Tick these in [`TODO.md`](../../TODO.md).
- **P0-1 devenv.** Keep `languages.rust.enable` and the existing `CLAUDE_CONFIG_DIR` `enterShell`. Add `pkgs.pkg-config` and `pkgs.systemd` (its `.dev` output provides `libsystemd.pc`). Add `scripts.build`, `scripts.run` and `scripts.token`, which wraps `cargo run -- token`. Do not add openssl. Optionally add `pkgs.opentelemetry-collector-contrib` for G7.
- **P0-2 Cargo.toml.** Package `skyg-linux-tracker` with `[[bin]] name = "slinuxm"` (DEC-1). Pin every crate needed by all phases:
- Runtime: `tokio` (full), `tokio-util` (`CancellationToken`), `futures`.
- HTTP: `axum` 0.8, `tower`, `tower-http` (`trace`).
- Data: `serde`, `serde_json`, `schemars`, `thiserror`, `jiff` or `chrono`.
- Config/CLI: `figment` (`toml`, `env`), `clap` (`derive`, `env`).
- MCP: `rmcp` (`server`, `macros`, `schemars`, `transport-streamable-http-server`).
- systemd/journal: `zbus`, `zbus_systemd` (`systemd1`), `systemd` (`journal`, default features off where possible).
- Docker: `bollard`, no TLS features needed for a unix socket.
- OTel: `opentelemetry`, `opentelemetry_sdk`, `opentelemetry-otlp` (`grpc-tonic`, `http-proto`, rustls), `tracing`, `tracing-subscriber`, `tracing-opentelemetry`.
- Auth: `sqlx` (`runtime-tokio`, `sqlite`), `sha2`, `rand`, `base64`.
- Optional: `async-trait`.
- Release profile: `lto = true`, `strip = true`, `codegen-units = 1`.
- **P0-3 API spike, max half a day.** Confirm against the pinned versions:
- The rmcp `StreamableHttpService::new` signature, `StreamableHttpServerConfig` fields, and whether it validates Origin or Host (DEC-8).
- zbus_systemd `ManagerProxy`, `UnitProxy`, `ServiceProxy`, `TimerProxy` names and property types.
- The `systemd` crate journal open API, `wait` with timeout, namespace support, and whether `Journal` is `Send`.
- bollard feature flags that avoid pulling openssl or rustls when only the unix socket is used.
- The opentelemetry-otlp builder API for headers and protocol selection.
- Record findings in the log below.
- **P0-4 config.rs.** Use figment `Serialized::defaults` < `Toml::file` < `Env::prefixed("SKYG_").split("__")` < CLI overrides. Key additions over the original design:
- `otel.headers` (map)
- `otel.metrics.units` / `otel.metrics.containers` (glob allowlists)
- `limits.max_lines` (default 1000), `limits.max_tail` (default 1000), `limits.max_follow_secs` (default 60)
- Validation: reject `auth.enabled = false` when `bind` is not loopback.
- **P0-5 config.example.toml.** Document every key with a comment and its env-var name.
- **P0-6 DTOs** in `src/sources/mod.rs`:
- `UnitSummary`, `UnitStatus` (service fields `Option`), `TimerInfo`
- `JournalQuery`, `JournalEntry`, `Priority` (0–7, parse from name or number)
- `ContainerSummary`, `ContainerStatus`, `LogLine { stream: Stdout|Stderr, ts, message }`
- All derive `Serialize`, `JsonSchema` and `Debug`. Timestamps serialize as RFC3339.
- **P0-7 Traits + fakes.** `SystemdSource`, `JournalSource`, `DockerSource`. Follow methods return `mpsc::Receiver` or a boxed `Stream`. `fake.rs` provides fixture-backed implementations for tests (DEC-2).
- **P0-8 error.rs.** `AppError { NotFound, BadRequest, Unauthorized, Unavailable, Internal }`. Implement `IntoResponse` (status + `{"error": {"code", "message"}}`) and `From<AppError> for rmcp::ErrorData`.
- **P0-9 state.rs.** `AppState { config: Arc<Config>, systemd: Arc<dyn SystemdSource>, journal: Arc<dyn JournalSource>, docker: Arc<dyn DockerSource>, db: Option<SqlitePool>, shutdown: CancellationToken }`. Must be `Clone`.
- **P0-10 Stubs with final signatures:**
- `api::router(AppState) -> Router`
- `mcp::service(AppState) -> StreamableHttpService<…>`
- `telemetry::init(&Config) -> Result<TelemetryGuard>`
- `auth::layer(AppState)`
- `auth::tokens::run(TokenCmd, &Config) -> Result<()>`
- `db::open(&Config) -> Result<SqlitePool>`
- Source constructors in `sources/{systemd,journal,docker}.rs`.
- `main.rs` dispatches `slinuxm serve` (default), `slinuxm token …`, and `--print-config`.
- **P0-11 Tests.** Defaults only, file overrides defaults, env overrides file, CLI overrides env. The auth-off + `0.0.0.0` combination is an error.
## Done when
- `cargo run -- serve --print-config` prints the merged config, and precedence works by hand.
- `cargo build` produces `target/debug/slinuxm`.
- `cargo clippy -- -D warnings`, `cargo fmt --check` and `cargo test` are clean.
- `cargo tree -i openssl` and `cargo tree -i openssl-sys` print nothing.
- Spike findings are recorded below and DEC-8 is updated.
## Risks
- Changing DTOs after Phase 1 starts causes merge pain in four places. Spend the time here.
- rmcp moves fast. Pin the exact version and note it in the log.
## Log
- 2026-10-15: Card created from the design review (see Design notes in ARCHITECTURE.md). DEC-1 decided: CLI is `slinuxm`.
- 2026-10-15: Implemented on branch `p0/foundation`. DEC-2 to DEC-7 accepted with the proposed defaults. DEC-8 resolved by the spike (below). Card moved to Review.
### Pinned versions (Cargo.lock is committed)
| Crate | Version | Notes |
|---|---|---|
| tokio / tokio-util / futures | 1.53.3 / 0.7.20 / 0.3.34 | |
| axum / tower / tower-http | 0.8.9 / 0.5.3 / 0.7.1 | `{name}` path syntax |
| rmcp | 3.5.1 | features `server`, `macros`, `schemars`, `transport-streamable-http-server` |
| zbus / zbus_systemd | 5.19.0 / 0.26200.0 | `zbus_systemd` features `systemd1` + `zbus-async-tokio` |
| systemd (journal) | 0.10.1 | features `journal`, `systemd_v245` (namespaces); default `bus` off |
| bollard | 0.21.1 | default features |
| opentelemetry / _sdk / -otlp | 0.33.1 | `tracing-opentelemetry` 0.34.0 |
| sqlx | 0.9.0 | `runtime-tokio`, `sqlite`, `migrate`, `macros` |
| serde / schemars / jiff | 1.0.229 / 1.2.3 (`jiff02`) / 0.2.38 | |
| figment / clap / toml | 0.10.19 / 4.6.7 / 1.1.8 | |
| sha2 / rand / base64 | 0.11.1 / 0.10.3 / 0.23.1 | newer major APIs than older docs |
### Spike findings (P0-3)
- **rmcp 3.5.1.** `StreamableHttpService::new(service_factory: impl Fn() -> Result<S, std::io::Error>, session_manager: Arc<M>, config: StreamableHttpServerConfig)`. Service type is `StreamableHttpService<S, LocalSessionManager>` (`rmcp::transport::streamable_http_server::{StreamableHttpService, StreamableHttpServerConfig, session::local::LocalSessionManager}`). It is a `tower_service::Service<Request<B>>` with `Error = Infallible`, so `Router::nest_service("/mcp", svc)` works. Config is `#[non_exhaustive]`: build with `StreamableHttpServerConfig::default()` + `with_*`. Fields: `sse_keep_alive`, `sse_retry`, `legacy_session_mode` (default true), `json_response`, `cancellation_token`, `allowed_hosts` (default loopback names), `allowed_origins` (default empty = off), `session_store`, `max_request_body_bytes`, `stateless_protocol_metadata_required`. Note the plan's `with_json_response(false)` still exists. `ServerHandler::get_info` returns `rmcp::model::ServerConfig` (`ServerInfo` is deprecated). Errors: `rmcp::ErrorData` with `resource_not_found`, `invalid_params`, `invalid_request`, `internal_error` constructors. **Host and Origin validation are built in (DEC-8).**
- **zbus_systemd 0.26200.0.** Proxies from `zbus_systemd::systemd1`: `ManagerProxy` (default service and path set), `UnitProxy`, `ServiceProxy`, `TimerProxy` (the last three have `assume_defaults = false`, so build with `.builder(&conn).path(unit_path)`). `ListUnits` returns `Vec<(name, description, load_state, active_state, sub_state, followed, unit_path: OwnedObjectPath, job_id: u32, job_type, job_path)>`; `list_units_filtered(states)` and `list_units_by_patterns(states, patterns)` exist. `get_unit` and `load_unit` return `OwnedObjectPath`. Unit props: `description`, `load_state`, `active_state`, `sub_state`, `active_enter_timestamp: u64` (usec). Service: `exec_main_status: i32`, `n_restarts: u32`, `result: String`. Timer: `unit: String`, `next_elapse_u_sec_realtime: u64`, `last_trigger_u_sec: u64`. There is no `ListTimers`. Timestamps are microseconds since epoch; 0 and `u64::MAX` mean unset. Needs the `zbus-async-tokio` feature to run on tokio.
- **systemd 0.10.1 journal.** `journal::OpenOptions::default().system(true).open()` and `.open_namespace(ns)` (feature `systemd_v245`) return `Journal`. `Journal` is **not `Send` nor `Sync`**: open it inside the `spawn_blocking` thread and keep it there. `wait(Option<Duration>) -> JournalWaitResult { Nop, Append, Invalidate }` supports a timeout. Other API: `seek_tail`, `previous_skip`, `next_entry`/`previous_entry` returning `JournalRecord = BTreeMap<String, String>`, `match_add(key, val)`, `match_or`, `match_and`, `timestamp_usec`, `cursor`. `next_entry` yields string values, so binary `MESSAGE` fields may need `get_data` for raw bytes.
- **bollard 0.21.1.** Default features (`http`, `pipe`) give the unix socket with no TLS, and no rustls or openssl enters the tree. `Docker::connect_with_unix(path, timeout, &API_DEFAULT_VERSION)` checks that the socket exists at connect time, hence lazy construction (G9).
- **opentelemetry-otlp 0.33.1.** Builders via `SpanExporter::builder()`, `MetricExporter::builder()`, `LogExporter::builder()`, then `.with_tonic()` or `.with_http()`. `WithExportConfig::{with_endpoint, with_protocol}`; `WithHttpConfig::with_headers(HashMap<String,String>)`; `WithTonicConfig::with_metadata(MetadataMap)`. The `http-proto` feature alone has **no HTTP client**; an explicit client feature is needed. We enabled `reqwest-rustls` (plus `tls-ring`, `tls-webpki-roots` for gRPC TLS). This pulls `aws-lc-sys` (C build) via reqwest but no openssl. If aws-lc is a problem, `hyper-client` gives plain-http only.
- **Cargo tree.** `cargo tree -i openssl` and `-i openssl-sys` print "did not match any packages".
### Contract notes and deviations
- Added `src/lib.rs`. The crate is lib + bin so unused stub items are not dead-code errors under `clippy -D warnings`. Surfaces import from `skyg_linux_tracker::...`.
- Added `telemetry::spawn_exporters(AppState) -> Vec<JoinHandle<()>>` beyond the listed `telemetry::init`, because metrics polling and log fan-out need the sources.
- Added `AppState::live(config, db, shutdown)` and `sources::fake::fake_state(config)` constructors.
- Source constructors are infallible and lazy: `DbusSystemd::new(&Config, CancellationToken)`, `SdJournal::new(..)`, `BollardDocker::new(..)`. Follow tasks get the shutdown token at construction, so trait `follow` methods take no token.
- `docker_logs` is split into `logs` (snapshot) and `follow_logs` (stream) on `DockerSource`.
- Added `server.allowed_hosts` / `server.allowed_origins` config keys for DEC-8.
- `Config` has no `Option` fields, so it serializes to TOML. Empty string or list means unset.
- `serve` currently validates config then exits with an error "server wiring is not implemented yet (P3 I1)". `--print-config` works with any subcommand and redacts `otel.headers` values.
- `config.example.toml` was drafted by goose (openrouter, gpt-4o-mini), then corrected by hand (invalid TOML on one line, comment layout). A test asserts it equals `Config::default()`.
+63
View File
@@ -0,0 +1,63 @@
# P3 — Integration & hardening
| Field | Value |
|---|---|
| Status | Done (I6 #5 deferred to Backlog) |
| Owner | Ari |
| Phase | 3 (single track) |
| Depends on | WS-E, WS-F, WS-G (and transitively A–D) |
| Blocks | Release |
| Owned files | `src/main.rs` (serve wiring), `systemd/skyg-linux-tracker.service`, `README.md`, `tests/` |
## Goal
Wire every surface into one axum server, prove the negative paths, and ship the unit file and docs. This is the only place the surfaces meet.
## Tasks
Tick these in [`TODO.md`](../../TODO.md).
- **I1** `slinuxm serve` wiring in `main.rs`:
1. Load config, then `telemetry::init`, then `db::open` when auth is enabled, then construct sources and `AppState`.
2. Build the protected router: `.nest("/api/v1", api::router(state))` and `.nest_service("/mcp", mcp::service(state))`, then `.layer(auth::layer(state))`.
3. Build the public router with `/healthz`.
4. `public.merge(protected).layer(TraceLayer::new_for_http())` (review R3).
5. Spawn the metrics and log export tasks when enabled.
6. On SIGINT or SIGTERM: cancel `state.shutdown` so SSE streams and background tasks end, run `axum::serve(...).with_graceful_shutdown`, then drop the telemetry guard to flush OTel (review G3).
- **I2** Integration tests in `tests/` covering Verification #6:
- A missing or invalid token gets 401 on `/api/v1/*` and `/mcp`. `/healthz` returns 200 without a token.
- `auth.enabled = false` with `bind = 0.0.0.0` makes startup exit non-zero with a clear message.
- A missing Docker socket gives 503 on docker routes while systemd routes still return 200.
- SIGTERM with an open SSE client exits within a few seconds.
- **I3** `systemd/skyg-linux-tracker.service`:
- `ExecStart=/usr/local/bin/slinuxm serve --config /etc/skyg-linux-tracker.toml`
- A dedicated `User=` (for example `skyg`), or `DynamicUser=yes` if the token CLI story still works. Decide and note it.
- `StateDirectory=skyg-linux-tracker`, `StateDirectoryMode=0700`
- `ProtectSystem=strict`, `ProtectHome=yes`, `PrivateTmp=yes`, `NoNewPrivileges=yes`, `ProtectKernelTunables=yes`, `ProtectControlGroups=yes`, `RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6`, `LockPersonality=yes`, `MemoryDenyWriteExecute=yes`
- `SupplementaryGroups=systemd-journal`. Add `docker` commented out with a root-equivalence warning (DEC-6).
- **I4** `README.md`:
- What it is and how to build it (`devenv shell`, `cargo build --release`).
- Install: binary, config, unit file.
- Config reference, or a pointer to `config.example.toml` and the env-var naming.
- Token workflow: `sudo -u skyg slinuxm token create <name>`.
- REST endpoint list, and MCP client setup with a bearer header example.
- OTel setup and the hardened Docker socket-proxy option.
- **I5** Release build checks:
- `cargo build --release`.
- `ldd target/release/slinuxm` lists only libc, libsystemd and libsystemd's own deps.
- Record the binary size in the log.
- **I6** Run Verification #1–#6 from the [verification checklist](../../ARCHITECTURE.md#verification) by hand on a real systemd + Docker host and record the results in the log.
## Done when
- I2 tests pass in `cargo test`.
- I6 results are logged. #1–#4 and #6 pass; #5 (OTel) is deferred to the Backlog (**I6-5**) because the external collector on minipc1 is down.
- The unit file starts the service on a clean host with only the binary, config and unit installed (met on minipc1, 2026-10-19).
## Log
- 2026-10-15: Card created.
- 2026-10-15: Implemented on `p3/integration`. I1-I5 done; I6 partial. Router assembly lives in `src/server.rs` (`build_router`, `serve_listener`) so tests and `main` share it. Shutdown: signal -> cancel token -> graceful serve ends -> await exporter handles -> `guard.shutdown().await` -> close pool. `token` subcommand does not init telemetry. Unit uses a static `User=skyg` (not DynamicUser) so `sudo -u skyg slinuxm token ...` hits the same DB. I2: 8 tests in `tests/integration.rs` (raw HTTP client, no new deps). Release binary 19 MB; `ldd`: libsystemd, libgcc_s, libm, libc only.
- I6 live results (release binary, loopback, real token via CLI, real systemd/journal/docker on this host): #1 build/tests pass; #2 serve + token create pass; #3 REST incl. `curl -N` journal SSE pass (docker log follow SSE not exercised live); #4 MCP initialize, tools/list, prompts/list, resources/list, 5 tool calls, no-token 401, bad Origin 403 pass via curl (MCP Inspector not run; not all 8 tools called); #5 OTel NOT RUN (otelcol-contrib not on PATH, devenv shell not started via `devenv up`); #6 pass via tests (401, auth-off+0.0.0.0 refuses, SIGTERM with open SSE exits 0 in <1s); missing-docker-socket 503 covered by WS-E fake test, not live. Unit file not installed on a clean host (only `systemd-analyze verify`, which cannot fully run here).
- 2026-10-15: Review follow-ups applied. Shutdown drain and exporter awaits are bounded (10s each) and the unit uses TimeoutStopSec=30; new test: SIGTERM with an open MCP session and an in-flight 60s `tail_journal` exits 0. Unit: SystemCallFilter, ProtectProc=invisible, ProcSubset=pid, RestrictSUIDSGID, UMask=0077; docker.service is `After=` only. Hardening was run via `systemd-run --user` with the same sandbox properties (REST systemd/journal/docker and SIGTERM all worked); not run as a system unit with User=skyg/StateDirectory. Host allowlist guard on `/api/v1` when auth is disabled (test). `server.port = 0` rejected (test). Integration tests extended (13 tests): unmatched paths (401 with auth on, route existence not disclosed), POST 401/405, GET/DELETE /mcp without token, revoked token, Basic scheme, good Origin, missing docker socket (503 vs 200). SIGTERM tests use env_clear; the pick-port-then-drop race remains (port 0 is now rejected by config).
- 2026-10-19: I6 re-run on minipc1 (real NixOS host) through the installed `skyg-linux-tracker.service` (in `/etc/systemd/system`, enabled, active since 15:58, 0 restarts). All 8 MCP tools were called live and passed: `list_systemd_units`, `list_containers`, `list_systemd_timers`, `get_journal_logs` (priority + lines), `get_systemd_unit` (including the `not_found` error), `get_container`, `get_container_logs`, `tail_journal`. Verification #1–#4 and #6 pass; the I3 clean-host install is satisfied. **#5 OTel is deferred to the Backlog (I6-5)**: `opentelemetry-collector.service` on minipc1 is failed (`var-lib-loki.mount` also failed), so `slinuxm` logs `TonicLogsClient export failed ... Connection refused`. `slinuxm` itself exports fine; the fault is external (X-1). Not covered live: docker log-follow SSE (B-1), REST/auth/TLS (B-2). Whether minipc1 is deployed from this repo's `nixosModules.default` is unknown. Card moved to Done.
+43
View File
@@ -0,0 +1,43 @@
# WS-A — systemd source
| Field | Value |
|---|---|
| Status | Done |
| Owner | — |
| Phase | 1 (parallel) |
| Depends on | P0 |
| Blocks | WS-E, WS-F, WS-G (to finish) |
| Owned files | `src/sources/systemd.rs`, `examples/systemd_dump.rs` |
## Goal
Implement `SystemdSource` over D-Bus with `zbus_systemd`. Every call hits the bus live, with no caching.
## Tasks
Tick these in [`TODO.md`](../../TODO.md).
- **A1** Create one `zbus::Connection::system()` at construction and share `ManagerProxy`.
- **A2** `list_units(state: Option<…>)`. Use `ListUnitsFiltered(states)` when a state is given, otherwise `ListUnits`. Map the tuple to `UnitSummary` (name, description, load/active/sub state).
- **A3** `list_timers()`. The Manager has no `ListTimers` method (review R1). Call `ListUnitsByPatterns([], ["*.timer"])`, then for each timer read `TimerProxy`: `NextElapseUSecRealtime`, `LastTriggerUSec`, `Unit` (the triggered unit). Convert µs since epoch to RFC3339. `0` or `u64::MAX` means `None`.
- **A4** `unit_status(name)`. Try `GetUnit(name)` first, falling back to `LoadUnit` only on `NoSuchUnit`. The fallback lets inactive unloaded units still report without side effects on loaded ones.
- `Unit` props: `ActiveState`, `SubState`, `LoadState`, `Description`, `ActiveEnterTimestamp`, `FragmentPath`.
- `Service` props, only when the name ends in `.service`: `ExecMainStatus`, `NRestarts`, `MainPID`.
- **A5** Validate names against systemd's charset (`[A-Za-z0-9:_.@\-\\]`, max 255) and return `BadRequest`. Map D-Bus `NoSuchUnit` to `NotFound`. Map bus connection failure to `Unavailable`.
- **A6** `examples/systemd_dump.rs` prints JSON for units, failed units, timers and one unit's status.
- **A7** Unit tests for µs-to-RFC3339 conversion, sentinel handling and tuple mapping.
## Done when
- `cargo run --example systemd_dump` prints sane JSON on a real host. Timers show next elapse times that match `systemctl list-timers`.
- `unit_status("nonexistent.service")` returns `NotFound` and `unit_status("../etc")` returns `BadRequest`.
## Notes
- Property reads cost one D-Bus round-trip each. Fetch with `GetAll` where zbus_systemd exposes it, or `join!` the reads.
- Unprivileged reads on the system bus are allowed. No polkit involvement is needed for these calls.
## Log
- 2026-10-15: Card created.
- 2026-10-15: Implemented on branch `ws-a/systemd-source`. Gates pass; example verified on live bus (timers match `systemctl list-timers`, NotFound/BadRequest verified). Beyond spec: LoadUnit result with load_state `not-found` also maps to NotFound; `unit_file_state`/`active_exit_time` populated.
+49
View File
@@ -0,0 +1,49 @@
# WS-B — journal source
| Field | Value |
|---|---|
| Status | Done |
| Owner | — |
| Phase | 1 (parallel) |
| Depends on | P0 (incl. spike findings on the `systemd` crate) |
| Blocks | WS-E, WS-F, WS-G (to finish) |
| Owned files | `src/sources/journal.rs`, `examples/journal_tail.rs` |
## Goal
Implement `JournalSource` over sd-journal via the `systemd` crate. It provides a bounded snapshot query and a cancellable live follow.
## Tasks
Tick these in [`TODO.md`](../../TODO.md).
- **B1** Run every sd-journal call inside `tokio::task::spawn_blocking` (review G1). Open the journal inside the blocking closure and never hold it across `.await`. Use `sources.journal_namespace` if the spike found namespace support. Otherwise log a warning when it is set and use the default journal.
- **B2** `query(JournalQuery)`:
- Unit filter via `match_add("_SYSTEMD_UNIT", unit)`.
- Priority filter `≤ N` by adding one `PRIORITY=k` match for each k in 0..=N. Matches on the same field are OR'ed by sd-journal.
- `seek_tail`, then step backward `lines` entries. `lines` is clamped to `limits.max_lines`.
- Reverse the result so it returns oldest-first.
- **B3** `follow(JournalQuery, CancellationToken) -> mpsc::Receiver<JournalEntry>`:
- Seek to tail and run a blocking loop of `wait(Some(500ms))`, then drain new entries with `next`.
- Each iteration checks `tx.is_closed()` and `token.is_cancelled()` and exits if either is true (review G2).
- Use `blocking_send` with a bounded channel, around 1024 entries. Decide on drop-or-block for slow consumers and document it.
- **B4** Map `__REALTIME_TIMESTAMP`, `PRIORITY`, `_SYSTEMD_UNIT`, `SYSLOG_IDENTIFIER`, `_PID`, `_HOSTNAME` and `MESSAGE` to `JournalEntry`. `MESSAGE` may be binary, so convert it with lossy UTF-8.
- **B5** `examples/journal_tail.rs --unit X [--priority err] [--lines N] [--follow]`.
- **B6** Test that dropping the receiver makes the blocking thread exit within about one second. Use a counter or `JoinHandle` to prove it.
## Done when
- The example snapshot output matches `journalctl -u X -p err -n N -o json` for the same filters.
- Follow prints new entries live. `logger -t test hello` shows up in under a second.
- B6 passes.
## Notes
- Reading the full system journal needs the `systemd-journal` group (or root). Without it you only see your own user's entries. Say so in the example's error output.
- If the `systemd` crate turns out unmaintained or broken during the spike, record the alternative in the P0 log before starting.
## Log
- 2026-10-15: Card created.
- 2026-10-15: Implemented B1-B6 on branch ws-b/journal-source. Query and follow verified on the live host. Unit filter matches `_SYSTEMD_UNIT` only; `UNIT` is used only as a fallback when mapping entries. Follow applies backpressure (no drops). `since` is applied during the backward backlog scan.
- 2026-10-15: Review fixes: unit filter is now (_SYSTEMD_UNIT=u OR UNIT=u), with PRIORITY matches repeated per disjunct. Verified against `journalctl -u hyprland-headless.service` style PID1 lines (same 5 entries, same timestamps). Offline unit tests cover match plan, ordering, `since`, and send exit on drop/cancel. Live follow tests are #[ignore]: `cargo test -- --ignored live_`.
+45
View File
@@ -0,0 +1,45 @@
# WS-C — docker source
| Field | Value |
|---|---|
| Status | Done |
| Owner | — |
| Phase | 1 (parallel) |
| Depends on | P0 |
| Blocks | WS-E, WS-F, WS-G (to finish) |
| Owned files | `src/sources/docker.rs`, `examples/docker_ps.rs` |
## Goal
Implement `DockerSource` over the Docker or Podman API with `bollard` on a unix socket. A missing Docker never breaks the rest of the service.
## Tasks
Tick these in [`TODO.md`](../../TODO.md).
- **C1** Construct with `Docker::connect_with_unix(socket, timeout, API_DEFAULT_VERSION)`. Construction must not fail when the socket is absent (review G9). Map connection-refused, not-found and permission-denied to `AppError::Unavailable` with a clear message on each call.
- **C2** `ps()`:
- `list_containers(all = true)`.
- Then `inspect_container` for each one concurrently, bounded with `buffer_unordered(8)`.
- Map `State.Health.Status`, `State.StartedAt` (to uptime seconds), `RestartCount`, image, and status.
- **C3** `container_status(name)` uses inspect only. A 404 maps to `NotFound`.
- **C4** `logs(name, follow, tail, since, token)` returns a boxed `Stream<Item = Result<LogLine>>`. Use `stdout`, `stderr` and `timestamps = true`, parse the timestamp prefix, and keep the stream type (stdout or stderr). The stream ends when dropped or when the token fires.
- **C5** Clamp `tail` to `limits.max_tail`.
- **C6** `examples/docker_ps.rs` prints `ps` as JSON. `--logs NAME [--follow]` streams logs.
- **C7** *(nice-to-have)* Run the example against `/run/podman/podman.sock` and note any field differences.
## Done when
- The example matches `docker ps -a` and shows health and uptime for containers with healthchecks.
- Following logs streams live, and Ctrl-C exits cleanly.
- With `docker_socket` pointed at a missing path, every call returns `Unavailable` and nothing panics.
## Notes
- Container names from the API have a leading `/`. Strip it in DTOs and accept names with or without it.
- `docker logs` only works for the `json-file`, `local` and `journald` log drivers. Others return an API error, which should map to `BadRequest` with the driver named.
## Log
- 2026-10-15: Card created.
- 2026-10-15: Implemented C1-C6 on branch `ws-c/docker-source`. Verified against the real docker socket (ps, logs, follow). C7 skipped: no Podman socket on this host. `ps` uses `buffered(8)` (ordered) instead of `buffer_unordered`. Names are validated to `[A-Za-z0-9_.-]`. Non-readable log drivers are rejected up front from inspect (BadRequest naming the driver).
+54
View File
@@ -0,0 +1,54 @@
# WS-D — auth + DB
| Field | Value |
|---|---|
| Status | Done |
| Owner | — |
| Phase | 1 (parallel) |
| Depends on | P0 |
| Blocks | WS-E, WS-F (to finish), P3 |
| Owned files | `src/db.rs`, `src/auth/mod.rs`, `src/auth/tokens.rs`, `migrations/` |
## Goal
Provide token storage in SQLite, the `slinuxm token` CLI, and one bearer-auth middleware that protects both `/api/v1` and `/mcp`.
## Tasks
Tick these in [`TODO.md`](../../TODO.md).
- **D1** `db::open`:
- Create the parent dir with mode `0700` if it is missing. In production, systemd `StateDirectory=` already creates it.
- Use `SqliteConnectOptions` with `create_if_missing`, `journal_mode(Wal)` and `busy_timeout(5s)`.
- Run `sqlx::migrate!()` from embedded `migrations/`.
- **D2** First migration: `tokens(id INTEGER PK, name TEXT NOT NULL UNIQUE, token_hash TEXT NOT NULL UNIQUE, created_at TEXT NOT NULL, revoked_at TEXT, last_used_at TEXT)`.
- **D3** `tokens.rs` with runtime `sqlx::query` / `query_as` (DEC-3):
- `create(name)` generates 32 bytes from the OS RNG and formats the token as `slinuxm_<base64url>`. Store `hex(sha256(token))` and print the plaintext once with a warning.
- `list()` shows name, created, revoked and last-used, never hashes.
- `revoke(name)` sets `revoked_at`. Revoking twice is a no-op with a message.
- **D4** Implement `auth::tokens::run(TokenCmd, &Config)` behind `slinuxm token create|list|revoke`. The `main.rs` dispatch already exists from P0, so do not edit `main.rs`.
- **D5** Middleware:
- Extract `Authorization: Bearer <t>`, hash it, and look up `WHERE token_hash = ? AND revoked_at IS NULL`.
- On success, optionally update `last_used_at`. Throttle that update to at most once a minute per token, or skip it.
- On failure, return `401` with `WWW-Authenticate: Bearer` and the standard JSON error body. Use the same response for `/mcp` (DEC-4).
- When `auth.enabled = false`, the layer is a pass-through.
- **D6** Expose the middleware so P3 applies it to the protected router only. `/healthz` must stay reachable without a token (review R3).
- **D7** Tests:
- Use an in-memory or tempdir DB.
- Create a token and confirm it is accepted. Revoke it and confirm it is rejected. Garbage and missing headers are rejected.
- Re-assert the P0 startup refusal for auth-off + non-loopback bind.
## Done when
- `slinuxm token create ci` prints a token. `slinuxm token list` shows it. A request with it passes the middleware.
- After `slinuxm token revoke ci`, the same request returns 401 without a server restart.
- D7 passes.
## Notes
- The CLI writes the same DB the server reads, so WAL mode is required. In production, run the CLI as the service user, for example `sudo -u skyg slinuxm token create …`. Document this in the README (I4).
## Log
- 2026-10-15: Card created.
- 2026-10-15: Implemented on branch `ws-d/auth-db`. Token fns take `&SqlitePool` (`create`, `list`, `revoke`, `verify`); `db::migrate` is public for in-memory test pools. `last_used_at` is throttled to once a minute. Lookup is by sha256 hex hash with `slinuxm_` prefix pre-check.
+51
View File
@@ -0,0 +1,51 @@
# WS-E — REST API
| Field | Value |
|---|---|
| Status | Done |
| Owner | — |
| Phase | 2 (parallel) |
| Depends on | P0 to start (fakes). A, B, C, D to finish. |
| Blocks | P3 |
| Owned files | `src/api/mod.rs`, `src/api/systemd.rs`, `src/api/journal.rs`, `src/api/docker.rs` |
## Goal
Build thin, read-only GET handlers over `AppState` sources, with SSE for follow endpoints. `api::router(state)` returns a router that P3 nests under `/api/v1`.
## Tasks
Tick these in [`TODO.md`](../../TODO.md).
- **E1** `api::router(AppState) -> Router`. Use axum 0.8 path syntax `/{name}`, because `:name` panics (review R4). GET only.
- **E2** Systemd routes:
- `/systemd/units?state=failed|active|…`
- `/systemd/units/{name}`
- `/systemd/timers`
- **E3** `/journal?unit=&priority=err|3&lines=`. Parse priority by name or number. Clamp `lines` and return 400 on bad input.
- **E4** `/journal/follow?unit=&priority=` returns `Sse` built from the source receiver:
- `KeepAlive::default()`.
- The stream ends on client disconnect, which drops the receiver and stops the blocking thread, or on `state.shutdown` (review G3).
- **E5** Docker routes:
- `/docker/ps`
- `/docker/containers/{name}`
- `/docker/containers/{name}/logs?tail=&since=&follow=`: JSON array when `follow=0`, SSE when `follow=1`. Each SSE event's `event:` is `stdout` or `stderr`.
- **E6** Provide a public `healthz` handler for P3 to mount outside auth. It returns `{"status":"ok","version":…}`.
- **E7** All errors go through `AppError`, giving one JSON shape everywhere.
- **E8** Tests with `fake.rs` sources via `tower::ServiceExt::oneshot`. Cover each route's happy path, 400, 404, and one SSE stream that yields events and then ends.
- **E9** When the fake docker returns `Unavailable`, docker routes return 503 and systemd and journal routes still return 200.
## Done when
- E8 and E9 pass.
- With real sources wired by P3, Verification #3 curl checks pass, including `curl -N` on both SSE endpoints.
## Notes
- Handlers stay thin: parse, call the source, serialize. No business logic.
- `tower-http` `TraceLayer` is applied in P3, not here.
## Log
- 2026-10-15: Card created.
- 2026-10-15: Implemented on branch `ws-e/rest-api`. E1-E9 done. Query errors, missing routes and source errors all use the `AppError` JSON shape. `lines`/`tail` are rejected at 0 and clamped to `limits.max_*`; `priority` parsed via `Priority::from_str`; `state` validated by the source (400). SSE ends on disconnect, `state.shutdown`, or `limits.max_follow_secs`. Journal SSE event name is `journal`; docker is `stdout`/`stderr`; data is the JSON DTO. `api::healthz` is public for P3 to mount at `/healthz`. `follow` and `all` accept 0/1/true/false.
+62
View File
@@ -0,0 +1,62 @@
# WS-F — MCP server
| Field | Value |
|---|---|
| Status | Done |
| Owner | — |
| Phase | 2 (parallel) |
| Depends on | P0 to start (fakes, spike findings). A, B, C, D to finish. |
| Blocks | P3 |
| Owned files | `src/mcp/mod.rs`, `src/mcp/prompts.rs`, `src/mcp/resources.rs` |
## Goal
Build an `rmcp` Streamable HTTP server exposing the same data as REST, plus agent-facing context: instructions, rich descriptions, prompts and resources. `mcp::service(state)` returns the Tower service that P3 mounts at `/mcp`.
## Tasks
Tick these in [`TODO.md`](../../TODO.md).
- **F1** `TrackerService` holding `AppState`, with `#[tool_router]`. There are 8 tools (review R5):
- `list_systemd_units { state? }`
- `get_systemd_unit { name }`
- `list_systemd_timers {}`
- `get_journal_logs { unit?, priority?, lines }`
- `tail_journal { unit, seconds }`
- `list_containers {}`
- `get_container { name }`
- `get_container_logs { name, tail }`
- Parameter structs derive `Deserialize` and `JsonSchema`.
- **F2** `tail_journal` clamps `seconds` to `limits.max_follow_secs` and stops early at an entry cap, for example `limits.max_lines`. It returns whatever it collected and never streams indefinitely.
- **F3** `get_container_logs` clamps `tail` to `limits.max_tail`.
- **F4** `ServerHandler::get_info` sets server name/version, capabilities for tools, prompts and resources, and **instructions**. The instructions describe the workflow: start with failed units or unhealthy containers, then pull logs for the specific unit or container, and use timers to explain scheduled jobs.
- **F5** `#[tool(description = …)]` strings explain:
- active/sub states (`failed`, `activating`, `auto-restart`)
- journal priority numbers and names
- Docker health states (`starting`, `healthy`, `unhealthy`, `none`)
- **F6** Prompts in `prompts.rs`:
- `diagnose_unit { name }`: status, then recent err+ logs, then restart count, then a suggested next look.
- `container_health_report {}`: list, then unhealthy and restarting containers, then their logs.
- **F7** Resources in `resources.rs`: `systemd://units/failed` and `docker://containers`. Implement list and read, resolved live on each read, with no subscriptions (DEC-5).
- **F8** `mcp::service(state)` builds `StreamableHttpService::new(|| Ok(TrackerService::new(state.clone())), LocalSessionManager::default().into(), config)`. Apply Origin validation per DEC-8.
- **F9** Tests against fake sources:
- A snapshot of the tool list (names + schemas), so description changes are reviewed deliberately.
- Each tool's happy path.
- `AppError` maps to MCP errors.
- `tail_journal` respects its time cap.
## Done when
- F9 passes.
- With P3 wiring, Verification #4 passes. MCP Inspector at `http://localhost:1984/mcp` with a bearer header lists 8 tools, 2 prompts and 2 resources, and every tool returns data. Without the header the client receives 401.
## Notes
- Tool handlers must not duplicate logic from `api/`. Both call `sources/` directly.
- Keep tool output JSON-shaped (serialized DTOs) so agents can parse it, with a short text summary line where it helps.
## Log
- 2026-10-15: Card created.
- 2026-10-15: Implemented on branch `ws-f/mcp-server`. Prompts and resources are hand-written `ServerHandler` methods (helpers in `prompts.rs`/`resources.rs` are testable without a request context). `tail_journal` follows with a 10-entry backlog, stops at `max_follow_secs` or `max_lines`. Origin validation is enforced (`enforce_origin_validation`) with loopback and bind origins on `server.port` plus configured extras. Tests use real time (1s cap) because tokio `test-util` is not enabled in Cargo.toml.
- 2026-10-15: Review fixes. Tool handlers return source errors as `CallToolResult::error` (is_error) with an actionable message; prompts/resources keep protocol errors. Log tools return `{entries, truncated, note}` capped at 256 KiB. `tail_journal` notes reader-closed, shutdown and entry-cap stops; `seconds` defaults to 10 (schema 0-3600, server clamps to `max_follow_secs`). Log content is documented as untrusted data; `diagnose_unit` validates `name`. Notes: rmcp accepts requests with no `Origin` header, so bearer auth is the real gate; Origin/Host checks defend against browser DNS-rebinding. With `server.bind = 0.0.0.0`, LAN clients need their Host names in `server.allowed_hosts` (a startup warning is logged otherwise).
+58
View File
@@ -0,0 +1,58 @@
# WS-G — OpenTelemetry export
| Field | Value |
|---|---|
| Status | Done |
| Owner | — |
| Phase | 2 (parallel) |
| Depends on | P0 to start (fakes, spike findings). A, B, C to finish. |
| Blocks | P3 |
| Owned files | `src/telemetry/mod.rs`, `src/telemetry/metrics_export.rs`, `src/telemetry/logs_export.rs`, `dev/otelcol.yaml` |
## Goal
Export traces, metrics and logs over a single OTLP exporter config. This is the only push path in the system. Metrics come from a poll loop, and logs fan out from the journal and Docker follow streams.
## Tasks
Tick these in [`TODO.md`](../../TODO.md).
- **G1** `telemetry::init(&Config) -> TelemetryGuard`:
- Build tracer, meter and logger providers from one exporter config: `otel.endpoint`, `otel.protocol` (`grpc` or `http`) and `otel.headers`.
- Resource attributes: `service.name`, `service.version`, `host.name`.
- When `otel.enabled = false`, return a no-op guard.
- **G2** Tracing subscriber: `fmt` to stderr (journald captures it under the unit) plus `tracing-opentelemetry` when enabled. Do **not** route the app's own logs into the OTel logs pipeline. That pipeline is for host logs.
- **G3** Metrics (review G11):
- A background task runs every `metrics_interval_secs`, calls `list_units`, `list_timers` and `ps`, and stores the result in an `ArcSwap` or `RwLock` snapshot.
- Observable gauges read the latest snapshot in their callbacks, so units or containers that disappear stop being reported.
- **G4** Metric set from [ARCHITECTURE.md](../../ARCHITECTURE.md#opentelemetry-telemetry):
- `systemd.unit.active`, `systemd.unit.failed`, `systemd.unit.restarts`, `systemd.timer.next_elapse_seconds`
- `docker.container.running`, `docker.container.healthy`, `docker.container.uptime_seconds`, `docker.container.restarts`
- Attribute names: `systemd.unit`, `systemd.sub_state`, `systemd.load_state`; `container.name`, `container.id`, `container.image.name` (semconv).
- Scope: apply `otel.metrics.units` / `otel.metrics.containers` globs. With an empty unit list, use the `*.service` + `*.timer` default (DEC-7).
- **G5** Logs (`otel.export_logs = true`, review G4):
- Journal: one `follow()` per configured unit, or one unfiltered follow when `otel.logs.journal_units` is empty. Map priority 0–7 to OTel severity (emerg/alert/crit to FATAL, err to ERROR, warning to WARN, notice/info to INFO, debug to DEBUG).
- Docker: a reconciler subscribes to Docker `events` (`start`, `die`, `destroy`). It starts and stops per-container `logs(follow)` tasks for allowlisted containers and does an initial list at startup. stderr maps to WARN and stdout to INFO.
- Every task restarts with exponential backoff on error. If Docker is unavailable, log once and retry slowly.
- Attributes: `systemd.unit` or `container.name`, plus `host.name` from the resource.
- **G6** On shutdown: cancel all tasks via `state.shutdown`, then `force_flush` and `shutdown` each provider, with a bounded timeout.
- **G7** `dev/otelcol.yaml` with an OTLP receiver (grpc 4317, http 4318) and a `debug` exporter at `verbosity: detailed`. Add a devenv `processes.otelcol` entry, coordinating the `devenv.nix` edit with the P0 owner.
## Done when
- Verification #5 passes. With `[otel] enabled = true`, the collector shows the gauges every interval and log records with correct severity and attributes.
- `logger -p user.err test` appears as an ERROR record.
- Starting a new container shows its logs without restarting the service.
- SIGTERM flushes pending records before exit.
## Notes
- opentelemetry-rust marks metrics and logs Stable and traces Beta. Pin versions together, because the `opentelemetry*` crates must match each other.
- Cardinality: one series per unit or container per attribute set. With the DEC-7 default, expect a few hundred series on a typical host.
## Log
- 2026-10-15: Card created.
- 2026-10-15: Review fixes F1-F8. Restart polling uses buffer_unordered(8) with a 15s budget; failed sections keep the previous snapshot; series carry identity attributes only; timer metric renamed `systemd.timer.next_elapse_timestamp_seconds`; units added; observations computed once per poll. New containers start from StartedAt, docker lines deduped on reconnect, own unit/identifier excluded from the journal follow (by name; cursor not used). `TelemetryGuard::shutdown().await` added; Drop is best-effort.
- P3 notes: (1) call `guard.shutdown().await` after awaiting the exporter handles. (2) devenv: add `processes.otelcol.exec = "otelcol-contrib --config dev/otelcol.yaml"` to devenv.nix (not done here, P0 owner). (3) Cargo.toml needs `reqwest-blocking-client` on opentelemetry-otlp for http/protobuf; the exporter-build test is strict and fails without it. (4) RUST_LOG/EnvFilter deferred.
- 2026-10-15: Implemented G1-G7 on `ws-g/otel`. Notes: no glob crate, so a small `*`/`?` matcher lives in `telemetry/mod.rs`. `DockerSource` has no events stream, so the log reconciler polls `ps(false)` every 5s (30s while Docker is unavailable) instead of subscribing to `events`. No `EnvFilter` (feature not enabled in Cargo.toml): level is fixed at INFO. `systemd.timer.next_elapse_seconds` exports absolute unix seconds. `systemd.unit.restarts` costs one `unit_status` per in-scope loaded service per interval. HTTP endpoints get `/v1/<signal>` appended unless already present. Live otelcol smoke test not run (no collector binary on PATH).
+65
View File
@@ -0,0 +1,65 @@
# WS-H — NixOS module + package
| Field | Value |
|---|---|
| Status | Review (gate outstanding) |
| Owner | Ari |
| Phase | 4 (single track, after release candidate) |
| Depends on | P3 (final unit hardening, `--print-config`, README); WS-I (TLS options) |
| Blocks | — |
| Owned files | `flake.nix`, `nix/package.nix`, `nix/module.nix`, `nix/tests/`, `README.md` (NixOS section only) |
## Goal
Install `skyg-linux-tracker` on NixOS as a systemd service that exposes the REST API and `/mcp` on **port 1984**, and install the `slinuxm` CLI system-wide so the operator can mint the first token.
## Tasks
Tick these in [`TODO.md`](../../TODO.md).
- **H1** `nix/package.nix`: `rustPlatform.buildRustPackage` from the committed `Cargo.lock`. `nativeBuildInputs = [ pkg-config cmake ]` (cmake for `aws-lc-sys`), `buildInputs = [ systemd ]`. Binary is `slinuxm`. Live-journal tests are `#[ignore]`, so the sandbox build runs `cargo test`.
- **H2** `flake.nix` exporting `packages.<system>.default` and `nixosModules.default`, plus `checks` wired to H8.
- **H3** `nix/module.nix` options under `services.skyg-linux-tracker`:
- `enable`, `package`
- `port` (default **1984**, same as the binary default)
- `bind` (default `"127.0.0.1"`; LAN exposure is set per install)
- `tls.enable`, `tls.certFile`, `tls.keyFile` (passed via systemd `LoadCredential`, DEC-12)
- `openFirewall` (default `false`)
- `allowedHosts` (extra Host/Origin entries for `/mcp`; the module adds `networking.hostName` automatically)
- `docker.enable` (default `false`; adds the `docker` group and sets the socket path, DEC-6)
- `settings` (freeform TOML via `pkgs.formats.toml`, merged over module defaults)
- `environmentFile` (for secrets such as `SKYG_OTEL__HEADERS__*`, kept out of the store)
- **H4** Service definition:
- dedicated user and group
- `StateDirectory=skyg-linux-tracker`, mode 0700 (matches `db_path`)
- `SupplementaryGroups=systemd-journal` (plus `docker` when enabled)
- the P3 hardening set (`NoNewPrivileges`, `ProtectSystem=strict`, `ProtectHome`, `PrivateTmp`, `RestrictAddressFamilies`, …), reusing `systemd/skyg-linux-tracker.service` rather than diverging
- config written to `/etc/skyg-linux-tracker.toml`, the default path the app already reads, so the CLI and the service always agree
- **H5** CLI install (required): `environment.systemPackages = [ cfg.package ]`, plain binary, no wrapper (DEC-10). Mint tokens with `sudo -u skyg-linux-tracker slinuxm token create <name>`; the module's NixOS notes and the README show this exact command. The root guard in `slinuxm` (token commands refuse to run as root against another user's DB and print the command above) is implemented here as a small change in the CLI code.
- **H6** Firewall: `openFirewall` opens the port globally. Also support an interface-scoped option (`networking.firewall.interfaces.<if>.allowedTCPPorts`), which is the recommended LAN setup. Assert at eval time that `bind` non-loopback is accompanied by auth enabled.
- **H7** README NixOS section: flake input, minimal config, token workflow (`sudo -u skyg-linux-tracker slinuxm token create <name>`), the exposure model (loopback + Tailscale or TLS proxy recommended, since bearer tokens over plain HTTP are readable on a LAN), `allowedHosts` for `bind = 0.0.0.0`.
- **H8** NixOS VM test (`nix/tests/`):
1. service starts and `curl :1984/healthz` returns 200 unauthenticated (a second case with TLS enabled and a test cert)
2. `/api/v1/systemd/units` returns 401 without a token
3. `sudo -u skyg-linux-tracker slinuxm token create t` prints a token, and the same request with it returns 200
4. `/mcp` initialize and tools/list work with the token
5. `slinuxm token revoke t` (same `sudo -u` form) makes the same request return 401 without a restart
6. bad Origin on `/mcp` returns 403
## Done when
- `nix build .#default` produces `bin/slinuxm`, and `ldd` shows only libc and libsystemd (plus deps). No openssl.
- `nix flake check` passes, including the VM test (H8).
- On a NixOS host, enabling the module with `port = 1984` exposes `:1984` as described, and `slinuxm token create` works from a normal login shell.
## Notes
- Port 1984 is also the binary default, so the module just passes it through.
- Tokens are printed once and never stored (DEC-11). Declarative/hash-imported tokens and activation-time `bootstrapTokens` are out of scope for v1 and are not offered.
- The module must not put secrets in the generated TOML. Anything sensitive goes through `environmentFile`.
## Log
- 2026-10-15: Card created from design discussion.
- 2026-10-19: Merged to `main` as PR #1 (34da657): `nix/package.nix`, `nix/module.nix`, `flake.nix`, `nix/tests/vm.nix`, and the README NixOS section. H1–H8 ticked in `TODO.md`. **Only the Gate remains** (`nix build`, `nix flake check` incl. the VM test, `ldd` check); it is being run and the result will be logged here. Card stays in Review until then.
- _WS-H gate result (`nix build`, `nix flake check` incl. VM test, `ldd`): pending. Append it here, then update the In progress / Review row and milestone M4 in [`TODO.md`](../../TODO.md)._
+44
View File
@@ -0,0 +1,44 @@
# WS-I — Native TLS
| Field | Value |
|---|---|
| Status | Done |
| Owner | — |
| Phase | 4 (single track, after P3, before WS-H) |
| Depends on | P3 |
| Blocks | WS-H (the NixOS module exposes the TLS options) |
| Owned files | `src/tls.rs`, `src/server.rs` (serve path only), `src/config.rs` (`server.tls` keys), `config.example.toml` (tls keys), `Cargo.toml` (tls deps, by agreed exception), `tests/tls.rs`, `README.md` (TLS section) |
## Goal
Serve REST and `/mcp` over HTTPS with certificates the operator provides, so bearer tokens are not readable on the LAN. No openssl.
## Tasks
Tick these in [`TODO.md`](../../TODO.md).
- **T1** Config: `server.tls.enabled` (default false), `server.tls.cert_file`, `server.tls.key_file`. Validation: enabled requires both paths, and both files must be readable at startup with a clear error. Document each key and its env var in `config.example.toml`.
- **T2** Serve over rustls. Pick the integration that keeps openssl out (`axum-server` with `tls-rustls`, or `tokio-rustls` with hyper-util). Pin the crypto provider so only one is linked. Cargo.toml change is a small separate commit.
- **T3** Reload certificate and key on SIGHUP without dropping connections. A bad new cert keeps the old one and logs an error.
- **T4** MCP allowlists: when TLS is on, `allowed_origins` default to `https://…`. `allowed_hosts` is unchanged.
- **T5** Startup warning when `bind` is non-loopback and TLS is off. Do not refuse.
- **T6** Tests: generate a self-signed cert in the test (dev-dependency such as `rcgen`). Cover HTTPS `/healthz`, 401 and 200 with a token over TLS, plain HTTP to the TLS port failing, SIGHUP reload serving the new cert, bad cert files failing startup, and graceful shutdown still within its bound.
- **T7** README TLS section, with a self-signed example and a note on ACME or sops certificate paths.
- **T8** Live check: `curl --cacert` against the running server, and one `/mcp` initialize over HTTPS.
## Done when
- `server.tls.enabled = true` serves HTTPS, and `cargo tree -i openssl` is still empty.
- Replacing the cert files and sending SIGHUP changes the served certificate without a restart.
- Clippy, fmt and tests are clean, and the T8 live check passes.
## Notes
- **TLS is optional (user requirement, 2026-10-16).** `server.tls.enabled` defaults to `false`. With it off the server behaves exactly as before: plain HTTP, no cert or key files read, MCP origins stay `http://`. It is configurable by file, env (`SKYG_SERVER__TLS__*`) and shown by `--print-config`. Existing tests pass unchanged with the default config.
- DEC-12 holds the rationale. The NixOS module (WS-H) passes certificate paths through systemd `LoadCredential` so the service user never needs read access to the original key file.
## Log
- 2026-10-16: Card created from design discussion.
- 2026-10-16: Implemented on `ws-i/tls`, T1-T8 done. **TLS is optional, default off** (user requirement): with `server.tls.enabled = false` the plain path is unchanged, the cert/key paths are ignored and never read, no rustls code runs, no provider is built, MCP origins stay `http://`; env (`SKYG_SERVER__TLS__*`) and `--print-config` cover the block. Design: `src/tls.rs` uses `tokio-rustls` + `hyper-util` (auto builder, HTTP/1.1 ALPN) with an accept loop and a `ReloadableCert` (`ResolvesServerCert` behind an RwLock) so SIGHUP swaps the cert for new handshakes without touching live connections; a bad pair logs an error and keeps the old one. The `ring` provider is passed explicitly to `ServerConfig::builder_with_provider`; no process-wide default is installed (aws-lc-rs was already in the tree via reqwest/OTLP; this adds no new provider or openssl). Cargo.toml: rustls, tokio-rustls, hyper-util (separate commit), rcgen dev-dependency. Shutdown: accept stops, each connection gets `graceful_shutdown`, bounded by the existing 10s drain (test: SIGTERM with an idle keep-alive and an open SSE stream over TLS exits 0). T5 warning is logged only when TLS is off and bind is non-loopback. Tests: 3 new unit tests (config validation, env, origins) and 8 in `tests/tls.rs`. Live check (debug binary, openssl-generated certs, curl): healthz, 401/200 REST, MCP initialize over HTTPS, plain HTTP rejected, SIGHUP changed the served fingerprint, a junk cert + SIGHUP kept the rotated one, SIGTERM exit 0.
- 2026-10-16: Review fixes. TLS listener bounded by a semaphore (`tls::Limits::max_conns`, default 1024; at the cap accepting pauses) and a 30s hyper `header_read_timeout` (covers post-handshake silence and idle keep-alive), accept-error warnings rate-limited to one per 10s, HTTP/1.1 only. Note: the auto builder's `serve_connection_with_upgrades` sniffs the protocol before hyper starts, which defeats the header timeout, so the TLS path uses `serve_connection` (no upgrades; none are used). A SIGHUP handler is now installed at the top of `serve` on both paths, so SIGHUP never kills the process; with TLS off it only logs. Cert and key are read separately on reload: rotation tooling must send SIGHUP after BOTH files are in place (a mismatch keeps the old pair; fix the files and send HUP again). Intermediates must be in the `cert_file` chain. Pre-existing and out of scope: the plain-HTTP path has no connection cap or header timeout either (axum::serve defaults), the same exhaustion exposure it had before WS-I. New tests: silent-connection header timeout, connection cap, SIGHUP with TLS off, key/cert mismatch on SIGHUP.