doc updates

This commit is contained in:
ari committed 2026-10-19 14:14:29 -04:00
1 parent 7e1bd4a9a4
commit a511ff532e
26 files changed
+573 -770

No files matched your search

+71 -120
View File
@@ -1,135 +1,86 @@
# TODO — skyg-linux-tracker
Master checklist. **Tick boxes here only.** Card status lives on [`plan/BOARD.md`](plan/BOARD.md). Scope, owned files and design notes for each workstream live in [`plan/workstreams/`](plan/workstreams/).
The single place for work tracking. **Tick boxes here only.** Design lives in [`ARCHITECTURE.md`](ARCHITECTURE.md). Archived workstream design notes and logs are in [`docs/workstreams/`](docs/workstreams/). Plans for the next big thing go in [`plan/`](plan/README.md).
Design: [`plan.md`](plan.md) · Review fixes: [`plan/review.md`](plan/review.md) · Open questions: [`plan/decisions.md`](plan/decisions.md)
_Last updated: 2026-10-19_
## Before starting
## Status
- [x] Resolve **DEC-1**: binary and CLI are `slinuxm`
- [x] Accept or override the proposed decisions DEC-2 to DEC-7 (all accepted)
The build is functionally complete and merged to `main`. REST, MCP, OTel, TLS and the NixOS module are implemented. Verification #1–#4 and #6 passed live on minipc1 (2026-10-19). The only item in flight is the WS-H nix gate. Everything else open is in the Backlog.
## Phase 0 — Foundation · [card](plan/workstreams/P0-foundation.md)
| Milestone | Gate | Status |
|---|---|---|
| M0 Skeleton | `slinuxm serve --print-config` works. Clippy and tests are clean. No openssl. | Done |
| M1 Sources live | All four Phase 1 examples run against a real host | Done (systemd, journal, docker verified live) |
| M2 Surfaces | Verification #3, #4 and #5 pass | Partial: #3 and #4 pass live (minipc1, 2026-10-19, all 8 MCP tools); #5 OTel deferred to I6-5 (collector on minipc1 is down) |
| M3 Release | Verification #1–#6 pass. Unit file and README shipped. | Partial: #1–#4 and #6 pass, unit installed on minipc1; needs #5 (I6-5) |
| M4 NixOS | `nix flake check` passes including the VM test. `:1984` serves and a token is minted with the system-installed CLI. | **Pending the WS-H gate** (result to be recorded here and in [WS-H log](docs/workstreams/WS-H-nixos.md#log)) |
| M5 TLS | HTTPS serves REST and `/mcp`, SIGHUP reloads certs, no openssl. | Done |
- [x] **P0-1** devenv: `pkg-config`, `systemd` (libsystemd.pc), `build`/`run`/`token` scripts. No openssl. `pkg-config --libs libsystemd` works in the shell.
- [x] **P0-2** `Cargo.toml` with `[[bin]] name = "slinuxm"` and every dependency pinned for all phases. rustls-only features. Release profile set to lto + strip.
- [x] **P0-3** Time-boxed API spike to confirm rmcp, zbus_systemd, systemd-journal and bollard APIs. Findings go in the card. Update DEC-8.
- [x] **P0-4** `config.rs`: figment layering (defaults < TOML < `SKYG_*` env with `__` nesting < CLI). Add `otel.headers`, `otel.metrics.*` allowlists and `limits.*`. Validate the auth-disabled + non-loopback combination.
- [x] **P0-5** `config.example.toml` covering every key, with comments
- [x] **P0-6** Shared DTOs in `src/sources/mod.rs` with serde + schemars derives and RFC3339 timestamps. Service-only fields are `Option`.
- [x] **P0-7** Source traits plus in-memory fakes (`src/sources/fake.rs`)
- [x] **P0-8** `error.rs`: `AppError` mapped to HTTP status + JSON body and to MCP `ErrorData`
- [x] **P0-9** `src/state.rs`: `AppState` (sources, optional DB pool, config, `CancellationToken`)
- [x] **P0-10** Stub every module with its final signatures. `main.rs` dispatches `serve` / `token` / `--print-config` to them.
- [x] **P0-11** Unit tests for config precedence and validation
- [x] **Gate:** `slinuxm serve --print-config` prints effective config. Build, clippy, fmt and test are clean. `cargo tree -i openssl` is empty.
## In progress / Review
## Phase 1 — Sources + auth (parallel)
| Item | Owner | State |
|---|---|---|
| [WS-H NixOS](docs/workstreams/WS-H-nixos.md) gate | Ari | PR #1 merged to `main` (34da657). Outstanding: `nix build`, `nix flake check` incl. VM test. Gate result: _pending_ |
### WS-A systemd · [card](plan/workstreams/WS-A-systemd.md)
- [x] **A1** Shared system-bus `zbus::Connection` + `ManagerProxy`
- [x] **A2** `list_units(state)` mapped to `UnitSummary`
- [x] **A3** `list_timers()` via `*.timer` units + `TimerProxy` props (no `ListTimers` method exists)
- [x] **A4** `unit_status(name)`: `GetUnit` only (read-only; `LoadUnit` is never called, an unloaded unit is NotFound). `Unit` props always, `Service` props for `.service` only.
- [x] **A5** Unit-name validation. `NoSuchUnit` maps to `NotFound`.
- [x] **A6** `examples/systemd_dump.rs` prints JSON from the live bus
- [x] **A7** Unit tests for timestamp conversion and property mapping
- [ ] **WS-H Gate:** `nix build` and `nix flake check` pass. `ldd` shows no openssl. A fresh NixOS host serves `:1984` and a token can be minted from a login shell.
### WS-B journal · [card](plan/workstreams/WS-B-journal.md)
- [x] **B1** All sd-journal access inside `spawn_blocking`. Optional namespace, depending on the spike.
- [x] **B2** `query(filter)`: unit, priority ≤ N, `lines` capped by `limits.max_lines`, returned in chronological order
- [x] **B3** `follow(filter)` returns an mpsc receiver. `wait()` with a timeout. Exits on receiver drop or shutdown.
- [x] **B4** Entry mapping with lossy UTF-8 for binary `MESSAGE`
- [x] **B5** `examples/journal_tail.rs`
- [x] **B6** Test: the follow thread exits after its receiver is dropped
## Backlog
Parked, not scheduled. Each has a reason. Nothing here blocks the current work. Tick here when picked up and finished.
| ID | Item | Reason parked |
|---|---|---|
| **C7** | *(optional)* Smoke test against the Podman socket (WS-C) | Nice-to-have; no Podman host in use |
| **I6-5** | Verification #5: OTel smoke test end to end (P3) | Waiting on the minipc1 collector fix (**X-1**); `slinuxm` itself exports fine |
| **B-1** | Live check of docker log-follow SSE (`/docker/containers/{name}/logs?follow=1`) | Not exercised in the live runs so far |
| **B-2** | Live check of REST, bearer auth and TLS against minipc1 | The MCP-tool run did not cover them |
| **X-1** | *(external)* Investigate failed `opentelemetry-collector.service` and `var-lib-loki.mount` on minipc1 | Outside this repo; blocks I6-5 |
| **F-1** | Cap concurrent follow streams | Security review 2026-10-16; not scheduled |
| **F-2** | Bound line and message length | Security review 2026-10-16; not scheduled |
| **F-3** | Unit hardening extras | Security review 2026-10-16; needs a clean-host test first |
### WS-C docker · [card](plan/workstreams/WS-C-docker.md)
- [x] **C1** Lazy `bollard::Docker` on the configured socket. Missing or denied socket maps to `Unavailable`.
- [x] **C2** `ps()`: list + concurrent bounded inspect (health, uptime, restarts, image)
- [x] **C3** `container_status(name)`
- [x] **C4** `logs(name, follow, tail, since)` stream that stops on drop or shutdown
- [x] **C5** `tail` capped by config
- [x] **C6** `examples/docker_ps.rs` + log follow
- [ ] **C7** *(nice-to-have)* Smoke test against the Podman socket
### WS-D auth + DB · [card](plan/workstreams/WS-D-auth.md)
- [x] **D1** `db.rs`: open-or-create, dir `0700`, WAL, `busy_timeout`, embedded migrations
- [x] **D2** Migration for `tokens(id, name UNIQUE, token_hash UNIQUE, created_at, revoked_at, last_used_at)`
- [x] **D3** `tokens.rs`: create (32 random bytes, `slinuxm_` prefix, printed once), list, revoke. Uses runtime queries.
- [x] **D4** `slinuxm token create|list|revoke` CLI wired via the Phase 0 stub
- [x] **D5** Bearer middleware: sha256 + indexed lookup of non-revoked tokens. 401 + `WWW-Authenticate` JSON on failure.
- [x] **D6** Middleware exposed as a layer for the protected router only. `/healthz` stays public.
- [x] **D7** Tests: create, accept, revoke, reject. Unknown token is rejected. Startup refuses auth-off + non-loopback.
## Phase 2 — Surfaces (parallel)
### WS-E REST API · [card](plan/workstreams/WS-E-rest.md)
- [x] **E1** `api::router(state)` using axum 0.8 `{name}` path syntax
- [x] **E2** `GET /systemd/units?state=`, `/systemd/units/{name}`, `/systemd/timers`
- [x] **E3** `GET /journal?unit=&priority=&lines=` with validation
- [x] **E4** `GET /journal/follow` SSE with keep-alive. Ends on disconnect or shutdown.
- [x] **E5** `GET /docker/ps`, `/docker/containers/{name}`, `/docker/containers/{name}/logs` (JSON, or SSE with `follow=1`)
- [x] **E6** `GET /healthz` on the public router
- [x] **E7** Consistent JSON error body from `AppError`
- [x] **E8** Handler tests against fake sources (`tower::ServiceExt::oneshot`). Name/state validation is tested via a validating stub, not the fakes.
- [x] **E9** Docker unavailable returns 503 on docker routes only
### WS-F MCP · [card](plan/workstreams/WS-F-mcp.md)
- [x] **F1** `TrackerService` with `#[tool_router]` and 8 tools
- [x] **F2** `tail_journal` capped by `limits.max_follow_secs` and an entry cap
- [x] **F3** `get_container_logs` tail cap
- [x] **F4** `ServerHandler::get_info`: instructions + capabilities (tools, prompts, resources)
- [x] **F5** Tool descriptions that explain systemd states, health statuses and priorities
- [x] **F6** Prompts `diagnose_unit { name }` and `container_health_report {}`
- [x] **F7** Resources `systemd://units/failed` and `docker://containers`, read live, no subscriptions
- [x] **F8** `mcp::service(state)` returns `StreamableHttpService` with Origin validation (DEC-8)
- [x] **F9** Tests: tool list snapshot + each tool against fakes
### WS-G OTel · [card](plan/workstreams/WS-G-otel.md)
- [x] **G1** `telemetry::init(cfg)`: OTLP grpc/http with headers. Resource has `service.name`, `service.version`, `host.name`. Returns a guard.
- [x] **G2** `tracing` subscriber: fmt to stderr + `tracing-opentelemetry` layer when enabled
- [x] **G3** Metrics poll loop feeds a snapshot. Observable gauges read it, so vanished series drop out.
- [x] **G4** Metric set from `plan.md` with semconv attribute names, allowlists and the DEC-7 default scope
- [x] **G5** Log export: journal follow + per-container docker follow, reconciled via the Docker events stream, with backoff
- [x] **G6** Shutdown cancels tasks and flushes all providers
- [x] **G7** `dev/otelcol.yaml` with a debug exporter, plus a devenv process to run it
## Phase 3 — Integration & hardening · [card](plan/workstreams/P3-integration.md)
- [x] **I1** `main.rs` wiring: protected router (`/api/v1` + `/mcp` + auth) merged with the public `/healthz`, `TraceLayer`, SIGTERM/SIGINT leading to cancel, drain and OTel flush
- [x] **I2** Automated negative tests (Verification #6)
- [x] **I3** `systemd/skyg-linux-tracker.service` running `slinuxm serve`: `StateDirectory=`, `ProtectSystem=strict`, `NoNewPrivileges`, `SupplementaryGroups=systemd-journal`, docker group commented (DEC-6) (verified only by systemd-analyze security --offline = 1.5 OK and a `systemd-run --user` run with the same sandbox properties; not installed as a system unit on a clean host)
- [x] **I4** README: install, config reference, token workflow, MCP client setup, hardened docker option
- [x] **I5** Release build: `ldd` shows only libc + libsystemd and its deps. Record binary size.
- [ ] **I6** Full manual Verification #1–#6 from `plan.md` passes on a real host
## Phase 4 — TLS and NixOS packaging
### WS-I TLS · [card](plan/workstreams/WS-I-tls.md)
- [x] **T1** Config `server.tls.{enabled,cert_file,key_file}` with validation and documented example keys
- [x] **T2** Serve over rustls with a single pinned crypto provider. No openssl.
- [x] **T3** Reload cert and key on SIGHUP without dropping connections. A bad cert keeps the old one.
- [x] **T4** MCP allowed origins use `https://` when TLS is on
- [x] **T5** Startup warning for non-loopback bind without TLS
- [x] **T6** Tests with a generated self-signed cert: HTTPS, 401/200 over TLS, plain HTTP rejected, SIGHUP reload, bad files, bounded shutdown
- [x] **T7** README TLS section
- [x] **T8** Live check with `curl --cacert` and an `/mcp` initialize over HTTPS
### WS-H NixOS · [card](plan/workstreams/WS-H-nixos.md)
- [x] **H1** `nix/package.nix`: `buildRustPackage` from `Cargo.lock` with pkg-config, cmake, systemd. Binary `slinuxm`.
- [x] **H2** `flake.nix` with `packages.default`, `nixosModules.default` and `checks`
- [x] **H3** `nix/module.nix` options: `enable`, `package`, `port` (default 1984), `bind` (default loopback), `tls.*`, `openFirewall`, `allowedHosts`, `docker.enable`, `settings`, `environmentFile`
- [x] **H4** Service: dedicated user, `StateDirectory` 0700, `systemd-journal` group, P3 hardening set, config at `/etc/skyg-linux-tracker.toml`
- [x] **H5** **CLI installed system-wide**, plain binary (DEC-10). Token commands refuse to run as root against another user's DB and print the `sudo -u skyg-linux-tracker …` command.
- [x] **H6** Firewall: global `openFirewall` plus interface-scoped option. Eval assertion: non-loopback bind requires auth.
- [x] **H7** README NixOS section: flake input, token workflow, exposure model, `allowedHosts`
- [x] **H8** NixOS VM test: healthz, 401, mint token via CLI (`sudo -u`), 200, `/mcp`, revoke, 401, bad Origin 403, plus a TLS case
- [ ] **Gate:** `nix build` and `nix flake check` pass. `ldd` shows no openssl. A fresh NixOS host serves `:1984` and a token can be minted from a login shell.
## Future
Logged from the final security review (2026-10-16). Not scheduled.
- [ ] **I6-5** Verification #5 OTel smoke test against a working collector (see the [verification checklist](ARCHITECTURE.md#verification))
- [ ] **B-1** Docker log-follow SSE live check
- [ ] **B-2** REST, auth and TLS live check against minipc1
- [ ] **X-1** *(external)* Fix or investigate `opentelemetry-collector.service` / `var-lib-loki.mount` on minipc1
- [ ] **F-1** Cap concurrent follow streams. Each journal follow, `tail_journal` call or docker follow pins a `spawn_blocking` thread for up to `limits.max_follow_secs`. Add a global semaphore shared by REST and MCP (suggested 16) returning 429/503 when full. A client that stops reading may hold the thread past the deadline (untested); add a write timeout or a hard `tokio::time::timeout` around the producer.
- [ ] **F-2** Bound line and message length. Docker `LineSplitter` and journal `MESSAGE` are buffered without limit before the 256 KiB result cap applies. Truncate per line (suggested 64 KiB, with a visible marker) and cap total bytes collected.
- [ ] **F-3** Unit hardening: `ProtectKernelLogs`, `ProtectHostname`, `RemoveIPC`, `LimitNOFILE`, `TasksMax`, `MemoryMax`, narrower `RestrictAddressFamilies`, `PrivateUsers` (check journal group access). Test under the hardened unit on a clean host before enabling. Document how the service user gets read access to the TLS key.
## Completed workstreams
Design notes, owned files and logs for each are archived under [`docs/workstreams/`](docs/workstreams/). Task IDs below are stable references.
| ID | What | Key outcome | Tasks | Completed |
|---|---|---|---|---|
| [P0](docs/workstreams/P0-foundation.md) | Foundation: devenv, Cargo, config, DTOs, source traits + fakes, errors, `AppState`, stubs | `serve --print-config` works; build, clippy, fmt, test clean; `cargo tree -i openssl` empty | P0-1..P0-11, Gate | merged to `main` |
| [WS-A](docs/workstreams/WS-A-systemd.md) | systemd source (zbus) | Units, timers (via `*.timer` + `Timer` props), read-only `GetUnit` status | A1..A7 | merged to `main` |
| [WS-B](docs/workstreams/WS-B-journal.md) | journal source (sd-journal) | Snapshot + follow in `spawn_blocking`; follow thread exits on receiver drop | B1..B6 | merged to `main` |
| [WS-C](docs/workstreams/WS-C-docker.md) | docker source (bollard) | Lazy socket, bounded concurrent inspect, log streams; missing socket maps to `Unavailable` | C1..C6 (C7 in Backlog) | merged to `main` |
| [WS-D](docs/workstreams/WS-D-auth.md) | auth + SQLite token DB | sha256 tokens, `slinuxm token create`, `list`, `revoke`, bearer middleware, 401 + `WWW-Authenticate` | D1..D7 | merged to `main` |
| [WS-E](docs/workstreams/WS-E-rest.md) | REST API | All `/api/v1` routes, SSE follow, `/healthz`, 503 on docker-only failures | E1..E9 | merged to `main` |
| [WS-F](docs/workstreams/WS-F-mcp.md) | MCP server | 8 tools, 2 prompts, 2 resources, instructions, Origin/Host validation | F1..F9 | merged to `main` |
| [WS-G](docs/workstreams/WS-G-otel.md) | OTel export | Traces, observable-gauge metrics, journal/docker log fan-out with reconcile, flush on shutdown, `dev/otelcol.yaml` | G1..G7 | merged to `main` |
| [P3](docs/workstreams/P3-integration.md) | Integration and hardening | One axum server, negative tests, hardened unit, README; I6 verified on minipc1 (#1–#4, #6) | I1..I6 (I6-5 in Backlog) | 2026-10-19 |
| [WS-I](docs/workstreams/WS-I-tls.md) | Native TLS (rustls) | `server.tls.*`, SIGHUP reload, HTTPS tests, live `curl --cacert` check | T1..T8 | merged to `main` |
| [WS-H](docs/workstreams/WS-H-nixos.md) | NixOS package, module, VM test | H1..H8 merged (PR #1, 34da657); **gate pending, see above** | H1..H8 | merged; gate open |
### Done (archive)
All ticked. One line per workstream; the full task text is on each card.
- **P0** — devenv scripts and libsystemd (P0-1); `Cargo.toml` with `slinuxm` bin, rustls-only, LTO+strip (P0-2); API spike (P0-3); figment config with `otel.headers`, `otel.metrics.*`, `limits.*` and validation (P0-4); `config.example.toml` (P0-5); DTOs with serde/schemars (P0-6); source traits + fakes (P0-7); `AppError` mapping (P0-8); `AppState` (P0-9); stubs and `main.rs` dispatch (P0-10); config tests (P0-11); gate.
- **WS-A** — bus connection (A1); `list_units` (A2); `list_timers` (A3); `unit_status` via `GetUnit` (A4); name validation, `NoSuchUnit` to NotFound (A5); `examples/systemd_dump.rs` (A6); mapping tests (A7).
- **WS-B** — `spawn_blocking` access (B1); `query` (B2); `follow` (B3); lossy UTF-8 mapping (B4); `examples/journal_tail.rs` (B5); follow-exit test (B6).
- **WS-C** — lazy bollard client (C1); `ps` (C2); `container_status` (C3); `logs` stream (C4); `tail` cap (C5); `examples/docker_ps.rs` (C6).
- **WS-D** — `db.rs` open-or-create, WAL (D1); tokens migration (D2); token CRUD (D3); CLI (D4); bearer middleware (D5); protected-router layer, `/healthz` public (D6); tests (D7).
- **WS-E** — router with `{name}` syntax (E1); systemd routes (E2); journal snapshot (E3); journal SSE (E4); docker routes (E5); `/healthz` (E6); JSON errors (E7); handler tests (E8); docker 503 (E9).
- **WS-F** — `TrackerService`, 8 tools (F1); `tail_journal` cap (F2); `get_container_logs` cap (F3); `get_info` instructions (F4); tool descriptions (F5); prompts (F6); resources (F7); `mcp::service` with Origin validation (F8); tests (F9).
- **WS-G** — `telemetry::init` (G1); tracing subscriber (G2); snapshot + observable gauges (G3); metric set and allowlists (G4); log export with reconcile (G5); shutdown flush (G6); `dev/otelcol.yaml` (G7).
- **P3** — serve wiring and shutdown (I1); negative tests (I2); systemd unit, verified with `systemd-analyze security --offline` (1.5 OK) and installed as a system unit on minipc1, 2026-10-19 (I3); README (I4); release build, `ldd` clean (I5); manual Verification #1–#4 and #6 on minipc1, 2026-10-19; #5 deferred as I6-5 (I6).
- **WS-I** — `server.tls.*` config (T1); rustls with pinned provider (T2); SIGHUP reload (T3); `https://` MCP origins (T4); non-loopback-without-TLS warning (T5); self-signed cert tests (T6); README TLS section (T7); live check (T8).
- **WS-H** — `nix/package.nix` (H1); `flake.nix` (H2); module options (H3); service definition (H4); system-wide CLI with root guard (H5); firewall options and eval assertion (H6); README NixOS section (H7); VM test (H8). Gate still open.
Early decisions (DEC-1..DEC-8) were accepted before work started; all decisions are recorded in [`ARCHITECTURE.md`](ARCHITECTURE.md#decisions).