Files
2026-10-21 00:12:33 -04:00

114 lines
24 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# TODO — skyg-linux-tracker
The single place for work tracking. **Tick boxes here only.** Design lives in [`ARCHITECTURE.md`](ARCHITECTURE.md). Archived workstream design notes and logs are in [`docs/workstreams/`](docs/workstreams/). Plans for the next big thing go in [`plan/`](plan/README.md).
_Last updated: 2026-10-21_
## Status
The build is functionally complete and merged to `main`. REST, MCP, OTel, TLS and the NixOS module are implemented. Verification #1–#4 and #6 passed live on minipc1 (2026-10-19). The WS-H nix gate passed on 2026-10-19. Phase 5 (consolidator, optional peers) and the generated OpenAPI doc are merged, and the binary serves it as a Redoc page at `/docs` (DEC-16); the Phase 5 record is archived in [`docs/workstreams/P5-consolidator/`](docs/workstreams/P5-consolidator/README.md). Everything open is in the Backlog.
| Milestone | Gate | Status |
| --------------- | ------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| M0 Skeleton | `slinuxm serve --print-config` works. Clippy and tests are clean. No openssl. | Done |
| M1 Sources live | All four Phase 1 examples run against a real host | Done (systemd, journal, docker verified live) |
| M2 Surfaces | Verification #3, #4 and #5 pass | Partial: #3 and #4 pass live (minipc1, 2026-10-19, all 8 MCP tools); #5 OTel deferred to I6-5 (collector on minipc1 is down) |
| M3 Release | Verification #1–#6 pass. Unit file and README shipped. | Partial: #1–#4 and #6 pass, unit installed on minipc1; needs #5 (I6-5) |
| M4 NixOS | `nix flake check` passes including the VM test. `:1984` serves and a token is minted with the system-installed CLI. | ✅ 2026-10-19: `nix build` and `nix flake check` (incl. VM test, plain + TLS) pass; see [WS-H log](docs/workstreams/WS-H-nixos.md#log) |
| M5 TLS | HTTPS serves REST and `/mcp`, SIGHUP reloads certs, no openssl. | Done |
| M6 Consolidator | Phase 5 exit criteria; peer-less instance unchanged (8 MCP tools); `nix flake check` incl. the peers VM test | Met, except two live checks still open: **P5-1** minipc1 cross-check needs a user-minted token, **P5-2** OTel peer-gauge check via `devenv up` not yet recorded (both in the Backlog) |
## In progress / Review
| Item | Owner | State |
| ---- | ----- | ----------------- |
| — | | Nothing in flight |
- [x] **WS-H Gate:** `nix build` and `nix flake check` pass. `ldd` shows no openssl. A fresh NixOS host serves `:1984` and a token can be minted from a login shell.
- [x] **V-1** Build-stamped version `YYYY.MM.DD-<branch>-<sha>` in `--version`, `/healthz`, `/api/v1/ping`, MCP server info and OTel `service.version` (DEC-17). `dev-test lint`, `cargo test`, `nix build` and both VM tests (`vm`, `vm peers`) pass.
- [x] **DOC-3** User docs restructured into a guide: slim `README.md`, plus `docs/how-it-works.md`, `docs/install.md` (Nix and NixOS first; the systemd unit is an example for manual installs), `docs/configuration.md`, `docs/operations.md` and `docs/development.md`. The example unit gains `ExecReload` (SIGHUP), matching the module. markdownlint and link check pass.
## Backlog
Parked, not scheduled. Each has a reason. Nothing here blocks the current work. Tick here when picked up and finished.
| ID | Item | Reason parked |
| --------- | ----------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| **C7** | _(optional)_ Smoke test against the Podman socket (WS-C) | Nice-to-have; no Podman host in use |
| **I6-5** | Verification #5: OTel smoke test end to end (P3) | Waiting on the minipc1 collector fix (**X-1**); `slinuxm` itself exports fine |
| **B-1** | Live check of docker log-follow SSE (`/docker/containers/{name}/logs?follow=1`) | Not exercised in the live runs so far |
| **B-2** | Live check of REST, bearer auth and TLS against minipc1 | The MCP-tool run did not cover them |
| **X-1** | _(external)_ Investigate failed `opentelemetry-collector.service` and `var-lib-loki.mount` on minipc1 | Outside this repo; blocks I6-5 |
| **F-1** | Cap concurrent follow streams | Security review 2026-10-16; not scheduled |
| **F-2** | Bound line and message length | Security review 2026-10-16; not scheduled |
| **F-3** | Unit hardening extras | Security review 2026-10-16; needs a clean-host test first |
| **P5-1** | minipc1 cross-check of the consolidator against a real peer | Needs a read-only token minted by the user on minipc1; agents never mint tokens on real hosts |
| **P5-2** | Record the OTel peer-gauge check (`devenv up`, Verification step 5 with peers) | Not yet run and recorded; no automated test asserts the exported peer metrics end to end |
| **D8** | Fleet aggregation (merged views across peers) | Out of scope for v1 (DEC-14: no aggregation, no chaining); needs a design for partial failure and ordering |
| **P5-3** | Scoped tokens (per-peer or read-class scopes) | Any valid inbound token may read peer data (DEC-14); scopes need a token schema change and a policy decision |
| **P5-4** | Peer health history | State is in memory only by design; history needs storage or an external TSDB, and the OTel gauges already cover it |
| **P5-5** | Runtime peer management (add/remove without editing config) | Peers are declarative so no write path or DB table exists (DEC-14); would need a write path against the read-only rule |
| **DOC-1** | Show the MCP tools (`x-mcp-tools`) in the Redoc page | Redoc renders REST operations only; needs the tool list generated into a tag description (or a section) in `openapi.rs`, which changes the spec |
| **DOC-2** | Config switch to turn the `/docs` viewer off | Static, public and data-free (DEC-16), so not needed yet; adds a config key, README and `config.example.toml` entries |
| **V-2** | Real branch name in Nix builds (stamp says `nix`) | A flake does not expose its branch; needs CI to pass it to `callPackage` or a release-tag convention |
- [ ] **C7** _(nice-to-have)_ Smoke test against the Podman socket
- [ ] **I6-5** Verification #5 OTel smoke test against a working collector (see the [verification checklist](ARCHITECTURE.md#verification))
- [ ] **B-1** Docker log-follow SSE live check
- [ ] **B-2** REST, auth and TLS live check against minipc1
- [ ] **X-1** _(external)_ Fix or investigate `opentelemetry-collector.service` / `var-lib-loki.mount` on minipc1
- [ ] **P5-1** minipc1 cross-check: run a consolidator against minipc1 with a read-only token the user mints there (`list_peers`, `get_peer`, one pass-through)
- [ ] **P5-2** OTel peer gauges: run `devenv up`, enable `[otel]` with peers, confirm `peer.up`, `peer.api.auth_ok`, `peer.check.duration_seconds`, `peer.last_change_timestamp_seconds` reach the collector, and record the result
- [ ] **D8** Fleet aggregation across peers
- [ ] **P5-3** Scoped tokens
- [ ] **P5-4** Peer health history
- [ ] **P5-5** Runtime peer management
- [ ] **DOC-1** MCP tools in the Redoc page
- [ ] **DOC-2** Config switch for the `/docs` viewer
- [ ] **V-2** Real branch name in Nix builds instead of the `nix` placeholder
- [ ] **F-1** Cap concurrent follow streams. Each journal follow, `tail_journal` call or docker follow pins a `spawn_blocking` thread for up to `limits.max_follow_secs`. Add a global semaphore shared by REST and MCP (suggested 16) returning 429/503 when full. A client that stops reading may hold the thread past the deadline (untested); add a write timeout or a hard `tokio::time::timeout` around the producer.
- [ ] **F-2** Bound line and message length. Docker `LineSplitter` and journal `MESSAGE` are buffered without limit before the 256 KiB result cap applies. Truncate per line (suggested 64 KiB, with a visible marker) and cap total bytes collected.
- [ ] **F-3** Unit hardening: `ProtectKernelLogs`, `ProtectHostname`, `RemoveIPC`, `LimitNOFILE`, `TasksMax`, `MemoryMax`, narrower `RestrictAddressFamilies`, `PrivateUsers` (check journal group access). Test under the hardened unit on a clean host before enabling. Document how the service user gets read access to the TLS key.
## Completed workstreams
Design notes, owned files and logs for each are archived under [`docs/workstreams/`](docs/workstreams/). Task IDs below are stable references.
| ID | What | Key outcome | Tasks | Completed |
| ---------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | ------------------------------ |
| [P0](docs/workstreams/P0-foundation.md) | Foundation: devenv, Cargo, config, DTOs, source traits + fakes, errors, `AppState`, stubs | `serve --print-config` works; build, clippy, fmt, test clean; `cargo tree -i openssl` empty | P0-1..P0-11, Gate | merged to `main` |
| [WS-A](docs/workstreams/WS-A-systemd.md) | systemd source (zbus) | Units, timers (via `*.timer` + `Timer` props), read-only `GetUnit` status | A1..A7 | merged to `main` |
| [WS-B](docs/workstreams/WS-B-journal.md) | journal source (sd-journal) | Snapshot + follow in `spawn_blocking`; follow thread exits on receiver drop | B1..B6 | merged to `main` |
| [WS-C](docs/workstreams/WS-C-docker.md) | docker source (bollard) | Lazy socket, bounded concurrent inspect, log streams; missing socket maps to `Unavailable` | C1..C6 (C7 in Backlog) | merged to `main` |
| [WS-D](docs/workstreams/WS-D-auth.md) | auth + SQLite token DB | sha256 tokens, `slinuxm token create`, `list`, `revoke`, bearer middleware, 401 + `WWW-Authenticate` | D1..D7 | merged to `main` |
| [WS-E](docs/workstreams/WS-E-rest.md) | REST API | All `/api/v1` routes, SSE follow, `/healthz`, 503 on docker-only failures | E1..E9 | merged to `main` |
| [WS-F](docs/workstreams/WS-F-mcp.md) | MCP server | 8 tools, 2 prompts, 2 resources, instructions, Origin/Host validation | F1..F9 | merged to `main` |
| [WS-G](docs/workstreams/WS-G-otel.md) | OTel export | Traces, observable-gauge metrics, journal/docker log fan-out with reconcile, flush on shutdown, `dev/otelcol.yaml` | G1..G7 | merged to `main` |
| [P3](docs/workstreams/P3-integration.md) | Integration and hardening | One axum server, negative tests, hardened unit, README; I6 verified on minipc1 (#1–#4, #6) | I1..I6 (I6-5 in Backlog) | 2026-10-19 |
| [WS-I](docs/workstreams/WS-I-tls.md) | Native TLS (rustls) | `server.tls.*`, SIGHUP reload, HTTPS tests, live `curl --cacert` check | T1..T8 | merged to `main` |
| [WS-H](docs/workstreams/WS-H-nixos.md) | NixOS package, module, VM test | H1..H8 merged (PR #1, 34da657); gate passed 2026-10-19 | H1..H8 | merged; gate passed 2026-10-19 |
| [P5](docs/workstreams/P5-consolidator/README.md) | Consolidator: optional peers (WS-T, A..F) | Declarative peers with TCP and API health checks, `/api/v1/peers` status and GET pass-through, `peer` argument and `list_peers`/`get_peer` (10 MCP tools with peers, 8 without), four OTel gauges, NixOS module option, SSRF guard, peers VM test; peer-less instance unchanged | T1..T7, A0..A4, B1..B5, C1..C4, D1..D3, E1..E2, F1..F4 (P5-1, P5-2 in Backlog) | 2026-10-20 |
| [WS-G (OpenAPI)](docs/workstreams/WS-G-openapi.md) | Generated OpenAPI 3.1 doc (stand-alone, after P5) | `utoipa` paths plus schemars bridge, `x-mcp-tools`, `GET /api/v1/openapi.json`, `slinuxm openapi [--peers]`, drift test, `docs/openapi.json` | G1..G4 | 2026-10-20 |
| [WS-H (final fixes)](docs/workstreams/WS-H-final-fixes.md) | Phase 5 final-review fixes | One token loader, one set of peer helpers, IPv4-mapped deny check, IPv6 `deny_cidrs`, client span kind, flaky test fixes, docs corrections | review items | 2026-10-20 |
### Done (archive)
All ticked. One line per workstream; the full task text is on each card.
- **P0** — devenv scripts and libsystemd (P0-1); `Cargo.toml` with `slinuxm` bin, rustls-only, LTO+strip (P0-2); API spike (P0-3); figment config with `otel.headers`, `otel.metrics.*`, `limits.*` and validation (P0-4); `config.example.toml` (P0-5); DTOs with serde/schemars (P0-6); source traits + fakes (P0-7); `AppError` mapping (P0-8); `AppState` (P0-9); stubs and `main.rs` dispatch (P0-10); config tests (P0-11); gate.
- **WS-A** — bus connection (A1); `list_units` (A2); `list_timers` (A3); `unit_status` via `GetUnit` (A4); name validation, `NoSuchUnit` to NotFound (A5); `examples/systemd_dump.rs` (A6); mapping tests (A7).
- **WS-B** — `spawn_blocking` access (B1); `query` (B2); `follow` (B3); lossy UTF-8 mapping (B4); `examples/journal_tail.rs` (B5); follow-exit test (B6).
- **WS-C** — lazy bollard client (C1); `ps` (C2); `container_status` (C3); `logs` stream (C4); `tail` cap (C5); `examples/docker_ps.rs` (C6).
- **WS-D** — `db.rs` open-or-create, WAL (D1); tokens migration (D2); token CRUD (D3); CLI (D4); bearer middleware (D5); protected-router layer, `/healthz` public (D6); tests (D7).
- **WS-E** — router with `{name}` syntax (E1); systemd routes (E2); journal snapshot (E3); journal SSE (E4); docker routes (E5); `/healthz` (E6); JSON errors (E7); handler tests (E8); docker 503 (E9).
- **WS-F** — `TrackerService`, 8 tools (F1); `tail_journal` cap (F2); `get_container_logs` cap (F3); `get_info` instructions (F4); tool descriptions (F5); prompts (F6); resources (F7); `mcp::service` with Origin validation (F8); tests (F9).
- **WS-G** — `telemetry::init` (G1); tracing subscriber (G2); snapshot + observable gauges (G3); metric set and allowlists (G4); log export with reconcile (G5); shutdown flush (G6); `dev/otelcol.yaml` (G7).
- **P3** — serve wiring and shutdown (I1); negative tests (I2); systemd unit, verified with `systemd-analyze security --offline` (1.5 OK) and installed as a system unit on minipc1, 2026-10-19 (I3); README (I4); release build, `ldd` clean (I5); manual Verification #1–#4 and #6 on minipc1, 2026-10-19; #5 deferred as I6-5 (I6).
- **WS-I** — `server.tls.*` config (T1); rustls with pinned provider (T2); SIGHUP reload (T3); `https://` MCP origins (T4); non-loopback-without-TLS warning (T5); self-signed cert tests (T6); README TLS section (T7); live check (T8).
- **WS-H** — `nix/package.nix` (H1); `flake.nix` (H2); module options (H3); service definition (H4); system-wide CLI with root guard (H5); firewall options and eval assertion (H6); README NixOS section (H7); VM test (H8). Gate passed 2026-10-19 (build, flake check, VM test).
- **P5** — runner `dev/dev-test.nu` and peers VM test (WS-T); config, validation, secrets loader, `peer` CLI (WS-A); registry, health loops, status model, SIGHUP reload, transport with SSRF guard (WS-B); status routes, pass-through, `x-slinuxm-peer`, error mapping (WS-C); `peer` argument, `list_peers`/`get_peer`, instructions (WS-D); gauges, client spans and propagation (WS-E); NixOS module, promoted VM test, `tests/peers.rs`, README and `docs/consolidator.md` (WS-F).
- **WS-G (OpenAPI)** — REST spec (G1); MCP `x-mcp-tools` (G2); route and `slinuxm openapi` (G3); drift guard and `docs/openapi.json` (G4).
- **WS-H (final fixes)** — see the [record](docs/workstreams/WS-H-final-fixes.md).
Early decisions (DEC-1..DEC-8) were accepted before work started; all decisions are recorded in [`ARCHITECTURE.md`](ARCHITECTURE.md#decisions).