added live redoc and slight refactors to tokens

This commit is contained in:
ari committed 2026-10-20 22:53:09 -04:00
1 parent 435a54a02d
commit b38c66bcc8
19 files changed
+836 -157

No files matched your search

+51 -47
View File
@@ -6,23 +6,23 @@ _Last updated: 2026-10-20_
## Status
The build is functionally complete and merged to `main`. REST, MCP, OTel, TLS and the NixOS module are implemented. Verification #1–#4 and #6 passed live on minipc1 (2026-10-19). The WS-H nix gate passed on 2026-10-19. Phase 5 (consolidator, optional peers) and the generated OpenAPI doc are merged; the Phase 5 record is archived in [`docs/workstreams/P5-consolidator/`](docs/workstreams/P5-consolidator/README.md). Everything open is in the Backlog.
The build is functionally complete and merged to `main`. REST, MCP, OTel, TLS and the NixOS module are implemented. Verification #1–#4 and #6 passed live on minipc1 (2026-10-19). The WS-H nix gate passed on 2026-10-19. Phase 5 (consolidator, optional peers) and the generated OpenAPI doc are merged, and the binary serves it as a Redoc page at `/docs` (DEC-16); the Phase 5 record is archived in [`docs/workstreams/P5-consolidator/`](docs/workstreams/P5-consolidator/README.md). Everything open is in the Backlog.
| Milestone | Gate | Status |
|---|---|---|
| M0 Skeleton | `slinuxm serve --print-config` works. Clippy and tests are clean. No openssl. | Done |
| M1 Sources live | All four Phase 1 examples run against a real host | Done (systemd, journal, docker verified live) |
| M2 Surfaces | Verification #3, #4 and #5 pass | Partial: #3 and #4 pass live (minipc1, 2026-10-19, all 8 MCP tools); #5 OTel deferred to I6-5 (collector on minipc1 is down) |
| M3 Release | Verification #1–#6 pass. Unit file and README shipped. | Partial: #1–#4 and #6 pass, unit installed on minipc1; needs #5 (I6-5) |
| M4 NixOS | `nix flake check` passes including the VM test. `:1984` serves and a token is minted with the system-installed CLI. | ✅ 2026-10-19: `nix build` and `nix flake check` (incl. VM test, plain + TLS) pass; see [WS-H log](docs/workstreams/WS-H-nixos.md#log) |
| M5 TLS | HTTPS serves REST and `/mcp`, SIGHUP reloads certs, no openssl. | Done |
| M6 Consolidator | Phase 5 exit criteria; peer-less instance unchanged (8 MCP tools); `nix flake check` incl. the peers VM test | Met, except two live checks still open: **P5-1** minipc1 cross-check needs a user-minted token, **P5-2** OTel peer-gauge check via `devenv up` not yet recorded (both in the Backlog) |
| Milestone | Gate | Status |
| --------------- | ------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| M0 Skeleton | `slinuxm serve --print-config` works. Clippy and tests are clean. No openssl. | Done |
| M1 Sources live | All four Phase 1 examples run against a real host | Done (systemd, journal, docker verified live) |
| M2 Surfaces | Verification #3, #4 and #5 pass | Partial: #3 and #4 pass live (minipc1, 2026-10-19, all 8 MCP tools); #5 OTel deferred to I6-5 (collector on minipc1 is down) |
| M3 Release | Verification #1–#6 pass. Unit file and README shipped. | Partial: #1–#4 and #6 pass, unit installed on minipc1; needs #5 (I6-5) |
| M4 NixOS | `nix flake check` passes including the VM test. `:1984` serves and a token is minted with the system-installed CLI. | ✅ 2026-10-19: `nix build` and `nix flake check` (incl. VM test, plain + TLS) pass; see [WS-H log](docs/workstreams/WS-H-nixos.md#log) |
| M5 TLS | HTTPS serves REST and `/mcp`, SIGHUP reloads certs, no openssl. | Done |
| M6 Consolidator | Phase 5 exit criteria; peer-less instance unchanged (8 MCP tools); `nix flake check` incl. the peers VM test | Met, except two live checks still open: **P5-1** minipc1 cross-check needs a user-minted token, **P5-2** OTel peer-gauge check via `devenv up` not yet recorded (both in the Backlog) |
## In progress / Review
| Item | Owner | State |
|---|---|---|
| — | | Nothing in flight |
| Item | Owner | State |
| ---- | ----- | ----------------- |
| — | | Nothing in flight |
- [x] **WS-H Gate:** `nix build` and `nix flake check` pass. `ldd` shows no openssl. A fresh NixOS host serves `:1984` and a token can be minted from a login shell.
@@ -30,34 +30,38 @@ The build is functionally complete and merged to `main`. REST, MCP, OTel, TLS an
Parked, not scheduled. Each has a reason. Nothing here blocks the current work. Tick here when picked up and finished.
| ID | Item | Reason parked |
|---|---|---|
| **C7** | *(optional)* Smoke test against the Podman socket (WS-C) | Nice-to-have; no Podman host in use |
| **I6-5** | Verification #5: OTel smoke test end to end (P3) | Waiting on the minipc1 collector fix (**X-1**); `slinuxm` itself exports fine |
| **B-1** | Live check of docker log-follow SSE (`/docker/containers/{name}/logs?follow=1`) | Not exercised in the live runs so far |
| **B-2** | Live check of REST, bearer auth and TLS against minipc1 | The MCP-tool run did not cover them |
| **X-1** | *(external)* Investigate failed `opentelemetry-collector.service` and `var-lib-loki.mount` on minipc1 | Outside this repo; blocks I6-5 |
| **F-1** | Cap concurrent follow streams | Security review 2026-10-16; not scheduled |
| **F-2** | Bound line and message length | Security review 2026-10-16; not scheduled |
| **F-3** | Unit hardening extras | Security review 2026-10-16; needs a clean-host test first |
| **P5-1** | minipc1 cross-check of the consolidator against a real peer | Needs a read-only token minted by the user on minipc1; agents never mint tokens on real hosts |
| **P5-2** | Record the OTel peer-gauge check (`devenv up`, Verification step 5 with peers) | Not yet run and recorded; no automated test asserts the exported peer metrics end to end |
| **D8** | Fleet aggregation (merged views across peers) | Out of scope for v1 (DEC-14: no aggregation, no chaining); needs a design for partial failure and ordering |
| **P5-3** | Scoped tokens (per-peer or read-class scopes) | Any valid inbound token may read peer data (DEC-14); scopes need a token schema change and a policy decision |
| **P5-4** | Peer health history | State is in memory only by design; history needs storage or an external TSDB, and the OTel gauges already cover it |
| **P5-5** | Runtime peer management (add/remove without editing config) | Peers are declarative so no write path or DB table exists (DEC-14); would need a write path against the read-only rule |
| ID | Item | Reason parked |
| --------- | ----------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| **C7** | _(optional)_ Smoke test against the Podman socket (WS-C) | Nice-to-have; no Podman host in use |
| **I6-5** | Verification #5: OTel smoke test end to end (P3) | Waiting on the minipc1 collector fix (**X-1**); `slinuxm` itself exports fine |
| **B-1** | Live check of docker log-follow SSE (`/docker/containers/{name}/logs?follow=1`) | Not exercised in the live runs so far |
| **B-2** | Live check of REST, bearer auth and TLS against minipc1 | The MCP-tool run did not cover them |
| **X-1** | _(external)_ Investigate failed `opentelemetry-collector.service` and `var-lib-loki.mount` on minipc1 | Outside this repo; blocks I6-5 |
| **F-1** | Cap concurrent follow streams | Security review 2026-10-16; not scheduled |
| **F-2** | Bound line and message length | Security review 2026-10-16; not scheduled |
| **F-3** | Unit hardening extras | Security review 2026-10-16; needs a clean-host test first |
| **P5-1** | minipc1 cross-check of the consolidator against a real peer | Needs a read-only token minted by the user on minipc1; agents never mint tokens on real hosts |
| **P5-2** | Record the OTel peer-gauge check (`devenv up`, Verification step 5 with peers) | Not yet run and recorded; no automated test asserts the exported peer metrics end to end |
| **D8** | Fleet aggregation (merged views across peers) | Out of scope for v1 (DEC-14: no aggregation, no chaining); needs a design for partial failure and ordering |
| **P5-3** | Scoped tokens (per-peer or read-class scopes) | Any valid inbound token may read peer data (DEC-14); scopes need a token schema change and a policy decision |
| **P5-4** | Peer health history | State is in memory only by design; history needs storage or an external TSDB, and the OTel gauges already cover it |
| **P5-5** | Runtime peer management (add/remove without editing config) | Peers are declarative so no write path or DB table exists (DEC-14); would need a write path against the read-only rule |
| **DOC-1** | Show the MCP tools (`x-mcp-tools`) in the Redoc page | Redoc renders REST operations only; needs the tool list generated into a tag description (or a section) in `openapi.rs`, which changes the spec |
| **DOC-2** | Config switch to turn the `/docs` viewer off | Static, public and data-free (DEC-16), so not needed yet; adds a config key, README and `config.example.toml` entries |
- [ ] **C7** *(nice-to-have)* Smoke test against the Podman socket
- [ ] **C7** _(nice-to-have)_ Smoke test against the Podman socket
- [ ] **I6-5** Verification #5 OTel smoke test against a working collector (see the [verification checklist](ARCHITECTURE.md#verification))
- [ ] **B-1** Docker log-follow SSE live check
- [ ] **B-2** REST, auth and TLS live check against minipc1
- [ ] **X-1** *(external)* Fix or investigate `opentelemetry-collector.service` / `var-lib-loki.mount` on minipc1
- [ ] **X-1** _(external)_ Fix or investigate `opentelemetry-collector.service` / `var-lib-loki.mount` on minipc1
- [ ] **P5-1** minipc1 cross-check: run a consolidator against minipc1 with a read-only token the user mints there (`list_peers`, `get_peer`, one pass-through)
- [ ] **P5-2** OTel peer gauges: run `devenv up`, enable `[otel]` with peers, confirm `peer.up`, `peer.api.auth_ok`, `peer.check.duration_seconds`, `peer.last_change_timestamp_seconds` reach the collector, and record the result
- [ ] **D8** Fleet aggregation across peers
- [ ] **P5-3** Scoped tokens
- [ ] **P5-4** Peer health history
- [ ] **P5-5** Runtime peer management
- [ ] **DOC-1** MCP tools in the Redoc page
- [ ] **DOC-2** Config switch for the `/docs` viewer
- [ ] **F-1** Cap concurrent follow streams. Each journal follow, `tail_journal` call or docker follow pins a `spawn_blocking` thread for up to `limits.max_follow_secs`. Add a global semaphore shared by REST and MCP (suggested 16) returning 429/503 when full. A client that stops reading may hold the thread past the deadline (untested); add a write timeout or a hard `tokio::time::timeout` around the producer.
- [ ] **F-2** Bound line and message length. Docker `LineSplitter` and journal `MESSAGE` are buffered without limit before the 256 KiB result cap applies. Truncate per line (suggested 64 KiB, with a visible marker) and cap total bytes collected.
- [ ] **F-3** Unit hardening: `ProtectKernelLogs`, `ProtectHostname`, `RemoveIPC`, `LimitNOFILE`, `TasksMax`, `MemoryMax`, narrower `RestrictAddressFamilies`, `PrivateUsers` (check journal group access). Test under the hardened unit on a clean host before enabling. Document how the service user gets read access to the TLS key.
@@ -66,22 +70,22 @@ Parked, not scheduled. Each has a reason. Nothing here blocks the current work.
Design notes, owned files and logs for each are archived under [`docs/workstreams/`](docs/workstreams/). Task IDs below are stable references.
| ID | What | Key outcome | Tasks | Completed |
|---|---|---|---|---|
| [P0](docs/workstreams/P0-foundation.md) | Foundation: devenv, Cargo, config, DTOs, source traits + fakes, errors, `AppState`, stubs | `serve --print-config` works; build, clippy, fmt, test clean; `cargo tree -i openssl` empty | P0-1..P0-11, Gate | merged to `main` |
| [WS-A](docs/workstreams/WS-A-systemd.md) | systemd source (zbus) | Units, timers (via `*.timer` + `Timer` props), read-only `GetUnit` status | A1..A7 | merged to `main` |
| [WS-B](docs/workstreams/WS-B-journal.md) | journal source (sd-journal) | Snapshot + follow in `spawn_blocking`; follow thread exits on receiver drop | B1..B6 | merged to `main` |
| [WS-C](docs/workstreams/WS-C-docker.md) | docker source (bollard) | Lazy socket, bounded concurrent inspect, log streams; missing socket maps to `Unavailable` | C1..C6 (C7 in Backlog) | merged to `main` |
| [WS-D](docs/workstreams/WS-D-auth.md) | auth + SQLite token DB | sha256 tokens, `slinuxm token create`, `list`, `revoke`, bearer middleware, 401 + `WWW-Authenticate` | D1..D7 | merged to `main` |
| [WS-E](docs/workstreams/WS-E-rest.md) | REST API | All `/api/v1` routes, SSE follow, `/healthz`, 503 on docker-only failures | E1..E9 | merged to `main` |
| [WS-F](docs/workstreams/WS-F-mcp.md) | MCP server | 8 tools, 2 prompts, 2 resources, instructions, Origin/Host validation | F1..F9 | merged to `main` |
| [WS-G](docs/workstreams/WS-G-otel.md) | OTel export | Traces, observable-gauge metrics, journal/docker log fan-out with reconcile, flush on shutdown, `dev/otelcol.yaml` | G1..G7 | merged to `main` |
| [P3](docs/workstreams/P3-integration.md) | Integration and hardening | One axum server, negative tests, hardened unit, README; I6 verified on minipc1 (#1–#4, #6) | I1..I6 (I6-5 in Backlog) | 2026-10-19 |
| [WS-I](docs/workstreams/WS-I-tls.md) | Native TLS (rustls) | `server.tls.*`, SIGHUP reload, HTTPS tests, live `curl --cacert` check | T1..T8 | merged to `main` |
| [WS-H](docs/workstreams/WS-H-nixos.md) | NixOS package, module, VM test | H1..H8 merged (PR #1, 34da657); gate passed 2026-10-19 | H1..H8 | merged; gate passed 2026-10-19 |
| [P5](docs/workstreams/P5-consolidator/README.md) | Consolidator: optional peers (WS-T, A..F) | Declarative peers with TCP and API health checks, `/api/v1/peers` status and GET pass-through, `peer` argument and `list_peers`/`get_peer` (10 MCP tools with peers, 8 without), four OTel gauges, NixOS module option, SSRF guard, peers VM test; peer-less instance unchanged | T1..T7, A0..A4, B1..B5, C1..C4, D1..D3, E1..E2, F1..F4 (P5-1, P5-2 in Backlog) | 2026-10-20 |
| [WS-G (OpenAPI)](docs/workstreams/WS-G-openapi.md) | Generated OpenAPI 3.1 doc (stand-alone, after P5) | `utoipa` paths plus schemars bridge, `x-mcp-tools`, `GET /api/v1/openapi.json`, `slinuxm openapi [--peers]`, drift test, `docs/openapi.json` | G1..G4 | 2026-10-20 |
| [WS-H (final fixes)](docs/workstreams/WS-H-final-fixes.md) | Phase 5 final-review fixes | One token loader, one set of peer helpers, IPv4-mapped deny check, IPv6 `deny_cidrs`, client span kind, flaky test fixes, docs corrections | review items | 2026-10-20 |
| ID | What | Key outcome | Tasks | Completed |
| ---------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | ------------------------------ |
| [P0](docs/workstreams/P0-foundation.md) | Foundation: devenv, Cargo, config, DTOs, source traits + fakes, errors, `AppState`, stubs | `serve --print-config` works; build, clippy, fmt, test clean; `cargo tree -i openssl` empty | P0-1..P0-11, Gate | merged to `main` |
| [WS-A](docs/workstreams/WS-A-systemd.md) | systemd source (zbus) | Units, timers (via `*.timer` + `Timer` props), read-only `GetUnit` status | A1..A7 | merged to `main` |
| [WS-B](docs/workstreams/WS-B-journal.md) | journal source (sd-journal) | Snapshot + follow in `spawn_blocking`; follow thread exits on receiver drop | B1..B6 | merged to `main` |
| [WS-C](docs/workstreams/WS-C-docker.md) | docker source (bollard) | Lazy socket, bounded concurrent inspect, log streams; missing socket maps to `Unavailable` | C1..C6 (C7 in Backlog) | merged to `main` |
| [WS-D](docs/workstreams/WS-D-auth.md) | auth + SQLite token DB | sha256 tokens, `slinuxm token create`, `list`, `revoke`, bearer middleware, 401 + `WWW-Authenticate` | D1..D7 | merged to `main` |
| [WS-E](docs/workstreams/WS-E-rest.md) | REST API | All `/api/v1` routes, SSE follow, `/healthz`, 503 on docker-only failures | E1..E9 | merged to `main` |
| [WS-F](docs/workstreams/WS-F-mcp.md) | MCP server | 8 tools, 2 prompts, 2 resources, instructions, Origin/Host validation | F1..F9 | merged to `main` |
| [WS-G](docs/workstreams/WS-G-otel.md) | OTel export | Traces, observable-gauge metrics, journal/docker log fan-out with reconcile, flush on shutdown, `dev/otelcol.yaml` | G1..G7 | merged to `main` |
| [P3](docs/workstreams/P3-integration.md) | Integration and hardening | One axum server, negative tests, hardened unit, README; I6 verified on minipc1 (#1–#4, #6) | I1..I6 (I6-5 in Backlog) | 2026-10-19 |
| [WS-I](docs/workstreams/WS-I-tls.md) | Native TLS (rustls) | `server.tls.*`, SIGHUP reload, HTTPS tests, live `curl --cacert` check | T1..T8 | merged to `main` |
| [WS-H](docs/workstreams/WS-H-nixos.md) | NixOS package, module, VM test | H1..H8 merged (PR #1, 34da657); gate passed 2026-10-19 | H1..H8 | merged; gate passed 2026-10-19 |
| [P5](docs/workstreams/P5-consolidator/README.md) | Consolidator: optional peers (WS-T, A..F) | Declarative peers with TCP and API health checks, `/api/v1/peers` status and GET pass-through, `peer` argument and `list_peers`/`get_peer` (10 MCP tools with peers, 8 without), four OTel gauges, NixOS module option, SSRF guard, peers VM test; peer-less instance unchanged | T1..T7, A0..A4, B1..B5, C1..C4, D1..D3, E1..E2, F1..F4 (P5-1, P5-2 in Backlog) | 2026-10-20 |
| [WS-G (OpenAPI)](docs/workstreams/WS-G-openapi.md) | Generated OpenAPI 3.1 doc (stand-alone, after P5) | `utoipa` paths plus schemars bridge, `x-mcp-tools`, `GET /api/v1/openapi.json`, `slinuxm openapi [--peers]`, drift test, `docs/openapi.json` | G1..G4 | 2026-10-20 |
| [WS-H (final fixes)](docs/workstreams/WS-H-final-fixes.md) | Phase 5 final-review fixes | One token loader, one set of peer helpers, IPv4-mapped deny check, IPv6 `deny_cidrs`, client span kind, flaky test fixes, docs corrections | review items | 2026-10-20 |
### Done (archive)