This commit is contained in:
ari committed 2026-10-21 00:12:33 -04:00
1 parent d8ad1c44c5
commit 72fccc34f8
14 files changed
+811 -261

No files matched your search

+3 -3
View File
@@ -12,7 +12,7 @@ Canonical guidance for AI coding agents in this repo (no CLAUDE.md; this file is
- `tests/`, `docs/openapi.json` (generated; regenerate with `cargo run -- openapi --peers > docs/openapi.json`), `examples/`, `migrations/`, `dev/otelcol.yaml`, `systemd/` (example unit), `flake.nix` + `nix/` (package, NixOS module, VM test), `config.example.toml`
- [`ARCHITECTURE.md`](ARCHITECTURE.md): design, decisions (DEC-n), gotchas, verification checklist. Read before structural changes.
- [`TODO.md`](TODO.md): all work tracking. [`plan/`](plan/README.md): plans for the next big phase only. [`docs/`](docs/README.md): long-form docs, archived workstream records.
- [`README.md`](README.md): user-facing install and usage.
- [`README.md`](README.md): user-facing overview; the user guide is `docs/how-it-works.md`, `install.md`, `configuration.md`, `operations.md` (development notes in `docs/development.md`).
## Commands
@@ -47,7 +47,7 @@ Git hooks run fmt, clippy, lint and a secret scan on commit and path-filtered te
- Default port is 1984, default bind is loopback. Do not change defaults for convenience.
- Comments: sparse, explain why not what. No workstream, task or review tags (`WS-x`, `G3`, `DEC-n`) in code comments; reference ARCHITECTURE.md by section if needed.
- The MCP tool count is 8 without peers and 10 with `peers.enabled` (`list_peers`, `get_peer`, and an optional `peer` argument on the other eight); the tool-list snapshot test guards both tables. Change tools deliberately.
- Keep `config.example.toml` and `README.md` in sync with config or CLI changes.
- Keep `config.example.toml`, `docs/configuration.md` and `docs/how-it-works.md` (command list) in sync with config or CLI changes; NixOS option changes go in `docs/install.md`.
## Safety
@@ -67,6 +67,6 @@ Git hooks run fmt, clippy, lint and a secret scan on commit and path-filtered te
1. `devenv shell -- dev-test all` (fmt, clippy, unit and integration tests) passes; that is `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings` and `cargo test`.
2. New behaviour has tests (fakes for handlers and tools; `tests/` for end to end).
3. Docs updated: `README.md`, `config.example.toml`, `ARCHITECTURE.md` as relevant.
3. Docs updated: `README.md`, the guides in `docs/`, `config.example.toml`, `ARCHITECTURE.md` as relevant.
4. `TODO.md` updated; any deferred work is in the Backlog with a reason.
5. No new openssl, no write path to the host, no secrets in the diff.
+2 -2
View File
@@ -2,7 +2,7 @@
skyg-linux-tracker (binary and CLI: `slinuxm`) is a read-only Linux system monitor in Rust, shipped as a single binary. It exposes systemd state, journald logs and Docker container state/logs over three surfaces that share one core: a **REST API**, an **MCP server** (Streamable HTTP) and **OpenTelemetry export**. Everything is read live from the source on every request. Nothing is collected or buffered, except a small SQLite database holding hashed API tokens.
User-facing usage is in [`README.md`](README.md). Work tracking is in [`TODO.md`](TODO.md). Per-workstream design logs from the original build and Phase 5 are archived in [`docs/workstreams/`](docs/workstreams/).
User-facing docs start at [`README.md`](README.md) and the guides in [`docs/`](docs/README.md). Work tracking is in [`TODO.md`](TODO.md). Per-workstream design logs from the original build and Phase 5 are archived in [`docs/workstreams/`](docs/workstreams/).
## Requirements
@@ -214,7 +214,7 @@ All accepted. Rationale is kept; the build history is in [`docs/workstreams/`](d
- **DEC-3 runtime queries.** `sqlx::query!` macros need `DATABASE_URL` at build time or a committed `.sqlx/` cache. With one table and about five queries, runtime `query`/`query_as` plus tests gives the same safety with no build-time DB.
- **DEC-4 HTTP 401.** The MCP authorization spec puts auth at the HTTP layer and clients expect 401 with `WWW-Authenticate: Bearer`. One middleware gives the same response for `/api/v1` and `/mcp`; `AppError::Unauthorized` renders it.
- **DEC-5 no subscriptions.** Subscriptions need change detection, i.e. background polling, which conflicts with "live reads only". Resources are resolved live on each read.
- **DEC-6 Docker socket.** `docker` group membership is root-equivalent. The unit ships with `SupplementaryGroups=docker` commented out; the hardened setup points `sources.docker_socket` at a read-only socket proxy (see the README). No code change is needed either way. The NixOS module's `docker.enable` adds the group explicitly.
- **DEC-6 Docker socket.** `docker` group membership is root-equivalent. The unit ships with `SupplementaryGroups=docker` commented out; the hardened setup points `sources.docker_socket` at a read-only socket proxy (see [docs/operations.md](docs/operations.md#docker-access)). No code change is needed either way. The NixOS module's `docker.enable` adds the group explicitly.
- **DEC-7 metrics scope.** "Empty allowlist = all units" would export every loaded unit (often 300+ including devices and mounts), so empty means `*.service` and `*.timer`.
- **DEC-8 Origin validation.** The P0 spike showed rmcp 3.5 validates both headers itself: `with_allowed_hosts` (default loopback names) and `with_allowed_origins` (default empty means off; requests without `Origin` always pass). No custom tower layer. Hosts: loopback names, `server.bind`, `server.allowed_hosts`. Origins: `http(s)://localhost:*`, `127.0.0.1:*`, `[::1]:*`, `<bind>:*` plus `server.allowed_origins`.
- **DEC-9 defaults.** Binary and NixOS module both default to port 1984 and `bind = "127.0.0.1"` (the binary default was 9109 until 2026-10-16). LAN exposure is an install-time choice: set `bind`, `openFirewall` or an interface-scoped rule, and `allowedHosts`.
+45 -251
View File
@@ -1,268 +1,62 @@
# skyg-linux-tracker
A small, read-only monitor for a single Linux host. One binary, `slinuxm`, serves:
A small, **read-only** monitor for a Linux host. One binary, `slinuxm`, tells you what systemd, the journal and Docker are doing, through:
- a REST API (with Server-Sent Events for follow endpoints),
- an MCP server (Streamable HTTP) so agents can inspect the host,
- optional OpenTelemetry export (metrics and logs over OTLP).
- a **REST API** (JSON, with Server-Sent Events for live logs),
- an **MCP server** so AI agents can inspect the host,
- optional **OpenTelemetry** export of metrics and logs.
Data sources are systemd (D-Bus), the journal (libsystemd) and the Docker Engine API (unix socket). Nothing here can start, stop or modify anything.
It reads everything live, keeps no data of its own beyond a database of hashed API tokens, and cannot start, stop or change anything on the host.
## Quick start
## Try it in a minute
Try it on your own machine in a minute. It needs no root and no install, and it keeps its token database in a temp directory.
No root, no install. The token database goes in a temp directory.
```sh
devenv shell && cargo build --release # or just: cargo build --release with the prerequisites below
cargo build --release # needs Rust, pkg-config, lld, libsystemd headers (or: devenv shell)
export SKYG_AUTH__DB_PATH=/tmp/slinuxm-demo/db.sqlite
BIN=target/release/slinuxm
TOKEN=$($BIN token create demo) # printed once, never stored in plaintext
TOKEN=$($BIN token create demo) # printed once, stored only as a hash
$BIN serve & # http://127.0.0.1:1984, loopback only
curl -s http://127.0.0.1:1984/healthz # public
curl -s http://127.0.0.1:1984/healthz
curl -s -H "Authorization: Bearer $TOKEN" 'http://127.0.0.1:1984/api/v1/systemd/units?state=failed'
curl -s -H "Authorization: Bearer $TOKEN" 'http://127.0.0.1:1984/api/v1/journal?priority=err&lines=20'
curl -s -H "Authorization: Bearer $TOKEN" http://127.0.0.1:1984/api/v1/docker/ps # 503 if Docker is not running
kill %1
```
Point an MCP client at `http://127.0.0.1:1984/mcp` with the same bearer header (see [MCP](#mcp)). Stop the server with `kill %1`.
The service listens on loopback by default. To reach it from another machine set `server.bind`, keep auth on, and use [TLS](#tls). For a permanent install as a systemd service, see [Install](#install) and [Tokens](#tokens).
## Build
```sh
devenv shell # toolchain, pkg-config, lld, libsystemd, otelcol-contrib
cargo build --release # target/release/slinuxm
```
Without devenv you need a Rust toolchain, `pkg-config`, the `lld` linker (`.cargo/config.toml` links with it) and the libsystemd development files.
### Version
The binary carries a build-stamped version, `YYYY.MM.DD-<branch>-<sha>` (for example `2026.10.21-main-435a54a`). `slinuxm --version`, `GET /healthz`, `GET /api/v1/ping`, the MCP server info and the OTel `service.version` attribute all report it. `Cargo.toml` keeps a semver `version`, which is what the OpenAPI document's `info.version` uses so that `docs/openapi.json` does not change on every commit.
- The date is the build day in UTC, the branch has every character outside `[A-Za-z0-9._-]` replaced by `-` (`feature/x` becomes `feature-x`, a detached HEAD becomes `detached`), and the sha is 7 characters, with `-dirty` appended when tracked files have uncommitted changes.
- Without git (a source tarball) the missing parts are `unknown`.
- `build.rs` reads `SLINUXM_GIT_SHA`, `SLINUXM_GIT_BRANCH` and `SLINUXM_BUILD_DATE` (`YYYY.MM.DD`) when set, and `SLINUXM_VERSION` replaces the whole string. `SOURCE_DATE_EPOCH` is ignored on purpose, because Nix dev shells export a fixed 1980 value.
- `nix build` has no `.git`, so the flake passes the commit date and short rev, and the branch is the fixed placeholder `nix` (a flake does not know its branch). Override it with `pkgs.callPackage ./nix/package.nix { gitBranch = "main"; … }`.
## Install
```sh
sudo install -m 0755 target/release/slinuxm /usr/local/bin/slinuxm
sudo useradd --system --no-create-home --shell /usr/sbin/nologin skyg
sudo install -m 0644 config.example.toml /etc/skyg-linux-tracker.toml # then edit
sudo install -m 0644 systemd/skyg-linux-tracker.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now skyg-linux-tracker
```
The unit runs as the `skyg` user with `SupplementaryGroups=systemd-journal` and a private `StateDirectory` (`/var/lib/skyg-linux-tracker`, mode 0700) holding the token database. Its sandbox options are listed in the unit file.
### Config
See `config.example.toml`; every value shown is the default. Precedence, lowest to highest: defaults, config file, `SKYG_*` environment variables (nesting with `__`, for example `SKYG_SERVER__PORT`), CLI flags. `slinuxm --print-config` prints the effective configuration with secrets redacted.
Startup refuses `auth.enabled = false` on a non-loopback `server.bind`.
## Tokens
All `/api/v1/*` and `/mcp` requests need `Authorization: Bearer <token>`. `/healthz` and the static `/docs` viewer are public. Tokens are stored hashed (sha256) in SQLite, and the plaintext is shown once.
Run the token CLI **as the service user** (the unit's `User=skyg`), i.e. `sudo -u <service user> slinuxm token create <name>`. The CLI refuses to run (with the matching `sudo -u <owner> ...` hint) when the effective uid, root included, does not own the database file or, if it does not exist yet, its existing state directory, so it cannot create or modify state owned by another user:
```sh
sudo -u skyg slinuxm token create my-laptop --config /etc/skyg-linux-tracker.toml
sudo -u skyg slinuxm token list --config /etc/skyg-linux-tracker.toml
sudo -u skyg slinuxm token revoke my-laptop --config /etc/skyg-linux-tracker.toml
```
Revocation takes effect immediately; no restart is needed.
Notes:
- `auth.db_path` must stay under the unit's `StateDirectory` (`/var/lib/skyg-linux-tracker`). The unit uses `ProtectSystem=strict`, so any other location needs a matching `ReadWritePaths=`.
- `token create` needs the state directory to exist, so start the service once first (systemd creates it with the right owner), then run the CLI as `skyg`.
- A state directory the server creates itself is made 0700 and the database file 0600 from creation. An existing directory is never chmod'd; a group/world-accessible one only logs a warning.
## REST
Base: `http://127.0.0.1:1984`, all routes are GET and read-only. `GET /healthz` is public; everything under `/api/v1` needs the bearer token. Routes: `systemd/units`, `systemd/units/{name}`, `systemd/timers`, `journal`, `journal/follow` (SSE), `docker/ps`, `docker/containers/{name}`, `docker/containers/{name}/logs` (JSON or SSE). Docker routes return `503` when Docker is unavailable. Errors are `{"error":{"code","message"}}`.
```sh
curl -H "Authorization: Bearer $TOKEN" 'http://127.0.0.1:1984/api/v1/systemd/units?state=failed'
curl -N -H "Authorization: Bearer $TOKEN" 'http://127.0.0.1:1984/api/v1/journal/follow?priority=err'
```
Full route table with parameters: [docs/api.md](docs/api.md).
The whole surface is also a generated OpenAPI 3.1 document, [`docs/openapi.json`](docs/openapi.json): `GET /api/v1/openapi.json` serves it (same bearer token), `slinuxm openapi [--peers]` prints it offline without reading any config or token, and `x-mcp-tools` in it lists every MCP tool with its JSON Schema. Regenerate with `cargo run -- openapi --peers > docs/openapi.json`; a test fails if the file is stale.
The same document is rendered at `http://127.0.0.1:1984/docs` with [Redoc](https://github.com/Redocly/redoc), served from the binary (no CDN, no outbound request). The page and its script are public and hold no host data; the page asks for your bearer token (kept in `sessionStorage`, sent only to this server) to fetch `/api/v1/openapi.json`, or loads straight away with `[auth] enabled = false`.
## MCP
Endpoint: `http://127.0.0.1:1984/mcp` (Streamable HTTP, same bearer token). Eight tools (`list_systemd_units`, `get_systemd_unit`, `list_systemd_timers`, `get_journal_logs`, `tail_journal`, `list_containers`, `get_container`, `get_container_logs`), two prompts, two resources and two resource templates. A [consolidator](#peers-consolidator) adds `list_peers` and `get_peer` and a `peer` argument on every tool, for ten tools. Log content is untrusted data and is labelled as such.
Claude Code: `claude mcp add --transport http linux-tracker http://127.0.0.1:1984/mcp --header "Authorization: Bearer $TOKEN"`.
`/mcp` validates `Origin` and `Host` against DNS rebinding. If you bind to `0.0.0.0` or serve behind a name, add the names clients use to `server.allowed_hosts` (for example `["monitor.lan:1984"]`) and browser origins to `server.allowed_origins`. Generic client config, Inspector and curl handshake, and the full Origin/Host rules: [docs/api.md](docs/api.md#mcp).
## TLS
TLS is optional and off by default; a non-loopback `bind` without TLS logs a startup warning. Enable it with a PEM chain and key:
```toml
[server.tls]
enabled = true
cert_file = "/etc/skyg/tls/cert.pem" # PEM chain, leaf first
key_file = "/etc/skyg/tls/key.pem" # PEM, PKCS#8, PKCS#1 or SEC1
```
REST and `/mcp` are then served over HTTPS only. Replace the files and send `SIGHUP` to rotate without dropping connections (a bad pair keeps the old certificate). rustls only; no openssl. Self-signed example, rotation caveats, ACME and sops paths: [docs/tls.md](docs/tls.md).
## Peers (consolidator)
One `slinuxm` can watch other instances: declare them as **peers** and it health-checks each one and passes read-only requests through to it, so one MCP endpoint or one REST base URL covers the fleet. Off by default; a peer-less instance keeps exactly the same routes, eight MCP tools, metrics and background tasks as before.
```toml
[peers]
enabled = true # other intervals, timeouts and caps: config.example.toml
[[peers.peer]]
name = "minipc1" # ^[a-z0-9][a-z0-9_-]{0,31}$, at most 64 peers
url = "https://minipc1:1984" # http:// to a non-loopback host also needs allow_plain_http = true
token_file = "/run/credentials/skyg-linux-tracker.service/peer-minipc1"
ca_file = "/etc/skyg/private-ca.pem" # optional extra trust anchor
[[peers.peer]]
name = "nas"
url = "http://nas:1984"
token_file = "/etc/skyg/peer-nas.token" # 0600, owned by the service user
allow_plain_http = true
```
Peers are only ever declared in configuration, and every address a peer's name resolves to is checked against `deny_cidrs` before a connection is made; the defaults refuse link-local and cloud-metadata addresses in both families (`169.254.0.0/16`, `fe80::/10`, `fd00:ec2::254/128`) and the unspecified address (`0.0.0.0/8`, `::/128`), while private LAN ranges stay allowed.
`token_file` holds a path, never the token. A consolidator has to replay the peer's plaintext bearer token, so the token lives in a file managed outside `slinuxm` (agenix on NixOS), is read into memory on every check and request, and never appears in the config, `--print-config`, the database or the logs. Mint **one token per consolidator on every peer** and use it only there (`sudo -u <service user> slinuxm token create consolidator-<host>`), so a single `token revoke` on a peer cuts that consolidator off and nothing else. Replacing the file is enough to rotate: the next check reads the new value, and no restart is needed. On NixOS also run `systemctl reload skyg-linux-tracker` so systemd re-copies the credential the unit holds (`RefreshOnReload=credentials`).
Two checks per peer, kept in memory and reset on restart:
- **server**: a TCP connect to the peer's port.
- **API**: an authenticated `GET /api/v1/ping`. A `401` means the API runs and our token does not: `api.up = true` with `api.auth = "rejected"`. That is reported to you as a `503` naming the peer, never as a `401` that would suggest re-authenticating with this host. A peer without the ping route falls back to public `/healthz` and reports `api.auth = "unknown"`.
```sh
curl -H "Authorization: Bearer $TOKEN" http://127.0.0.1:1984/api/v1/peers
curl -H "Authorization: Bearer $TOKEN" 'http://127.0.0.1:1984/api/v1/peers/minipc1/systemd/units?state=failed'
sudo -u skyg slinuxm peer list --config /etc/skyg-linux-tracker.toml
sudo -u skyg slinuxm peer check minipc1 --config /etc/skyg-linux-tracker.toml
```
`GET /api/v1/peers` lists every peer, `GET /api/v1/peers/{peer}` one of them. `GET /api/v1/peers/{peer}/...` is a GET-only, allowlisted pass-through (`systemd/...`, `journal`, `journal/follow`, `docker/...`): the peer's answer comes back unchanged with `x-slinuxm-peer: <peer>`, a peer that is down, rejects our token or times out is a `503` naming it, `x-slinuxm-hop` from a caller is a `400`, and unknown peers are a `404`. With peers enabled MCP grows `list_peers` and `get_peer`, and every other tool accepts an optional `peer` argument (ten tools instead of eight). There is no aggregation across peers in this version: ask one peer at a time. OTel gains `peer.up`, `peer.api.auth_ok`, `peer.check.duration_seconds` and `peer.last_change_timestamp_seconds`.
On NixOS the module declares peers and delivers each token through systemd `LoadCredential`, so the service user never needs to read the agenix output:
```nix
services.skyg-linux-tracker.peers = {
enable = true;
peers.minipc1 = {
url = "https://minipc1:1984";
tokenFile = config.age.secrets.peer-minipc1.path; # a string path, never a path literal
};
};
```
Worked agenix setup, the threat model, the firewall note (every peer must accept the consolidator's address on its port), reload and shutdown semantics: [docs/consolidator.md](docs/consolidator.md). The peers tests are `tests/peers.rs` (in-process, no VM) and the three-machine NixOS VM test: `devenv shell -- dev-test integration peers`, and for the VM test `devenv shell -- dev-test vm peers`, which is `nix build .#checks.x86_64-linux.vm-peers -L`.
## NixOS
The flake exports `packages.<system>.default` and `nixosModules.default`.
```nix
# flake.nix
inputs.skyg-linux-tracker.url = "github:<owner>/linux-system-monitor";
# in your nixosSystem modules
modules = [
skyg-linux-tracker.nixosModules.default
{ services.skyg-linux-tracker.enable = true; }
];
```
That runs the service as the `skyg-linux-tracker` user on `127.0.0.1:1984` with the same sandbox as `systemd/skyg-linux-tracker.service`, writes `/etc/skyg-linux-tracker.toml` (the path the CLI reads by default) and installs `slinuxm` system-wide. Mint the first token as the service user:
```sh
sudo -u skyg-linux-tracker slinuxm token create my-laptop
```
Options (under `services.skyg-linux-tracker`): `port` (1984), `bind` (`"127.0.0.1"`), `tls.{enable,certFile,keyFile}` (passed through `LoadCredential`, so the service user never reads the original key; `systemctl reload skyg-linux-tracker` re-reads them; for ACME add `security.acme.certs.<name>.reloadServices = [ "skyg-linux-tracker" ]`), `openFirewall` (all interfaces), `firewallInterfaces` (recommended for a LAN), `allowedHosts`, `docker.enable` (adds the root-equivalent `docker` group), `settings` (freeform TOML merged over the module defaults) and `environmentFile` (secrets such as `SKYG_OTEL__HEADERS__AUTHORIZATION`, kept out of the store and out of the generated TOML).
Exposure: keep the default loopback bind and reach it over Tailscale or a TLS reverse proxy. Bearer tokens over plain HTTP are readable on a LAN. A non-loopback `bind` with `auth.enabled = false` fails at evaluation time. For `bind = "0.0.0.0"` add the names clients use to `allowedHosts` (`networking.hostName` is added for you), and prefer `firewallInterfaces = [ "enp3s0" ]` over `openFirewall`.
```nix
services.skyg-linux-tracker = {
enable = true;
bind = "0.0.0.0";
firewallInterfaces = [ "enp3s0" ];
allowedHosts = [ "monitor.lan:1984" ];
tls = { enable = true; certFile = "/var/lib/acme/monitor.lan/fullchain.pem"; keyFile = "/var/lib/acme/monitor.lan/key.pem"; };
};
```
`nix flake check` runs a two-machine VM test (plain and TLS) covering healthz, 401, token mint and revoke through the CLI, `/mcp` and the Origin check.
## Docker access
Reading the Docker socket directly requires group `docker`, which is root-equivalent. The unit file has this commented out. Prefer a read-only socket proxy that allows only `GET /containers/*`, `/events` and `/version`, for example [tecnativa/docker-socket-proxy](https://github.com/Tecnativa/docker-socket-proxy) with `CONTAINERS=1 EVENTS=1 POST=0`, exposed as a unix socket (or loopback TCP through a socat bridge), then:
```toml
[sources]
docker_socket = "/run/docker-proxy/docker.sock"
```
A stock install does not add the `docker` group, so out of the box the Docker routes return `503` until you either grant access or configure a proxy. If the socket is missing or unreadable, Docker routes return `503` and everything else keeps working.
## OpenTelemetry
```toml
[otel]
enabled = true
endpoint = "http://localhost:4317" # gRPC; or http://localhost:4318 with protocol = "http/protobuf"
export_logs = true
```
Metrics are polled every `metrics_interval_secs` (systemd unit active/failed/restarts and timer next elapse; container running/healthy/uptime/restarts). Logs fan out from the journal and from Docker containers. For a local collector with a debug exporter: `devenv up` (runs `otelcol-contrib --config dev/otelcol.yaml`). On SIGTERM pending records are flushed before exit.
## Verifying a release binary
```sh
ldd /usr/local/bin/slinuxm # libc, libm, libgcc_s, libsystemd and its dependencies only
```
## Testing
Checks run through one runner, `dev/dev-test.nu`. `devenv shell -- dev-test …` runs it from any shell, including an interactive one. `devenv shell -- dev-test --help` lists every subcommand and the cargo or nix command behind it.
```sh
devenv shell -- dev-test unit # cargo test --lib --bins
devenv shell -- dev-test integration [filter] [--nocapture] # cargo test --test '*' (integration targets only)
devenv shell -- dev-test lint # cargo fmt --check, clippy -D warnings, no openssl
devenv shell -- dev-test vm [vm|peers] # NixOS VM test (slow, needs /dev/kvm)
devenv shell -- dev-test all # lint, unit, integration; stops at the first failure
devenv shell -- dev-test ci # all plus the vm test
devenv shell -- smoke # same as: cargo test --test binary_smoke -- --nocapture
```
The smoke test runs the real built `slinuxm` binary against a throwaway SQLite database in a temp directory: it creates a token with the CLI, starts `serve` on a free loopback port, exercises `/healthz`, REST and MCP with and without the token, revokes the token, and checks a clean SIGTERM exit and a 0600 database. It needs no root and no network. Checks that depend on host data (systemd over D-Bus, the journal) are skipped with a printed reason when the source is unavailable; Docker is never required.
## Development
```sh
devenv shell -- dev-test all # fmt, clippy, unit and integration tests
```
`devenv shell -- dev-test --help` shows the underlying commands (`cargo fmt --check`, `cargo clippy --all-targets -- -D warnings`, `cargo test --lib --bins`, `cargo test --test '*'`). Nushell: `use dev/dev-test.nu *` gives `dev-test unit`, `dev-test vm peers`, ... with tab completion. zsh users add `source <(nu dev/dev-test.nu completions zsh)` to `.zshrc` after `compinit`. An interactive `devenv shell` installs bash completion. The integration tests in `tests/` start the real router on an ephemeral port with fake sources. Design notes live in [ARCHITECTURE.md](ARCHITECTURE.md); work tracking in [TODO.md](TODO.md); extended docs in [docs/](docs/README.md).
Contributors: enter `devenv shell` to install the git hooks; details in [docs/hooks.md](docs/hooks.md).
Or with Nix: `nix run git+ssh://gitea@gitea.app.internal/Hermenix/skyg-linux-tracker -- --help`.
## Documentation
Read in this order:
| Guide | What it covers |
| --- | --- |
| [How it works](docs/how-it-works.md) | What the binary does when it runs, what it reads, auth, signals, every command |
| [Install](docs/install.md) | **Nix and NixOS module**, or from source with an example systemd unit |
| [Configuration](docs/configuration.md) | Config file, environment variables, CLI flags, every section |
| [Operations](docs/operations.md) | Tokens, MCP clients, network exposure, Docker access, OpenTelemetry, troubleshooting |
Reference:
| Doc | What it covers |
| --- | --- |
| [API reference](docs/api.md) | Every REST route and MCP tool; also served by the binary at `/docs` |
| [TLS](docs/tls.md) | HTTPS, certificate rotation, ACME |
| [Consolidator](docs/consolidator.md) | One instance watching others (peers) |
| [`config.example.toml`](config.example.toml) | Every config key with its default |
| [Development](docs/development.md) | Building, tests, git hooks, version stamp |
| [ARCHITECTURE.md](ARCHITECTURE.md) | Internals and design decisions |
## At a glance
| | |
| --- | --- |
| Binary | `slinuxm` (Linux, links libsystemd; no openssl) |
| Listens on | `127.0.0.1:1984` by default |
| Config | `/etc/skyg-linux-tracker.toml`, `SKYG_*` env vars, CLI flags |
| State | `/var/lib/skyg-linux-tracker/db.sqlite` (hashed tokens only) |
| Auth | Bearer tokens, `slinuxm token create <name>` as the service user |
| Sources | systemd over D-Bus, the journal via libsystemd, Docker over its unix socket |
+1
View File
@@ -26,6 +26,7 @@ The build is functionally complete and merged to `main`. REST, MCP, OTel, TLS an
- [x] **WS-H Gate:** `nix build` and `nix flake check` pass. `ldd` shows no openssl. A fresh NixOS host serves `:1984` and a token can be minted from a login shell.
- [x] **V-1** Build-stamped version `YYYY.MM.DD-<branch>-<sha>` in `--version`, `/healthz`, `/api/v1/ping`, MCP server info and OTel `service.version` (DEC-17). `dev-test lint`, `cargo test`, `nix build` and both VM tests (`vm`, `vm peers`) pass.
- [x] **DOC-3** User docs restructured into a guide: slim `README.md`, plus `docs/how-it-works.md`, `docs/install.md` (Nix and NixOS first; the systemd unit is an example for manual installs), `docs/configuration.md`, `docs/operations.md` and `docs/development.md`. The example unit gains `ExecReload` (SIGHUP), matching the module. markdownlint and link check pass.
## Backlog
+21 -1
View File
@@ -1,6 +1,15 @@
# Documentation index
Root files: [`README.md`](../README.md) (install and usage), [`ARCHITECTURE.md`](../ARCHITECTURE.md) (design, decisions, verification), [`TODO.md`](../TODO.md) (work tracking), [`AGENTS.md`](../AGENTS.md) (guidance for AI coding agents). Plans for the next big phase go in [`plan/`](../plan/README.md).
Start with the project [README](../README.md).
## User guide
Read in order:
1. [`how-it-works.md`](how-it-works.md): what the binary does, what it reads and serves, auth, lifecycle, signals, commands
2. [`install.md`](install.md): Nix package, NixOS module, or from source with an example systemd unit
3. [`configuration.md`](configuration.md): layers (file, env, flags), every section, validation, what reloads
4. [`operations.md`](operations.md): tokens, MCP clients, network exposure, Docker access, OpenTelemetry, troubleshooting
## Reference
@@ -11,6 +20,17 @@ Root files: [`README.md`](../README.md) (install and usage), [`ARCHITECTURE.md`]
| [`tls.md`](tls.md) | Native TLS: configuration, self-signed example, SIGHUP rotation, ACME and sops paths |
| [`consolidator.md`](consolidator.md) | Peers (consolidator): config, agenix tokens, threat model, firewall, reload |
## Contributors
| Doc | Contents |
| -------------------------------------- | ------------------------------------------------------------ |
| [`development.md`](development.md) | Toolchain, tests, version stamp, OpenAPI regeneration |
| [`hooks.md`](hooks.md) | Git hooks and flake checks |
| [`ARCHITECTURE.md`](../ARCHITECTURE.md) | Design, decisions, verification |
| [`TODO.md`](../TODO.md) | Work tracking |
| [`AGENTS.md`](../AGENTS.md) | Guidance for AI coding agents |
| [`plan/`](../plan/README.md) | Plans for the next big phase |
## Workstream records (archived)
> **Archived.** The files in [`workstreams/`](workstreams/) are historical records of how the project was built (scope, owned files, design notes, logs). They are not maintained. Current design is in [`ARCHITECTURE.md`](../ARCHITECTURE.md) and open work is in [`TODO.md`](../TODO.md).
+1 -1
View File
@@ -1,6 +1,6 @@
# REST and MCP reference
Full endpoint and MCP client reference. The short version is in the [README](../README.md); the design is in [ARCHITECTURE.md](../ARCHITECTURE.md#rest-api).
Full endpoint and MCP client reference. Everyday usage is in [operations.md](operations.md); the design is in [ARCHITECTURE.md](../ARCHITECTURE.md#rest-api).
## Generated reference (OpenAPI)
+129
View File
@@ -0,0 +1,129 @@
# Configuration
Every setting has a default, so `slinuxm serve` runs with no config file at all. [`config.example.toml`](../config.example.toml) lists every key with its default and its environment variable; this page explains how the layers combine and what each section does.
## Where settings come from
Four layers. A later layer wins over an earlier one, key by key:
1. **Built-in defaults**
2. **Config file**: `--config <file>`, or `SKYG_CONFIG`, or `/etc/skyg-linux-tracker.toml` if it exists
3. **Environment variables**: `SKYG_` plus the key path in capitals, with `__` between levels
4. **CLI flags**: `--bind`, `--port`, `--auth-enabled`, `--db-path`, `--docker-socket`, `--journal-namespace`, `--otel-enabled`, `--otel-endpoint`
So `server.port` can be set as `port = 2000` under `[server]`, as `SKYG_SERVER__PORT=2000`, or as `--port 2000`.
| Key | Environment variable |
| --- | --- |
| `server.port` | `SKYG_SERVER__PORT` |
| `server.tls.cert_file` | `SKYG_SERVER__TLS__CERT_FILE` |
| `otel.headers.authorization` | `SKYG_OTEL__HEADERS__AUTHORIZATION` |
| `otel.metrics.units` (a list) | `SKYG_OTEL__METRICS__UNITS='["nginx.service"]'` |
To see the result of all four layers:
```sh
slinuxm --print-config # secrets such as OTel headers are redacted
slinuxm --print-config --config ./test.toml
```
The CLI commands (`token`, `peer`) read the same layers, which is why they find the service's database without extra flags when run on the same host.
## Sections
### `[server]`: where it listens
| Key | Default | Meaning |
| --- | --- | --- |
| `bind` | `"127.0.0.1"` | Listen address. Use `"0.0.0.0"` (IPv4) or `"::"` (dual stack) to accept remote clients. |
| `port` | `1984` | Listen port. |
| `allowed_hosts` | `[]` | Extra `Host` header values accepted on `/mcp` (loopback names and `bind` are always accepted), e.g. `["monitor.lan:1984"]`. |
| `allowed_origins` | `[]` | Extra browser `Origin` values accepted on `/mcp`, with scheme, e.g. `["https://app.example.com"]`. |
`/mcp` checks `Host` and `Origin` to block DNS-rebinding attacks from browsers. If clients reach the server under a name, list that name in `allowed_hosts`.
### `[server.tls]`: HTTPS
| Key | Default | Meaning |
| --- | --- | --- |
| `enabled` | `false` | Serve HTTPS only on the port. |
| `cert_file` | `""` | PEM certificate chain, leaf first. |
| `key_file` | `""` | PEM private key (PKCS#8, PKCS#1 or SEC1). |
Both files are re-read on SIGHUP. Details, self-signed example and rotation: [tls.md](tls.md).
### `[auth]`: tokens
| Key | Default | Meaning |
| --- | --- | --- |
| `enabled` | `true` | Require a bearer token on `/api/v1` and `/mcp`. `false` is allowed on a loopback `bind` only. |
| `db_path` | `"/var/lib/skyg-linux-tracker/db.sqlite"` | The token database. Under the example systemd unit it must stay under `/var/lib/skyg-linux-tracker`. |
### `[sources]`: what it reads
| Key | Default | Meaning |
| --- | --- | --- |
| `docker_socket` | `"/var/run/docker.sock"` | Docker Engine API socket. Point it at a read-only socket proxy for a hardened setup. |
| `journal_namespace` | `""` | A journald namespace to read; empty means the default journal. |
### `[limits]`: request bounds
| Key | Default | Meaning |
| --- | --- | --- |
| `max_lines` | `1000` | Most journal lines one request can return. |
| `max_tail` | `1000` | Most container log lines one request can return. |
| `max_follow_secs` | `60` | Longest bounded follow (the MCP `tail_journal` tool, and proxied follows). |
### `[otel]`: OpenTelemetry export
| Key | Default | Meaning |
| --- | --- | --- |
| `enabled` | `false` | Master switch for OTLP export. |
| `endpoint` | `"http://localhost:4317"` | Collector address. Port 4317 for gRPC, 4318 for HTTP. |
| `protocol` | `"grpc"` | `"grpc"` or `"http/protobuf"`. |
| `service_name` | `"skyg-linux-tracker"` | The `service.name` resource attribute. |
| `export_unit_metrics` | `true` | Export unit and container status as gauges (when `enabled`). |
| `export_logs` | `false` | Forward journal and container logs as OTel logs. |
| `metrics_interval_secs` | `30` | How often the metrics are polled. |
| `[otel.headers]` | empty | Extra OTLP headers, e.g. `authorization`. Put secret values in the environment, not the file. |
| `[otel.logs] journal_units` | `[]` | Units whose logs are exported; empty means all. |
| `[otel.logs] docker_containers` | `[]` | Containers whose logs are exported; empty means all. |
| `[otel.metrics] units` | `[]` | Glob allowlist of units; empty means `*.service` and `*.timer` only. |
| `[otel.metrics] containers` | `[]` | Glob allowlist of containers; empty means all. |
What is exported: [operations.md](operations.md#opentelemetry).
### `[peers]`: consolidator mode
Off by default. When on, this instance health-checks other `slinuxm` instances declared as `[[peers.peer]]` entries and proxies read-only requests to them. All keys, validation rules and the credential model: [consolidator.md](consolidator.md#configuration).
## Validation
Every command, `--print-config` included, refuses a configuration that is unsafe or cannot work, with a message naming the key:
- `auth.enabled = false` with a non-loopback `server.bind`;
- `server.port = 0`, any `limits.*` = 0, or `otel.metrics_interval_secs = 0`;
- `server.tls.enabled = true` without both `cert_file` and `key_file`;
- an invalid `[peers]` section (see [consolidator.md](consolidator.md#configuration)).
`serve` additionally refuses to start when the TLS files do not load or do not match, or when an enabled peer's token file is missing or readable by others.
A non-loopback `bind` without TLS is allowed but logs a warning: bearer tokens then travel in cleartext.
## What applies without a restart
| Change | Applied by |
| --- | --- |
| New, revoked or rotated tokens | immediately; no signal needed |
| TLS certificate and key contents | `SIGHUP` (`systemctl reload skyg-linux-tracker`) |
| `[peers]` entries, intervals, timeouts, `fail_threshold` | `SIGHUP` |
| Everything else | restart |
## Secrets
Keep secrets out of the config file, which is usually world-readable (`/etc`) or in the Nix store:
- OTel headers: `SKYG_OTEL__HEADERS__<NAME>` in an `EnvironmentFile=` (NixOS: `environmentFile`).
- TLS key: a file readable only by the service user, or a systemd credential (NixOS: `tls.keyFile`).
- Peer tokens: `token_file` paths only; the file holds the token (NixOS: `peers.peers.<name>.tokenFile`).
- API tokens are never in the config; they live hashed in the database.
+68
View File
@@ -0,0 +1,68 @@
# Development
Building, testing and changing `slinuxm`. Design and internals are in [ARCHITECTURE.md](../ARCHITECTURE.md); open work is in [TODO.md](../TODO.md).
## Toolchain
```sh
devenv shell # Rust, pkg-config, lld, libsystemd, nushell, otelcol-contrib; installs the git hooks
cargo build --release # target/release/slinuxm
```
Without devenv you need a Rust toolchain, `pkg-config`, the `lld` linker (`.cargo/config.toml` links with it) and the libsystemd development files.
`examples/` holds three small programs (`systemd_dump`, `journal_tail`, `docker_ps`) that run one source against the live host: `cargo run --example journal_tail`.
## Tests
All checks go through one runner, `dev/dev-test.nu`:
```sh
devenv shell -- dev-test lint # cargo fmt --check, clippy -D warnings, no openssl in the tree
devenv shell -- dev-test unit # cargo test --lib --bins
devenv shell -- dev-test integration [filter] [--nocapture] # cargo test --test '*'
devenv shell -- dev-test all # lint, unit, integration; stops at the first failure
devenv shell -- dev-test vm [vm|peers] # NixOS VM tests (slow, needs /dev/kvm)
devenv shell -- dev-test ci # all plus the VM test
devenv shell -- smoke # the real binary end to end
```
`devenv shell -- dev-test --help` shows the cargo or nix command behind each.
- Unit and integration tests use in-memory fakes for systemd, the journal and Docker, so they need no root, no Docker and no network.
- The smoke test builds and runs the real binary against a throwaway database: creates a token, serves on a free port, exercises `/healthz`, REST and MCP with and without the token, revokes it and checks a clean SIGTERM exit. Checks that need host data are skipped with a reason when the source is unavailable.
Shell completion for `dev-test`: Nushell `use dev/dev-test.nu *`; zsh `source <(nu dev/dev-test.nu completions zsh)` after `compinit`; an interactive `devenv shell` sets up bash. More in [consolidator.md](consolidator.md#testing).
## Git hooks
Entering `devenv shell` installs hooks that run formatting, clippy, linters and a secret scan on commit, and path-selected tests on push. Details: [hooks.md](hooks.md).
## Version stamp
The binary reports `YYYY.MM.DD-<branch>-<sha>`, for example `2026.10.21-main-435a54a`, in `--version`, `/healthz`, `/api/v1/ping`, MCP server info and the OTel `service.version`.
- The date is the build day (UTC). Characters outside `[A-Za-z0-9._-]` in the branch become `-`; a detached HEAD is `detached`. The sha gets `-dirty` when tracked files have uncommitted changes. Without git, missing parts are `unknown`.
- `SLINUXM_GIT_SHA`, `SLINUXM_GIT_BRANCH` and `SLINUXM_BUILD_DATE` override the parts; `SLINUXM_VERSION` replaces the whole string. `SOURCE_DATE_EPOCH` is ignored (Nix dev shells set it to 1980).
- `nix build` uses the commit date and short rev, with the branch fixed to `nix`. Override with `pkgs.callPackage ./nix/package.nix { gitBranch = "main"; … }`.
- `Cargo.toml` keeps a semver version, used as `info.version` in the OpenAPI document so it does not change on every commit.
## OpenAPI document
[`docs/openapi.json`](openapi.json) is generated from the code. Regenerate it after changing a route, an MCP tool, a response type or the crate version:
```sh
cargo run -- openapi --peers > docs/openapi.json
```
`tests/openapi.rs` fails when it is stale.
## Rules that matter when changing code
The full list is in [AGENTS.md](../AGENTS.md); the ones that bite:
- Read-only against the host. No code path may change a unit, a container or a file outside the state directory.
- `src/sources/` is the only code that talks to D-Bus, journald or Docker. REST, MCP and OTel call into it.
- No openssl anywhere in the dependency tree (`cargo tree -i openssl` must be empty).
- Keep `config.example.toml`, [configuration.md](configuration.md) and the NixOS module in step when adding config.
- Keep `systemd/skyg-linux-tracker.service` and the module's `serviceConfig` in step.
+1 -1
View File
@@ -56,4 +56,4 @@ A later `dev-test` runner can replace the `cargo test` calls in the `chk_test-*`
## Proposed pointers
- AGENTS.md (Commands): `Git hooks run fmt, clippy, lint and a secret scan on commit and path-filtered tests on push; see docs/hooks.md.`
- README.md (Development): `Contributors: enter devenv shell to install the git hooks; details in docs/hooks.md.`
- docs/development.md (Git hooks): `Entering devenv shell installs hooks ...; details in hooks.md.`
+143
View File
@@ -0,0 +1,143 @@
# How it works
What the `slinuxm` binary is, what it does when it runs, and what it will never do. Read this first; the other guides assume it.
## One binary, one process
`slinuxm` is a single, self-contained executable. It does two kinds of work:
- **Run the server** (`slinuxm serve`, the default when no subcommand is given). A long-running process that answers HTTP requests about the host it runs on.
- **Run a one-shot command** (`slinuxm token ...`, `slinuxm peer ...`, `slinuxm openapi`, `--print-config`). These start, do one thing and exit. They never start a server.
There is no agent/daemon split, no separate database server and no web assets on disk: the REST API, the MCP server, the OpenTelemetry exporter and the `/docs` viewer are all inside the binary. Its only dynamic dependencies are libc, libm, libgcc_s and libsystemd.
## What the server reads
Everything is read **live, on each request**, from three sources on the local host:
| Source | How it is read | Needs |
| --- | --- | --- |
| systemd units and timers | D-Bus, system bus (`org.freedesktop.systemd1`) | nothing special; any user can read unit state |
| The journal | libsystemd (`sd-journal`) | membership of the `systemd-journal` group to see all logs |
| Docker containers and logs | Docker Engine API over a unix socket | access to the socket (see [Docker access](operations.md#docker-access)) |
Nothing is cached or collected in the background for REST or MCP. If Docker is not running, only the Docker routes fail (with `503`); everything else keeps working.
The only thing the server **stores** is a small SQLite database of API tokens (hashed). That database is the whole of its state.
## What it serves
All on one port (default `1984`, bound to `127.0.0.1` by default):
```text
client (curl, browser, MCP client, another slinuxm)
│ HTTP or HTTPS, HTTP/1.1
▼
┌───────────────────── slinuxm serve ─────────────────────┐
│ /healthz, /docs public, no host data │
│ │
│ bearer-token check ──► token database (SQLite) │
│ ├─ /api/v1/* REST, JSON, SSE for "follow" │
│ └─ /mcp MCP server (Streamable HTTP) │
│ │ │
│ ▼ │
│ sources: systemd · journal · Docker │
│ │
│ optional background tasks: │
│ OTel exporter ──push──► your OTLP collector │
│ peer health checks ──► other slinuxm instances │
└─────────────────────────────────────────────────────────┘
```
- **REST API** under `/api/v1`: GET only. Lists and details for units, timers, journal entries and containers. "Follow" endpoints (journal, container logs) stream as Server-Sent Events. Full table: [api.md](api.md).
- **MCP server** at `/mcp`: the same data as tools, prompts and resources for AI agents. REST and MCP call the same code, so they always return the same data.
- **OpenTelemetry** (off by default): pushes unit and container status as metrics and journal and container logs as OTel logs to a collector you point it at. This is the one part that polls on a timer.
- **`/healthz`**: public liveness check with the version.
- **`/docs`**: a built-in API viewer (Redoc). The page is public; it asks for a token before it fetches the API description.
- **Peers** (off by default): one instance can health-check other `slinuxm` instances and pass read-only requests through to them. See [consolidator.md](consolidator.md).
## Read-only by design
No code path can start, stop, restart or modify a unit, a container or a file outside the state directory. Every route is GET. The only writes the binary ever makes are to its own token database. This is why it is safe to hand a token to an AI agent.
Log text from the journal and from containers is produced by other programs. It is returned as untrusted data and labelled as such to MCP clients.
## Authentication
Every request to `/api/v1/*` and `/mcp` must carry `Authorization: Bearer <token>`.
- You create tokens with `slinuxm token create <name>`. The plaintext is printed **once**; only a sha256 hash is stored.
- On each request the server hashes the presented token and looks it up. Revoking or rotating a token takes effect on the next request; no restart.
- A missing or wrong token gets `401` with `WWW-Authenticate: Bearer`, the same for REST and MCP.
- Every token can read everything. There are no scopes.
Auth can be switched off (`auth.enabled = false`), but only while the server listens on loopback; startup refuses otherwise.
## Lifecycle of `slinuxm serve`
1. **Load configuration**: built-in defaults, then the config file, then `SKYG_*` environment variables, then CLI flags (later wins). Invalid combinations stop here with an error. See [configuration.md](configuration.md).
2. **Check peer token files** (only when peers are enabled).
3. **Start telemetry**: logging to stderr, plus OTLP exporters when `otel.enabled`.
4. **Open the token database** (when auth is on), creating it if needed: directory mode 0700, file mode 0600.
5. **Load the TLS certificate and key** (when TLS is on). A missing or mismatched pair stops startup.
6. **Bind the listener**. Connections are capped at 1024; a connection that sends no request within 30 s is closed.
7. **Start background tasks**: OTel exporters and peer health checks, if enabled.
8. **Serve** until a stop signal.
Logs go to stderr, so under systemd they land in the journal (`journalctl -u skyg-linux-tracker`).
## Signals
| Signal | Effect |
| --- | --- |
| `SIGTERM` / `SIGINT` | Graceful stop: stop accepting, end open SSE streams, drain requests (at most 10 s), stop exporters (at most 10 s), flush OTel, close the database. A systemd stop timeout of 30 s covers all of it. |
| `SIGHUP` | Reload without dropping connections: re-reads the TLS certificate and key, and the `[peers]` section. A bad new certificate pair is rejected and the old one kept. Everything else (port, bind, auth, sources, OTel, limits) needs a restart. |
## Commands
```text
slinuxm [GLOBAL OPTIONS] [COMMAND]
Commands:
serve Run the server (default when no command is given)
token create <name> Create a token and print it once
token list List tokens: name, created, revoked, last used (never the secret)
token revoke <name> Revoke a token; effective immediately
token rotate <name> Replace a token's secret, print the new one once; re-enables a revoked token
peer list List configured peers (config only)
peer check <name> Probe one peer once over TCP and the authenticated API
openapi [--peers] Print the OpenAPI description and exit (reads no config)
Global options:
--config <FILE> Config file (default /etc/skyg-linux-tracker.toml if present; env SKYG_CONFIG)
--print-config Print the effective configuration, secrets redacted, and exit
--bind <IP> server.bind
--port <PORT> server.port
--auth-enabled <BOOL> auth.enabled
--db-path <FILE> auth.db_path
--docker-socket <FILE> sources.docker_socket
--journal-namespace <NAME> sources.journal_namespace
--otel-enabled <BOOL> otel.enabled
--otel-endpoint <URL> otel.endpoint
--version Print the build version (YYYY.MM.DD-<branch>-<sha>)
```
`token` commands open the token database directly, so they must run **as the user that owns it**, normally the service user (`sudo -u <service user> slinuxm token ...`). They refuse to run as anyone else, root included, and print the right `sudo -u` command. This prevents root from creating database files the service can no longer open.
## Files it touches
| Path | Purpose | Default |
| --- | --- | --- |
| Config file | read at start (and the `[peers]` section on SIGHUP) | `/etc/skyg-linux-tracker.toml` |
| Token database | read and written | `/var/lib/skyg-linux-tracker/db.sqlite` |
| TLS certificate and key | read at start and on SIGHUP | none (TLS off) |
| Docker socket | read | `/var/run/docker.sock` |
| Peer token files | read on every peer check and request | none (peers off) |
## Where to go next
- Install it: [install.md](install.md)
- Configure it: [configuration.md](configuration.md)
- Run it day to day (tokens, network exposure, Docker, OTel, troubleshooting): [operations.md](operations.md)
- Every route and MCP tool: [api.md](api.md)
- Internals and design decisions: [ARCHITECTURE.md](../ARCHITECTURE.md)
+249
View File
@@ -0,0 +1,249 @@
# Install
Three ways to run `slinuxm`, from most to least managed:
| Path | Use when | Service user |
| --- | --- | --- |
| [NixOS module](#nixos-module) | the host runs NixOS | `skyg-linux-tracker` (created for you) |
| [Nix package, any distro](#nix-package-on-any-linux) | you have Nix but not NixOS | you choose |
| [From source, manual systemd](#from-source-with-a-manual-systemd-unit) | any systemd-based Linux | `skyg` in the example |
Whatever the path, the result is the same binary listening on `127.0.0.1:1984` with auth on. Read [How it works](how-it-works.md) first if you have not.
**Requirements on the host:** Linux with systemd (libsystemd is linked in). Docker is optional.
## Nix
The repository is a flake. It provides:
| Output | What it is |
| --- | --- |
| `packages.<system>.default` | the `slinuxm` binary (`x86_64-linux`, `aarch64-linux`) |
| `nixosModules.default` | the `services.skyg-linux-tracker` NixOS module, with the package preset |
The flake reference used below is the repository's git remote:
```text
git+ssh://gitea@gitea.app.internal/Hermenix/skyg-linux-tracker
```
From a local checkout, use `.` (or `path:/path/to/checkout`) instead.
### Nix package on any Linux
```sh
nix run git+ssh://gitea@gitea.app.internal/Hermenix/skyg-linux-tracker -- --version # try without installing
nix profile install git+ssh://gitea@gitea.app.internal/Hermenix/skyg-linux-tracker # slinuxm on your PATH
nix build # from a checkout: ./result/bin/slinuxm
```
That gives you the binary only. To run it as a service on a non-NixOS host, follow [From source](#from-source-with-a-manual-systemd-unit) from step 2, and point `ExecStart=` in the unit at the Nix-installed binary (for example `/nix/var/nix/profiles/default/bin/slinuxm`) instead of `/usr/local/bin/slinuxm`.
The Nix build stamps the version as `<commit date>-nix-<short sha>`; a flake cannot see the branch name, so the branch is always `nix`.
### NixOS module
#### 1. Add the flake input and the module
```nix
# flake.nix of your system configuration
{
inputs.skyg-linux-tracker.url = "git+ssh://gitea@gitea.app.internal/Hermenix/skyg-linux-tracker";
outputs = { nixpkgs, skyg-linux-tracker, ... }: {
nixosConfigurations.myhost = nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
modules = [
./configuration.nix
skyg-linux-tracker.nixosModules.default
];
};
};
}
```
```nix
# configuration.nix
{
services.skyg-linux-tracker.enable = true;
}
```
`nixos-rebuild switch`, and you have:
- a `skyg-linux-tracker` system user and group, in `systemd-journal`;
- a hardened `skyg-linux-tracker.service` listening on `127.0.0.1:1984`;
- the generated config at `/etc/skyg-linux-tracker.toml` (the path the CLI reads by default, so `slinuxm` commands and the service agree);
- the token database under `/var/lib/skyg-linux-tracker` (mode 0700);
- `slinuxm` on the system `PATH`.
#### 2. Create the first token
```sh
sudo -u skyg-linux-tracker slinuxm token create my-laptop
```
The token is printed once. Test it:
```sh
curl -s -H "Authorization: Bearer $TOKEN" 'http://127.0.0.1:1984/api/v1/systemd/units?state=failed'
```
#### 3. Options
All under `services.skyg-linux-tracker`:
| Option | Default | Meaning |
| --- | --- | --- |
| `enable` | `false` | Run the service. |
| `package` | this flake's package | The package that provides `slinuxm`. |
| `port` | `1984` | TCP port for REST and `/mcp`. |
| `bind` | `"127.0.0.1"` | Listen address. A non-loopback address requires auth (the default), checked at evaluation. |
| `openFirewall` | `false` | Open `port` on all interfaces. |
| `firewallInterfaces` | `[ ]` | Open `port` on these interfaces only (preferred on a LAN), e.g. `[ "enp3s0" ]`. |
| `allowedHosts` | `[ ]` | Extra `Host`/`Origin` names accepted on `/mcp`, e.g. `[ "monitor.lan:1984" ]`. The host's own name is added automatically. |
| `tls.enable` | `false` | Serve HTTPS. |
| `tls.certFile`, `tls.keyFile` | `null` | PEM chain and key, as **strings** (absolute paths outside the store). Delivered through `LoadCredential`, so the service user never reads the originals. |
| `docker.enable` | `false` | Add the service user to the `docker` group. Root-equivalent; requires `virtualisation.docker.enable`. See [Docker access](operations.md#docker-access). |
| `settings` | `{ }` | Any config file key, as Nix attributes, merged over the module defaults (see [configuration.md](configuration.md)). Never put secrets here. |
| `environmentFile` | `null` | systemd `EnvironmentFile` for secrets such as `SKYG_OTEL__HEADERS__AUTHORIZATION`. Kept out of the store. |
| `peers.enable`, `peers.peers.<name>`, `peers.settings` | off | Consolidator mode. See [consolidator.md](consolidator.md#nixos-with-agenix). |
#### 4. Common setups
Reachable from the LAN, with TLS from ACME:
```nix
services.skyg-linux-tracker = {
enable = true;
bind = "0.0.0.0";
firewallInterfaces = [ "enp3s0" ];
allowedHosts = [ "monitor.lan:1984" ];
tls = {
enable = true;
certFile = "/var/lib/acme/monitor.lan/fullchain.pem";
keyFile = "/var/lib/acme/monitor.lan/key.pem";
};
};
security.acme.certs."monitor.lan".reloadServices = [ "skyg-linux-tracker" ];
```
Exporting to an OpenTelemetry collector, with the collector's auth header kept out of the store:
```nix
services.skyg-linux-tracker = {
enable = true;
settings.otel = {
enabled = true;
endpoint = "http://otel.lan:4317";
export_logs = true;
};
environmentFile = "/run/secrets/skyg-linux-tracker.env"; # SKYG_OTEL__HEADERS__AUTHORIZATION=Bearer ...
};
```
Tighter limits, a Docker socket proxy:
```nix
services.skyg-linux-tracker = {
enable = true;
settings = {
limits.max_lines = 500;
sources.docker_socket = "/run/docker-proxy/docker.sock";
};
};
```
#### 5. What a `nixos-rebuild switch` does to a running service
The module decides between a reload and a restart for you:
- **Reload (SIGHUP)**, no dropped connections: a change to a peer's `url` or `caFile`, or to the peer intervals and timeouts. A renewed certificate at the same path is picked up by `systemctl reload skyg-linux-tracker` (or ACME's `reloadServices`).
- **Restart**: everything else, including `port`, `bind`, auth, `tls.*` paths, `settings.otel`, `settings.limits`, sources, `peers.enable`, and adding, removing or re-pointing the `tokenFile` of a peer.
### Nix checks
`nix flake check` builds the package (running the test suite), evaluates the module against a set of cases, and boots NixOS VM tests: a two-machine test (plain and TLS) and a three-machine consolidator test. The VM tests need `/dev/kvm`.
## From source, with a manual systemd unit
For any systemd-based distribution without Nix. The unit file in this repository, [`systemd/skyg-linux-tracker.service`](../systemd/skyg-linux-tracker.service), is an **example** of a hardened unit: copy it and adapt it to your host. The NixOS module generates its own equivalent; you do not need this file on NixOS.
### 1. Build
You need a Rust toolchain, `pkg-config`, the `lld` linker (the repository's `.cargo/config.toml` links with it) and the libsystemd development files (`libsystemd-dev` on Debian/Ubuntu, `systemd-devel` on Fedora). With [devenv](https://devenv.sh), `devenv shell` provides all of them.
```sh
cargo build --release # produces target/release/slinuxm
```
### 2. Install the binary, user and config
```sh
sudo install -m 0755 target/release/slinuxm /usr/local/bin/slinuxm
sudo useradd --system --no-create-home --shell /usr/sbin/nologin skyg
sudo install -m 0644 config.example.toml /etc/skyg-linux-tracker.toml
```
`config.example.toml` lists every key with its default, so the copied file already describes a working setup. Edit it as needed ([configuration.md](configuration.md)).
### 3. Install the example unit
```sh
sudo install -m 0644 systemd/skyg-linux-tracker.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now skyg-linux-tracker
```
What the example sets up, and what you may need to change:
| In the unit | Why | Adapt if |
| --- | --- | --- |
| `ExecStart=/usr/local/bin/slinuxm serve --config /etc/skyg-linux-tracker.toml` | runs the server with the installed config | the binary or config lives elsewhere |
| `ExecReload=` sends SIGHUP | `systemctl reload` rotates TLS certificates and reloads peers | |
| `User=skyg`, `Group=skyg` | a static user, so `sudo -u skyg slinuxm token ...` opens the same database as the service | you chose another user name |
| `SupplementaryGroups=systemd-journal` | read the whole journal | |
| `#SupplementaryGroups=docker` (commented out) | Docker access is root-equivalent | you accept that risk; prefer a socket proxy |
| `StateDirectory=skyg-linux-tracker`, mode 0700 | systemd creates `/var/lib/skyg-linux-tracker` owned by the service user | you move `auth.db_path`: then also add `ReadWritePaths=` for the new directory |
| `ProtectSystem=strict` and the rest of the hardening block | the process can write nothing but its state directory | rarely; `systemd-analyze security skyg-linux-tracker` scores it about 1.5 |
| `TimeoutStopSec=30` | room for the graceful drain and OTel flush | |
Override settings without editing the file using `systemctl edit skyg-linux-tracker` (a drop-in), for example to add `Environment=SKYG_SERVER__PORT=2000` or an `EnvironmentFile=` for secrets.
### 4. Create the first token
The service creates the state directory on first start, so start it first (step 3), then:
```sh
sudo -u skyg slinuxm token create my-laptop
```
## Without a service
Nothing requires systemd to manage the process. For a quick look on your own machine, with the token database in a temp directory and no root:
```sh
export SKYG_AUTH__DB_PATH=/tmp/slinuxm-demo/db.sqlite
TOKEN=$(slinuxm token create demo)
slinuxm serve # Ctrl-C to stop
```
As a normal user you see your own journal entries only, unless you are in `systemd-journal`.
## Verify an install
```sh
curl -s http://127.0.0.1:1984/healthz # {"status":"ok","version":"..."}
curl -s -H "Authorization: Bearer $TOKEN" http://127.0.0.1:1984/api/v1/ping
slinuxm --print-config # effective config, secrets redacted
journalctl -u skyg-linux-tracker -n 50 # the server's own log
ldd "$(command -v slinuxm)" # libc, libm, libgcc_s, libsystemd only
```
Then connect an MCP client: [operations.md](operations.md#connect-an-mcp-client).
## Upgrade and uninstall
- **NixOS:** update the flake input (`nix flake update skyg-linux-tracker`) and switch. The token database is kept.
- **Manual:** replace `/usr/local/bin/slinuxm` and `systemctl restart skyg-linux-tracker`. The database migrates itself on start.
- **Uninstall (manual):** `systemctl disable --now skyg-linux-tracker`, then remove the unit, the binary, `/etc/skyg-linux-tracker.toml`, `/var/lib/skyg-linux-tracker` and the `skyg` user.
+139
View File
@@ -0,0 +1,139 @@
# Operations
Day-to-day tasks for a running install. The service user below is `skyg-linux-tracker` (NixOS) or `skyg` (the manual example unit); substitute yours.
## Tokens
```sh
sudo -u skyg-linux-tracker slinuxm token create my-laptop # prints the token once
sudo -u skyg-linux-tracker slinuxm token list # name, created, revoked, last used
sudo -u skyg-linux-tracker slinuxm token rotate my-laptop # new secret, printed once; the old one stops working
sudo -u skyg-linux-tracker slinuxm token revoke my-laptop
```
- Tokens look like `slinuxm_...`. Only a sha256 hash is stored, so a lost token cannot be recovered: rotate it.
- Every change takes effect on the next request. No restart, no reload.
- `rotate` also re-enables a revoked token under the same name.
- Use one token per client (laptop, agent, consolidator), so you can revoke one without touching the others.
- Run the commands **as the service user**. Run as anyone else, root included, they refuse and print the right `sudo -u` command. The first `token create` needs the state directory, which the service creates on its first start.
- If the config is not at `/etc/skyg-linux-tracker.toml`, add `--config <file>`.
## Use the API
```sh
TOKEN=slinuxm_...
BASE=http://127.0.0.1:1984
curl -s $BASE/healthz # public
curl -s -H "Authorization: Bearer $TOKEN" "$BASE/api/v1/systemd/units?state=failed"
curl -s -H "Authorization: Bearer $TOKEN" "$BASE/api/v1/systemd/timers"
curl -s -H "Authorization: Bearer $TOKEN" "$BASE/api/v1/journal?unit=nginx.service&priority=err&lines=50"
curl -N -H "Authorization: Bearer $TOKEN" "$BASE/api/v1/journal/follow?priority=warning" # live stream (SSE)
curl -s -H "Authorization: Bearer $TOKEN" "$BASE/api/v1/docker/ps"
```
Every route, parameter and error: [api.md](api.md). The same reference is browsable at `$BASE/docs` (paste a token when asked), and `slinuxm openapi` prints it as OpenAPI 3.1.
## Connect an MCP client
Endpoint `http://127.0.0.1:1984/mcp`, Streamable HTTP, with the bearer header.
Claude Code:
```sh
claude mcp add --transport http linux-tracker http://127.0.0.1:1984/mcp \
--header "Authorization: Bearer $TOKEN"
```
Any client that takes a JSON config with headers:
```json
{
"mcpServers": {
"linux-tracker": {
"url": "http://127.0.0.1:1984/mcp",
"headers": { "Authorization": "Bearer slinuxm_..." }
}
}
}
```
The server offers eight tools (`list_systemd_units`, `get_systemd_unit`, `list_systemd_timers`, `get_journal_logs`, `tail_journal`, `list_containers`, `get_container`, `get_container_logs`), two prompts, two resources and two resource templates. Details and a raw curl handshake: [api.md](api.md#mcp).
## Expose it beyond localhost
The default is loopback only. The safest way to reach it from elsewhere is to leave it there and use an SSH tunnel, Tailscale or a TLS reverse proxy. To listen on the network directly:
1. Set `server.bind` (`"0.0.0.0"` or `"::"`). Auth must stay on.
2. Turn on TLS ([tls.md](tls.md)), unless the network is trusted: bearer tokens are readable in plain HTTP.
3. Add the names clients use to `server.allowed_hosts` (e.g. `"monitor.lan:1984"`), or `/mcp` rejects them.
4. Open the port in the firewall, ideally on the LAN interface only.
On NixOS that is `bind`, `tls.*`, `allowedHosts` and `firewallInterfaces`; see [install.md](install.md#4-common-setups).
## Docker access
Reading the Docker socket needs either:
- **A read-only socket proxy (recommended).** For example [tecnativa/docker-socket-proxy](https://github.com/Tecnativa/docker-socket-proxy) with `CONTAINERS=1 EVENTS=1 POST=0`, exposed as a unix socket. Then:
```toml
[sources]
docker_socket = "/run/docker-proxy/docker.sock"
```
- **The `docker` group.** Simple, but membership is root-equivalent: anyone who controls the process could start a privileged container. Manual unit: uncomment `SupplementaryGroups=docker`. NixOS: `docker.enable = true`.
Out of the box neither is set up, so the Docker routes and tools return `503` while the rest works.
## OpenTelemetry
```toml
[otel]
enabled = true
endpoint = "http://collector:4317" # or :4318 with protocol = "http/protobuf"
export_logs = true # also forward logs, not only metrics
```
**Metrics** (every `metrics_interval_secs`, gauges):
| Metric | Meaning |
| --- | --- |
| `systemd.unit.active`, `systemd.unit.failed` | 0/1 per unit |
| `systemd.unit.restarts` | restart count (services) |
| `systemd.timer.next_elapse_timestamp_seconds` | next run of a timer, unix time |
| `docker.container.running`, `docker.container.healthy` | 0/1 per container |
| `docker.container.uptime_seconds`, `docker.container.restarts` | per container |
| `peer.*` | peer health, with peers enabled ([consolidator.md](consolidator.md#opentelemetry)) |
By default only `*.service` and `*.timer` units are exported; set `[otel.metrics] units` to a glob list to change that.
**Logs** (`export_logs = true`): journal entries and container stdout/stderr, with severity mapped from the journal priority (stderr is WARN, stdout INFO). Narrow them with `[otel.logs]`.
**Traces**: one span per HTTP and MCP request.
For a local test collector that prints everything it receives: `devenv up` in a checkout.
## Reload, restart, stop
```sh
systemctl reload skyg-linux-tracker # SIGHUP: re-read the TLS pair and [peers]; connections stay up
systemctl restart skyg-linux-tracker # anything else in the config
systemctl stop skyg-linux-tracker # graceful: open streams end, requests drain, OTel flushes
```
## Troubleshooting
The server logs to stderr; under systemd: `journalctl -u skyg-linux-tracker`.
| Symptom | Likely cause |
| --- | --- |
| `401` on every request | Missing `Authorization: Bearer` header, or the token was revoked or rotated. `/healthz` works without a token. |
| `503` on `/api/v1/docker/*` only | No access to the Docker socket, or Docker is not running. See [Docker access](#docker-access). |
| `403` on REST with auth off | With `auth.enabled = false`, REST accepts only loopback `Host` names plus `allowed_hosts`. |
| MCP client fails to connect from another host | Its `Host` (or browser `Origin`) is not in `server.allowed_hosts` / `allowed_origins`. |
| `token create` refuses with a `sudo -u` hint | You ran it as the wrong user, root included. Use the command it prints. |
| `token create` cannot find the database | The service has never started (no state directory yet), or `--config` points at a different file. |
| Journal results are short or empty | The service user is not in `systemd-journal`, or `sources.journal_namespace` names the wrong namespace. |
| Server refuses to start | Read the first error line: it names the config key, the TLS file or the peer token file. `slinuxm --print-config` shows the effective config. |
| Port changed in config, nothing happened | Port, bind and auth need a restart, not a reload. |
+1 -1
View File
@@ -1,6 +1,6 @@
# TLS
Reference for native TLS. Summary in the [README](../README.md#tls); design in [ARCHITECTURE.md](../ARCHITECTURE.md#tls).
Reference for native TLS. Settings overview in [configuration.md](configuration.md#servertls-https); design in [ARCHITECTURE.md](../ARCHITECTURE.md#tls).
TLS is optional and off by default. With `server.tls.enabled = false` (the default) the server speaks plain HTTP exactly as before, the certificate paths are ignored and never read, and no TLS code runs. A non-loopback `bind` without TLS logs a startup warning (it is not refused).
+8 -1
View File
@@ -1,3 +1,8 @@
# Example unit for a manual (non-NixOS) install of slinuxm. Copy it to
# /etc/systemd/system/ and adapt the binary path, config path and user to your
# host; docs/install.md walks through it. On NixOS the module generates its own
# equivalent unit, so this file is not used there.
[Unit]
Description=skyg-linux-tracker: read-only systemd, journal and Docker monitor (REST, MCP, OTel)
After=network.target docker.service
@@ -5,6 +10,8 @@ After=network.target docker.service
[Service]
Type=simple
ExecStart=/usr/local/bin/slinuxm serve --config /etc/skyg-linux-tracker.toml
# SIGHUP re-reads the TLS certificate and the [peers] section without a restart.
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
@@ -19,7 +26,7 @@ Group=skyg
SupplementaryGroups=systemd-journal
# WARNING: membership of the docker group is root-equivalent on the host (the
# Docker API can start privileged containers). Prefer pointing
# sources.docker_socket at a read-only socket proxy (see README) and leave this
# sources.docker_socket at a read-only socket proxy (see docs/operations.md) and leave this
# commented out. Uncomment only if you accept that risk (see ARCHITECTURE.md, Decisions).
#SupplementaryGroups=docker