Files
skyg-linux-tracker/README.md
T
ari 04992e202d F: correct the peers docs against the finished behaviour
docs/consolidator.md: the Testing section described the pre-feature test
(no tests/peers.rs, the old tests.<system>.peers attribute, a note about
assertions failing by design) and is rewritten for both layers, the
module's eval check included. A shutdown subsection backs the README's
promise, the agenix examples say why a consolidator would bind 0.0.0.0
(remote callers; peers need no inbound port), and the VM test asserts
the chained-request status is exactly 400 as documented. README: token
rotation on NixOS also needs a reload for the LoadCredential copy.
2026-10-20 15:04:09 -04:00

255 lines
16 KiB
Markdown

# skyg-linux-tracker
A small, read-only monitor for a single Linux host. One binary, `slinuxm`, serves:
- a REST API (with Server-Sent Events for follow endpoints),
- an MCP server (Streamable HTTP) so agents can inspect the host,
- optional OpenTelemetry export (metrics and logs over OTLP).
Data sources are systemd (D-Bus), the journal (libsystemd) and the Docker Engine API (unix socket). Nothing here can start, stop or modify anything.
## Quick start
Try it on your own machine in a minute. It needs no root and no install, and it keeps its token database in a temp directory.
```sh
devenv shell && cargo build --release # or just: cargo build --release with the prerequisites below
export SKYG_AUTH__DB_PATH=/tmp/slinuxm-demo/db.sqlite
BIN=target/release/slinuxm
TOKEN=$($BIN token create demo) # printed once, never stored in plaintext
$BIN serve & # http://127.0.0.1:1984, loopback only
curl -s http://127.0.0.1:1984/healthz # public
curl -s -H "Authorization: Bearer $TOKEN" 'http://127.0.0.1:1984/api/v1/systemd/units?state=failed'
curl -s -H "Authorization: Bearer $TOKEN" 'http://127.0.0.1:1984/api/v1/journal?priority=err&lines=20'
curl -s -H "Authorization: Bearer $TOKEN" http://127.0.0.1:1984/api/v1/docker/ps # 503 if Docker is not running
```
Point an MCP client at `http://127.0.0.1:1984/mcp` with the same bearer header (see [MCP](#mcp)). Stop the server with `kill %1`.
The service listens on loopback by default. To reach it from another machine set `server.bind`, keep auth on, and use [TLS](#tls). For a permanent install as a systemd service, see [Install](#install) and [Tokens](#tokens).
## Build
```sh
devenv shell # toolchain, pkg-config, libsystemd, otelcol-contrib
cargo build --release # target/release/slinuxm
```
Without devenv you need a Rust toolchain, `pkg-config` and the libsystemd development files.
## Install
```sh
sudo install -m 0755 target/release/slinuxm /usr/local/bin/slinuxm
sudo useradd --system --no-create-home --shell /usr/sbin/nologin skyg
sudo install -m 0644 config.example.toml /etc/skyg-linux-tracker.toml # then edit
sudo install -m 0644 systemd/skyg-linux-tracker.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now skyg-linux-tracker
```
The unit runs as the `skyg` user with `SupplementaryGroups=systemd-journal` and a private `StateDirectory` (`/var/lib/skyg-linux-tracker`, mode 0700) holding the token database. Its sandbox options are listed in the unit file.
### Config
See `config.example.toml`; every value shown is the default. Precedence, lowest to highest: defaults, config file, `SKYG_*` environment variables (nesting with `__`, for example `SKYG_SERVER__PORT`), CLI flags. `slinuxm --print-config` prints the effective configuration with secrets redacted.
Startup refuses `auth.enabled = false` on a non-loopback `server.bind`.
## Tokens
All `/api/v1/*` and `/mcp` requests need `Authorization: Bearer <token>`. `/healthz` is public. Tokens are stored hashed (sha256) in SQLite, and the plaintext is shown once.
Run the token CLI **as the service user** (the unit's `User=skyg`), i.e. `sudo -u <service user> slinuxm token create <name>`. The CLI refuses to run (with the matching `sudo -u <owner> ...` hint) when the effective uid, root included, does not own the database file or, if it does not exist yet, its existing state directory, so it cannot create or modify state owned by another user:
```sh
sudo -u skyg slinuxm token create my-laptop --config /etc/skyg-linux-tracker.toml
sudo -u skyg slinuxm token list --config /etc/skyg-linux-tracker.toml
sudo -u skyg slinuxm token revoke my-laptop --config /etc/skyg-linux-tracker.toml
```
Revocation takes effect immediately; no restart is needed.
Notes:
- `auth.db_path` must stay under the unit's `StateDirectory` (`/var/lib/skyg-linux-tracker`). The unit uses `ProtectSystem=strict`, so any other location needs a matching `ReadWritePaths=`.
- `token create` needs the state directory to exist, so start the service once first (systemd creates it with the right owner), then run the CLI as `skyg`.
- A state directory the server creates itself is made 0700 and the database file 0600 from creation. An existing directory is never chmod'd; a group/world-accessible one only logs a warning.
## REST
Base: `http://127.0.0.1:1984`, all routes are GET and read-only. `GET /healthz` is public; everything under `/api/v1` needs the bearer token. Routes: `systemd/units`, `systemd/units/{name}`, `systemd/timers`, `journal`, `journal/follow` (SSE), `docker/ps`, `docker/containers/{name}`, `docker/containers/{name}/logs` (JSON or SSE). Docker routes return `503` when Docker is unavailable. Errors are `{"error":{"code","message"}}`.
```sh
curl -H "Authorization: Bearer $TOKEN" 'http://127.0.0.1:1984/api/v1/systemd/units?state=failed'
curl -N -H "Authorization: Bearer $TOKEN" 'http://127.0.0.1:1984/api/v1/journal/follow?priority=err'
```
Full route table with parameters: [docs/api.md](docs/api.md).
## MCP
Endpoint: `http://127.0.0.1:1984/mcp` (Streamable HTTP, same bearer token). Eight tools (`list_systemd_units`, `get_systemd_unit`, `list_systemd_timers`, `get_journal_logs`, `tail_journal`, `list_containers`, `get_container`, `get_container_logs`), two prompts, two resources and two resource templates. A [consolidator](#peers-consolidator) adds `list_peers` and `get_peer` and a `peer` argument on every tool, for ten tools. Log content is untrusted data and is labelled as such.
Claude Code: `claude mcp add --transport http linux-tracker http://127.0.0.1:1984/mcp --header "Authorization: Bearer $TOKEN"`.
`/mcp` validates `Origin` and `Host` against DNS rebinding. If you bind to `0.0.0.0` or serve behind a name, add the names clients use to `server.allowed_hosts` (for example `["monitor.lan:1984"]`) and browser origins to `server.allowed_origins`. Generic client config, Inspector and curl handshake, and the full Origin/Host rules: [docs/api.md](docs/api.md#mcp).
## TLS
TLS is optional and off by default; a non-loopback `bind` without TLS logs a startup warning. Enable it with a PEM chain and key:
```toml
[server.tls]
enabled = true
cert_file = "/etc/skyg/tls/cert.pem" # PEM chain, leaf first
key_file = "/etc/skyg/tls/key.pem" # PEM, PKCS#8, PKCS#1 or SEC1
```
REST and `/mcp` are then served over HTTPS only. Replace the files and send `SIGHUP` to rotate without dropping connections (a bad pair keeps the old certificate). rustls only; no openssl. Self-signed example, rotation caveats, ACME and sops paths: [docs/tls.md](docs/tls.md).
## Peers (consolidator)
One `slinuxm` can watch other instances: declare them as **peers** and it health-checks each one and passes read-only requests through to it, so one MCP endpoint or one REST base URL covers the fleet. Off by default; a peer-less instance keeps exactly the same routes, eight MCP tools, metrics and background tasks as before.
```toml
[peers]
enabled = true # other intervals, timeouts and caps: config.example.toml
[[peers.peer]]
name = "minipc1" # ^[a-z0-9][a-z0-9_-]{0,31}$, at most 64 peers
url = "https://minipc1:1984" # http:// to a non-loopback host also needs allow_plain_http = true
token_file = "/run/credentials/skyg-linux-tracker.service/peer-minipc1"
ca_file = "/etc/skyg/private-ca.pem" # optional extra trust anchor
[[peers.peer]]
name = "nas"
url = "http://nas:1984"
token_file = "/etc/skyg/peer-nas.token" # 0600, owned by the service user
allow_plain_http = true
```
`token_file` holds a path, never the token. A consolidator has to replay the peer's plaintext bearer token, so the token lives in a file managed outside `slinuxm` (agenix on NixOS), is read into memory on every check and request, and never appears in the config, `--print-config`, the database or the logs. Mint **one token per consolidator on every peer** and use it only there (`sudo -u <service user> slinuxm token create consolidator-<host>`), so a single `token revoke` on a peer cuts that consolidator off and nothing else. Replacing the file is enough to rotate: the next check reads the new value, and no restart is needed. On NixOS also run `systemctl reload skyg-linux-tracker` so systemd re-copies the credential the unit holds (`RefreshOnReload=credentials`).
Two checks per peer, kept in memory and reset on restart:
- **server**: a TCP connect to the peer's port.
- **API**: an authenticated `GET /api/v1/ping`. A `401` means the API runs and our token does not: `api.up = true` with `api.auth = "rejected"`. That is reported to you as a `503` naming the peer, never as a `401` that would suggest re-authenticating with this host. A peer without the ping route falls back to public `/healthz` and reports `api.auth = "unknown"`.
```sh
curl -H "Authorization: Bearer $TOKEN" http://127.0.0.1:1984/api/v1/peers
curl -H "Authorization: Bearer $TOKEN" 'http://127.0.0.1:1984/api/v1/peers/minipc1/systemd/units?state=failed'
sudo -u skyg slinuxm peer list --config /etc/skyg-linux-tracker.toml
sudo -u skyg slinuxm peer check minipc1 --config /etc/skyg-linux-tracker.toml
```
`GET /api/v1/peers` lists every peer, `GET /api/v1/peers/{peer}` one of them. `GET /api/v1/peers/{peer}/...` is a GET-only, allowlisted pass-through (`systemd/...`, `journal`, `journal/follow`, `docker/...`): the peer's answer comes back unchanged with `x-slinuxm-peer: <peer>`, a peer that is down, rejects our token or times out is a `503` naming it, `x-slinuxm-hop` from a caller is a `400`, and unknown peers are a `404`. With peers enabled MCP grows `list_peers` and `get_peer`, and every other tool accepts an optional `peer` argument (ten tools instead of eight). There is no aggregation across peers in this version: ask one peer at a time. OTel gains `peer.up`, `peer.api.auth_ok`, `peer.check.duration_seconds` and `peer.last_change_timestamp_seconds`.
On NixOS the module declares peers and delivers each token through systemd `LoadCredential`, so the service user never needs to read the agenix output:
```nix
services.skyg-linux-tracker.peers = {
enable = true;
peers.minipc1 = {
url = "https://minipc1:1984";
tokenFile = config.age.secrets.peer-minipc1.path; # a string path, never a path literal
};
};
```
Worked agenix setup, the threat model, the firewall note (every peer must accept the consolidator's address on its port), reload and shutdown semantics: [docs/consolidator.md](docs/consolidator.md). The peers tests are `tests/peers.rs` (in-process, no VM) and the three-machine NixOS VM test: `devenv shell -- dev-test integration peers`, and for the VM test `devenv shell -- dev-test vm peers`, which is `nix build .#checks.x86_64-linux.vm-peers -L`.
## NixOS
The flake exports `packages.<system>.default` and `nixosModules.default`.
```nix
# flake.nix
inputs.skyg-linux-tracker.url = "github:<owner>/linux-system-monitor";
# in your nixosSystem modules
modules = [
skyg-linux-tracker.nixosModules.default
{ services.skyg-linux-tracker.enable = true; }
];
```
That runs the service as the `skyg-linux-tracker` user on `127.0.0.1:1984` with the same sandbox as `systemd/skyg-linux-tracker.service`, writes `/etc/skyg-linux-tracker.toml` (the path the CLI reads by default) and installs `slinuxm` system-wide. Mint the first token as the service user:
```sh
sudo -u skyg-linux-tracker slinuxm token create my-laptop
```
Options (under `services.skyg-linux-tracker`): `port` (1984), `bind` (`"127.0.0.1"`), `tls.{enable,certFile,keyFile}` (passed through `LoadCredential`, so the service user never reads the original key; `systemctl reload skyg-linux-tracker` re-reads them; for ACME add `security.acme.certs.<name>.reloadServices = [ "skyg-linux-tracker" ]`), `openFirewall` (all interfaces), `firewallInterfaces` (recommended for a LAN), `allowedHosts`, `docker.enable` (adds the root-equivalent `docker` group), `settings` (freeform TOML merged over the module defaults) and `environmentFile` (secrets such as `SKYG_OTEL__HEADERS__AUTHORIZATION`, kept out of the store and out of the generated TOML).
Exposure: keep the default loopback bind and reach it over Tailscale or a TLS reverse proxy. Bearer tokens over plain HTTP are readable on a LAN. A non-loopback `bind` with `auth.enabled = false` fails at evaluation time. For `bind = "0.0.0.0"` add the names clients use to `allowedHosts` (`networking.hostName` is added for you), and prefer `firewallInterfaces = [ "enp3s0" ]` over `openFirewall`.
```nix
services.skyg-linux-tracker = {
enable = true;
bind = "0.0.0.0";
firewallInterfaces = [ "enp3s0" ];
allowedHosts = [ "monitor.lan:1984" ];
tls = { enable = true; certFile = "/var/lib/acme/monitor.lan/fullchain.pem"; keyFile = "/var/lib/acme/monitor.lan/key.pem"; };
};
```
`nix flake check` runs a two-machine VM test (plain and TLS) covering healthz, 401, token mint and revoke through the CLI, `/mcp` and the Origin check.
## Docker access
Reading the Docker socket directly requires group `docker`, which is root-equivalent. The unit file has this commented out. Prefer a read-only socket proxy that allows only `GET /containers/*`, `/events` and `/version`, for example [tecnativa/docker-socket-proxy](https://github.com/Tecnativa/docker-socket-proxy) with `CONTAINERS=1 EVENTS=1 POST=0`, exposed as a unix socket (or loopback TCP through a socat bridge), then:
```toml
[sources]
docker_socket = "/run/docker-proxy/docker.sock"
```
A stock install does not add the `docker` group, so out of the box the Docker routes return `503` until you either grant access or configure a proxy. If the socket is missing or unreadable, Docker routes return `503` and everything else keeps working.
## OpenTelemetry
```toml
[otel]
enabled = true
endpoint = "http://localhost:4317" # gRPC; or http://localhost:4318 with protocol = "http/protobuf"
export_logs = true
```
Metrics are polled every `metrics_interval_secs` (systemd unit active/failed/restarts and timer next elapse; container running/healthy/uptime/restarts). Logs fan out from the journal and from Docker containers. For a local collector with a debug exporter: `devenv up` (runs `otelcol-contrib --config dev/otelcol.yaml`). On SIGTERM pending records are flushed before exit.
## Verifying a release binary
```sh
ldd /usr/local/bin/slinuxm # libc, libm, libgcc_s, libsystemd and its dependencies only
```
## Testing
Checks run through one runner, `dev/dev-test.nu`. `devenv shell -- dev-test …` runs it from any shell, including an interactive one. `devenv shell -- dev-test --help` lists every subcommand and the cargo or nix command behind it.
```sh
devenv shell -- dev-test unit # cargo test --lib --bins
devenv shell -- dev-test integration [filter] [--nocapture] # cargo test --test '*' (integration targets only)
devenv shell -- dev-test lint # cargo fmt --check, clippy -D warnings, no openssl
devenv shell -- dev-test vm [vm|peers] # NixOS VM test (slow, needs /dev/kvm)
devenv shell -- dev-test all # lint, unit, integration; stops at the first failure
devenv shell -- dev-test ci # all plus the vm test
devenv shell -- smoke # same as: cargo test --test binary_smoke -- --nocapture
```
The smoke test runs the real built `slinuxm` binary against a throwaway SQLite database in a temp directory: it creates a token with the CLI, starts `serve` on a free loopback port, exercises `/healthz`, REST and MCP with and without the token, revokes the token, and checks a clean SIGTERM exit and a 0600 database. It needs no root and no network. Checks that depend on host data (systemd over D-Bus, the journal) are skipped with a printed reason when the source is unavailable; Docker is never required.
## Development
```sh
devenv shell -- dev-test all # fmt, clippy, unit and integration tests
```
`devenv shell -- dev-test --help` shows the underlying commands (`cargo fmt --check`, `cargo clippy --all-targets -- -D warnings`, `cargo test --lib --bins`, `cargo test --test '*'`). Nushell: `use dev/dev-test.nu *` gives `dev-test unit`, `dev-test vm peers`, ... with tab completion. zsh users add `source <(nu dev/dev-test.nu completions zsh)` to `.zshrc` after `compinit`. An interactive `devenv shell` installs bash completion. The integration tests in `tests/` start the real router on an ephemeral port with fake sources. Design notes live in [ARCHITECTURE.md](ARCHITECTURE.md); work tracking in [TODO.md](TODO.md); extended docs in [docs/](docs/README.md).
Contributors: enter `devenv shell` to install the git hooks; details in [docs/hooks.md](docs/hooks.md).