# Operations Day-to-day tasks for a running install. The service user below is `skyg-linux-tracker` (NixOS) or `skyg` (the manual example unit); substitute yours. ## Tokens ```sh sudo -u skyg-linux-tracker slinuxm token create my-laptop # prints the token once sudo -u skyg-linux-tracker slinuxm token list # name, created, revoked, last used sudo -u skyg-linux-tracker slinuxm token rotate my-laptop # new secret, printed once; the old one stops working sudo -u skyg-linux-tracker slinuxm token revoke my-laptop ``` - Tokens look like `slinuxm_...`. Only a sha256 hash is stored, so a lost token cannot be recovered: rotate it. - Every change takes effect on the next request. No restart, no reload. - `rotate` also re-enables a revoked token under the same name. - Use one token per client (laptop, agent, consolidator), so you can revoke one without touching the others. - Run the commands **as the service user**. Run as anyone else, root included, they refuse and print the right `sudo -u` command. The first `token create` needs the state directory, which the service creates on its first start. - If the config is not at `/etc/skyg-linux-tracker.toml`, add `--config `. ## Use the API ```sh TOKEN=slinuxm_... BASE=http://127.0.0.1:1984 curl -s $BASE/healthz # public curl -s -H "Authorization: Bearer $TOKEN" "$BASE/api/v1/systemd/units?state=failed" curl -s -H "Authorization: Bearer $TOKEN" "$BASE/api/v1/systemd/timers" curl -s -H "Authorization: Bearer $TOKEN" "$BASE/api/v1/journal?unit=nginx.service&priority=err&lines=50" curl -N -H "Authorization: Bearer $TOKEN" "$BASE/api/v1/journal/follow?priority=warning" # live stream (SSE) curl -s -H "Authorization: Bearer $TOKEN" "$BASE/api/v1/docker/ps" ``` Every route, parameter and error: [api.md](api.md). The same reference is browsable at `$BASE/docs` (paste a token when asked), and `slinuxm openapi` prints it as OpenAPI 3.1. ## Connect an MCP client Endpoint `http://127.0.0.1:1984/mcp`, Streamable HTTP, with the bearer header. Claude Code: ```sh claude mcp add --transport http linux-tracker http://127.0.0.1:1984/mcp \ --header "Authorization: Bearer $TOKEN" ``` Any client that takes a JSON config with headers: ```json { "mcpServers": { "linux-tracker": { "url": "http://127.0.0.1:1984/mcp", "headers": { "Authorization": "Bearer slinuxm_..." } } } } ``` The server offers eight tools (`list_systemd_units`, `get_systemd_unit`, `list_systemd_timers`, `get_journal_logs`, `tail_journal`, `list_containers`, `get_container`, `get_container_logs`), two prompts, two resources and two resource templates. Details and a raw curl handshake: [api.md](api.md#mcp). ## Expose it beyond localhost The default is loopback only. The safest way to reach it from elsewhere is to leave it there and use an SSH tunnel, Tailscale or a TLS reverse proxy. To listen on the network directly: 1. Set `server.bind` (`"0.0.0.0"` or `"::"`). Auth must stay on. 2. Turn on TLS ([tls.md](tls.md)), unless the network is trusted: bearer tokens are readable in plain HTTP. 3. Add the names clients use to `server.allowed_hosts` (e.g. `"monitor.lan:1984"`), or `/mcp` rejects them. 4. Open the port in the firewall, ideally on the LAN interface only. On NixOS that is `bind`, `tls.*`, `allowedHosts` and `firewallInterfaces`; see [install.md](install.md#4-common-setups). ## Docker access Reading the Docker socket needs either: - **A read-only socket proxy (recommended).** For example [tecnativa/docker-socket-proxy](https://github.com/Tecnativa/docker-socket-proxy) with `CONTAINERS=1 EVENTS=1 POST=0`, exposed as a unix socket. Then: ```toml [sources] docker_socket = "/run/docker-proxy/docker.sock" ``` - **The `docker` group.** Simple, but membership is root-equivalent: anyone who controls the process could start a privileged container. Manual unit: uncomment `SupplementaryGroups=docker`. NixOS: `docker.enable = true`. Out of the box neither is set up, so the Docker routes and tools return `503` while the rest works. ## OpenTelemetry ```toml [otel] enabled = true endpoint = "http://collector:4317" # or :4318 with protocol = "http/protobuf" export_logs = true # also forward logs, not only metrics ``` **Metrics** (every `metrics_interval_secs`, gauges): | Metric | Meaning | | --- | --- | | `systemd.unit.active`, `systemd.unit.failed` | 0/1 per unit | | `systemd.unit.restarts` | restart count (services) | | `systemd.timer.next_elapse_timestamp_seconds` | next run of a timer, unix time | | `docker.container.running`, `docker.container.healthy` | 0/1 per container | | `docker.container.uptime_seconds`, `docker.container.restarts` | per container | | `peer.*` | peer health, with peers enabled ([consolidator.md](consolidator.md#opentelemetry)) | By default only `*.service` and `*.timer` units are exported; set `[otel.metrics] units` to a glob list to change that. **Logs** (`export_logs = true`): journal entries and container stdout/stderr, with severity mapped from the journal priority (stderr is WARN, stdout INFO). Narrow them with `[otel.logs]`. **Traces**: one span per HTTP and MCP request. For a local test collector that prints everything it receives: `devenv up` in a checkout. ## Reload, restart, stop ```sh systemctl reload skyg-linux-tracker # SIGHUP: re-read the TLS pair and [peers]; connections stay up systemctl restart skyg-linux-tracker # anything else in the config systemctl stop skyg-linux-tracker # graceful: open streams end, requests drain, OTel flushes ``` ## Troubleshooting The server logs to stderr; under systemd: `journalctl -u skyg-linux-tracker`. | Symptom | Likely cause | | --- | --- | | `401` on every request | Missing `Authorization: Bearer` header, or the token was revoked or rotated. `/healthz` works without a token. | | `503` on `/api/v1/docker/*` only | No access to the Docker socket, or Docker is not running. See [Docker access](#docker-access). | | `403` on REST with auth off | With `auth.enabled = false`, REST accepts only loopback `Host` names plus `allowed_hosts`. | | MCP client fails to connect from another host | Its `Host` (or browser `Origin`) is not in `server.allowed_hosts` / `allowed_origins`. | | `token create` refuses with a `sudo -u` hint | You ran it as the wrong user, root included. Use the command it prints. | | `token create` cannot find the database | The service has never started (no state directory yet), or `--config` points at a different file. | | Journal results are short or empty | The service user is not in `systemd-journal`, or `sources.journal_namespace` names the wrong namespace. | | Server refuses to start | Read the first error line: it names the config key, the TLS file or the peer token file. `slinuxm --print-config` shows the effective config. | | Port changed in config, nothing happened | Port, bind and auth need a restart, not a reload. |