Files
2026-10-21 00:12:33 -04:00

140 lines
6.8 KiB
Markdown

# Operations
Day-to-day tasks for a running install. The service user below is `skyg-linux-tracker` (NixOS) or `skyg` (the manual example unit); substitute yours.
## Tokens
```sh
sudo -u skyg-linux-tracker slinuxm token create my-laptop # prints the token once
sudo -u skyg-linux-tracker slinuxm token list # name, created, revoked, last used
sudo -u skyg-linux-tracker slinuxm token rotate my-laptop # new secret, printed once; the old one stops working
sudo -u skyg-linux-tracker slinuxm token revoke my-laptop
```
- Tokens look like `slinuxm_...`. Only a sha256 hash is stored, so a lost token cannot be recovered: rotate it.
- Every change takes effect on the next request. No restart, no reload.
- `rotate` also re-enables a revoked token under the same name.
- Use one token per client (laptop, agent, consolidator), so you can revoke one without touching the others.
- Run the commands **as the service user**. Run as anyone else, root included, they refuse and print the right `sudo -u` command. The first `token create` needs the state directory, which the service creates on its first start.
- If the config is not at `/etc/skyg-linux-tracker.toml`, add `--config <file>`.
## Use the API
```sh
TOKEN=slinuxm_...
BASE=http://127.0.0.1:1984
curl -s $BASE/healthz # public
curl -s -H "Authorization: Bearer $TOKEN" "$BASE/api/v1/systemd/units?state=failed"
curl -s -H "Authorization: Bearer $TOKEN" "$BASE/api/v1/systemd/timers"
curl -s -H "Authorization: Bearer $TOKEN" "$BASE/api/v1/journal?unit=nginx.service&priority=err&lines=50"
curl -N -H "Authorization: Bearer $TOKEN" "$BASE/api/v1/journal/follow?priority=warning" # live stream (SSE)
curl -s -H "Authorization: Bearer $TOKEN" "$BASE/api/v1/docker/ps"
```
Every route, parameter and error: [api.md](api.md). The same reference is browsable at `$BASE/docs` (paste a token when asked), and `slinuxm openapi` prints it as OpenAPI 3.1.
## Connect an MCP client
Endpoint `http://127.0.0.1:1984/mcp`, Streamable HTTP, with the bearer header.
Claude Code:
```sh
claude mcp add --transport http linux-tracker http://127.0.0.1:1984/mcp \
--header "Authorization: Bearer $TOKEN"
```
Any client that takes a JSON config with headers:
```json
{
"mcpServers": {
"linux-tracker": {
"url": "http://127.0.0.1:1984/mcp",
"headers": { "Authorization": "Bearer slinuxm_..." }
}
}
}
```
The server offers eight tools (`list_systemd_units`, `get_systemd_unit`, `list_systemd_timers`, `get_journal_logs`, `tail_journal`, `list_containers`, `get_container`, `get_container_logs`), two prompts, two resources and two resource templates. Details and a raw curl handshake: [api.md](api.md#mcp).
## Expose it beyond localhost
The default is loopback only. The safest way to reach it from elsewhere is to leave it there and use an SSH tunnel, Tailscale or a TLS reverse proxy. To listen on the network directly:
1. Set `server.bind` (`"0.0.0.0"` or `"::"`). Auth must stay on.
2. Turn on TLS ([tls.md](tls.md)), unless the network is trusted: bearer tokens are readable in plain HTTP.
3. Add the names clients use to `server.allowed_hosts` (e.g. `"monitor.lan:1984"`), or `/mcp` rejects them.
4. Open the port in the firewall, ideally on the LAN interface only.
On NixOS that is `bind`, `tls.*`, `allowedHosts` and `firewallInterfaces`; see [install.md](install.md#4-common-setups).
## Docker access
Reading the Docker socket needs either:
- **A read-only socket proxy (recommended).** For example [tecnativa/docker-socket-proxy](https://github.com/Tecnativa/docker-socket-proxy) with `CONTAINERS=1 EVENTS=1 POST=0`, exposed as a unix socket. Then:
```toml
[sources]
docker_socket = "/run/docker-proxy/docker.sock"
```
- **The `docker` group.** Simple, but membership is root-equivalent: anyone who controls the process could start a privileged container. Manual unit: uncomment `SupplementaryGroups=docker`. NixOS: `docker.enable = true`.
Out of the box neither is set up, so the Docker routes and tools return `503` while the rest works.
## OpenTelemetry
```toml
[otel]
enabled = true
endpoint = "http://collector:4317" # or :4318 with protocol = "http/protobuf"
export_logs = true # also forward logs, not only metrics
```
**Metrics** (every `metrics_interval_secs`, gauges):
| Metric | Meaning |
| --- | --- |
| `systemd.unit.active`, `systemd.unit.failed` | 0/1 per unit |
| `systemd.unit.restarts` | restart count (services) |
| `systemd.timer.next_elapse_timestamp_seconds` | next run of a timer, unix time |
| `docker.container.running`, `docker.container.healthy` | 0/1 per container |
| `docker.container.uptime_seconds`, `docker.container.restarts` | per container |
| `peer.*` | peer health, with peers enabled ([consolidator.md](consolidator.md#opentelemetry)) |
By default only `*.service` and `*.timer` units are exported; set `[otel.metrics] units` to a glob list to change that.
**Logs** (`export_logs = true`): journal entries and container stdout/stderr, with severity mapped from the journal priority (stderr is WARN, stdout INFO). Narrow them with `[otel.logs]`.
**Traces**: one span per HTTP and MCP request.
For a local test collector that prints everything it receives: `devenv up` in a checkout.
## Reload, restart, stop
```sh
systemctl reload skyg-linux-tracker # SIGHUP: re-read the TLS pair and [peers]; connections stay up
systemctl restart skyg-linux-tracker # anything else in the config
systemctl stop skyg-linux-tracker # graceful: open streams end, requests drain, OTel flushes
```
## Troubleshooting
The server logs to stderr; under systemd: `journalctl -u skyg-linux-tracker`.
| Symptom | Likely cause |
| --- | --- |
| `401` on every request | Missing `Authorization: Bearer` header, or the token was revoked or rotated. `/healthz` works without a token. |
| `503` on `/api/v1/docker/*` only | No access to the Docker socket, or Docker is not running. See [Docker access](#docker-access). |
| `403` on REST with auth off | With `auth.enabled = false`, REST accepts only loopback `Host` names plus `allowed_hosts`. |
| MCP client fails to connect from another host | Its `Host` (or browser `Origin`) is not in `server.allowed_hosts` / `allowed_origins`. |
| `token create` refuses with a `sudo -u` hint | You ran it as the wrong user, root included. Use the command it prints. |
| `token create` cannot find the database | The service has never started (no state directory yet), or `--config` points at a different file. |
| Journal results are short or empty | The service user is not in `systemd-journal`, or `sources.journal_namespace` names the wrong namespace. |
| Server refuses to start | Read the first error line: it names the config key, the TLS file or the peer token file. `slinuxm --print-config` shows the effective config. |
| Port changed in config, nothing happened | Port, bind and auth need a restart, not a reload. |