Files
skyg-linux-tracker/docs/operations.md
T
2026-10-21 00:12:33 -04:00

6.8 KiB

Operations

Day-to-day tasks for a running install. The service user below is skyg-linux-tracker (NixOS) or skyg (the manual example unit); substitute yours.

Tokens

sudo -u skyg-linux-tracker slinuxm token create my-laptop    # prints the token once
sudo -u skyg-linux-tracker slinuxm token list                # name, created, revoked, last used
sudo -u skyg-linux-tracker slinuxm token rotate my-laptop    # new secret, printed once; the old one stops working
sudo -u skyg-linux-tracker slinuxm token revoke my-laptop
  • Tokens look like slinuxm_.... Only a sha256 hash is stored, so a lost token cannot be recovered: rotate it.
  • Every change takes effect on the next request. No restart, no reload.
  • rotate also re-enables a revoked token under the same name.
  • Use one token per client (laptop, agent, consolidator), so you can revoke one without touching the others.
  • Run the commands as the service user. Run as anyone else, root included, they refuse and print the right sudo -u command. The first token create needs the state directory, which the service creates on its first start.
  • If the config is not at /etc/skyg-linux-tracker.toml, add --config <file>.

Use the API

TOKEN=slinuxm_...
BASE=http://127.0.0.1:1984

curl -s $BASE/healthz                                                    # public
curl -s -H "Authorization: Bearer $TOKEN" "$BASE/api/v1/systemd/units?state=failed"
curl -s -H "Authorization: Bearer $TOKEN" "$BASE/api/v1/systemd/timers"
curl -s -H "Authorization: Bearer $TOKEN" "$BASE/api/v1/journal?unit=nginx.service&priority=err&lines=50"
curl -N -H "Authorization: Bearer $TOKEN" "$BASE/api/v1/journal/follow?priority=warning"   # live stream (SSE)
curl -s -H "Authorization: Bearer $TOKEN" "$BASE/api/v1/docker/ps"

Every route, parameter and error: api.md. The same reference is browsable at $BASE/docs (paste a token when asked), and slinuxm openapi prints it as OpenAPI 3.1.

Connect an MCP client

Endpoint http://127.0.0.1:1984/mcp, Streamable HTTP, with the bearer header.

Claude Code:

claude mcp add --transport http linux-tracker http://127.0.0.1:1984/mcp \
  --header "Authorization: Bearer $TOKEN"

Any client that takes a JSON config with headers:

{
  "mcpServers": {
    "linux-tracker": {
      "url": "http://127.0.0.1:1984/mcp",
      "headers": { "Authorization": "Bearer slinuxm_..." }
    }
  }
}

The server offers eight tools (list_systemd_units, get_systemd_unit, list_systemd_timers, get_journal_logs, tail_journal, list_containers, get_container, get_container_logs), two prompts, two resources and two resource templates. Details and a raw curl handshake: api.md.

Expose it beyond localhost

The default is loopback only. The safest way to reach it from elsewhere is to leave it there and use an SSH tunnel, Tailscale or a TLS reverse proxy. To listen on the network directly:

  1. Set server.bind ("0.0.0.0" or "::"). Auth must stay on.
  2. Turn on TLS (tls.md), unless the network is trusted: bearer tokens are readable in plain HTTP.
  3. Add the names clients use to server.allowed_hosts (e.g. "monitor.lan:1984"), or /mcp rejects them.
  4. Open the port in the firewall, ideally on the LAN interface only.

On NixOS that is bind, tls.*, allowedHosts and firewallInterfaces; see install.md.

Docker access

Reading the Docker socket needs either:

  • A read-only socket proxy (recommended). For example tecnativa/docker-socket-proxy with CONTAINERS=1 EVENTS=1 POST=0, exposed as a unix socket. Then:

    [sources]
    docker_socket = "/run/docker-proxy/docker.sock"
    
  • The docker group. Simple, but membership is root-equivalent: anyone who controls the process could start a privileged container. Manual unit: uncomment SupplementaryGroups=docker. NixOS: docker.enable = true.

Out of the box neither is set up, so the Docker routes and tools return 503 while the rest works.

OpenTelemetry

[otel]
enabled = true
endpoint = "http://collector:4317"   # or :4318 with protocol = "http/protobuf"
export_logs = true                   # also forward logs, not only metrics

Metrics (every metrics_interval_secs, gauges):

Metric Meaning
systemd.unit.active, systemd.unit.failed 0/1 per unit
systemd.unit.restarts restart count (services)
systemd.timer.next_elapse_timestamp_seconds next run of a timer, unix time
docker.container.running, docker.container.healthy 0/1 per container
docker.container.uptime_seconds, docker.container.restarts per container
peer.* peer health, with peers enabled (consolidator.md)

By default only *.service and *.timer units are exported; set [otel.metrics] units to a glob list to change that.

Logs (export_logs = true): journal entries and container stdout/stderr, with severity mapped from the journal priority (stderr is WARN, stdout INFO). Narrow them with [otel.logs].

Traces: one span per HTTP and MCP request.

For a local test collector that prints everything it receives: devenv up in a checkout.

Reload, restart, stop

systemctl reload skyg-linux-tracker    # SIGHUP: re-read the TLS pair and [peers]; connections stay up
systemctl restart skyg-linux-tracker   # anything else in the config
systemctl stop skyg-linux-tracker      # graceful: open streams end, requests drain, OTel flushes

Troubleshooting

The server logs to stderr; under systemd: journalctl -u skyg-linux-tracker.

Symptom Likely cause
401 on every request Missing Authorization: Bearer header, or the token was revoked or rotated. /healthz works without a token.
503 on /api/v1/docker/* only No access to the Docker socket, or Docker is not running. See Docker access.
403 on REST with auth off With auth.enabled = false, REST accepts only loopback Host names plus allowed_hosts.
MCP client fails to connect from another host Its Host (or browser Origin) is not in server.allowed_hosts / allowed_origins.
token create refuses with a sudo -u hint You ran it as the wrong user, root included. Use the command it prints.
token create cannot find the database The service has never started (no state directory yet), or --config points at a different file.
Journal results are short or empty The service user is not in systemd-journal, or sources.journal_namespace names the wrong namespace.
Server refuses to start Read the first error line: it names the config key, the TLS file or the peer token file. slinuxm --print-config shows the effective config.
Port changed in config, nothing happened Port, bind and auth need a restart, not a reload.