Files
skyg-linux-tracker/docs/consolidator.md
T
ari 4d83c2eff5 F: F4 peers docs, agenix example and tool-count note
README peers section, docs/consolidator.md (threat model, tokens, firewall,
worked agenix example, reload/restart), the MCP tool count in both tables,
and config.example.toml commentary.
2026-10-20 15:04:09 -04:00

18 KiB

Consolidator (peers)

One slinuxm can be a consolidator for other slinuxm instances. Each one it is told about is a peer: it health-checks the peer in two ways and passes read-only requests through to it, so one MCP endpoint or one REST base URL covers the fleet. The feature is off by default ([peers] enabled = false), and a peer-less instance behaves exactly as before: same routes, eight MCP tools, same metrics, no extra tasks or config.

Everything here is declarative: peers come from the config file, nothing writes to a peer, and no request parameter can supply a URL. Runtime peer management (peer add, a peers table) does not exist.

Tests: tests/peers.rs runs the whole path in-process (no VM), and a three-machine NixOS VM test covers boot, TLS, rotation and outages. Both go through the repository's runner, devenv shell -- dev-test …; see Testing below.

Model

Check What it proves How
server The peer's port accepts a TCP connection one connect per tcp_interval_secs, bounded by tcp_timeout_secs
api The peer's REST API runs and accepts our token authenticated GET /api/v1/ping every api_interval_secs, bounded by api_timeout_secs

A check flips down after fail_threshold consecutive failures and up again on the first success. All state is in memory (since, last_checked, last_error, latency_ms), so a restart of the consolidator resets it; history belongs in your OTel backend.

last_error is a class, never peer text or a URL: unreachable, timeout, dns, tls_error, http_<code>, unauthorized, oversize, saturated, token_file.

Reading server and api

server.up api.up api.auth Meaning
true true ok healthy
true true rejected the API runs, our token is refused: revoked, rotated on the peer, or the file is still a placeholder. Mint a new token on the peer and replace the file (auth_ok is 0 in OTel while peer.up is 1 for both checks)
true true unknown the peer predates GET /api/v1/ping; the check fell back to public /healthz, so liveness is proven but not our credential
true false unknown the port answers but the API does not (tls_error for a certificate problem, timeout, http_5xx)
false false unknown nothing answers on the port: the peer is down or unreachable (also the case when the name resolves to an address the peer does not listen on, see Address families)

A peer that rejects our token is reported to your callers as a 503 naming the peer, never as a 401: re-authenticating with this host would not help.

Configuration

[peers]
enabled = false                 # default: no peer routes, tools, tasks or metrics
tcp_interval_secs = 10
tcp_timeout_secs = 3            # must be below tcp_interval_secs
api_interval_secs = 30
api_timeout_secs = 5            # must be below api_interval_secs
fail_threshold = 2              # consecutive failures before a check flips down
request_timeout_secs = 10       # one proxied request, connect to body
max_response_bytes = 8388608    # buffered bodies above this are refused
max_concurrent_per_peer = 4     # proxied requests in flight per peer
max_concurrent_total = 16       # ... across all peers
deny_cidrs = ["169.254.0.0/16", "0.0.0.0/8"]   # resolved peer addresses in these ranges are refused

[[peers.peer]]
name = "minipc1"                # ^[a-z0-9][a-z0-9_-]{0,31}$, unique, at most 64
url = "https://minipc1:1984"    # http or https, no userinfo, query or fragment
token_file = "/run/credentials/skyg-linux-tracker.service/peer-minipc1"   # absolute
ca_file = "/etc/skyg/private-ca.pem"   # optional extra trust anchor
allow_plain_http = false        # required true for http:// to a non-loopback host
enabled = true                  # false keeps the declaration without contacting the peer

Scalar keys accept environment overrides (SKYG_PEERS__ENABLED, SKYG_PEERS__API_INTERVAL_SECS, ...); [[peers.peer]] entries are file-only. slinuxm --print-config prints token_file paths, never their contents. Validation refuses: a timeout that is not below its interval, more than 64 peers, duplicate names, a bad name, a relative token_file or ca_file, a non-http(s) URL, userinfo/query/fragment in a URL, an IP literal in deny_cidrs, and plain http to a non-loopback host without allow_plain_http = true (which also logs a warning: the bearer token travels in cleartext). enabled = true with no peers logs a warning.

deny_cidrs is checked against the resolved address at connect time, so a name that resolves into a denied range is refused as unreachable. Private LAN ranges are deliberately allowed; the default denies only link-local/metadata and the unspecified range.

REST and MCP

All under the inbound auth of this host; nothing is mounted while peers are off.

Route Behaviour
GET /api/v1/peers {"peers": [...]}, sorted by name, disabled peers included
GET /api/v1/peers/{peer} one status object; unknown peer is 404
GET /api/v1/peers/{peer}/... GET-only pass-through of an allowlisted route

The status object is {name, enabled, url, server{up, since, last_checked, last_error, latency_ms}, api{up, since, last_checked, last_error, latency_ms, auth, peer_version}}, times in RFC 3339 UTC.

The pass-through allowlist is systemd/..., journal, journal/follow, docker/..., with the query parameters the local routes accept (state, unit, priority, lines, since, all, tail, follow). A caller may include the /api/v1 prefix; it is stripped. Anything else is a 400, .. and encoded slashes are rejected, ping, peers/... and /healthz are not reachable through it, and methods other than GET are 405. The peer's 2xx answer comes back unchanged (status, Content-Type, body; SSE stays a stream) with x-slinuxm-peer: <peer>; every error produced for a peer carries that header too, except the 405 for a non-GET method, which the router answers before the peer route runs. Consolidated failures use the existing error shape:

Peer outcome Answer to your caller
peer 404 404 not_found, peer <name>: not found on peer
peer 400/422 400 bad_request
peer rejects our token (401/403) 503 unavailable, peer <name>: peer rejected credentials
timeout, refused, DNS, TLS, 5xx, oversize, saturated, token file 503 unavailable with the class
malformed peer JSON 500 internal
unknown peer 404 not_found

No caller header is forwarded except traceparent; the Authorization header is always the peer's own token from token_file. A caller that sends x-slinuxm-hop is refused with 400 (no chained consolidators).

With peers enabled, MCP exposes list_peers, get_peer and the resource peers://status (same JSON as the list), every one of the eight existing tools gains an optional peer argument, and the container_health_report and diagnose_unit prompts gain a peer argument. Ten tools instead of eight, and only on the instances that need them. Peer-sourced text is labelled untrusted data, as local log text is.

OpenTelemetry

Four observable gauges, registered only with peers enabled and otel.metrics.enabled, read from the registry snapshot on each scrape: peer.up (attributes peer.name, peer.check = server/api), peer.api.auth_ok (peer.name), peer.check.duration_seconds (peer.name, peer.check) and peer.last_change_timestamp_seconds (peer.name, peer.check). No URL, version or token in attributes. A removed peer disappears from the next scrape. Outbound proxied requests carry a span named peer.request with peer.name, and the caller's traceparent is passed through, so a trace spans the fleet.

Credentials and threat model

DEC-11 stores inbound tokens only as sha256 hashes. A consolidator cannot do that: it has to replay the peer's plaintext bearer token. DEC-13 therefore makes outbound peer tokens externally managed files:

  • slinuxm owns no copy of a peer token: not in SQLite, not in config.toml, not in --print-config, the logs or a support bundle. The config holds a path; the process reads the file into a redacted Secret on every check and request and drops it.
  • On NixOS the file is an agenix secret decrypted at activation to /run/agenix/... (root-only) and handed to the service through systemd LoadCredential, so the service user never needs read access to the agenix output. Outside NixOS, put the file somewhere only the service user can read: a group- or world-readable token file is refused (token_file class) unless it sits in $CREDENTIALS_DIRECTORY.
  • Rotation is a file replacement; the next check reads the new value. On NixOS also systemctl reload (SIGHUP) so systemd re-copies the credential (RefreshOnReload=credentials).
  • Use one token per consolidator on every peer (consolidator-<host>, read-only by nature since every token is read-only): slinuxm token revoke consolidator-<host> on a peer cuts exactly that consolidator off, and nothing else.

What it does not protect: root on the consolidator (it can read the decrypted file), and the consolidator's own token as an asset — it reaches every peer's data, which is why tokens are per consolidator, revocable, and why scoping tokens per peer is a follow-up, not a v1 feature.

Peers are only ever declared by whoever manages the config (root or Nix). No route, parameter or MCP argument supplies a URL, so the consolidator cannot be steered at an arbitrary target. Redirects from a peer are never followed (503 http_3xx), TLS is verified against the webpki roots plus the optional ca_file, and there is no skip-verify switch.

NixOS with agenix

The module renders the peers and the credentials; you only declare them:

services.skyg-linux-tracker = {
  enable = true;
  bind = "0.0.0.0";
  openFirewall = true;                       # or firewallInterfaces = [ "enp3s0" ];
  peers = {
    enable = true;
    settings = { api_interval_secs = 30; fail_threshold = 2; };   # [peers] tuning, snake_case
    peers = {
      minipc1 = { url = "https://minipc1:1984"; tokenFile = config.age.secrets.peer-minipc1.path; };
      nas = {
        url = "http://nas:1984";
        tokenFile = config.age.secrets.peer-nas.path;
        allowPlainHttp = true;               # the token then travels in cleartext in your LAN
      };
    };
  };
};

A worked agenix setup for host monitor, which consolidates minipc1 (HTTPS with a private CA) and nas (plain HTTP in a trusted LAN).

On each peer, mint a token for this consolidator. It has to be a dedicated token: revoking it cuts off exactly this consolidator.

# on minipc1 and on nas
sudo -u skyg-linux-tracker slinuxm token create consolidator-monitor

Register the encrypted files next to your other agenix secrets and encrypt them for the consolidator host. Only the ciphertext goes into git.

# secrets/secrets.nix (agenix)
{
  # The token minipc1 issued to `monitor`. Only `monitor` needs to decrypt it:
  # the consolidator replays it, no other host reads it.
  "peer-minipc1.age".publicKeys = [ monitor ];
  "peer-nas.age".publicKeys = [ monitor ];
}
agenix -e secrets/peer-minipc1.age     # paste the token printed on minipc1
agenix -e secrets/peer-nas.age         # paste the token printed on nas
{ config, inputs, ... }: {
  imports = [ inputs.agenix.nixosModules.default ];

  # Decrypted at activation to /run/agenix/peer-<name>, root-only. The module
  # hands the file to the service through LoadCredential, so the
  # `skyg-linux-tracker` user never reads it directly.
  age.secrets.peer-minipc1 = {
    file = ./secrets/peer-minipc1.age;
    owner = "root";
    mode = "0400";
  };
  age.secrets.peer-nas = { file = ./secrets/peer-nas.age; owner = "root"; mode = "0400"; };

  services.skyg-linux-tracker = {
    enable = true;
    bind = "0.0.0.0";
    openFirewall = true;
    peers = {
      enable = true;
      peers = {
        minipc1 = {
          url = "https://minipc1:1984";
          tokenFile = config.age.secrets.peer-minipc1.path;   # a string path, not a path literal
          caFile = "/etc/ssl/private-ca.pem";                  # readable by the service user
        };
        nas = {
          url = "http://nas:1984";
          tokenFile = config.age.secrets.peer-nas.path;
          allowPlainHttp = true;
        };
      };
    };
  };
}

The module turns that into one [[peers.peer]] entry per enabled peer, with token_file = /run/credentials/skyg-linux-tracker.service/peer-<name>, and one LoadCredential=peer-<name>:<tokenFile> line. Consequences worth knowing:

  • tokenFile must be a string (config.age.secrets...path), not a path literal: a nix path or "${./secret}" interpolates a store path and is rejected at evaluation. Such a secret has already been copied into the local store by then, so rotate it.
  • caFile is not passed through LoadCredential: the service user reads it directly, so it must be readable by skyg-linux-tracker. A root-only file such as the agenix output makes every check and proxied request to that peer fail as tls_error. Store paths are fine for a public CA.
  • Peers with enable = false are rendered nowhere, so no credential is shipped for them.
  • peers.settings is freeform [peers] tuning; it must not contain enabled, peer or a token.

Reload or restart

switch-to-configuration reloads (SIGHUP) when only the config file changed, and restarts when the unit file changed. SIGHUP re-reads [peers], so editing a peer's url, caFile or the intervals applies without a restart, and systemd re-copies the credentials (RefreshOnReload=credentials), so a rotated token is picked up as well. Adding, removing, enabling or disabling a peer, or changing its tokenFile, changes the LoadCredential= lines and therefore restarts the service — the safe outcome. enabled, deny_cidrs, max_response_bytes and the two concurrency limits also need a restart (the routes are mounted once and the limits live in the transport), and the module's restart trigger makes a switch do that instead of silently keeping the old values.

Outside NixOS the same split applies by hand: SIGHUP re-reads the TLS pair and [peers] (peer entries, intervals, timeouts, fail_threshold); everything else needs a restart.

Firewall and networking

Every peer must accept the consolidator's address on its port; the consolidator only needs outbound access. On a peer's NixOS host that is openFirewall = true or firewallInterfaces = [ "enp3s0" ]; a peer bound to 127.0.0.1 is not reachable from another machine, so a consolidator needs the peer to bind a LAN address (with auth on, which the module enforces) and either TLS or a trusted network.

Address families

A peer is reached at the address its name resolves to at connect time, and the first resolved address wins. A peer that listens on IPv4 only, with a name that also resolves to an IPv6 address (as a NixOS test network does), reads as unreachable; declare it by IP address or let it listen on both families (bind = "::" accepts IPv4-mapped connections too). DNS rebinding is not a risk here because the target is pinned to the address that was checked against deny_cidrs.

Limits and non-goals

  • No aggregation: peer = "*" and fleet-wide views are a follow-up, because partial failures need a result shape of their own. Ask one peer at a time.
  • No chaining: a consolidator's request carries x-slinuxm-hop, so a peer that is itself a consolidator refuses it. This prevents loops and amplification.
  • No write path: proxying is GET-only and allowlisted, and nothing here starts, stops or changes a unit or a container.
  • No history: peer state is in memory; use OTel for trends.
  • No runtime management: peers are declarative, reloaded on SIGHUP.
  • Each peer exports its own telemetry; peers' OTel signals are not re-exported by the consolidator.

Testing

The peers acceptance test is a NixOS VM test. Three machines run the real slinuxm: alpha and beta consolidate each other over plain HTTP, and alpha also declares the TLS-only gamma with a private CA. It needs KVM and takes minutes.

devenv shell -- dev-test vm peers    # nix build .#tests.x86_64-linux.peers -L

Against a tree without the consolidator the test stops at the first peers assertion; that is expected, the test is written ahead of the feature. dev/dev-test.nu is the repository's test runner. devenv shell -- dev-test --help lists its subcommands and the command behind each. In an interactive devenv shell, dev-test … is that same script.

In Nushell the runner is a module. Load it once, for example in config.nu, and dev-test unit, dev-test vm peers, ... complete natively. Integration-test name completion is cached in target/skyg-test-filters.cache at the repo root (not relative to the current directory, so completion works from any subdirectory), keyed on the newest mtime of the Rust sources and the Cargo manifests. A failing check raises a catchable error instead of exiting, so a red dev-test unit does not close the Nushell session that loaded the module; script mode still exits non-zero.

use /home/you/src/skyg-linux-tracker/dev/dev-test.nu *    # absolute path to dev/dev-test.nu

zsh and bash completion comes from the same command table. The generated script only registers completion for the external command dev-test. An interactive devenv shell sources the bash script. That hook runs in bash and direnv will not source it either, so zsh users add the following to .zshrc after compinit:

source <(nu /home/you/src/skyg-linux-tracker/dev/dev-test.nu completions zsh)

bash, from the repo checkout:

source <(nu dev/dev-test.nu completions bash)

nu dev/dev-test.nu self-check fails if the command table, the defined subcommands, the help text, the cargo argv, the filter cache or the shell completion drift apart.