Files
skyg-linux-tracker/docs/consolidator.md
T

20 KiB

Consolidator (peers)

One slinuxm can be a consolidator for other slinuxm instances. Each one it is told about is a peer: it health-checks the peer in two ways and passes read-only requests through to it, so one MCP endpoint or one REST base URL covers the fleet. The feature is off by default ([peers] enabled = false), and a peer-less instance behaves exactly as before: same routes, eight MCP tools, same metrics, no extra tasks or config.

Everything here is declarative: peers come from the config file, nothing writes to a peer, and no request parameter can supply a URL. Runtime peer management (peer add, a peers table) does not exist.

Tests: tests/peers.rs runs the whole path in-process (no VM), and a three-machine NixOS VM test covers boot, TLS, rotation and outages. Both go through the repository's runner, devenv shell -- dev-test …; see Testing below.

Model

Check What it proves How
server The peer's port accepts a TCP connection one connect per tcp_interval_secs, bounded by tcp_timeout_secs
api The peer's REST API runs and accepts our token authenticated GET /api/v1/ping every api_interval_secs, bounded by api_timeout_secs

A check flips down after fail_threshold consecutive failures and up again on the first success. All state is in memory (since, last_checked, last_error, latency_ms), so a restart of the consolidator resets it; history belongs in your OTel backend.

last_error is a class, never peer text or a URL: unreachable, timeout, dns, tls_error, http_<code>, unauthorized, oversize, saturated, token_file.

Reading server and api

server.up api.up api.auth Meaning
true true ok healthy
true true rejected the API runs, our token is refused: revoked, rotated on the peer, or the file is still a placeholder. Mint a new token on the peer and replace the file (auth_ok is 0 in OTel while peer.up is 1 for both checks)
true true unknown the peer predates GET /api/v1/ping; the check fell back to public /healthz, so liveness is proven but not our credential
true false unknown the port answers but the API does not (tls_error for a certificate problem, timeout, http_5xx)
false false unknown no resolved address answers on the port: the peer is down or unreachable (see Address families)

A peer that rejects our token is reported to your callers as a 503 naming the peer, never as a 401: re-authenticating with this host would not help.

Configuration

[peers]
enabled = false                 # default: no peer routes, tools, tasks or metrics
tcp_interval_secs = 10
tcp_timeout_secs = 3            # must be below tcp_interval_secs
api_interval_secs = 30
api_timeout_secs = 5            # must be below api_interval_secs
fail_threshold = 2              # consecutive failures before a check flips down
request_timeout_secs = 10       # one proxied request, connect to body
max_response_bytes = 8388608    # buffered bodies above this are refused
max_concurrent_per_peer = 4     # proxied requests in flight per peer
max_concurrent_total = 16       # ... across all peers
deny_cidrs = ["169.254.0.0/16", "0.0.0.0/8", "fe80::/10", "::/128", "fd00:ec2::254/128"]  # resolved peer addresses in these ranges are refused

[[peers.peer]]
name = "minipc1"                # ^[a-z0-9][a-z0-9_-]{0,31}$, unique, at most 64
url = "https://minipc1:1984"    # http or https, no userinfo, query or fragment
token_file = "/run/credentials/skyg-linux-tracker.service/peer-minipc1"   # absolute
ca_file = "/etc/skyg/private-ca.pem"   # optional extra trust anchor
allow_plain_http = false        # required true for http:// to a non-loopback host
enabled = true                  # false keeps the declaration without contacting the peer

Scalar keys accept environment overrides (SKYG_PEERS__ENABLED, SKYG_PEERS__API_INTERVAL_SECS, ...); [[peers.peer]] entries are file-only. slinuxm --print-config prints token_file paths, never their contents. Validation refuses: a timeout that is not below its interval, more than 64 peers, duplicate names, a bad name, a relative token_file or ca_file, a non-http(s) URL, userinfo/query/fragment in a URL, an IP literal in deny_cidrs, and plain http to a non-loopback host without allow_plain_http = true (which also logs a warning: the bearer token travels in cleartext). enabled = true with no peers logs a warning.

deny_cidrs is checked against every resolved address at connect time, so a name where any address lands in a denied range is refused as unreachable. IPv4-mapped addresses are compared in their IPv4 form, so ::ffff:169.254.169.254 is caught by 169.254.0.0/16. Private LAN ranges are deliberately allowed; the defaults deny only link-local and cloud-metadata addresses in both families (169.254.0.0/16, fe80::/10, fd00:ec2::254/128) and the unspecified address (0.0.0.0/8, ::/128).

REST and MCP

All under the inbound auth of this host; nothing is mounted while peers are off.

Route Behaviour
GET /api/v1/peers {"peers": [...]}, sorted by name, disabled peers included
GET /api/v1/peers/{peer} one status object; unknown peer is 404
GET /api/v1/peers/{peer}/... GET-only pass-through of an allowlisted route

The status object is {name, enabled, url, server{up, since, last_checked, last_error, latency_ms}, api{up, since, last_checked, last_error, latency_ms, auth, peer_version}}, times in RFC 3339 UTC.

The pass-through allowlist is systemd/..., journal, journal/follow, docker/..., with the query parameters the local routes accept (state, unit, priority, lines, since, all, tail, follow). A caller may include the /api/v1 prefix or leave it out; the peer always receives the route under its own /api/v1. Anything else is a 400, .. and encoded slashes are rejected, ping, peers/... and /healthz are not reachable through it, and methods other than GET are 405. The peer's 2xx answer comes back unchanged (status, Content-Type, body; SSE stays a stream) with x-slinuxm-peer: <peer>; every error produced for a peer carries that header too, except the 405 for a non-GET method, which the router answers before the peer route runs. Consolidated failures use the existing error shape:

Peer outcome Answer to your caller
peer 404 404 not_found, peer <name>: not found on peer
peer 400/422 400 bad_request
peer rejects our token (401/403) 503 unavailable, peer <name>: peer rejected credentials
timeout, refused, DNS, TLS, 5xx, oversize, saturated, token file 503 unavailable with the class
malformed peer JSON 500 internal
unknown peer 404 not_found

No caller header is forwarded except traceparent; the Authorization header is always the peer's own token from token_file. A caller that sends x-slinuxm-hop is refused with 400 (no chained consolidators).

With peers enabled, MCP exposes list_peers, get_peer and the resource peers://status (same JSON as the list), every one of the eight existing tools gains an optional peer argument, and the container_health_report and diagnose_unit prompts gain a peer argument. Ten tools instead of eight, and only on the instances that need them. Peer-sourced text is labelled untrusted data, as local log text is.

OpenTelemetry

Four observable gauges, registered only with peers enabled and otel.metrics.enabled, read from the registry snapshot on each scrape: peer.up (attributes peer.name, peer.check = server/api), peer.api.auth_ok (peer.name), peer.check.duration_seconds (peer.name, peer.check) and peer.last_change_timestamp_seconds (peer.name, peer.check). No URL, version or token in attributes. A removed peer disappears from the next scrape. Outbound proxied requests carry a span named peer.request with peer.name, and the caller's traceparent is passed through, so a trace spans the fleet.

Credentials and threat model

DEC-11 stores inbound tokens only as sha256 hashes. A consolidator cannot do that: it has to replay the peer's plaintext bearer token. DEC-13 therefore makes outbound peer tokens externally managed files:

  • slinuxm owns no copy of a peer token: not in SQLite, not in config.toml, not in --print-config, the logs or a support bundle. The config holds a path; the process reads the file into a redacted Secret on every check and request and drops it.
  • On NixOS the file is an agenix secret decrypted at activation to /run/agenix/... (root-only) and handed to the service through systemd LoadCredential, so the service user never needs read access to the agenix output. Outside NixOS, put the file somewhere only the service user can read: a group- or world-accessible token file is refused (token_file class) unless it sits in $CREDENTIALS_DIRECTORY.
  • Rotation is a file replacement; the next check reads the new value. On NixOS also systemctl reload (SIGHUP) so systemd re-copies the credential (RefreshOnReload=credentials).
  • Use one token per consolidator on every peer (consolidator-<host>, read-only by nature since every token is read-only): slinuxm token revoke consolidator-<host> on a peer cuts exactly that consolidator off, and nothing else.

What it does not protect: root on the consolidator (it can read the decrypted file), and the consolidator's own token as an asset — it reaches every peer's data, which is why tokens are per consolidator, revocable, and why scoping tokens per peer is a follow-up, not a v1 feature.

Peers are only ever declared by whoever manages the config (root or Nix). No route, parameter or MCP argument supplies a URL, so the consolidator cannot be steered at an arbitrary target. Redirects from a peer are never followed (503 http_3xx), TLS is verified against the webpki roots plus the optional ca_file, and there is no skip-verify switch.

NixOS with agenix

The module renders the peers and the credentials; you only declare them:

services.skyg-linux-tracker = {
  enable = true;
  bind = "0.0.0.0";                          # inbound, only if callers reach this host from other machines
  openFirewall = true;                       # or firewallInterfaces = [ "enp3s0" ];
  peers = {
    enable = true;
    settings = { api_interval_secs = 30; fail_threshold = 2; };   # [peers] tuning, snake_case
    peers = {
      minipc1 = { url = "https://minipc1:1984"; tokenFile = config.age.secrets.peer-minipc1.path; };
      nas = {
        url = "http://nas:1984";
        tokenFile = config.age.secrets.peer-nas.path;
        allowPlainHttp = true;               # the token then travels in cleartext in your LAN
      };
    };
  };
};

A worked agenix setup for host monitor, which consolidates minipc1 (HTTPS with a private CA) and nas (plain HTTP in a trusted LAN).

On each peer, mint a token for this consolidator. It has to be a dedicated token: revoking it cuts off exactly this consolidator.

# on minipc1 and on nas
sudo -u skyg-linux-tracker slinuxm token create consolidator-monitor

Register the encrypted files next to your other agenix secrets and encrypt them for the consolidator host. Only the ciphertext goes into git.

# secrets/secrets.nix (agenix)
{
  # The token minipc1 issued to `monitor`. Only `monitor` needs to decrypt it:
  # the consolidator replays it, no other host reads it.
  "peer-minipc1.age".publicKeys = [ monitor ];
  "peer-nas.age".publicKeys = [ monitor ];
}
agenix -e secrets/peer-minipc1.age     # paste the token printed on minipc1
agenix -e secrets/peer-nas.age         # paste the token printed on nas
{ config, inputs, ... }: {
  imports = [ inputs.agenix.nixosModules.default ];

  # Decrypted at activation to /run/agenix/peer-<name>, root-only. The module
  # hands the file to the service through LoadCredential, so the
  # `skyg-linux-tracker` user never reads it directly.
  age.secrets.peer-minipc1 = {
    file = ./secrets/peer-minipc1.age;
    owner = "root";
    mode = "0400";
  };
  age.secrets.peer-nas = { file = ./secrets/peer-nas.age; owner = "root"; mode = "0400"; };

  services.skyg-linux-tracker = {
    enable = true;
    bind = "0.0.0.0";                        # inbound for remote callers; peers need no inbound port
    openFirewall = true;
    peers = {
      enable = true;
      peers = {
        minipc1 = {
          url = "https://minipc1:1984";
          tokenFile = config.age.secrets.peer-minipc1.path;   # a string path, not a path literal
          caFile = "/etc/ssl/private-ca.pem";                  # readable by the service user
        };
        nas = {
          url = "http://nas:1984";
          tokenFile = config.age.secrets.peer-nas.path;
          allowPlainHttp = true;
        };
      };
    };
  };
}

The module turns that into one [[peers.peer]] entry per enabled peer, with token_file = /run/credentials/skyg-linux-tracker.service/peer-<name>, and one LoadCredential=peer-<name>:<tokenFile> line. Consequences worth knowing:

  • tokenFile must be a string (config.age.secrets...path), not a path literal: a nix path or "${./secret}" interpolates a store path and is rejected at evaluation. Such a secret has already been copied into the local store by then, so rotate it.
  • caFile is not passed through LoadCredential: the service user reads it directly, so it must be readable by skyg-linux-tracker. A root-only file such as the agenix output makes every check and proxied request to that peer fail as tls_error. Store paths are fine for a public CA.
  • Peers with enable = false are rendered nowhere, so no credential is shipped for them.
  • peers.settings is freeform [peers] tuning; it must not contain enabled, peer or a token.

Reload or restart

switch-to-configuration reloads (SIGHUP) when only the config file changed, and restarts when the unit file changed. SIGHUP re-reads [peers], so editing a peer's url, caFile or the intervals applies without a restart, and systemd re-copies the credentials (RefreshOnReload=credentials), so a rotated token is picked up as well. Adding, removing, enabling or disabling a peer, or changing its tokenFile, changes the LoadCredential= lines and therefore restarts the service — the safe outcome. enabled, deny_cidrs, max_response_bytes and the two concurrency limits also need a restart (the routes are mounted once and the limits live in the transport), and the module's restart trigger makes a switch do that instead of silently keeping the old values.

Outside NixOS the same split applies by hand: SIGHUP re-reads the TLS pair and [peers] (peer entries, intervals, timeouts, fail_threshold); everything else needs a restart.

Shutdown

SIGTERM stops the listener, cancels the shared token and then drains. The health loops end on that token, and a proxied follow stream is closed rather than held until the peer decides to stop sending: shutdown ends the SSE stream to your caller even when the peer keeps its socket open (tests/peers.rs asserts exactly that against a listener that never closes). A buffered proxied request is bounded by request_timeout_secs and a follow stream by the peer's own cap, so no peer connection outlives the process by more than one timeout.

Firewall and networking

Every peer must accept the consolidator's address on its port; the consolidator only needs outbound access. On a peer's NixOS host that is openFirewall = true or firewallInterfaces = [ "enp3s0" ]; a peer bound to 127.0.0.1 is not reachable from another machine, so a consolidator needs the peer to bind a LAN address (with auth on, which the module enforces) and either TLS or a trusted network.

Address families

A peer's name is resolved at connect time and every resolved address is used: the health probe tries them in order within its timeout, and the HTTP client pins the whole list so the connection falls through to the next address when the first one refuses or does not answer. A peer that listens on IPv4 only is therefore reached even when its name also resolves to an IPv6 address (as a NixOS test network or a dual-stack LAN does). Both families and a single family work; bind = "::" on a peer is dual stack, bind = "0.0.0.0" is IPv4 only.

DNS rebinding is still not a risk: the set is resolved once per request and pinned, and a name where any resolved address falls in deny_cidrs is refused as unreachable rather than dialled at the permitted addresses.

Limits and non-goals

  • No aggregation: peer = "*" and fleet-wide views are a follow-up, because partial failures need a result shape of their own. Ask one peer at a time.
  • No chaining: a consolidator's request carries x-slinuxm-hop, so a peer that is itself a consolidator refuses it. This prevents loops and amplification.
  • No write path: proxying is GET-only and allowlisted, and nothing here starts, stops or changes a unit or a container.
  • No history: peer state is in memory; use OTel for trends.
  • No runtime management: peers are declarative, reloaded on SIGHUP.
  • Each peer exports its own telemetry; peers' OTel signals are not re-exported by the consolidator.

Testing

Two layers, neither of which needs a VM for the first:

  • tests/peers.rs drives the whole path in-process: a consolidator served by the real router and registry with the real outbound client, against peers that are either another in-process slinuxm (fake sources, real auth with a minted token) or a hand-written listener for a peer that misbehaves. It covers a plain peer and a TLS peer with its own CA, a peer signed by the wrong CA, a redirect, an oversize (declared and chunked) body, a silent peer, a slow-loris peer, the 64-peer cap, shutdown ending a proxied follow, and a peer declared by name that listens on one address family only.
  • The NixOS VM test boots three machines running the real slinuxm: alpha and beta consolidate each other over plain HTTP, and alpha also declares the TLS-only gamma with a private CA (bound IPv4 only, so the address-family fallback stays guarded). It asserts status, fan-in, proxied data against a direct call, MCP parity, token install/rotation/revocation, peer removal on SIGHUP and an outage, all with MainPID unchanged where a reload is promised. nix flake check also runs checks.module-peers, an evaluation-only case list for the module: every rendering and assertion fails the check without booting anything.
devenv shell -- dev-test integration peers # cargo test --test peers
devenv shell -- dev-test vm peers          # nix build .#checks.x86_64-linux.vm-peers -L

dev/dev-test.nu is the repository's test runner. devenv shell -- dev-test --help lists its subcommands and the command behind each. In an interactive devenv shell, dev-test … is that same script.

In Nushell the runner is a module. Load it once, for example in config.nu, and dev-test unit, dev-test vm peers, ... complete natively. Integration-test name completion is cached in target/skyg-test-filters.cache at the repo root (not relative to the current directory, so completion works from any subdirectory), keyed on the newest mtime of the Rust sources and the Cargo manifests. A failing check raises a catchable error instead of exiting, so a red dev-test unit does not close the Nushell session that loaded the module; script mode still exits non-zero.

use /home/you/src/skyg-linux-tracker/dev/dev-test.nu *    # absolute path to dev/dev-test.nu

zsh and bash completion comes from the same command table. The generated script only registers completion for the external command dev-test. An interactive devenv shell sources the bash script. That hook runs in bash and direnv will not source it either, so zsh users add the following to .zshrc after compinit:

source <(nu /home/you/src/skyg-linux-tracker/dev/dev-test.nu completions zsh)

bash, from the repo checkout:

source <(nu dev/dev-test.nu completions bash)

nu dev/dev-test.nu self-check fails if the command table, the defined subcommands, the help text, the cargo argv, the filter cache or the shell completion drift apart.