# Consolidator (peers) One `slinuxm` can be a **consolidator** for other `slinuxm` instances. Each one it is told about is a *peer*: it health-checks the peer in two ways and passes read-only requests through to it, so one MCP endpoint or one REST base URL covers the fleet. The feature is off by default (`[peers] enabled = false`), and a peer-less instance behaves exactly as before: same routes, eight MCP tools, same metrics, no extra tasks or config. Everything here is declarative: peers come from the config file, nothing writes to a peer, and no request parameter can supply a URL. Runtime peer management (`peer add`, a peers table) does not exist. Tests: `tests/peers.rs` runs the whole path in-process (no VM), and a three-machine NixOS VM test covers boot, TLS, rotation and outages. Both go through the repository's runner, `devenv shell -- dev-test …`; see [Testing](#testing) below. ## Model | Check | What it proves | How | |---|---|---| | `server` | The peer's port accepts a TCP connection | one connect per `tcp_interval_secs`, bounded by `tcp_timeout_secs` | | `api` | The peer's REST API runs **and** accepts our token | authenticated `GET /api/v1/ping` every `api_interval_secs`, bounded by `api_timeout_secs` | A check flips down after `fail_threshold` consecutive failures and up again on the first success. All state is in memory (`since`, `last_checked`, `last_error`, `latency_ms`), so a restart of the consolidator resets it; history belongs in your OTel backend. `last_error` is a class, never peer text or a URL: `unreachable`, `timeout`, `dns`, `tls_error`, `http_`, `unauthorized`, `oversize`, `saturated`, `token_file`. ### Reading `server` and `api` | `server.up` | `api.up` | `api.auth` | Meaning | |---|---|---|---| | `true` | `true` | `ok` | healthy | | `true` | `true` | `rejected` | the API runs, **our token is refused**: revoked, rotated on the peer, or the file is still a placeholder. Mint a new token on the peer and replace the file (`auth_ok` is 0 in OTel while `peer.up` is 1 for both checks) | | `true` | `true` | `unknown` | the peer predates `GET /api/v1/ping`; the check fell back to public `/healthz`, so liveness is proven but not our credential | | `true` | `false` | `unknown` | the port answers but the API does not (`tls_error` for a certificate problem, `timeout`, `http_5xx`) | | `false` | `false` | `unknown` | nothing answers on the port: the peer is down or unreachable (also the case when the name resolves to an address the peer does not listen on, see [Address families](#address-families)) | A peer that rejects our token is reported to *your* callers as a `503` naming the peer, never as a `401`: re-authenticating with this host would not help. ## Configuration ```toml [peers] enabled = false # default: no peer routes, tools, tasks or metrics tcp_interval_secs = 10 tcp_timeout_secs = 3 # must be below tcp_interval_secs api_interval_secs = 30 api_timeout_secs = 5 # must be below api_interval_secs fail_threshold = 2 # consecutive failures before a check flips down request_timeout_secs = 10 # one proxied request, connect to body max_response_bytes = 8388608 # buffered bodies above this are refused max_concurrent_per_peer = 4 # proxied requests in flight per peer max_concurrent_total = 16 # ... across all peers deny_cidrs = ["169.254.0.0/16", "0.0.0.0/8"] # resolved peer addresses in these ranges are refused [[peers.peer]] name = "minipc1" # ^[a-z0-9][a-z0-9_-]{0,31}$, unique, at most 64 url = "https://minipc1:1984" # http or https, no userinfo, query or fragment token_file = "/run/credentials/skyg-linux-tracker.service/peer-minipc1" # absolute ca_file = "/etc/skyg/private-ca.pem" # optional extra trust anchor allow_plain_http = false # required true for http:// to a non-loopback host enabled = true # false keeps the declaration without contacting the peer ``` Scalar keys accept environment overrides (`SKYG_PEERS__ENABLED`, `SKYG_PEERS__API_INTERVAL_SECS`, ...); `[[peers.peer]]` entries are file-only. `slinuxm --print-config` prints `token_file` paths, never their contents. Validation refuses: a timeout that is not below its interval, more than 64 peers, duplicate names, a bad name, a relative `token_file` or `ca_file`, a non-http(s) URL, userinfo/query/fragment in a URL, an IP literal in `deny_cidrs`, and plain `http` to a non-loopback host without `allow_plain_http = true` (which also logs a warning: the bearer token travels in cleartext). `enabled = true` with no peers logs a warning. `deny_cidrs` is checked against the **resolved** address at connect time, so a name that resolves into a denied range is refused as `unreachable`. Private LAN ranges are deliberately allowed; the default denies only link-local/metadata and the unspecified range. ## REST and MCP All under the inbound auth of this host; nothing is mounted while peers are off. | Route | Behaviour | |---|---| | `GET /api/v1/peers` | `{"peers": [...]}`, sorted by name, disabled peers included | | `GET /api/v1/peers/{peer}` | one status object; unknown peer is `404` | | `GET /api/v1/peers/{peer}/...` | GET-only pass-through of an allowlisted route | The status object is `{name, enabled, url, server{up, since, last_checked, last_error, latency_ms}, api{up, since, last_checked, last_error, latency_ms, auth, peer_version}}`, times in RFC 3339 UTC. The pass-through allowlist is `systemd/...`, `journal`, `journal/follow`, `docker/...`, with the query parameters the local routes accept (`state`, `unit`, `priority`, `lines`, `since`, `all`, `tail`, `follow`). A caller may include the `/api/v1` prefix; it is stripped. Anything else is a `400`, `..` and encoded slashes are rejected, `ping`, `peers/...` and `/healthz` are not reachable through it, and methods other than GET are `405`. The peer's 2xx answer comes back unchanged (status, `Content-Type`, body; SSE stays a stream) with `x-slinuxm-peer: `; every error produced for a peer carries that header too, except the `405` for a non-GET method, which the router answers before the peer route runs. Consolidated failures use the existing error shape: | Peer outcome | Answer to your caller | |---|---| | peer `404` | `404 not_found`, `peer : not found on peer` | | peer `400`/`422` | `400 bad_request` | | peer rejects our token (`401`/`403`) | `503 unavailable`, `peer : peer rejected credentials` | | timeout, refused, DNS, TLS, `5xx`, oversize, saturated, token file | `503 unavailable` with the class | | malformed peer JSON | `500 internal` | | unknown peer | `404 not_found` | No caller header is forwarded except `traceparent`; the Authorization header is always the peer's own token from `token_file`. A caller that sends `x-slinuxm-hop` is refused with `400` (no chained consolidators). With peers enabled, MCP exposes `list_peers`, `get_peer` and the resource `peers://status` (same JSON as the list), every one of the eight existing tools gains an optional `peer` argument, and the `container_health_report` and `diagnose_unit` prompts gain a `peer` argument. Ten tools instead of eight, and only on the instances that need them. Peer-sourced text is labelled untrusted data, as local log text is. ## OpenTelemetry Four observable gauges, registered only with peers enabled and `otel.metrics.enabled`, read from the registry snapshot on each scrape: `peer.up` (attributes `peer.name`, `peer.check` = `server`/`api`), `peer.api.auth_ok` (`peer.name`), `peer.check.duration_seconds` (`peer.name`, `peer.check`) and `peer.last_change_timestamp_seconds` (`peer.name`, `peer.check`). No URL, version or token in attributes. A removed peer disappears from the next scrape. Outbound proxied requests carry a span named `peer.request` with `peer.name`, and the caller's `traceparent` is passed through, so a trace spans the fleet. ## Credentials and threat model DEC-11 stores *inbound* tokens only as sha256 hashes. A consolidator cannot do that: it has to replay the peer's plaintext bearer token. DEC-13 therefore makes **outbound** peer tokens externally managed files: - `slinuxm` owns no copy of a peer token: not in SQLite, not in `config.toml`, not in `--print-config`, the logs or a support bundle. The config holds a path; the process reads the file into a redacted `Secret` on every check and request and drops it. - On NixOS the file is an agenix secret decrypted at activation to `/run/agenix/...` (root-only) and handed to the service through systemd `LoadCredential`, so the service user never needs read access to the agenix output. Outside NixOS, put the file somewhere only the service user can read: a group- or world-readable token file is refused (`token_file` class) unless it sits in `$CREDENTIALS_DIRECTORY`. - Rotation is a file replacement; the next check reads the new value. On NixOS also `systemctl reload` (SIGHUP) so systemd re-copies the credential (`RefreshOnReload=credentials`). - Use **one token per consolidator on every peer** (`consolidator-`, read-only by nature since every token is read-only): `slinuxm token revoke consolidator-` on a peer cuts exactly that consolidator off, and nothing else. What it does not protect: root on the consolidator (it can read the decrypted file), and the consolidator's own token as an asset — it reaches every peer's data, which is why tokens are per consolidator, revocable, and why scoping tokens per peer is a follow-up, not a v1 feature. Peers are only ever declared by whoever manages the config (root or Nix). No route, parameter or MCP argument supplies a URL, so the consolidator cannot be steered at an arbitrary target. Redirects from a peer are never followed (`503 http_3xx`), TLS is verified against the webpki roots plus the optional `ca_file`, and there is no skip-verify switch. ## NixOS with agenix The module renders the peers and the credentials; you only declare them: ```nix services.skyg-linux-tracker = { enable = true; bind = "0.0.0.0"; openFirewall = true; # or firewallInterfaces = [ "enp3s0" ]; peers = { enable = true; settings = { api_interval_secs = 30; fail_threshold = 2; }; # [peers] tuning, snake_case peers = { minipc1 = { url = "https://minipc1:1984"; tokenFile = config.age.secrets.peer-minipc1.path; }; nas = { url = "http://nas:1984"; tokenFile = config.age.secrets.peer-nas.path; allowPlainHttp = true; # the token then travels in cleartext in your LAN }; }; }; }; ``` A worked agenix setup for host `monitor`, which consolidates `minipc1` (HTTPS with a private CA) and `nas` (plain HTTP in a trusted LAN). On each peer, mint a token for this consolidator. It has to be a dedicated token: revoking it cuts off exactly this consolidator. ```sh # on minipc1 and on nas sudo -u skyg-linux-tracker slinuxm token create consolidator-monitor ``` Register the encrypted files next to your other agenix secrets and encrypt them for the consolidator host. Only the ciphertext goes into git. ```nix # secrets/secrets.nix (agenix) { # The token minipc1 issued to `monitor`. Only `monitor` needs to decrypt it: # the consolidator replays it, no other host reads it. "peer-minipc1.age".publicKeys = [ monitor ]; "peer-nas.age".publicKeys = [ monitor ]; } ``` ```sh agenix -e secrets/peer-minipc1.age # paste the token printed on minipc1 agenix -e secrets/peer-nas.age # paste the token printed on nas ``` ```nix { config, inputs, ... }: { imports = [ inputs.agenix.nixosModules.default ]; # Decrypted at activation to /run/agenix/peer-, root-only. The module # hands the file to the service through LoadCredential, so the # `skyg-linux-tracker` user never reads it directly. age.secrets.peer-minipc1 = { file = ./secrets/peer-minipc1.age; owner = "root"; mode = "0400"; }; age.secrets.peer-nas = { file = ./secrets/peer-nas.age; owner = "root"; mode = "0400"; }; services.skyg-linux-tracker = { enable = true; bind = "0.0.0.0"; openFirewall = true; peers = { enable = true; peers = { minipc1 = { url = "https://minipc1:1984"; tokenFile = config.age.secrets.peer-minipc1.path; # a string path, not a path literal caFile = "/etc/ssl/private-ca.pem"; # readable by the service user }; nas = { url = "http://nas:1984"; tokenFile = config.age.secrets.peer-nas.path; allowPlainHttp = true; }; }; }; }; } ``` The module turns that into one `[[peers.peer]]` entry per enabled peer, with `token_file = /run/credentials/skyg-linux-tracker.service/peer-`, and one `LoadCredential=peer-:` line. Consequences worth knowing: - `tokenFile` must be a **string** (`config.age.secrets...path`), not a path literal: a `nix` path or `"${./secret}"` interpolates a store path and is rejected at evaluation. Such a secret has already been copied into the local store by then, so rotate it. - `caFile` is **not** passed through `LoadCredential`: the service user reads it directly, so it must be readable by `skyg-linux-tracker`. A root-only file such as the agenix output makes every check and proxied request to that peer fail as `tls_error`. Store paths are fine for a public CA. - Peers with `enable = false` are rendered nowhere, so no credential is shipped for them. - `peers.settings` is freeform `[peers]` tuning; it must not contain `enabled`, `peer` or a token. ### Reload or restart `switch-to-configuration` reloads (SIGHUP) when only the config file changed, and restarts when the unit file changed. SIGHUP re-reads `[peers]`, so editing a peer's `url`, `caFile` or the intervals applies without a restart, and systemd re-copies the credentials (`RefreshOnReload=credentials`), so a rotated token is picked up as well. Adding, removing, enabling or disabling a peer, or changing its `tokenFile`, changes the `LoadCredential=` lines and therefore **restarts** the service — the safe outcome. `enabled`, `deny_cidrs`, `max_response_bytes` and the two concurrency limits also need a restart (the routes are mounted once and the limits live in the transport), and the module's restart trigger makes a `switch` do that instead of silently keeping the old values. Outside NixOS the same split applies by hand: SIGHUP re-reads the TLS pair and `[peers]` (peer entries, intervals, timeouts, `fail_threshold`); everything else needs a restart. ## Firewall and networking Every **peer** must accept the consolidator's address on its port; the consolidator only needs outbound access. On a peer's NixOS host that is `openFirewall = true` or `firewallInterfaces = [ "enp3s0" ]`; a peer bound to `127.0.0.1` is not reachable from another machine, so a consolidator needs the peer to bind a LAN address (with auth on, which the module enforces) and either TLS or a trusted network. ### Address families A peer is reached at the address its name resolves to at connect time, and the first resolved address wins. A peer that listens on IPv4 only, with a name that also resolves to an IPv6 address (as a NixOS test network does), reads as `unreachable`; declare it by IP address or let it listen on both families (`bind = "::"` accepts IPv4-mapped connections too). DNS rebinding is not a risk here because the target is pinned to the address that was checked against `deny_cidrs`. ## Limits and non-goals - No aggregation: `peer = "*"` and fleet-wide views are a follow-up, because partial failures need a result shape of their own. Ask one peer at a time. - No chaining: a consolidator's request carries `x-slinuxm-hop`, so a peer that is itself a consolidator refuses it. This prevents loops and amplification. - No write path: proxying is GET-only and allowlisted, and nothing here starts, stops or changes a unit or a container. - No history: peer state is in memory; use OTel for trends. - No runtime management: peers are declarative, reloaded on SIGHUP. - Each peer exports its own telemetry; peers' OTel signals are not re-exported by the consolidator. ## Testing The peers acceptance test is a NixOS VM test. Three machines run the real `slinuxm`: `alpha` and `beta` consolidate each other over plain HTTP, and `alpha` also declares the TLS-only `gamma` with a private CA. It needs KVM and takes minutes. ```sh devenv shell -- dev-test vm peers # nix build .#tests.x86_64-linux.peers -L ``` Against a tree without the consolidator the test stops at the first peers assertion; that is expected, the test is written ahead of the feature. `dev/dev-test.nu` is the repository's test runner. `devenv shell -- dev-test --help` lists its subcommands and the command behind each. In an interactive `devenv shell`, `dev-test …` is that same script. In Nushell the runner is a module. Load it once, for example in `config.nu`, and `dev-test unit`, `dev-test vm peers`, ... complete natively. Integration-test name completion is cached in `target/skyg-test-filters.cache` at the repo root (not relative to the current directory, so completion works from any subdirectory), keyed on the newest mtime of the Rust sources and the Cargo manifests. A failing check raises a catchable error instead of exiting, so a red `dev-test unit` does not close the Nushell session that loaded the module; script mode still exits non-zero. ```nu use /home/you/src/skyg-linux-tracker/dev/dev-test.nu * # absolute path to dev/dev-test.nu ``` zsh and bash completion comes from the same command table. The generated script only registers completion for the external command `dev-test`. An interactive `devenv shell` sources the bash script. That hook runs in bash and direnv will not source it either, so zsh users add the following to `.zshrc` after `compinit`: ```sh source <(nu /home/you/src/skyg-linux-tracker/dev/dev-test.nu completions zsh) ``` bash, from the repo checkout: ```sh source <(nu dev/dev-test.nu completions bash) ``` `nu dev/dev-test.nu self-check` fails if the command table, the defined subcommands, the help text, the cargo argv, the filter cache or the shell completion drift apart.