Files
skyg-linux-tracker/docs/consolidator.md
T
ari dde6a12ce4 F: try every resolved peer address instead of the first
A peer whose name resolves to both families but which listens on only one
of them was reported unreachable, which the documented 0.0.0.0 binding
produces on a dual-stack LAN and in the NixOS test network.

resolve_target now returns the whole address set (still refusing the name
if any address is denied), the HTTP client pins it with
resolve_to_addrs so the connector falls through to the next address, and
tcp_check tries each address in order with a per-attempt share of the
remaining deadline. ClientKey keys on the set.

Regression cover: a client test that resolves localhost against a
single-family listener, a tests/peers.rs case declaring the peer as
http://localhost:<port>, and the VM test's TLS peer gamma now binds
0.0.0.0 (IPv4 only) while alpha and beta stay dual stack. Docs and the
contracts section 10 address note describe the new behaviour.
2026-10-20 15:04:09 -04:00

253 lines
18 KiB
Markdown

# Consolidator (peers)
One `slinuxm` can be a **consolidator** for other `slinuxm` instances. Each one it is told about is a *peer*: it health-checks the peer in two ways and passes read-only requests through to it, so one MCP endpoint or one REST base URL covers the fleet. The feature is off by default (`[peers] enabled = false`), and a peer-less instance behaves exactly as before: same routes, eight MCP tools, same metrics, no extra tasks or config.
Everything here is declarative: peers come from the config file, nothing writes to a peer, and no request parameter can supply a URL. Runtime peer management (`peer add`, a peers table) does not exist.
Tests: `tests/peers.rs` runs the whole path in-process (no VM), and a three-machine NixOS VM test covers boot, TLS, rotation and outages. Both go through the repository's runner, `devenv shell -- dev-test …`; see [Testing](#testing) below.
## Model
| Check | What it proves | How |
|---|---|---|
| `server` | The peer's port accepts a TCP connection | one connect per `tcp_interval_secs`, bounded by `tcp_timeout_secs` |
| `api` | The peer's REST API runs **and** accepts our token | authenticated `GET /api/v1/ping` every `api_interval_secs`, bounded by `api_timeout_secs` |
A check flips down after `fail_threshold` consecutive failures and up again on the first success. All state is in memory (`since`, `last_checked`, `last_error`, `latency_ms`), so a restart of the consolidator resets it; history belongs in your OTel backend.
`last_error` is a class, never peer text or a URL: `unreachable`, `timeout`, `dns`, `tls_error`, `http_<code>`, `unauthorized`, `oversize`, `saturated`, `token_file`.
### Reading `server` and `api`
| `server.up` | `api.up` | `api.auth` | Meaning |
|---|---|---|---|
| `true` | `true` | `ok` | healthy |
| `true` | `true` | `rejected` | the API runs, **our token is refused**: revoked, rotated on the peer, or the file is still a placeholder. Mint a new token on the peer and replace the file (`auth_ok` is 0 in OTel while `peer.up` is 1 for both checks) |
| `true` | `true` | `unknown` | the peer predates `GET /api/v1/ping`; the check fell back to public `/healthz`, so liveness is proven but not our credential |
| `true` | `false` | `unknown` | the port answers but the API does not (`tls_error` for a certificate problem, `timeout`, `http_5xx`) |
| `false` | `false` | `unknown` | no resolved address answers on the port: the peer is down or unreachable (see [Address families](#address-families)) |
A peer that rejects our token is reported to *your* callers as a `503` naming the peer, never as a `401`: re-authenticating with this host would not help.
## Configuration
```toml
[peers]
enabled = false # default: no peer routes, tools, tasks or metrics
tcp_interval_secs = 10
tcp_timeout_secs = 3 # must be below tcp_interval_secs
api_interval_secs = 30
api_timeout_secs = 5 # must be below api_interval_secs
fail_threshold = 2 # consecutive failures before a check flips down
request_timeout_secs = 10 # one proxied request, connect to body
max_response_bytes = 8388608 # buffered bodies above this are refused
max_concurrent_per_peer = 4 # proxied requests in flight per peer
max_concurrent_total = 16 # ... across all peers
deny_cidrs = ["169.254.0.0/16", "0.0.0.0/8"] # resolved peer addresses in these ranges are refused
[[peers.peer]]
name = "minipc1" # ^[a-z0-9][a-z0-9_-]{0,31}$, unique, at most 64
url = "https://minipc1:1984" # http or https, no userinfo, query or fragment
token_file = "/run/credentials/skyg-linux-tracker.service/peer-minipc1" # absolute
ca_file = "/etc/skyg/private-ca.pem" # optional extra trust anchor
allow_plain_http = false # required true for http:// to a non-loopback host
enabled = true # false keeps the declaration without contacting the peer
```
Scalar keys accept environment overrides (`SKYG_PEERS__ENABLED`, `SKYG_PEERS__API_INTERVAL_SECS`, ...); `[[peers.peer]]` entries are file-only. `slinuxm --print-config` prints `token_file` paths, never their contents. Validation refuses: a timeout that is not below its interval, more than 64 peers, duplicate names, a bad name, a relative `token_file` or `ca_file`, a non-http(s) URL, userinfo/query/fragment in a URL, an IP literal in `deny_cidrs`, and plain `http` to a non-loopback host without `allow_plain_http = true` (which also logs a warning: the bearer token travels in cleartext). `enabled = true` with no peers logs a warning.
`deny_cidrs` is checked against every **resolved** address at connect time, so a name where any address lands in a denied range is refused as `unreachable`. Private LAN ranges are deliberately allowed; the default denies only link-local/metadata and the unspecified range.
## REST and MCP
All under the inbound auth of this host; nothing is mounted while peers are off.
| Route | Behaviour |
|---|---|
| `GET /api/v1/peers` | `{"peers": [...]}`, sorted by name, disabled peers included |
| `GET /api/v1/peers/{peer}` | one status object; unknown peer is `404` |
| `GET /api/v1/peers/{peer}/...` | GET-only pass-through of an allowlisted route |
The status object is `{name, enabled, url, server{up, since, last_checked, last_error, latency_ms}, api{up, since, last_checked, last_error, latency_ms, auth, peer_version}}`, times in RFC 3339 UTC.
The pass-through allowlist is `systemd/...`, `journal`, `journal/follow`, `docker/...`, with the query parameters the local routes accept (`state`, `unit`, `priority`, `lines`, `since`, `all`, `tail`, `follow`). A caller may include the `/api/v1` prefix; it is stripped. Anything else is a `400`, `..` and encoded slashes are rejected, `ping`, `peers/...` and `/healthz` are not reachable through it, and methods other than GET are `405`. The peer's 2xx answer comes back unchanged (status, `Content-Type`, body; SSE stays a stream) with `x-slinuxm-peer: <peer>`; every error produced for a peer carries that header too, except the `405` for a non-GET method, which the router answers before the peer route runs. Consolidated failures use the existing error shape:
| Peer outcome | Answer to your caller |
|---|---|
| peer `404` | `404 not_found`, `peer <name>: not found on peer` |
| peer `400`/`422` | `400 bad_request` |
| peer rejects our token (`401`/`403`) | `503 unavailable`, `peer <name>: peer rejected credentials` |
| timeout, refused, DNS, TLS, `5xx`, oversize, saturated, token file | `503 unavailable` with the class |
| malformed peer JSON | `500 internal` |
| unknown peer | `404 not_found` |
No caller header is forwarded except `traceparent`; the Authorization header is always the peer's own token from `token_file`. A caller that sends `x-slinuxm-hop` is refused with `400` (no chained consolidators).
With peers enabled, MCP exposes `list_peers`, `get_peer` and the resource `peers://status` (same JSON as the list), every one of the eight existing tools gains an optional `peer` argument, and the `container_health_report` and `diagnose_unit` prompts gain a `peer` argument. Ten tools instead of eight, and only on the instances that need them. Peer-sourced text is labelled untrusted data, as local log text is.
## OpenTelemetry
Four observable gauges, registered only with peers enabled and `otel.metrics.enabled`, read from the registry snapshot on each scrape: `peer.up` (attributes `peer.name`, `peer.check` = `server`/`api`), `peer.api.auth_ok` (`peer.name`), `peer.check.duration_seconds` (`peer.name`, `peer.check`) and `peer.last_change_timestamp_seconds` (`peer.name`, `peer.check`). No URL, version or token in attributes. A removed peer disappears from the next scrape. Outbound proxied requests carry a span named `peer.request` with `peer.name`, and the caller's `traceparent` is passed through, so a trace spans the fleet.
## Credentials and threat model
DEC-11 stores *inbound* tokens only as sha256 hashes. A consolidator cannot do that: it has to replay the peer's plaintext bearer token. DEC-13 therefore makes **outbound** peer tokens externally managed files:
- `slinuxm` owns no copy of a peer token: not in SQLite, not in `config.toml`, not in `--print-config`, the logs or a support bundle. The config holds a path; the process reads the file into a redacted `Secret` on every check and request and drops it.
- On NixOS the file is an agenix secret decrypted at activation to `/run/agenix/...` (root-only) and handed to the service through systemd `LoadCredential`, so the service user never needs read access to the agenix output. Outside NixOS, put the file somewhere only the service user can read: a group- or world-readable token file is refused (`token_file` class) unless it sits in `$CREDENTIALS_DIRECTORY`.
- Rotation is a file replacement; the next check reads the new value. On NixOS also `systemctl reload` (SIGHUP) so systemd re-copies the credential (`RefreshOnReload=credentials`).
- Use **one token per consolidator on every peer** (`consolidator-<host>`, read-only by nature since every token is read-only): `slinuxm token revoke consolidator-<host>` on a peer cuts exactly that consolidator off, and nothing else.
What it does not protect: root on the consolidator (it can read the decrypted file), and the consolidator's own token as an asset — it reaches every peer's data, which is why tokens are per consolidator, revocable, and why scoping tokens per peer is a follow-up, not a v1 feature.
Peers are only ever declared by whoever manages the config (root or Nix). No route, parameter or MCP argument supplies a URL, so the consolidator cannot be steered at an arbitrary target. Redirects from a peer are never followed (`503 http_3xx`), TLS is verified against the webpki roots plus the optional `ca_file`, and there is no skip-verify switch.
## NixOS with agenix
The module renders the peers and the credentials; you only declare them:
```nix
services.skyg-linux-tracker = {
enable = true;
bind = "0.0.0.0";
openFirewall = true; # or firewallInterfaces = [ "enp3s0" ];
peers = {
enable = true;
settings = { api_interval_secs = 30; fail_threshold = 2; }; # [peers] tuning, snake_case
peers = {
minipc1 = { url = "https://minipc1:1984"; tokenFile = config.age.secrets.peer-minipc1.path; };
nas = {
url = "http://nas:1984";
tokenFile = config.age.secrets.peer-nas.path;
allowPlainHttp = true; # the token then travels in cleartext in your LAN
};
};
};
};
```
A worked agenix setup for host `monitor`, which consolidates `minipc1` (HTTPS with a private CA) and `nas` (plain HTTP in a trusted LAN).
On each peer, mint a token for this consolidator. It has to be a dedicated token: revoking it cuts off exactly this consolidator.
```sh
# on minipc1 and on nas
sudo -u skyg-linux-tracker slinuxm token create consolidator-monitor
```
Register the encrypted files next to your other agenix secrets and encrypt them for the consolidator host. Only the ciphertext goes into git.
```nix
# secrets/secrets.nix (agenix)
{
# The token minipc1 issued to `monitor`. Only `monitor` needs to decrypt it:
# the consolidator replays it, no other host reads it.
"peer-minipc1.age".publicKeys = [ monitor ];
"peer-nas.age".publicKeys = [ monitor ];
}
```
```sh
agenix -e secrets/peer-minipc1.age # paste the token printed on minipc1
agenix -e secrets/peer-nas.age # paste the token printed on nas
```
```nix
{ config, inputs, ... }: {
imports = [ inputs.agenix.nixosModules.default ];
# Decrypted at activation to /run/agenix/peer-<name>, root-only. The module
# hands the file to the service through LoadCredential, so the
# `skyg-linux-tracker` user never reads it directly.
age.secrets.peer-minipc1 = {
file = ./secrets/peer-minipc1.age;
owner = "root";
mode = "0400";
};
age.secrets.peer-nas = { file = ./secrets/peer-nas.age; owner = "root"; mode = "0400"; };
services.skyg-linux-tracker = {
enable = true;
bind = "0.0.0.0";
openFirewall = true;
peers = {
enable = true;
peers = {
minipc1 = {
url = "https://minipc1:1984";
tokenFile = config.age.secrets.peer-minipc1.path; # a string path, not a path literal
caFile = "/etc/ssl/private-ca.pem"; # readable by the service user
};
nas = {
url = "http://nas:1984";
tokenFile = config.age.secrets.peer-nas.path;
allowPlainHttp = true;
};
};
};
};
}
```
The module turns that into one `[[peers.peer]]` entry per enabled peer, with `token_file = /run/credentials/skyg-linux-tracker.service/peer-<name>`, and one `LoadCredential=peer-<name>:<tokenFile>` line. Consequences worth knowing:
- `tokenFile` must be a **string** (`config.age.secrets...path`), not a path literal: a `nix` path or `"${./secret}"` interpolates a store path and is rejected at evaluation. Such a secret has already been copied into the local store by then, so rotate it.
- `caFile` is **not** passed through `LoadCredential`: the service user reads it directly, so it must be readable by `skyg-linux-tracker`. A root-only file such as the agenix output makes every check and proxied request to that peer fail as `tls_error`. Store paths are fine for a public CA.
- Peers with `enable = false` are rendered nowhere, so no credential is shipped for them.
- `peers.settings` is freeform `[peers]` tuning; it must not contain `enabled`, `peer` or a token.
### Reload or restart
`switch-to-configuration` reloads (SIGHUP) when only the config file changed, and restarts when the unit file changed. SIGHUP re-reads `[peers]`, so editing a peer's `url`, `caFile` or the intervals applies without a restart, and systemd re-copies the credentials (`RefreshOnReload=credentials`), so a rotated token is picked up as well. Adding, removing, enabling or disabling a peer, or changing its `tokenFile`, changes the `LoadCredential=` lines and therefore **restarts** the service — the safe outcome. `enabled`, `deny_cidrs`, `max_response_bytes` and the two concurrency limits also need a restart (the routes are mounted once and the limits live in the transport), and the module's restart trigger makes a `switch` do that instead of silently keeping the old values.
Outside NixOS the same split applies by hand: SIGHUP re-reads the TLS pair and `[peers]` (peer entries, intervals, timeouts, `fail_threshold`); everything else needs a restart.
## Firewall and networking
Every **peer** must accept the consolidator's address on its port; the consolidator only needs outbound access. On a peer's NixOS host that is `openFirewall = true` or `firewallInterfaces = [ "enp3s0" ]`; a peer bound to `127.0.0.1` is not reachable from another machine, so a consolidator needs the peer to bind a LAN address (with auth on, which the module enforces) and either TLS or a trusted network.
### Address families
A peer's name is resolved at connect time and **every** resolved address is used: the health probe tries them in order within its timeout, and the HTTP client pins the whole list so the connection falls through to the next address when the first one refuses or does not answer. A peer that listens on IPv4 only is therefore reached even when its name also resolves to an IPv6 address (as a NixOS test network or a dual-stack LAN does). Both families and a single family work; `bind = "::"` on a peer is dual stack, `bind = "0.0.0.0"` is IPv4 only.
DNS rebinding is still not a risk: the set is resolved once per request and pinned, and a name where **any** resolved address falls in `deny_cidrs` is refused as `unreachable` rather than dialled at the permitted addresses.
## Limits and non-goals
- No aggregation: `peer = "*"` and fleet-wide views are a follow-up, because partial failures need a result shape of their own. Ask one peer at a time.
- No chaining: a consolidator's request carries `x-slinuxm-hop`, so a peer that is itself a consolidator refuses it. This prevents loops and amplification.
- No write path: proxying is GET-only and allowlisted, and nothing here starts, stops or changes a unit or a container.
- No history: peer state is in memory; use OTel for trends.
- No runtime management: peers are declarative, reloaded on SIGHUP.
- Each peer exports its own telemetry; peers' OTel signals are not re-exported by the consolidator.
## Testing
The peers acceptance test is a NixOS VM test. Three machines run the real `slinuxm`: `alpha` and `beta` consolidate each other over plain HTTP, and `alpha` also declares the TLS-only `gamma` with a private CA. It needs KVM and takes minutes.
```sh
devenv shell -- dev-test vm peers # nix build .#tests.x86_64-linux.peers -L
```
Against a tree without the consolidator the test stops at the first peers assertion; that is expected, the test is written ahead of the feature. `dev/dev-test.nu` is the repository's test runner. `devenv shell -- dev-test --help` lists its subcommands and the command behind each. In an interactive `devenv shell`, `dev-test …` is that same script.
In Nushell the runner is a module. Load it once, for example in `config.nu`, and `dev-test unit`, `dev-test vm peers`, ... complete natively. Integration-test name completion is cached in `target/skyg-test-filters.cache` at the repo root (not relative to the current directory, so completion works from any subdirectory), keyed on the newest mtime of the Rust sources and the Cargo manifests. A failing check raises a catchable error instead of exiting, so a red `dev-test unit` does not close the Nushell session that loaded the module; script mode still exits non-zero.
```nu
use /home/you/src/skyg-linux-tracker/dev/dev-test.nu * # absolute path to dev/dev-test.nu
```
zsh and bash completion comes from the same command table. The generated script only registers completion for the external command `dev-test`. An interactive `devenv shell` sources the bash script. That hook runs in bash and direnv will not source it either, so zsh users add the following to `.zshrc` after `compinit`:
```sh
source <(nu /home/you/src/skyg-linux-tracker/dev/dev-test.nu completions zsh)
```
bash, from the repo checkout:
```sh
source <(nu dev/dev-test.nu completions bash)
```
`nu dev/dev-test.nu self-check` fails if the command table, the defined subcommands, the help text, the cargo argv, the filter cache or the shell completion drift apart.