README peers section, docs/consolidator.md (threat model, tokens, firewall, worked agenix example, reload/restart), the MCP tool count in both tables, and config.example.toml commentary.
251 lines
18 KiB
Markdown
251 lines
18 KiB
Markdown
# Consolidator (peers)
|
|
|
|
One `slinuxm` can be a **consolidator** for other `slinuxm` instances. Each one it is told about is a *peer*: it health-checks the peer in two ways and passes read-only requests through to it, so one MCP endpoint or one REST base URL covers the fleet. The feature is off by default (`[peers] enabled = false`), and a peer-less instance behaves exactly as before: same routes, eight MCP tools, same metrics, no extra tasks or config.
|
|
|
|
Everything here is declarative: peers come from the config file, nothing writes to a peer, and no request parameter can supply a URL. Runtime peer management (`peer add`, a peers table) does not exist.
|
|
|
|
Tests: `tests/peers.rs` runs the whole path in-process (no VM), and a three-machine NixOS VM test covers boot, TLS, rotation and outages. Both go through the repository's runner, `devenv shell -- dev-test …`; see [Testing](#testing) below.
|
|
|
|
## Model
|
|
|
|
| Check | What it proves | How |
|
|
|---|---|---|
|
|
| `server` | The peer's port accepts a TCP connection | one connect per `tcp_interval_secs`, bounded by `tcp_timeout_secs` |
|
|
| `api` | The peer's REST API runs **and** accepts our token | authenticated `GET /api/v1/ping` every `api_interval_secs`, bounded by `api_timeout_secs` |
|
|
|
|
A check flips down after `fail_threshold` consecutive failures and up again on the first success. All state is in memory (`since`, `last_checked`, `last_error`, `latency_ms`), so a restart of the consolidator resets it; history belongs in your OTel backend.
|
|
|
|
`last_error` is a class, never peer text or a URL: `unreachable`, `timeout`, `dns`, `tls_error`, `http_<code>`, `unauthorized`, `oversize`, `saturated`, `token_file`.
|
|
|
|
### Reading `server` and `api`
|
|
|
|
| `server.up` | `api.up` | `api.auth` | Meaning |
|
|
|---|---|---|---|
|
|
| `true` | `true` | `ok` | healthy |
|
|
| `true` | `true` | `rejected` | the API runs, **our token is refused**: revoked, rotated on the peer, or the file is still a placeholder. Mint a new token on the peer and replace the file (`auth_ok` is 0 in OTel while `peer.up` is 1 for both checks) |
|
|
| `true` | `true` | `unknown` | the peer predates `GET /api/v1/ping`; the check fell back to public `/healthz`, so liveness is proven but not our credential |
|
|
| `true` | `false` | `unknown` | the port answers but the API does not (`tls_error` for a certificate problem, `timeout`, `http_5xx`) |
|
|
| `false` | `false` | `unknown` | nothing answers on the port: the peer is down or unreachable (also the case when the name resolves to an address the peer does not listen on, see [Address families](#address-families)) |
|
|
|
|
A peer that rejects our token is reported to *your* callers as a `503` naming the peer, never as a `401`: re-authenticating with this host would not help.
|
|
|
|
## Configuration
|
|
|
|
```toml
|
|
[peers]
|
|
enabled = false # default: no peer routes, tools, tasks or metrics
|
|
tcp_interval_secs = 10
|
|
tcp_timeout_secs = 3 # must be below tcp_interval_secs
|
|
api_interval_secs = 30
|
|
api_timeout_secs = 5 # must be below api_interval_secs
|
|
fail_threshold = 2 # consecutive failures before a check flips down
|
|
request_timeout_secs = 10 # one proxied request, connect to body
|
|
max_response_bytes = 8388608 # buffered bodies above this are refused
|
|
max_concurrent_per_peer = 4 # proxied requests in flight per peer
|
|
max_concurrent_total = 16 # ... across all peers
|
|
deny_cidrs = ["169.254.0.0/16", "0.0.0.0/8"] # resolved peer addresses in these ranges are refused
|
|
|
|
[[peers.peer]]
|
|
name = "minipc1" # ^[a-z0-9][a-z0-9_-]{0,31}$, unique, at most 64
|
|
url = "https://minipc1:1984" # http or https, no userinfo, query or fragment
|
|
token_file = "/run/credentials/skyg-linux-tracker.service/peer-minipc1" # absolute
|
|
ca_file = "/etc/skyg/private-ca.pem" # optional extra trust anchor
|
|
allow_plain_http = false # required true for http:// to a non-loopback host
|
|
enabled = true # false keeps the declaration without contacting the peer
|
|
```
|
|
|
|
Scalar keys accept environment overrides (`SKYG_PEERS__ENABLED`, `SKYG_PEERS__API_INTERVAL_SECS`, ...); `[[peers.peer]]` entries are file-only. `slinuxm --print-config` prints `token_file` paths, never their contents. Validation refuses: a timeout that is not below its interval, more than 64 peers, duplicate names, a bad name, a relative `token_file` or `ca_file`, a non-http(s) URL, userinfo/query/fragment in a URL, an IP literal in `deny_cidrs`, and plain `http` to a non-loopback host without `allow_plain_http = true` (which also logs a warning: the bearer token travels in cleartext). `enabled = true` with no peers logs a warning.
|
|
|
|
`deny_cidrs` is checked against the **resolved** address at connect time, so a name that resolves into a denied range is refused as `unreachable`. Private LAN ranges are deliberately allowed; the default denies only link-local/metadata and the unspecified range.
|
|
|
|
## REST and MCP
|
|
|
|
All under the inbound auth of this host; nothing is mounted while peers are off.
|
|
|
|
| Route | Behaviour |
|
|
|---|---|
|
|
| `GET /api/v1/peers` | `{"peers": [...]}`, sorted by name, disabled peers included |
|
|
| `GET /api/v1/peers/{peer}` | one status object; unknown peer is `404` |
|
|
| `GET /api/v1/peers/{peer}/...` | GET-only pass-through of an allowlisted route |
|
|
|
|
The status object is `{name, enabled, url, server{up, since, last_checked, last_error, latency_ms}, api{up, since, last_checked, last_error, latency_ms, auth, peer_version}}`, times in RFC 3339 UTC.
|
|
|
|
The pass-through allowlist is `systemd/...`, `journal`, `journal/follow`, `docker/...`, with the query parameters the local routes accept (`state`, `unit`, `priority`, `lines`, `since`, `all`, `tail`, `follow`). A caller may include the `/api/v1` prefix; it is stripped. Anything else is a `400`, `..` and encoded slashes are rejected, `ping`, `peers/...` and `/healthz` are not reachable through it, and methods other than GET are `405`. The peer's 2xx answer comes back unchanged (status, `Content-Type`, body; SSE stays a stream) with `x-slinuxm-peer: <peer>`; every error produced for a peer carries that header too, except the `405` for a non-GET method, which the router answers before the peer route runs. Consolidated failures use the existing error shape:
|
|
|
|
| Peer outcome | Answer to your caller |
|
|
|---|---|
|
|
| peer `404` | `404 not_found`, `peer <name>: not found on peer` |
|
|
| peer `400`/`422` | `400 bad_request` |
|
|
| peer rejects our token (`401`/`403`) | `503 unavailable`, `peer <name>: peer rejected credentials` |
|
|
| timeout, refused, DNS, TLS, `5xx`, oversize, saturated, token file | `503 unavailable` with the class |
|
|
| malformed peer JSON | `500 internal` |
|
|
| unknown peer | `404 not_found` |
|
|
|
|
No caller header is forwarded except `traceparent`; the Authorization header is always the peer's own token from `token_file`. A caller that sends `x-slinuxm-hop` is refused with `400` (no chained consolidators).
|
|
|
|
With peers enabled, MCP exposes `list_peers`, `get_peer` and the resource `peers://status` (same JSON as the list), every one of the eight existing tools gains an optional `peer` argument, and the `container_health_report` and `diagnose_unit` prompts gain a `peer` argument. Ten tools instead of eight, and only on the instances that need them. Peer-sourced text is labelled untrusted data, as local log text is.
|
|
|
|
## OpenTelemetry
|
|
|
|
Four observable gauges, registered only with peers enabled and `otel.metrics.enabled`, read from the registry snapshot on each scrape: `peer.up` (attributes `peer.name`, `peer.check` = `server`/`api`), `peer.api.auth_ok` (`peer.name`), `peer.check.duration_seconds` (`peer.name`, `peer.check`) and `peer.last_change_timestamp_seconds` (`peer.name`, `peer.check`). No URL, version or token in attributes. A removed peer disappears from the next scrape. Outbound proxied requests carry a span named `peer.request` with `peer.name`, and the caller's `traceparent` is passed through, so a trace spans the fleet.
|
|
|
|
## Credentials and threat model
|
|
|
|
DEC-11 stores *inbound* tokens only as sha256 hashes. A consolidator cannot do that: it has to replay the peer's plaintext bearer token. DEC-13 therefore makes **outbound** peer tokens externally managed files:
|
|
|
|
- `slinuxm` owns no copy of a peer token: not in SQLite, not in `config.toml`, not in `--print-config`, the logs or a support bundle. The config holds a path; the process reads the file into a redacted `Secret` on every check and request and drops it.
|
|
- On NixOS the file is an agenix secret decrypted at activation to `/run/agenix/...` (root-only) and handed to the service through systemd `LoadCredential`, so the service user never needs read access to the agenix output. Outside NixOS, put the file somewhere only the service user can read: a group- or world-readable token file is refused (`token_file` class) unless it sits in `$CREDENTIALS_DIRECTORY`.
|
|
- Rotation is a file replacement; the next check reads the new value. On NixOS also `systemctl reload` (SIGHUP) so systemd re-copies the credential (`RefreshOnReload=credentials`).
|
|
- Use **one token per consolidator on every peer** (`consolidator-<host>`, read-only by nature since every token is read-only): `slinuxm token revoke consolidator-<host>` on a peer cuts exactly that consolidator off, and nothing else.
|
|
|
|
What it does not protect: root on the consolidator (it can read the decrypted file), and the consolidator's own token as an asset — it reaches every peer's data, which is why tokens are per consolidator, revocable, and why scoping tokens per peer is a follow-up, not a v1 feature.
|
|
|
|
Peers are only ever declared by whoever manages the config (root or Nix). No route, parameter or MCP argument supplies a URL, so the consolidator cannot be steered at an arbitrary target. Redirects from a peer are never followed (`503 http_3xx`), TLS is verified against the webpki roots plus the optional `ca_file`, and there is no skip-verify switch.
|
|
|
|
## NixOS with agenix
|
|
|
|
The module renders the peers and the credentials; you only declare them:
|
|
|
|
```nix
|
|
services.skyg-linux-tracker = {
|
|
enable = true;
|
|
bind = "0.0.0.0";
|
|
openFirewall = true; # or firewallInterfaces = [ "enp3s0" ];
|
|
peers = {
|
|
enable = true;
|
|
settings = { api_interval_secs = 30; fail_threshold = 2; }; # [peers] tuning, snake_case
|
|
peers = {
|
|
minipc1 = { url = "https://minipc1:1984"; tokenFile = config.age.secrets.peer-minipc1.path; };
|
|
nas = {
|
|
url = "http://nas:1984";
|
|
tokenFile = config.age.secrets.peer-nas.path;
|
|
allowPlainHttp = true; # the token then travels in cleartext in your LAN
|
|
};
|
|
};
|
|
};
|
|
};
|
|
```
|
|
|
|
A worked agenix setup for host `monitor`, which consolidates `minipc1` (HTTPS with a private CA) and `nas` (plain HTTP in a trusted LAN).
|
|
|
|
On each peer, mint a token for this consolidator. It has to be a dedicated token: revoking it cuts off exactly this consolidator.
|
|
|
|
```sh
|
|
# on minipc1 and on nas
|
|
sudo -u skyg-linux-tracker slinuxm token create consolidator-monitor
|
|
```
|
|
|
|
Register the encrypted files next to your other agenix secrets and encrypt them for the consolidator host. Only the ciphertext goes into git.
|
|
|
|
```nix
|
|
# secrets/secrets.nix (agenix)
|
|
{
|
|
# The token minipc1 issued to `monitor`. Only `monitor` needs to decrypt it:
|
|
# the consolidator replays it, no other host reads it.
|
|
"peer-minipc1.age".publicKeys = [ monitor ];
|
|
"peer-nas.age".publicKeys = [ monitor ];
|
|
}
|
|
```
|
|
|
|
```sh
|
|
agenix -e secrets/peer-minipc1.age # paste the token printed on minipc1
|
|
agenix -e secrets/peer-nas.age # paste the token printed on nas
|
|
```
|
|
|
|
```nix
|
|
{ config, inputs, ... }: {
|
|
imports = [ inputs.agenix.nixosModules.default ];
|
|
|
|
# Decrypted at activation to /run/agenix/peer-<name>, root-only. The module
|
|
# hands the file to the service through LoadCredential, so the
|
|
# `skyg-linux-tracker` user never reads it directly.
|
|
age.secrets.peer-minipc1 = {
|
|
file = ./secrets/peer-minipc1.age;
|
|
owner = "root";
|
|
mode = "0400";
|
|
};
|
|
age.secrets.peer-nas = { file = ./secrets/peer-nas.age; owner = "root"; mode = "0400"; };
|
|
|
|
services.skyg-linux-tracker = {
|
|
enable = true;
|
|
bind = "0.0.0.0";
|
|
openFirewall = true;
|
|
peers = {
|
|
enable = true;
|
|
peers = {
|
|
minipc1 = {
|
|
url = "https://minipc1:1984";
|
|
tokenFile = config.age.secrets.peer-minipc1.path; # a string path, not a path literal
|
|
caFile = "/etc/ssl/private-ca.pem"; # readable by the service user
|
|
};
|
|
nas = {
|
|
url = "http://nas:1984";
|
|
tokenFile = config.age.secrets.peer-nas.path;
|
|
allowPlainHttp = true;
|
|
};
|
|
};
|
|
};
|
|
};
|
|
}
|
|
```
|
|
|
|
The module turns that into one `[[peers.peer]]` entry per enabled peer, with `token_file = /run/credentials/skyg-linux-tracker.service/peer-<name>`, and one `LoadCredential=peer-<name>:<tokenFile>` line. Consequences worth knowing:
|
|
|
|
- `tokenFile` must be a **string** (`config.age.secrets...path`), not a path literal: a `nix` path or `"${./secret}"` interpolates a store path and is rejected at evaluation. Such a secret has already been copied into the local store by then, so rotate it.
|
|
- `caFile` is **not** passed through `LoadCredential`: the service user reads it directly, so it must be readable by `skyg-linux-tracker`. A root-only file such as the agenix output makes every check and proxied request to that peer fail as `tls_error`. Store paths are fine for a public CA.
|
|
- Peers with `enable = false` are rendered nowhere, so no credential is shipped for them.
|
|
- `peers.settings` is freeform `[peers]` tuning; it must not contain `enabled`, `peer` or a token.
|
|
|
|
### Reload or restart
|
|
|
|
`switch-to-configuration` reloads (SIGHUP) when only the config file changed, and restarts when the unit file changed. SIGHUP re-reads `[peers]`, so editing a peer's `url`, `caFile` or the intervals applies without a restart, and systemd re-copies the credentials (`RefreshOnReload=credentials`), so a rotated token is picked up as well. Adding, removing, enabling or disabling a peer, or changing its `tokenFile`, changes the `LoadCredential=` lines and therefore **restarts** the service — the safe outcome. `enabled`, `deny_cidrs`, `max_response_bytes` and the two concurrency limits also need a restart (the routes are mounted once and the limits live in the transport), and the module's restart trigger makes a `switch` do that instead of silently keeping the old values.
|
|
|
|
Outside NixOS the same split applies by hand: SIGHUP re-reads the TLS pair and `[peers]` (peer entries, intervals, timeouts, `fail_threshold`); everything else needs a restart.
|
|
|
|
## Firewall and networking
|
|
|
|
Every **peer** must accept the consolidator's address on its port; the consolidator only needs outbound access. On a peer's NixOS host that is `openFirewall = true` or `firewallInterfaces = [ "enp3s0" ]`; a peer bound to `127.0.0.1` is not reachable from another machine, so a consolidator needs the peer to bind a LAN address (with auth on, which the module enforces) and either TLS or a trusted network.
|
|
|
|
### Address families
|
|
|
|
A peer is reached at the address its name resolves to at connect time, and the first resolved address wins. A peer that listens on IPv4 only, with a name that also resolves to an IPv6 address (as a NixOS test network does), reads as `unreachable`; declare it by IP address or let it listen on both families (`bind = "::"` accepts IPv4-mapped connections too). DNS rebinding is not a risk here because the target is pinned to the address that was checked against `deny_cidrs`.
|
|
|
|
## Limits and non-goals
|
|
|
|
- No aggregation: `peer = "*"` and fleet-wide views are a follow-up, because partial failures need a result shape of their own. Ask one peer at a time.
|
|
- No chaining: a consolidator's request carries `x-slinuxm-hop`, so a peer that is itself a consolidator refuses it. This prevents loops and amplification.
|
|
- No write path: proxying is GET-only and allowlisted, and nothing here starts, stops or changes a unit or a container.
|
|
- No history: peer state is in memory; use OTel for trends.
|
|
- No runtime management: peers are declarative, reloaded on SIGHUP.
|
|
- Each peer exports its own telemetry; peers' OTel signals are not re-exported by the consolidator.
|
|
|
|
## Testing
|
|
|
|
The peers acceptance test is a NixOS VM test. Three machines run the real `slinuxm`: `alpha` and `beta` consolidate each other over plain HTTP, and `alpha` also declares the TLS-only `gamma` with a private CA. It needs KVM and takes minutes.
|
|
|
|
```sh
|
|
devenv shell -- dev-test vm peers # nix build .#tests.x86_64-linux.peers -L
|
|
```
|
|
|
|
Against a tree without the consolidator the test stops at the first peers assertion; that is expected, the test is written ahead of the feature. `dev/dev-test.nu` is the repository's test runner. `devenv shell -- dev-test --help` lists its subcommands and the command behind each. In an interactive `devenv shell`, `dev-test …` is that same script.
|
|
|
|
In Nushell the runner is a module. Load it once, for example in `config.nu`, and `dev-test unit`, `dev-test vm peers`, ... complete natively. Integration-test name completion is cached in `target/skyg-test-filters.cache` at the repo root (not relative to the current directory, so completion works from any subdirectory), keyed on the newest mtime of the Rust sources and the Cargo manifests. A failing check raises a catchable error instead of exiting, so a red `dev-test unit` does not close the Nushell session that loaded the module; script mode still exits non-zero.
|
|
|
|
```nu
|
|
use /home/you/src/skyg-linux-tracker/dev/dev-test.nu * # absolute path to dev/dev-test.nu
|
|
```
|
|
|
|
zsh and bash completion comes from the same command table. The generated script only registers completion for the external command `dev-test`. An interactive `devenv shell` sources the bash script. That hook runs in bash and direnv will not source it either, so zsh users add the following to `.zshrc` after `compinit`:
|
|
|
|
```sh
|
|
source <(nu /home/you/src/skyg-linux-tracker/dev/dev-test.nu completions zsh)
|
|
```
|
|
|
|
bash, from the repo checkout:
|
|
|
|
```sh
|
|
source <(nu dev/dev-test.nu completions bash)
|
|
```
|
|
|
|
`nu dev/dev-test.nu self-check` fails if the command table, the defined subcommands, the help text, the cargo argv, the filter cache or the shell completion drift apart.
|