261 lines
20 KiB
Markdown
261 lines
20 KiB
Markdown
# Consolidator (peers)
|
|
|
|
One `slinuxm` can be a **consolidator** for other `slinuxm` instances. Each one it is told about is a *peer*: it health-checks the peer in two ways and passes read-only requests through to it, so one MCP endpoint or one REST base URL covers the fleet. The feature is off by default (`[peers] enabled = false`), and a peer-less instance behaves exactly as before: same routes, eight MCP tools, same metrics, no extra tasks or config.
|
|
|
|
Everything here is declarative: peers come from the config file, nothing writes to a peer, and no request parameter can supply a URL. Runtime peer management (`peer add`, a peers table) does not exist.
|
|
|
|
Tests: `tests/peers.rs` runs the whole path in-process (no VM), and a three-machine NixOS VM test covers boot, TLS, rotation and outages. Both go through the repository's runner, `devenv shell -- dev-test …`; see [Testing](#testing) below.
|
|
|
|
## Model
|
|
|
|
| Check | What it proves | How |
|
|
|---|---|---|
|
|
| `server` | The peer's port accepts a TCP connection | one connect per `tcp_interval_secs`, bounded by `tcp_timeout_secs` |
|
|
| `api` | The peer's REST API runs **and** accepts our token | authenticated `GET /api/v1/ping` every `api_interval_secs`, bounded by `api_timeout_secs` |
|
|
|
|
A check flips down after `fail_threshold` consecutive failures and up again on the first success. All state is in memory (`since`, `last_checked`, `last_error`, `latency_ms`), so a restart of the consolidator resets it; history belongs in your OTel backend.
|
|
|
|
`last_error` is a class, never peer text or a URL: `unreachable`, `timeout`, `dns`, `tls_error`, `http_<code>`, `unauthorized`, `oversize`, `saturated`, `token_file`.
|
|
|
|
### Reading `server` and `api`
|
|
|
|
| `server.up` | `api.up` | `api.auth` | Meaning |
|
|
|---|---|---|---|
|
|
| `true` | `true` | `ok` | healthy |
|
|
| `true` | `true` | `rejected` | the API runs, **our token is refused**: revoked, rotated on the peer, or the file is still a placeholder. Mint a new token on the peer and replace the file (`auth_ok` is 0 in OTel while `peer.up` is 1 for both checks) |
|
|
| `true` | `true` | `unknown` | the peer predates `GET /api/v1/ping`; the check fell back to public `/healthz`, so liveness is proven but not our credential |
|
|
| `true` | `false` | `unknown` | the port answers but the API does not (`tls_error` for a certificate problem, `timeout`, `http_5xx`) |
|
|
| `false` | `false` | `unknown` | no resolved address answers on the port: the peer is down or unreachable (see [Address families](#address-families)) |
|
|
|
|
A peer that rejects our token is reported to *your* callers as a `503` naming the peer, never as a `401`: re-authenticating with this host would not help.
|
|
|
|
## Configuration
|
|
|
|
```toml
|
|
[peers]
|
|
enabled = false # default: no peer routes, tools, tasks or metrics
|
|
tcp_interval_secs = 10
|
|
tcp_timeout_secs = 3 # must be below tcp_interval_secs
|
|
api_interval_secs = 30
|
|
api_timeout_secs = 5 # must be below api_interval_secs
|
|
fail_threshold = 2 # consecutive failures before a check flips down
|
|
request_timeout_secs = 10 # one proxied request, connect to body
|
|
max_response_bytes = 8388608 # buffered bodies above this are refused
|
|
max_concurrent_per_peer = 4 # proxied requests in flight per peer
|
|
max_concurrent_total = 16 # ... across all peers
|
|
deny_cidrs = ["169.254.0.0/16", "0.0.0.0/8", "fe80::/10", "::/128", "fd00:ec2::254/128"] # resolved peer addresses in these ranges are refused
|
|
|
|
[[peers.peer]]
|
|
name = "minipc1" # ^[a-z0-9][a-z0-9_-]{0,31}$, unique, at most 64
|
|
url = "https://minipc1:1984" # http or https, no userinfo, query or fragment
|
|
token_file = "/run/credentials/skyg-linux-tracker.service/peer-minipc1" # absolute
|
|
ca_file = "/etc/skyg/private-ca.pem" # optional extra trust anchor
|
|
allow_plain_http = false # required true for http:// to a non-loopback host
|
|
enabled = true # false keeps the declaration without contacting the peer
|
|
```
|
|
|
|
Scalar keys accept environment overrides (`SKYG_PEERS__ENABLED`, `SKYG_PEERS__API_INTERVAL_SECS`, ...); `[[peers.peer]]` entries are file-only. `slinuxm --print-config` prints `token_file` paths, never their contents. Validation refuses: a timeout that is not below its interval, more than 64 peers, duplicate names, a bad name, a relative `token_file` or `ca_file`, a non-http(s) URL, userinfo/query/fragment in a URL, an IP literal in `deny_cidrs`, and plain `http` to a non-loopback host without `allow_plain_http = true` (which also logs a warning: the bearer token travels in cleartext). `enabled = true` with no peers logs a warning.
|
|
|
|
`deny_cidrs` is checked against every **resolved** address at connect time, so a name where any address lands in a denied range is refused as `unreachable`. IPv4-mapped addresses are compared in their IPv4 form, so `::ffff:169.254.169.254` is caught by `169.254.0.0/16`. Private LAN ranges are deliberately allowed; the defaults deny only link-local and cloud-metadata addresses in both families (`169.254.0.0/16`, `fe80::/10`, `fd00:ec2::254/128`) and the unspecified address (`0.0.0.0/8`, `::/128`).
|
|
|
|
## REST and MCP
|
|
|
|
All under the inbound auth of this host; nothing is mounted while peers are off.
|
|
|
|
| Route | Behaviour |
|
|
|---|---|
|
|
| `GET /api/v1/peers` | `{"peers": [...]}`, sorted by name, disabled peers included |
|
|
| `GET /api/v1/peers/{peer}` | one status object; unknown peer is `404` |
|
|
| `GET /api/v1/peers/{peer}/...` | GET-only pass-through of an allowlisted route |
|
|
|
|
The status object is `{name, enabled, url, server{up, since, last_checked, last_error, latency_ms}, api{up, since, last_checked, last_error, latency_ms, auth, peer_version}}`, times in RFC 3339 UTC.
|
|
|
|
The pass-through allowlist is `systemd/...`, `journal`, `journal/follow`, `docker/...`, with the query parameters the local routes accept (`state`, `unit`, `priority`, `lines`, `since`, `all`, `tail`, `follow`). A caller may include the `/api/v1` prefix or leave it out; the peer always receives the route under its own `/api/v1`. Anything else is a `400`, `..` and encoded slashes are rejected, `ping`, `peers/...` and `/healthz` are not reachable through it, and methods other than GET are `405`. The peer's 2xx answer comes back unchanged (status, `Content-Type`, body; SSE stays a stream) with `x-slinuxm-peer: <peer>`; every error produced for a peer carries that header too, except the `405` for a non-GET method, which the router answers before the peer route runs. Consolidated failures use the existing error shape:
|
|
|
|
| Peer outcome | Answer to your caller |
|
|
|---|---|
|
|
| peer `404` | `404 not_found`, `peer <name>: not found on peer` |
|
|
| peer `400`/`422` | `400 bad_request` |
|
|
| peer rejects our token (`401`/`403`) | `503 unavailable`, `peer <name>: peer rejected credentials` |
|
|
| timeout, refused, DNS, TLS, `5xx`, oversize, saturated, token file | `503 unavailable` with the class |
|
|
| malformed peer JSON | `500 internal` |
|
|
| unknown peer | `404 not_found` |
|
|
|
|
No caller header is forwarded except `traceparent`; the Authorization header is always the peer's own token from `token_file`. A caller that sends `x-slinuxm-hop` is refused with `400` (no chained consolidators).
|
|
|
|
With peers enabled, MCP exposes `list_peers`, `get_peer` and the resource `peers://status` (same JSON as the list), every one of the eight existing tools gains an optional `peer` argument, and the `container_health_report` and `diagnose_unit` prompts gain a `peer` argument. Ten tools instead of eight, and only on the instances that need them. Peer-sourced text is labelled untrusted data, as local log text is.
|
|
|
|
## OpenTelemetry
|
|
|
|
Four observable gauges, registered only with peers enabled and OTLP metrics on (`otel.enabled = true` together with `otel.export_unit_metrics = true`), read from the registry snapshot on each scrape: `peer.up` (attributes `peer.name`, `peer.check` = `server`/`api`), `peer.api.auth_ok` (`peer.name`), `peer.check.duration_seconds` (`peer.name`, `peer.check`) and `peer.last_change_timestamp_seconds` (`peer.name`, `peer.check`). No URL, version or token in attributes. A removed peer disappears from the next scrape, and a peer declared with `enabled = false` has no series at all: its checks never run, so a `0` would turn an intended switch-off into a false alarm. Outbound proxied requests carry a client span named `peer.request` with `peer.name`, and the caller's `traceparent` is passed through, so a trace spans the fleet.
|
|
|
|
## Credentials and threat model
|
|
|
|
DEC-11 stores *inbound* tokens only as sha256 hashes. A consolidator cannot do that: it has to replay the peer's plaintext bearer token. DEC-13 therefore makes **outbound** peer tokens externally managed files:
|
|
|
|
- `slinuxm` owns no copy of a peer token: not in SQLite, not in `config.toml`, not in `--print-config`, the logs or a support bundle. The config holds a path; the process reads the file into a redacted `Secret` on every check and request and drops it.
|
|
- On NixOS the file is an agenix secret decrypted at activation to `/run/agenix/...` (root-only) and handed to the service through systemd `LoadCredential`, so the service user never needs read access to the agenix output. Outside NixOS, put the file somewhere only the service user can read: a group- or world-accessible token file is refused (`token_file` class) unless it sits in `$CREDENTIALS_DIRECTORY`.
|
|
- Rotation is a file replacement; the next check reads the new value. On NixOS also `systemctl reload` (SIGHUP) so systemd re-copies the credential (`RefreshOnReload=credentials`).
|
|
- Use **one token per consolidator on every peer** (`consolidator-<host>`, read-only by nature since every token is read-only): `slinuxm token revoke consolidator-<host>` on a peer cuts exactly that consolidator off, and nothing else.
|
|
|
|
What it does not protect: root on the consolidator (it can read the decrypted file), and the consolidator's own token as an asset — it reaches every peer's data, which is why tokens are per consolidator, revocable, and why scoping tokens per peer is a follow-up, not a v1 feature.
|
|
|
|
Peers are only ever declared by whoever manages the config (root or Nix). No route, parameter or MCP argument supplies a URL, so the consolidator cannot be steered at an arbitrary target. Redirects from a peer are never followed (`503 http_3xx`), TLS is verified against the webpki roots plus the optional `ca_file`, and there is no skip-verify switch.
|
|
|
|
## NixOS with agenix
|
|
|
|
The module renders the peers and the credentials; you only declare them:
|
|
|
|
```nix
|
|
services.skyg-linux-tracker = {
|
|
enable = true;
|
|
bind = "0.0.0.0"; # inbound, only if callers reach this host from other machines
|
|
openFirewall = true; # or firewallInterfaces = [ "enp3s0" ];
|
|
peers = {
|
|
enable = true;
|
|
settings = { api_interval_secs = 30; fail_threshold = 2; }; # [peers] tuning, snake_case
|
|
peers = {
|
|
minipc1 = { url = "https://minipc1:1984"; tokenFile = config.age.secrets.peer-minipc1.path; };
|
|
nas = {
|
|
url = "http://nas:1984";
|
|
tokenFile = config.age.secrets.peer-nas.path;
|
|
allowPlainHttp = true; # the token then travels in cleartext in your LAN
|
|
};
|
|
};
|
|
};
|
|
};
|
|
```
|
|
|
|
A worked agenix setup for host `monitor`, which consolidates `minipc1` (HTTPS with a private CA) and `nas` (plain HTTP in a trusted LAN).
|
|
|
|
On each peer, mint a token for this consolidator. It has to be a dedicated token: revoking it cuts off exactly this consolidator.
|
|
|
|
```sh
|
|
# on minipc1 and on nas
|
|
sudo -u skyg-linux-tracker slinuxm token create consolidator-monitor
|
|
```
|
|
|
|
Register the encrypted files next to your other agenix secrets and encrypt them for the consolidator host. Only the ciphertext goes into git.
|
|
|
|
```nix
|
|
# secrets/secrets.nix (agenix)
|
|
{
|
|
# The token minipc1 issued to `monitor`. Only `monitor` needs to decrypt it:
|
|
# the consolidator replays it, no other host reads it.
|
|
"peer-minipc1.age".publicKeys = [ monitor ];
|
|
"peer-nas.age".publicKeys = [ monitor ];
|
|
}
|
|
```
|
|
|
|
```sh
|
|
agenix -e secrets/peer-minipc1.age # paste the token printed on minipc1
|
|
agenix -e secrets/peer-nas.age # paste the token printed on nas
|
|
```
|
|
|
|
```nix
|
|
{ config, inputs, ... }: {
|
|
imports = [ inputs.agenix.nixosModules.default ];
|
|
|
|
# Decrypted at activation to /run/agenix/peer-<name>, root-only. The module
|
|
# hands the file to the service through LoadCredential, so the
|
|
# `skyg-linux-tracker` user never reads it directly.
|
|
age.secrets.peer-minipc1 = {
|
|
file = ./secrets/peer-minipc1.age;
|
|
owner = "root";
|
|
mode = "0400";
|
|
};
|
|
age.secrets.peer-nas = { file = ./secrets/peer-nas.age; owner = "root"; mode = "0400"; };
|
|
|
|
services.skyg-linux-tracker = {
|
|
enable = true;
|
|
bind = "0.0.0.0"; # inbound for remote callers; peers need no inbound port
|
|
openFirewall = true;
|
|
peers = {
|
|
enable = true;
|
|
peers = {
|
|
minipc1 = {
|
|
url = "https://minipc1:1984";
|
|
tokenFile = config.age.secrets.peer-minipc1.path; # a string path, not a path literal
|
|
caFile = "/etc/ssl/private-ca.pem"; # readable by the service user
|
|
};
|
|
nas = {
|
|
url = "http://nas:1984";
|
|
tokenFile = config.age.secrets.peer-nas.path;
|
|
allowPlainHttp = true;
|
|
};
|
|
};
|
|
};
|
|
};
|
|
}
|
|
```
|
|
|
|
The module turns that into one `[[peers.peer]]` entry per enabled peer, with `token_file = /run/credentials/skyg-linux-tracker.service/peer-<name>`, and one `LoadCredential=peer-<name>:<tokenFile>` line. Consequences worth knowing:
|
|
|
|
- `tokenFile` must be a **string** (`config.age.secrets...path`), not a path literal: a `nix` path or `"${./secret}"` interpolates a store path and is rejected at evaluation. Such a secret has already been copied into the local store by then, so rotate it.
|
|
- `caFile` is **not** passed through `LoadCredential`: the service user reads it directly, so it must be readable by `skyg-linux-tracker`. A root-only file such as the agenix output makes every check and proxied request to that peer fail as `tls_error`. Store paths are fine for a public CA.
|
|
- Peers with `enable = false` are rendered nowhere, so no credential is shipped for them.
|
|
- `peers.settings` is freeform `[peers]` tuning; it must not contain `enabled`, `peer` or a token.
|
|
|
|
### Reload or restart
|
|
|
|
`switch-to-configuration` reloads (SIGHUP) when only the config file changed, and restarts when the unit file changed. SIGHUP re-reads `[peers]`, so editing a peer's `url`, `caFile` or the intervals applies without a restart, and systemd re-copies the credentials (`RefreshOnReload=credentials`), so a rotated token is picked up as well. Adding, removing, enabling or disabling a peer, or changing its `tokenFile`, changes the `LoadCredential=` lines and therefore **restarts** the service — the safe outcome. `enabled`, `deny_cidrs`, `max_response_bytes` and the two concurrency limits also need a restart (the routes are mounted once and the limits live in the transport), and the module's restart trigger makes a `switch` do that instead of silently keeping the old values.
|
|
|
|
Outside NixOS the same split applies by hand: SIGHUP re-reads the TLS pair and `[peers]` (peer entries, intervals, timeouts, `fail_threshold`); everything else needs a restart.
|
|
|
|
### Shutdown
|
|
|
|
SIGTERM stops the listener, cancels the shared token and then drains. The health loops end on that token, and a proxied follow stream is closed rather than held until the peer decides to stop sending: shutdown ends the SSE stream to your caller even when the peer keeps its socket open (`tests/peers.rs` asserts exactly that against a listener that never closes). A buffered proxied request is bounded by `request_timeout_secs` and a follow stream by the peer's own cap, so no peer connection outlives the process by more than one timeout.
|
|
|
|
## Firewall and networking
|
|
|
|
Every **peer** must accept the consolidator's address on its port; the consolidator only needs outbound access. On a peer's NixOS host that is `openFirewall = true` or `firewallInterfaces = [ "enp3s0" ]`; a peer bound to `127.0.0.1` is not reachable from another machine, so a consolidator needs the peer to bind a LAN address (with auth on, which the module enforces) and either TLS or a trusted network.
|
|
|
|
### Address families
|
|
|
|
A peer's name is resolved at connect time and **every** resolved address is used: the health probe tries them in order within its timeout, and the HTTP client pins the whole list so the connection falls through to the next address when the first one refuses or does not answer. A peer that listens on IPv4 only is therefore reached even when its name also resolves to an IPv6 address (as a NixOS test network or a dual-stack LAN does). Both families and a single family work; `bind = "::"` on a peer is dual stack, `bind = "0.0.0.0"` is IPv4 only.
|
|
|
|
DNS rebinding is still not a risk: the set is resolved once per request and pinned, and a name where **any** resolved address falls in `deny_cidrs` is refused as `unreachable` rather than dialled at the permitted addresses.
|
|
|
|
## Limits and non-goals
|
|
|
|
- No aggregation: `peer = "*"` and fleet-wide views are a follow-up, because partial failures need a result shape of their own. Ask one peer at a time.
|
|
- No chaining: a consolidator's request carries `x-slinuxm-hop`, so a peer that is itself a consolidator refuses it. This prevents loops and amplification.
|
|
- No write path: proxying is GET-only and allowlisted, and nothing here starts, stops or changes a unit or a container.
|
|
- No history: peer state is in memory; use OTel for trends.
|
|
- No runtime management: peers are declarative, reloaded on SIGHUP.
|
|
- Each peer exports its own telemetry; peers' OTel signals are not re-exported by the consolidator.
|
|
|
|
## Testing
|
|
|
|
Two layers, neither of which needs a VM for the first:
|
|
|
|
- `tests/peers.rs` drives the whole path in-process: a consolidator served by the real router and registry with the real outbound client, against peers that are either another in-process `slinuxm` (fake sources, real auth with a minted token) or a hand-written listener for a peer that misbehaves. It covers a plain peer and a TLS peer with its own CA, a peer signed by the wrong CA, a redirect, an oversize (declared and chunked) body, a silent peer, a slow-loris peer, the 64-peer cap, shutdown ending a proxied follow, and a peer declared by name that listens on one address family only.
|
|
- The NixOS VM test boots three machines running the real `slinuxm`: `alpha` and `beta` consolidate each other over plain HTTP, and `alpha` also declares the TLS-only `gamma` with a private CA (bound IPv4 only, so the address-family fallback stays guarded). It asserts status, fan-in, proxied data against a direct call, MCP parity, token install/rotation/revocation, peer removal on SIGHUP and an outage, all with `MainPID` unchanged where a reload is promised. `nix flake check` also runs `checks.module-peers`, an evaluation-only case list for the module: every rendering and assertion fails the check without booting anything.
|
|
|
|
```sh
|
|
devenv shell -- dev-test integration peers # cargo test --test peers
|
|
devenv shell -- dev-test vm peers # nix build .#checks.x86_64-linux.vm-peers -L
|
|
```
|
|
|
|
`dev/dev-test.nu` is the repository's test runner. `devenv shell -- dev-test --help` lists its subcommands and the command behind each. In an interactive `devenv shell`, `dev-test …` is that same script.
|
|
|
|
In Nushell the runner is a module. Load it once, for example in `config.nu`, and `dev-test unit`, `dev-test vm peers`, ... complete natively. Integration-test name completion is cached in `target/skyg-test-filters.cache` at the repo root (not relative to the current directory, so completion works from any subdirectory), keyed on the newest mtime of the Rust sources and the Cargo manifests. A failing check raises a catchable error instead of exiting, so a red `dev-test unit` does not close the Nushell session that loaded the module; script mode still exits non-zero.
|
|
|
|
```nu
|
|
use /home/you/src/skyg-linux-tracker/dev/dev-test.nu * # absolute path to dev/dev-test.nu
|
|
```
|
|
|
|
zsh and bash completion comes from the same command table. The generated script only registers completion for the external command `dev-test`. An interactive `devenv shell` sources the bash script. That hook runs in bash and direnv will not source it either, so zsh users add the following to `.zshrc` after `compinit`:
|
|
|
|
```sh
|
|
source <(nu /home/you/src/skyg-linux-tracker/dev/dev-test.nu completions zsh)
|
|
```
|
|
|
|
bash, from the repo checkout:
|
|
|
|
```sh
|
|
source <(nu dev/dev-test.nu completions bash)
|
|
```
|
|
|
|
`nu dev/dev-test.nu self-check` fails if the command table, the defined subcommands, the help text, the cargo argv, the filter cache or the shell completion drift apart.
|