F: try every resolved peer address instead of the first
A peer whose name resolves to both families but which listens on only one of them was reported unreachable, which the documented 0.0.0.0 binding produces on a dual-stack LAN and in the NixOS test network. resolve_target now returns the whole address set (still refusing the name if any address is denied), the HTTP client pins it with resolve_to_addrs so the connector falls through to the next address, and tcp_check tries each address in order with a per-attempt share of the remaining deadline. ClientKey keys on the set. Regression cover: a client test that resolves localhost against a single-family listener, a tests/peers.rs case declaring the peer as http://localhost:<port>, and the VM test's TLS peer gamma now binds 0.0.0.0 (IPv4 only) while alpha and beta stay dual stack. Docs and the contracts section 10 address note describe the new behaviour.
This commit is contained in:
1 parent
3ae8e74a1d
commit
dde6a12ce4
5 files changed
+146
-34
No files matched your search
@@ -25,7 +25,7 @@ A check flips down after `fail_threshold` consecutive failures and up again on t
|
||||
| `true` | `true` | `rejected` | the API runs, **our token is refused**: revoked, rotated on the peer, or the file is still a placeholder. Mint a new token on the peer and replace the file (`auth_ok` is 0 in OTel while `peer.up` is 1 for both checks) |
|
||||
| `true` | `true` | `unknown` | the peer predates `GET /api/v1/ping`; the check fell back to public `/healthz`, so liveness is proven but not our credential |
|
||||
| `true` | `false` | `unknown` | the port answers but the API does not (`tls_error` for a certificate problem, `timeout`, `http_5xx`) |
|
||||
| `false` | `false` | `unknown` | nothing answers on the port: the peer is down or unreachable (also the case when the name resolves to an address the peer does not listen on, see [Address families](#address-families)) |
|
||||
| `false` | `false` | `unknown` | no resolved address answers on the port: the peer is down or unreachable (see [Address families](#address-families)) |
|
||||
|
||||
A peer that rejects our token is reported to *your* callers as a `503` naming the peer, never as a `401`: re-authenticating with this host would not help.
|
||||
|
||||
@@ -56,7 +56,7 @@ enabled = true # false keeps the declaration without contacting
|
||||
|
||||
Scalar keys accept environment overrides (`SKYG_PEERS__ENABLED`, `SKYG_PEERS__API_INTERVAL_SECS`, ...); `[[peers.peer]]` entries are file-only. `slinuxm --print-config` prints `token_file` paths, never their contents. Validation refuses: a timeout that is not below its interval, more than 64 peers, duplicate names, a bad name, a relative `token_file` or `ca_file`, a non-http(s) URL, userinfo/query/fragment in a URL, an IP literal in `deny_cidrs`, and plain `http` to a non-loopback host without `allow_plain_http = true` (which also logs a warning: the bearer token travels in cleartext). `enabled = true` with no peers logs a warning.
|
||||
|
||||
`deny_cidrs` is checked against the **resolved** address at connect time, so a name that resolves into a denied range is refused as `unreachable`. Private LAN ranges are deliberately allowed; the default denies only link-local/metadata and the unspecified range.
|
||||
`deny_cidrs` is checked against every **resolved** address at connect time, so a name where any address lands in a denied range is refused as `unreachable`. Private LAN ranges are deliberately allowed; the default denies only link-local/metadata and the unspecified range.
|
||||
|
||||
## REST and MCP
|
||||
|
||||
@@ -208,7 +208,9 @@ Every **peer** must accept the consolidator's address on its port; the consolida
|
||||
|
||||
### Address families
|
||||
|
||||
A peer is reached at the address its name resolves to at connect time, and the first resolved address wins. A peer that listens on IPv4 only, with a name that also resolves to an IPv6 address (as a NixOS test network does), reads as `unreachable`; declare it by IP address or let it listen on both families (`bind = "::"` accepts IPv4-mapped connections too). DNS rebinding is not a risk here because the target is pinned to the address that was checked against `deny_cidrs`.
|
||||
A peer's name is resolved at connect time and **every** resolved address is used: the health probe tries them in order within its timeout, and the HTTP client pins the whole list so the connection falls through to the next address when the first one refuses or does not answer. A peer that listens on IPv4 only is therefore reached even when its name also resolves to an IPv6 address (as a NixOS test network or a dual-stack LAN does). Both families and a single family work; `bind = "::"` on a peer is dual stack, `bind = "0.0.0.0"` is IPv4 only.
|
||||
|
||||
DNS rebinding is still not a risk: the set is resolved once per request and pinned, and a name where **any** resolved address falls in `deny_cidrs` is refused as `unreachable` rather than dialled at the permitted addresses.
|
||||
|
||||
## Limits and non-goals
|
||||
|
||||
|
||||
+8
-7
@@ -51,15 +51,14 @@ let
|
||||
};
|
||||
|
||||
# The test network gives every node an IPv4 and an IPv6 address in
|
||||
# /etc/hosts, and name resolution returns the IPv6 one first, so the nodes
|
||||
# listen on both families. A peer that only listens on one of them is
|
||||
# reported as unreachable.
|
||||
mkNode = { peers }: { ... }: {
|
||||
# /etc/hosts, so a peer name resolves to two addresses. The consolidators
|
||||
# listen on both families; gamma listens on IPv4 only, which is the case that
|
||||
# used to read as unreachable when the transport pinned the first address.
|
||||
mkNode = { peers, bind ? "::" }: { ... }: {
|
||||
imports = [ module ];
|
||||
services.skyg-linux-tracker = {
|
||||
enable = true;
|
||||
inherit port;
|
||||
bind = "::";
|
||||
inherit port bind;
|
||||
openFirewall = true;
|
||||
# A node without peers keeps the feature off: `peers.enable` requires
|
||||
# at least one peer.
|
||||
@@ -117,7 +116,9 @@ pkgs.testers.runNixOSTest {
|
||||
};
|
||||
|
||||
nodes.gamma = { ... }: {
|
||||
imports = [ (mkNode { peers = { }; }) ];
|
||||
# IPv4 only: alpha still reaches it by name, because every resolved
|
||||
# address is tried.
|
||||
imports = [ (mkNode { peers = { }; bind = "0.0.0.0"; }) ];
|
||||
services.skyg-linux-tracker.tls = {
|
||||
enable = true;
|
||||
certFile = "${certs}/cert.pem";
|
||||
|
||||
@@ -317,10 +317,10 @@ Rendering: one `[[peers.peer]]` per enabled peer with `token_file = "/run/creden
|
||||
|
||||
The VM test declares its peers with the module's first-class options (F2), so the TOML contract above must stay compatible with what they render:
|
||||
|
||||
- Hosts: `alpha` (peers `beta` over plain HTTP, `gamma` over HTTPS with `ca_file`), `beta` (peer `alpha`), `gamma` (TLS-only, no peers). Port 1984, bind `::` (dual stack, see the address note below).
|
||||
- Hosts: `alpha` (peers `beta` over plain HTTP, `gamma` over HTTPS with `ca_file`), `beta` (peer `alpha`), `gamma` (TLS-only, no peers). Port 1984; `alpha` and `beta` bind `::` (dual stack) and `gamma` binds `0.0.0.0` (IPv4 only), so a single-family peer stays covered (see the address note below).
|
||||
- Config comes through the module: `peers.enable = true`, `peers.settings = { tcp_interval_secs = 2; tcp_timeout_secs = 1; api_interval_secs = 3; api_timeout_secs = 2; fail_threshold = 2; request_timeout_secs = 4; }` and `peers.peers.<name> = { url; tokenFile; caFile ? null; allowPlainHttp }`. The module renders `[peers]` plus one `[[peers.peer]]` per enabled peer with keys `name`, `url`, `token_file`, `allow_plain_http`, optional `ca_file`.
|
||||
- `token_file = /run/credentials/skyg-linux-tracker.service/peer-<name>` (the module adds `LoadCredential=peer-<name>:/var/lib/peer-tokens/<name>` and `RefreshOnReload=credentials`); the file holds the placeholder `pending` until the test installs a real token, so the placeholder must pass A3 loading (non-empty, under `$CREDENTIALS_DIRECTORY` so the permission check is skipped) and be reported as `auth = "rejected"`, not as a config error.
|
||||
- Address note: a test node's name resolves to both its IPv4 and its IPv6 address, and resolution returns the IPv6 one first, so each node binds `::` and the peer URLs keep the hostnames. A peer reachable only on IPv4 is reported as `unreachable` (the transport connects to the first resolved address); see the ws-f Log.
|
||||
- Address note: a test node's name resolves to both its IPv4 and its IPv6 address. The transport uses **every** resolved address (the probe tries them in order inside its timeout, the client pins the whole set, and one denied address refuses the name), so a peer is found on whichever family it listens on. `alpha` and `beta` bind `::` (dual stack) and the TLS peer `gamma` binds `0.0.0.0`, so the test covers both a dual-stack and an IPv4-only peer; the peer URLs keep the hostnames.
|
||||
- Token rotation = `mv` a new file into place and `systemctl reload skyg-linux-tracker.service` (SIGHUP with `RefreshOnReload=credentials`); status must recover without a restart. SIGHUP must therefore re-read token files and keep the registry (B4).
|
||||
- Timeouts (1 s / 2 s) are below intervals (2 s / 3 s), as validation requires; `request_timeout_secs = 4` and the test's `elapsed < 15` bound the proxy call when a peer VM is down.
|
||||
- Assertions the contract must satisfy: `GET /api/v1/peers/<p>` JSON has `server.up`, `api.up`, `api.auth` in {`ok`,`rejected`}; `GET /api/v1/peers` body contains each peer name; proxied `GET /api/v1/peers/<p>/systemd/units[/<unit>.service]` returns the peer's data with response header `x-slinuxm-peer: <p>`; `POST`/`DELETE` on a proxy path -> 405; `x-slinuxm-hop: 1` -> 4xx; unknown peer -> 404; peer rejected/down -> 503 with the peer name in the body and the caller's own token still valid; MCP `list_peers` output contains the peer names; MCP `list_systemd_units` with `{"peer": "<p>"}` returns the peer's units.
|
||||
|
||||
+108
-22
@@ -66,14 +66,14 @@ enum CaKey {
|
||||
Extra(Vec<u8>),
|
||||
}
|
||||
|
||||
/// One client per CA plus pinned address. Pinning the resolved address both
|
||||
/// keeps DNS rebinding from slipping past the deny list and makes the check
|
||||
/// and the connection use the same address.
|
||||
/// One client per CA plus pinned address set. Pinning every address that
|
||||
/// passed the deny check both keeps DNS rebinding from slipping past it and
|
||||
/// lets the connector try the next address when the first one does not answer.
|
||||
#[derive(Clone, PartialEq, Eq, Hash)]
|
||||
struct ClientKey {
|
||||
ca: CaKey,
|
||||
host: Option<String>,
|
||||
addr: SocketAddr,
|
||||
addrs: Vec<SocketAddr>,
|
||||
}
|
||||
|
||||
struct Permits {
|
||||
@@ -151,7 +151,7 @@ impl ReqwestTransport {
|
||||
&self,
|
||||
ca_pem: Option<Vec<u8>>,
|
||||
host: Option<&str>,
|
||||
addr: SocketAddr,
|
||||
addrs: &[SocketAddr],
|
||||
) -> Result<reqwest::Client, PeerError> {
|
||||
let key = ClientKey {
|
||||
ca: match &ca_pem {
|
||||
@@ -159,7 +159,7 @@ impl ReqwestTransport {
|
||||
Some(pem) => CaKey::Extra(pem.clone()),
|
||||
},
|
||||
host: host.map(str::to_owned),
|
||||
addr,
|
||||
addrs: addrs.to_vec(),
|
||||
};
|
||||
let mut cache = Self::lock(&self.clients);
|
||||
if let Some(client) = cache.get(&key) {
|
||||
@@ -168,7 +168,7 @@ impl ReqwestTransport {
|
||||
if cache.len() >= crate::peers::types::MAX_PEERS {
|
||||
cache.clear();
|
||||
}
|
||||
let client = build_client(ca_pem.as_deref(), host, addr)?;
|
||||
let client = build_client(ca_pem.as_deref(), host, addrs)?;
|
||||
cache.insert(key, client.clone());
|
||||
Ok(client)
|
||||
}
|
||||
@@ -222,12 +222,14 @@ fn denied(ip: IpAddr, deny: &[IpNet]) -> bool {
|
||||
}
|
||||
|
||||
/// Resolve the peer host and reject denied addresses before any connection is
|
||||
/// made. The returned address is pinned into the HTTP client; an IP literal
|
||||
/// needs no pin because there is no name to rebind.
|
||||
/// made. Every address is returned so a peer that listens on one family is not
|
||||
/// reported as down when its name also resolves to the other; the set is
|
||||
/// pinned into the HTTP client. An IP literal needs no pin because there is no
|
||||
/// name to rebind.
|
||||
async fn resolve_target(
|
||||
peer: &PeerConfig,
|
||||
deny: &[IpNet],
|
||||
) -> Result<(Option<String>, SocketAddr), PeerError> {
|
||||
) -> Result<(Option<String>, Vec<SocketAddr>), PeerError> {
|
||||
validate_target(peer)?;
|
||||
let port = peer
|
||||
.url
|
||||
@@ -242,9 +244,11 @@ async fn resolve_target(
|
||||
.collect(),
|
||||
None => return Err(PeerError::Unreachable),
|
||||
};
|
||||
let Some(addr) = addrs.first().copied() else {
|
||||
if addrs.is_empty() {
|
||||
return Err(PeerError::Dns);
|
||||
};
|
||||
}
|
||||
// One denied address refuses the whole name: picking a "safe" address out
|
||||
// of the set would let a hostile resolver keep the denied one around.
|
||||
if addrs.iter().any(|addr| denied(addr.ip(), deny)) {
|
||||
return Err(PeerError::Unreachable);
|
||||
}
|
||||
@@ -252,14 +256,14 @@ async fn resolve_target(
|
||||
Some(url::Host::Domain(name)) => Some(name.to_owned()),
|
||||
_ => None,
|
||||
};
|
||||
Ok((pin, addr))
|
||||
Ok((pin, addrs))
|
||||
}
|
||||
|
||||
/// Ring is passed in so this process never calls `install_default`.
|
||||
fn build_client(
|
||||
extra_ca: Option<&[u8]>,
|
||||
host: Option<&str>,
|
||||
addr: SocketAddr,
|
||||
addrs: &[SocketAddr],
|
||||
) -> Result<reqwest::Client, PeerError> {
|
||||
let tls = rustls_config(extra_ca)?;
|
||||
let mut builder = reqwest::Client::builder()
|
||||
@@ -268,7 +272,10 @@ fn build_client(
|
||||
.http1_only()
|
||||
.tls_backend_preconfigured(tls);
|
||||
if let Some(host) = host {
|
||||
builder = builder.resolve(host, addr);
|
||||
// The connector walks the pinned list, so the address the deny check
|
||||
// approved is the only kind that can be dialled, and an address family
|
||||
// the peer does not listen on no longer hides one it does.
|
||||
builder = builder.resolve_to_addrs(host, addrs);
|
||||
}
|
||||
builder.build().map_err(|_| PeerError::Tls)
|
||||
}
|
||||
@@ -524,11 +531,26 @@ impl PeerTransport for ReqwestTransport {
|
||||
let deadline = tokio::time::Instant::now() + timeout;
|
||||
within(deadline, async {
|
||||
let deny = self.deny_list();
|
||||
let (_, addr) = resolve_target(peer, &deny).await?;
|
||||
let started = tokio::time::Instant::now();
|
||||
let stream = tokio::net::TcpStream::connect(addr).await.map_err(map_io)?;
|
||||
drop(stream);
|
||||
Ok(started.elapsed())
|
||||
let (_, addrs) = resolve_target(peer, &deny).await?;
|
||||
let mut left = addrs.len();
|
||||
let mut last = PeerError::Unreachable;
|
||||
for addr in addrs {
|
||||
// Each address gets its share of what remains, so a blackholed
|
||||
// first address cannot eat the deadline the others need.
|
||||
let now = tokio::time::Instant::now();
|
||||
let share = deadline.saturating_duration_since(now) / left as u32;
|
||||
left -= 1;
|
||||
let started = tokio::time::Instant::now();
|
||||
match tokio::time::timeout(share, tokio::net::TcpStream::connect(addr)).await {
|
||||
Ok(Ok(stream)) => {
|
||||
drop(stream);
|
||||
return Ok(started.elapsed());
|
||||
}
|
||||
Ok(Err(err)) => last = map_io(err),
|
||||
Err(_) => last = PeerError::Timeout,
|
||||
}
|
||||
}
|
||||
Err(last)
|
||||
})
|
||||
.await
|
||||
}
|
||||
@@ -546,7 +568,7 @@ impl PeerTransport for ReqwestTransport {
|
||||
let (resp, permits, max) = within(deadline, async {
|
||||
let url = request_url(peer, &req)?;
|
||||
let deny = self.deny_list();
|
||||
let (pin, addr) = resolve_target(peer, &deny).await?;
|
||||
let (pin, addrs) = resolve_target(peer, &deny).await?;
|
||||
self.warn_plain_http(peer);
|
||||
let token_path = peer.token_file.clone();
|
||||
let secret = tokio::task::spawn_blocking(move || load_token(&token_path))
|
||||
@@ -570,7 +592,7 @@ impl PeerTransport for ReqwestTransport {
|
||||
} else {
|
||||
Some(self.acquire(&peer.name)?)
|
||||
};
|
||||
let client = self.client_for(ca, pin.as_deref(), addr)?;
|
||||
let client = self.client_for(ca, pin.as_deref(), &addrs)?;
|
||||
let max = self.max_bytes();
|
||||
let mut builder = client.get(url);
|
||||
for (name, value) in &req.headers {
|
||||
@@ -1196,6 +1218,70 @@ mod tests {
|
||||
let _ = std::fs::remove_file(&token);
|
||||
}
|
||||
|
||||
/// A name is resolved to every address, and one denied address refuses
|
||||
/// the name rather than being skipped in favour of a permitted one.
|
||||
#[tokio::test]
|
||||
async fn every_resolved_address_is_returned_and_checked() {
|
||||
let token = write_token("t", 0o600);
|
||||
let mut p = peer(9, &token, None);
|
||||
p.url = url("http://localhost:9");
|
||||
let (pin, addrs) = resolve_target(&p, &[]).await.unwrap();
|
||||
assert_eq!(pin.as_deref(), Some("localhost"));
|
||||
assert!(!addrs.is_empty());
|
||||
|
||||
// `localhost` may resolve to both loopback families. Denying one of
|
||||
// them must refuse the whole name.
|
||||
let denied_family = if addrs.iter().any(|a| a.is_ipv4()) {
|
||||
"127.0.0.0/8"
|
||||
} else {
|
||||
"::1/128"
|
||||
};
|
||||
let deny = deny_networks(&[denied_family.to_owned()]).unwrap();
|
||||
assert!(matches!(
|
||||
resolve_target(&p, &deny).await,
|
||||
Err(PeerError::Unreachable)
|
||||
));
|
||||
let _ = std::fs::remove_file(&token);
|
||||
}
|
||||
|
||||
/// A peer declared by name must be reached on the family it listens on:
|
||||
/// the listener takes one family only, so whichever family the resolver
|
||||
/// puts first, the other one has to be tried as well.
|
||||
#[tokio::test]
|
||||
async fn tcp_check_tries_every_resolved_address() {
|
||||
let token = write_token("t", 0o600);
|
||||
let transport = ReqwestTransport::new(&limits(4, 8, 1024)).unwrap();
|
||||
let mut tried = Vec::new();
|
||||
for bind in ["127.0.0.1", "[::1]"] {
|
||||
let Ok(listener) = TcpListener::bind(format!("{bind}:0")).await else {
|
||||
continue; // this environment has no IPv6 loopback
|
||||
};
|
||||
let port = listener.local_addr().unwrap().port();
|
||||
tokio::spawn(async move {
|
||||
while let Ok((mut sock, _)) = listener.accept().await {
|
||||
let _ = read_req(&mut sock).await;
|
||||
let _ = sock
|
||||
.write_all(
|
||||
b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\nConnection: close\r\n\r\nok",
|
||||
)
|
||||
.await;
|
||||
}
|
||||
});
|
||||
let mut p = peer(0, &token, None);
|
||||
p.url = url(&format!("http://localhost:{port}"));
|
||||
assert!(
|
||||
transport
|
||||
.tcp_check(&p, Duration::from_secs(2))
|
||||
.await
|
||||
.is_ok(),
|
||||
"{bind}"
|
||||
);
|
||||
tried.push(bind);
|
||||
}
|
||||
assert!(!tried.is_empty(), "no loopback family available");
|
||||
let _ = std::fs::remove_file(&token);
|
||||
}
|
||||
|
||||
/// The request timeout bounds preparation, not a follow stream: chunks
|
||||
/// keep arriving long after it would have expired.
|
||||
#[tokio::test]
|
||||
|
||||
@@ -471,6 +471,29 @@ async fn a_plain_peer_serves_its_own_data_through_the_proxy() {
|
||||
peer.stop().await;
|
||||
}
|
||||
|
||||
/// Declared by name, not by IP literal: `localhost` resolves to both loopback
|
||||
/// families while the peer listens on IPv4 only, so a transport that pins just
|
||||
/// the first resolved address reports this peer down.
|
||||
#[tokio::test]
|
||||
async fn a_peer_declared_by_name_is_reached_on_the_family_it_listens_on() {
|
||||
let dir = TempDir::new("localhost");
|
||||
let peer = Peer::start().await;
|
||||
let token = peer.token_file(&dir);
|
||||
let url = format!("http://localhost:{}", peer.port);
|
||||
let consolidator = Consolidator::start(one_peer("beta", url, None, &token)).await;
|
||||
|
||||
let status = consolidator
|
||||
.wait_status("beta", |s| s.server.up && s.api.up)
|
||||
.await;
|
||||
assert_eq!(status.api.auth, AuthState::Ok);
|
||||
let (code, _, body) = consolidator
|
||||
.get("/api/v1/peers/beta/systemd/units", None)
|
||||
.await;
|
||||
assert_eq!(code, 200, "{body}");
|
||||
consolidator.stop().await;
|
||||
peer.stop().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_tls_peer_is_trusted_through_its_ca_file() {
|
||||
let dir = TempDir::new("tls");
|
||||
|
||||
Reference in new issue
Block a user