F: F3 end-to-end peer tests
Real registry and transport against in-process peers and hand-written listeners: plain, TLS with an rcgen CA, a wrong CA, redirects, oversize bodies, a silent peer, the 64-peer cap and shutdown mid-follow. Two proxy cases are ignored: the proxied path is missing its /api/v1 prefix.
This commit is contained in:
1 parent
ca000d973f
commit
66ee41a672
3 files changed
+686
No files matched your search
@@ -68,3 +68,8 @@ Behaviour tests with the fake transport for every error mapping row in D9; 405 o
|
||||
- Handoff to WS-D: `PeerRequest.query` carries decoded pairs. A `unit` argument reaches the proxy as the unit name itself (`dev-disk-by\x2did.device`, backslash included) and needs no encoding on D's side; the tool schema takes the plain name, so an already percent-encoded string such as `dev-disk-by%5Cx2did.device` is not a unit name and is `400`.
|
||||
- Non-blocking 3: a `..` inside a single segment is not traversal, because segments cannot contain `/` and the forwarded path is re-encoded from them. Pinned by allowlist cases `systemd/units/foo..bar.mount` and `docker/containers/foo..bar`, plus a rejection case for an encoded `%2e%2e` inside one segment (`systemd/units/foo%2e%2ebar.mount`), which `encoding_forbidden` catches.
|
||||
- `docs/api.md` states the `unit` query rule and the `%5C`/literal-backslash forms.
|
||||
- 2026-10-20 (from WS-F, **blocking defect found by the F3 end-to-end test**): every proxied call to a healthy peer answers `404` and the consolidator maps it to `peer <name>: not found on peer`.
|
||||
- Cause: `proxy::get` hands `PeerRequest.path` to the transport as the stripped route (`/systemd/units`), and `ReqwestTransport::request_url` builds `{peer.url.path()}/{path}`, i.e. `http://<peer>:1984/systemd/units`. An unmodified peer serves its REST routes only under `/api/v1` (`server::build_router` nests `api::router` at `/api/v1`), so the peer's own 404 comes back. Health checks are unaffected: `health::probe_api` passes `/api/v1/ping`, which is the `PeerRequest.path` form `contracts.md` section 2 specifies ("e.g. `/api/v1/systemd/units`").
|
||||
- Reproduction (real transport, real peer, no VM): in the WS-F worktree, `cargo test --test peers` — the tests `a_plain_peer_serves_its_own_data_through_the_proxy` and `shutdown_ends_a_proxied_follow` are `#[ignore]`d with this reason and state the contract (proxied units equal a direct call to the peer; the proxied follow ends on shutdown). Removing the attribute is the acceptance check for this fix. It also blocks the proxy subtest of `checks.vm-peers` (`nix/tests/peers.nix`), which expects `200` from `GET /api/v1/peers/beta/systemd/units`.
|
||||
- Fix, one place: prefix the path the transport sees with `/api/v1` in `proxy::get` (`path: format!("/api/v1{path}")` before building `PeerRequest`), keeping the stripped route for the allowlist and the forward form. Then the `proxy.rs` module doc ("the transport always sees one form" of the stripped path) and the unit tests asserting `forwarded(&fake) == "/systemd/units"` follow the new form. Doing it in the transport instead would leave the probes' `/api/v1/ping` double-prefixed, so the proxy side is the safer place.
|
||||
- Not fixed here: `src/**` is outside WS-F's Owned files.
|
||||
@@ -76,3 +76,8 @@ Shippable, documented, verified.
|
||||
- Verified here (no nix **builds** in this worktree, per AGENTS.md): `nix eval` of `nix/tests/module-peers.nix` after the rebase, 40/40 cases; `nix eval` of `checks.x86_64-linux.{module-peers,vm-peers}.drvPath` (both instantiate, so the test nodes, the module assertions and the first-class rendering are all evaluated); `nix-instantiate --parse` on `flake.nix`, `nix/tests/peers.nix`; `python3 -m py_compile` on the assembled `testScript`.
|
||||
- **For the orchestrator to run** (the nix gate): `nix flake check -L` (now includes `checks.vm-peers` and `checks.module-peers`), or `nix build .#checks.x86_64-linux.vm-peers -L` on its own.
|
||||
- **Seam, blocking for `dev-test vm peers` (WS-T owns `dev/dev-test.nu`)**: `vm-steps` builds `.#($attr).($system).($name)` with `"peers" => "tests"`, so after the move it must build `.#checks.($system).vm-peers`. Suggested change: keep `let attr = "checks"` and map the build name (`"peers" => "vm-peers"`, `_ => $name`). `dev-test ci` is unaffected (it runs `vm`), as is `nix flake check`.
|
||||
- 2026-10-20: **F3 done** (commit `F: F3 end-to-end peer tests`).
|
||||
- New `tests/peers.rs`, one integration binary: a consolidator served by the real router (`server::build_router` + `server::serve_listener`) with the real `PeerRegistry`/`ReqwestTransport`, against peers that are either another in-process `slinuxm` (fake sources, real auth with a minted token) or a hand-written listener for the cases a real peer cannot produce. Token files are 0600 temporaries in `$TMPDIR`; `tests/common/mod.rs` is reused for `request`/`parse_response` and not modified (its items would be dead code in the other test binaries).
|
||||
- Cases: plain peer (status `server.up`/`api.up`/`auth ok`, `peer_version`, list shape, 401 without a caller token, 404 for an unknown peer, 405 on POST, both checks flipping when the peer dies, proxy 503 naming the peer while the caller's token keeps working); TLS peer with an `rcgen` CA and a leaf for `127.0.0.1` via `ca_file` (`api.up` needs a verified handshake and our token); a peer signed by a different CA (`server.up` true, `api.up` false, `last_error = tls_error`, proxy 503 `tls_error`); a `302` (503 `http_302`, and the `Location` target sees **zero** connections); an oversize body (503 `oversize` with `max_response_bytes = 64`); a silent peer (503 `timeout`, bounded); the 64-peer cap (`Config::validate`, the path `serve` uses: 64 accepted, 65 refused); and shutdown ending a proxied follow.
|
||||
- **Blocking finding, reported to WS-C's Log (2026-10-20)**: the proxy hands the transport the stripped route (`/systemd/units`) while peers serve `/api/v1/systemd/units`, so every proxied call to a healthy peer answers `peer <name>: not found on peer`. Health checks are fine (`/api/v1/ping`). Two tests (`a_plain_peer_serves_its_own_data_through_the_proxy`, `shutdown_ends_a_proxied_follow`) are `#[ignore]`d with that reason and state the contract; removing the attribute is the acceptance check once C (or B) prefixes the path. The same defect will fail the proxy subtest of `checks.vm-peers`.
|
||||
- Verified: `cargo test --test peers` → 7 passed, 2 ignored.
|
||||
+676
@@ -0,0 +1,676 @@
|
||||
//! End-to-end peer behaviour: a consolidator served by the real router and
|
||||
//! registry, with each peer served in-process by another `slinuxm` router
|
||||
//! (fake sources) or by a hand-written listener when the peer has to misbehave.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::atomic::{AtomicUsize, Ordering};
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
use skyg_linux_tracker::auth::tokens;
|
||||
use skyg_linux_tracker::config::Config;
|
||||
use skyg_linux_tracker::peers::registry::PeerRegistry;
|
||||
use skyg_linux_tracker::peers::types::{AuthState, PeerConfig, PeerName, PeerStatus, PeersConfig};
|
||||
use skyg_linux_tracker::server;
|
||||
use skyg_linux_tracker::sources::fake::fake_state;
|
||||
use skyg_linux_tracker::tls;
|
||||
use sqlx::sqlite::SqlitePoolOptions;
|
||||
use tokio::io::{AsyncReadExt, AsyncWriteExt};
|
||||
use tokio::net::TcpListener;
|
||||
use tokio::task::JoinHandle;
|
||||
use tokio_util::sync::CancellationToken;
|
||||
|
||||
mod common;
|
||||
|
||||
/// Deadline for one proxied request; well above the configured timeouts.
|
||||
const EXCHANGE: Duration = Duration::from_secs(20);
|
||||
/// Deadline for a peer to flip its status.
|
||||
const FLIP: Duration = Duration::from_secs(10);
|
||||
|
||||
struct TempDir(PathBuf);
|
||||
|
||||
impl TempDir {
|
||||
fn new(tag: &str) -> Self {
|
||||
static N: AtomicUsize = AtomicUsize::new(0);
|
||||
let p = std::env::temp_dir().join(format!(
|
||||
"slinuxm-peers-e2e-{tag}-{}-{}",
|
||||
std::process::id(),
|
||||
N.fetch_add(1, Ordering::SeqCst)
|
||||
));
|
||||
std::fs::create_dir_all(&p).unwrap();
|
||||
Self(p)
|
||||
}
|
||||
|
||||
/// A token file as a credential manager leaves it: private to this user.
|
||||
fn token(&self, name: &str, contents: &str) -> PathBuf {
|
||||
use std::io::Write;
|
||||
use std::os::unix::fs::OpenOptionsExt;
|
||||
let path = self.0.join(name);
|
||||
std::fs::OpenOptions::new()
|
||||
.write(true)
|
||||
.create(true)
|
||||
.truncate(true)
|
||||
.mode(0o600)
|
||||
.open(&path)
|
||||
.unwrap()
|
||||
.write_all(contents.as_bytes())
|
||||
.unwrap();
|
||||
path
|
||||
}
|
||||
|
||||
fn file(&self, name: &str, contents: &str) -> PathBuf {
|
||||
let path = self.0.join(name);
|
||||
std::fs::write(&path, contents).unwrap();
|
||||
path
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for TempDir {
|
||||
fn drop(&mut self) {
|
||||
let _ = std::fs::remove_dir_all(&self.0);
|
||||
}
|
||||
}
|
||||
|
||||
/// One request over a fresh connection, read to EOF.
|
||||
async fn send(port: u16, raw: &str) -> (u16, String, String) {
|
||||
let mut sock = tokio::net::TcpStream::connect(("127.0.0.1", port))
|
||||
.await
|
||||
.unwrap();
|
||||
sock.write_all(raw.as_bytes()).await.unwrap();
|
||||
let mut buf = Vec::new();
|
||||
tokio::time::timeout(EXCHANGE, sock.read_to_end(&mut buf))
|
||||
.await
|
||||
.expect("request did not finish")
|
||||
.unwrap();
|
||||
common::parse_response(&buf)
|
||||
}
|
||||
|
||||
fn get_raw(path: &str, token: &str, accept: Option<&str>) -> String {
|
||||
let auth = format!("Bearer {token}");
|
||||
let mut headers = vec![("Authorization", auth.as_str())];
|
||||
if let Some(accept) = accept {
|
||||
headers.push(("Accept", accept));
|
||||
}
|
||||
common::request("GET", path, "localhost", &headers, None)
|
||||
}
|
||||
|
||||
/// A second `slinuxm` acting as the peer: real router, fake sources.
|
||||
struct Peer {
|
||||
port: u16,
|
||||
/// The token this peer accepts.
|
||||
token: String,
|
||||
shutdown: CancellationToken,
|
||||
join: JoinHandle<anyhow::Result<()>>,
|
||||
}
|
||||
|
||||
impl Peer {
|
||||
async fn start() -> Self {
|
||||
Self::spawn(None).await
|
||||
}
|
||||
|
||||
async fn start_tls(cert: &Path, key: &Path) -> Self {
|
||||
Self::spawn(Some((cert, key))).await
|
||||
}
|
||||
|
||||
async fn spawn(pair: Option<(&Path, &Path)>) -> Self {
|
||||
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let port = listener.local_addr().unwrap().port();
|
||||
let mut config = Config::default();
|
||||
config.server.port = port;
|
||||
config.auth.enabled = true;
|
||||
if let Some((cert, key)) = pair {
|
||||
config.server.tls.enabled = true;
|
||||
config.server.tls.cert_file = cert.to_path_buf();
|
||||
config.server.tls.key_file = key.to_path_buf();
|
||||
}
|
||||
let mut state = fake_state(config);
|
||||
let pool = SqlitePoolOptions::new()
|
||||
.max_connections(1)
|
||||
.connect("sqlite::memory:")
|
||||
.await
|
||||
.unwrap();
|
||||
skyg_linux_tracker::db::migrate(&pool).await.unwrap();
|
||||
let token = tokens::create(&pool, "consolidator").await.unwrap();
|
||||
state.db = Some(pool);
|
||||
let acceptor = pair.map(|_| {
|
||||
let cert = tls::ReloadableCert::load(&state.config.server.tls).unwrap();
|
||||
tls::acceptor(cert).unwrap()
|
||||
});
|
||||
let shutdown = state.shutdown.clone();
|
||||
let join = tokio::spawn(server::serve_listener(
|
||||
listener,
|
||||
acceptor,
|
||||
server::build_router(state),
|
||||
shutdown.clone(),
|
||||
server::Limits::default(),
|
||||
));
|
||||
Self {
|
||||
port,
|
||||
token,
|
||||
shutdown,
|
||||
join,
|
||||
}
|
||||
}
|
||||
|
||||
fn url(&self, scheme: &str) -> String {
|
||||
format!("{scheme}://127.0.0.1:{}", self.port)
|
||||
}
|
||||
|
||||
fn token_file(&self, dir: &TempDir) -> PathBuf {
|
||||
dir.token("peer-token", &format!("{}\n", self.token))
|
||||
}
|
||||
|
||||
async fn stop(self) {
|
||||
self.shutdown.cancel();
|
||||
let _ = self.join.await;
|
||||
}
|
||||
}
|
||||
|
||||
/// The consolidator under test: real registry, real router, own caller token.
|
||||
struct Consolidator {
|
||||
port: u16,
|
||||
token: String,
|
||||
registry: Arc<PeerRegistry>,
|
||||
shutdown: CancellationToken,
|
||||
join: JoinHandle<anyhow::Result<()>>,
|
||||
}
|
||||
|
||||
impl Consolidator {
|
||||
async fn start(peers: PeersConfig) -> Self {
|
||||
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let port = listener.local_addr().unwrap().port();
|
||||
let mut config = Config::default();
|
||||
config.server.port = port;
|
||||
config.auth.enabled = true;
|
||||
config.peers = peers.clone();
|
||||
let mut state = fake_state(config);
|
||||
let pool = SqlitePoolOptions::new()
|
||||
.max_connections(1)
|
||||
.connect("sqlite::memory:")
|
||||
.await
|
||||
.unwrap();
|
||||
skyg_linux_tracker::db::migrate(&pool).await.unwrap();
|
||||
let token = tokens::create(&pool, "caller").await.unwrap();
|
||||
state.db = Some(pool);
|
||||
let shutdown = state.shutdown.clone();
|
||||
let registry = PeerRegistry::start(peers, shutdown.clone()).unwrap();
|
||||
state.peers = Some(registry.clone());
|
||||
let join = tokio::spawn(server::serve_listener(
|
||||
listener,
|
||||
None,
|
||||
server::build_router(state),
|
||||
shutdown.clone(),
|
||||
server::Limits::default(),
|
||||
));
|
||||
Self {
|
||||
port,
|
||||
token,
|
||||
registry,
|
||||
shutdown,
|
||||
join,
|
||||
}
|
||||
}
|
||||
|
||||
/// One GET with the caller's own token unless `token` is given.
|
||||
async fn get(&self, path: &str, token: Option<&str>) -> (u16, String, String) {
|
||||
let raw = get_raw(path, token.unwrap_or(&self.token), None);
|
||||
send(self.port, &raw).await
|
||||
}
|
||||
|
||||
async fn wait_status(&self, peer: &str, ok: impl Fn(&PeerStatus) -> bool) -> PeerStatus {
|
||||
let deadline = tokio::time::Instant::now() + FLIP;
|
||||
loop {
|
||||
let status = self.registry.status(peer).unwrap();
|
||||
if ok(&status) {
|
||||
return status;
|
||||
}
|
||||
assert!(
|
||||
tokio::time::Instant::now() < deadline,
|
||||
"{peer} stayed {status:?}"
|
||||
);
|
||||
tokio::time::sleep(Duration::from_millis(50)).await;
|
||||
}
|
||||
}
|
||||
|
||||
async fn stop(self) {
|
||||
self.shutdown.cancel();
|
||||
let _ = self.join.await;
|
||||
self.registry.shutdown().await;
|
||||
}
|
||||
}
|
||||
|
||||
fn peers_config(entries: &[(&str, String, Option<PathBuf>)]) -> PeersConfig {
|
||||
PeersConfig {
|
||||
enabled: true,
|
||||
tcp_interval_secs: 2,
|
||||
tcp_timeout_secs: 1,
|
||||
api_interval_secs: 2,
|
||||
api_timeout_secs: 1,
|
||||
fail_threshold: 1,
|
||||
request_timeout_secs: 2,
|
||||
peer: entries
|
||||
.iter()
|
||||
.map(|(name, url, ca)| PeerConfig {
|
||||
name: PeerName::parse(name).unwrap(),
|
||||
url: url::Url::parse(url).unwrap(),
|
||||
token_file: PathBuf::from(format!("/run/credentials/peer-{name}")),
|
||||
ca_file: ca.clone(),
|
||||
allow_plain_http: true,
|
||||
enabled: true,
|
||||
})
|
||||
.collect(),
|
||||
..PeersConfig::default()
|
||||
}
|
||||
}
|
||||
|
||||
/// One peer entry, with its token file replaced by `path`.
|
||||
fn one_peer(name: &str, url: String, ca: Option<PathBuf>, path: &Path) -> PeersConfig {
|
||||
let mut cfg = peers_config(&[(name, url, ca)]);
|
||||
cfg.peer[0].token_file = path.to_path_buf();
|
||||
cfg
|
||||
}
|
||||
|
||||
fn json(body: &str) -> serde_json::Value {
|
||||
serde_json::from_str(body).unwrap_or_else(|e| panic!("not JSON ({e}): {body}"))
|
||||
}
|
||||
|
||||
/// A CA plus a leaf the consolidator can be told to trust.
|
||||
struct Ca {
|
||||
params: rcgen::CertificateParams,
|
||||
key: rcgen::KeyPair,
|
||||
cert_pem: String,
|
||||
}
|
||||
|
||||
impl Ca {
|
||||
fn new(name: &str) -> Self {
|
||||
let mut params = rcgen::CertificateParams::new(Vec::new()).unwrap();
|
||||
params.is_ca = rcgen::IsCa::Ca(rcgen::BasicConstraints::Unconstrained);
|
||||
params
|
||||
.distinguished_name
|
||||
.push(rcgen::DnType::CommonName, name);
|
||||
let key = rcgen::KeyPair::generate().unwrap();
|
||||
let cert_pem = params.self_signed(&key).unwrap().pem();
|
||||
Self {
|
||||
params,
|
||||
key,
|
||||
cert_pem,
|
||||
}
|
||||
}
|
||||
|
||||
/// Writes the CA and a leaf for 127.0.0.1 into `dir`; returns
|
||||
/// (ca, cert, key) paths. Consumes the CA: it holds the signing key.
|
||||
fn leaf_for_loopback(self, dir: &TempDir, tag: &str) -> (PathBuf, PathBuf, PathBuf) {
|
||||
let params = rcgen::CertificateParams::new(vec!["127.0.0.1".to_owned()]).unwrap();
|
||||
let key = rcgen::KeyPair::generate().unwrap();
|
||||
let issuer = rcgen::Issuer::new(self.params, self.key);
|
||||
let cert = params.signed_by(&key, &issuer).unwrap();
|
||||
(
|
||||
dir.file(&format!("{tag}-ca.pem"), &self.cert_pem),
|
||||
dir.file(&format!("{tag}-cert.pem"), &cert.pem()),
|
||||
dir.file(&format!("{tag}-key.pem"), &key.serialize_pem()),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
/// A hand-written listener for the cases a real `slinuxm` cannot produce.
|
||||
///
|
||||
/// `respond` returns the raw response for each accepted connection, or
|
||||
/// `None` to accept and answer nothing.
|
||||
struct Listener {
|
||||
port: u16,
|
||||
hits: Arc<AtomicUsize>,
|
||||
stop: CancellationToken,
|
||||
}
|
||||
|
||||
impl Listener {
|
||||
async fn start<F>(respond: F) -> Self
|
||||
where
|
||||
F: Fn() -> Option<Vec<u8>> + Send + Sync + 'static,
|
||||
{
|
||||
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let port = listener.local_addr().unwrap().port();
|
||||
let stop = CancellationToken::new();
|
||||
let hits = Arc::new(AtomicUsize::new(0));
|
||||
let respond = Arc::new(respond);
|
||||
let counter = hits.clone();
|
||||
let cancelled = stop.clone();
|
||||
tokio::spawn(async move {
|
||||
let mut held: Vec<tokio::net::TcpStream> = Vec::new();
|
||||
loop {
|
||||
let accepted = tokio::select! {
|
||||
_ = cancelled.cancelled() => return,
|
||||
accepted = listener.accept() => accepted,
|
||||
};
|
||||
let Ok((mut sock, _)) = accepted else {
|
||||
return;
|
||||
};
|
||||
counter.fetch_add(1, Ordering::SeqCst);
|
||||
let Some(body) = respond() else {
|
||||
held.push(sock);
|
||||
continue;
|
||||
};
|
||||
tokio::spawn(async move {
|
||||
// Read the head first so the client is not reset mid-send.
|
||||
let mut buf = [0u8; 1024];
|
||||
let _ =
|
||||
tokio::time::timeout(Duration::from_millis(500), sock.read(&mut buf)).await;
|
||||
let _ = sock.write_all(&body).await;
|
||||
});
|
||||
}
|
||||
});
|
||||
Self { port, hits, stop }
|
||||
}
|
||||
|
||||
fn hits(&self) -> usize {
|
||||
self.hits.load(Ordering::SeqCst)
|
||||
}
|
||||
|
||||
fn url(&self, scheme: &str) -> String {
|
||||
format!("{scheme}://127.0.0.1:{}", self.port)
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for Listener {
|
||||
fn drop(&mut self) {
|
||||
self.stop.cancel();
|
||||
}
|
||||
}
|
||||
|
||||
fn response(status: &str, extra: &str, body: &[u8]) -> Vec<u8> {
|
||||
let mut out = format!(
|
||||
"HTTP/1.1 {status}\r\nContent-Length: {}\r\nConnection: close\r\n{extra}\r\n",
|
||||
body.len()
|
||||
)
|
||||
.into_bytes();
|
||||
out.extend_from_slice(body);
|
||||
out
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_plain_peer_reports_up_and_auth_ok() {
|
||||
let dir = TempDir::new("plain");
|
||||
let peer = Peer::start().await;
|
||||
let token = peer.token_file(&dir);
|
||||
let consolidator =
|
||||
Consolidator::start(one_peer("beta", peer.url("http"), None, &token)).await;
|
||||
|
||||
let mut status = consolidator
|
||||
.wait_status("beta", |s| s.server.up && s.api.up)
|
||||
.await;
|
||||
assert_eq!(status.api.auth, AuthState::Ok);
|
||||
assert!(status.api.peer_version.is_some(), "{status:?}");
|
||||
|
||||
let (code, _, body) = consolidator.get("/api/v1/peers", None).await;
|
||||
assert_eq!(code, 200);
|
||||
let listed = json(&body);
|
||||
assert_eq!(listed["peers"][0]["name"], "beta");
|
||||
assert_eq!(listed["peers"][0]["api"]["auth"], "ok");
|
||||
assert_eq!(listed["peers"][0]["url"], peer.url("http"));
|
||||
|
||||
// The caller still needs a token of this host, and unknown names are
|
||||
// local 404s.
|
||||
let (code, _, _) = consolidator
|
||||
.get("/api/v1/peers/beta/systemd/units", Some("wrong"))
|
||||
.await;
|
||||
assert_eq!(code, 401);
|
||||
let (code, _, body) = consolidator.get("/api/v1/peers/nope", None).await;
|
||||
assert_eq!(code, 404, "{body}");
|
||||
assert!(body.contains("unknown peer"), "{body}");
|
||||
|
||||
let auth = format!("Bearer {}", consolidator.token);
|
||||
let raw = common::request(
|
||||
"POST",
|
||||
"/api/v1/peers/beta/systemd/units",
|
||||
"localhost",
|
||||
&[("Authorization", &auth)],
|
||||
Some("{}"),
|
||||
);
|
||||
assert_eq!(send(consolidator.port, &raw).await.0, 405);
|
||||
|
||||
// A dead peer flips both checks; the proxy fails with the peer's name and
|
||||
// the caller's own token keeps working.
|
||||
peer.stop().await;
|
||||
status = consolidator
|
||||
.wait_status("beta", |s| !s.server.up && !s.api.up)
|
||||
.await;
|
||||
assert_eq!(status.api.auth, AuthState::Unknown);
|
||||
let (code, _, body) = consolidator
|
||||
.get("/api/v1/peers/beta/systemd/units", None)
|
||||
.await;
|
||||
assert_eq!(code, 503, "{body}");
|
||||
assert!(body.contains("beta"), "{body}");
|
||||
assert_eq!(consolidator.get("/api/v1/systemd/units", None).await.0, 200);
|
||||
consolidator.stop().await;
|
||||
}
|
||||
|
||||
// Blocked by a defect outside this workstream: the proxy hands the transport
|
||||
// the stripped path ("/systemd/units") while an unmodified peer only serves
|
||||
// "/api/v1/systemd/units", so every proxied call to a healthy peer answers
|
||||
// "peer beta: not found on peer". The test states the contract; the Log entry
|
||||
// in plan/phase-5-consolidator/ws-f-nixos-docs.md has the details to route the
|
||||
// one-line fix to the owning stream.
|
||||
#[ignore = "blocked: the proxy drops the /api/v1 prefix of the forwarded path"]
|
||||
#[tokio::test]
|
||||
async fn a_plain_peer_serves_its_own_data_through_the_proxy() {
|
||||
let dir = TempDir::new("proxy");
|
||||
let peer = Peer::start().await;
|
||||
let token = peer.token_file(&dir);
|
||||
let consolidator =
|
||||
Consolidator::start(one_peer("beta", peer.url("http"), None, &token)).await;
|
||||
consolidator
|
||||
.wait_status("beta", |s| s.server.up && s.api.up)
|
||||
.await;
|
||||
|
||||
// The proxied answer is the peer's own: compare it with a direct call.
|
||||
let raw = get_raw("/api/v1/systemd/units", &peer.token, None);
|
||||
let (direct_code, _, direct) = send(peer.port, &raw).await;
|
||||
assert_eq!(direct_code, 200);
|
||||
let (code, head, proxied) = consolidator
|
||||
.get("/api/v1/peers/beta/systemd/units", None)
|
||||
.await;
|
||||
assert_eq!(code, 200, "{proxied}");
|
||||
assert!(head.contains("x-slinuxm-peer: beta"), "{head}");
|
||||
assert_eq!(json(&proxied), json(&direct));
|
||||
// An allowlisted query reaches the peer too.
|
||||
let (code, _, failed) = consolidator
|
||||
.get("/api/v1/peers/beta/systemd/units?state=failed", None)
|
||||
.await;
|
||||
assert_eq!(code, 200, "{failed}");
|
||||
consolidator.stop().await;
|
||||
peer.stop().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_tls_peer_is_trusted_through_its_ca_file() {
|
||||
let dir = TempDir::new("tls");
|
||||
let ca = Ca::new("slinuxm-e2e-ca");
|
||||
let (ca_path, cert, key) = ca.leaf_for_loopback(&dir, "good");
|
||||
let peer = Peer::start_tls(&cert, &key).await;
|
||||
let token = peer.token_file(&dir);
|
||||
let consolidator = Consolidator::start(one_peer(
|
||||
"beta",
|
||||
peer.url("https"),
|
||||
Some(ca_path),
|
||||
&token,
|
||||
))
|
||||
.await;
|
||||
|
||||
// `api.up` needs a verified handshake and our token: that is the CA file
|
||||
// working end to end.
|
||||
let status = consolidator
|
||||
.wait_status("beta", |s| s.server.up && s.api.up)
|
||||
.await;
|
||||
assert_eq!(status.api.auth, AuthState::Ok);
|
||||
assert!(status.url.starts_with("https://"), "{status:?}");
|
||||
peer.stop().await;
|
||||
consolidator.stop().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_peer_signed_by_another_ca_is_a_tls_error() {
|
||||
let dir = TempDir::new("wrong-ca");
|
||||
let trusted = Ca::new("trusted-ca");
|
||||
let other = Ca::new("other-ca");
|
||||
let (trusted_ca, _, _) = trusted.leaf_for_loopback(&dir, "trusted");
|
||||
let (_, cert, key) = other.leaf_for_loopback(&dir, "other");
|
||||
let peer = Peer::start_tls(&cert, &key).await;
|
||||
let token = peer.token_file(&dir);
|
||||
let consolidator =
|
||||
Consolidator::start(one_peer("beta", peer.url("https"), Some(trusted_ca), &token)).await;
|
||||
|
||||
// The port answers, the handshake does not: a certificate fault must not
|
||||
// read as an outage.
|
||||
let status = consolidator
|
||||
.wait_status("beta", |s| s.api.last_checked.is_some() && !s.api.up)
|
||||
.await;
|
||||
assert!(status.server.up, "{status:?}");
|
||||
assert_eq!(status.api.last_error.as_deref(), Some("tls_error"));
|
||||
let (code, _, body) = consolidator
|
||||
.get("/api/v1/peers/beta/systemd/units", None)
|
||||
.await;
|
||||
assert_eq!(code, 503);
|
||||
assert!(body.contains("tls_error"), "{body}");
|
||||
assert!(body.contains("beta"), "{body}");
|
||||
peer.stop().await;
|
||||
consolidator.stop().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_redirect_is_reported_and_never_followed() {
|
||||
let dir = TempDir::new("redirect");
|
||||
let target = Listener::start(|| Some(response("200 OK", "", b"{}"))).await;
|
||||
let location = format!("{}/elsewhere", target.url("http"));
|
||||
let source = Listener::start(move || {
|
||||
Some(response(
|
||||
"302 Found",
|
||||
&format!("Location: {location}\r\n"),
|
||||
b"moved",
|
||||
))
|
||||
})
|
||||
.await;
|
||||
let token = dir.token("peer-token", "redirect-token\n");
|
||||
let consolidator =
|
||||
Consolidator::start(one_peer("beta", source.url("http"), None, &token)).await;
|
||||
|
||||
let (code, head, body) = consolidator
|
||||
.get("/api/v1/peers/beta/systemd/units", None)
|
||||
.await;
|
||||
assert_eq!(code, 503, "{body}");
|
||||
assert!(body.contains("http_302"), "{body}");
|
||||
assert!(head.contains("x-slinuxm-peer: beta"), "{head}");
|
||||
// Let the health loop probe as well, then check the Location was never
|
||||
// acted on.
|
||||
tokio::time::sleep(Duration::from_millis(500)).await;
|
||||
consolidator
|
||||
.wait_status("beta", |s| !s.api.up)
|
||||
.await;
|
||||
assert!(source.hits() > 0);
|
||||
assert_eq!(target.hits(), 0, "the redirect was followed");
|
||||
consolidator.stop().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn an_oversize_body_is_refused() {
|
||||
let dir = TempDir::new("oversize");
|
||||
let source = Listener::start(|| Some(response("200 OK", "", &[b'x'; 4096]))).await;
|
||||
let token = dir.token("peer-token", "big-token\n");
|
||||
let mut cfg = one_peer("beta", source.url("http"), None, &token);
|
||||
cfg.max_response_bytes = 64;
|
||||
let consolidator = Consolidator::start(cfg).await;
|
||||
|
||||
let (code, _, body) = consolidator
|
||||
.get("/api/v1/peers/beta/systemd/units", None)
|
||||
.await;
|
||||
assert_eq!(code, 503, "{body}");
|
||||
assert!(body.contains("oversize"), "{body}");
|
||||
consolidator.stop().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_silent_peer_times_out_within_the_request_timeout() {
|
||||
let dir = TempDir::new("slow");
|
||||
let source = Listener::start(|| None).await;
|
||||
let token = dir.token("peer-token", "slow-token\n");
|
||||
let mut cfg = one_peer("beta", source.url("http"), None, &token);
|
||||
cfg.request_timeout_secs = 1;
|
||||
let consolidator = Consolidator::start(cfg).await;
|
||||
|
||||
let started = std::time::Instant::now();
|
||||
let (code, _, body) = consolidator
|
||||
.get("/api/v1/peers/beta/systemd/units", None)
|
||||
.await;
|
||||
let elapsed = started.elapsed();
|
||||
assert_eq!(code, 503, "{body}");
|
||||
assert!(body.contains("timeout"), "{body}");
|
||||
assert!(elapsed < Duration::from_secs(10), "took {elapsed:?}");
|
||||
consolidator.stop().await;
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn at_most_64_peers_may_be_configured() {
|
||||
let peer = |i: usize| PeerConfig {
|
||||
name: PeerName::parse(&format!("peer{i}")).unwrap(),
|
||||
url: url::Url::parse(&format!("http://127.0.0.1:{}", 2000 + i)).unwrap(),
|
||||
token_file: PathBuf::from(format!("/run/credentials/peer{i}")),
|
||||
ca_file: None,
|
||||
allow_plain_http: true,
|
||||
enabled: true,
|
||||
};
|
||||
let mut config = Config::default();
|
||||
config.server.bind = "127.0.0.1".parse().unwrap();
|
||||
config.peers = PeersConfig {
|
||||
enabled: true,
|
||||
peer: (0..skyg_linux_tracker::peers::types::MAX_PEERS)
|
||||
.map(peer)
|
||||
.collect(),
|
||||
..PeersConfig::default()
|
||||
};
|
||||
config.validate().expect("64 peers are accepted");
|
||||
|
||||
config.peers.peer.push(peer(64));
|
||||
let err = config.validate().unwrap_err().to_string();
|
||||
assert!(err.contains("64"), "{err}");
|
||||
}
|
||||
|
||||
#[ignore = "blocked: the proxy drops the /api/v1 prefix of the forwarded path"]
|
||||
#[tokio::test]
|
||||
async fn shutdown_ends_a_proxied_follow() {
|
||||
let dir = TempDir::new("follow");
|
||||
let peer = Peer::start().await;
|
||||
let token = peer.token_file(&dir);
|
||||
let consolidator = Consolidator::start(one_peer("beta", peer.url("http"), None, &token)).await;
|
||||
consolidator
|
||||
.wait_status("beta", |s| s.server.up && s.api.up)
|
||||
.await;
|
||||
|
||||
let raw = get_raw(
|
||||
"/api/v1/peers/beta/journal/follow",
|
||||
&consolidator.token,
|
||||
Some("text/event-stream"),
|
||||
);
|
||||
let mut sock = tokio::net::TcpStream::connect(("127.0.0.1", consolidator.port))
|
||||
.await
|
||||
.unwrap();
|
||||
sock.write_all(raw.as_bytes()).await.unwrap();
|
||||
let mut buf = [0u8; 1024];
|
||||
let n = tokio::time::timeout(EXCHANGE, sock.read(&mut buf))
|
||||
.await
|
||||
.expect("the follow stream never started")
|
||||
.unwrap();
|
||||
assert!(n > 0);
|
||||
assert!(String::from_utf8_lossy(&buf[..n]).contains("200 OK"));
|
||||
|
||||
// The stream is open and idle on the peer; shutdown must end it.
|
||||
consolidator.shutdown.cancel();
|
||||
let mut rest = Vec::new();
|
||||
tokio::time::timeout(Duration::from_secs(10), sock.read_to_end(&mut rest))
|
||||
.await
|
||||
.expect("the proxied follow outlived shutdown")
|
||||
.unwrap();
|
||||
tokio::time::timeout(Duration::from_secs(10), consolidator.join)
|
||||
.await
|
||||
.expect("the server did not drain after shutdown")
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
peer.stop().await;
|
||||
}
|
||||
Reference in new issue
Block a user