WS-I: review fixes (connection cap, header timeout, SIGHUP on plain path, HTTP/1.1 only)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
c5972b762f
commit
9f2bbc7d14
6 files changed
+252
-18
No files matched your search
@@ -135,7 +135,7 @@ openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 -nodes -days
|
||||
curl --cacert cert.pem -H "Authorization: Bearer $TOKEN" https://localhost:9109/api/v1/systemd/units
|
||||
```
|
||||
|
||||
Rotation: replace the files and send `SIGHUP` (`kill -HUP $(pidof slinuxm)`, or `systemctl kill -s HUP <unit>`). New connections get the new certificate; established ones are not dropped. If the new pair is unreadable or mismatched, the old certificate stays and an error is logged. For ACME, point `cert_file` and `key_file` at the client's output (for example certbot's `live/<name>/fullchain.pem` and `privkey.pem`, or the NixOS `security.acme` directory) and send SIGHUP from the renewal hook. For sops-nix or systemd `LoadCredential`, point at the decrypted or credential path (`/run/credentials/<unit>/...`) so the service user needs no access to the original key file.
|
||||
Rotation: replace the files and send `SIGHUP` (`kill -HUP $(pidof slinuxm)`, or `systemctl kill -s HUP <unit>`). New connections get the new certificate; established ones are not dropped. The certificate and key are read as two separate files, so rotation tooling must send SIGHUP only after BOTH are in place; if the pair is unreadable or mismatched (for example the cert was replaced but the key not yet), the old certificate stays, an error is logged, and you send SIGHUP again once the files are consistent. Any intermediate certificates must be in the `cert_file` chain. The TLS listener is HTTP/1.1 only, caps concurrent connections at 1024 and closes connections that send no request header within 30 seconds. SIGHUP is harmless when TLS is disabled (it is logged and ignored). For ACME, point `cert_file` and `key_file` at the client's output (for example certbot's `live/<name>/fullchain.pem` and `privkey.pem`, or the NixOS `security.acme` directory) and send SIGHUP from the renewal hook. For sops-nix or systemd `LoadCredential`, point at the decrypted or credential path (`/run/credentials/<unit>/...`) so the service user needs no access to the original key file.
|
||||
|
||||
## Docker access
|
||||
|
||||
|
||||
+1
-1
@@ -58,4 +58,4 @@ Move a card by cutting its row into another column. Keep the card's own **Status
|
||||
| M2 Surfaces | Verification #3, #4 and #5 pass | 🟨 #3 and #4 verified live on 2026-10-15; #5 (OTel smoke test) not run, no otelcol-contrib binary |
|
||||
| M3 Release | Verification #1–#6 pass. Unit file and README shipped. | ⬜ |
|
||||
| M4 NixOS | `nix flake check` passes including the VM test. `:1984` serves and a token is minted with the system-installed CLI. | ⬜ |
|
||||
| M5 TLS | HTTPS serves REST and `/mcp`, SIGHUP reloads certs, no openssl. Verified on `ws-i/tls` (unmerged). | ✅ |
|
||||
| M5 TLS | HTTPS serves REST and `/mcp`, SIGHUP reloads certs, no openssl. | 🟨 verified on ws-i/tls, not merged |
|
||||
@@ -40,3 +40,4 @@ Tick these in [`TODO.md`](../../TODO.md).
|
||||
|
||||
- 2026-10-16: Card created from design discussion.
|
||||
- 2026-10-16: Implemented on `ws-i/tls`, T1-T8 done. **TLS is optional, default off** (user requirement): with `server.tls.enabled = false` the plain path is unchanged, the cert/key paths are ignored and never read, no rustls code runs, no provider is built, MCP origins stay `http://`; env (`SKYG_SERVER__TLS__*`) and `--print-config` cover the block. Design: `src/tls.rs` uses `tokio-rustls` + `hyper-util` (auto builder, HTTP/1.1 ALPN) with an accept loop and a `ReloadableCert` (`ResolvesServerCert` behind an RwLock) so SIGHUP swaps the cert for new handshakes without touching live connections; a bad pair logs an error and keeps the old one. The `ring` provider is passed explicitly to `ServerConfig::builder_with_provider`; no process-wide default is installed (aws-lc-rs was already in the tree via reqwest/OTLP; this adds no new provider or openssl). Cargo.toml: rustls, tokio-rustls, hyper-util (separate commit), rcgen dev-dependency. Shutdown: accept stops, each connection gets `graceful_shutdown`, bounded by the existing 10s drain (test: SIGTERM with an idle keep-alive and an open SSE stream over TLS exits 0). T5 warning is logged only when TLS is off and bind is non-loopback. Tests: 3 new unit tests (config validation, env, origins) and 8 in `tests/tls.rs`. Live check (debug binary, openssl-generated certs, curl): healthz, 401/200 REST, MCP initialize over HTTPS, plain HTTP rejected, SIGHUP changed the served fingerprint, a junk cert + SIGHUP kept the rotated one, SIGTERM exit 0.
|
||||
- 2026-10-16: Review fixes. TLS listener bounded by a semaphore (`tls::Limits::max_conns`, default 1024; at the cap accepting pauses) and a 30s hyper `header_read_timeout` (covers post-handshake silence and idle keep-alive), accept-error warnings rate-limited to one per 10s, HTTP/1.1 only. Note: the auto builder's `serve_connection_with_upgrades` sniffs the protocol before hyper starts, which defeats the header timeout, so the TLS path uses `serve_connection` (no upgrades; none are used). A SIGHUP handler is now installed at the top of `serve` on both paths, so SIGHUP never kills the process; with TLS off it only logs. Cert and key are read separately on reload: rotation tooling must send SIGHUP after BOTH files are in place (a mismatch keeps the old pair; fix the files and send HUP again). Intermediates must be in the `cert_file` chain. Pre-existing and out of scope: the plain-HTTP path has no connection cap or header timeout either (axum::serve defaults), the same exhaustion exposure it had before WS-I. New tests: silent-connection header timeout, connection cap, SIGHUP with TLS off, key/cert mismatch on SIGHUP.
|
||||
+27
-3
@@ -105,6 +105,10 @@ const DRAIN_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10);
|
||||
const EXPORTER_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10);
|
||||
|
||||
async fn serve(config: Config) -> Result<()> {
|
||||
// Install SIGHUP handling before anything else: the default action would
|
||||
// terminate the process, with or without TLS.
|
||||
let hup = tokio::signal::unix::signal(tokio::signal::unix::SignalKind::hangup())
|
||||
.context("installing SIGHUP handler")?;
|
||||
let guard = telemetry::init(&config).context("initialising telemetry")?;
|
||||
let db = if config.auth.enabled {
|
||||
Some(db::open(&config).await?)
|
||||
@@ -136,8 +140,22 @@ async fn serve(config: Config) -> Result<()> {
|
||||
let exporters = telemetry::spawn_exporters(state.clone());
|
||||
let pool = state.db.clone();
|
||||
let app = server::build_router(state);
|
||||
if let Some((_, cert)) = &tls {
|
||||
tls::spawn_sighup_reload(cert.clone(), shutdown.clone());
|
||||
match &tls {
|
||||
Some((_, cert)) => tls::spawn_sighup_reload(cert.clone(), hup, shutdown.clone()),
|
||||
None => {
|
||||
// Plain path: SIGHUP is ignored (logged only), nothing to reload.
|
||||
let stop = shutdown.clone();
|
||||
let mut hup = hup;
|
||||
tokio::spawn(async move {
|
||||
loop {
|
||||
tokio::select! {
|
||||
_ = stop.cancelled() => break,
|
||||
r = hup.recv() => if r.is_none() { break },
|
||||
}
|
||||
tracing::info!("SIGHUP received; nothing to reload (TLS is disabled)");
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
let signal_token = shutdown.clone();
|
||||
@@ -152,7 +170,13 @@ async fn serve(config: Config) -> Result<()> {
|
||||
// immediately. Bound the drain so a stuck client cannot hang SIGTERM.
|
||||
let serving: std::pin::Pin<Box<dyn std::future::Future<Output = Result<()>> + Send>> = match tls
|
||||
{
|
||||
Some((acceptor, _)) => Box::pin(tls::serve_tls(listener, acceptor, app, shutdown.clone())),
|
||||
Some((acceptor, _)) => Box::pin(tls::serve_tls(
|
||||
listener,
|
||||
acceptor,
|
||||
app,
|
||||
shutdown.clone(),
|
||||
tls::Limits::default(),
|
||||
)),
|
||||
None => Box::pin(server::serve_listener(listener, app, shutdown.clone())),
|
||||
};
|
||||
tokio::pin!(serving);
|
||||
|
||||
+56
-13
@@ -13,7 +13,7 @@ use std::time::Duration;
|
||||
|
||||
use anyhow::{anyhow, Context, Result};
|
||||
use axum::Router;
|
||||
use hyper_util::rt::{TokioExecutor, TokioIo};
|
||||
use hyper_util::rt::{TokioExecutor, TokioIo, TokioTimer};
|
||||
use hyper_util::server::conn::auto::Builder;
|
||||
use hyper_util::service::TowerToHyperService;
|
||||
use rustls::crypto::ring as provider;
|
||||
@@ -22,7 +22,9 @@ use rustls::pki_types::{CertificateDer, PrivateKeyDer};
|
||||
use rustls::server::{ClientHello, ResolvesServerCert};
|
||||
use rustls::sign::CertifiedKey;
|
||||
use rustls::ServerConfig;
|
||||
use tokio::sync::Semaphore;
|
||||
use tokio::task::JoinSet;
|
||||
use tokio::time::Instant;
|
||||
use tokio_rustls::TlsAcceptor;
|
||||
use tokio_util::sync::CancellationToken;
|
||||
|
||||
@@ -106,15 +108,13 @@ pub fn acceptor(cert: Arc<ReloadableCert>) -> Result<TlsAcceptor> {
|
||||
}
|
||||
|
||||
/// Reload on every SIGHUP until `shutdown`. Failures are logged, never fatal.
|
||||
pub fn spawn_sighup_reload(cert: Arc<ReloadableCert>, shutdown: CancellationToken) {
|
||||
use tokio::signal::unix::{signal, SignalKind};
|
||||
let mut hup = match signal(SignalKind::hangup()) {
|
||||
Ok(s) => s,
|
||||
Err(e) => {
|
||||
tracing::error!(error = %e, "cannot install SIGHUP handler; TLS reload disabled");
|
||||
return;
|
||||
}
|
||||
};
|
||||
/// `hup` is installed by the caller before binding so no SIGHUP can kill the
|
||||
/// process in the meantime.
|
||||
pub fn spawn_sighup_reload(
|
||||
cert: Arc<ReloadableCert>,
|
||||
mut hup: tokio::signal::unix::Signal,
|
||||
shutdown: CancellationToken,
|
||||
) {
|
||||
tokio::spawn(async move {
|
||||
loop {
|
||||
tokio::select! {
|
||||
@@ -132,6 +132,26 @@ pub fn spawn_sighup_reload(cert: Arc<ReloadableCert>, shutdown: CancellationToke
|
||||
});
|
||||
}
|
||||
|
||||
/// Connection limits for the TLS listener.
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
pub struct Limits {
|
||||
/// Maximum concurrent connections (handshaking or established). At the
|
||||
/// cap, accepting pauses until a connection closes.
|
||||
pub max_conns: usize,
|
||||
/// Time allowed to receive a request header (also covers idle
|
||||
/// keep-alive and post-handshake silence).
|
||||
pub header_timeout: Duration,
|
||||
}
|
||||
|
||||
impl Default for Limits {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
max_conns: 1024,
|
||||
header_timeout: Duration::from_secs(30),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Serve `app` over TLS until `shutdown` is cancelled, then drain: stop
|
||||
/// accepting, ask each connection to finish gracefully, and wait. The caller
|
||||
/// bounds the drain by dropping this future (which aborts the connections).
|
||||
@@ -140,11 +160,23 @@ pub async fn serve_tls(
|
||||
acceptor: TlsAcceptor,
|
||||
app: Router,
|
||||
shutdown: CancellationToken,
|
||||
limits: Limits,
|
||||
) -> Result<()> {
|
||||
let addr = listener.local_addr().context("local_addr")?;
|
||||
tracing::info!(%addr, tls = true, "listening");
|
||||
let mut conns: JoinSet<()> = JoinSet::new();
|
||||
let slots = Arc::new(Semaphore::new(limits.max_conns));
|
||||
let mut last_accept_warn: Option<Instant> = None;
|
||||
loop {
|
||||
// Wait for a free slot first, so at the cap we stop accepting.
|
||||
let permit = tokio::select! {
|
||||
_ = shutdown.cancelled() => break,
|
||||
Some(_) = conns.join_next(), if !conns.is_empty() => continue,
|
||||
p = slots.clone().acquire_owned() => match p {
|
||||
Ok(p) => p,
|
||||
Err(_) => break,
|
||||
},
|
||||
};
|
||||
tokio::select! {
|
||||
_ = shutdown.cancelled() => break,
|
||||
Some(_) = conns.join_next(), if !conns.is_empty() => {}
|
||||
@@ -152,7 +184,10 @@ pub async fn serve_tls(
|
||||
let (tcp, peer) = match r {
|
||||
Ok(x) => x,
|
||||
Err(e) => {
|
||||
tracing::warn!(error = %e, "accept failed");
|
||||
if last_accept_warn.is_none_or(|t| t.elapsed() > Duration::from_secs(10)) {
|
||||
tracing::warn!(error = %e, "accept failed (further failures suppressed for 10s)");
|
||||
last_accept_warn = Some(Instant::now());
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(100)).await;
|
||||
continue;
|
||||
}
|
||||
@@ -161,6 +196,7 @@ pub async fn serve_tls(
|
||||
let svc = TowerToHyperService::new(app.clone());
|
||||
let stop = shutdown.clone();
|
||||
conns.spawn(async move {
|
||||
let _permit = permit;
|
||||
let tls = match tokio::time::timeout(HANDSHAKE_TIMEOUT, acceptor.accept(tcp)).await {
|
||||
Ok(Ok(t)) => t,
|
||||
Ok(Err(e)) => {
|
||||
@@ -172,8 +208,15 @@ pub async fn serve_tls(
|
||||
return;
|
||||
}
|
||||
};
|
||||
let builder = Builder::new(TokioExecutor::new());
|
||||
let conn = builder.serve_connection_with_upgrades(TokioIo::new(tls), svc);
|
||||
let mut builder = Builder::new(TokioExecutor::new()).http1_only();
|
||||
builder
|
||||
.http1()
|
||||
.timer(TokioTimer::new())
|
||||
.header_read_timeout(limits.header_timeout);
|
||||
// Plain serve_connection: the auto builder's _with_upgrades
|
||||
// variant sniffs the protocol first, which would defeat the
|
||||
// header timeout for silent clients.
|
||||
let conn = builder.serve_connection(TokioIo::new(tls), svc);
|
||||
tokio::pin!(conn);
|
||||
tokio::select! {
|
||||
r = conn.as_mut() => {
|
||||
|
||||
+166
@@ -497,3 +497,169 @@ async fn sigterm_with_open_tls_connections_exits_promptly() {
|
||||
tokio::time::sleep(Duration::from_millis(50)).await;
|
||||
}
|
||||
}
|
||||
|
||||
/// In-process TLS listener with explicit limits.
|
||||
async fn inproc(
|
||||
pair: &Pair,
|
||||
limits: skyg_linux_tracker::tls::Limits,
|
||||
) -> (u16, tokio_util::sync::CancellationToken, tempdir::Dir) {
|
||||
use skyg_linux_tracker::{config::Config, server, sources::fake::fake_state, tls};
|
||||
let dir = tempdir::Dir::new("inproc");
|
||||
let (c, k) = write_pair(&dir.0, pair);
|
||||
let mut cfg = Config::default();
|
||||
cfg.auth.enabled = false;
|
||||
cfg.server.tls.enabled = true;
|
||||
cfg.server.tls.cert_file = c;
|
||||
cfg.server.tls.key_file = k;
|
||||
let cert = tls::ReloadableCert::load(&cfg.server.tls).unwrap();
|
||||
let acceptor = tls::acceptor(cert).unwrap();
|
||||
let state = fake_state(cfg);
|
||||
let shutdown = state.shutdown.clone();
|
||||
let app = server::build_router(state);
|
||||
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let port = listener.local_addr().unwrap().port();
|
||||
tokio::spawn(tls::serve_tls(
|
||||
listener,
|
||||
acceptor,
|
||||
app,
|
||||
shutdown.clone(),
|
||||
limits,
|
||||
));
|
||||
(port, shutdown, dir)
|
||||
}
|
||||
|
||||
async fn handshake(
|
||||
port: u16,
|
||||
pair: &Pair,
|
||||
) -> std::io::Result<tokio_rustls::client::TlsStream<TcpStream>> {
|
||||
let tcp = TcpStream::connect(("127.0.0.1", port)).await?;
|
||||
connector(pair)
|
||||
.connect(ServerName::try_from("localhost").unwrap(), tcp)
|
||||
.await
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn silent_connection_closed_after_header_timeout() {
|
||||
let pair = gen_pair();
|
||||
let (port, shutdown, _d) = inproc(
|
||||
&pair,
|
||||
skyg_linux_tracker::tls::Limits {
|
||||
max_conns: 16,
|
||||
header_timeout: Duration::from_millis(500),
|
||||
},
|
||||
)
|
||||
.await;
|
||||
let mut c = handshake(port, &pair).await.unwrap();
|
||||
let start = Instant::now();
|
||||
let mut buf = [0u8; 256];
|
||||
let r = tokio::time::timeout(Duration::from_secs(5), async {
|
||||
loop {
|
||||
match c.read(&mut buf).await {
|
||||
Ok(0) | Err(_) => break,
|
||||
Ok(_) => {}
|
||||
}
|
||||
}
|
||||
})
|
||||
.await;
|
||||
assert!(r.is_ok(), "silent post-handshake connection was not closed");
|
||||
assert!(start.elapsed() < Duration::from_secs(4));
|
||||
shutdown.cancel();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn connection_cap_is_enforced() {
|
||||
let pair = gen_pair();
|
||||
let (port, shutdown, _d) = inproc(
|
||||
&pair,
|
||||
skyg_linux_tracker::tls::Limits {
|
||||
max_conns: 2,
|
||||
header_timeout: Duration::from_secs(30),
|
||||
},
|
||||
)
|
||||
.await;
|
||||
let c1 = handshake(port, &pair).await.unwrap();
|
||||
let c2 = handshake(port, &pair).await.unwrap();
|
||||
let third = tokio::time::timeout(Duration::from_millis(800), handshake(port, &pair)).await;
|
||||
assert!(third.is_err(), "third connection was served beyond the cap");
|
||||
drop(c1);
|
||||
let c3 = tokio::time::timeout(Duration::from_secs(5), handshake(port, &pair)).await;
|
||||
assert!(c3.is_ok(), "slot was not released after a close");
|
||||
drop(c2);
|
||||
shutdown.cancel();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sighup_with_tls_disabled_keeps_serving() {
|
||||
use std::io::{Read, Write};
|
||||
let dir = tempdir::Dir::new("plainhup");
|
||||
let port = free_port();
|
||||
let f = dir.0.join("c.toml");
|
||||
std::fs::write(
|
||||
&f,
|
||||
format!(
|
||||
"[server]\nport = {port}\n[auth]\ndb_path = {:?}\n",
|
||||
dir.0.join("db.sqlite").display().to_string()
|
||||
),
|
||||
)
|
||||
.unwrap();
|
||||
let mut child = Command::new(BIN)
|
||||
.args(["serve", "--config"])
|
||||
.arg(&f)
|
||||
.env_clear()
|
||||
.stderr(Stdio::null())
|
||||
.spawn()
|
||||
.unwrap();
|
||||
let get = || -> Option<String> {
|
||||
let mut c = std::net::TcpStream::connect(("127.0.0.1", port)).ok()?;
|
||||
c.write_all(b"GET /healthz HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\n\r\n")
|
||||
.ok()?;
|
||||
let mut s = String::new();
|
||||
let _ = c.read_to_string(&mut s);
|
||||
Some(s)
|
||||
};
|
||||
for _ in 0..100 {
|
||||
if get().is_some_and(|s| s.contains("200 OK")) {
|
||||
break;
|
||||
}
|
||||
std::thread::sleep(Duration::from_millis(100));
|
||||
}
|
||||
signal(&child, "-HUP");
|
||||
std::thread::sleep(Duration::from_millis(500));
|
||||
assert!(
|
||||
child.try_wait().unwrap().is_none(),
|
||||
"SIGHUP killed the process"
|
||||
);
|
||||
assert!(get().is_some_and(|s| s.contains("200 OK")));
|
||||
let _ = child.kill();
|
||||
let _ = child.wait();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn mismatched_pair_on_sighup_keeps_old_cert() {
|
||||
let a = gen_pair();
|
||||
let b = gen_pair();
|
||||
let s = start(&a);
|
||||
let fp_a = https(s.port, &a, "GET", "/healthz", &[], None)
|
||||
.await
|
||||
.peer_fp;
|
||||
// New cert B with old key A: mismatch, old pair kept.
|
||||
std::fs::write(s.dir.0.join("cert.pem"), &b.cert_pem).unwrap();
|
||||
signal(&s.child, "-HUP");
|
||||
tokio::time::sleep(Duration::from_millis(500)).await;
|
||||
let r = https(s.port, &a, "GET", "/healthz", &[], None).await;
|
||||
assert_eq!(r.peer_fp, fp_a);
|
||||
// Completing the rotation (key B) and sending HUP again works.
|
||||
std::fs::write(s.dir.0.join("key.pem"), &b.key_pem).unwrap();
|
||||
signal(&s.child, "-HUP");
|
||||
let mut fp = String::new();
|
||||
for _ in 0..30 {
|
||||
tokio::time::sleep(Duration::from_millis(200)).await;
|
||||
fp = https(s.port, &b, "GET", "/healthz", &[], None)
|
||||
.await
|
||||
.peer_fp;
|
||||
if fp != fp_a {
|
||||
break;
|
||||
}
|
||||
}
|
||||
assert_ne!(fp, fp_a);
|
||||
}
|
||||
Reference in new issue
Block a user