Add hardened systemd-resolved DNS with boot verification and journal recovery,
enabled only on specific hosts:
- hl-bigbox1, hl-bigbox2 (experienced DNS resilience issues)
- hl-minipc1, hl-minipc2, hl-minipc3 (minipc cluster)
- hl-fwdesk, hl-fws1 (desktop/workstation nodes)
Changes:
1. New module: modules/nixos/common/dns/default.nix
- systemd-resolved with LLMNR fallback and DNSSEC tolerance
- Hardened journald with persistent storage and auto-recovery
- Boot-time DNS verification service with 5 retries
- Systemd watchdog for automatic reboot on DNS failure
2. Default to false, enabled explicitly on target hosts only
3. Updated host configs to enable dns resilience module
The relay image is ghcr.io/block/buzz:main with autoUpdate and
BUZZ_AUTO_MIGRATE=true. On 2026-11-02 an auto-update pulled a newer build
whose migrations include 0078_artifact_sync_indexes.sql. That migration
refuses to create its four indexes inline once the target table is
populated -- it raises "prebuild <index> concurrently" and points at
docs/artifact-sync-deployment.md. channel_members is populated on a live
relay, so migration 78 aborts, the relay exits 1 and has been
crash-looping ever since. The schema sits at 0077, so every earlier
migration (including 0077_artifact_sync_timestamps) is applied; rolling
the image back is not an option because sqlx rejects a schema that is
ahead of the binary's known migrations.
Fix the gate instead of the image: a one-shot compose service,
relay-index-prebuild, runs on the postgres image after postgres reports
healthy and builds the four indexes CONCURRENTLY. Each index is checked
first -- missing, valid, or invalid (present but not
indisvalid/indisready/indislive); an invalid leftover is dropped
concurrently before the rebuild, and an already-valid index is left
alone, so re-runs are cheap and safe. The relay now waits on
relay-index-prebuild completing successfully, next to its existing
postgres/redis/minio/minio-init conditions.
Authored by hermenix.