115 Commits
Author SHA1 Message Date
Hermenix 7a973aaee6 diagnix: add diag-pstore
/sys/fs/pstore is root-only and holds the kernel's crash records from
the previous boot, the only trace a hard freeze leaves when the journal
is cut off. diag-pstore prints them via the existing NOPASSWD sudo list.
ABOUTME updated for both diagnix changes.
2026-11-20 01:34:41 +00:00
Hermenix 5c48ef9c1e diagnix: let the hermenix VM log in
hermenix had no shell route to lab hosts, so a hard freeze (bigbox1,
Nov 19 18:21 ET) could only be read through slinuxm's warning-level
journal window, which misses the info-level tail before the crash.

- hosts.nix: a dedicated hermenix-diagnix key, separate from the
  hermenix agenix key, accepted only from hl-fwdesk's addresses
  (10.0.10.11, 10.0.101.101) and the guest address 10.77.0.2.
- default.nix: keyLine takes the key; new
  skyg.nixos.common.diagnix.hermenix.enable (default true) adds the
  hermenix key line with the same restrict,pty + slice limits.
- hl-fwdesk hermenix.nix: lanAccess to 10.0.10.0/24 port 22.
2026-11-20 01:34:41 +00:00
ari f9835c150f cleaning up after my clanker 2026-11-17 14:57:04 -05:00
ari e9479f9be1 cleaning up after my clanker 2026-11-17 14:43:55 -05:00
Hermenix ff770c1d03 dns-resilience: Hardened systemd-resolved with scoped host enablement
Add hardened systemd-resolved DNS with boot verification and journal recovery,
enabled only on specific hosts:
- hl-bigbox1, hl-bigbox2 (experienced DNS resilience issues)
- hl-minipc1, hl-minipc2, hl-minipc3 (minipc cluster)
- hl-fwdesk, hl-fws1 (desktop/workstation nodes)

Changes:
1. New module: modules/nixos/common/dns/default.nix
   - systemd-resolved with LLMNR fallback and DNSSEC tolerance
   - Hardened journald with persistent storage and auto-recovery
   - Boot-time DNS verification service with 5 retries
   - Systemd watchdog for automatic reboot on DNS failure

2. Default to false, enabled explicitly on target hosts only
3. Updated host configs to enable dns resilience module
2026-11-17 18:25:20 +00:00
ariandnBot 10d85b2a91 Push builds to bigbox2 over ssh-ng from a background queue
bigbox2 runs nix-serve, which only serves its own store: every
'nix copy --to http://bigbox2.lab.internal:5000' from the post-build hook
ended in HTTP 404 after compressing the whole closure, so nothing ever
landed and the next build tried the same paths again. A post-build-hook
also blocks the build until it returns, which held up every build on
fwdesk (nbot's VM checks spent most of a cold CI run there).

- cacheReceive (on bigbox2): nix.sshServe with write over ssh-ng, for
  fwdesk's ssh host key; nix-ssh is trusted, and nix-serve signs the
  pushed paths when it serves them.
- cachePush: the hook only queues $OUT_PATHS. nix-cache-push.service
  drains the queue with nix copy to ssh-ng://nix-ssh@bigbox2.lab.internal
  as the host key, with bigbox2's host key pinned; a path unit starts it,
  and a failed push is retried every 10 minutes by a timer.

Co-Authored-By: nBot
2026-11-12 10:12:07 -05:00
ari 09c6fe2235 debug agent 2026-11-11 17:02:36 -05:00
Hermenix a8bc08de43 container-services: queue the env-reload restart instead of blocking on it
The env-reload unit is a oneshot whose ExecStart was a blocking
systemctl restart of the group unit. When the stack's cold start failed
transiently (container-services-buzz on minipc1, where buzz-relay-1 came
up unhealthy and compose exited 1), the group unit recovered on its own a
few seconds later through its own Restart=on-failure, but the reload unit
had already exited 1 with no restart policy and stayed failed until the
next env file write.

The trigger unit should queue the restart, not own the health of the whole
stack: pass --no-block. Real failures still surface on the group unit,
which carries Restart=on-failure and the start limit and is what the fleet
scan reports.
2026-11-09 15:04:30 +00:00
ari 9856e11416 clean up lid watchers 2026-10-29 14:18:27 -04:00
ari fd733cdc21 Genix rebuilds + hyprland version fix 2026-10-29 14:09:21 -04:00
ari 39f062a7af safeguarding k3s & removing fwmgr from machines that have hardware with no upstream updates 2026-10-21 14:40:10 -04:00
ari e719674851 killing exporters 2026-10-16 16:59:09 -04:00
ari 1e6cf7251f full deprecation 2026-10-16 15:07:11 -04:00
ari 17b14b33e5 making sure files will cause a restart 2026-10-03 00:57:28 -04:00
ari 4f396d93c4 new dns routing 2026-09-29 11:23:25 -04:00
ari 045702e909 dns refactor 2026-09-26 02:41:43 -04:00
ari 09e2e03423 single file for dns 2026-09-26 01:47:42 -04:00
ari f1e71fa1e9 configured portainer 2026-09-22 22:21:12 -04:00
ari cbba7075a7 fixing ip routing for tailscale outside 2026-09-22 14:29:50 -04:00
ari 7d10ad7d61 fixing some minor things 2026-09-22 11:13:58 -04:00
ari f1d907a438 bye ssh notify and hi buzz 2026-09-21 12:32:13 -04:00
ari f8444b1549 dns + keybinding consolidation 2026-09-21 10:52:32 -04:00
ari df46627173 some security patches + buzz updates 2026-09-20 20:55:17 -04:00
ari 7a3d64e645 clean up 2026-09-20 20:32:33 -04:00
ari 8cb5978f3a fixing podmand virt 2026-09-20 20:20:25 -04:00
ari beb83f21b3 buz support 2026-09-20 20:13:39 -04:00
ari 5d727c26cd suspend fixes 2026-09-20 17:16:04 -04:00
ari 2563492fa4 new ds4 from custom fork 2026-09-18 21:10:19 -04:00
ari a781b759f4 refactor of ssh-notify 2026-09-18 16:34:31 -04:00
ari a7b9273296 new static ip and merge approach 2026-09-16 22:02:04 -04:00
ari 7cd2b2ca20 macvlan 2026-09-16 16:18:25 -04:00
ari bdfcb9e814 clean up & consoldiation of custom pkgs from flakes & herdr 2026-09-09 11:46:46 -04:00
ari be306f2e30 bigbox2: fixed incorrect boot image + added overrides for cache dump 2026-09-02 14:22:29 -04:00
ari 6dd651d156 bigbox2 as cache machine approach #1 2026-09-02 13:37:09 -04:00
ari 3c86efa4ff clean up & hyprland experiments
- Cleaned up docs
- removed hyprglass
- added various ways to do scroll view
2026-08-28 12:58:27 -04:00
ari 2a2bdd0b4b moving minipc2 services to nix 2026-08-27 15:15:06 -04:00
ari 92d10f3f09 colibri 2026-08-07 17:08:22 -04:00
ari ad0f2e5459 simpler fusuma 2026-07-28 13:43:20 -04:00
ari 464477cfe7 updated niri settings 2026-07-28 13:18:43 -04:00
ari e5c264d3ab compose fix + boot all script 2026-07-19 09:26:06 -04:00
ari 0fa5c3bc68 ssh libnotify forward 2026-07-19 09:25:55 -04:00
ari 26bcf95f30 ssh libnotify forward 2026-07-19 09:14:01 -04:00
ari 7a0a9e6091 added max settings to ssh 2026-07-18 19:50:11 -04:00
ari 916799509c latest container work 2026-07-17 20:27:21 -04:00
ari 3d20e0f709 fixing containers 2026-07-15 17:46:01 -04:00
ari 1e8cf01461 initial work on iu + more compose refactoring 2026-07-13 22:25:23 -04:00
ari 10958450ba getting temp under control 2026-07-11 19:48:23 -04:00
ari 896d7480c1 autoupdates 2026-07-09 01:02:10 -04:00
ari c82088b35a cleaner volumes 2026-07-08 22:22:43 -04:00
ari 5242850fbc missing defaults 2026-07-01 18:07:52 -04:00