Commit Graph
1080 Commits
Author SHA1 Message Date
ari ce49e41635 bumps 2026-11-22 15:36:30 -05:00
ari 519ccbe76c Merge branch 'main' of gitea.app.internal:ari/nixos-setup 2026-11-22 10:37:21 -05:00
ari 3def9ce938 Merge pull request 'hermenix: stack2 — wait for healthy gluetun before qbittorrent' (#14) from hermes/nixos-setup:hermenix/stack2-gluetun-health into main
Reviewed-on: http://gitea.app.internal:3000/ari/nixos-setup/pulls/14
2026-11-22 15:37:14 +00:00
ari 2cdf089d75 Merge branch 'main' of gitea.app.internal:ari/nixos-setup 2026-11-22 10:31:51 -05:00
ari e0d0f0d155 fwbook fixes + nbot bump; 2026-11-22 10:31:39 -05:00
Hermenix 2e540a2224 hl-bigbox1: cap CPU idle at C1
bigbox1 (Ryzen 5 4600G, B550 GAMING X V2, BIOS FC) hard-froze 5 times
in 16 days (Nov 4 19:07, Nov 16 01:22, Nov 17 02:03, Nov 17 22:59,
Nov 19 18:20 ET), always while idle, each time with no log line and an
"uncleanly shut down" journal on the next boot. That matches the known
Ryzen deep-idle (C6) freeze.

acpi_idle exposes C1 (MWAIT), C2 and C3 (lat 350us, the deep state);
processor.max_cstate=1 keeps cores out of C2/C3. Cost: a few watts
more at idle.

The runtime watchdog (common.dns, SP5100 TCO, 30s) was armed during the
Nov 19 boot and did not reset the box, so it is not a safety net here
and is left as is.
2026-11-21 02:58:14 +00:00
ari 89484504e4 unami 2026-11-20 16:29:54 -05:00
ari 6fbc61c7ea slack bump 2026-11-20 16:08:51 -05:00
ari 1dc3ae47f9 Serve the LoopRail site from Caddy on minipc3 2026-11-20 15:09:26 -05:00
ari a092e23f6e bumps 2026-11-19 21:47:48 -05:00
Hermenix 7a973aaee6 diagnix: add diag-pstore
/sys/fs/pstore is root-only and holds the kernel's crash records from
the previous boot, the only trace a hard freeze leaves when the journal
is cut off. diag-pstore prints them via the existing NOPASSWD sudo list.
ABOUTME updated for both diagnix changes.
2026-11-20 01:34:41 +00:00
Hermenix 5c48ef9c1e diagnix: let the hermenix VM log in
hermenix had no shell route to lab hosts, so a hard freeze (bigbox1,
Nov 19 18:21 ET) could only be read through slinuxm's warning-level
journal window, which misses the info-level tail before the crash.

- hosts.nix: a dedicated hermenix-diagnix key, separate from the
  hermenix agenix key, accepted only from hl-fwdesk's addresses
  (10.0.10.11, 10.0.101.101) and the guest address 10.77.0.2.
- default.nix: keyLine takes the key; new
  skyg.nixos.common.diagnix.hermenix.enable (default true) adds the
  hermenix key line with the same restrict,pty + slice limits.
- hl-fwdesk hermenix.nix: lanAccess to 10.0.10.0/24 port 22.
2026-11-20 01:34:41 +00:00
hermenix 01f039a5ce stack2: wait for healthy gluetun before starting qbittorrent
qbittorrent shares gluetun's network. When it starts before the VPN
tunnel is up it never reaches the swarm, and torrents sit on
"downloading metadata" until it is restarted by hand (seen 2026-11-19:
qbit up 12:37:40, tun0 only up 12:38:57 after a failed first server).

- qbittorrent depends_on gluetun with condition: service_healthy and
  restart: true, so it waits for the tunnel and restarts with gluetun
- drop HEALTH_VPN_DURATION_INITIAL, which gluetun v3.41 reports as obsolete

Re-encrypted to the same recipients as secrets.nix: ari, minipc2, hermenix.
2026-11-19 13:06:44 +00:00
ari 4c9d8f88cb fixing stack2 & 1 2026-11-19 07:47:26 -05:00
ari c8223aec40 fixing stack2 & 1 2026-11-19 07:34:36 -05:00
ari 6abc047879 fixing stack2 2026-11-19 07:18:28 -05:00
ari a48af90346 version bumps 2026-11-18 16:14:21 -05:00
ari 32ea399d89 github runners 2026-11-18 12:50:45 -05:00
ari 705f2eb4c6 precommit 2026-11-18 12:22:04 -05:00
ari 5139a55daa version bumps 2026-11-18 12:20:19 -05:00
ari f9835c150f cleaning up after my clanker 2026-11-17 14:57:04 -05:00
ari e9479f9be1 cleaning up after my clanker 2026-11-17 14:43:55 -05:00
ari a1ebcfb6aa converting loops 2026-11-17 14:22:28 -05:00
ari 956158729e Merge branch 'main' of gitea.app.internal:ari/nixos-setup 2026-11-17 14:10:17 -05:00
ari 87aea41843 less frequent updates 2026-11-17 14:10:00 -05:00
ari f3d03f4c20 looprails 2026-11-17 13:52:08 -05:00
Hermenix ff770c1d03 dns-resilience: Hardened systemd-resolved with scoped host enablement
Add hardened systemd-resolved DNS with boot verification and journal recovery,
enabled only on specific hosts:
- hl-bigbox1, hl-bigbox2 (experienced DNS resilience issues)
- hl-minipc1, hl-minipc2, hl-minipc3 (minipc cluster)
- hl-fwdesk, hl-fws1 (desktop/workstation nodes)

Changes:
1. New module: modules/nixos/common/dns/default.nix
   - systemd-resolved with LLMNR fallback and DNSSEC tolerance
   - Hardened journald with persistent storage and auto-recovery
   - Boot-time DNS verification service with 5 retries
   - Systemd watchdog for automatic reboot on DNS failure

2. Default to false, enabled explicitly on target hosts only
3. Updated host configs to enable dns resilience module
2026-11-17 18:25:20 +00:00
ari d7875b588e multi-slackssss!! 2026-11-16 10:06:48 -05:00
ari 5881b9c8be general clean up of nbot 2026-11-15 15:15:01 -05:00
ari d7bd46c0cd clean up 2026-11-14 22:58:25 -05:00
ari b583c550f3 clean up 2026-11-14 22:57:54 -05:00
ari 08207da02b looprails 2026-11-12 17:19:14 -05:00
ari 3efec42e72 clean up 2026-11-12 16:53:01 -05:00
ari fb5503ec36 Merge remote-tracking branch 'github-hermenix/fix/cache-push-ssh' 2026-11-12 11:17:33 -05:00
ari be6e7492b3 latest nbot 2026-11-12 11:17:21 -05:00
ariandnBot 10d85b2a91 Push builds to bigbox2 over ssh-ng from a background queue
bigbox2 runs nix-serve, which only serves its own store: every
'nix copy --to http://bigbox2.lab.internal:5000' from the post-build hook
ended in HTTP 404 after compressing the whole closure, so nothing ever
landed and the next build tried the same paths again. A post-build-hook
also blocks the build until it returns, which held up every build on
fwdesk (nbot's VM checks spent most of a cold CI run there).

- cacheReceive (on bigbox2): nix.sshServe with write over ssh-ng, for
  fwdesk's ssh host key; nix-ssh is trusted, and nix-serve signs the
  pushed paths when it serves them.
- cachePush: the hook only queues $OUT_PATHS. nix-cache-push.service
  drains the queue with nix copy to ssh-ng://nix-ssh@bigbox2.lab.internal
  as the host key, with bigbox2's host key pinned; a path unit starts it,
  and a failed push is retried every 10 minutes by a timer.

Co-Authored-By: nBot
2026-11-12 10:12:07 -05:00
ari 6db82ed10a Merge branch 'main' of gitea.app.internal:ari/nixos-setup 2026-11-11 17:02:43 -05:00
ari 09c6fe2235 debug agent 2026-11-11 17:02:36 -05:00
ari 2547ec45b4 clear up 2026-11-11 12:03:09 -05:00
hermenix 26856003d4 llama-cpp: add CUDA support and bigbox1 RTX 3060 service config
- Extend pkgs/llama-cpp to support CUDA backend (backend="cuda") for NVIDIA GPUs
- Add llama-cpp-cuda variant to modules/core/default.nix overlay
- Create hosts/hl-bigbox1/services/llama-cpp.nix with RTX 3060 tuning:
  * gpuLayers=30 (conserve 12GB VRAM for context)
  * parallel=2 (handle concurrent requests on smaller VRAM)
  * Same Qwen3.8-27B + DFlash2 drafter as fwdesk
- Wire service into bigbox1 services/default.nix
- Disabled by default (weights download manual)
2026-11-11 16:02:41 +00:00
hermenix ce92096042 llama-cpp: llama-server with DFlash2 speculative decoding for gfx1151
v0.8.0 is the first upstream release with DFlash2 drafters; nixpkgs has 0.6.0 (unstable) and an older tree (stable), so the engine is built here. HIP build for Strix Halo. The NixOS module exposes llama-server behind skyg.nixos.server.services.llama-cpp; fwdesk wires it to Qwen3.8-27B + its DFlash2 drafter, disabled until the weights are on disk.
2026-11-11 16:01:32 +00:00
ari 62c41a6f8e clean up pkgs 2026-11-11 10:35:30 -05:00
ari d8ea131327 clean up pkgs 2026-11-10 16:38:18 -05:00
ari a3178f0507 latest work 2026-11-10 15:00:43 -05:00
ari 3b44ae1354 workflow bump 2026-11-10 14:47:10 -05:00
ari e72789fbb1 bump the version 2026-11-10 10:35:09 -05:00
ari f06921593d Merge branch 'main' of gitea.app.internal:ari/nixos-setup 2026-11-10 10:04:27 -05:00
ari 416c648f99 local fixes 2026-11-10 10:03:09 -05:00
Hermenix 25a21d2f66 nbot: bump to c2491e9 (4fdd292 → c2491e9)
Consolidate environmentFile handling in both hosts via nbot module.
2026-11-10 14:55:28 +00:00
ari dfd4ca06a2 registering nbot wf 2026-11-09 16:13:26 -05:00