Commit Graph
1068 Commits
Author SHA1 Message Date
hermenix 01f039a5ce stack2: wait for healthy gluetun before starting qbittorrent
qbittorrent shares gluetun's network. When it starts before the VPN
tunnel is up it never reaches the swarm, and torrents sit on
"downloading metadata" until it is restarted by hand (seen 2026-11-19:
qbit up 12:37:40, tun0 only up 12:38:57 after a failed first server).

- qbittorrent depends_on gluetun with condition: service_healthy and
  restart: true, so it waits for the tunnel and restarts with gluetun
- drop HEALTH_VPN_DURATION_INITIAL, which gluetun v3.41 reports as obsolete

Re-encrypted to the same recipients as secrets.nix: ari, minipc2, hermenix.
2026-11-19 13:06:44 +00:00
ari 4c9d8f88cb fixing stack2 & 1 2026-11-19 07:47:26 -05:00
ari c8223aec40 fixing stack2 & 1 2026-11-19 07:34:36 -05:00
ari 6abc047879 fixing stack2 2026-11-19 07:18:28 -05:00
ari a48af90346 version bumps 2026-11-18 16:14:21 -05:00
ari 32ea399d89 github runners 2026-11-18 12:50:45 -05:00
ari 705f2eb4c6 precommit 2026-11-18 12:22:04 -05:00
ari 5139a55daa version bumps 2026-11-18 12:20:19 -05:00
ari f9835c150f cleaning up after my clanker 2026-11-17 14:57:04 -05:00
ari e9479f9be1 cleaning up after my clanker 2026-11-17 14:43:55 -05:00
ari a1ebcfb6aa converting loops 2026-11-17 14:22:28 -05:00
ari 956158729e Merge branch 'main' of gitea.app.internal:ari/nixos-setup 2026-11-17 14:10:17 -05:00
ari 87aea41843 less frequent updates 2026-11-17 14:10:00 -05:00
ari f3d03f4c20 looprails 2026-11-17 13:52:08 -05:00
Hermenix ff770c1d03 dns-resilience: Hardened systemd-resolved with scoped host enablement
Add hardened systemd-resolved DNS with boot verification and journal recovery,
enabled only on specific hosts:
- hl-bigbox1, hl-bigbox2 (experienced DNS resilience issues)
- hl-minipc1, hl-minipc2, hl-minipc3 (minipc cluster)
- hl-fwdesk, hl-fws1 (desktop/workstation nodes)

Changes:
1. New module: modules/nixos/common/dns/default.nix
   - systemd-resolved with LLMNR fallback and DNSSEC tolerance
   - Hardened journald with persistent storage and auto-recovery
   - Boot-time DNS verification service with 5 retries
   - Systemd watchdog for automatic reboot on DNS failure

2. Default to false, enabled explicitly on target hosts only
3. Updated host configs to enable dns resilience module
2026-11-17 18:25:20 +00:00
ari d7875b588e multi-slackssss!! 2026-11-16 10:06:48 -05:00
ari 5881b9c8be general clean up of nbot 2026-11-15 15:15:01 -05:00
ari d7bd46c0cd clean up 2026-11-14 22:58:25 -05:00
ari b583c550f3 clean up 2026-11-14 22:57:54 -05:00
ari 08207da02b looprails 2026-11-12 17:19:14 -05:00
ari 3efec42e72 clean up 2026-11-12 16:53:01 -05:00
ari fb5503ec36 Merge remote-tracking branch 'github-hermenix/fix/cache-push-ssh' 2026-11-12 11:17:33 -05:00
ari be6e7492b3 latest nbot 2026-11-12 11:17:21 -05:00
ariandnBot 10d85b2a91 Push builds to bigbox2 over ssh-ng from a background queue
bigbox2 runs nix-serve, which only serves its own store: every
'nix copy --to http://bigbox2.lab.internal:5000' from the post-build hook
ended in HTTP 404 after compressing the whole closure, so nothing ever
landed and the next build tried the same paths again. A post-build-hook
also blocks the build until it returns, which held up every build on
fwdesk (nbot's VM checks spent most of a cold CI run there).

- cacheReceive (on bigbox2): nix.sshServe with write over ssh-ng, for
  fwdesk's ssh host key; nix-ssh is trusted, and nix-serve signs the
  pushed paths when it serves them.
- cachePush: the hook only queues $OUT_PATHS. nix-cache-push.service
  drains the queue with nix copy to ssh-ng://nix-ssh@bigbox2.lab.internal
  as the host key, with bigbox2's host key pinned; a path unit starts it,
  and a failed push is retried every 10 minutes by a timer.

Co-Authored-By: nBot
2026-11-12 10:12:07 -05:00
ari 6db82ed10a Merge branch 'main' of gitea.app.internal:ari/nixos-setup 2026-11-11 17:02:43 -05:00
ari 09c6fe2235 debug agent 2026-11-11 17:02:36 -05:00
ari 2547ec45b4 clear up 2026-11-11 12:03:09 -05:00
hermenix 26856003d4 llama-cpp: add CUDA support and bigbox1 RTX 3060 service config
- Extend pkgs/llama-cpp to support CUDA backend (backend="cuda") for NVIDIA GPUs
- Add llama-cpp-cuda variant to modules/core/default.nix overlay
- Create hosts/hl-bigbox1/services/llama-cpp.nix with RTX 3060 tuning:
  * gpuLayers=30 (conserve 12GB VRAM for context)
  * parallel=2 (handle concurrent requests on smaller VRAM)
  * Same Qwen3.8-27B + DFlash2 drafter as fwdesk
- Wire service into bigbox1 services/default.nix
- Disabled by default (weights download manual)
2026-11-11 16:02:41 +00:00
hermenix ce92096042 llama-cpp: llama-server with DFlash2 speculative decoding for gfx1151
v0.8.0 is the first upstream release with DFlash2 drafters; nixpkgs has 0.6.0 (unstable) and an older tree (stable), so the engine is built here. HIP build for Strix Halo. The NixOS module exposes llama-server behind skyg.nixos.server.services.llama-cpp; fwdesk wires it to Qwen3.8-27B + its DFlash2 drafter, disabled until the weights are on disk.
2026-11-11 16:01:32 +00:00
ari 62c41a6f8e clean up pkgs 2026-11-11 10:35:30 -05:00
ari d8ea131327 clean up pkgs 2026-11-10 16:38:18 -05:00
ari a3178f0507 latest work 2026-11-10 15:00:43 -05:00
ari 3b44ae1354 workflow bump 2026-11-10 14:47:10 -05:00
ari e72789fbb1 bump the version 2026-11-10 10:35:09 -05:00
ari f06921593d Merge branch 'main' of gitea.app.internal:ari/nixos-setup 2026-11-10 10:04:27 -05:00
ari 416c648f99 local fixes 2026-11-10 10:03:09 -05:00
Hermenix 25a21d2f66 nbot: bump to c2491e9 (4fdd292 → c2491e9)
Consolidate environmentFile handling in both hosts via nbot module.
2026-11-10 14:55:28 +00:00
ari dfd4ca06a2 registering nbot wf 2026-11-09 16:13:26 -05:00
ari d337d7605e workflows 2026-11-09 16:06:03 -05:00
ari f2d1ce5744 nbot bump 2026-11-09 13:22:40 -05:00
Hermenix a8bc08de43 container-services: queue the env-reload restart instead of blocking on it
The env-reload unit is a oneshot whose ExecStart was a blocking
systemctl restart of the group unit. When the stack's cold start failed
transiently (container-services-buzz on minipc1, where buzz-relay-1 came
up unhealthy and compose exited 1), the group unit recovered on its own a
few seconds later through its own Restart=on-failure, but the reload unit
had already exited 1 with no restart policy and stayed failed until the
next env file write.

The trigger unit should queue the restart, not own the health of the whole
stack: pass --no-block. Real failures still surface on the group unit,
which carries Restart=on-failure and the start limit and is what the fleet
scan reports.
2026-11-09 15:04:30 +00:00
ari 0c265f7a74 latest nbot 2026-11-08 12:23:04 -05:00
Hermenix d6b79f8d8d nbot: bump to 45c6a48 and enable kvm
nbot 04ade8b -> 45c6a48 (revCount 654 -> 857). The host module gained a new
kvm option to pass -kvm to the nbot binary, which binds /dev/kvm into every
container on the node for workloads that need to run VMs.

KVM is enabled on both fwdesk (master) and fws1 (slave).
2026-11-08 15:23:29 +00:00
ari 39fc85e8f9 latest nbot 2026-11-07 10:51:47 -05:00
ari 088de60af5 cleaning up old bots 2026-11-07 10:50:23 -05:00
ari eb94f5a034 nbot: bump to a6451cb and set the new drive and cluster flags
nbot 8daeca5 -> a6451cb (revCount 492 -> 652). The two config shape
changes in that range are allow_subdir on a drive and master_fingerprint
as a list; summaries.enabled is now ignored because summaries are always
on. The host module gained drives and stateDir, which feed
RequiresMountsFor= so the unit waits for its disks.

The nbot input stays on git+ssh: the repo is private, so git+https
cannot fetch it without credentials.
2026-11-07 15:02:56 +00:00
ari 91d6511f01 nbot updates 2026-11-06 12:59:31 -05:00
ari e3f01e3c35 cleaning up old bots 2026-11-05 15:21:01 -05:00
ari 880c85facb added more capacity 2026-11-05 15:00:35 -05:00
ari 6ff3b48fcf new secret 2026-11-05 12:30:08 -05:00