Commit Graph
1061 Commits
Author SHA1 Message Date
ari 5139a55daa version bumps 2026-11-18 12:20:19 -05:00
ari f9835c150f cleaning up after my clanker 2026-11-17 14:57:04 -05:00
ari e9479f9be1 cleaning up after my clanker 2026-11-17 14:43:55 -05:00
ari a1ebcfb6aa converting loops 2026-11-17 14:22:28 -05:00
ari 956158729e Merge branch 'main' of gitea.app.internal:ari/nixos-setup 2026-11-17 14:10:17 -05:00
ari 87aea41843 less frequent updates 2026-11-17 14:10:00 -05:00
ari f3d03f4c20 looprails 2026-11-17 13:52:08 -05:00
Hermenix ff770c1d03 dns-resilience: Hardened systemd-resolved with scoped host enablement
Add hardened systemd-resolved DNS with boot verification and journal recovery,
enabled only on specific hosts:
- hl-bigbox1, hl-bigbox2 (experienced DNS resilience issues)
- hl-minipc1, hl-minipc2, hl-minipc3 (minipc cluster)
- hl-fwdesk, hl-fws1 (desktop/workstation nodes)

Changes:
1. New module: modules/nixos/common/dns/default.nix
   - systemd-resolved with LLMNR fallback and DNSSEC tolerance
   - Hardened journald with persistent storage and auto-recovery
   - Boot-time DNS verification service with 5 retries
   - Systemd watchdog for automatic reboot on DNS failure

2. Default to false, enabled explicitly on target hosts only
3. Updated host configs to enable dns resilience module
2026-11-17 18:25:20 +00:00
ari d7875b588e multi-slackssss!! 2026-11-16 10:06:48 -05:00
ari 5881b9c8be general clean up of nbot 2026-11-15 15:15:01 -05:00
ari d7bd46c0cd clean up 2026-11-14 22:58:25 -05:00
ari b583c550f3 clean up 2026-11-14 22:57:54 -05:00
ari 08207da02b looprails 2026-11-12 17:19:14 -05:00
ari 3efec42e72 clean up 2026-11-12 16:53:01 -05:00
ari fb5503ec36 Merge remote-tracking branch 'github-hermenix/fix/cache-push-ssh' 2026-11-12 11:17:33 -05:00
ari be6e7492b3 latest nbot 2026-11-12 11:17:21 -05:00
ariandnBot 10d85b2a91 Push builds to bigbox2 over ssh-ng from a background queue
bigbox2 runs nix-serve, which only serves its own store: every
'nix copy --to http://bigbox2.lab.internal:5000' from the post-build hook
ended in HTTP 404 after compressing the whole closure, so nothing ever
landed and the next build tried the same paths again. A post-build-hook
also blocks the build until it returns, which held up every build on
fwdesk (nbot's VM checks spent most of a cold CI run there).

- cacheReceive (on bigbox2): nix.sshServe with write over ssh-ng, for
  fwdesk's ssh host key; nix-ssh is trusted, and nix-serve signs the
  pushed paths when it serves them.
- cachePush: the hook only queues $OUT_PATHS. nix-cache-push.service
  drains the queue with nix copy to ssh-ng://nix-ssh@bigbox2.lab.internal
  as the host key, with bigbox2's host key pinned; a path unit starts it,
  and a failed push is retried every 10 minutes by a timer.

Co-Authored-By: nBot
2026-11-12 10:12:07 -05:00
ari 6db82ed10a Merge branch 'main' of gitea.app.internal:ari/nixos-setup 2026-11-11 17:02:43 -05:00
ari 09c6fe2235 debug agent 2026-11-11 17:02:36 -05:00
ari 2547ec45b4 clear up 2026-11-11 12:03:09 -05:00
hermenix 26856003d4 llama-cpp: add CUDA support and bigbox1 RTX 3060 service config
- Extend pkgs/llama-cpp to support CUDA backend (backend="cuda") for NVIDIA GPUs
- Add llama-cpp-cuda variant to modules/core/default.nix overlay
- Create hosts/hl-bigbox1/services/llama-cpp.nix with RTX 3060 tuning:
  * gpuLayers=30 (conserve 12GB VRAM for context)
  * parallel=2 (handle concurrent requests on smaller VRAM)
  * Same Qwen3.8-27B + DFlash2 drafter as fwdesk
- Wire service into bigbox1 services/default.nix
- Disabled by default (weights download manual)
2026-11-11 16:02:41 +00:00
hermenix ce92096042 llama-cpp: llama-server with DFlash2 speculative decoding for gfx1151
v0.8.0 is the first upstream release with DFlash2 drafters; nixpkgs has 0.6.0 (unstable) and an older tree (stable), so the engine is built here. HIP build for Strix Halo. The NixOS module exposes llama-server behind skyg.nixos.server.services.llama-cpp; fwdesk wires it to Qwen3.8-27B + its DFlash2 drafter, disabled until the weights are on disk.
2026-11-11 16:01:32 +00:00
ari 62c41a6f8e clean up pkgs 2026-11-11 10:35:30 -05:00
ari d8ea131327 clean up pkgs 2026-11-10 16:38:18 -05:00
ari a3178f0507 latest work 2026-11-10 15:00:43 -05:00
ari 3b44ae1354 workflow bump 2026-11-10 14:47:10 -05:00
ari e72789fbb1 bump the version 2026-11-10 10:35:09 -05:00
ari f06921593d Merge branch 'main' of gitea.app.internal:ari/nixos-setup 2026-11-10 10:04:27 -05:00
ari 416c648f99 local fixes 2026-11-10 10:03:09 -05:00
Hermenix 25a21d2f66 nbot: bump to c2491e9 (4fdd292 → c2491e9)
Consolidate environmentFile handling in both hosts via nbot module.
2026-11-10 14:55:28 +00:00
ari dfd4ca06a2 registering nbot wf 2026-11-09 16:13:26 -05:00
ari d337d7605e workflows 2026-11-09 16:06:03 -05:00
ari f2d1ce5744 nbot bump 2026-11-09 13:22:40 -05:00
Hermenix a8bc08de43 container-services: queue the env-reload restart instead of blocking on it
The env-reload unit is a oneshot whose ExecStart was a blocking
systemctl restart of the group unit. When the stack's cold start failed
transiently (container-services-buzz on minipc1, where buzz-relay-1 came
up unhealthy and compose exited 1), the group unit recovered on its own a
few seconds later through its own Restart=on-failure, but the reload unit
had already exited 1 with no restart policy and stayed failed until the
next env file write.

The trigger unit should queue the restart, not own the health of the whole
stack: pass --no-block. Real failures still surface on the group unit,
which carries Restart=on-failure and the start limit and is what the fleet
scan reports.
2026-11-09 15:04:30 +00:00
ari 0c265f7a74 latest nbot 2026-11-08 12:23:04 -05:00
Hermenix d6b79f8d8d nbot: bump to 45c6a48 and enable kvm
nbot 04ade8b -> 45c6a48 (revCount 654 -> 857). The host module gained a new
kvm option to pass -kvm to the nbot binary, which binds /dev/kvm into every
container on the node for workloads that need to run VMs.

KVM is enabled on both fwdesk (master) and fws1 (slave).
2026-11-08 15:23:29 +00:00
ari 39fc85e8f9 latest nbot 2026-11-07 10:51:47 -05:00
ari 088de60af5 cleaning up old bots 2026-11-07 10:50:23 -05:00
ari eb94f5a034 nbot: bump to a6451cb and set the new drive and cluster flags
nbot 8daeca5 -> a6451cb (revCount 492 -> 652). The two config shape
changes in that range are allow_subdir on a drive and master_fingerprint
as a list; summaries.enabled is now ignored because summaries are always
on. The host module gained drives and stateDir, which feed
RequiresMountsFor= so the unit waits for its disks.

The nbot input stays on git+ssh: the repo is private, so git+https
cannot fetch it without credentials.
2026-11-07 15:02:56 +00:00
ari 91d6511f01 nbot updates 2026-11-06 12:59:31 -05:00
ari e3f01e3c35 cleaning up old bots 2026-11-05 15:21:01 -05:00
ari 880c85facb added more capacity 2026-11-05 15:00:35 -05:00
ari 6ff3b48fcf new secret 2026-11-05 12:30:08 -05:00
ari eb862ebf7e fixing stack2 + nbot changes 2026-11-05 12:22:23 -05:00
ari fa5343f211 Merge branch 'main' of gitea.app.internal:ari/nixos-setup 2026-11-04 08:20:37 -05:00
ari 82480e88f4 latest ping for nbot 2026-11-04 08:19:02 -05:00
ari a60b842ee2 more nbot stuff 2026-11-03 19:21:50 -05:00
ari 2d09b6c69f nbots 2026-11-03 15:25:54 -05:00
ari e2bff381b9 nbots 2026-11-03 14:02:31 -05:00
ari 2b83e9c606 nbots 2026-11-03 10:22:40 -05:00