1073 Commits
Author SHA1 Message Date
ari 89484504e4 unami 2026-11-20 16:29:54 -05:00
ari 6fbc61c7ea slack bump 2026-11-20 16:08:51 -05:00
ari 1dc3ae47f9 Serve the LoopRail site from Caddy on minipc3 2026-11-20 15:09:26 -05:00
ari a092e23f6e bumps 2026-11-19 21:47:48 -05:00
Hermenix 7a973aaee6 diagnix: add diag-pstore
/sys/fs/pstore is root-only and holds the kernel's crash records from
the previous boot, the only trace a hard freeze leaves when the journal
is cut off. diag-pstore prints them via the existing NOPASSWD sudo list.
ABOUTME updated for both diagnix changes.
2026-11-20 01:34:41 +00:00
Hermenix 5c48ef9c1e diagnix: let the hermenix VM log in
hermenix had no shell route to lab hosts, so a hard freeze (bigbox1,
Nov 19 18:21 ET) could only be read through slinuxm's warning-level
journal window, which misses the info-level tail before the crash.

- hosts.nix: a dedicated hermenix-diagnix key, separate from the
  hermenix agenix key, accepted only from hl-fwdesk's addresses
  (10.0.10.11, 10.0.101.101) and the guest address 10.77.0.2.
- default.nix: keyLine takes the key; new
  skyg.nixos.common.diagnix.hermenix.enable (default true) adds the
  hermenix key line with the same restrict,pty + slice limits.
- hl-fwdesk hermenix.nix: lanAccess to 10.0.10.0/24 port 22.
2026-11-20 01:34:41 +00:00
ari 4c9d8f88cb fixing stack2 & 1 2026-11-19 07:47:26 -05:00
ari c8223aec40 fixing stack2 & 1 2026-11-19 07:34:36 -05:00
ari 6abc047879 fixing stack2 2026-11-19 07:18:28 -05:00
ari a48af90346 version bumps 2026-11-18 16:14:21 -05:00
ari 32ea399d89 github runners 2026-11-18 12:50:45 -05:00
ari 705f2eb4c6 precommit 2026-11-18 12:22:04 -05:00
ari 5139a55daa version bumps 2026-11-18 12:20:19 -05:00
ari f9835c150f cleaning up after my clanker 2026-11-17 14:57:04 -05:00
ari e9479f9be1 cleaning up after my clanker 2026-11-17 14:43:55 -05:00
ari a1ebcfb6aa converting loops 2026-11-17 14:22:28 -05:00
ari 956158729e Merge branch 'main' of gitea.app.internal:ari/nixos-setup 2026-11-17 14:10:17 -05:00
ari 87aea41843 less frequent updates 2026-11-17 14:10:00 -05:00
ari f3d03f4c20 looprails 2026-11-17 13:52:08 -05:00
Hermenix ff770c1d03 dns-resilience: Hardened systemd-resolved with scoped host enablement
Add hardened systemd-resolved DNS with boot verification and journal recovery,
enabled only on specific hosts:
- hl-bigbox1, hl-bigbox2 (experienced DNS resilience issues)
- hl-minipc1, hl-minipc2, hl-minipc3 (minipc cluster)
- hl-fwdesk, hl-fws1 (desktop/workstation nodes)

Changes:
1. New module: modules/nixos/common/dns/default.nix
   - systemd-resolved with LLMNR fallback and DNSSEC tolerance
   - Hardened journald with persistent storage and auto-recovery
   - Boot-time DNS verification service with 5 retries
   - Systemd watchdog for automatic reboot on DNS failure

2. Default to false, enabled explicitly on target hosts only
3. Updated host configs to enable dns resilience module
2026-11-17 18:25:20 +00:00
ari d7875b588e multi-slackssss!! 2026-11-16 10:06:48 -05:00
ari 5881b9c8be general clean up of nbot 2026-11-15 15:15:01 -05:00
ari d7bd46c0cd clean up 2026-11-14 22:58:25 -05:00
ari b583c550f3 clean up 2026-11-14 22:57:54 -05:00
ari 08207da02b looprails 2026-11-12 17:19:14 -05:00
ari 3efec42e72 clean up 2026-11-12 16:53:01 -05:00
ari fb5503ec36 Merge remote-tracking branch 'github-hermenix/fix/cache-push-ssh' 2026-11-12 11:17:33 -05:00
ari be6e7492b3 latest nbot 2026-11-12 11:17:21 -05:00
ariandnBot 10d85b2a91 Push builds to bigbox2 over ssh-ng from a background queue
bigbox2 runs nix-serve, which only serves its own store: every
'nix copy --to http://bigbox2.lab.internal:5000' from the post-build hook
ended in HTTP 404 after compressing the whole closure, so nothing ever
landed and the next build tried the same paths again. A post-build-hook
also blocks the build until it returns, which held up every build on
fwdesk (nbot's VM checks spent most of a cold CI run there).

- cacheReceive (on bigbox2): nix.sshServe with write over ssh-ng, for
  fwdesk's ssh host key; nix-ssh is trusted, and nix-serve signs the
  pushed paths when it serves them.
- cachePush: the hook only queues $OUT_PATHS. nix-cache-push.service
  drains the queue with nix copy to ssh-ng://nix-ssh@bigbox2.lab.internal
  as the host key, with bigbox2's host key pinned; a path unit starts it,
  and a failed push is retried every 10 minutes by a timer.

Co-Authored-By: nBot
2026-11-12 10:12:07 -05:00
ari 6db82ed10a Merge branch 'main' of gitea.app.internal:ari/nixos-setup 2026-11-11 17:02:43 -05:00
ari 09c6fe2235 debug agent 2026-11-11 17:02:36 -05:00
ari 2547ec45b4 clear up 2026-11-11 12:03:09 -05:00
hermenix 26856003d4 llama-cpp: add CUDA support and bigbox1 RTX 3060 service config
- Extend pkgs/llama-cpp to support CUDA backend (backend="cuda") for NVIDIA GPUs
- Add llama-cpp-cuda variant to modules/core/default.nix overlay
- Create hosts/hl-bigbox1/services/llama-cpp.nix with RTX 3060 tuning:
  * gpuLayers=30 (conserve 12GB VRAM for context)
  * parallel=2 (handle concurrent requests on smaller VRAM)
  * Same Qwen3.8-27B + DFlash2 drafter as fwdesk
- Wire service into bigbox1 services/default.nix
- Disabled by default (weights download manual)
2026-11-11 16:02:41 +00:00
hermenix ce92096042 llama-cpp: llama-server with DFlash2 speculative decoding for gfx1151
v0.8.0 is the first upstream release with DFlash2 drafters; nixpkgs has 0.6.0 (unstable) and an older tree (stable), so the engine is built here. HIP build for Strix Halo. The NixOS module exposes llama-server behind skyg.nixos.server.services.llama-cpp; fwdesk wires it to Qwen3.8-27B + its DFlash2 drafter, disabled until the weights are on disk.
2026-11-11 16:01:32 +00:00
ari 62c41a6f8e clean up pkgs 2026-11-11 10:35:30 -05:00
ari d8ea131327 clean up pkgs 2026-11-10 16:38:18 -05:00
ari a3178f0507 latest work 2026-11-10 15:00:43 -05:00
ari 3b44ae1354 workflow bump 2026-11-10 14:47:10 -05:00
ari e72789fbb1 bump the version 2026-11-10 10:35:09 -05:00
ari f06921593d Merge branch 'main' of gitea.app.internal:ari/nixos-setup 2026-11-10 10:04:27 -05:00
ari 416c648f99 local fixes 2026-11-10 10:03:09 -05:00
Hermenix 25a21d2f66 nbot: bump to c2491e9 (4fdd292 → c2491e9)
Consolidate environmentFile handling in both hosts via nbot module.
2026-11-10 14:55:28 +00:00
ari dfd4ca06a2 registering nbot wf 2026-11-09 16:13:26 -05:00
ari d337d7605e workflows 2026-11-09 16:06:03 -05:00
ari f2d1ce5744 nbot bump 2026-11-09 13:22:40 -05:00
Hermenix a8bc08de43 container-services: queue the env-reload restart instead of blocking on it
The env-reload unit is a oneshot whose ExecStart was a blocking
systemctl restart of the group unit. When the stack's cold start failed
transiently (container-services-buzz on minipc1, where buzz-relay-1 came
up unhealthy and compose exited 1), the group unit recovered on its own a
few seconds later through its own Restart=on-failure, but the reload unit
had already exited 1 with no restart policy and stayed failed until the
next env file write.

The trigger unit should queue the restart, not own the health of the whole
stack: pass --no-block. Real failures still surface on the group unit,
which carries Restart=on-failure and the start limit and is what the fleet
scan reports.
2026-11-09 15:04:30 +00:00
ari 0c265f7a74 latest nbot 2026-11-08 12:23:04 -05:00
Hermenix d6b79f8d8d nbot: bump to 45c6a48 and enable kvm
nbot 04ade8b -> 45c6a48 (revCount 654 -> 857). The host module gained a new
kvm option to pass -kvm to the nbot binary, which binds /dev/kvm into every
container on the node for workloads that need to run VMs.

KVM is enabled on both fwdesk (master) and fws1 (slave).
2026-11-08 15:23:29 +00:00
ari 39fc85e8f9 latest nbot 2026-11-07 10:51:47 -05:00
ari 088de60af5 cleaning up old bots 2026-11-07 10:50:23 -05:00