575 Commits
Author SHA1 Message Date
ari 89484504e4 unami 2026-11-20 16:29:54 -05:00
ari 6fbc61c7ea slack bump 2026-11-20 16:08:51 -05:00
ari 1dc3ae47f9 Serve the LoopRail site from Caddy on minipc3 2026-11-20 15:09:26 -05:00
Hermenix 5c48ef9c1e diagnix: let the hermenix VM log in
hermenix had no shell route to lab hosts, so a hard freeze (bigbox1,
Nov 19 18:21 ET) could only be read through slinuxm's warning-level
journal window, which misses the info-level tail before the crash.

- hosts.nix: a dedicated hermenix-diagnix key, separate from the
  hermenix agenix key, accepted only from hl-fwdesk's addresses
  (10.0.10.11, 10.0.101.101) and the guest address 10.77.0.2.
- default.nix: keyLine takes the key; new
  skyg.nixos.common.diagnix.hermenix.enable (default true) adds the
  hermenix key line with the same restrict,pty + slice limits.
- hl-fwdesk hermenix.nix: lanAccess to 10.0.10.0/24 port 22.
2026-11-20 01:34:41 +00:00
ari 32ea399d89 github runners 2026-11-18 12:50:45 -05:00
ari 956158729e Merge branch 'main' of gitea.app.internal:ari/nixos-setup 2026-11-17 14:10:17 -05:00
ari f3d03f4c20 looprails 2026-11-17 13:52:08 -05:00
Hermenix ff770c1d03 dns-resilience: Hardened systemd-resolved with scoped host enablement
Add hardened systemd-resolved DNS with boot verification and journal recovery,
enabled only on specific hosts:
- hl-bigbox1, hl-bigbox2 (experienced DNS resilience issues)
- hl-minipc1, hl-minipc2, hl-minipc3 (minipc cluster)
- hl-fwdesk, hl-fws1 (desktop/workstation nodes)

Changes:
1. New module: modules/nixos/common/dns/default.nix
   - systemd-resolved with LLMNR fallback and DNSSEC tolerance
   - Hardened journald with persistent storage and auto-recovery
   - Boot-time DNS verification service with 5 retries
   - Systemd watchdog for automatic reboot on DNS failure

2. Default to false, enabled explicitly on target hosts only
3. Updated host configs to enable dns resilience module
2026-11-17 18:25:20 +00:00
ari d7875b588e multi-slackssss!! 2026-11-16 10:06:48 -05:00
ari 5881b9c8be general clean up of nbot 2026-11-15 15:15:01 -05:00
ari d7bd46c0cd clean up 2026-11-14 22:58:25 -05:00
ari b583c550f3 clean up 2026-11-14 22:57:54 -05:00
ari 08207da02b looprails 2026-11-12 17:19:14 -05:00
ari 3efec42e72 clean up 2026-11-12 16:53:01 -05:00
ari fb5503ec36 Merge remote-tracking branch 'github-hermenix/fix/cache-push-ssh' 2026-11-12 11:17:33 -05:00
ari be6e7492b3 latest nbot 2026-11-12 11:17:21 -05:00
ariandnBot 10d85b2a91 Push builds to bigbox2 over ssh-ng from a background queue
bigbox2 runs nix-serve, which only serves its own store: every
'nix copy --to http://bigbox2.lab.internal:5000' from the post-build hook
ended in HTTP 404 after compressing the whole closure, so nothing ever
landed and the next build tried the same paths again. A post-build-hook
also blocks the build until it returns, which held up every build on
fwdesk (nbot's VM checks spent most of a cold CI run there).

- cacheReceive (on bigbox2): nix.sshServe with write over ssh-ng, for
  fwdesk's ssh host key; nix-ssh is trusted, and nix-serve signs the
  pushed paths when it serves them.
- cachePush: the hook only queues $OUT_PATHS. nix-cache-push.service
  drains the queue with nix copy to ssh-ng://nix-ssh@bigbox2.lab.internal
  as the host key, with bigbox2's host key pinned; a path unit starts it,
  and a failed push is retried every 10 minutes by a timer.

Co-Authored-By: nBot
2026-11-12 10:12:07 -05:00
ari 6db82ed10a Merge branch 'main' of gitea.app.internal:ari/nixos-setup 2026-11-11 17:02:43 -05:00
ari 09c6fe2235 debug agent 2026-11-11 17:02:36 -05:00
ari 2547ec45b4 clear up 2026-11-11 12:03:09 -05:00
hermenix 26856003d4 llama-cpp: add CUDA support and bigbox1 RTX 3060 service config
- Extend pkgs/llama-cpp to support CUDA backend (backend="cuda") for NVIDIA GPUs
- Add llama-cpp-cuda variant to modules/core/default.nix overlay
- Create hosts/hl-bigbox1/services/llama-cpp.nix with RTX 3060 tuning:
  * gpuLayers=30 (conserve 12GB VRAM for context)
  * parallel=2 (handle concurrent requests on smaller VRAM)
  * Same Qwen3.8-27B + DFlash2 drafter as fwdesk
- Wire service into bigbox1 services/default.nix
- Disabled by default (weights download manual)
2026-11-11 16:02:41 +00:00
hermenix ce92096042 llama-cpp: llama-server with DFlash2 speculative decoding for gfx1151
v0.8.0 is the first upstream release with DFlash2 drafters; nixpkgs has 0.6.0 (unstable) and an older tree (stable), so the engine is built here. HIP build for Strix Halo. The NixOS module exposes llama-server behind skyg.nixos.server.services.llama-cpp; fwdesk wires it to Qwen3.8-27B + its DFlash2 drafter, disabled until the weights are on disk.
2026-11-11 16:01:32 +00:00
ari 62c41a6f8e clean up pkgs 2026-11-11 10:35:30 -05:00
ari d8ea131327 clean up pkgs 2026-11-10 16:38:18 -05:00
ari a3178f0507 latest work 2026-11-10 15:00:43 -05:00
ari e72789fbb1 bump the version 2026-11-10 10:35:09 -05:00
ari f06921593d Merge branch 'main' of gitea.app.internal:ari/nixos-setup 2026-11-10 10:04:27 -05:00
ari 416c648f99 local fixes 2026-11-10 10:03:09 -05:00
Hermenix 25a21d2f66 nbot: bump to c2491e9 (4fdd292 → c2491e9)
Consolidate environmentFile handling in both hosts via nbot module.
2026-11-10 14:55:28 +00:00
ari d337d7605e workflows 2026-11-09 16:06:03 -05:00
Hermenix d6b79f8d8d nbot: bump to 45c6a48 and enable kvm
nbot 04ade8b -> 45c6a48 (revCount 654 -> 857). The host module gained a new
kvm option to pass -kvm to the nbot binary, which binds /dev/kvm into every
container on the node for workloads that need to run VMs.

KVM is enabled on both fwdesk (master) and fws1 (slave).
2026-11-08 15:23:29 +00:00
ari 39fc85e8f9 latest nbot 2026-11-07 10:51:47 -05:00
ari 088de60af5 cleaning up old bots 2026-11-07 10:50:23 -05:00
ari eb94f5a034 nbot: bump to a6451cb and set the new drive and cluster flags
nbot 8daeca5 -> a6451cb (revCount 492 -> 652). The two config shape
changes in that range are allow_subdir on a drive and master_fingerprint
as a list; summaries.enabled is now ignored because summaries are always
on. The host module gained drives and stateDir, which feed
RequiresMountsFor= so the unit waits for its disks.

The nbot input stays on git+ssh: the repo is private, so git+https
cannot fetch it without credentials.
2026-11-07 15:02:56 +00:00
ari 91d6511f01 nbot updates 2026-11-06 12:59:31 -05:00
ari e3f01e3c35 cleaning up old bots 2026-11-05 15:21:01 -05:00
ari 880c85facb added more capacity 2026-11-05 15:00:35 -05:00
ari eb862ebf7e fixing stack2 + nbot changes 2026-11-05 12:22:23 -05:00
ari fa5343f211 Merge branch 'main' of gitea.app.internal:ari/nixos-setup 2026-11-04 08:20:37 -05:00
ari a60b842ee2 more nbot stuff 2026-11-03 19:21:50 -05:00
ari e2bff381b9 nbots 2026-11-03 14:02:31 -05:00
ari 2b83e9c606 nbots 2026-11-03 10:22:40 -05:00
Hermenix 1c420283f4 buzz: prebuild artifact-sync indexes before relay migrates
The relay image is ghcr.io/block/buzz:main with autoUpdate and
BUZZ_AUTO_MIGRATE=true. On 2026-11-02 an auto-update pulled a newer build
whose migrations include 0078_artifact_sync_indexes.sql. That migration
refuses to create its four indexes inline once the target table is
populated -- it raises "prebuild <index> concurrently" and points at
docs/artifact-sync-deployment.md. channel_members is populated on a live
relay, so migration 78 aborts, the relay exits 1 and has been
crash-looping ever since. The schema sits at 0077, so every earlier
migration (including 0077_artifact_sync_timestamps) is applied; rolling
the image back is not an option because sqlx rejects a schema that is
ahead of the binary's known migrations.

Fix the gate instead of the image: a one-shot compose service,
relay-index-prebuild, runs on the postgres image after postgres reports
healthy and builds the four indexes CONCURRENTLY. Each index is checked
first -- missing, valid, or invalid (present but not
indisvalid/indisready/indislive); an invalid leftover is dropped
concurrently before the rebuild, and an already-valid index is left
alone, so re-runs are cheap and safe. The relay now waits on
relay-index-prebuild completing successfully, next to its existing
postgres/redis/minio/minio-init conditions.

Authored by hermenix.
2026-11-03 04:32:29 +00:00
ariandClaude Opus 5.5 2a2b1d8421 bazel-build: remote-only builds for slackerbot containers
- Containers get an enforcing bazel wrapper: remote spawn strategy, no
  local fallback, and refused override flags. Gateway rc is written at boot.
- NativeLink: ExecPaths for the DynamicUser state dir (was noexec).
- Bazel server exits after 5 idle minutes; shared per-host repo cache
  (/repo-cache, idmap); goose hints say a local build means trouble.
- Removed the /handoff -> /outputs materializer; outputs download into
  bazel-bin again.
- slackerbot containers: 4G/1 CPU, up to 8G/2 CPUs.
- fws1: disable networkd wait-online.
- slackerbot 4db99ea (send_file symlinked home, trimmed tool lines).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0119iKoN3fSvRGUAYHupPW4U
2026-11-01 23:14:35 -05:00
ari 594a1c1901 bazel related limit issues 2026-11-01 16:17:50 -05:00
ari dd4965f030 bazel 2026-11-01 14:26:16 -05:00
ari a12f10d964 slackerbot test 1 2026-11-01 13:48:03 -05:00
ari 9856e11416 clean up lid watchers 2026-10-29 14:18:27 -04:00
ari fd733cdc21 Genix rebuilds + hyprland version fix 2026-10-29 14:09:21 -04:00
ari cd7c908fde updated secrets 2026-10-28 13:44:54 -04:00