582 Commits
Author SHA1 Message Date
ari 89484504e4 unami 2026-11-20 16:29:54 -05:00
ari 1dc3ae47f9 Serve the LoopRail site from Caddy on minipc3 2026-11-20 15:09:26 -05:00
Hermenix 7a973aaee6 diagnix: add diag-pstore
/sys/fs/pstore is root-only and holds the kernel's crash records from
the previous boot, the only trace a hard freeze leaves when the journal
is cut off. diag-pstore prints them via the existing NOPASSWD sudo list.
ABOUTME updated for both diagnix changes.
2026-11-20 01:34:41 +00:00
Hermenix 5c48ef9c1e diagnix: let the hermenix VM log in
hermenix had no shell route to lab hosts, so a hard freeze (bigbox1,
Nov 19 18:21 ET) could only be read through slinuxm's warning-level
journal window, which misses the info-level tail before the crash.

- hosts.nix: a dedicated hermenix-diagnix key, separate from the
  hermenix agenix key, accepted only from hl-fwdesk's addresses
  (10.0.10.11, 10.0.101.101) and the guest address 10.77.0.2.
- default.nix: keyLine takes the key; new
  skyg.nixos.common.diagnix.hermenix.enable (default true) adds the
  hermenix key line with the same restrict,pty + slice limits.
- hl-fwdesk hermenix.nix: lanAccess to 10.0.10.0/24 port 22.
2026-11-20 01:34:41 +00:00
ari f9835c150f cleaning up after my clanker 2026-11-17 14:57:04 -05:00
ari e9479f9be1 cleaning up after my clanker 2026-11-17 14:43:55 -05:00
Hermenix ff770c1d03 dns-resilience: Hardened systemd-resolved with scoped host enablement
Add hardened systemd-resolved DNS with boot verification and journal recovery,
enabled only on specific hosts:
- hl-bigbox1, hl-bigbox2 (experienced DNS resilience issues)
- hl-minipc1, hl-minipc2, hl-minipc3 (minipc cluster)
- hl-fwdesk, hl-fws1 (desktop/workstation nodes)

Changes:
1. New module: modules/nixos/common/dns/default.nix
   - systemd-resolved with LLMNR fallback and DNSSEC tolerance
   - Hardened journald with persistent storage and auto-recovery
   - Boot-time DNS verification service with 5 retries
   - Systemd watchdog for automatic reboot on DNS failure

2. Default to false, enabled explicitly on target hosts only
3. Updated host configs to enable dns resilience module
2026-11-17 18:25:20 +00:00
ari fb5503ec36 Merge remote-tracking branch 'github-hermenix/fix/cache-push-ssh' 2026-11-12 11:17:33 -05:00
ari be6e7492b3 latest nbot 2026-11-12 11:17:21 -05:00
ariandnBot 10d85b2a91 Push builds to bigbox2 over ssh-ng from a background queue
bigbox2 runs nix-serve, which only serves its own store: every
'nix copy --to http://bigbox2.lab.internal:5000' from the post-build hook
ended in HTTP 404 after compressing the whole closure, so nothing ever
landed and the next build tried the same paths again. A post-build-hook
also blocks the build until it returns, which held up every build on
fwdesk (nbot's VM checks spent most of a cold CI run there).

- cacheReceive (on bigbox2): nix.sshServe with write over ssh-ng, for
  fwdesk's ssh host key; nix-ssh is trusted, and nix-serve signs the
  pushed paths when it serves them.
- cachePush: the hook only queues $OUT_PATHS. nix-cache-push.service
  drains the queue with nix copy to ssh-ng://nix-ssh@bigbox2.lab.internal
  as the host key, with bigbox2's host key pinned; a path unit starts it,
  and a failed push is retried every 10 minutes by a timer.

Co-Authored-By: nBot
2026-11-12 10:12:07 -05:00
ari 6db82ed10a Merge branch 'main' of gitea.app.internal:ari/nixos-setup 2026-11-11 17:02:43 -05:00
ari 09c6fe2235 debug agent 2026-11-11 17:02:36 -05:00
hermenix 26856003d4 llama-cpp: add CUDA support and bigbox1 RTX 3060 service config
- Extend pkgs/llama-cpp to support CUDA backend (backend="cuda") for NVIDIA GPUs
- Add llama-cpp-cuda variant to modules/core/default.nix overlay
- Create hosts/hl-bigbox1/services/llama-cpp.nix with RTX 3060 tuning:
  * gpuLayers=30 (conserve 12GB VRAM for context)
  * parallel=2 (handle concurrent requests on smaller VRAM)
  * Same Qwen3.8-27B + DFlash2 drafter as fwdesk
- Wire service into bigbox1 services/default.nix
- Disabled by default (weights download manual)
2026-11-11 16:02:41 +00:00
hermenix ce92096042 llama-cpp: llama-server with DFlash2 speculative decoding for gfx1151
v0.8.0 is the first upstream release with DFlash2 drafters; nixpkgs has 0.6.0 (unstable) and an older tree (stable), so the engine is built here. HIP build for Strix Halo. The NixOS module exposes llama-server behind skyg.nixos.server.services.llama-cpp; fwdesk wires it to Qwen3.8-27B + its DFlash2 drafter, disabled until the weights are on disk.
2026-11-11 16:01:32 +00:00
ari 416c648f99 local fixes 2026-11-10 10:03:09 -05:00
Hermenix a8bc08de43 container-services: queue the env-reload restart instead of blocking on it
The env-reload unit is a oneshot whose ExecStart was a blocking
systemctl restart of the group unit. When the stack's cold start failed
transiently (container-services-buzz on minipc1, where buzz-relay-1 came
up unhealthy and compose exited 1), the group unit recovered on its own a
few seconds later through its own Restart=on-failure, but the reload unit
had already exited 1 with no restart policy and stayed failed until the
next env file write.

The trigger unit should queue the restart, not own the health of the whole
stack: pass --no-block. Real failures still surface on the group unit,
which carries Restart=on-failure and the start limit and is what the fleet
scan reports.
2026-11-09 15:04:30 +00:00
ari e3f01e3c35 cleaning up old bots 2026-11-05 15:21:01 -05:00
ari a60b842ee2 more nbot stuff 2026-11-03 19:21:50 -05:00
ari 2b83e9c606 nbots 2026-11-03 10:22:40 -05:00
ariandClaude Opus 5.5 8fec1dd4bb bazel-build: simpler idmap note and TODO
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0119iKoN3fSvRGUAYHupPW4U
2026-11-01 23:26:20 -05:00
ariandClaude Opus 5.5 18f4639bb7 bazel-build: clarify what idmap does on the repo cache bind
Container root stays an unprivileged host user; idmap only maps file
owners on that mount. Add a TODO for sharing the cache without
root-owned files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0119iKoN3fSvRGUAYHupPW4U
2026-11-01 23:24:40 -05:00
ariandClaude Opus 5.5 fdf7ecb3e4 bazel-build: drop the handoff note from the header
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0119iKoN3fSvRGUAYHupPW4U
2026-11-01 23:20:17 -05:00
ariandClaude Opus 5.5 2a2b1d8421 bazel-build: remote-only builds for slackerbot containers
- Containers get an enforcing bazel wrapper: remote spawn strategy, no
  local fallback, and refused override flags. Gateway rc is written at boot.
- NativeLink: ExecPaths for the DynamicUser state dir (was noexec).
- Bazel server exits after 5 idle minutes; shared per-host repo cache
  (/repo-cache, idmap); goose hints say a local build means trouble.
- Removed the /handoff -> /outputs materializer; outputs download into
  bazel-bin again.
- slackerbot containers: 4G/1 CPU, up to 8G/2 CPUs.
- fws1: disable networkd wait-online.
- slackerbot 4db99ea (send_file symlinked home, trimmed tool lines).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0119iKoN3fSvRGUAYHupPW4U
2026-11-01 23:14:35 -05:00
ari 594a1c1901 bazel related limit issues 2026-11-01 16:17:50 -05:00
ari dd4965f030 bazel 2026-11-01 14:26:16 -05:00
ari 9856e11416 clean up lid watchers 2026-10-29 14:18:27 -04:00
ari fd733cdc21 Genix rebuilds + hyprland version fix 2026-10-29 14:09:21 -04:00
ari 62e73eb4d2 using seperate robotnix 2026-10-27 13:28:27 -04:00
ari 0efbc5817b setup siblix on buzz working 2026-10-27 09:56:57 -04:00
ari 3a2cb09d07 added sibli support for buz 2026-10-26 16:18:48 -04:00
ari bfdcd45746 split bots 2026-10-26 14:36:03 -04:00
ari eecad5cc4a Merge branch 'main' of gitea.app.internal:ari/nixos-setup 2026-10-26 10:01:30 -04:00
ari 95b6885f4e bump the version 2026-10-26 09:59:27 -04:00
Hermenix 86f9ae010e robotnix: give the robot root a /usr tree before nspawn starts
robotnix-genix.service on hub was in start-limit-hit with exec_main_status 1.
systemd-nspawn refused the root: "Directory /run/robotnix/genix/root doesnt
look like it has an OS tree (/usr/ directory is missing). Refusing." The storage
and seed units were healthy (image mounted, "installation finished!"), so the
tree existed and was populated - it was just missing /usr.

The seed script ends with a tail block that runs on every invocation, not only
when it installs: it creates etc, var/lib, root and home, and touches os-release
and machine-id. That list was copied from nixos-containers but dropped its first
line, mkdir -p "$root/usr/bin" (nixos-containers.nix:102). nspawn treats the
presence of /usr as the marker of an OS tree, so every start died before the
init ran, and because the unit restarts on failure it burned through the start
limit instead of recovering.

Add that mkdir with an explicit 0755 mode to the same tail block. Because the
block is outside the first-install branch, the directory is created on the next
storage+seed run for an already seeded root, so this needs no robotnix reseed
and loses no data. The module layout is unchanged.

ABOUTME.md gains a Notes/decisions bullet recording why the root gets /usr/bin
and that no reseed is required.
2026-10-26 00:45:42 +00:00
ari eb6380ddfe Robotnix v1 2026-10-23 15:26:04 -04:00
ari 39f062a7af safeguarding k3s & removing fwmgr from machines that have hardware with no upstream updates 2026-10-21 14:40:10 -04:00
ari 5d2ff990ae limit builds 2026-10-21 09:34:30 -04:00
ari 4a72048f0d bye tether & hello new flakes 2026-10-21 00:46:09 -04:00
ari b0da186ed3 trying to fix reauth on fprint 2026-10-20 16:28:45 -04:00
ari 246f2cf360 bye superset 2026-10-20 15:05:27 -04:00
ari 4f3a95b42c tracking 2026-10-19 14:49:47 -04:00
ari e719674851 killing exporters 2026-10-16 16:59:09 -04:00
ari 1e6cf7251f full deprecation 2026-10-16 15:07:11 -04:00
ari faeeb885b6 immich in parallel 2026-10-16 10:08:33 -04:00
ari 8a253f82c9 migrating immich 2026-10-13 19:52:54 -04:00
ari cd48b65656 migrating immich 2026-10-13 19:52:27 -04:00
ari 42b8cf4e4f upgrades and hhstream 2026-10-10 22:30:01 -04:00
ari 17b14b33e5 making sure files will cause a restart 2026-10-03 00:57:28 -04:00
ari a7c1b1cc0c public gitea + nixarchy work 2026-10-01 13:45:17 -04:00
ari 8bd66c9975 nitw-gc 2026-09-30 15:58:41 -04:00