Commit Graph
328 Commits
Author SHA1 Message Date
Hermenix 7a973aaee6 diagnix: add diag-pstore
/sys/fs/pstore is root-only and holds the kernel's crash records from
the previous boot, the only trace a hard freeze leaves when the journal
is cut off. diag-pstore prints them via the existing NOPASSWD sudo list.
ABOUTME updated for both diagnix changes.
2026-11-20 01:34:41 +00:00
Hermenix 5c48ef9c1e diagnix: let the hermenix VM log in
hermenix had no shell route to lab hosts, so a hard freeze (bigbox1,
Nov 19 18:21 ET) could only be read through slinuxm's warning-level
journal window, which misses the info-level tail before the crash.

- hosts.nix: a dedicated hermenix-diagnix key, separate from the
  hermenix agenix key, accepted only from hl-fwdesk's addresses
  (10.0.10.11, 10.0.101.101) and the guest address 10.77.0.2.
- default.nix: keyLine takes the key; new
  skyg.nixos.common.diagnix.hermenix.enable (default true) adds the
  hermenix key line with the same restrict,pty + slice limits.
- hl-fwdesk hermenix.nix: lanAccess to 10.0.10.0/24 port 22.
2026-11-20 01:34:41 +00:00
ari f9835c150f cleaning up after my clanker 2026-11-17 14:57:04 -05:00
ari e9479f9be1 cleaning up after my clanker 2026-11-17 14:43:55 -05:00
Hermenix ff770c1d03 dns-resilience: Hardened systemd-resolved with scoped host enablement
Add hardened systemd-resolved DNS with boot verification and journal recovery,
enabled only on specific hosts:
- hl-bigbox1, hl-bigbox2 (experienced DNS resilience issues)
- hl-minipc1, hl-minipc2, hl-minipc3 (minipc cluster)
- hl-fwdesk, hl-fws1 (desktop/workstation nodes)

Changes:
1. New module: modules/nixos/common/dns/default.nix
   - systemd-resolved with LLMNR fallback and DNSSEC tolerance
   - Hardened journald with persistent storage and auto-recovery
   - Boot-time DNS verification service with 5 retries
   - Systemd watchdog for automatic reboot on DNS failure

2. Default to false, enabled explicitly on target hosts only
3. Updated host configs to enable dns resilience module
2026-11-17 18:25:20 +00:00
ari fb5503ec36 Merge remote-tracking branch 'github-hermenix/fix/cache-push-ssh' 2026-11-12 11:17:33 -05:00
ari be6e7492b3 latest nbot 2026-11-12 11:17:21 -05:00
ariandnBot 10d85b2a91 Push builds to bigbox2 over ssh-ng from a background queue
bigbox2 runs nix-serve, which only serves its own store: every
'nix copy --to http://bigbox2.lab.internal:5000' from the post-build hook
ended in HTTP 404 after compressing the whole closure, so nothing ever
landed and the next build tried the same paths again. A post-build-hook
also blocks the build until it returns, which held up every build on
fwdesk (nbot's VM checks spent most of a cold CI run there).

- cacheReceive (on bigbox2): nix.sshServe with write over ssh-ng, for
  fwdesk's ssh host key; nix-ssh is trusted, and nix-serve signs the
  pushed paths when it serves them.
- cachePush: the hook only queues $OUT_PATHS. nix-cache-push.service
  drains the queue with nix copy to ssh-ng://nix-ssh@bigbox2.lab.internal
  as the host key, with bigbox2's host key pinned; a path unit starts it,
  and a failed push is retried every 10 minutes by a timer.

Co-Authored-By: nBot
2026-11-12 10:12:07 -05:00
ari 6db82ed10a Merge branch 'main' of gitea.app.internal:ari/nixos-setup 2026-11-11 17:02:43 -05:00
ari 09c6fe2235 debug agent 2026-11-11 17:02:36 -05:00
hermenix ce92096042 llama-cpp: llama-server with DFlash2 speculative decoding for gfx1151
v0.8.0 is the first upstream release with DFlash2 drafters; nixpkgs has 0.6.0 (unstable) and an older tree (stable), so the engine is built here. HIP build for Strix Halo. The NixOS module exposes llama-server behind skyg.nixos.server.services.llama-cpp; fwdesk wires it to Qwen3.8-27B + its DFlash2 drafter, disabled until the weights are on disk.
2026-11-11 16:01:32 +00:00
ari 416c648f99 local fixes 2026-11-10 10:03:09 -05:00
Hermenix a8bc08de43 container-services: queue the env-reload restart instead of blocking on it
The env-reload unit is a oneshot whose ExecStart was a blocking
systemctl restart of the group unit. When the stack's cold start failed
transiently (container-services-buzz on minipc1, where buzz-relay-1 came
up unhealthy and compose exited 1), the group unit recovered on its own a
few seconds later through its own Restart=on-failure, but the reload unit
had already exited 1 with no restart policy and stayed failed until the
next env file write.

The trigger unit should queue the restart, not own the health of the whole
stack: pass --no-block. Real failures still surface on the group unit,
which carries Restart=on-failure and the start limit and is what the fleet
scan reports.
2026-11-09 15:04:30 +00:00
ari e3f01e3c35 cleaning up old bots 2026-11-05 15:21:01 -05:00
ari a60b842ee2 more nbot stuff 2026-11-03 19:21:50 -05:00
ari 2b83e9c606 nbots 2026-11-03 10:22:40 -05:00
ariandClaude Opus 5.5 8fec1dd4bb bazel-build: simpler idmap note and TODO
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0119iKoN3fSvRGUAYHupPW4U
2026-11-01 23:26:20 -05:00
ariandClaude Opus 5.5 18f4639bb7 bazel-build: clarify what idmap does on the repo cache bind
Container root stays an unprivileged host user; idmap only maps file
owners on that mount. Add a TODO for sharing the cache without
root-owned files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0119iKoN3fSvRGUAYHupPW4U
2026-11-01 23:24:40 -05:00
ariandClaude Opus 5.5 fdf7ecb3e4 bazel-build: drop the handoff note from the header
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0119iKoN3fSvRGUAYHupPW4U
2026-11-01 23:20:17 -05:00
ariandClaude Opus 5.5 2a2b1d8421 bazel-build: remote-only builds for slackerbot containers
- Containers get an enforcing bazel wrapper: remote spawn strategy, no
  local fallback, and refused override flags. Gateway rc is written at boot.
- NativeLink: ExecPaths for the DynamicUser state dir (was noexec).
- Bazel server exits after 5 idle minutes; shared per-host repo cache
  (/repo-cache, idmap); goose hints say a local build means trouble.
- Removed the /handoff -> /outputs materializer; outputs download into
  bazel-bin again.
- slackerbot containers: 4G/1 CPU, up to 8G/2 CPUs.
- fws1: disable networkd wait-online.
- slackerbot 4db99ea (send_file symlinked home, trimmed tool lines).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0119iKoN3fSvRGUAYHupPW4U
2026-11-01 23:14:35 -05:00
ari 594a1c1901 bazel related limit issues 2026-11-01 16:17:50 -05:00
ari dd4965f030 bazel 2026-11-01 14:26:16 -05:00
ari 9856e11416 clean up lid watchers 2026-10-29 14:18:27 -04:00
ari fd733cdc21 Genix rebuilds + hyprland version fix 2026-10-29 14:09:21 -04:00
ari 62e73eb4d2 using seperate robotnix 2026-10-27 13:28:27 -04:00
ari 0efbc5817b setup siblix on buzz working 2026-10-27 09:56:57 -04:00
ari 3a2cb09d07 added sibli support for buz 2026-10-26 16:18:48 -04:00
ari bfdcd45746 split bots 2026-10-26 14:36:03 -04:00
Hermenix 86f9ae010e robotnix: give the robot root a /usr tree before nspawn starts
robotnix-genix.service on hub was in start-limit-hit with exec_main_status 1.
systemd-nspawn refused the root: "Directory /run/robotnix/genix/root doesnt
look like it has an OS tree (/usr/ directory is missing). Refusing." The storage
and seed units were healthy (image mounted, "installation finished!"), so the
tree existed and was populated - it was just missing /usr.

The seed script ends with a tail block that runs on every invocation, not only
when it installs: it creates etc, var/lib, root and home, and touches os-release
and machine-id. That list was copied from nixos-containers but dropped its first
line, mkdir -p "$root/usr/bin" (nixos-containers.nix:102). nspawn treats the
presence of /usr as the marker of an OS tree, so every start died before the
init ran, and because the unit restarts on failure it burned through the start
limit instead of recovering.

Add that mkdir with an explicit 0755 mode to the same tail block. Because the
block is outside the first-install branch, the directory is created on the next
storage+seed run for an already seeded root, so this needs no robotnix reseed
and loses no data. The module layout is unchanged.

ABOUTME.md gains a Notes/decisions bullet recording why the root gets /usr/bin
and that no reseed is required.
2026-10-26 00:45:42 +00:00
ari eb6380ddfe Robotnix v1 2026-10-23 15:26:04 -04:00
ari 39f062a7af safeguarding k3s & removing fwmgr from machines that have hardware with no upstream updates 2026-10-21 14:40:10 -04:00
ari b0da186ed3 trying to fix reauth on fprint 2026-10-20 16:28:45 -04:00
ari 4f3a95b42c tracking 2026-10-19 14:49:47 -04:00
ari e719674851 killing exporters 2026-10-16 16:59:09 -04:00
ari 1e6cf7251f full deprecation 2026-10-16 15:07:11 -04:00
ari cd48b65656 migrating immich 2026-10-13 19:52:27 -04:00
ari 17b14b33e5 making sure files will cause a restart 2026-10-03 00:57:28 -04:00
ari a7c1b1cc0c public gitea + nixarchy work 2026-10-01 13:45:17 -04:00
ari d514939e37 adding more logging for crashes + hermes + bye niri gestures 2026-09-29 12:41:34 -04:00
ari 4f396d93c4 new dns routing 2026-09-29 11:23:25 -04:00
ari 045702e909 dns refactor 2026-09-26 02:41:43 -04:00
ari 09e2e03423 single file for dns 2026-09-26 01:47:42 -04:00
ari b5bde292d2 refactoring hosts too big 2026-09-25 14:51:13 -04:00
ari 507d603ef0 clean up of ds4 (using my fork of it instead) + update scripts!!! 2026-09-25 10:25:13 -04:00
ari f1e71fa1e9 configured portainer 2026-09-22 22:21:12 -04:00
ari a33f1284dc portainer management 2026-09-22 15:17:35 -04:00
ari cbba7075a7 fixing ip routing for tailscale outside 2026-09-22 14:29:50 -04:00
ari 060d427d04 version bump + bind fixes 2026-09-22 13:33:02 -04:00
ari 7d10ad7d61 fixing some minor things 2026-09-22 11:13:58 -04:00
ari 217b122c67 buzz working 2026-09-22 09:58:48 -04:00