Commit Graph
1028 Commits
Author SHA1 Message Date
Hermenix a8bc08de43 container-services: queue the env-reload restart instead of blocking on it
The env-reload unit is a oneshot whose ExecStart was a blocking
systemctl restart of the group unit. When the stack's cold start failed
transiently (container-services-buzz on minipc1, where buzz-relay-1 came
up unhealthy and compose exited 1), the group unit recovered on its own a
few seconds later through its own Restart=on-failure, but the reload unit
had already exited 1 with no restart policy and stayed failed until the
next env file write.

The trigger unit should queue the restart, not own the health of the whole
stack: pass --no-block. Real failures still surface on the group unit,
which carries Restart=on-failure and the start limit and is what the fleet
scan reports.
2026-11-09 15:04:30 +00:00
ari 0c265f7a74 latest nbot 2026-11-08 12:23:04 -05:00
Hermenix d6b79f8d8d nbot: bump to 45c6a48 and enable kvm
nbot 04ade8b -> 45c6a48 (revCount 654 -> 857). The host module gained a new
kvm option to pass -kvm to the nbot binary, which binds /dev/kvm into every
container on the node for workloads that need to run VMs.

KVM is enabled on both fwdesk (master) and fws1 (slave).
2026-11-08 15:23:29 +00:00
ari 39fc85e8f9 latest nbot 2026-11-07 10:51:47 -05:00
ari 088de60af5 cleaning up old bots 2026-11-07 10:50:23 -05:00
ari eb94f5a034 nbot: bump to a6451cb and set the new drive and cluster flags
nbot 8daeca5 -> a6451cb (revCount 492 -> 652). The two config shape
changes in that range are allow_subdir on a drive and master_fingerprint
as a list; summaries.enabled is now ignored because summaries are always
on. The host module gained drives and stateDir, which feed
RequiresMountsFor= so the unit waits for its disks.

The nbot input stays on git+ssh: the repo is private, so git+https
cannot fetch it without credentials.
2026-11-07 15:02:56 +00:00
ari 91d6511f01 nbot updates 2026-11-06 12:59:31 -05:00
ari e3f01e3c35 cleaning up old bots 2026-11-05 15:21:01 -05:00
ari 880c85facb added more capacity 2026-11-05 15:00:35 -05:00
ari 6ff3b48fcf new secret 2026-11-05 12:30:08 -05:00
ari eb862ebf7e fixing stack2 + nbot changes 2026-11-05 12:22:23 -05:00
ari fa5343f211 Merge branch 'main' of gitea.app.internal:ari/nixos-setup 2026-11-04 08:20:37 -05:00
ari 82480e88f4 latest ping for nbot 2026-11-04 08:19:02 -05:00
ari a60b842ee2 more nbot stuff 2026-11-03 19:21:50 -05:00
ari 2d09b6c69f nbots 2026-11-03 15:25:54 -05:00
ari e2bff381b9 nbots 2026-11-03 14:02:31 -05:00
ari 2b83e9c606 nbots 2026-11-03 10:22:40 -05:00
Hermenix 1c420283f4 buzz: prebuild artifact-sync indexes before relay migrates
The relay image is ghcr.io/block/buzz:main with autoUpdate and
BUZZ_AUTO_MIGRATE=true. On 2026-11-02 an auto-update pulled a newer build
whose migrations include 0078_artifact_sync_indexes.sql. That migration
refuses to create its four indexes inline once the target table is
populated -- it raises "prebuild <index> concurrently" and points at
docs/artifact-sync-deployment.md. channel_members is populated on a live
relay, so migration 78 aborts, the relay exits 1 and has been
crash-looping ever since. The schema sits at 0077, so every earlier
migration (including 0077_artifact_sync_timestamps) is applied; rolling
the image back is not an option because sqlx rejects a schema that is
ahead of the binary's known migrations.

Fix the gate instead of the image: a one-shot compose service,
relay-index-prebuild, runs on the postgres image after postgres reports
healthy and builds the four indexes CONCURRENTLY. Each index is checked
first -- missing, valid, or invalid (present but not
indisvalid/indisready/indislive); an invalid leftover is dropped
concurrently before the rebuild, and an already-valid index is left
alone, so re-runs are cheap and safe. The relay now waits on
relay-index-prebuild completing successfully, next to its existing
postgres/redis/minio/minio-init conditions.

Authored by hermenix.
2026-11-03 04:32:29 +00:00
ariandClaude Opus 5.5 8fec1dd4bb bazel-build: simpler idmap note and TODO
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0119iKoN3fSvRGUAYHupPW4U
2026-11-01 23:26:20 -05:00
ariandClaude Opus 5.5 18f4639bb7 bazel-build: clarify what idmap does on the repo cache bind
Container root stays an unprivileged host user; idmap only maps file
owners on that mount. Add a TODO for sharing the cache without
root-owned files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0119iKoN3fSvRGUAYHupPW4U
2026-11-01 23:24:40 -05:00
ariandClaude Opus 5.5 fdf7ecb3e4 bazel-build: drop the handoff note from the header
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0119iKoN3fSvRGUAYHupPW4U
2026-11-01 23:20:17 -05:00
ariandClaude Opus 5.5 2a2b1d8421 bazel-build: remote-only builds for slackerbot containers
- Containers get an enforcing bazel wrapper: remote spawn strategy, no
  local fallback, and refused override flags. Gateway rc is written at boot.
- NativeLink: ExecPaths for the DynamicUser state dir (was noexec).
- Bazel server exits after 5 idle minutes; shared per-host repo cache
  (/repo-cache, idmap); goose hints say a local build means trouble.
- Removed the /handoff -> /outputs materializer; outputs download into
  bazel-bin again.
- slackerbot containers: 4G/1 CPU, up to 8G/2 CPUs.
- fws1: disable networkd wait-online.
- slackerbot 4db99ea (send_file symlinked home, trimmed tool lines).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0119iKoN3fSvRGUAYHupPW4U
2026-11-01 23:14:35 -05:00
ari 594a1c1901 bazel related limit issues 2026-11-01 16:17:50 -05:00
ari dd4965f030 bazel 2026-11-01 14:26:16 -05:00
ari a12f10d964 slackerbot test 1 2026-11-01 13:48:03 -05:00
ari 9856e11416 clean up lid watchers 2026-10-29 14:18:27 -04:00
ari fd733cdc21 Genix rebuilds + hyprland version fix 2026-10-29 14:09:21 -04:00
ari cd7c908fde updated secrets 2026-10-28 13:44:54 -04:00
ari 62e73eb4d2 using seperate robotnix 2026-10-27 13:28:27 -04:00
ari 0efbc5817b setup siblix on buzz working 2026-10-27 09:56:57 -04:00
ari 2e8529fff0 buzz & precommits 2026-10-26 16:26:06 -04:00
ari 3a2cb09d07 added sibli support for buz 2026-10-26 16:18:48 -04:00
ari bfdcd45746 split bots 2026-10-26 14:36:03 -04:00
ari 9217e61675 scrypted hf token 2026-10-26 10:25:19 -04:00
ari 6f0efacfca added visibility to hermenix 2026-10-26 10:03:36 -04:00
ari eecad5cc4a Merge branch 'main' of gitea.app.internal:ari/nixos-setup 2026-10-26 10:01:30 -04:00
ari 6ec9594f92 Merge pull request 'hl-minipc2: bound the journal, give Scrypted the iGPU, add a hermenix secrets recipient' (#1) from hermes/nixos-setup:fix/minipc2-scrypted into main
Reviewed-on: http://gitea.app.internal:3000/ari/nixos-setup/pulls/1
2026-10-26 14:01:14 +00:00
ari 567a85b946 Merge pull request 'robotnix: create /usr/bin in the robot root so nspawn can start genix' (#2) from hermes/nixos-setup:fix/lab-robotnix-genix-ostree into main
Reviewed-on: http://gitea.app.internal:3000/ari/nixos-setup/pulls/2
2026-10-26 14:00:57 +00:00
ari 95b6885f4e bump the version 2026-10-26 09:59:27 -04:00
Hermenix 57fbf70b7c hl-fwdesk: ignore hermes VM fabric links in networkd wait-online
systemd-networkd-wait-online fails on every boot of hl-fwdesk with
"Timeout occurred while waiting for network connectivity" and
status=1/FAILURE.  The hermes VM runs on this host via services.hermenix
(ari/hermenix flake input), and its fabric interfaces -- the bridge
hermes-br and the tap vm-hermes -- are managed by systemd-networkd
(10-hermes-br.network, 11-vm-hermes.network).  networkd-managed links
default to RequiredForOnline=yes, but those two links only gain carrier
when the VM starts, which on the sampled boot was 0.2s after the 120s
wait-online timeout fired.  The only networkd link that never blocks is
robotnix's own bridge, and the physical uplink enp191s0 is owned by
NetworkManager, so it is invisible to networkd.  The result is a unit
that always burns the full timeout and then fails.

hermenix now passes --ignore=hermes-br --ignore=vm-hermes to the unit,
so it no longer treats the VM's fabric links as a condition for
"online": with every candidate ignored the wait returns immediately
instead of blocking and then failing.  The unit itself stays enabled
and anyInterface is untouched.

The durable fix belongs in ari/hermenix: set RequiredForOnline = "no" on
the hermes-br and vm-hermes .network files, the same way
modules/nixos/server/robotnix/network.nix already does for the robotnix
fabric.  That repository is out of scope for this change; once it lands
this ignoredInterfaces entry can be dropped.

Found by hermenix from the fleet monitor's persistent failure record
for unit:hub:systemd-networkd-wait-online.service.
2026-10-26 00:46:08 +00:00
Hermenix 86f9ae010e robotnix: give the robot root a /usr tree before nspawn starts
robotnix-genix.service on hub was in start-limit-hit with exec_main_status 1.
systemd-nspawn refused the root: "Directory /run/robotnix/genix/root doesnt
look like it has an OS tree (/usr/ directory is missing). Refusing." The storage
and seed units were healthy (image mounted, "installation finished!"), so the
tree existed and was populated - it was just missing /usr.

The seed script ends with a tail block that runs on every invocation, not only
when it installs: it creates etc, var/lib, root and home, and touches os-release
and machine-id. That list was copied from nixos-containers but dropped its first
line, mkdir -p "$root/usr/bin" (nixos-containers.nix:102). nspawn treats the
presence of /usr as the marker of an OS tree, so every start died before the
init ran, and because the unit restarts on failure it burned through the start
limit instead of recovering.

Add that mkdir with an explicit 0755 mode to the same tail block. Because the
block is outside the first-install branch, the directory is created on the next
storage+seed run for an already seeded root, so this needs no robotnix reseed
and loses no data. The module layout is unchanged.

ABOUTME.md gains a Notes/decisions bullet recording why the root gets /usr/bin
and that no reseed is required.
2026-10-26 00:45:42 +00:00
Hermenix 520f804738 secrets: declare a hermenix recipient 2026-10-26 00:28:45 +00:00
Hermenix 9b537c9f62 hl-minipc2/scrypted: pass /dev/dri into the container
Without it VAAPI cannot be enumerated and the 4608x1728 H.265 stream is
software-decoded, which is what times out Living Room analysis.
2026-10-26 00:28:45 +00:00
Hermenix bb3c73b310 hl-minipc2: cap the systemd journal at 1G
One stuck Scrypted prune loop was writing ~180k lines (62 MB) a day and
pushed every other fault out of a four-minute window.
2026-10-26 00:28:45 +00:00
ari eb6380ddfe Robotnix v1 2026-10-23 15:26:04 -04:00
ari 0cf504ad27 remiving fwmgr from bigboxes too 2026-10-21 14:43:02 -04:00
ari 39f062a7af safeguarding k3s & removing fwmgr from machines that have hardware with no upstream updates 2026-10-21 14:40:10 -04:00
ari 4c8efd185d updates 2026-10-21 11:04:51 -04:00
ari 5d2ff990ae limit builds 2026-10-21 09:34:30 -04:00
ari 4a72048f0d bye tether & hello new flakes 2026-10-21 00:46:09 -04:00