50 Commits
Author SHA1 Message Date
ari 89484504e4 unami 2026-11-20 16:29:54 -05:00
ari 6fbc61c7ea slack bump 2026-11-20 16:08:51 -05:00
ari 1dc3ae47f9 Serve the LoopRail site from Caddy on minipc3 2026-11-20 15:09:26 -05:00
ari a092e23f6e bumps 2026-11-19 21:47:48 -05:00
Hermenix 7a973aaee6 diagnix: add diag-pstore
/sys/fs/pstore is root-only and holds the kernel's crash records from
the previous boot, the only trace a hard freeze leaves when the journal
is cut off. diag-pstore prints them via the existing NOPASSWD sudo list.
ABOUTME updated for both diagnix changes.
2026-11-20 01:34:41 +00:00
Hermenix 5c48ef9c1e diagnix: let the hermenix VM log in
hermenix had no shell route to lab hosts, so a hard freeze (bigbox1,
Nov 19 18:21 ET) could only be read through slinuxm's warning-level
journal window, which misses the info-level tail before the crash.

- hosts.nix: a dedicated hermenix-diagnix key, separate from the
  hermenix agenix key, accepted only from hl-fwdesk's addresses
  (10.0.10.11, 10.0.101.101) and the guest address 10.77.0.2.
- default.nix: keyLine takes the key; new
  skyg.nixos.common.diagnix.hermenix.enable (default true) adds the
  hermenix key line with the same restrict,pty + slice limits.
- hl-fwdesk hermenix.nix: lanAccess to 10.0.10.0/24 port 22.
2026-11-20 01:34:41 +00:00
ari 4c9d8f88cb fixing stack2 & 1 2026-11-19 07:47:26 -05:00
ari c8223aec40 fixing stack2 & 1 2026-11-19 07:34:36 -05:00
ari 6abc047879 fixing stack2 2026-11-19 07:18:28 -05:00
ari a48af90346 version bumps 2026-11-18 16:14:21 -05:00
ari 32ea399d89 github runners 2026-11-18 12:50:45 -05:00
ari 705f2eb4c6 precommit 2026-11-18 12:22:04 -05:00
ari 5139a55daa version bumps 2026-11-18 12:20:19 -05:00
ari f9835c150f cleaning up after my clanker 2026-11-17 14:57:04 -05:00
ari e9479f9be1 cleaning up after my clanker 2026-11-17 14:43:55 -05:00
ari a1ebcfb6aa converting loops 2026-11-17 14:22:28 -05:00
ari 956158729e Merge branch 'main' of gitea.app.internal:ari/nixos-setup 2026-11-17 14:10:17 -05:00
ari 87aea41843 less frequent updates 2026-11-17 14:10:00 -05:00
ari f3d03f4c20 looprails 2026-11-17 13:52:08 -05:00
Hermenix ff770c1d03 dns-resilience: Hardened systemd-resolved with scoped host enablement
Add hardened systemd-resolved DNS with boot verification and journal recovery,
enabled only on specific hosts:
- hl-bigbox1, hl-bigbox2 (experienced DNS resilience issues)
- hl-minipc1, hl-minipc2, hl-minipc3 (minipc cluster)
- hl-fwdesk, hl-fws1 (desktop/workstation nodes)

Changes:
1. New module: modules/nixos/common/dns/default.nix
   - systemd-resolved with LLMNR fallback and DNSSEC tolerance
   - Hardened journald with persistent storage and auto-recovery
   - Boot-time DNS verification service with 5 retries
   - Systemd watchdog for automatic reboot on DNS failure

2. Default to false, enabled explicitly on target hosts only
3. Updated host configs to enable dns resilience module
2026-11-17 18:25:20 +00:00
ari d7875b588e multi-slackssss!! 2026-11-16 10:06:48 -05:00
ari 5881b9c8be general clean up of nbot 2026-11-15 15:15:01 -05:00
ari d7bd46c0cd clean up 2026-11-14 22:58:25 -05:00
ari b583c550f3 clean up 2026-11-14 22:57:54 -05:00
ari 08207da02b looprails 2026-11-12 17:19:14 -05:00
ari 3efec42e72 clean up 2026-11-12 16:53:01 -05:00
ari fb5503ec36 Merge remote-tracking branch 'github-hermenix/fix/cache-push-ssh' 2026-11-12 11:17:33 -05:00
ari be6e7492b3 latest nbot 2026-11-12 11:17:21 -05:00
ariandnBot 10d85b2a91 Push builds to bigbox2 over ssh-ng from a background queue
bigbox2 runs nix-serve, which only serves its own store: every
'nix copy --to http://bigbox2.lab.internal:5000' from the post-build hook
ended in HTTP 404 after compressing the whole closure, so nothing ever
landed and the next build tried the same paths again. A post-build-hook
also blocks the build until it returns, which held up every build on
fwdesk (nbot's VM checks spent most of a cold CI run there).

- cacheReceive (on bigbox2): nix.sshServe with write over ssh-ng, for
  fwdesk's ssh host key; nix-ssh is trusted, and nix-serve signs the
  pushed paths when it serves them.
- cachePush: the hook only queues $OUT_PATHS. nix-cache-push.service
  drains the queue with nix copy to ssh-ng://nix-ssh@bigbox2.lab.internal
  as the host key, with bigbox2's host key pinned; a path unit starts it,
  and a failed push is retried every 10 minutes by a timer.

Co-Authored-By: nBot
2026-11-12 10:12:07 -05:00
ari 6db82ed10a Merge branch 'main' of gitea.app.internal:ari/nixos-setup 2026-11-11 17:02:43 -05:00
ari 09c6fe2235 debug agent 2026-11-11 17:02:36 -05:00
ari 2547ec45b4 clear up 2026-11-11 12:03:09 -05:00
hermenix 26856003d4 llama-cpp: add CUDA support and bigbox1 RTX 3060 service config
- Extend pkgs/llama-cpp to support CUDA backend (backend="cuda") for NVIDIA GPUs
- Add llama-cpp-cuda variant to modules/core/default.nix overlay
- Create hosts/hl-bigbox1/services/llama-cpp.nix with RTX 3060 tuning:
  * gpuLayers=30 (conserve 12GB VRAM for context)
  * parallel=2 (handle concurrent requests on smaller VRAM)
  * Same Qwen3.8-27B + DFlash2 drafter as fwdesk
- Wire service into bigbox1 services/default.nix
- Disabled by default (weights download manual)
2026-11-11 16:02:41 +00:00
hermenix ce92096042 llama-cpp: llama-server with DFlash2 speculative decoding for gfx1151
v0.8.0 is the first upstream release with DFlash2 drafters; nixpkgs has 0.6.0 (unstable) and an older tree (stable), so the engine is built here. HIP build for Strix Halo. The NixOS module exposes llama-server behind skyg.nixos.server.services.llama-cpp; fwdesk wires it to Qwen3.8-27B + its DFlash2 drafter, disabled until the weights are on disk.
2026-11-11 16:01:32 +00:00
ari 62c41a6f8e clean up pkgs 2026-11-11 10:35:30 -05:00
ari d8ea131327 clean up pkgs 2026-11-10 16:38:18 -05:00
ari a3178f0507 latest work 2026-11-10 15:00:43 -05:00
ari 3b44ae1354 workflow bump 2026-11-10 14:47:10 -05:00
ari e72789fbb1 bump the version 2026-11-10 10:35:09 -05:00
ari f06921593d Merge branch 'main' of gitea.app.internal:ari/nixos-setup 2026-11-10 10:04:27 -05:00
ari 416c648f99 local fixes 2026-11-10 10:03:09 -05:00
Hermenix 25a21d2f66 nbot: bump to c2491e9 (4fdd292 → c2491e9)
Consolidate environmentFile handling in both hosts via nbot module.
2026-11-10 14:55:28 +00:00
ari dfd4ca06a2 registering nbot wf 2026-11-09 16:13:26 -05:00
ari d337d7605e workflows 2026-11-09 16:06:03 -05:00
ari f2d1ce5744 nbot bump 2026-11-09 13:22:40 -05:00
Hermenix a8bc08de43 container-services: queue the env-reload restart instead of blocking on it
The env-reload unit is a oneshot whose ExecStart was a blocking
systemctl restart of the group unit. When the stack's cold start failed
transiently (container-services-buzz on minipc1, where buzz-relay-1 came
up unhealthy and compose exited 1), the group unit recovered on its own a
few seconds later through its own Restart=on-failure, but the reload unit
had already exited 1 with no restart policy and stayed failed until the
next env file write.

The trigger unit should queue the restart, not own the health of the whole
stack: pass --no-block. Real failures still surface on the group unit,
which carries Restart=on-failure and the start limit and is what the fleet
scan reports.
2026-11-09 15:04:30 +00:00
ari 0c265f7a74 latest nbot 2026-11-08 12:23:04 -05:00
Hermenix d6b79f8d8d nbot: bump to 45c6a48 and enable kvm
nbot 04ade8b -> 45c6a48 (revCount 654 -> 857). The host module gained a new
kvm option to pass -kvm to the nbot binary, which binds /dev/kvm into every
container on the node for workloads that need to run VMs.

KVM is enabled on both fwdesk (master) and fws1 (slave).
2026-11-08 15:23:29 +00:00
ari 39fc85e8f9 latest nbot 2026-11-07 10:51:47 -05:00
ari 088de60af5 cleaning up old bots 2026-11-07 10:50:23 -05:00
71 changed files with 1692 additions and 239 deletions

No files matched your search

+18
View File
@@ -0,0 +1,18 @@
# .agents/looprails
Looprail files for this repo.
## journal-triage
`journal-triage.toml` reads the journald log of this machine, has Claude Opus triage it
against this repo, and writes a report ranked by urgency to `.tmp/journal-reports/<host>-<date>.md` (gitignored).
- `journal-triage/collect.sh`: pulls warnings (and worse), failed units and a digest of
unique messages into the run folder.
- `journal-triage/triage.md`: the brief for Opus. It reads `GUIDELINES.md`, traces each issue to
`hosts/<host>/` or `modules/`, and suggests fixes as repo changes. It only reads and suggests:
no edits, no builds, no sudo.
- Inputs: `since` (default `24 hours ago`), `priority` (`err`, `warning`, `notice`),
`host` (default `fwbook`), `repo` (default `/home/ari/nixos-setup`).
Run: `looprail run .agents/looprails/journal-triage.toml`, or ask the agent to run it.
+45
View File
@@ -0,0 +1,45 @@
name = "journal-triage"
description = "Collect journald warnings, triage them with Claude Opus against this repo, and write an urgency-ranked report with fixes as repo changes."
[inputs.static]
since = { type = "string", description = "How far back to read the journal.", min_length = 1, default = "24 hours ago" }
priority = { type = "enum", description = "Lowest log priority to include.", values = ["err", "warning", "notice"], default = "warning" }
host = { type = "string", description = "Name of the host folder under hosts/ to trace issues to.", min_length = 1, default = "fwbook" }
repo = { type = "string", description = "Path of the NixOS config repo.", min_length = 1, default = "/home/ari/nixos-setup" }
[[step.collect]]
kind = "shell"
description = "Collects journal warnings and failed units into a digest."
cwd = "{{ inputs.repo }}"
run = ["sh .agents/looprails/journal-triage/collect.sh {{ inputs.since }} {{ inputs.priority }}"]
[step.collect.outputs]
raw_log = { type = "string", description = "Path of the full raw journal log." }
digest = { type = "string", description = "Path of the deduplicated digest." }
lines = { type = "int", description = "Number of raw log lines." }
unique = { type = "int", description = "Number of distinct messages." }
[[step.triage]]
kind = "agent"
description = "Has Opus triage the log against the repo and suggest fixes."
needs = ["collect"]
agent = { command = ["claude-agent-acp"], model = "opus" }
cwd = "{{ inputs.repo }}"
brief = "journal-triage/triage.md"
failover = "agent"
[step.triage.outputs]
summary = { type = "string", description = "Markdown report ranked by urgency.", min_length = 1 }
issues = { type = "list", description = "The issues found, with urgency, fix and files.", items = { title = { type = "string" }, urgency = { type = "enum", values = ["critical", "high", "medium", "low", "noise"] }, fix = { type = "string" }, files = { type = "list" } } }
[[step.report]]
kind = "shell"
description = "Writes the report to .tmp/journal-reports."
needs = ["triage"]
run = [
"R={{ inputs.repo }}; mkdir -p \"$R/.tmp/journal-reports\"",
"S={{ steps.triage.outputs.summary }}; H={{ inputs.host }}; R={{ inputs.repo }}; f=\"$R/.tmp/journal-reports/$H-$(date +%Y-%m-%d-%H%M).md\"; printf '%s\\n' \"$S\" > \"$f\"; echo \"report=$f\" >> \"$NBOT_LR_OUTPUTS\"; echo \"Report written to $f\"",
]
[step.report.outputs]
report = { type = "string", description = "Path of the written report." }
+26
View File
@@ -0,0 +1,26 @@
#!/bin/sh
# Collect journald warnings+ and failed units into the run folder, plus a deduplicated digest.
set -eu
since="$1"; priority="$2"
dir="$NBOT_LR_RUN_DIR"
raw="$dir/journal-raw.log"; digest="$dir/journal-digest.txt"
journalctl -p "$priority" --since "$since" --no-pager -q -o short-iso > "$raw" || true
{
echo "=== Failed systemd units ==="
systemctl --failed --no-legend --plain || true
systemctl --user --failed --no-legend --plain 2>/dev/null || true
echo
echo "=== Boots in window ==="
journalctl --list-boots --no-pager -q | tail -5 || true
echo
echo "=== Unique messages (count, source: message; numbers/hex normalized), most frequent first ==="
cut -d' ' -f3- "$raw" \
| sed -E 's/\[[0-9]+\]//; s/0x[0-9a-fA-F]+/0xN/g; s/[0-9]+/N/g' \
| sort | uniq -c | sort -rn | head -400
} > "$digest"
lines=$(wc -l < "$raw")
uniq=$(grep -c '^ *[0-9]' "$digest" || true)
printf 'raw_log=%s\ndigest=%s\nlines=%s\nunique=%s\n' "$raw" "$digest" "$lines" "$uniq" >> "$NBOT_LR_OUTPUTS"
@@ -0,0 +1,65 @@
You are triaging the systemd journal of the NixOS host `{{ inputs.host }}`, and tracing each issue
back to the NixOS config repo that builds it: {{ inputs.repo }} (your working folder).
Window: since {{ inputs.since }}, priority {{ inputs.priority }} and worse.
{{ steps.collect.outputs.lines }} log lines, {{ steps.collect.outputs.unique }} distinct messages.
- Digest (failed units, boots, deduplicated messages with counts): {{ steps.collect.outputs.digest }}
- Full raw log (timestamps, for context): {{ steps.collect.outputs.raw_log }}
## Rules: read and suggest only
- Change nothing. Do not edit, create or delete files, in the repo or anywhere else.
- Do not run builds or switches: no `nixos-rebuild`, `nh`, `skyg`, `nix build`, `nix flake check`,
`nixos-anywhere`, docker/podman builds. No sudo. No git commands that change state.
- You may run read-only commands: `systemctl status`, `journalctl -u`, `ls`, `cat`, `grep`, `rg`,
`git log`, `git diff`, `nix-instantiate --parse`.
- Never read anything in `secrets/`.
## Steps
1. Read `GUIDELINES.md` in the repo. Your suggested changes must follow it.
2. Read the digest, then grep the raw log for detail where needed.
3. Group related messages into issues. Ignore harmless boot-time noise unless it points to
something real; list it briefly under "noise" so the reader knows it was seen.
4. For each real issue, find where it comes from in the repo: start with `hosts/{{ inputs.host }}/`,
then `modules/`, `configs/`, `pkgs/` and `flake.nix`. Check whether another host
(`hosts/*/`) already solves the same problem, and reuse that pattern.
Note when the cause is outside the repo (a hand-written file in `~/.config`, state on disk).
## How to write fixes
- Give each fix as a short suggested Nix change with its repo path, e.g.
`hosts/{{ inputs.host }}/systemd.nix` plus a small snippet.
- Follow the repo's layout: put config in the matching host sub-file (`hardware.nix`, `systemd.nix`,
`packages.nix`, `networking.nix`, `services.nix` or `services/<name>.nix`, `home/default.nix`).
Custom options live under `skyg.*`; check `modules/lib.nix` before naming one.
- Never suggest `imports = [...]` in host files.
- Never suggest editing `flake.lock`, anything in `hosts/*/hardware-configuration/` or
`*.hardware-configuration.nix`, or `secrets/`. If the cause lives there, say so and suggest an
override in the right host sub-file instead (e.g. `fileSystems."/boot".options` in `hardware.nix`
with `lib.mkForce`).
- When a fix is imperative (a one-off command), say it needs to be run by hand, and prefer a
declarative option when one exists.
- Remind the reader that they rebuild themselves; do not tell them a change has been applied.
## Urgency
- critical: data loss, security, hardware failing, system unbootable or core service down
- high: a service failing or crash-looping, repeated errors with user-visible effect
- medium: degraded behaviour, misconfiguration, something that will bite later
- low: cosmetic or rare warnings worth a look
- noise: benign, safe to ignore
## Output
`summary` is a Markdown report:
- a one-paragraph overview
- sections by urgency, critical first; each issue with evidence (unit, sample message, count,
first/last seen), likely cause, the repo files involved, and fixes as suggested repo changes
- a short "Repo recommendations" section: patterns across hosts or modules worth changing
(duplication, hand-made units that should be declarative, a fix that would help other hosts too)
Keep it concise, with plain words.
`issues` is the same issues as a list; `files` holds the repo paths involved (empty if none).
+8
View File
@@ -0,0 +1,8 @@
---
name: nbot-wf
description: When anything to do with workflows comes up, use the nbot-wf MCP tools; if they are not available, run nbot-wf install and restart the session.
---
When anything to do with workflows comes up, use the nbot-wf MCP tools; if they are not available, run `nbot-wf install` and restart the session.
The `get_workflow_guide` tool explains how to find, run, watch, answer, stop, resume, write and update a workflow.
+3
View File
@@ -0,0 +1,3 @@
[mcp_servers.nbot-wf]
command = "nbot-wf"
args = ["mcp"]
+3
View File
@@ -45,3 +45,6 @@ docs
# Zed local settings (machine-local MCP tokens live here)
.zed/settings.local.json
.nbot-wf
+10
View File
@@ -0,0 +1,10 @@
{
"mcpServers": {
"nbot-wf": {
"command": "nbot-wf",
"args": [
"mcp"
]
}
}
}
+1 -1
View File
@@ -1 +1 @@
/nix/store/5z8yjkavrsi0zy030pcbz60ndxy248vy-pre-commit-config.json
/nix/store/mbdhxcacyi218765m4p2b53ww4fhrlsf-pre-commit-config.json
Generated
+83 -11
View File
@@ -1579,7 +1579,7 @@
"hyprutils": "hyprutils_3",
"hyprwayland-scanner": "hyprwayland-scanner_3",
"hyprwire": "hyprwire_3",
"nixpkgs": "nixpkgs_11",
"nixpkgs": "nixpkgs_12",
"pre-commit-hooks": "pre-commit-hooks_3",
"systems": "systems_3",
"xdph": "xdph_2"
@@ -2136,6 +2136,45 @@
"type": "github"
}
},
"looprail": {
"inputs": {
"nixpkgs": [
"nixpkgs-unstable"
]
},
"locked": {
"lastModified": 1795206835,
"narHash": "sha256-9MP7M7Ifw6nU+K+38rymjG+z6a1jSJSQxzvMzFNcwoM=",
"ref": "refs/heads/main",
"rev": "d6b3a8eababef6389895e126814659476e2ae1ee",
"revCount": 261,
"type": "git",
"url": "ssh://git@github.com/SkyG-Org/LoopRail.git"
},
"original": {
"type": "git",
"url": "ssh://git@github.com/SkyG-Org/LoopRail.git"
}
},
"looprail_2": {
"inputs": {
"nixpkgs": "nixpkgs_11"
},
"locked": {
"lastModified": 1795060300,
"narHash": "sha256-T+k1mIiNGYPQFfvorraVlEy5qDIqMHtdprkbWFCOJ7k=",
"ref": "refs/heads/main",
"rev": "70be15ae9d29264dd85c892b86209af486a92e6a",
"revCount": 136,
"type": "git",
"url": "ssh://git@github.com/SkyG-Org/LoopRail.git"
},
"original": {
"rev": "70be15ae9d29264dd85c892b86209af486a92e6a",
"type": "git",
"url": "ssh://git@github.com/SkyG-Org/LoopRail.git"
}
},
"mcp-servers-nix": {
"inputs": {
"nixpkgs": [
@@ -2199,22 +2238,23 @@
},
"nbot": {
"inputs": {
"looprail": "looprail_2",
"nixpkgs": [
"nixpkgs-unstable"
]
},
"locked": {
"lastModified": 1794057087,
"narHash": "sha256-UlrEHt0pFD2AXuleAKUmf6lD2Pih8+jCbxWxpEz05LA=",
"lastModified": 1795132051,
"narHash": "sha256-xPfYpDSNbLda+oKo/MIKEua20dKh/XoB4Tx5uUbeF5Q=",
"ref": "refs/heads/main",
"rev": "a6451cb0fa240d23f280c991d3d21e2784618b42",
"revCount": 652,
"rev": "87839b5992e215f43e69833359b8c76e89027495",
"revCount": 1702,
"type": "git",
"url": "ssh://git@github.com/asosnovsky/nbot.git"
"url": "ssh://git@github.com/SkyG-Org/nBot.git"
},
"original": {
"type": "git",
"url": "ssh://git@github.com/asosnovsky/nbot.git"
"url": "ssh://git@github.com/SkyG-Org/nBot.git"
}
},
"nix": {
@@ -2551,7 +2591,7 @@
},
"nixos-hardware_2": {
"inputs": {
"nixpkgs": "nixpkgs_12"
"nixpkgs": "nixpkgs_13"
},
"locked": {
"lastModified": 1792491056,
@@ -2648,6 +2688,22 @@
"type": "github"
}
},
"nixpkgs-slack": {
"locked": {
"lastModified": 1794707698,
"narHash": "sha256-5KAH1e8u/12aNUz79IA8xUlrzrwHIaxISaGoxsB/MyA=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "713bc1926b25ad17376e552e339faeea73cf86dc",
"type": "github"
},
"original": {
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "713bc1926b25ad17376e552e339faeea73cf86dc",
"type": "github"
}
},
"nixpkgs-stable": {
"locked": {
"lastModified": 1789009968,
@@ -2695,6 +2751,20 @@
}
},
"nixpkgs_11": {
"locked": {
"lastModified": 1794537146,
"narHash": "sha256-Wpz/7x/YWnDIBu7yj6ixe8j6fIMsidd9Y+o1PqPF3iM=",
"rev": "d5cc6833999746cffdfe0598f7bc801a57b887a9",
"revCount": 1119549,
"type": "tarball",
"url": "https://api.flakehub.com/f/pinned/NixOS/nixpkgs/0.1.1119549%2Brev-d5cc6833999746cffdfe0598f7bc801a57b887a9/01a1de0c-832c-7c20-8d39-1fe64a0165e2/source.tar.gz"
},
"original": {
"type": "tarball",
"url": "https://flakehub.com/f/NixOS/nixpkgs/0.1"
}
},
"nixpkgs_12": {
"locked": {
"lastModified": 1788404924,
"narHash": "sha256-lhEhY8X5EgkQ/eg6IFz4cc8jRuSYSvnxx1al7d1dvZ0=",
@@ -2710,7 +2780,7 @@
"type": "github"
}
},
"nixpkgs_12": {
"nixpkgs_13": {
"locked": {
"lastModified": 1789546076,
"narHash": "sha256-vWkSk5bbfTqdtMoSgD9FshACO8JCvXTFi+3cqEp0mH0=",
@@ -2723,7 +2793,7 @@
"url": "https://channels.nixos.org/nixos-unstable/nixexprs.tar.xz"
}
},
"nixpkgs_13": {
"nixpkgs_14": {
"locked": {
"lastModified": 1792503283,
"narHash": "sha256-vhe2jyvRCgufWPqDuDq1S3t3ypE+ka46IFutV02xKso=",
@@ -3092,6 +3162,7 @@
"home-manager": "home-manager_2",
"hyprland": "hyprland",
"hyprlauncher": "hyprlauncher",
"looprail": "looprail",
"my-nixpkgs": "my-nixpkgs",
"nbot": "nbot",
"nix-darwin": "nix-darwin",
@@ -3099,7 +3170,8 @@
"nix-index-database": "nix-index-database",
"nixarchy": "nixarchy",
"nixos-hardware": "nixos-hardware_2",
"nixpkgs": "nixpkgs_13",
"nixpkgs": "nixpkgs_14",
"nixpkgs-slack": "nixpkgs-slack",
"nixpkgs-unstable": "nixpkgs-unstable",
"noctalia": "noctalia",
"skyg-linux-tracker": "skyg-linux-tracker",
+25 -6
View File
@@ -4,16 +4,19 @@
skygqts.url = "git+ssh://gitea@gitea.app.internal:22/ari/skygqts.git";
skygqts.inputs.nixpkgs.follows = "nixpkgs";
hermenix.url = "git+ssh://gitea@gitea.app.internal:22/ari/hermenix.git";
nbot = {
url = "git+ssh://git@github.com/SkyG-Org/nBot.git";
inputs.nixpkgs.follows = "nixpkgs-unstable";
};
looprail = {
url = "git+ssh://git@github.com/SkyG-Org/LoopRail.git";
inputs.nixpkgs.follows = "nixpkgs-unstable";
};
# Read-only host monitor (systemd, journal, Docker) over REST + MCP.
skyg-linux-tracker = {
url = "git+https://gitea.skyg.ca/ari/skyg-linux-tracker.git";
inputs.nixpkgs.follows = "nixpkgs";
};
# Slack bot that runs goose agents in nspawn containers.
nbot = {
url = "git+ssh://git@github.com/asosnovsky/nbot.git";
inputs.nixpkgs.follows = "nixpkgs-unstable";
};
# Headless Hyprland session streamed over a Unix socket.
hhstream = {
url = "git+ssh://gitea@gitea.app.internal:22/Hermenix/hyprland-headless-stream-plugin.git";
@@ -31,6 +34,8 @@
determinate.url = "https://flakehub.com/f/DeterminateSystems/determinate/*";
determinate.inputs.nixpkgs.follows = "nixpkgs";
nixpkgs.url = "https://flakehub.com/f/NixOS/nixpkgs/0";
# Slack 4.53.287, NixOS/nixpkgs#557737.
nixpkgs-slack.url = "github:NixOS/nixpkgs/713bc1926b25ad17376e552e339faeea73cf86dc";
nixpkgs-unstable.url = "https://flakehub.com/f/NixOS/nixpkgs/0.1";
# my fork of nixpkgs
my-nixpkgs.url = "github:asosnovsky/nixpkgs/ds4-init";
@@ -114,8 +119,10 @@
, hermenix
, skyg-linux-tracker
, nbot
, looprail
, hhstream
, nixpkgs-unstable
, nixpkgs-slack
, flox
, determinate
, nixos-hardware
@@ -183,6 +190,7 @@
buildbot-gitea-oauth-secret = ./secrets/buildbot-gitea-oauth-secret.age;
immich-shared = ./secrets/immich-shared.age;
grafana-secret-key = ./secrets/grafana-secret-key.age;
umami-app-secret = ./secrets/umami-app-secret.age;
linux-tracker-peer-minipc1 = ./secrets/linux-tracker-peer-minipc1.age;
linux-tracker-peer-minipc2 = ./secrets/linux-tracker-peer-minipc2.age;
linux-tracker-peer-minipc3 = ./secrets/linux-tracker-peer-minipc3.age;
@@ -194,11 +202,13 @@
nbot-goose = ./secrets/nbot-goose.age;
nbot-cluster = ./secrets/nbot-cluster.age;
nbot-cluster-hl-fws1 = ./secrets/nbot-cluster-hl-fws1.age;
github-runner-token-fws1 = ./secrets/github-runner-token-fws1.age;
};
inherit
hyprlauncher
hyprland
nixpkgs-unstable
nixpkgs-slack
my-nixpkgs
dms
dank-greeter
@@ -210,6 +220,7 @@
hermenix
skyg-linux-tracker
nbot
looprail
hhstream
flox
determinate
@@ -232,14 +243,21 @@
let
pkgs = nixpkgs.legacyPackages.${system};
qtDocs = pkgs.qt6.qtdoc;
# Hook: run skyg pkgs update --all before every commit; fail if it bumped
# Hook: run skyg pkgs update --all at most once a week; fail if it bumped
# anything under pkgs/ so bumps get reviewed and staged, never auto-committed.
pkgs-update-script = pkgs.writeShellScriptBin "skyg-pkgs-update" ''
set -euo pipefail
export PATH=${pkgs.bash}/bin:${pkgs.coreutils}/bin:${pkgs.curl}/bin:${pkgs.jq}/bin:${pkgs.git}/bin:${pkgs.gnused}/bin:${pkgs.gawk}/bin:$PATH
root=$(${pkgs.git}/bin/git rev-parse --show-toplevel)
cd "$root"
stamp=$(${pkgs.git}/bin/git rev-parse --git-path pkgs-update-last-success)
now=$(date +%s)
last_run=$(date -r "$stamp" +%s 2>/dev/null || echo 0)
if (( now - last_run < 604800 )); then
exit 0
fi
${pkgs.nushell}/bin/nu --commands 'use ./bin/lib/cmds.nu *; skyg pkgs update --all'
touch "$stamp"
if ! ${pkgs.git}/bin/git diff --quiet -- pkgs/; then
echo ""
echo "pkgs-update: version bumps applied under pkgs/ — review 'git diff pkgs/',"
@@ -339,6 +357,7 @@
buzz-desktop = lib.pkgs.${system}.buzz-desktop;
delta-editor = lib.pkgs.${system}.delta-editor;
colibri = lib.pkgs.${system}.colibri;
llama-cpp-halo = lib.pkgs.${system}.llama-cpp-halo;
herdr = herdr.packages.${system}.default;
}
);
+2 -2
View File
@@ -36,8 +36,8 @@ hosts/
| `hl-bigbox1/` | hl-bigbox1 | AI/compute server (NVIDIA + AMD); Immich machine-learning (CUDA) |
| `hl-bigbox2/` | hl-bigbox2 | NFS storage server; Immich server/db (docker, `immich.home.sosnovsky.ca`) |
| `hl-minipc1/` | hl-minipc1 | Core infra: Gitea (native or containerized), nix-serve, registry, Buzz (docker, `buzz-manage` CLI), monitoring (Grafana, Prometheus, Loki, OTel collector) |
| `hl-minipc2/` | hl-minipc2 | Container/media server |
| `hl-minipc3/` | hl-minipc3 | K3s worker node; central Caddy reverse-proxy (`*.home.sosnovsky.ca`, plus `gitea.skyg.ca` HTTP/SSH); buildbot-nix CI master + worker (2 cores) |
| `hl-minipc2/` | hl-minipc2 | Container/media server; Umami for LoopRail telemetry (own nspawn container, `10.0.101.14`, reachable only from Caddy) |
| `hl-minipc3/` | hl-minipc3 | K3s worker node; central Caddy reverse-proxy (`*.home.sosnovsky.ca`, plus `gitea.skyg.ca` HTTP/SSH); serves LoopRail docs at `looprails.skyg.ca`, plus its guarded telemetry route `/t/v1` → Umami and the LAN-only `umami.home.sosnovsky.ca`; buildbot-nix CI master + worker (2 cores) |
| `hl-pi1/` | hl-pi1 | Raspberry Pi CM4 (aarch64) — headless server, SSH only |
| `hl-pi2/` | hl-pi2 | Raspberry Pi 4 (aarch64) — headless server, SSH only |
| `hl-terra1/` | hl-terra1 | NFS storage + Jellyfin backup |
+2
View File
@@ -3,10 +3,12 @@
imports = [
./hardware-configuration
./packages.nix
./slack.nix
./services.nix
./networking.nix
./hardware.nix
./systemd.nix
./omarchy.nix
];
skyg = {
+1 -1
View File
@@ -19,7 +19,7 @@ in
"x-scheme-handler/mailto" = "chromium-browser.desktop";
"x-scheme-handler/sgnl" = "signal.desktop";
"x-scheme-handler/signalcaptcha" = "signal.desktop";
"x-scheme-handler/slack" = "slack.desktop";
"x-scheme-handler/slack" = "slack-link-picker.desktop";
"x-scheme-handler/terminal" = "com.mitchellh.ghostty.desktop";
"application/pdf" = "chromium-browser.desktop";
"application/xhtml+xml" = "chromium-browser.desktop";
+34
View File
@@ -0,0 +1,34 @@
{ pkgs, nixarchy, skygUtils, config, ... }:
let
# Omarchy shell build
dedent = "${pkgs.gawk}/bin/awk 'BEGIN{m=9999} {l[NR]=$0; match($0,/^ */); if ($0 ~ /[^ ]/ && RLENGTH<m) m=RLENGTH} END{for(i=1;i<=NR;i++) print substr(l[i],m+1)}'";
omarchy = ((pkgs.extend nixarchy.overlays.default).omarchy).overrideAttrs (old: {
installPhase = builtins.replaceStrings
[ "python3 -c '" "' $menu\n" ]
[ "python3 -c \"$(printf '%s' '" "' | ${dedent})\" $menu\n" ]
old.installPhase;
});
# The shell's runtime deps, minus hyprland (we keep the flake's Hyprland).
# Omarchy runtime deps
omarchyRuntime = builtins.filter
(d: !(pkgs.lib.hasPrefix "hyprland-" (d.name or "")))
omarchy.passthru.runtimeDeps;
in
{
# Omarchy env vars (shell resolves paths via OMARCHY_PATH)
environment.sessionVariables.OMARCHY_PATH = "${omarchy}/share/omarchy";
environment.sessionVariables.OMARCHY_SCREENSHOT_EDITOR = "satty-edit";
# Omarchy config link: ~/.config/omarchy -> configs/fwbook/omarchy
system.userActivationScripts.omarchyConfig.text =
skygUtils.makeHyperlinkScriptToConfigs {
filePath = "fwbook/omarchy";
targetPath = "omarchy";
configSource = "/home/${config.skyg.user.name}/nixos-setup/configs";
};
# Packages
environment.systemPackages =
[ omarchy ]
++ omarchyRuntime;
}
+81 -125
View File
@@ -1,24 +1,5 @@
{ pkgs, unstablePkgs, hermes-agent, nixarchy, skygUtils, config, ... }:
{ pkgs, unstablePkgs, looprail, ... }:
let
# Omarchy (nixarchy) shell, hand-wired — no nixarchy modules. The shell is
# launched from configs/fwbook/hypr/conf/autostart.lua (omarchy-launch-shell).
#
# Workaround for an upstream bug in nixarchy v4.0.4-1: the omarchy package's
# installPhase is one '' string whose least-indented lines are the shell-quoted
# omarchy-theme-set patterns (indent 4), so Nix dedents the whole string by 4
# and the inline `python3 -c '...'` menu check stays indented -> IndentationError.
# Wrap that one invocation so its code is dedented before Python sees it.
dedent = "${pkgs.gawk}/bin/awk 'BEGIN{m=9999} {l[NR]=$0; match($0,/^ */); if ($0 ~ /[^ ]/ && RLENGTH<m) m=RLENGTH} END{for(i=1;i<=NR;i++) print substr(l[i],m+1)}'";
omarchy = ((pkgs.extend nixarchy.overlays.default).omarchy).overrideAttrs (old: {
installPhase = builtins.replaceStrings
[ "python3 -c '" "' $menu\n" ]
[ "python3 -c \"$(printf '%s' '" "' | ${dedent})\" $menu\n" ]
old.installPhase;
});
# The shell's runtime deps, minus hyprland (we keep the flake's Hyprland).
omarchyRuntime = builtins.filter
(d: !(pkgs.lib.hasPrefix "hyprland-" (d.name or "")))
omarchy.passthru.runtimeDeps;
gdk = pkgs.google-cloud-sdk.withExtraComponents (
with pkgs.google-cloud-sdk.components;
[
@@ -26,10 +7,6 @@ let
kubectl
]
);
# Resolve bundles its own Qt5, which has no wayland platform plugin. The
# Hyprland session exports QT_QPA_PLATFORM=wayland globally
# (configs/fwbook/hypr/conf/env.lua), so Resolve aborts at startup. Force
# xcb — Resolve is X11-only and runs under XWayland anyway.
davinci-resolve-wrapped = pkgs.symlinkJoin {
name = "davinci-resolve";
paths = [ pkgs.davinci-resolve ];
@@ -40,142 +17,121 @@ let
};
in
{
# The shell resolves everything through OMARCHY_PATH.
environment.sessionVariables.OMARCHY_PATH = "${omarchy}/share/omarchy";
environment.sessionVariables.OMARCHY_SCREENSHOT_EDITOR = "satty-edit";
# ~/.config/omarchy -> configs/fwbook/omarchy (same pattern as hypr/noctalia).
system.userActivationScripts.omarchyConfig.text =
skygUtils.makeHyperlinkScriptToConfigs {
filePath = "fwbook/omarchy";
targetPath = "omarchy";
configSource = "/home/${config.skyg.user.name}/nixos-setup/configs";
};
environment.systemPackages =
[ omarchy ]
++ omarchyRuntime
# Theme application: omarchy-theme-set-gnome writes gsettings, and every
# theme names a Yaru icon variant.
++ (with pkgs; [
(with pkgs; [
# Desktop & theme
glib
gsettings-desktop-schemas
gnome-themes-extra
yaru-theme
adwaita-icon-theme
])
++ (with pkgs; [
oreo-cursors-plus
# Screen & images
grim
slurp
wl-clipboard
tesseract
wl-screenrec
gpu-screen-recorder
gifski
imagemagick
zbar
curl
translate-shell
wl-screenrec
tesseract
# Media & creative
obs-studio
davinci-resolve-wrapped
shotcut
ffmpeg
gifski
libusb1
lshw
jq
parted
gparted-full
f3
nixd
kdePackages.qtdeclarative
krita
gimp-with-plugins
simple-scan
orca-slicer
# Chat & meetings
zoom-us
discord
betterdiscordctl
signal-desktop
telegram-desktop
# Office
onlyoffice-desktopeditors
darling-dmg
piper-tts
ffmpeg
oreo-cursors-plus
postgresql
google-cloud-sdk
awscli
openfortivpn
openfortivpn-webview
openfortivpn-webview-qt
nodejs
# Wine
wineWow64Packages.stable
winetricks
wineWow64Packages.waylandFull
obs-studio
davinci-resolve-wrapped
grim
slurp
wl-clipboard
tesseract
imagemagick
zbar
curl
translate-shell
wl-screenrec
ffmpeg
gifski
jq
# Cloud, VPN & data
postgresql
google-cloud-sdk
gdk
awscli
openfortivpn
openfortivpn-webview
openfortivpn-webview-qt
bitwarden-cli
krita
gimp-with-plugins
shotcut
simple-scan
zoom-us
betterdiscordctl
discord
signal-desktop
telegram-desktop
# Disks & hardware
libusb1
lshw
parted
gparted-full
f3
rpi-imager
rpiboot
libimobiledevice
ifuse
idevicerestore
# Languages & dev tools
nodejs
python313
python314
uv
pipx
cargo
rustc
go
pipx
rust-analyzer
vscode
unstablePkgs.zed-editor-fhs
unstablePkgs.et
unstablePkgs.devenv
nixd
kdePackages.qtdeclarative
just
rpi-imager
rpiboot
code-cursor-fhs
nix-prefetch
orca-slicer
devcontainer
gpu-screen-recorder
unstablePkgs.devcontainer
unstablePkgs.devenv
flox_dev
darling-dmg
bitwarden-cli
onlyoffice-desktopeditors
nix-prefetch
gh
gdk
unstablePkgs.slack
bubblewrap
delta-editor
libimobiledevice
ifuse
idevicerestore
# Editors
unstablePkgs.vscode
unstablePkgs.zed-editor-fhs
code-cursor-fhs
# Agents & LLM tools
unstablePkgs.claude-code
unstablePkgs.claude-agent-acp
unstablePkgs.grok-build
unstablePkgs.pi-coding-agent
unstablePkgs.goose-cli
unstablePkgs.claude-agent-acp
unstablePkgs.claude-code
buzz-desktop
bubblewrap
delta-editor
herdr
hermes-agent.packages.${pkgs.stdenv.hostPlatform.system}.default
skygqts
unstablePkgs.herdr
unstablePkgs.codex
unstablePkgs.codex-acp
looprail.packages.${pkgs.stdenv.hostPlatform.system}.looprail
hermenix
unstablePkgs.pi-coding-agent
# Remote & CLI tools
unstablePkgs.et
curl
jq
translate-shell
buzz-desktop
skygqts
]);
}
+65
View File
@@ -0,0 +1,65 @@
{ pkgs, nixpkgs-slack, ... }:
let
slack = (import nixpkgs-slack {
system = pkgs.stdenv.hostPlatform.system;
config.allowUnfreePredicate = pkg: pkgs.lib.getName pkg == "slack";
}).slack;
slackIcon = "${slack}/share/icons/hicolor/512x512/apps/slack.png";
makeSlack =
{ id, name, color }:
let
icon = pkgs.runCommand "slack-${id}-icon.png" { nativeBuildInputs = [ pkgs.imagemagick ]; } ''
magick ${slackIcon} -modulate 100,0 -fill '${color}' -tint 100 $out
'';
bin = pkgs.writeShellScriptBin "slack-${id}" ''
exec ${slack}/bin/slack --user-data-dir="''${XDG_CONFIG_HOME:-$HOME/.config}/Slack-${name}" "$@"
'';
desktop = pkgs.makeDesktopItem {
name = "slack-${id}";
desktopName = "Slack (${name})";
exec = "${bin}/bin/slack-${id} -s %U";
inherit icon;
startupWMClass = "Slack";
categories = [ "Network" "InstantMessaging" ];
};
in
pkgs.symlinkJoin {
name = "slack-${id}";
paths = [ bin desktop ];
};
slack-sibli = makeSlack { id = "sibli"; name = "Sibli"; color = "#2EB67D"; };
slack-personal = makeSlack { id = "personal"; name = "Personal"; color = "#4A154B"; };
# Opens slack:// links in the Slack you pick.
slack-link-picker =
let
bin = pkgs.writeShellScriptBin "slack-link-picker" ''
choice=$(printf '%s\n' "Slack" "Slack (Sibli)" "Slack (Personal)" \
| ${pkgs.rofi}/bin/rofi -dmenu -i -p "Open Slack link in")
case "$choice" in
"Slack") exec ${slack}/bin/slack -s "$@" ;;
"Slack (Sibli)") exec ${slack-sibli}/bin/slack-sibli -s "$@" ;;
"Slack (Personal)") exec ${slack-personal}/bin/slack-personal -s "$@" ;;
esac
'';
desktop = pkgs.makeDesktopItem {
name = "slack-link-picker";
desktopName = "Slack Link Picker";
exec = "${bin}/bin/slack-link-picker %U";
icon = "slack";
noDisplay = true;
mimeTypes = [ "x-scheme-handler/slack" ];
};
in
pkgs.symlinkJoin {
name = "slack-link-picker";
paths = [ bin desktop ];
};
in
{
environment.systemPackages = [
slack-sibli
slack-personal
slack-link-picker
];
}
+6
View File
@@ -1,4 +1,10 @@
{ ... }:
{
systemd.services.systemd-udev-settle.enable = false;
# Snapper needs a .snapshots subvolume in each snapshotted volume
systemd.tmpfiles.rules = [
"v /.snapshots 0750 root root -"
"v /home/.snapshots 0750 root root -"
];
}
+2
View File
@@ -17,7 +17,9 @@
gnome.enable = false;
};
common.ssh-server.enable = true;
common.diagnix.enable = true;
common.dns.enable = true;
common.hardware = {
nvidia.enable = true;
amdgpu.enable = true;
+1
View File
@@ -5,5 +5,6 @@
./portainer.nix
./timers.nix
./linux-tracker.nix
./llama-cpp.nix
];
}
+37
View File
@@ -0,0 +1,37 @@
{ ... }:
{
# llama.cpp with DFlash2 speculative decoding on RTX 3060 (12GB VRAM).
#
# Off by default: it needs ~19 GB of weights on disk. Fetch the files, then
# flip enable:
#
# sudo install -d -o ari -g users /var/lib/llama-cpp
# cd /var/lib/llama-cpp
# curl -L -O https://huggingface.co/ggml-org/Qwen3.8-27B-GGUF/resolve/main/Qwen3.8-27B-Q4_K_M.gguf
# curl -L -O https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2-GGUF/resolve/main/Qwen3.8-27B-DFlash2-Q8_0.gguf
#
# Measured expectation: ~15-25 tok/s (RTX 3060 has less bandwidth than Strix
# Halo, but the dedicated VRAM is snappier for inference).
skyg.nixos.server.services.llama-cpp = {
enable = false;
package = "llama-cpp-cuda";
model = "/var/lib/llama-cpp/Qwen3.8-27B-Q4_K_M.gguf";
draftModel = "/var/lib/llama-cpp/Qwen3.8-27B-DFlash2-Q8_0.gguf";
specType = "draft-dflash";
specDraftNMax = 7;
ctx = 32768;
kvCacheType = "q8_0";
cacheReuse = 256;
# RTX 3060: 12GB VRAM, offload most layers but leave headroom for context.
gpuLayers = 30;
parallel = 2;
host = "0.0.0.0";
port = 8081;
openFirewall = true;
};
}
+4
View File
@@ -13,12 +13,16 @@
user.enable = true;
nixos.common = {
ssh-server.enable = true;
diagnix.enable = true;
dns.enable = true;
containers.networks.ipvlanLab = {
enable = true;
parent = "enp3s0";
};
cachePush.enable = false;
cacheReceive.enable = true;
};
networkDrives = {
+8 -1
View File
@@ -1,4 +1,4 @@
{ pkgs, unstablePkgs, myPkgs, my-nixpkgs, ... }:
{ pkgs, my-nixpkgs, ... }:
let
ports = {
tabby = 11029;
@@ -40,11 +40,18 @@ in
environment.sessionVariables.ELECTRON_OZONE_PLATFORM_HINT = "wayland";
programs.nh.clean = {
dates = "daily";
extraArgs = "--keep-since 2d --keep 5";
};
skyg = {
user.enable = true;
core.qemu.enable = true;
nixos = {
common.ssh-server.enable = true;
common.diagnix.enable = true;
common.dns.enable = true;
common.cachePush.enable = true;
common.hardware = {
pipewire.enable = true;
+1
View File
@@ -1,6 +1,7 @@
{
imports = [
./ds4.nix
./llama-cpp.nix
./ollama.nix
./libretranslate.nix
./hermenix.nix
+1 -1
View File
@@ -4,7 +4,7 @@ let
in
{
services.ds4 = {
enable = true;
enable = false;
package = (myPkgs.ds4-rocm.override { rocmArch = "gfx1151"; });
user = "ari";
group = "users";
+1
View File
@@ -41,6 +41,7 @@ in
{ address = "10.0.101.3"; port = 8080; }
{ address = "10.0.101.3"; port = 9102; }
{ address = "10.0.101.8"; port = 3000; }
{ address = "10.0.10.0/24"; port = 22; }
];
publish = [
+32
View File
@@ -0,0 +1,32 @@
{ ... }:
{
# llama.cpp with DFlash2 speculative decoding.
#
# Off by default: it needs ~19 GB of weights on disk and a first build of the
# ROCm package. Fetch the files, then flip enable:
#
# sudo install -d -o ari -g users /var/lib/llama-cpp
# cd /var/lib/llama-cpp
# curl -L -O https://huggingface.co/ggml-org/Qwen3.8-27B-GGUF/resolve/main/Qwen3.8-27B-Q4_K_M.gguf
# curl -L -O https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2-GGUF/resolve/main/Qwen3.8-27B-DFlash2-Q8_0.gguf
#
# Measured expectation for this box: ~25-40 tok/s, not the 380 tok/s in the
# post that prompted this — that figure is a 3090 copying a document.
skyg.nixos.server.services.llama-cpp = {
enable = false;
model = "/var/lib/llama-cpp/Qwen3.8-27B-Q4_K_M.gguf";
draftModel = "/var/lib/llama-cpp/Qwen3.8-27B-DFlash2-Q8_0.gguf";
specType = "draft-dflash";
specDraftNMax = 7;
ctx = 32768;
kvCacheType = "q8_0";
cacheReuse = 256;
host = "0.0.0.0";
port = 8081;
openFirewall = true;
};
}
+16 -44
View File
@@ -2,6 +2,7 @@
let
ports = { cluster = 7946; };
homes = "/data/nbot";
goCache = "/data/nbot-cache/go";
secrets = [ "nbot-slack" "nbot-goose" "nbot-cluster" ];
settings = {
# The system each container boots (nbot's agent + gh); updated on every rebuild.
@@ -15,7 +16,12 @@ let
allow_indefinite = true;
boot_timeout = "2m";
# Summaries are always on now; only the model and its timeout are set.
# Go caches shared by all containers on this node (see ABOUTME). Env vars are in agent_env.
rw_binds = [
"${goCache}/mod:/var/cache/go/mod:idmap"
"${goCache}/build:/var/cache/go/build:idmap"
];
summaries = {
timeout = "10s";
};
@@ -23,49 +29,14 @@ let
# Goose settings sent to every container (also on fws1).
# Values come from the nbot-goose secret.
agent_env = {
GOOSE_PROVIDER = "$GOOSE_PROVIDER";
GOOSE_MODEL = "$GOOSE_MODEL";
OPENROUTER_API_KEY = "$OPENROUTER_API_KEY";
CLAUDE_CODE_OAUTH_TOKEN = "$CLAUDE_CODE_OAUTH_TOKEN";
COMMAND_CODE_API_KEY = "$COMMAND_CODE_API_KEY";
GH_TOKEN = "$GH_TOKEN";
DIAGNIX_KEY = "$DIAGNIX_KEY";
GOMODCACHE = "/var/cache/go/mod";
GOCACHE = "/var/cache/go/build";
};
model_providers = [
{
id = "claude-acp";
name = "Claude ACP (Ari Credentials)";
key_env = "CLAUDE_CODE_OAUTH_TOKEN";
models = [ "default" "sonnet" "haiku" "opus" ];
users = [ "U0CBGSS4U6A" ];
}
{
id = "claude-oauth";
name = "Claude OAuth";
personal = true;
upstream = "https://api.anthropic.com";
engine = "claude-code";
login = "claude";
models = [ "default" "opus" "sonnet" "haiku" "fable" ];
}
# {
# id = "codex";
# name = "Codex (OpenAI API key)";
# personal = true;
# upstream = "https://api.openai.com";
# engine = "openai";
# auth = "bearer";
# models = [ "gpt-6.1-codex" "gpt-6.1" ];
# }
# {
# id = "xai";
# name = "Grok";
# personal = true;
# upstream = "https://api.x.ai";
# engine = "openai";
# auth = "bearer";
# models = ["grok-4"];
# }
{
id = "custom_commandcode";
name = "Command Code";
@@ -92,9 +63,6 @@ let
max_container = { memory_mb = 16384; cpus = 4; };
host_reserve = { memory_mb = 32768; cpus = 4.0; };
storage = { min = "2G"; max = "5G"; limit = "20G"; };
# allow_subdir: the folder is a plain folder on the /data disk. The unit
# waits for that disk (services.nbot.drives), which is what the check in
# nbot is for.
drives = [{ path = homes; reserve = "40G"; allow_subdir = true; }];
cluster = {
@@ -121,7 +89,7 @@ in
enable = true;
mode = "master";
clusterPort = ports.cluster;
# The unit waits for these mounts before it starts.
kvm = true;
drives = [ homes ];
stateDir = "/var/lib/nbot";
configFile = (pkgs.formats.toml { }).generate "nbot.toml" settings;
@@ -135,5 +103,9 @@ in
networking.networkmanager.unmanaged = [ "interface-name:ve-*" ];
networking.dhcpcd.denyInterfaces = [ "ve-*" ];
systemd.tmpfiles.rules = [ "d ${homes} 0700 root root -" ];
systemd.tmpfiles.rules = [
"d ${homes} 0700 root root -"
"d ${goCache}/mod 0700 root root -"
"d ${goCache}/build 0700 root root -"
];
}
+7
View File
@@ -11,6 +11,8 @@
user.enable = true;
nixos = {
common.ssh-server.enable = true;
common.diagnix.enable = true;
common.dns.enable = true;
common.hardware.pipewire.enable = true;
desktop = {
enable = true;
@@ -53,6 +55,11 @@
playerctl
];
programs.nh.clean = {
dates = "daily";
extraArgs = "--keep-since 2d --keep 5";
};
services.flatpak.enable = true;
services.flatpak.packages = [ "com.spotify.Client" ];
+1 -1
View File
@@ -1,6 +1,6 @@
{ ... }:
{
imports = [ ./nbot.nix ];
imports = [ ./nbot.nix ./github-runners.nix ];
services.pipewire = {
extraConfig.pipewire-pulse."10-raop-discover" = {
+78
View File
@@ -0,0 +1,78 @@
{ config, lib, skyg-secrets, ... }:
let
concurrency = 4;
runnerNames = map (i: "runner-${toString i}") (lib.range 1 concurrency);
stateVersion = config.system.stateVersion;
cache = "/var/cache/github-runners/go";
credentials = "/run/github-runner-registration";
# The same numeric identity owns the bind-mounted caches on the host and in every container.
runnerUser = {
users.github-runner = {
uid = 915;
isSystemUser = true;
group = "github-runner";
};
groups.github-runner.gid = 915;
};
in
{
age.secrets.github-runner-token-fws1 = {
file = skyg-secrets.github-runner-token-fws1;
path = "${credentials}/token";
# The containers mount this directory and must see token rotations.
symlink = false;
mode = "0400";
};
users = runnerUser;
systemd.tmpfiles.rules = [
"d ${credentials} 0700 root root -"
"d ${cache} 0755 root root -"
"d ${cache}/build 0700 github-runner github-runner -"
"d ${cache}/mod 0700 github-runner github-runner -"
];
systemd.services = lib.genAttrs (map (name: "container@${name}") runnerNames) (_: {
serviceConfig = {
MemoryLow = "4G";
MemoryMax = "8G";
CPUQuota = "200%";
};
});
containers = lib.genAttrs runnerNames (name: {
autoStart = true;
bindMounts = {
"${credentials}".isReadOnly = true;
"/var/cache/go" = {
hostPath = cache;
isReadOnly = false;
};
};
config = { pkgs, ... }: {
system.stateVersion = stateVersion;
users = runnerUser;
nix.settings.experimental-features = [ "nix-command" "flakes" ];
services.github-runners.${name} = {
enable = true;
inherit name;
replace = true;
url = "https://github.com/SkyG-Org";
tokenFile = "${credentials}/token";
user = "github-runner";
group = "github-runner";
extraLabels = [ "fws1" "nixos" "go" ];
extraPackages = [ pkgs.go ];
extraEnvironment = {
NIX_REMOTE = "daemon";
GOCACHE = "/var/cache/go/build";
GOMODCACHE = "/var/cache/go/mod";
};
serviceOverrides.ReadWritePaths = [ "/var/cache/go" ];
};
};
});
}
+13 -5
View File
@@ -1,6 +1,7 @@
{ config, pkgs, skyg-secrets, ... }:
let
homes = "/var/lib/nbot/homes";
goCache = "/var/lib/nbot-cache/go";
settings = {
# The system each container boots (nbot's agent + gh); updated on every rebuild.
system = "${config.skyg.nixos.server.services.nbot-agent.system}";
@@ -13,11 +14,16 @@ let
allow_indefinite = true;
boot_timeout = "2m";
# Go caches shared by all containers on this node (see ABOUTME). Env vars come from the master.
rw_binds = [
"${goCache}/mod:/var/cache/go/mod:idmap"
"${goCache}/build:/var/cache/go/build:idmap"
];
container = { memory_mb = 4096; cpus = 1.0; };
max_container = { memory_mb = 51200; cpus = 6; };
host_reserve = { memory_mb = 8192; cpus = 2.0; };
storage = { min = "2G"; max = "10G"; limit = "50G"; };
# allow_subdir: the folder is a plain folder on the root disk.
drives = [{ path = homes; reserve = "200G"; allow_subdir = true; }];
# Worker of the master on fwdesk. Slack tokens and goose settings come
@@ -26,8 +32,6 @@ let
cluster = {
name = "hl-fws1";
master = "wss://fwdesk.lab.internal:7946";
# A list, so the master's certificate can be rotated with no time where
# a slave refuses the master: add the new fingerprint, then drop the old.
master_fingerprint = [ "ce08c6b0f3aec855cc8b936d81449c1a34074d181089b26acbdddacfca60af1d" ];
};
};
@@ -43,7 +47,7 @@ in
services.nbot = {
enable = true;
mode = "slave";
# The unit waits for these mounts before it starts.
kvm = true;
drives = [ homes ];
stateDir = "/var/lib/nbot";
configFile = (pkgs.formats.toml { }).generate "nbot.toml" settings;
@@ -57,5 +61,9 @@ in
# so wait-online would always time out.
systemd.network.wait-online.enable = false;
systemd.tmpfiles.rules = [ "d ${homes} 0700 root root -" ];
systemd.tmpfiles.rules = [
"d ${homes} 0700 root root -"
"d ${goCache}/mod 0700 root root -"
"d ${goCache}/build 0700 root root -"
];
}
+2
View File
@@ -37,6 +37,8 @@ in
skyg.user.enable = true;
skyg.nixos.common.ssh-server.enable = true;
skyg.nixos.common.diagnix.enable = true;
skyg.nixos.common.dns.enable = true;
skyg.server.dns.routing = {
enable = false;
openFirewall = true;
+2
View File
@@ -8,6 +8,8 @@
skyg.user.enable = true;
skyg.nixos.common.ssh-server.enable = true;
skyg.nixos.common.diagnix.enable = true;
skyg.nixos.common.dns.enable = true;
skyg.nixos.common.containers.networks.ipvlanLab = {
enable = true;
parent = "enp2s0";
+1
View File
@@ -7,5 +7,6 @@
./stack2.nix
./timers.nix
./linux-tracker.nix
./umami.nix
];
}
+74
View File
@@ -0,0 +1,74 @@
{ config, skyg-secrets, ... }:
# Umami collects LoopRail's telemetry. Caddy on minipc3 sends it the public
# POST looprails.skyg.ca/t/v1 and the LAN-only dashboard. It runs in its own
# nspawn container with its own LAN address, and only Caddy can reach it.
let
app = config.skyg.internalNetworkingMap.apps.umami;
caddy = config.skyg.internalNetworkingMap.apps.caddy;
appNetwork = config.skyg.internalNetworkingMap.appNetwork;
parent = config.skyg.nixos.common.containers.networks.ipvlanLab.parent;
# nspawn names the ipvlan link iv-<parent>.
lanIf = "iv-${parent}";
port = 3000;
secretPath = "/run/secrets/umami-app-secret";
hostStateVersion = config.system.stateVersion;
in
{
age.secrets.umami-app-secret.file = skyg-secrets.umami-app-secret;
containers.umami = {
autoStart = true;
# Own network: an ipvlan link on the LAN, like the docker apps get.
privateNetwork = true;
extraFlags = [ "--network-ipvlan=${parent}" ];
bindMounts.${secretPath} = {
hostPath = config.age.secrets.umami-app-secret.path;
isReadOnly = true;
};
config = { lib, ... }: {
system.stateVersion = hostStateVersion;
networking = {
useDHCP = false;
interfaces.${lanIf}.ipv4.addresses = [{
address = app.ip;
prefixLength = lib.toInt (lib.last (lib.splitString "/" appNetwork.subnet));
}];
defaultGateway = { address = appNetwork.gateway; interface = lanIf; };
nftables.enable = true;
firewall = {
enable = true;
# Only Caddy may reach Umami, at most ~20 new connections a second.
# Caddy opens one connection per telemetry request, so this caps requests.
extraInputRules = ''
ip saddr ${caddy.ip} tcp dport ${toString port} ct state new limit rate 20/second burst 100 packets accept
'';
};
};
# Postgres only on its local socket.
services.postgresql.settings.listen_addresses = lib.mkForce "";
services.umami = {
enable = true;
createPostgresqlDatabase = true;
settings = {
APP_SECRET_FILE = secretPath;
HOSTNAME = "0.0.0.0";
PORT = port;
DISABLE_UPDATES = true;
DISABLE_TELEMETRY = true;
# looprail's User-Agent would look like a bot.
DISABLE_BOT_CHECK = true;
};
};
};
};
# A flood can't starve the rest of the host.
systemd.services."container@umami".serviceConfig = {
MemoryMax = "1G";
CPUQuota = "100%";
};
}
+2
View File
@@ -8,6 +8,8 @@
skyg.user.enable = true;
skyg.nixos.common.ssh-server.enable = true;
skyg.nixos.common.diagnix.enable = true;
skyg.nixos.common.dns.enable = true;
skyg.server.dns.routing = {
enable = false;
openFirewall = true;
+48
View File
@@ -55,6 +55,54 @@ gitea.skyg.ca {
reverse_proxy {$GITEA_IP}:80
}
looprails.skyg.ca {
tls /etc/caddy/tls/gitea/fullchain.pem /etc/caddy/tls/gitea/key.pem
# Telemetry: only looprail's own requests reach Umami.
@telemetry {
method POST
path /t/v1
header Content-Type application/json*
header User-Agent looprail/*
header X-Looprail-Telemetry {$LOOPRAIL_TELEMETRY_KEY}
}
handle @telemetry {
request_body {
max_size 256KB
}
rewrite * /api/batch
reverse_proxy {$UMAMI_IP}:3000 {
# Umami never sees the sender's IP.
header_up -X-Forwarded-For
header_up -X-Real-IP
header_up -Forwarded
header_up -CF-Connecting-IP
header_up -True-Client-IP
header_up -X-Client-IP
header_up -X-Looprail-Telemetry
# One connection per request, so Umami's rate limit counts requests.
transport http {
keepalive off
}
}
}
handle /t/* {
respond 404
}
handle {
root * /srv/looprails
redir /looprail.html / permanent
file_server
}
}
# Umami dashboard for LoopRail telemetry, LAN only.
umami.home.sosnovsky.ca {
import lan_only
reverse_proxy {$UMAMI_IP}:3000
}
# Buzz: straight to its own caddy, LAN only.
buzz.home.sosnovsky.ca {
import lan_only
+1
View File
@@ -11,6 +11,7 @@
orderBefore = [ "container-services-caddy" ];
};
certs."gitea.skyg.ca" = {
extraDomains = [ "looprails.skyg.ca" ];
postRun = "${pkgs.docker}/bin/docker restart caddy-caddy-1";
orderBefore = [ "container-services-caddy" ];
};
+12 -1
View File
@@ -1,5 +1,6 @@
{ config, ... }:
{ config, looprail, pkgs, ... }:
let
looprailPackages = looprail.packages.${pkgs.stdenv.hostPlatform.system};
app = config.skyg.internalNetworkingMap.apps.caddy;
nvr = config.skyg.internalNetworkingMap.apps.nvr;
jellyfin = config.skyg.internalNetworkingMap.apps.jellyfin;
@@ -7,6 +8,10 @@ let
portainer = config.skyg.internalNetworkingMap.apps.portainer;
buzz = config.skyg.internalNetworkingMap.apps.buzz;
publicGitea = config.skyg.internalNetworkingMap.apps.public-gitea;
umami = config.skyg.internalNetworkingMap.apps.umami;
# Sent by release looprail binaries on telemetry posts. It ships in the
# binary, so it only filters out scanners; it is not a secret.
looprailTelemetryKey = "70c685d238cc9ec7229e557a8de7b0d25527b1a402805251";
certDomain = "home.sosnovsky.ca";
giteaCertDomain = "gitea.skyg.ca";
# k8s Traefik ingress — fallback for hosts caddy doesn't handle itself.
@@ -34,9 +39,15 @@ in
PORTAINER_IP = portainer.ip;
JELLYFIN_IP = jellyfin.ip;
BUZZ_IP = buzz.ip;
UMAMI_IP = umami.ip;
LOOPRAIL_TELEMETRY_KEY = looprailTelemetryKey;
K8S_INGRESS = k8sIngress;
};
volumes = [
(if looprailPackages ? looprail-site then
"${looprailPackages.looprail-site}:/srv/looprails:ro"
else
"${looprail}/docs/looprail.html:/srv/looprails/index.html:ro")
"/var/lib/acme/${certDomain}/fullchain.pem:/etc/caddy/tls/fullchain.pem:ro"
"/var/lib/acme/${certDomain}/key.pem:/etc/caddy/tls/key.pem:ro"
"/var/lib/acme/${giteaCertDomain}/fullchain.pem:/etc/caddy/tls/gitea/fullchain.pem:ro"
+1
View File
@@ -13,6 +13,7 @@
minimal = true;
fonts.minimal = true;
ssh-server.enable = true;
diagnix.enable = true;
containers.enable = false;
networking.nfsServer.enable = false;
hardware.pipewire.enable = true;
+1
View File
@@ -10,6 +10,7 @@
user.enable = true;
nixos = {
common.ssh-server.enable = true;
common.diagnix.enable = true;
common.hardware.pipewire.enable = true;
desktop = {
enable = true;
+1
View File
@@ -20,6 +20,7 @@ in
skyg.user.enable = true;
skyg.nixos.common.ssh-server.enable = true;
skyg.nixos.common.diagnix.enable = true;
skyg.server.dns.routing = {
enable = false;
openFirewall = true;
+4
View File
@@ -22,6 +22,10 @@
# colibrì (JustVugg/colibri) — GLM-5.2/OLMoE local inference, see pkgs/colibri.
colibri = final.callPackage ../../pkgs/colibri { };
colibri-rocm = final.callPackage ../../pkgs/colibri { backend = "rocm"; };
# llama.cpp v0.8.0 with DFlash2 speculative decoding, see pkgs/llama-cpp.
llama-cpp-halo = final.callPackage ../../pkgs/llama-cpp { };
llama-cpp-halo-cpu = final.callPackage ../../pkgs/llama-cpp { backend = "cpu"; };
llama-cpp-cuda = final.callPackage ../../pkgs/llama-cpp { backend = "cuda"; };
# Touchscreen gesture bridge for niri, see pkgs/niri-touchscreen-gestures.
niri-touchscreen-gestures = final.callPackage ../../pkgs/niri-touchscreen-gestures { };
# Buzz Desktop AppImage wrapper, see pkgs/buzz-desktop.
+3
View File
@@ -112,6 +112,7 @@ in
immich = { ip = "10.0.101.10"; };
immich-ml = { ip = "10.0.101.11"; };
gilix = { ip = "10.0.101.13"; };
umami = { ip = "10.0.101.14"; };
jellyfin = { ip = "10.0.101.5"; };
nvr = { ip = "10.0.101.6"; };
portainer = { ip = "10.0.101.241"; };
@@ -123,6 +124,8 @@ in
"portainer.home.sosnovsky.ca"
"immich.home.sosnovsky.ca"
"gitea.skyg.ca"
"looprails.skyg.ca"
"umami.home.sosnovsky.ca"
];
};
};
+19
View File
@@ -15,6 +15,7 @@ common/
├── user.nix # skyg.user.createSystemUser — creates the system user/groups + zsh shell
├── fonts.nix # System font packages + fontconfig defaults (Fira Code, Noto, …); skyg.nixos.common.fonts.minimal
├── ssh-server.nix # skyg.nixos.common.ssh-server — hardened sshd + master public keys (list)
├── diagnix/ # skyg.nixos.common.diagnix — read-only user nbot uses to diagnose hosts
├── qemu.nix # skyg.core.qemu — QEMU/quickemu + SPICE guest tooling
├── binary-cache.nix # skyg.nixos.common.cachePush — push finished builds to the bigbox2 cache
@@ -38,6 +39,23 @@ common/
- **`ssh-server.nix`** ships the master public keys (`masterPubKeys`, a list so a key can be
rotated by adding the new one before dropping the old) and a hardened sshd config
(key-only auth, long keepalives). Disabled by default.
- **`diagnix/`** adds a read-only `diagnix` user (uid 2950) that nbot uses over SSH, via
`lab-ssh` in the agent system (`server/services/nbot-agent.nix`). It reads logs
(`systemd-journal` group), units, `/proc` and the store, and can't change anything. It
can't read root-only files. Through sudo it can run only the fixed `diag-*` scripts
(docker ps/logs, smart, ports, firewall, btrfs, io, pstore). The key is `restrict,pty` and only
works from the nbot hosts (`hosts.nix`). `diag-pstore` prints the kernel's saved crash
records from `/sys/fs/pstore` (root-only), the one trace a hard freeze can leave behind.
The hermenix VM has its own key (`hosts.nix` → `hermenix`), with the same limits, accepted
only from hl-fwdesk's addresses (the VM is NAT'ed out of fwdesk). Turn it off per host
with `skyg.nixos.common.diagnix.hermenix.enable = false`. fwdesk's `hermenix.nix` lets
the VM reach port 22 on `10.0.10.0/24`. Hosts running nbot also let their own containers
reach port 22 and accept them from private IPs outside the LAN. Their IPs are random, so
they can't be listed. Resources are capped (1G memory, 1 CPU, 256 tasks). The Nix daemon
is left open on purpose: an allow-list would break nix-serve/buildbot.
The private key (base64) is `DIAGNIX_KEY` in the `nbot-goose` secret; fwdesk passes it to
every container via `agent_env`.
`hosts.nix` also holds each host's address and host key; the agent pins those keys.
## Option Namespace
@@ -46,6 +64,7 @@ skyg.core.hostName → networking.nix
skyg.user.createSystemUser → user.nix
skyg.core.qemu.* → qemu.nix
skyg.nixos.common.ssh-server.* → ssh-server.nix
skyg.nixos.common.diagnix.* → diagnix/
skyg.nixos.common.cachePush.* → binary-cache.nix
skyg.nixos.common.minimal → core.nix (skip heavy always-on system packages)
+115 -21
View File
@@ -1,35 +1,129 @@
{ config, lib, pkgs, ... }:
let
cfg = config.skyg.nixos.common.cachePush;
recv = config.skyg.nixos.common.cacheReceive;
lab = import ./diagnix/hosts.nix;
# The hook writes into tmp and moves into queue, which the path unit watches.
# A push that fails moves its files to retry, for the timer.
state = "/var/lib/nix-cache-push";
queue = "${state}/queue";
retry = "${state}/retry";
# nix-serve on bigbox2 only serves its own store and can't take uploads, so
# pushes go over ssh-ng into that store, as the host's root key.
sshOpts = "-i /etc/ssh/ssh_host_ed25519_key -o StrictHostKeyChecking=yes -o BatchMode=yes -o ConnectTimeout=10";
drain = pkgs.writeShellScript "nix-cache-push-drain" ''
set -u
shopt -s nullglob
new=(${queue}/*)
old=(${retry}/*)
files=("''${new[@]}" "''${old[@]}")
[ ''${#files[@]} -eq 0 ] && exit 0
paths=()
for f in "''${files[@]}"; do
for p in $(cat "$f"); do
[ -e "$p" ] && paths+=("$p")
done
done
if [ ''${#paths[@]} -gt 0 ]; then
if ! NIX_SSHOPTS="${sshOpts}" ${config.nix.package}/bin/nix copy --to "${cfg.url}" "''${paths[@]}"; then
[ ''${#new[@]} -gt 0 ] && mv "''${new[@]}" ${retry}/
exit 1
fi
fi
rm -f "''${files[@]}"
'';
in
{
options.skyg.nixos.common.cachePush = {
enable = lib.mkEnableOption "push finished builds to the bigbox2 binary cache";
url = lib.mkOption {
type = lib.types.str;
default = "http://bigbox2.lab.internal:5000";
default = "ssh-ng://nix-ssh@bigbox2.lab.internal";
};
host = lib.mkOption {
type = lib.types.str;
default = "bigbox2.lab.internal";
description = "The cache's ssh host, whose host key is pinned from diagnix/hosts.nix.";
};
};
config = lib.mkIf cfg.enable {
# Push every finished build to the cache right after it finishes.
nix.settings.post-build-hook = pkgs.writeShellScript "nix-post-build-hook" ''
set -u
if [ -n "''${OUT_PATHS:-}" ]; then
nix copy --to "${cfg.url}" $OUT_PATHS \
|| echo "post-build-hook: failed to push to ${cfg.url}" >&2
fi
'';
# Manual/backstop push command, available on every host's PATH.
environment.systemPackages = [
(pkgs.writeShellScriptBin "push-cache" ''
set -eu
if [ "''$#" -eq 0 ]; then
set -- /run/current-system
fi
exec nix copy --to "${cfg.url}" "''$@"
'')
];
options.skyg.nixos.common.cacheReceive = {
enable = lib.mkEnableOption "accept pushes into this host's store over ssh-ng (nix-ssh)";
pushers = lib.mkOption {
type = lib.types.listOf lib.types.str;
default = [ "hl-fwdesk" ];
description = "Hosts in diagnix/hosts.nix whose ssh host key may push.";
};
};
config = lib.mkMerge [
(lib.mkIf cfg.enable {
# The hook only queues the paths: Nix waits for a post-build-hook before it
# goes on, so copying here held up every build for the whole upload.
nix.settings.post-build-hook = pkgs.writeShellScript "nix-post-build-hook" ''
set -u
if [ -n "''${OUT_PATHS:-}" ]; then
tmp=$(mktemp -p ${state}/tmp) || exit 0
echo "$OUT_PATHS" > "$tmp" && mv "$tmp" "${queue}/$(date +%s%N)-$$"
fi
'';
systemd.tmpfiles.rules = map (d: "d ${d} 0700 root root -") [ state "${state}/tmp" queue retry ];
programs.ssh.knownHosts.nix-cache-push = {
hostNames = [ cfg.host ];
publicKey = (lib.findFirst (h: h.address == cfg.host) null (lib.attrValues lab.hosts)).hostKey;
};
systemd.services.nix-cache-push = {
description = "Push queued builds to ${cfg.url}";
after = [ "network-online.target" ];
wants = [ "network-online.target" ];
path = [ config.programs.ssh.package ];
serviceConfig = {
Type = "oneshot";
ExecStart = drain;
Nice = 10;
IOSchedulingClass = "idle";
};
};
# Runs when the hook queues a build, and every 10 minutes for the retries.
systemd.paths.nix-cache-push = {
wantedBy = [ "multi-user.target" ];
pathConfig.DirectoryNotEmpty = queue;
};
systemd.timers.nix-cache-push = {
wantedBy = [ "timers.target" ];
timerConfig.OnUnitInactiveSec = "10min";
timerConfig.OnBootSec = "5min";
};
# Manual/backstop push command, available on every host's PATH.
environment.systemPackages = [
(pkgs.writeShellScriptBin "push-cache" ''
set -eu
if [ "''$#" -eq 0 ]; then
set -- /run/current-system
fi
NIX_SSHOPTS="${sshOpts}" exec nix copy --to "${cfg.url}" "''$@"
'')
];
})
(lib.mkIf recv.enable {
nix.sshServe = {
enable = true;
write = true;
protocol = "ssh-ng";
# Pushed paths aren't signed; nix-serve signs them with this host's key
# when it serves them.
trusted = true;
keys = map (h: lab.hosts.${h}.hostKey) recv.pushers;
};
})
];
}
@@ -52,5 +52,7 @@ If adding a new per-service option:
- The group's unit sets `restartTriggers` to the files-write script, so editing any
`files` entry restarts the stack on the next switch (services that read files at
startup, like Gitea templates, otherwise keep serving the old content)
- Env file changes trigger a path unit that restarts the stack (`container-services-<group>-env-reload.*`)
- Env file changes trigger a path unit that restarts the stack (`container-services-<group>-env-reload.*`);
the restart is queued with `systemctl restart --no-block`, so the reload unit does not wait for or
inherit the health of the stack it restarted
- Compose files are rendered to the store and staged to `/var/lib/container-services/<group>/compose.yml`
@@ -22,6 +22,7 @@
"network-online.target"
"agenix.service"
"remote-fs.target"
"dns-boot-check.service"
]
++ lib.optional hasFiles fileServiceDep;
requires = [ runtimeService ]
+2
View File
@@ -11,6 +11,8 @@
./qemu.nix
./binary-cache.nix
./pritunl
./diagnix
./dns
];
config = {
environment.sessionVariables.EDITOR = "vi";
+130
View File
@@ -0,0 +1,130 @@
# diagnix: a read-only user nbot logs in as to diagnose a host.
# Sees logs, units, /proc and /nix/store; can't change the system.
{ config, lib, pkgs, options, ... }:
let
cfg = config.skyg.nixos.common.diagnix;
data = import ./hosts.nix;
uid = 2950;
# This host runs nbot, so its own containers connect from random private IPs.
hasNbot = options.services ? nbot && config.services.nbot.enable;
keyLine = from: key: ''restrict,pty,from="${from}" ${key}'';
runtime =
if config.virtualisation.docker.enable then "${config.virtualisation.docker.package}/bin/docker"
else if config.virtualisation.podman.enable then "${config.virtualisation.podman.package}/bin/podman"
else null;
rt = toString runtime;
# Fixed root scripts diagnix may run with sudo. Anything taking input checks it.
script = name: inputs: text: pkgs.writeShellApplication {
inherit name text;
runtimeInputs = inputs;
};
scripts = [
(script "diag-smart" [ pkgs.smartmontools pkgs.nvme-cli ] ''
smartctl --scan | while read -r dev _; do
echo "=== $dev"; smartctl -a "$dev" || true
done
for dev in /dev/nvme[0-9]; do
[ -e "$dev" ] || continue
echo "=== $dev"; nvme smart-log "$dev" || true
done
'')
(script "diag-ports" [ pkgs.iproute2 ] "ss -tulpn")
(script "diag-firewall" [ pkgs.iptables pkgs.nftables ] ''
iptables -S || true; iptables -t nat -S || true; ip6tables -S || true
nft list ruleset || true
'')
(script "diag-btrfs" [ pkgs.btrfs-progs pkgs.util-linux ] ''
findmnt -rn -t btrfs -o TARGET | sort -u | while read -r mnt; do
echo "=== $mnt"; btrfs filesystem usage "$mnt" || true; btrfs device stats "$mnt" || true
done
'')
(script "diag-io" [ pkgs.iotop ] "iotop -b -o -n 3 -d 1")
(script "diag-pstore" [ ] ''
shopt -s nullglob
files=(/sys/fs/pstore/*)
if [ ''${#files[@]} -eq 0 ]; then echo "pstore is empty"; exit 0; fi
for f in "''${files[@]}"; do
echo "=== $f"; cat "$f" || true
done
'')
] ++ lib.optionals (runtime != null) [
(script "diag-docker-ps" [ ] ''
${rt} ps -a; ${rt} stats --no-stream; ${rt} system df
'')
(script "diag-docker-logs" [ ] ''
if [ $# -ne 1 ] || ! [[ "$1" =~ ^[A-Za-z0-9][A-Za-z0-9_.-]*$ ]]; then
echo "usage: diag-docker-logs <container-name>" >&2; exit 2
fi
${rt} logs --tail 500 "$1" 2>&1
'')
];
in
{
options.skyg.nixos.common.diagnix = {
enable = lib.mkEnableOption "the read-only diagnix user for nbot";
publicKey = lib.mkOption {
type = lib.types.str;
default = data.publicKey;
description = "nbot's client public key for diagnix.";
};
hermenix.enable = lib.mkOption {
type = lib.types.bool;
default = true;
description = "Let the hermenix VM log in as diagnix too.";
};
};
config = lib.mkIf cfg.enable {
users.groups.diagnix.gid = uid;
users.users.diagnix = {
inherit uid;
isNormalUser = true;
group = "diagnix";
description = "nbot read-only diagnostics";
shell = pkgs.bash;
extraGroups = [ "systemd-journal" ];
openssh.authorizedKeys.keys =
[ (keyLine (lib.concatStringsSep "," data.nbotHosts) cfg.publicKey) ]
# Containers on this host: any private IP outside the LAN.
++ lib.optional hasNbot (keyLine "!${data.lanSubnet},10.0.0.0/8,172.16.0.0/12,192.168.0.0/16" cfg.publicKey)
++ lib.optional cfg.hermenix.enable (keyLine (lib.concatStringsSep "," data.hermenix.from) data.hermenix.publicKey);
};
# A runaway command can't starve the host.
systemd.slices."user-${toString uid}" = {
overrideStrategy = "asDropin";
sliceConfig = { MemoryMax = "1G"; CPUQuota = "100%"; TasksMax = 256; };
};
security.sudo.extraRules = [{
users = [ "diagnix" ];
commands = map (s: { command = lib.getExe s; options = [ "NOPASSWD" ]; }) scripts;
}];
programs.mtr.enable = true;
environment.systemPackages = scripts ++ (with pkgs; [
lsof
pciutils
usbutils
dnsutils
sysstat
ethtool
lm_sensors
htop
]);
# nbot drops all container traffic to the host except a few ports; let SSH in too.
networking.firewall = lib.mkIf (hasNbot && !config.networking.nftables.enable) {
extraCommands = lib.mkAfter ''
iptables -C INPUT -i ve-+ -p tcp --dport 22 -j ACCEPT 2>/dev/null || iptables -I INPUT 1 -i ve-+ -p tcp --dport 22 -j ACCEPT
'';
extraStopCommands = ''
iptables -D INPUT -i ve-+ -p tcp --dport 22 -j ACCEPT 2>/dev/null || true
'';
};
};
}
+30
View File
@@ -0,0 +1,30 @@
# Lab hosts diagnix lives on, shared by the host module and nbot's agent system.
# hostKey is the host's ssh_host_ed25519 public key (from secrets.nix);
# null means the agent saves it on first connect.
{
# LAN addresses of the hosts that run nbot. The key only works from these.
nbotHosts = [ "10.0.10.11" "10.0.10.13" ]; # hl-fwdesk, hl-fws1
lanSubnet = "10.0.0.0/16";
# The nbot client public key. Private half (base64) is DIAGNIX_KEY in the
# nbot-goose secret, sent to containers via agent_env on fwdesk.
publicKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPKsgkysVa3BiJQkUjiyESwjoi9ZMmUiWEB66JlbgIrO nbot-diagnix";
hermenix = {
publicKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKSYYLaAYJKifV3oOwnF7taNV4ko+A+A/r4oHgpKBwRW hermenix-diagnix";
from = [ "10.0.10.11" "10.0.101.101" "10.77.0.2" ];
};
hosts = {
hl-bigbox1 = { address = "bigbox1.lab.internal"; hostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFl5xiH4aFlmDWdoU+KmERLYOeOQK2aKP7jenY2G3rFb"; };
hl-bigbox2 = { address = "bigbox2.lab.internal"; hostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIL6l6VqQ23PkGkWVABBoLp1KA6PzkX5oKTy5RBHbQXet"; };
hl-fwdesk = { address = "fwdesk.lab.internal"; hostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHbBcdMpNPaELUzHRIoRkTUQde+mBQ87Ra2R+974FuUy"; };
hl-fws1 = { address = "fws1.lab.internal"; hostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM6r0AIJyTlE+U6K97GijzAP4jNaAueVlwXbeMpEMcpe"; };
hl-minipc1 = { address = "minipc1.lab.internal"; hostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIP42vtV9Qo7q5F9M/rCGD/TIZG3d0mm1uP6lMvy3zF/a"; };
hl-minipc2 = { address = "minipc2.lab.internal"; hostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDqi2WT0pa8m+kZNN2Rrg1F01zwTvkCwD3wjSOBNvSjz"; };
hl-minipc3 = { address = "minipc3.lab.internal"; hostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEbvoe/UvhpRSWVXMCyj/qzuR7WnJdNF7CYRmmJE3yYu"; };
hl-pi1 = { address = "pi1.lab.internal"; hostKey = null; };
hl-pi2 = { address = "pi2.lab.internal"; hostKey = null; };
hl-terra1 = { address = "terra1.lab.internal"; hostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHBKI8gqd7DDubShle4D3JGq61BP1NBKCZ81V3C9uyym"; };
};
}
+35
View File
@@ -0,0 +1,35 @@
# Boot-time DNS check
Six hosts opt into this module, including hl-bigbox1. It was introduced after the reported 79-minute DNS outage on that host on 2026-11-16. The incident account suspected journal corruption after an unclean shutdown. This module keeps logs for investigation.
## Configuration
```nix
skyg.nixos.common.dns.enable = true;
```
| Option under `skyg.nixos.common.dns` | Default | Purpose |
| --- | --- | --- |
| `enable` | `false` | Enable resolved, persistent journal storage, and the boot check. |
| `bootCheckName` | `"portainer.app.internal"` | Name expected to resolve to an IPv4 address on this host's network. |
| `bootCheckRetries` | `5` | Maximum lookup attempts, spaced one second apart after each failure. |
| `watchdogEnabled` | `true` | Configure systemd to feed a hardware watchdog when available. |
| `watchdogTimeout` | `30` | Runtime watchdog timeout in seconds if systemd stops feeding it. |
## Boot behavior
After network-online.target and systemd-resolved start, dns-boot-check queries the host stub at 127.0.0.53 for the configured name. A returned IPv4 address passes the check. A cached answer can pass. Choose a record available independently of containers starting on the same host.
The container service groups are ordered after the check. An exhausted retry budget leaves the diagnostic unit failed and allows the groups to start. A DNS failure alone does not trigger a reboot. The probe runs once per boot and can be rerun manually.
The journal persists across boots. The module sets a 500M persistent storage limit, a 100M runtime limit and a 30-day retention limit. Host configuration can override these limits.
## Investigating a failure
```bash
journalctl -b -u dns-boot-check -u systemd-resolved
resolvectl status
dig +time=2 +tries=1 @127.0.0.53 portainer.app.internal A
```
Use the configured name if the host overrides the default. Check the DNS status and answer section, and compare the upstream servers and routing domains in `resolvectl status` with the intended network. After correcting the problem, run `systemctl restart dns-boot-check` and check `systemctl status dns-boot-check`. For a failure on the previous boot, use `journalctl -b -1` with the same unit filters.
+89
View File
@@ -0,0 +1,89 @@
{ config, lib, pkgs, ... }:
let
cfg = config.skyg.nixos.common.dns;
in
{
options.skyg.nixos.common.dns = {
enable = lib.mkEnableOption "systemd-resolved with a boot-time DNS diagnostic";
bootCheckName = lib.mkOption {
type = lib.types.strMatching "[a-zA-Z0-9][a-zA-Z0-9.-]*";
default = "portainer.app.internal";
description = "Internal DNS name expected to have an IPv4 address at boot.";
};
bootCheckRetries = lib.mkOption {
type = lib.types.ints.positive;
default = 5;
description = "Maximum number of DNS lookup attempts at boot.";
};
watchdogEnabled = lib.mkOption {
type = lib.types.bool;
default = true;
description = "Enable the system manager's hardware watchdog.";
};
watchdogTimeout = lib.mkOption {
type = lib.types.ints.positive;
default = 30;
description = "Runtime hardware watchdog timeout in seconds.";
};
};
config = lib.mkIf cfg.enable {
services.resolved = {
enable = true;
settings.Resolve = {
LLMNR = "yes";
DNSSEC = "allow-downgrade";
DNSOverTLS = false;
Domains = config.networking.search ++ [ "~." ];
};
};
services.journald = {
storage = "persistent";
extraConfig = lib.mkBefore ''
SystemMaxUse=500M
SystemMaxFileSize=50M
RuntimeMaxUse=100M
RuntimeMaxFileSize=10M
MaxRetentionSec=30day
MaxFileSec=7day
'';
};
systemd.settings.Manager = lib.mkIf cfg.watchdogEnabled {
RuntimeWatchdogSec = "${toString cfg.watchdogTimeout}s";
RebootWatchdogSec = "${toString (cfg.watchdogTimeout + 10)}s";
};
systemd.services.dns-boot-check = {
description = "Check internal DNS resolution at boot";
wantedBy = [ "multi-user.target" ];
wants = [ "network-online.target" ];
after = [ "network-online.target" "systemd-resolved.service" ];
requires = [ "systemd-resolved.service" ];
path = [ pkgs.dnsutils pkgs.gnugrep pkgs.coreutils ];
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
};
script = ''
name=${lib.escapeShellArg cfg.bootCheckName}
attempts=${toString cfg.bootCheckRetries}
for ((attempt=1; attempt<=attempts; attempt++)); do
if answer=$(dig +short +time=2 +tries=1 @127.0.0.53 -q "$name" -t A) &&
grep -Eq '^([0-9]{1,3}\.){3}[0-9]{1,3}$' <<< "$answer"; then
echo "$name resolved through 127.0.0.53"
exit 0
fi
echo "No IPv4 answer for $name (attempt $attempt/$attempts)" >&2
if ((attempt < attempts)); then sleep 1; fi
done
exit 1
'';
};
};
}
+2 -6
View File
@@ -72,12 +72,8 @@ let
"x-scheme-handler/http" = "chromium-browser.desktop";
"x-scheme-handler/https" = "chromium-browser.desktop";
};
portal = {
enable = true;
extraPortals = [
pkgs.xdg-desktop-portal
];
};
# Backends come from each desktop module (niri, hyprland, kde, ...)
portal.enable = true;
};
};
in
+13
View File
@@ -16,6 +16,7 @@ services/
├── nbot-agent.nix # skyg.nixos.server.services.nbot-agent — nbot's agent system + gh, for nbot's `system`
├── signal-cli.nix # skyg.nixos.server.services.signal-cli — signal-cli HTTP daemon (Hermes Signal bridge)
├── colibri.nix # skyg.nixos.server.services.colibri — coli serve HTTP API (GLM-5.2/OLMoE local inference)
├── llama-cpp.nix # skyg.nixos.server.services.llama-cpp — llama-server (GGUF, DFlash2 speculative decoding)
└── comfyui/ # skyg.nixos.server.services.comfyui — ComfyUI with ROCm/CUDA GPU support
```
@@ -27,10 +28,21 @@ services/
repo unchanged. `gh` reads `GH_TOKEN` from `agent_env`, and git over HTTPS to github.com
uses it through `gh`. Every container gets that token (chosen on purpose, 2026-11-03),
so anyone who can prompt an agent can use it or ask it to print the token.
- **Shared Go caches** (2026-11-11, set in `hosts/hl-{fwdesk,fws1}/services/nbot.nix`, not
here): each node bind-mounts one `GOMODCACHE` and `GOCACHE` into every container under
`/var/cache/go/`, with the env vars in the master's `agent_env`. Go locks these, so sharing
is safe. Only Go caches are shared, not all of `~/.cache`: other tools break or leak logins
when shared, and it would skip each home's size limit. Trade-off: any container can change
build results the others use. The caches have no size limit; clear with `go clean -cache`.
- **`colibri.nix`** runs `coli serve` (colibrì's OpenAI-compatible API) as a native systemd
service. `package` picks the backend (`pkgs.colibri`
cpu / `pkgs.colibri-rocm`); `environmentFile` supplies `COLI_API_KEY` via agenix instead of
a CLI flag.
- **`llama-cpp.nix`** runs `llama-server` as a native systemd service. `draftModel` plus
`specType = "draft-dflash"` turns on DFlash2 speculative decoding; `cacheReuse` turns on
prefix reuse for repeated prompts (agents). Model files are read from disk, never packaged —
the unit refuses to start with a clear message when one is missing. Enabled on fwdesk
(off by default: ~19 GB of weights, first build is the ROCm package).
## Option Namespace
@@ -41,6 +53,7 @@ skyg.nixos.server.services.jellyfin.enable
skyg.nixos.server.services.nbot-agent.system
skyg.nixos.server.services.signal-cli.{enable,host,port,configDir,account,environmentFile,openFirewall}
skyg.nixos.server.services.colibri.{enable,package,model,host,port,environmentFile,openFirewall,...}
skyg.nixos.server.services.llama-cpp.{enable,package,model,draftModel,specType,specDraftNMax,cacheReuse,ctx,host,port,openFirewall,...}
skyg.nixos.server.services.comfyui.{enable,mode,port,openFirewall}
```
@@ -4,6 +4,7 @@
./audiobookshelf.nix
./ai-services.nix
./colibri.nix
./llama-cpp.nix
./comfyui
./jellyfin.nix
./nbot-agent.nix
+252
View File
@@ -0,0 +1,252 @@
{ config, lib, pkgs, ... }:
with lib;
let
cfg = config.skyg.nixos.server.services.llama-cpp;
skygUser = config.skyg.user;
# Target model plus optional drafter. The drafter is only useful for a
# strategy that matches its architecture: DFlash2 sidecars want
# `--spec-type draft-dflash`.
modelFiles = [ cfg.model ] ++ optional (cfg.draftModel != null) cfg.draftModel;
serverArgs =
[
"--model"
cfg.model
"--host"
cfg.host
"--port"
(toString cfg.port)
"--ctx-size"
(toString cfg.ctx)
"--n-gpu-layers"
(toString cfg.gpuLayers)
"--parallel"
(toString cfg.parallel)
"--flash-attn"
cfg.flashAttention
"--cache-type-k"
cfg.kvCacheType
"--cache-type-v"
cfg.kvCacheType
# Qwen-style tool calls arrive as Jinja templates.
"--jinja"
]
++ optionals (cfg.draftModel != null) [
"--model-draft"
cfg.draftModel
]
++ optionals (cfg.specType != null) [
"--spec-type"
cfg.specType
]
++ optionals (cfg.specDraftNMax != null) [
"--spec-draft-n-max"
(toString cfg.specDraftNMax)
]
# Reuse the longest matching prefix of the previous prompt instead of
# reprocessing it. This is the "prefix caching" half of the speedup, and it
# works with any model.
++ optionals (cfg.cacheReuse != null) [
"--cache-reuse"
(toString cfg.cacheReuse)
]
++ cfg.extraArgs;
checkModels = pkgs.writeShellScript "llama-cpp-check-models" ''
set -eu
${concatMapStringsSep "\n" (
f: ''
if [ ! -f ${escapeShellArg f} ]; then
echo "llama-cpp: missing model file: ${f}" >&2
exit 1
fi
''
) modelFiles}
'';
in
{
options.skyg.nixos.server.services.llama-cpp = {
enable = mkEnableOption "the llama.cpp HTTP API (llama-server)";
package = mkOption {
type = types.package;
default = pkgs.llama-cpp-halo;
defaultText = literalExpression "pkgs.llama-cpp-halo";
description = ''
Server package. `pkgs.llama-cpp-halo` is the ROCm build for gfx1151;
`pkgs.llama-cpp-halo-cpu` is the same tree without the GPU backend.
'';
};
model = mkOption {
type = types.str;
example = "/var/lib/llama-cpp/Qwen3.8-27B-Q4_K_M.gguf";
description = "Path to the target model (GGUF).";
};
draftModel = mkOption {
type = types.nullOr types.str;
default = null;
example = "/var/lib/llama-cpp/Qwen3.8-27B-DFlash2-Q8_0.gguf";
description = ''
Optional draft model (GGUF). A drafter must be trained for the exact
target model — a mismatch loads and then drafts nothing useful.
'';
};
specType = mkOption {
type = types.nullOr types.str;
default = null;
example = "draft-dflash";
description = ''
Speculative decoding strategy passed to `--spec-type`. Use
`draft-dflash` for a DFlash/DFlash2 drafter, `draft-mtp` for a model's
own MTP heads, or `ngram-simple` for prompt-copying workloads (code
edits, document rewriting) where no drafter is needed at all.
'';
};
specDraftNMax = mkOption {
type = types.nullOr types.int;
default = null;
example = 7;
description = ''
Tokens drafted per verification step. Clamped by the drafter's trained
block size; 7 is the value upstream uses for DFlash2.
'';
};
cacheReuse = mkOption {
type = types.nullOr types.int;
default = null;
example = 256;
description = ''
`--cache-reuse`: minimum run of tokens to reuse from a matching prompt
prefix. 256 is a reasonable default for coding agents.
'';
};
ctx = mkOption {
type = types.int;
default = 32768;
description = "Context size in tokens.";
};
kvCacheType = mkOption {
type = types.enum [
"f16"
"bf16"
"q8_0"
"q4_0"
];
default = "q8_0";
description = "KV cache type for both K and V (q8_0 halves KV memory).";
};
flashAttention = mkOption {
type = types.enum [
"on"
"off"
"auto"
];
default = "auto";
description = "`--flash-attn` value.";
};
gpuLayers = mkOption {
type = types.int;
default = 99;
description = ''
Layers offloaded to the GPU. On Strix Halo the GPU shares system RAM,
so all layers fit and 99 is the right value.
'';
};
parallel = mkOption {
type = types.int;
default = 1;
description = "Parallel request slots.";
};
host = mkOption {
type = types.str;
default = "127.0.0.1";
example = "0.0.0.0";
description = "Address llama-server binds to.";
};
port = mkOption {
type = types.port;
default = 8081;
description = "TCP port for the HTTP API.";
};
user = mkOption {
type = types.str;
default = skygUser.name;
description = "User to run the server as. Needs the render/video groups for GPU access.";
};
group = mkOption {
type = types.str;
default = "users";
description = "Group to run the server as.";
};
environment = mkOption {
type = types.attrsOf types.str;
default = {
# Same workaround the box's Ollama and ds4 units use.
HSA_ENABLE_SDMA = "0";
};
description = "Environment variables for the service.";
};
extraArgs = mkOption {
type = types.listOf types.str;
default = [ ];
example = [
"--metrics"
"--no-mmap"
];
description = "Extra arguments appended to the llama-server command line.";
};
openFirewall = mkEnableOption "the llama-server port in the firewall";
};
config = mkIf cfg.enable {
assertions = [
{
assertion = cfg.specType == null || cfg.draftModel != null || cfg.specType != "draft-dflash";
message = "services.llama-cpp: specType = \"draft-dflash\" needs a draftModel.";
}
];
systemd.services.llama-cpp = {
description = "llama.cpp server (local LLM inference)";
wantedBy = [ "multi-user.target" ];
after = [ "network.target" ];
environment = cfg.environment;
serviceConfig = {
User = cfg.user;
Group = cfg.group;
ExecStartPre = checkModels;
ExecStart = "${lib.getExe' cfg.package "llama-server"} ${escapeShellArgs serverArgs}";
Restart = "on-failure";
RestartSec = 5;
# Model files live here; the unit may create it, weights are downloaded
# by hand (they are tens of GB and not part of the system closure).
StateDirectory = "llama-cpp";
WorkingDirectory = "/var/lib/llama-cpp";
LimitNOFILE = 65536;
TimeoutStopSec = 120;
};
};
networking.firewall.allowedTCPPorts = mkIf cfg.openFirewall [ cfg.port ];
};
}
+31 -4
View File
@@ -1,11 +1,38 @@
# nbot's agent system with our additions. Use it for nbot's `system`.
# nbot's agent system with our additions (gh, lab-ssh). Use it for nbot's `system`.
{ lib, nbot, ... }:
with lib;
let
lab = import ../../common/diagnix/hosts.nix;
# lab-ssh <host> [cmd...]: log in to a lab host as the read-only diagnix user.
# The key comes base64-encoded in $DIAGNIX_KEY (agent_env, from the nbot-goose secret).
labSsh = pkgs: pkgs.writeShellApplication {
name = "lab-ssh";
runtimeInputs = [ pkgs.openssh pkgs.coreutils ];
text = ''
host="''${1:-}"
case "$host" in
${concatStrings (mapAttrsToList (name: h: ''
${name}|${removePrefix "hl-" name}) addr=${h.address}; check=${if h.hostKey == null then "accept-new" else "yes"} ;;
'') lab.hosts)}
*) echo "usage: lab-ssh <host> [command...]; hosts: ${concatStringsSep " " (attrNames lab.hosts)}" >&2; exit 2 ;;
esac
shift
: "''${DIAGNIX_KEY:?DIAGNIX_KEY missing from agent_env}"
key="$HOME/.ssh/diagnix"
mkdir -p "$HOME/.ssh" && chmod 700 "$HOME/.ssh"
(umask 077; base64 -d <<<"$DIAGNIX_KEY" > "$key")
exec ssh -i "$key" -o IdentitiesOnly=yes -o StrictHostKeyChecking="$check" "diagnix@$addr" "$@"
'';
};
agentModule = { pkgs, ... }: {
environment.systemPackages = [ pkgs.gh ];
environment.systemPackages = [ pkgs.gh (labSsh pkgs) ];
programs.ssh.knownHosts = mapAttrs
(name: h: { hostNames = [ name h.address ]; publicKey = h.hostKey; })
(filterAttrs (_: h: h.hostKey != null) lab.hosts);
# gh reads GH_TOKEN (from agent_env); git over HTTPS uses it through gh.
programs.git = {
enable = true;
@@ -15,13 +42,13 @@ let
in
{
options.skyg.nixos.server.services.nbot-agent.system = mkOption {
description = "nbot's agent system with gh added. Use for nbot's `system`.";
description = "nbot's agent system with gh and lab-ssh added. Use for nbot's `system`.";
readOnly = true;
type = types.package;
default = (nbot.nixosConfigurations.agent.extendModules {
modules = [ agentModule ];
}).config.system.build.toplevel;
# Keeps the manual from building it on every host.
defaultText = literalExpression "nbot agent + gh";
defaultText = literalExpression "nbot agent + gh + lab-ssh";
};
}
+2 -1
View File
@@ -12,6 +12,7 @@ Packages here are exposed via an overlay defined in `modules/core/default.nix`.
| `buzz-desktop/` | Block Buzz desktop (Type-2 AppImage from `block/buzz`). Wrapped with `appimageTools.wrapType2` plus extra FHS libs (`elfutils`, `zstd`, WebKit/tray, GStreamer good/bad) so it runs on NixOS. Exposed as `pkgs.buzz-desktop` (`buzz-desktop` binary + desktop entry / icons). |
| `buzz-cli/` | Buzz CLI (`buzz`, `crates/buzz-cli` of `block/buzz`), built with `rustPlatform.buildRustPackage` from the same tag as `buzz-desktop`. Hermes's Buzz platform shells out to it. Bump with `update.sh` (sets the source hash, resets `cargoHash` to `lib.fakeHash`; build once and paste the `got:` hash). Robots call it with `pkgs.callPackage` (their nixpkgs has no host overlay). |
| `delta-editor/` | Delta Editor (Zed Industries' AI-native editor, `delta.dev`) prebuilt tarball. No stable URL exists — the release API returns a short-lived signed R2 link — so `src` is a fixed-output derivation that resolves the API at build time. Interpreter/RPATH patched (`--force-rpath`, `dontPatchELF`), dlopen'd graphics libs (`libglvnd`, `vulkan-loader`, `wayland`) added via wrapper, `XKB_CONFIG_ROOT` set for the bundled libxkbcommon. Exposed as `pkgs.delta-editor` (`delta-editor` binary + desktop entry / icons); named `-editor` to avoid clashing with the `delta` git diff pager. |
| `llama-cpp/` | llama.cpp pinned to v0.8.0, the first release with DFlash2 drafters (`--spec-type draft-dflash`). nixpkgs only carries v0.6.0 in the unstable input and an older tree in the stable one, so the server is built here. HIP build for Strix Halo (`CMAKE_HIP_ARCHITECTURES=gfx1151`, ROCm clang++, `autoPatchelfHook`). Backend-parameterized: `pkgs.llama-cpp-halo` (rocm, default) and `pkgs.llama-cpp-halo-cpu` (same tree, no GPU backend). Web UI is off — it downloads prebuilt assets at build time, which the sandbox cannot do. Model weights are not packaged. |
## Adding a New Package
@@ -32,7 +33,7 @@ Each package can have an `update.sh` script that automates version bumps.
- **`skyg pkgs update --all`** — auto-detect the latest version for all packages that have `update.sh`.
- **`skyg pkgs update <pkg-name>`** — auto-detect the latest version for a single package.
- **`skyg pkgs update <pkg-name> <version>`** — bump a specific package to a specific version.
- **Pre-commit hook** — a `pkgs-update` hook (defined in `flake.nix`) runs `skyg pkgs update --all` before every commit and blocks the commit if it changed anything under `pkgs/` (review + `git add pkgs/`, then commit again).
- **Pre-commit hook** — a `pkgs-update` hook (defined in `flake.nix`) runs `skyg pkgs update --all` on the first commit attempt and then at most once every seven days. It records successful runs in Git's local metadata; failed updates are retried on the next commit. If it changed anything under `pkgs/`, it blocks the commit (review + `git add pkgs/`, then commit again without rerunning the update).
Each `update.sh` supports two modes:
+3 -3
View File
@@ -13,17 +13,17 @@
rustPlatform.buildRustPackage rec {
pname = "buzz-cli";
version = "0.5.28";
version = "0.5.30";
src = fetchFromGitHub {
owner = "block";
repo = "buzz";
rev = "desktop-v${version}";
hash = "sha256-NEsLQwcoQ2zTgznFgGFGNgYPnoclicNVycSEjl7I0Wo=";
hash = "sha256-Ay6zUeHHYO92/Fs4FGHm1+QQUoUEQjYyOC3dDg5+7Ss=";
};
# The workspace has git deps in [patch.crates-io]; cargoHash covers them.
cargoHash = "sha256-UQ8qNqoeDwkwhmVZebi++gb2GP61dr1Zv2QU9mJ6KJ8=";
cargoHash = lib.fakeHash;
# Only this crate; the workspace has 30+ others (relay, db, ...).
cargoBuildFlags = [ "-p" "buzz-cli" ];
+2 -2
View File
@@ -23,11 +23,11 @@
let
pname = "buzz-desktop";
version = "0.5.28";
version = "0.5.30";
src = fetchurl {
url = "https://github.com/block/buzz/releases/download/desktop-v${version}/Buzz_${version}_amd64.AppImage";
hash = "sha256-YCI6wn2HQiJqC149teH5dEGyZ/9jq7kFeTsz/mVoh0w=";
hash = "sha256-AAZc110TQPDucGamgOUgimlXw4mjquzBgvTwc6oWzZo=";
};
extracted = appimageTools.extractType2 {
+2 -2
View File
@@ -29,7 +29,7 @@
let
pname = "delta-editor";
version = "0.24.0";
version = "0.28.0";
apiUrl = "https://delta.dev/api/releases/nightly/${version}/asset?asset=delta&os=linux&arch=x86_64";
@@ -38,7 +38,7 @@ let
inherit version;
outputHashMode = "flat";
outputHash = "sha256-nWmR0N0HG/mvxmLCGInfBqdFWZ678CqNUSq0EIkE71o=";
outputHash = "sha256-LryJftc5QdosIIaMnuX/PDcqXe3GHyOSbYfX0ZuhCBM=";
nativeBuildInputs = [ curl jq cacert ];
SSL_CERT_FILE = "${cacert}/etc/ssl/certs/ca-bundle.crt";
+108
View File
@@ -0,0 +1,108 @@
{
lib,
stdenv,
fetchFromGitHub,
cmake,
ninja,
pkg-config,
autoPatchelfHook,
rocmPackages,
cuda_packages,
# "rocm" builds the gfx1151 engine (Strix Halo). "cuda" builds for NVIDIA
# GPUs (e.g., RTX 3060). "cpu" is the same tree without a GPU backend.
backend ? "rocm",
rocmArch ? "gfx1151",
}:
# llama.cpp, pinned to the first release that carries DFlash2 drafters
# (v0.8.0, 2026-10-28). nixpkgs only has v0.6.0 in the unstable input and an
# older tree in the stable one, and DFlash2 landed upstream after both — hence
# this package. Supports ROCm (AMD, Strix Halo) and CUDA (NVIDIA, RTX 3060).
#
# To bump: set `version` and `rev`, refresh `hash` with
# `nix-prefetch-url --unpack https://github.com/ggml-org/llama.cpp/archive/refs/tags/vX.Y.Z.tar.gz`
# then `nix hash convert --to sri <hash>`. Check the release notes for a new
# `--spec-type` before assuming a drafter still loads.
assert lib.elem backend [
"cpu"
"rocm"
"cuda"
];
let
rocmInputs = [
rocmPackages.clr
rocmPackages.rocm-runtime
rocmPackages.hipblas
rocmPackages.rocblas
];
cudaInputs = [
cuda_packages.cudatoolkit
cuda_packages.cudnn
];
in
stdenv.mkDerivation {
pname = "llama-cpp" + lib.optionalString (backend == "rocm") "-halo" + lib.optionalString (backend == "cuda") "-cuda" + lib.optionalString (backend == "cpu") "-cpu";
version = "0.8.0";
src = fetchFromGitHub {
owner = "ggml-org";
repo = "llama.cpp";
tag = "v0.8.0";
hash = "sha256-WQAtD8Hos6fEtgPQId7nZiLGyP4asAMKJYEXZid1nqc=";
};
nativeBuildInputs = [
cmake
ninja
pkg-config
]
++ lib.optionals (backend == "rocm") [
autoPatchelfHook
# Provides clang++ and the HIP headers; llama.cpp wants ROCm's own clang.
rocmPackages.clr
]
++ lib.optionals (backend == "cuda") [
autoPatchelfHook
cuda_packages.cudatoolkit
];
buildInputs = lib.optionals (backend == "rocm") rocmInputs ++ lib.optionals (backend == "cuda") cudaInputs;
# The web UI is a prebuilt asset downloaded at build time, which the sandbox
# cannot do. The API does not need it.
cmakeFlags = [
(lib.cmakeBool "GGML_NATIVE" false)
(lib.cmakeBool "BUILD_SHARED_LIBS" false)
(lib.cmakeBool "LLAMA_BUILD_SERVER" true)
(lib.cmakeBool "LLAMA_BUILD_TESTS" false)
(lib.cmakeBool "LLAMA_BUILD_EXAMPLES" false)
(lib.cmakeBool "LLAMA_BUILD_APP" false)
(lib.cmakeBool "LLAMA_BUILD_UI" false)
(lib.cmakeBool "LLAMA_USE_PREBUILT_UI" false)
(lib.cmakeBool "LLAMA_OPENSSL" false)
# Upstream reads these from git, which the release tarball does not carry.
(lib.cmakeFeature "LLAMA_BUILD_NUMBER" "31309")
(lib.cmakeFeature "LLAMA_BUILD_COMMIT" "8642541b2")
]
++ lib.optionals (backend == "rocm") [
(lib.cmakeBool "GGML_HIP" true)
(lib.cmakeFeature "CMAKE_HIP_COMPILER" "${rocmPackages.clr.hipClangPath}/clang++")
(lib.cmakeFeature "CMAKE_HIP_ARCHITECTURES" rocmArch)
]
++ lib.optionals (backend == "cuda") [
(lib.cmakeBool "GGML_CUDA" true)
(lib.cmakeFeature "CMAKE_CUDA_COMPILER" "${cuda_packages.cudatoolkit}/bin/nvcc")
];
doCheck = false;
meta = {
description = "llama.cpp server (DFlash2 speculative decoding) with ROCm/CUDA support";
homepage = "https://github.com/ggml-org/llama.cpp";
license = lib.licenses.mit;
platforms = lib.platforms.linux;
mainProgram = "llama-server";
};
}
Binary file not shown.
+6
View File
@@ -9,6 +9,7 @@ let
bigbox2 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIL6l6VqQ23PkGkWVABBoLp1KA6PzkX5oKTy5RBHbQXet root@hl-bigbox2";
hermenix = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPfMrT6dDPw3+J22k3X5aR9ERQ8RMRLeBGgqQjmuL7EI hermenix@lab";
fws1 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM6r0AIJyTlE+U6K97GijzAP4jNaAueVlwXbeMpEMcpe root@hl-fws1";
diagnix = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPKsgkysVa3BiJQkUjiyESwjoi9ZMmUiWEB66JlbgIrO nbot-diagnix";
in
{
"secrets/dns-addresses.conf.age".publicKeys = [
@@ -34,6 +35,8 @@ in
"secrets/nbot-cluster.age".publicKeys = [ ari fwdesk ];
"secrets/nbot-cluster-hl-fws1.age".publicKeys = [ ari fws1 ];
"secrets/github-runner-token-fws1.age".publicKeys = [ ari fws1 ];
# Buzz stack env (minipc1)
"secrets/buzz-env.age".publicKeys = [ ari minipc1 ];
@@ -77,4 +80,7 @@ in
"secrets/linux-tracker-peer-bigbox1.age".publicKeys = [ ari fwdesk ];
"secrets/linux-tracker-peer-bigbox2.age".publicKeys = [ ari fwdesk ];
"secrets/linux-tracker-peer-terra1.age".publicKeys = [ ari fwdesk ];
# Unami App
"secrets/umami-app-secret.age".publicKeys = [ ari minipc2 ];
}
+7
View File
@@ -0,0 +1,7 @@
age-encryption.org/v1
-> ssh-ed25519 CIR0mA L83QWLnE5GVGD0exCETcRL+nDKb42ItcG3N2cpPaGgs
9PiiapcU18WKrwvAqgiZNjuYQEo61sykwVZtvJ8JS+8
-> ssh-ed25519 KFBpLw QTpLVyI872MepWcJyeFQBR6RB+xVy9lK28kshgl+IiQ
dFKmJDffaC2Ig8fEwb9kJ0tdedK24J/CnjK40Yor5HQ
--- WeEsbTKGDvopQNxQpXMCTL4hhqmhljQb7bv5c4oaHXw
Bžþoú#” µ×gÆE+‹O…¹^ ÙÅ,%LO÷¦uàYJÃh‚ø‚ç\É"¹ÑçÞËŠí¥ã
Binary file not shown.
Binary file not shown.
Binary file not shown.
+7
View File
@@ -0,0 +1,7 @@
age-encryption.org/v1
-> ssh-ed25519 CIR0mA 8VtMsT7T4XfvUbix/ZBQjNtYNBvEsznysusG/JjlSV0
NvrhVNN2F95A+H9szc8eIwL7SbLT8UEngEXnSbRV1QM
-> ssh-ed25519 ug+sMw gCL0H0WywGuFMkmihmK1IRe5jw16cMZ4PK19+8WL+Tc
TBpUBZoZORO8x0sLifLF49LQv6gimFU15+iCX6gXCFU
--- TRnC3PR8LvnH6uOl5GW1rE+T0UNoJtgkQ8Lmm3sueks
>ÜF׈H'¯K“û XÅR_0Ì™—’>Õ p¤.KÆ872s+iÛE\Ä‘Þ?dA.YNÐR¸)‚`°ÁF!