- Containers get an enforcing bazel wrapper: remote spawn strategy, no local fallback, and refused override flags. Gateway rc is written at boot. - NativeLink: ExecPaths for the DynamicUser state dir (was noexec). - Bazel server exits after 5 idle minutes; shared per-host repo cache (/repo-cache, idmap); goose hints say a local build means trouble. - Removed the /handoff -> /outputs materializer; outputs download into bazel-bin again. - slackerbot containers: 4G/1 CPU, up to 8G/2 CPUs. - fws1: disable networkd wait-online. - slackerbot 4db99ea (send_file symlinked home, trimmed tool lines). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0119iKoN3fSvRGUAYHupPW4U
5.5 KiB
5.5 KiB
modules/nixos/server/services/
Self-hosted application services. default.nix imports each one; every service is an
independent skyg.nixos.server.services.<name>.enable flag. Most run as OCI containers via
virtualisation.oci-containers (so they honour the Docker/Podman runtime chosen in
common/containers/), often backed by NFS volumes.
Files
services/
├── default.nix # imports the services below
├── ai-services.nix # skyg.nixos.server.services.ai — Wyoming (whisper/piper/wake)
├── bazel-build.nix # skyg.nixos.server.services.bazel-build — NativeLink Bazel build server for slackerbot containers
├── audiobookshelf.nix # skyg.nixos.server.services.audiobookshelf — audiobook/podcast server
├── jellyfin.nix # skyg.nixos.server.services.jellyfin — media server (uid/gid 7777)
├── signal-cli.nix # skyg.nixos.server.services.signal-cli — signal-cli HTTP daemon (Hermes Signal bridge)
├── colibri.nix # skyg.nixos.server.services.colibri — coli serve HTTP API (GLM-5.2/OLMoE local inference)
└── comfyui/ # skyg.nixos.server.services.comfyui — ComfyUI with ROCm/CUDA GPU support
Notable details
ai-services.nixopenwakeword as containers and enables NixOS-native Wyoming faster-whisper + piper. Opens 11434/10200/10300/10400.bazel-build.nixruns NativeLink (from thenativelinkflake input, pinned to v1.8.3 because its config format changes) as a remote cache and executor for slackerbot containers. Used on hl-fws1 and hl-fwdesk. Goal: as little as possible runs in a container. Measured on slacker-bot (1 CPU container, 2026-11-01): local 458 s, remote with no cache 65 s, remote with warm cache 17 s.- NativeLink uses
DynamicUser, whose state folder systemd mountsnoexec;ExecPathsre-allows it, or every remote action fails with "Permission denied". - Containers get their own
bazel(agentBazel), with no plain bazel/bazelisk on PATH. At bootbazel-remote-rcwrites the gateway into/run/bazel-remote.bazelrc, which/etc/bazel.bazelrctry-imports. Only remote actions are allowed (--spawn_strategy=remote, no local fallback), so a missing file or unreachable host fails the build instead of building in the container. The wrapper refuses command-line flags that change this (--spawn_strategy,--strategy*,--remote_executor,--bazelrc, ...). Flags in a repo's own.bazelrcare not checked. - Final outputs are downloaded into
bazel-bin(--remote_download_toplevel).rbuildis justbazel build. A/handoff→/outputspath that hardlinked outputs from the cache was tried and removed: outputs are a few MB over a local link, so it saved nothing (download ~20 s vs handoff ~344 s). Most of that was a lock bug: Bazel's server inherits open file handles, so a lock held on an fd in the wrapper stays held until the server exits. Close such fds for Bazel (9>&-). - Still in the container: goose, the Bazel server (512M heap, exits after 5 idle minutes) and repository rules, which Bazel can't run remotely.
- Downloads share one cache per host (
/var/lib/bazel-repo-cache→/repo-cache, sha256-checked, 30-day cleanup). Fetched repo folders stay per container (--repo_contents_cache), since their contents are not checked. - Containers use private user IDs, so writable binds (
/repo-cache) useidmap; without it the folder shows asnobodyand the container can't write. - Global goose hints (
/root/.config/goose/.goosehints→/etc/goose/goosehints): a local build means something is wrong (stop and report); build Go through Bazel. - Exposes
agentSystem(slackerbot's agent + bazel, go, bazelrc, rbuild) andbinds(slackerbotro_binds/rw_binds). Slackerbot creates missing bind source folders, or containers fail to start.bazelis a bazelisk wrapper on the host and in containers: a repo's.bazelversionwins, otherwise 8.7.0 (USE_BAZEL_FALLBACK_VERSION). Bazel 8.7.0 is fetched once per node by Nix (official release, sha256-checked) and the wrapper enters it into each user's bazelisk cache as a link, so containers share that one read-only copy and never download it. Other versions download into each container's own/root/.cache/bazelisk. Upstream Bazel only runs on NixOS throughprograms.nix-ld, enabled on the host and in containers. BumpbazelVersionandbazelSha256together.
- NativeLink uses
colibri.nixrunscoli serve(colibrì's OpenAI-compatible API) as a native systemd service.packagepicks the backend (pkgs.colibricpu /pkgs.colibri-rocm);environmentFilesuppliesCOLI_API_KEYvia agenix instead of a CLI flag.
Option Namespace
skyg.nixos.server.services.ai.enable
skyg.nixos.server.services.audiobookshelf.{enable,package,configDir,dataDir}
skyg.nixos.server.services.bazel-build.{enable,cpuCount,cacheGB,port,agentSystem,binds}
skyg.nixos.server.services.jellyfin.enable
skyg.nixos.server.services.signal-cli.{enable,host,port,configDir,account,environmentFile,openFirewall}
skyg.nixos.server.services.colibri.{enable,package,model,host,port,environmentFile,openFirewall,...}
skyg.nixos.server.services.comfyui.{enable,mode,port,openFirewall}
Conventions
- One file per service; gate everything behind
enable(defaultfalse). - Expose an
openFirewalloption instead of opening ports unconditionally. - Secrets (tokens, credentials) come from agenix — never hardcode them.
- See
GUIDELINES.mdinmodules/nixos/server/for the full service-authoring pattern.