llama-cpp: add CUDA support and bigbox1 RTX 3060 service config

- Extend pkgs/llama-cpp to support CUDA backend (backend="cuda") for NVIDIA GPUs
- Add llama-cpp-cuda variant to modules/core/default.nix overlay
- Create hosts/hl-bigbox1/services/llama-cpp.nix with RTX 3060 tuning:
  * gpuLayers=30 (conserve 12GB VRAM for context)
  * parallel=2 (handle concurrent requests on smaller VRAM)
  * Same Qwen3.8-27B + DFlash2 drafter as fwdesk
- Wire service into bigbox1 services/default.nix
- Disabled by default (weights download manual)
This commit is contained in:
hermenix committed 2026-11-11 16:02:41 +00:00
1 parent ce92096042
commit 26856003d4
4 files changed
+60 -7

No files matched your search

+2 -1
View File
@@ -22,9 +22,10 @@
# colibrì (JustVugg/colibri) — GLM-5.2/OLMoE local inference, see pkgs/colibri.
colibri = final.callPackage ../../pkgs/colibri { };
colibri-rocm = final.callPackage ../../pkgs/colibri { backend = "rocm"; };
# llama.cpp v0.8.0 for gfx1151 — DFlash2 speculative decoding, see pkgs/llama-cpp.
# llama.cpp v0.8.0 with DFlash2 speculative decoding, see pkgs/llama-cpp.
llama-cpp-halo = final.callPackage ../../pkgs/llama-cpp { };
llama-cpp-halo-cpu = final.callPackage ../../pkgs/llama-cpp { backend = "cpu"; };
llama-cpp-cuda = final.callPackage ../../pkgs/llama-cpp { backend = "cuda"; };
# Touchscreen gesture bridge for niri, see pkgs/niri-touchscreen-gestures.
niri-touchscreen-gestures = final.callPackage ../../pkgs/niri-touchscreen-gestures { };
# Buzz Desktop AppImage wrapper, see pkgs/buzz-desktop.