llama-cpp: add CUDA support and bigbox1 RTX 3060 service config
- Extend pkgs/llama-cpp to support CUDA backend (backend="cuda") for NVIDIA GPUs - Add llama-cpp-cuda variant to modules/core/default.nix overlay - Create hosts/hl-bigbox1/services/llama-cpp.nix with RTX 3060 tuning: * gpuLayers=30 (conserve 12GB VRAM for context) * parallel=2 (handle concurrent requests on smaller VRAM) * Same Qwen3.8-27B + DFlash2 drafter as fwdesk - Wire service into bigbox1 services/default.nix - Disabled by default (weights download manual)
This commit is contained in:
1 parent
ce92096042
commit
26856003d4
4 files changed
+60
-7
No files matched your search
@@ -22,9 +22,10 @@
|
||||
# colibrì (JustVugg/colibri) — GLM-5.2/OLMoE local inference, see pkgs/colibri.
|
||||
colibri = final.callPackage ../../pkgs/colibri { };
|
||||
colibri-rocm = final.callPackage ../../pkgs/colibri { backend = "rocm"; };
|
||||
# llama.cpp v0.8.0 for gfx1151 — DFlash2 speculative decoding, see pkgs/llama-cpp.
|
||||
# llama.cpp v0.8.0 with DFlash2 speculative decoding, see pkgs/llama-cpp.
|
||||
llama-cpp-halo = final.callPackage ../../pkgs/llama-cpp { };
|
||||
llama-cpp-halo-cpu = final.callPackage ../../pkgs/llama-cpp { backend = "cpu"; };
|
||||
llama-cpp-cuda = final.callPackage ../../pkgs/llama-cpp { backend = "cuda"; };
|
||||
# Touchscreen gesture bridge for niri, see pkgs/niri-touchscreen-gestures.
|
||||
niri-touchscreen-gestures = final.callPackage ../../pkgs/niri-touchscreen-gestures { };
|
||||
# Buzz Desktop AppImage wrapper, see pkgs/buzz-desktop.
|
||||
|
||||
Reference in new issue
Block a user