- Extend pkgs/llama-cpp to support CUDA backend (backend="cuda") for NVIDIA GPUs - Add llama-cpp-cuda variant to modules/core/default.nix overlay - Create hosts/hl-bigbox1/services/llama-cpp.nix with RTX 3060 tuning: * gpuLayers=30 (conserve 12GB VRAM for context) * parallel=2 (handle concurrent requests on smaller VRAM) * Same Qwen3.8-27B + DFlash2 drafter as fwdesk - Wire service into bigbox1 services/default.nix - Disabled by default (weights download manual)
11 lines
143 B
Nix
11 lines
143 B
Nix
{
|
|
imports = [
|
|
./jellyfin.nix
|
|
./immich-ml.nix
|
|
./portainer.nix
|
|
./timers.nix
|
|
./linux-tracker.nix
|
|
./llama-cpp.nix
|
|
];
|
|
}
|