diff --git a/pkgs/ABOUTME.md b/pkgs/ABOUTME.md index eba85b7..930e1cf 100644 --- a/pkgs/ABOUTME.md +++ b/pkgs/ABOUTME.md @@ -8,6 +8,7 @@ Packages here are exposed via an overlay defined in `modules/core/default.nix`. | Package | Description | |---|---| | `grok-cli/` | xAI Grok CLI — prebuilt static binary, fetched per-platform. Exposed as `pkgs.grok-cli`. | +| `ds4/` | DwarfStar (`antirez/ds4`) DeepSeek V4 local inference engine, built from source. Backend-parameterized (`backend = "cpu" \| "rocm" \| "cuda"`); tracks `main` and builds per-host. Exposed as `pkgs.ds4` (cpu, default), `pkgs.ds4-rocm` (Strix Halo / gfx1151), `pkgs.ds4-cuda`. Installs the `ds4`, `ds4-server`, `ds4-bench`, `ds4-eval`, `ds4-agent` binaries plus a `ds4-download-model` helper for fetching GGUF weights. Model weights are a runtime concern and not packaged. | ## Adding a New Package diff --git a/pkgs/ds4/default.nix b/pkgs/ds4/default.nix index cc6e5dc..c7e4c11 100644 --- a/pkgs/ds4/default.nix +++ b/pkgs/ds4/default.nix @@ -1,6 +1,9 @@ { lib , stdenv , fetchFromGitHub +, makeWrapper +, curl +, cacert , rocmPackages , cudaPackages # backend selects the build target / toolchain. @@ -120,7 +123,8 @@ stdenv.mkDerivation (finalAttrs: { enableParallelBuilding = true; nativeBuildInputs = - lib.optionals (backend == "rocm") [ rocmPackages.clr ] + [ makeWrapper ] + ++ lib.optionals (backend == "rocm") [ rocmPackages.clr ] ++ lib.optionals (backend == "cuda") [ cudaPackages.cuda_nvcc ]; buildInputs = @@ -142,9 +146,27 @@ stdenv.mkDerivation (finalAttrs: { installPhase = '' runHook preInstall install -Dm755 -t "$out/bin" ds4 ds4-server ds4-bench ds4-eval ds4-agent + + # Wire upstream's GGUF downloader in as `ds4-download-model`. Patch its + # project-root detection so the gguf dir and the `ds4flash.gguf` symlink + # land in a writable location ($DS4_HOME, default: cwd) instead of the + # read-only store dir that `dirname $0` would resolve to here. + install -Dm755 download_model.sh "$out/bin/ds4-download-model" + substituteInPlace "$out/bin/ds4-download-model" \ + --replace-fail 'ROOT=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)' 'ROOT=''${DS4_HOME:-$PWD}' \ + --replace-warn './download_model.sh' 'ds4-download-model' runHook postInstall ''; + # `ds4-download-model` shells out to curl (and optionally the `hf` CLI for the + # huge PRO files); make curl available and point it at a CA bundle if the + # environment doesn't already set one. + postFixup = '' + wrapProgram "$out/bin/ds4-download-model" \ + --prefix PATH : ${lib.makeBinPath [ curl ]} \ + --set-default SSL_CERT_FILE ${cacert}/etc/ssl/certs/ca-bundle.crt + ''; + # Only the CPU variant is guaranteed to run without a GPU/device present, so # restrict the smoke check to it. GPU variants are validated on real hardware. doInstallCheck = backend == "cpu"; diff --git a/pkgs/ds4/plan.md b/pkgs/ds4/plan.md deleted file mode 100644 index da0bbc0..0000000 --- a/pkgs/ds4/plan.md +++ /dev/null @@ -1,326 +0,0 @@ -# Plan: Package `antirez/ds4` (DwarfStar) as a Nix package - -This document is a self-contained implementation plan for adding the -[`antirez/ds4`](https://github.com/antirez/ds4) inference engine ("DwarfStar") -to this repo's custom package set under `pkgs/ds4`. It is written so another -agent (or future me) can pick it up cold. - -## Goal - -A single, **backend-parameterized** derivation at `pkgs/ds4/default.nix` that -builds ds4 from source for a chosen GPU backend, wired into the repo's overlay. -Backends in scope: - -- **ROCm** (`make strix-halo`, AMD Strix Halo / `gfx1151`). -- **CUDA** (`make cuda-generic`, or `make cuda CUDA_ARCH=sm_NN` for an explicit - arch). -- **CPU** (`make cpu`) as the safe default and cheapest smoke build. - -(Which backend is installed where is intentionally deferred to the final -**Host wiring & deployment** section.) - -Decisions already made by the repo owner: - -1. **Parameterize** the package by target/backend (one `default.nix`, selected - via a `backend` argument). -2. **Build per host** — i.e. compile natively with `-march=native`/`-mcpu=native` - (the upstream Makefile default). We are *not* pinning a portable baseline - `-march`. This means the resulting store paths are CPU-specific; that's - acceptable here. (Implication: don't expect these variants to be cache-shared - across differing CPUs, and the CUDA/ROCm variants generally must be built on a - machine with the toolchain anyway.) -3. **Track `main`** — pin the current `main` HEAD commit (below). No upstream - releases/tags exist. - -## Upstream facts (verified from the repo on 2026-06-30) - -- **License:** MIT. -- **Pinned commit (`main` HEAD at planning time):** - `80ebbc396aee40eedc1d829222f3362d10fa4c6c` -- **Build system:** hand-written `Makefile`, no `install` target. Languages: - C (C99), CUDA, Objective-C (Metal, macOS only), Metal shaders, some C++. -- **Binaries produced (all backends):** `ds4`, `ds4-server`, `ds4-bench`, - `ds4-eval`, `ds4-agent`. -- **Metal is macOS-only** and irrelevant to our NixOS Linux hosts. - -### Makefile behavior that matters for packaging - -From the upstream `Makefile` (Linux branch, i.e. `uname -s` != `Darwin`): - -```make -CC ?= cc -NATIVE_CPU_FLAG ?= -march=native # Linux; -mcpu=native on Darwin -CFLAGS ?= -O3 -ffast-math -g $(NATIVE_CPU_FLAG) -Wall -Wextra -std=c99 -CFLAGS += -D_GNU_SOURCE -fno-finite-math-only -LDLIBS ?= -lm -pthread - -# CUDA -CUDA_HOME ?= /usr/local/cuda -NVCC ?= $(CUDA_HOME)/bin/nvcc -CUDA_ARCH ?= # empty unless overridden -NVCCFLAGS ?= -O3 -g -lineinfo --use_fast_math $(NVCC_ARCH_FLAGS) \ - -Xcompiler $(NATIVE_CPU_FLAG) -Xcompiler -pthread -CUDA_LDLIBS ?= -lm -Xcompiler -pthread \ - -L$(CUDA_HOME)/targets/sbsa-linux/lib -L$(CUDA_HOME)/lib64 \ - -lcudart -lcublas - -# ROCm -HIPCC ?= $(shell command -v hipcc 2>/dev/null || echo /opt/rocm/bin/hipcc) -ROCM_ARCH ?= gfx1151 -ROCM_CFLAGS ?= -O3 -ffast-math -g -fno-finite-math-only -pthread \ - -D__HIP_PLATFORM_AMD__ -Wno-unused-command-line-argument \ - --offload-arch=$(ROCM_ARCH) -ROCM_LDLIBS ?= -lm -pthread -lhipblas -lhipblaslt -``` - -Relevant phony targets (Linux): - -- `make cuda-generic` → `$(MAKE) -B ds4 ds4-server ds4-bench ds4-eval ds4-agent CUDA_ARCH=native` -- `make cuda-spark` → same but `CUDA_ARCH=` (empty; DGX Spark/GB10) -- `make cuda CUDA_ARCH=sm_NN` → explicit arch (errors if `CUDA_ARCH` empty) -- `make strix-halo` (alias `make rocm`) → rebuilds the 5 binaries with - `CORE_OBJS="ds4.o ds4_distributed.o ds4_ssd.o ds4_rocm.o"`, - `CFLAGS="… -DDS4_ROCM_BUILD"`, `DS4_LINK="$(HIPCC) $(ROCM_CFLAGS)"`, - `DS4_LINK_LIBS="$(ROCM_LDLIBS)"`. -- `make cpu` → CPU-only reference build (`-DDS4_NO_GPU`), no GPU libs. Useful - as a cheap CI/portable smoke build. -- `make test` → builds `ds4_test`, `ds4_agent_test`, `ds4-eval`, a dot-product - test, and runs them. The CUDA path links tests with `nvcc`. - -Key gotchas: - -1. **No `install` target** → we must write our own `installPhase` copying the 5 - binaries into `$out/bin`. -2. **`-march=native`** is on by default. Per decision #2 we keep it (build per - host). For the optional `cpu` smoke variant built on the remote builder, the - resulting binary is CPU-specific to the builder — fine for a smoke test, but - do **not** distribute a `cpu` variant to heterogeneous machines without - overriding `NATIVE_CPU_FLAG`. -3. **CUDA library paths** are hardcoded to `/usr/local/cuda/...` and an - `sbsa-linux` (aarch64) target dir. Under Nix we must point `CUDA_HOME`/`NVCC` - at the nixpkgs CUDA toolkit and likely override `CUDA_LDLIBS` so the `-L` - paths resolve inside the store. Expect to set `NVCC`, `CUDA_HOME`, and a - corrected `CUDA_LDLIBS` via `makeFlags`. -4. **ROCm** finds `hipcc` via `command -v hipcc`, so having `rocmPackages.clr` - (provides `hipcc`) on `PATH` (`nativeBuildInputs`) is enough. `ROCM_ARCH` - defaults to `gfx1151` (exactly the Strix Halo target). The backend uses - **rocWMMA**; on Ubuntu upstream notes the distro packages miss - `rocwmma/internal/` headers (see `STRIXHALO.md`). Under Nix, - `rocmPackages.rocwmma` should provide the complete header tree — verify - during implementation (ROCm phase). -5. **`-ffast-math` / `-fno-finite-math-only`** are upstream defaults; leave - them. Standard `fortify`/`strip` Nix hardening should be fine but watch for - warnings; can disable hardening flags if the build complains. -6. Runtime: on Linux the binaries are self-contained (Metal `.metal` runtime - files are macOS-only). GGUF model weights and `download_model.sh` are a - **runtime** concern, NOT packaged. The agent/server accept `--chdir` and - `-m ` for model selection. - -## Repo conventions (how packages are added here) - -See `pkgs/ABOUTME.md`. The pattern is: - -1. Create `pkgs//default.nix` (standard derivation; `callPackage`-style - function taking its inputs as arguments). -2. Register in the overlay in `modules/core/default.nix`: - ```nix - nixpkgs.overlays = [ - (final: _prev: { - grok-cli = final.callPackage ../../pkgs/grok-cli { }; - # add ds4 entries here - }) - # ... - ]; - ``` -3. Reference as `pkgs.` anywhere in the flake / host modules. -4. Optionally expose as a flake output under `packages` in `flake.nix` - (see the existing `grok-cli = lib.pkgs.${system}.grok-cli;` line). - -`pkgs/grok-cli/default.nix` is the existing reference example (but note it is a -prebuilt binary using `stdenvNoCC` + `fetchurl`; **ds4 is build-from-source and -must use `stdenv.mkDerivation` + `fetchFromGitHub`**). - ---- - -## Implementation phases - -### Phase 1 — Source pinning -- [ ] Write the `fetchFromGitHub` `src` with: - - `owner = "antirez"; repo = "ds4";` - - `rev = "80ebbc396aee40eedc1d829222f3362d10fa4c6c";` (current `main` HEAD; - bump as desired since we track `main`). - - `hash` — start with `lib.fakeHash`, run a build, copy the real SRI hash - from the error. (Equivalent to the `nix-prefetch`/`fakeHash` flow noted in - `pkgs/ABOUTME.md`.) - -### Phase 2 — Core derivation skeleton (`pkgs/ds4/default.nix`) -Parameterized function. Sketch (to be refined during implementation): - -```nix -{ lib -, stdenv -, fetchFromGitHub -, rocmPackages -, cudaPackages -# backend selects the build target / toolchain -, backend ? "cpu" # one of "cpu" | "rocm" | "cuda" -, cudaArch ? null # e.g. "sm_89"; null => cuda-generic (CUDA_ARCH=native) -, rocmArch ? "gfx1151" -}: - -assert lib.elem backend [ "cpu" "rocm" "cuda" ]; - -stdenv.mkDerivation (finalAttrs: { - pname = "ds4"; - version = "unstable-2026-06-17"; # date of pinned commit - src = fetchFromGitHub { /* Phase 1 */ }; - - enableParallelBuilding = true; - - # nativeBuildInputs / buildInputs / makeFlags / buildPhase vary by backend - # (ROCm / CUDA / CPU phases). - - installPhase = '' - runHook preInstall - install -Dm755 -t "$out/bin" ds4 ds4-server ds4-bench ds4-eval ds4-agent - runHook postInstall - ''; - - meta = { - description = "DeepSeek V4 Flash/PRO local inference engine (DwarfStar)"; - homepage = "https://github.com/antirez/ds4"; - license = lib.licenses.mit; - sourceProvenance = [ lib.sourceTypes.fromSource ]; - platforms = lib.platforms.linux; # Metal/macOS out of scope - mainProgram = "ds4"; - }; -}) -``` - -- [ ] Implement the skeleton with a `let buildTarget = { cpu = "cpu"; rocm = - "strix-halo"; cuda = if cudaArch == null then "cuda-generic" else "cuda"; }` - style mapping, and a per-backend `makeFlags`/`buildFlags`. -- [ ] Keep `-march=native` (do **not** override `NATIVE_CPU_FLAG`) per decision #2. - -### Phase 3 — ROCm backend -- [ ] `nativeBuildInputs`: `rocmPackages.clr` (provides `hipcc`). -- [ ] `buildInputs`: `rocmPackages` libs needed by `ds4_rocm.cu` and link flags - (`-lhipblas -lhipblaslt`): at least `clr`, `hipblas`, `hipblaslt`, - `rocblas`, `rocwmma`, plus header-only `hipcub`/`rocprim` as required. -- [ ] Build: `make strix-halo` (or pass the equivalent `makeFlags`). `ROCM_ARCH` - defaults to `gfx1151`; allow override via `rocmArch` arg. -- [ ] **Verify rocWMMA headers**: ensure `rocwmma/internal/*` resolves from the - nixpkgs `rocwmma` package (upstream `STRIXHALO.md` warns the Ubuntu pkgs - are incomplete). If missing, add the include path or the right package. -- [ ] May need `ROCM_PATH`/include/lib flags so `hipcc` finds the store-located - ROCm libs; adjust `ROCM_CFLAGS`/`ROCM_LDLIBS` via `makeFlags` if needed. - -### Phase 4 — CUDA backend -- [ ] `nativeBuildInputs`: `cudaPackages.cuda_nvcc`. -- [ ] `buildInputs`: `cudaPackages.cuda_cudart`, `cudaPackages.libcublas`. -- [ ] Override the hardcoded CUDA paths via `makeFlags`: - - `CUDA_HOME = ...` / `NVCC = "${cudaPackages.cuda_nvcc}/bin/nvcc"`, - - a corrected `CUDA_LDLIBS` that points `-L` at the nixpkgs cudart/cublas - `lib` dirs instead of `/usr/local/cuda/...` and `sbsa-linux` (that path is - aarch64; an x86_64 target should use `lib64`/store libs). -- [ ] Build target: `cuda-generic` (`CUDA_ARCH=native`) by default, or `cuda` - with an explicit `cudaArch` (preferred for reproducibility / when the - build host differs from the GPU host). -- [ ] CUDA is unfree → the package's closure is unfree. Keep `meta.license = - lib.licenses.mit` for the source itself, but be aware `allowUnfree` is - required to build/realize this variant. - -### Phase 5 — CPU backend (default) -- [ ] `make cpu`; no GPU inputs. Cheapest build, good for a portable smoke test - and as the safe default `backend = "cpu"`. -- [ ] Add `doInstallCheck` running `$out/bin/ds4 --help` (only meaningful for - the CPU variant; GPU variants may require a device at runtime, so guard - or skip the check for them). - -### Phase 6 — Overlay wiring (`modules/core/default.nix`) -Add to the existing `nixpkgs.overlays` first entry: -```nix -(final: _prev: { - grok-cli = final.callPackage ../../pkgs/grok-cli { }; - ds4 = final.callPackage ../../pkgs/ds4 { }; # cpu default - ds4-rocm = final.callPackage ../../pkgs/ds4 { backend = "rocm"; }; - ds4-cuda = final.callPackage ../../pkgs/ds4 { backend = "cuda"; }; -}) -``` -- [ ] Confirm `rocmPackages` / `cudaPackages` resolve via `callPackage` (they - are top-level attrs in nixpkgs, so they will). - -### Phase 7 — Flake output (optional) -- [ ] In `flake.nix` `packages` output, alongside `grok-cli`, optionally expose: - `ds4 = lib.pkgs.${system}.ds4;` (CPU variant — buildable on any system). - The GPU variants are awkward as generic flake outputs (need toolchains), - so leave them overlay-only unless wanted. - -### Phase 8 — Validation -Order by cheapness / portability: -- [ ] `nix build .#ds4` (or via a throwaway `nix-build -E`) — CPU variant, - builds on the default builder. Resolve the `src` hash here (Phase 1). -- [ ] `nix build` the ROCm variant (build on a machine with the ROCm toolchain). - Smoke: `ds4 --help`. -- [ ] `nix build` the CUDA variant (build on a machine with the CUDA toolchain). - Smoke: `ds4 --help`. -- [ ] `nixpkgs-fmt` the new/edited Nix files (repo uses it as a pre-commit - hook and devshell formatter). -- [ ] Update `pkgs/ABOUTME.md` contents table with the new `ds4` entries. - -### Phase 9 — Docs / cleanup -- [ ] Add a short header comment in `pkgs/ds4/default.nix` (like `grok-cli`) - explaining backends, the `-march=native` "build per host" choice, the - `main`-tracking pin, and how to bump (`rev` + refresh `hash`). -- [ ] Delete or keep this `plan.md` (suggest keeping until implementation lands, - then optionally remove). - -## Risks / unknowns to resolve during implementation - -- **CUDA path overrides**: the Makefile's `CUDA_LDLIBS` hardcodes - `targets/sbsa-linux/lib` (aarch64) + `/usr/local/cuda`. On x86_64 nixpkgs - this must be replaced. Highest-risk part of the CUDA backend. -- **rocWMMA internal headers** under nixpkgs (ROCm phase) — verify availability. -- **Hardening flags**: `-ffast-math` + Nix default hardening may produce - warnings; if the build breaks, selectively disable (`hardeningDisable`). -- **Build-per-host vs. binary cache**: with `-march=native`, store paths are - CPU-specific; ensure builds happen on (or for) the right host. - ---- - -## Host wiring & deployment (do this last) - -All host-specific assignment and system enablement lives here, after the -package itself builds and is wired into the overlay. - -Hosts are defined in `flake.nix` under `nixosConfigurations`. System-level -packages for a host are typically added in the host's module (or a shared -module it imports); locate where each host declares `environment.systemPackages` -/ module package lists and add the appropriate `pkgs.ds4-*` there. - -### `hl-fwdesk` — ROCm -- Module: `hosts/hl-fwdesk.nix` (uses - `framework-desktop-amd-ai-max-300-series`); the box is AMD Ryzen AI Max 300 - (Strix Halo, `gfx1151`) — exactly the ROCm/`make strix-halo` target. -- [ ] Add `pkgs.ds4-rocm` to the host's package list. -- [ ] Confirm ROCm is enabled on the host (drivers, `hardware.graphics` / - ROCm runtime) before expecting the GPU path to work. -- [ ] System enablement required to *run* it (not part of the package), per - `STRIXHALO.md`: - - `/dev/kfd` + DRM render-node access (user in `render`/`video` groups). - - GTT memory kernel params so the full 128 GB is GPU-visible: - `amd_iommu=off amdgpu.gttsize=126976 ttm.pages_limit=32505856 - ttm.page_pool_size=32505856` (in `boot.kernelParams`). - -### `hl-bigbox1` — CUDA -- Module: `hosts/hl-bigbox1.nix`. Host has an NVIDIA GPU. -- [ ] Add `pkgs.ds4-cuda` to the host's package list. -- [ ] Confirm the exact NVIDIA GPU/arch to decide between `cuda-generic` - (`CUDA_ARCH=native`) and a pinned `CUDA_ARCH=sm_NN` - (e.g. `sm_86`/`sm_89`/`sm_90`). Pinning is preferred when the build host - differs from the GPU host. -- [ ] Ensure `nixpkgs.config.allowUnfree` (and CUDA driver/`hardware.graphics` - support) is set for this host — the CUDA closure is unfree. -- [ ] The configured remote builder `root@bigbox1.lab.internal` is x86_64 and - may double as the native CUDA build host (convenient, avoids - cross-CPU `-march=native` mismatches). diff --git a/pkgs/ds4/user-guide.md b/pkgs/ds4/user-guide.md new file mode 100644 index 0000000..c7d2a1d --- /dev/null +++ b/pkgs/ds4/user-guide.md @@ -0,0 +1,75 @@ +# ds4 (DwarfStar) — user guide + +A from-source build of [`antirez/ds4`](https://github.com/antirez/ds4), a local +inference engine for **DeepSeek V4 Flash/PRO**. This is a short operational +guide; see the [upstream README](https://github.com/antirez/ds4) for the full +documentation (server API, agent, distributed inference, tuning, etc.). + +## What's installed + +The package builds for a chosen GPU backend (`cpu`, `rocm`, or `cuda`) and +installs these binaries: + +| Binary | What it does | +|---|---| +| `ds4` | Interactive CLI chat / one-shot `-p "..."` | +| `ds4-server` | OpenAI/Anthropic/Responses-compatible HTTP API | +| `ds4-agent` | Native terminal coding agent (alpha) | +| `ds4-bench` | Throughput benchmarking | +| `ds4-eval` | Capability/regression eval harness | +| `ds4-download-model` | Helper to fetch GGUF weights (see below) | + +There is **no graphical UI**. `ds4-server` exposes a URL you can point an +OpenAI-compatible client at. + +## 1. Get the weights + +Weights are **not** packaged. They live at the project's Hugging Face repo +[`antirez/deepseek-v4-gguf`](https://huggingface.co/antirez/deepseek-v4-gguf) +(public, MIT, no token needed) and only these custom GGUFs work — not arbitrary +DeepSeek GGUFs. + +`ds4-download-model` writes into `./gguf` and links `./ds4flash.gguf` in the +current directory (override the root with `DS4_HOME`, or the gguf dir with +`DS4_GGUF_DIR`): + +```sh +ds4-download-model q2-imatrix # ~81 GB, the pick for 96/128 GB machines +ds4-download-model --help # all targets (q4-imatrix, pro-*, mtp, ...) +``` + +Pick by machine memory: `q2-imatrix` (128 GB), `q4-imatrix` (≥256 GB), +`pro-*` (512 GB / distributed). The small/Flash quants download via `curl`; the +huge **PRO** files need the `hf` CLI (not bundled — `nix shell nixpkgs#huggingface-hub`). + +## 2. Run it + +```sh +# interactive chat (defaults to ./ds4flash.gguf in the current dir) +ds4 + +# explicit model path +ds4 -m /path/to/gguf/DeepSeek-V4-Flash-IQ2XXS-...-imatrix.gguf + +# HTTP server on the LAN +ds4-server -m ./ds4flash.gguf \ + --ctx 100000 --kv-disk-dir ./kv --kv-disk-space-mb 8192 --host 0.0.0.0 +``` + +Then point a client at `http://:8000/v1` (endpoints: `/v1/chat/completions`, +`/v1/messages`, `/v1/responses`, ...). Use `--cors` for browser clients. + +## Backend notes + +- **rocm** targets a specific GPU arch (default `gfx1151`, AMD Strix Halo). At + runtime the user needs `/dev/kfd` + render-node access, and large models need + the GTT memory kernel params from upstream's `STRIXHALO.md`. On Strix Halo, + export `HSA_ENABLE_SDMA=0` (SDMA is buggy on its unified memory). +- **cuda** is unfree (CUDA closure) and needs an NVIDIA driver at runtime. +- **cpu** is a reference/debug path only — not for production inference. + +## Maintenance + +This tracks upstream `main` (no releases). To bump, set `rev` in `default.nix` +and refresh `hash`. Binaries are built per-host (`-march=native`), so build each +GPU variant on its target host. See `default.nix`'s header comment for details.