ds4 works
This commit is contained in:
1 parent
e68b8f8fb5
commit
7d4c0457ba
4 files changed
+99
-327
No files matched your search
@@ -8,6 +8,7 @@ Packages here are exposed via an overlay defined in `modules/core/default.nix`.
|
||||
| Package | Description |
|
||||
|---|---|
|
||||
| `grok-cli/` | xAI Grok CLI — prebuilt static binary, fetched per-platform. Exposed as `pkgs.grok-cli`. |
|
||||
| `ds4/` | DwarfStar (`antirez/ds4`) DeepSeek V4 local inference engine, built from source. Backend-parameterized (`backend = "cpu" \| "rocm" \| "cuda"`); tracks `main` and builds per-host. Exposed as `pkgs.ds4` (cpu, default), `pkgs.ds4-rocm` (Strix Halo / gfx1151), `pkgs.ds4-cuda`. Installs the `ds4`, `ds4-server`, `ds4-bench`, `ds4-eval`, `ds4-agent` binaries plus a `ds4-download-model` helper for fetching GGUF weights. Model weights are a runtime concern and not packaged. |
|
||||
|
||||
## Adding a New Package
|
||||
|
||||
|
||||
+23
-1
@@ -1,6 +1,9 @@
|
||||
{ lib
|
||||
, stdenv
|
||||
, fetchFromGitHub
|
||||
, makeWrapper
|
||||
, curl
|
||||
, cacert
|
||||
, rocmPackages
|
||||
, cudaPackages
|
||||
# backend selects the build target / toolchain.
|
||||
@@ -120,7 +123,8 @@ stdenv.mkDerivation (finalAttrs: {
|
||||
enableParallelBuilding = true;
|
||||
|
||||
nativeBuildInputs =
|
||||
lib.optionals (backend == "rocm") [ rocmPackages.clr ]
|
||||
[ makeWrapper ]
|
||||
++ lib.optionals (backend == "rocm") [ rocmPackages.clr ]
|
||||
++ lib.optionals (backend == "cuda") [ cudaPackages.cuda_nvcc ];
|
||||
|
||||
buildInputs =
|
||||
@@ -142,9 +146,27 @@ stdenv.mkDerivation (finalAttrs: {
|
||||
installPhase = ''
|
||||
runHook preInstall
|
||||
install -Dm755 -t "$out/bin" ds4 ds4-server ds4-bench ds4-eval ds4-agent
|
||||
|
||||
# Wire upstream's GGUF downloader in as `ds4-download-model`. Patch its
|
||||
# project-root detection so the gguf dir and the `ds4flash.gguf` symlink
|
||||
# land in a writable location ($DS4_HOME, default: cwd) instead of the
|
||||
# read-only store dir that `dirname $0` would resolve to here.
|
||||
install -Dm755 download_model.sh "$out/bin/ds4-download-model"
|
||||
substituteInPlace "$out/bin/ds4-download-model" \
|
||||
--replace-fail 'ROOT=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)' 'ROOT=''${DS4_HOME:-$PWD}' \
|
||||
--replace-warn './download_model.sh' 'ds4-download-model'
|
||||
runHook postInstall
|
||||
'';
|
||||
|
||||
# `ds4-download-model` shells out to curl (and optionally the `hf` CLI for the
|
||||
# huge PRO files); make curl available and point it at a CA bundle if the
|
||||
# environment doesn't already set one.
|
||||
postFixup = ''
|
||||
wrapProgram "$out/bin/ds4-download-model" \
|
||||
--prefix PATH : ${lib.makeBinPath [ curl ]} \
|
||||
--set-default SSL_CERT_FILE ${cacert}/etc/ssl/certs/ca-bundle.crt
|
||||
'';
|
||||
|
||||
# Only the CPU variant is guaranteed to run without a GPU/device present, so
|
||||
# restrict the smoke check to it. GPU variants are validated on real hardware.
|
||||
doInstallCheck = backend == "cpu";
|
||||
|
||||
@@ -1,326 +0,0 @@
|
||||
# Plan: Package `antirez/ds4` (DwarfStar) as a Nix package
|
||||
|
||||
This document is a self-contained implementation plan for adding the
|
||||
[`antirez/ds4`](https://github.com/antirez/ds4) inference engine ("DwarfStar")
|
||||
to this repo's custom package set under `pkgs/ds4`. It is written so another
|
||||
agent (or future me) can pick it up cold.
|
||||
|
||||
## Goal
|
||||
|
||||
A single, **backend-parameterized** derivation at `pkgs/ds4/default.nix` that
|
||||
builds ds4 from source for a chosen GPU backend, wired into the repo's overlay.
|
||||
Backends in scope:
|
||||
|
||||
- **ROCm** (`make strix-halo`, AMD Strix Halo / `gfx1151`).
|
||||
- **CUDA** (`make cuda-generic`, or `make cuda CUDA_ARCH=sm_NN` for an explicit
|
||||
arch).
|
||||
- **CPU** (`make cpu`) as the safe default and cheapest smoke build.
|
||||
|
||||
(Which backend is installed where is intentionally deferred to the final
|
||||
**Host wiring & deployment** section.)
|
||||
|
||||
Decisions already made by the repo owner:
|
||||
|
||||
1. **Parameterize** the package by target/backend (one `default.nix`, selected
|
||||
via a `backend` argument).
|
||||
2. **Build per host** — i.e. compile natively with `-march=native`/`-mcpu=native`
|
||||
(the upstream Makefile default). We are *not* pinning a portable baseline
|
||||
`-march`. This means the resulting store paths are CPU-specific; that's
|
||||
acceptable here. (Implication: don't expect these variants to be cache-shared
|
||||
across differing CPUs, and the CUDA/ROCm variants generally must be built on a
|
||||
machine with the toolchain anyway.)
|
||||
3. **Track `main`** — pin the current `main` HEAD commit (below). No upstream
|
||||
releases/tags exist.
|
||||
|
||||
## Upstream facts (verified from the repo on 2026-06-30)
|
||||
|
||||
- **License:** MIT.
|
||||
- **Pinned commit (`main` HEAD at planning time):**
|
||||
`80ebbc396aee40eedc1d829222f3362d10fa4c6c`
|
||||
- **Build system:** hand-written `Makefile`, no `install` target. Languages:
|
||||
C (C99), CUDA, Objective-C (Metal, macOS only), Metal shaders, some C++.
|
||||
- **Binaries produced (all backends):** `ds4`, `ds4-server`, `ds4-bench`,
|
||||
`ds4-eval`, `ds4-agent`.
|
||||
- **Metal is macOS-only** and irrelevant to our NixOS Linux hosts.
|
||||
|
||||
### Makefile behavior that matters for packaging
|
||||
|
||||
From the upstream `Makefile` (Linux branch, i.e. `uname -s` != `Darwin`):
|
||||
|
||||
```make
|
||||
CC ?= cc
|
||||
NATIVE_CPU_FLAG ?= -march=native # Linux; -mcpu=native on Darwin
|
||||
CFLAGS ?= -O3 -ffast-math -g $(NATIVE_CPU_FLAG) -Wall -Wextra -std=c99
|
||||
CFLAGS += -D_GNU_SOURCE -fno-finite-math-only
|
||||
LDLIBS ?= -lm -pthread
|
||||
|
||||
# CUDA
|
||||
CUDA_HOME ?= /usr/local/cuda
|
||||
NVCC ?= $(CUDA_HOME)/bin/nvcc
|
||||
CUDA_ARCH ?= # empty unless overridden
|
||||
NVCCFLAGS ?= -O3 -g -lineinfo --use_fast_math $(NVCC_ARCH_FLAGS) \
|
||||
-Xcompiler $(NATIVE_CPU_FLAG) -Xcompiler -pthread
|
||||
CUDA_LDLIBS ?= -lm -Xcompiler -pthread \
|
||||
-L$(CUDA_HOME)/targets/sbsa-linux/lib -L$(CUDA_HOME)/lib64 \
|
||||
-lcudart -lcublas
|
||||
|
||||
# ROCm
|
||||
HIPCC ?= $(shell command -v hipcc 2>/dev/null || echo /opt/rocm/bin/hipcc)
|
||||
ROCM_ARCH ?= gfx1151
|
||||
ROCM_CFLAGS ?= -O3 -ffast-math -g -fno-finite-math-only -pthread \
|
||||
-D__HIP_PLATFORM_AMD__ -Wno-unused-command-line-argument \
|
||||
--offload-arch=$(ROCM_ARCH)
|
||||
ROCM_LDLIBS ?= -lm -pthread -lhipblas -lhipblaslt
|
||||
```
|
||||
|
||||
Relevant phony targets (Linux):
|
||||
|
||||
- `make cuda-generic` → `$(MAKE) -B ds4 ds4-server ds4-bench ds4-eval ds4-agent CUDA_ARCH=native`
|
||||
- `make cuda-spark` → same but `CUDA_ARCH=` (empty; DGX Spark/GB10)
|
||||
- `make cuda CUDA_ARCH=sm_NN` → explicit arch (errors if `CUDA_ARCH` empty)
|
||||
- `make strix-halo` (alias `make rocm`) → rebuilds the 5 binaries with
|
||||
`CORE_OBJS="ds4.o ds4_distributed.o ds4_ssd.o ds4_rocm.o"`,
|
||||
`CFLAGS="… -DDS4_ROCM_BUILD"`, `DS4_LINK="$(HIPCC) $(ROCM_CFLAGS)"`,
|
||||
`DS4_LINK_LIBS="$(ROCM_LDLIBS)"`.
|
||||
- `make cpu` → CPU-only reference build (`-DDS4_NO_GPU`), no GPU libs. Useful
|
||||
as a cheap CI/portable smoke build.
|
||||
- `make test` → builds `ds4_test`, `ds4_agent_test`, `ds4-eval`, a dot-product
|
||||
test, and runs them. The CUDA path links tests with `nvcc`.
|
||||
|
||||
Key gotchas:
|
||||
|
||||
1. **No `install` target** → we must write our own `installPhase` copying the 5
|
||||
binaries into `$out/bin`.
|
||||
2. **`-march=native`** is on by default. Per decision #2 we keep it (build per
|
||||
host). For the optional `cpu` smoke variant built on the remote builder, the
|
||||
resulting binary is CPU-specific to the builder — fine for a smoke test, but
|
||||
do **not** distribute a `cpu` variant to heterogeneous machines without
|
||||
overriding `NATIVE_CPU_FLAG`.
|
||||
3. **CUDA library paths** are hardcoded to `/usr/local/cuda/...` and an
|
||||
`sbsa-linux` (aarch64) target dir. Under Nix we must point `CUDA_HOME`/`NVCC`
|
||||
at the nixpkgs CUDA toolkit and likely override `CUDA_LDLIBS` so the `-L`
|
||||
paths resolve inside the store. Expect to set `NVCC`, `CUDA_HOME`, and a
|
||||
corrected `CUDA_LDLIBS` via `makeFlags`.
|
||||
4. **ROCm** finds `hipcc` via `command -v hipcc`, so having `rocmPackages.clr`
|
||||
(provides `hipcc`) on `PATH` (`nativeBuildInputs`) is enough. `ROCM_ARCH`
|
||||
defaults to `gfx1151` (exactly the Strix Halo target). The backend uses
|
||||
**rocWMMA**; on Ubuntu upstream notes the distro packages miss
|
||||
`rocwmma/internal/` headers (see `STRIXHALO.md`). Under Nix,
|
||||
`rocmPackages.rocwmma` should provide the complete header tree — verify
|
||||
during implementation (ROCm phase).
|
||||
5. **`-ffast-math` / `-fno-finite-math-only`** are upstream defaults; leave
|
||||
them. Standard `fortify`/`strip` Nix hardening should be fine but watch for
|
||||
warnings; can disable hardening flags if the build complains.
|
||||
6. Runtime: on Linux the binaries are self-contained (Metal `.metal` runtime
|
||||
files are macOS-only). GGUF model weights and `download_model.sh` are a
|
||||
**runtime** concern, NOT packaged. The agent/server accept `--chdir` and
|
||||
`-m <path>` for model selection.
|
||||
|
||||
## Repo conventions (how packages are added here)
|
||||
|
||||
See `pkgs/ABOUTME.md`. The pattern is:
|
||||
|
||||
1. Create `pkgs/<name>/default.nix` (standard derivation; `callPackage`-style
|
||||
function taking its inputs as arguments).
|
||||
2. Register in the overlay in `modules/core/default.nix`:
|
||||
```nix
|
||||
nixpkgs.overlays = [
|
||||
(final: _prev: {
|
||||
grok-cli = final.callPackage ../../pkgs/grok-cli { };
|
||||
# add ds4 entries here
|
||||
})
|
||||
# ...
|
||||
];
|
||||
```
|
||||
3. Reference as `pkgs.<name>` anywhere in the flake / host modules.
|
||||
4. Optionally expose as a flake output under `packages` in `flake.nix`
|
||||
(see the existing `grok-cli = lib.pkgs.${system}.grok-cli;` line).
|
||||
|
||||
`pkgs/grok-cli/default.nix` is the existing reference example (but note it is a
|
||||
prebuilt binary using `stdenvNoCC` + `fetchurl`; **ds4 is build-from-source and
|
||||
must use `stdenv.mkDerivation` + `fetchFromGitHub`**).
|
||||
|
||||
---
|
||||
|
||||
## Implementation phases
|
||||
|
||||
### Phase 1 — Source pinning
|
||||
- [ ] Write the `fetchFromGitHub` `src` with:
|
||||
- `owner = "antirez"; repo = "ds4";`
|
||||
- `rev = "80ebbc396aee40eedc1d829222f3362d10fa4c6c";` (current `main` HEAD;
|
||||
bump as desired since we track `main`).
|
||||
- `hash` — start with `lib.fakeHash`, run a build, copy the real SRI hash
|
||||
from the error. (Equivalent to the `nix-prefetch`/`fakeHash` flow noted in
|
||||
`pkgs/ABOUTME.md`.)
|
||||
|
||||
### Phase 2 — Core derivation skeleton (`pkgs/ds4/default.nix`)
|
||||
Parameterized function. Sketch (to be refined during implementation):
|
||||
|
||||
```nix
|
||||
{ lib
|
||||
, stdenv
|
||||
, fetchFromGitHub
|
||||
, rocmPackages
|
||||
, cudaPackages
|
||||
# backend selects the build target / toolchain
|
||||
, backend ? "cpu" # one of "cpu" | "rocm" | "cuda"
|
||||
, cudaArch ? null # e.g. "sm_89"; null => cuda-generic (CUDA_ARCH=native)
|
||||
, rocmArch ? "gfx1151"
|
||||
}:
|
||||
|
||||
assert lib.elem backend [ "cpu" "rocm" "cuda" ];
|
||||
|
||||
stdenv.mkDerivation (finalAttrs: {
|
||||
pname = "ds4";
|
||||
version = "unstable-2026-06-17"; # date of pinned commit
|
||||
src = fetchFromGitHub { /* Phase 1 */ };
|
||||
|
||||
enableParallelBuilding = true;
|
||||
|
||||
# nativeBuildInputs / buildInputs / makeFlags / buildPhase vary by backend
|
||||
# (ROCm / CUDA / CPU phases).
|
||||
|
||||
installPhase = ''
|
||||
runHook preInstall
|
||||
install -Dm755 -t "$out/bin" ds4 ds4-server ds4-bench ds4-eval ds4-agent
|
||||
runHook postInstall
|
||||
'';
|
||||
|
||||
meta = {
|
||||
description = "DeepSeek V4 Flash/PRO local inference engine (DwarfStar)";
|
||||
homepage = "https://github.com/antirez/ds4";
|
||||
license = lib.licenses.mit;
|
||||
sourceProvenance = [ lib.sourceTypes.fromSource ];
|
||||
platforms = lib.platforms.linux; # Metal/macOS out of scope
|
||||
mainProgram = "ds4";
|
||||
};
|
||||
})
|
||||
```
|
||||
|
||||
- [ ] Implement the skeleton with a `let buildTarget = { cpu = "cpu"; rocm =
|
||||
"strix-halo"; cuda = if cudaArch == null then "cuda-generic" else "cuda"; }`
|
||||
style mapping, and a per-backend `makeFlags`/`buildFlags`.
|
||||
- [ ] Keep `-march=native` (do **not** override `NATIVE_CPU_FLAG`) per decision #2.
|
||||
|
||||
### Phase 3 — ROCm backend
|
||||
- [ ] `nativeBuildInputs`: `rocmPackages.clr` (provides `hipcc`).
|
||||
- [ ] `buildInputs`: `rocmPackages` libs needed by `ds4_rocm.cu` and link flags
|
||||
(`-lhipblas -lhipblaslt`): at least `clr`, `hipblas`, `hipblaslt`,
|
||||
`rocblas`, `rocwmma`, plus header-only `hipcub`/`rocprim` as required.
|
||||
- [ ] Build: `make strix-halo` (or pass the equivalent `makeFlags`). `ROCM_ARCH`
|
||||
defaults to `gfx1151`; allow override via `rocmArch` arg.
|
||||
- [ ] **Verify rocWMMA headers**: ensure `rocwmma/internal/*` resolves from the
|
||||
nixpkgs `rocwmma` package (upstream `STRIXHALO.md` warns the Ubuntu pkgs
|
||||
are incomplete). If missing, add the include path or the right package.
|
||||
- [ ] May need `ROCM_PATH`/include/lib flags so `hipcc` finds the store-located
|
||||
ROCm libs; adjust `ROCM_CFLAGS`/`ROCM_LDLIBS` via `makeFlags` if needed.
|
||||
|
||||
### Phase 4 — CUDA backend
|
||||
- [ ] `nativeBuildInputs`: `cudaPackages.cuda_nvcc`.
|
||||
- [ ] `buildInputs`: `cudaPackages.cuda_cudart`, `cudaPackages.libcublas`.
|
||||
- [ ] Override the hardcoded CUDA paths via `makeFlags`:
|
||||
- `CUDA_HOME = ...` / `NVCC = "${cudaPackages.cuda_nvcc}/bin/nvcc"`,
|
||||
- a corrected `CUDA_LDLIBS` that points `-L` at the nixpkgs cudart/cublas
|
||||
`lib` dirs instead of `/usr/local/cuda/...` and `sbsa-linux` (that path is
|
||||
aarch64; an x86_64 target should use `lib64`/store libs).
|
||||
- [ ] Build target: `cuda-generic` (`CUDA_ARCH=native`) by default, or `cuda`
|
||||
with an explicit `cudaArch` (preferred for reproducibility / when the
|
||||
build host differs from the GPU host).
|
||||
- [ ] CUDA is unfree → the package's closure is unfree. Keep `meta.license =
|
||||
lib.licenses.mit` for the source itself, but be aware `allowUnfree` is
|
||||
required to build/realize this variant.
|
||||
|
||||
### Phase 5 — CPU backend (default)
|
||||
- [ ] `make cpu`; no GPU inputs. Cheapest build, good for a portable smoke test
|
||||
and as the safe default `backend = "cpu"`.
|
||||
- [ ] Add `doInstallCheck` running `$out/bin/ds4 --help` (only meaningful for
|
||||
the CPU variant; GPU variants may require a device at runtime, so guard
|
||||
or skip the check for them).
|
||||
|
||||
### Phase 6 — Overlay wiring (`modules/core/default.nix`)
|
||||
Add to the existing `nixpkgs.overlays` first entry:
|
||||
```nix
|
||||
(final: _prev: {
|
||||
grok-cli = final.callPackage ../../pkgs/grok-cli { };
|
||||
ds4 = final.callPackage ../../pkgs/ds4 { }; # cpu default
|
||||
ds4-rocm = final.callPackage ../../pkgs/ds4 { backend = "rocm"; };
|
||||
ds4-cuda = final.callPackage ../../pkgs/ds4 { backend = "cuda"; };
|
||||
})
|
||||
```
|
||||
- [ ] Confirm `rocmPackages` / `cudaPackages` resolve via `callPackage` (they
|
||||
are top-level attrs in nixpkgs, so they will).
|
||||
|
||||
### Phase 7 — Flake output (optional)
|
||||
- [ ] In `flake.nix` `packages` output, alongside `grok-cli`, optionally expose:
|
||||
`ds4 = lib.pkgs.${system}.ds4;` (CPU variant — buildable on any system).
|
||||
The GPU variants are awkward as generic flake outputs (need toolchains),
|
||||
so leave them overlay-only unless wanted.
|
||||
|
||||
### Phase 8 — Validation
|
||||
Order by cheapness / portability:
|
||||
- [ ] `nix build .#ds4` (or via a throwaway `nix-build -E`) — CPU variant,
|
||||
builds on the default builder. Resolve the `src` hash here (Phase 1).
|
||||
- [ ] `nix build` the ROCm variant (build on a machine with the ROCm toolchain).
|
||||
Smoke: `ds4 --help`.
|
||||
- [ ] `nix build` the CUDA variant (build on a machine with the CUDA toolchain).
|
||||
Smoke: `ds4 --help`.
|
||||
- [ ] `nixpkgs-fmt` the new/edited Nix files (repo uses it as a pre-commit
|
||||
hook and devshell formatter).
|
||||
- [ ] Update `pkgs/ABOUTME.md` contents table with the new `ds4` entries.
|
||||
|
||||
### Phase 9 — Docs / cleanup
|
||||
- [ ] Add a short header comment in `pkgs/ds4/default.nix` (like `grok-cli`)
|
||||
explaining backends, the `-march=native` "build per host" choice, the
|
||||
`main`-tracking pin, and how to bump (`rev` + refresh `hash`).
|
||||
- [ ] Delete or keep this `plan.md` (suggest keeping until implementation lands,
|
||||
then optionally remove).
|
||||
|
||||
## Risks / unknowns to resolve during implementation
|
||||
|
||||
- **CUDA path overrides**: the Makefile's `CUDA_LDLIBS` hardcodes
|
||||
`targets/sbsa-linux/lib` (aarch64) + `/usr/local/cuda`. On x86_64 nixpkgs
|
||||
this must be replaced. Highest-risk part of the CUDA backend.
|
||||
- **rocWMMA internal headers** under nixpkgs (ROCm phase) — verify availability.
|
||||
- **Hardening flags**: `-ffast-math` + Nix default hardening may produce
|
||||
warnings; if the build breaks, selectively disable (`hardeningDisable`).
|
||||
- **Build-per-host vs. binary cache**: with `-march=native`, store paths are
|
||||
CPU-specific; ensure builds happen on (or for) the right host.
|
||||
|
||||
---
|
||||
|
||||
## Host wiring & deployment (do this last)
|
||||
|
||||
All host-specific assignment and system enablement lives here, after the
|
||||
package itself builds and is wired into the overlay.
|
||||
|
||||
Hosts are defined in `flake.nix` under `nixosConfigurations`. System-level
|
||||
packages for a host are typically added in the host's module (or a shared
|
||||
module it imports); locate where each host declares `environment.systemPackages`
|
||||
/ module package lists and add the appropriate `pkgs.ds4-*` there.
|
||||
|
||||
### `hl-fwdesk` — ROCm
|
||||
- Module: `hosts/hl-fwdesk.nix` (uses
|
||||
`framework-desktop-amd-ai-max-300-series`); the box is AMD Ryzen AI Max 300
|
||||
(Strix Halo, `gfx1151`) — exactly the ROCm/`make strix-halo` target.
|
||||
- [ ] Add `pkgs.ds4-rocm` to the host's package list.
|
||||
- [ ] Confirm ROCm is enabled on the host (drivers, `hardware.graphics` /
|
||||
ROCm runtime) before expecting the GPU path to work.
|
||||
- [ ] System enablement required to *run* it (not part of the package), per
|
||||
`STRIXHALO.md`:
|
||||
- `/dev/kfd` + DRM render-node access (user in `render`/`video` groups).
|
||||
- GTT memory kernel params so the full 128 GB is GPU-visible:
|
||||
`amd_iommu=off amdgpu.gttsize=126976 ttm.pages_limit=32505856
|
||||
ttm.page_pool_size=32505856` (in `boot.kernelParams`).
|
||||
|
||||
### `hl-bigbox1` — CUDA
|
||||
- Module: `hosts/hl-bigbox1.nix`. Host has an NVIDIA GPU.
|
||||
- [ ] Add `pkgs.ds4-cuda` to the host's package list.
|
||||
- [ ] Confirm the exact NVIDIA GPU/arch to decide between `cuda-generic`
|
||||
(`CUDA_ARCH=native`) and a pinned `CUDA_ARCH=sm_NN`
|
||||
(e.g. `sm_86`/`sm_89`/`sm_90`). Pinning is preferred when the build host
|
||||
differs from the GPU host.
|
||||
- [ ] Ensure `nixpkgs.config.allowUnfree` (and CUDA driver/`hardware.graphics`
|
||||
support) is set for this host — the CUDA closure is unfree.
|
||||
- [ ] The configured remote builder `root@bigbox1.lab.internal` is x86_64 and
|
||||
may double as the native CUDA build host (convenient, avoids
|
||||
cross-CPU `-march=native` mismatches).
|
||||
@@ -0,0 +1,75 @@
|
||||
# ds4 (DwarfStar) — user guide
|
||||
|
||||
A from-source build of [`antirez/ds4`](https://github.com/antirez/ds4), a local
|
||||
inference engine for **DeepSeek V4 Flash/PRO**. This is a short operational
|
||||
guide; see the [upstream README](https://github.com/antirez/ds4) for the full
|
||||
documentation (server API, agent, distributed inference, tuning, etc.).
|
||||
|
||||
## What's installed
|
||||
|
||||
The package builds for a chosen GPU backend (`cpu`, `rocm`, or `cuda`) and
|
||||
installs these binaries:
|
||||
|
||||
| Binary | What it does |
|
||||
|---|---|
|
||||
| `ds4` | Interactive CLI chat / one-shot `-p "..."` |
|
||||
| `ds4-server` | OpenAI/Anthropic/Responses-compatible HTTP API |
|
||||
| `ds4-agent` | Native terminal coding agent (alpha) |
|
||||
| `ds4-bench` | Throughput benchmarking |
|
||||
| `ds4-eval` | Capability/regression eval harness |
|
||||
| `ds4-download-model` | Helper to fetch GGUF weights (see below) |
|
||||
|
||||
There is **no graphical UI**. `ds4-server` exposes a URL you can point an
|
||||
OpenAI-compatible client at.
|
||||
|
||||
## 1. Get the weights
|
||||
|
||||
Weights are **not** packaged. They live at the project's Hugging Face repo
|
||||
[`antirez/deepseek-v4-gguf`](https://huggingface.co/antirez/deepseek-v4-gguf)
|
||||
(public, MIT, no token needed) and only these custom GGUFs work — not arbitrary
|
||||
DeepSeek GGUFs.
|
||||
|
||||
`ds4-download-model` writes into `./gguf` and links `./ds4flash.gguf` in the
|
||||
current directory (override the root with `DS4_HOME`, or the gguf dir with
|
||||
`DS4_GGUF_DIR`):
|
||||
|
||||
```sh
|
||||
ds4-download-model q2-imatrix # ~81 GB, the pick for 96/128 GB machines
|
||||
ds4-download-model --help # all targets (q4-imatrix, pro-*, mtp, ...)
|
||||
```
|
||||
|
||||
Pick by machine memory: `q2-imatrix` (128 GB), `q4-imatrix` (≥256 GB),
|
||||
`pro-*` (512 GB / distributed). The small/Flash quants download via `curl`; the
|
||||
huge **PRO** files need the `hf` CLI (not bundled — `nix shell nixpkgs#huggingface-hub`).
|
||||
|
||||
## 2. Run it
|
||||
|
||||
```sh
|
||||
# interactive chat (defaults to ./ds4flash.gguf in the current dir)
|
||||
ds4
|
||||
|
||||
# explicit model path
|
||||
ds4 -m /path/to/gguf/DeepSeek-V4-Flash-IQ2XXS-...-imatrix.gguf
|
||||
|
||||
# HTTP server on the LAN
|
||||
ds4-server -m ./ds4flash.gguf \
|
||||
--ctx 100000 --kv-disk-dir ./kv --kv-disk-space-mb 8192 --host 0.0.0.0
|
||||
```
|
||||
|
||||
Then point a client at `http://<host>:8000/v1` (endpoints: `/v1/chat/completions`,
|
||||
`/v1/messages`, `/v1/responses`, ...). Use `--cors` for browser clients.
|
||||
|
||||
## Backend notes
|
||||
|
||||
- **rocm** targets a specific GPU arch (default `gfx1151`, AMD Strix Halo). At
|
||||
runtime the user needs `/dev/kfd` + render-node access, and large models need
|
||||
the GTT memory kernel params from upstream's `STRIXHALO.md`. On Strix Halo,
|
||||
export `HSA_ENABLE_SDMA=0` (SDMA is buggy on its unified memory).
|
||||
- **cuda** is unfree (CUDA closure) and needs an NVIDIA driver at runtime.
|
||||
- **cpu** is a reference/debug path only — not for production inference.
|
||||
|
||||
## Maintenance
|
||||
|
||||
This tracks upstream `main` (no releases). To bump, set `rev` in `default.nix`
|
||||
and refresh `hash`. Binaries are built per-host (`-march=native`), so build each
|
||||
GPU variant on its target host. See `default.nix`'s header comment for details.
|
||||
Reference in new issue
Block a user