Files
nixos-setup/pkgs/ds4/user-guide.md
T
2026-08-05 16:42:57 -04:00

4.2 KiB

ds4 (DwarfStar) — user guide

A from-source build of antirez/ds4, a local inference engine for DeepSeek V4 Flash/PRO. This is a short operational guide; see the upstream README for the full documentation (server API, agent, distributed inference, tuning, etc.).

What's installed

The package builds for a chosen GPU backend (cpu, rocm, or cuda) and installs these binaries:

Binary What it does
ds4 Interactive CLI chat / one-shot -p "..."
ds4-server OpenAI/Anthropic/Responses-compatible HTTP API
ds4-agent Native terminal coding agent (alpha)
ds4-bench Throughput benchmarking
ds4-eval Capability/regression eval harness
ds4-download-model Helper to fetch GGUF weights (see below)

There is no graphical UI. ds4-server exposes a URL you can point an OpenAI-compatible client at.

1. Get the weights

Weights are not packaged. They live at the project's Hugging Face repos (antirez/deepseek-v4-gguf for DeepSeek V4, antirez/GLM-5.2-GGUF and unsloth/GLM-5.2-GGUF for GLM 5.2) — public, MIT, no token needed — and only these custom GGUFs work, not arbitrary DeepSeek/GLM GGUFs.

ds4-download-model writes into ./gguf and links ./ds4flash.gguf in the current directory (override the root with DS4_HOME, or the gguf dir with DS4_GGUF_DIR):

# DeepSeek V4 Flash (0731 checkpoints)
ds4-download-model ds4f-q2          # ~81 GB, the pick for 96/128 GB machines
ds4-download-model ds4f-q2-q4      # ~98 GB, last 6 expert layers at q4
ds4-download-model ds4f-q4         # ~153 GB, for >=256 GB machines
ds4-download-model ds4f-mxfp4      # ~156 GB, native MXFP4 (Metal/CUDA)
ds4-download-model ds4f-dspark     # ~6 GB, optional DSpark speculative draft

# DeepSeek V4 PRO (512 GB / distributed)
ds4-download-model pro-q2-imatrix          # single-file PRO q2
ds4-download-model pro-q4-split            # both halves of PRO Q4 split

# GLM 5.2
ds4-download-model glm-unsloth-q4         # Unsloth UD-Q4_K_XL, 11 shards
ds4-download-model glm-antirez-iq2xxs     # routed IQ2_XXS, reduced-memory
ds4-download-model glm-antirez-q2         # routed Q2_K, ~262 GB
ds4-download-model glm-antirez-q4         # routed Q4_K, ~434 GB

ds4-download-model --help                 # full target list

Pick by machine memory: ds4f-q2 (96/128 GB), ds4f-q4 (>=256 GB), pro-* (512 GB / distributed), glm-* (per-target sizes above). The small Flash quants download via curl; the MXFP4, PRO, and GLM files need the hf CLI (not bundled — nix shell nixpkgs#huggingface-hub).

2. Run it

# interactive chat (defaults to ./ds4flash.gguf in the current dir)
ds4

# explicit model path
ds4 -m /path/to/gguf/DeepSeek-V4-Flash-IQ2XXS-...-imatrix-0731.gguf

# DSpark speculative decoding (greedy only; needs the ds4f-dspark support GGUF)
ds4 -m ./ds4flash.gguf \
  --dspark --mtp ./gguf/DeepSeek-V4-Flash-DSpark-support-0731.gguf --temp 0

# GLM 5.2 (experimental greedy MTP via --glm-mtp)
ds4 -m gguf/GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf --glm-mtp-timing --temp 0

# HTTP server on the LAN
ds4-server -m ./ds4flash.gguf \
  --ctx 100000 --kv-disk-dir ./kv --kv-disk-space-mb 8192 --host 0.0.0.0

Then point a client at http://<host>:8000/v1 (endpoints: /v1/chat/completions, /v1/messages, /v1/responses, ...). Use --cors for browser clients.

Backend notes

  • rocm targets a specific GPU arch (default gfx1151, AMD Strix Halo). At runtime the user needs /dev/kfd + render-node access, and large models need the GTT memory kernel params from upstream's STRIXHALO.md. On Strix Halo, export HSA_ENABLE_SDMA=0 (SDMA is buggy on its unified memory).
  • cuda is unfree (CUDA closure) and needs an NVIDIA driver at runtime.
  • cpu is a reference/debug path only — not for production inference.

Maintenance

This tracks upstream main (no releases). To bump, set rev in default.nix and refresh hash. Binaries are built per-host (-march=native), so build each GPU variant on its target host. See default.nix's header comment for details.