ds4

README

ds4go

ds4go is a zero-CGO Go wrapper for the ds4 inference engine. Applications using ds4go loads a pre-built libds4 shared library at runtime with github.com/ebitengine/purego. The shared library owns hardware acceleration. Use a Metal, CUDA, or CPU build of ds4 that matches your machine and model.

ds4 itself is an inference engine focused on the DeepSeek v4 Flash model targeting machines with 96G or more of GPU-accessible RAM.

We try to maintain parity with the upstream ds4 library, wrapping its C API. We build slightly-opinionated tools to facilitate using ds4.

Motivation

C is a wonderful language for low-level, high-performance, portable code; a clean C API can be wrapped and used by other laguages. Golang is a wonderful language for systems and tools development, and generally more friendly for developers, esepecially when creating networked applications. LLMs are great at programming both. We take the high-performance C engine of ds4 and allow Golang to directly utilize it, simplifying local LLM application development.

Install

Install the ds4go CLI with Homebrew or the Go toolchain:

# Homebrew (macOS/Linux)
brew install nimblemarkets/tap/ds4go

# or with the Go toolchain
go install github.com/NimbleMarkets/ds4go/cmd/ds4go@latest

To use ds4go as a library:

go get github.com/NimbleMarkets/ds4go

Once the CLI is installed, fetch a prebuilt native libds4 from GitHub Releases:

ds4go install --backend auto

The installer downloads from github.com/NimbleMarkets/ds4 by default. Use --repo, --version, --backend, or --url to select a fork, release, build, or direct archive. It installs into $DS4_DIR/lib, defaulting to ~/.ds4/lib.

DS4_DIR is the ds4 home directory used by ds4go tooling:

$DS4_DIR/lib/      native shared libraries
$DS4_DIR/models/   GGUF model files

Manage curated DeepSeek V4 Flash models with:

ds4go model list
ds4go model download q2-imatrix
ds4go model set q2-imatrix

The default model path for commands and examples is $DS4_DIR/models/ds4flash.gguf.

Place the shared library in ~/.ds4/lib/, next to your executable, or point at it explicitly (the working directory is not searched, to avoid loading a planted library):

export DS4_LIB=/absolute/path/to/libds4.dylib
# or
export DS4_DIR=/opt/ds4

Platform defaults are:

Platform	Library
macOS	libds4.dylib
Linux	libds4.so
Windows	libds4.dll

Usage

import ds4 "github.com/NimbleMarkets/ds4go"

engine, err := ds4.NewEngine(ds4.EngineOptions{
    ModelPath: "/models/ds4flash.gguf",
    Backend:   ds4.BackendMetal,
})
if err != nil {
    panic(err)
}
defer engine.Close()

session, err := engine.NewSession(32768)
if err != nil {
    panic(err)
}
defer session.Close()

prompt, err := engine.EncodeChatPrompt("", "Explain Redis streams briefly.", ds4.ThinkHigh)
if err != nil {
    panic(err)
}
defer prompt.Free()

_, err = ds4.Generator{Engine: engine, Session: session}.GenerateTokens(prompt, ds4.GenerateOptions{
    MaxTokens: 128,
    StopOnEOS: true,
    OnToken: func(token int) {
        text, _ := engine.TokenText(token)
        fmt.Print(text)
    },
})

CLI

go run ./cmd/ds4go prompt --model ./ds4flash.gguf -p "Explain Redis streams in one paragraph."
go run ./cmd/ds4go prompt --model ./ds4flash.gguf

cmd/ds4go prompt and the examples accept the same arguments as the upstream ds4 C programs, parsed with pflag so options take the --option form. cmd/ds4go prompt, examples/simple, and examples/chat mirror the ds4 CLI (ds4_cli.c); examples/openai-compatible mirrors ds4-server (ds4_server.c). Run any of them with --help for the full list.

The only addition with no C equivalent is --lib, which points at the libds4 shared library the pure-Go wrapper loads at runtime (empty falls back to DS4_LIB or DS4_DIR/lib).

$ ds4go help cheat
ds4go — command cheat sheet

  ├── completion      Generate the autocompletion script for the specified shell
  │   ├── bash        Generate the autocompletion script for bash
  │   ├── fish        Generate the autocompletion script for fish
  │   ├── powershell  Generate the autocompletion script for powershell
  │   └── zsh         Generate the autocompletion script for zsh
  │
  ├── install  Download a prebuilt libds4 shared library
  │
  ├── model         Browse, download, and manage curated ds4 models
  │   ├── delete    Delete a downloaded model from disk
  │   ├── download  Download a curated model from Hugging Face
  │   ├── info      Show details for a curated model
  │   ├── list      List installed and available models
  │   └── set       Set the default chat model
  │
  └── prompt  Run prompt or interactive chat inference

Run 'ds4go help <command>' for detailed usage.

Examples

go run ./examples/simple --model ./ds4flash.gguf
go run ./examples/chat --model ./ds4flash.gguf
go run ./examples/openai-compatible --model ./ds4flash.gguf --host 127.0.0.1 --port 8000

The OpenAI-compatible example exposes POST /v1/chat/completions for a minimal local test server.

API Coverage

Most users should import the root package ds4 from github.com/NimbleMarkets/ds4go. It provides Go-native runtime policy and convenience helpers on top of the raw API.

The strict binding layer lives in package ds4api, imported as github.com/NimbleMarkets/ds4go/ds4api. It mirrors the public ds4.h API: engines, sessions, token vectors, chat prompt rendering, tokenization, logprob helpers, MTP metadata, directional steering options, snapshot/payload save-load, and DS4 context-memory helpers. APIs that take FILE * use the package's opaque ds4api.File wrapper around a C FILE*.

ds4_log is exposed as LogString, which safely calls it with a fixed "%s" format. Arbitrary C varargs are intentionally not surfaced as a Go variadic API.

Signal Safety

Do not use signal.NotifyContext around C FFI calls. SIGINT (Ctrl+C) can be delivered to any OS thread, including C worker threads inside libds4 (Metal, CUDA, or CPU). When that happens the C runtime aborts and the process segfaults.

Safe cancellation is programmatic only — pass a context.Context to GenerateOptions.Context and cancel it from Go code. The generator checks ctx.Done() between tokens, so cancellation never interrupts an active FFI call:

ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()

_, err = ds4.Generator{Engine: engine, Session: session}.GenerateTokens(prompt, ds4.GenerateOptions{
    MaxTokens: 128,
    Context:   ctx,
    OnToken: func(token int) {
        text, _ := engine.TokenText(token)
        fmt.Print(text)
    },
})

This is exactly how examples/openai-compatible handles client disconnects — it wires r.Context() into generation so the engine stops cleanly when the HTTP connection drops.

Notes

Bindings are generated by hand against the public ds4 header at https://github.com/antirez/ds4/blob/main/ds4.h.

Inference runs in-process. The Golang wrapper adds FFI calls but does not proxy tokens through a server or copy model weights. Prefill, generation, Metal/CUDA/CPU execution, MTP, KV reuse, and disk KV payload serialization are all handled by the loaded ds4 shared library.

Open Collaboration

We welcome contributions and feedback. Please adhere to our Code of Conduct when engaging our community.

• GitHub Issues
• GitHub Pull Requests

Acknowledgements

Thanks to @antirez for his work on ds4 and for his local-LLM advocacy. Thanks to DeepSeek for their public contributions.

License

Released under the MIT License, see LICENSE.txt.

Copyright (c) 2026 Neomantra Corp.

---

Made with ❤ and 🔥 by the team behind Nimble.Markets.

Documentation

Overview

Package ds4 provides Go-native conveniences for the ds4 inference engine.

The lower-level github.com/NimbleMarkets/ds4go/ds4api package is the strict purego wrapper around ds4.h. This package owns runtime policy such as default paths, friendly diagnostics, and small convenience entry points.

Index

• Constants
• func ApplyMTPDefaults(opts *EngineOptions)
• func DefaultDir() string
• func DefaultLibraryDir() string
• func DefaultLibraryPath() string
• func DefaultMTPPath() string
• func DefaultModelPath() string
• func EnrichEngineOpenError(err error) error
• func Load(path string) (*ds4api.Library, error)
• func NewEngine(opts ds4api.EngineOptions) (*ds4api.Engine, error)
• func SetDefaultLibrary(lib *ds4api.Library)
• type Backend
• type Engine
• type EngineOptions
• type GenerateOptions
• type Generator
  • func (g Generator) Continue(opts GenerateOptions) ([]int, error)
  • func (g Generator) Generate(prompt []int, opts GenerateOptions) ([]int, error)
  • func (g Generator) GenerateString(prompt string, opts GenerateOptions) (string, error)
  • func (g Generator) GenerateTokens(prompt *ds4api.Tokens, opts GenerateOptions) ([]int, error)
• type Library
• type Session
• type ThinkMode
• type TokenEmitFunc
• type Tokens

Constants

const (
	// DefaultTemperature is ds4's default sampling temperature.
	DefaultTemperature = ds4api.DefaultTemperature
	// DefaultTopP is ds4's default nucleus sampling probability.
	DefaultTopP = ds4api.DefaultTopP
	// DefaultMinP is ds4's default minimum relative-probability filter.
	DefaultMinP = ds4api.DefaultMinP

	// BackendMetal selects the Metal backend.
	BackendMetal = ds4api.BackendMetal
	// BackendCUDA selects the CUDA backend.
	BackendCUDA = ds4api.BackendCUDA
	// BackendCPU selects the CPU reference backend.
	BackendCPU = ds4api.BackendCPU

	// ThinkNone disables thinking markers in chat prompts.
	ThinkNone = ds4api.ThinkNone
	// ThinkHigh enables ordinary high-effort thinking.
	ThinkHigh = ds4api.ThinkHigh
	// ThinkMax enables the maximum-effort thinking prefix when the context is large enough.
	ThinkMax = ds4api.ThinkMax

	// DefaultMTPDraftTokens is the default number of draft tokens speculative
	// decoding generates per step when MTP is enabled. A value of 0 disables
	// speculative decoding; set it explicitly to enable MTP.
	DefaultMTPDraftTokens = 0
	// DefaultMTPMargin is the default minimum margin (in tokens) between the
	// draft model's accepted sequence and the full target model output.
	DefaultMTPMargin = 3
)

Functions

func ApplyMTPDefaults

func ApplyMTPDefaults(opts *EngineOptions)

ApplyMTPDefaults populates MTPPath, MTPDraftTokens, and MTPMargin with sensible defaults when an MTP model is installed. It only fills fields that are currently empty or zero, so explicit caller settings are respected.

func DefaultDir

func DefaultDir() string

DefaultDir returns the ds4go data directory.

DS4_DIR overrides the default. When DS4_DIR is unset, DefaultDir returns "$HOME/.ds4" when the user home directory can be determined, otherwise ".ds4".

func DefaultLibraryDir

func DefaultLibraryDir() string

DefaultLibraryDir returns the directory where libds4 is installed by default: the "lib" subdirectory of DefaultDir.

func DefaultLibraryPath

func DefaultLibraryPath() string

DefaultLibraryPath returns the preferred libds4 shared-library path.

Search order is DS4_LIB, DS4_DIR/lib, executable-local paths, and finally the platform library name for system loader lookup.

The current working directory is deliberately NOT searched: loading a shared library from the CWD would let an attacker who can write a file into a directory the user happens to run ds4go from plant a malicious libds4 and gain code execution (binary planting). Use DS4_LIB or DS4_DIR to load a library from a non-default location.

func DefaultMTPPath

func DefaultMTPPath() string

DefaultMTPPath returns the path to the installed MTP companion model, or empty string if it is not present.

func DefaultModelPath

func DefaultModelPath() string

DefaultModelPath returns the path to the default model symlink.

The default model is a symlink at $DS4_DIR/models/<DefaultModelSymlink> that points to the active downloaded model. Use ds4go model set to switch it.

func EnrichEngineOpenError

func EnrichEngineOpenError(err error) error

EnrichEngineOpenError adds process names to ds4 engine-open errors that mention lock-holder PIDs.

func Load

func Load(path string) (*ds4api.Library, error)

Load loads libds4 using ds4go's runtime path policy.

Passing an empty path searches DS4_LIB, DS4_DIR/lib, executable-local library locations, and finally the platform loader path. The current working directory is not searched; see DefaultLibraryPath.

func NewEngine

func NewEngine(opts ds4api.EngineOptions) (*ds4api.Engine, error)

NewEngine loads the default libds4 shared library and opens a ds4 engine.

func SetDefaultLibrary

func SetDefaultLibrary(lib *ds4api.Library)

SetDefaultLibrary makes lib the low-level package default library.

Types

type Backend

type Backend = ds4api.Backend

Backend selects the accelerator implementation compiled into libds4.

type Engine

type Engine = ds4api.Engine

Engine wraps a ds4_engine.

type EngineOptions

type EngineOptions = ds4api.EngineOptions

EngineOptions configures ds4_engine_open.

type GenerateOptions

type GenerateOptions struct {
	// MaxTokens is the maximum number of tokens to generate.
	MaxTokens int
	// Temperature controls sampling. Values <= 0 use argmax.
	Temperature float32
	// TopK limits sampling to the best k tokens when Temperature > 0.
	TopK int
	// TopP applies nucleus sampling when Temperature > 0.
	TopP float32
	// MinP applies minimum probability sampling when Temperature > 0.
	MinP float32
	// Seed seeds ds4's sampler. A zero seed is valid and deterministic.
	Seed uint64
	// StopOnEOS stops generation when ds4 emits the engine EOS token.
	StopOnEOS bool
	// ExcludeToken asks argmax generation to skip a specific token id.
	ExcludeToken int
	// OnToken streams generated tokens. Returning normally continues generation.
	OnToken ds4api.TokenEmitFunc
	// Context, when non-nil, can be cancelled to stop generation gracefully
	// before the next token is sampled.
	Context context.Context
}

GenerateOptions controls Go-native session generation helpers.

type Generator

type Generator struct {
	Engine  *ds4api.Engine
	Session *ds4api.Session
}

Generator binds a ds4 engine and session for Go-native generation helpers.

func (Generator) Continue

func (g Generator) Continue(opts GenerateOptions) ([]int, error)

Continue generates tokens from the current session logits.

func (Generator) Generate

func (g Generator) Generate(prompt []int, opts GenerateOptions) ([]int, error)

Generate synchronizes to prompt and generates tokens from the session.

func (Generator) GenerateString

func (g Generator) GenerateString(prompt string, opts GenerateOptions) (string, error)

GenerateString tokenizes prompt, generates, and decodes the generated text.

func (Generator) GenerateTokens

func (g Generator) GenerateTokens(prompt *ds4api.Tokens, opts GenerateOptions) ([]int, error)

GenerateTokens synchronizes to prompt and generates tokens from the session.

type Library

type Library = ds4api.Library

Library is a loaded libds4 shared library.

type Session

type Session = ds4api.Session

Session wraps a ds4_session.

type ThinkMode

type ThinkMode = ds4api.ThinkMode

ThinkMode controls ds4's rendered chat thinking mode.

type TokenEmitFunc

type TokenEmitFunc = ds4api.TokenEmitFunc

TokenEmitFunc is called when ds4 emits a generated token.

type Tokens

type Tokens = ds4api.Tokens

Tokens owns a ds4_tokens value allocated by libds4.