ds4api

package
v0.5.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 11, 2026 License: MIT Imports: 15 Imported by: 0

Documentation

Overview

Package ds4api test infrastructure: a pure-Go mock of libds4.

NewMockLibrary creates a Library whose raw function pointers are Go implementations backed by in-memory state. This lets tests exercise the ds4go binding and generator layers without loading a real shared library.

Package ds4api provides pure-Go bindings for the ds4 inference engine.

The package does not use cgo. It loads a user-provided libds4 shared library at runtime through purego and wraps the public API from ds4.h.

Index

Constants

View Source 
const (
	// DefaultTemperature is ds4's default sampling temperature.
	DefaultTemperature float32 = 1.0
	// DefaultTopP is ds4's default nucleus sampling probability.
	DefaultTopP float32 = 1.0
	// DefaultMinP is ds4's default minimum relative-probability filter.
	DefaultMinP float32 = 0.05
)

Sampling defaults mirror the DS4_DEFAULT_* macros in ds4.h. The default sampler keeps top-p at 1.0 and uses min-p as the active filter.

Variables

View Source 
var ErrCancelNotSupported = errors.New("ds4: session cancellation is not supported by the loaded library (missing symbol)")

ErrCancelNotSupported is returned by Session.SetCancel when the loaded libds4 build does not export ds4_session_set_cancel.

View Source 
var ErrDistributedNotSupported = errors.New("ds4: distributed inference is not supported by the loaded library (missing symbols)")

ErrDistributedNotSupported is returned by the distributed inference and layer-slice session methods when the loaded libds4 build does not export the ds4_session distributed symbols. Callers can check with errors.Is and fall back to single-node execution.

View Source 
var ErrSessionSyncInterrupted = errors.New("ds4: session sync interrupted")

ErrSessionSyncInterrupted is returned when ds4_session_sync stops because the installed cancel callback requested cooperative cancellation.

View Source 
var ErrSteeringNotSupported = errors.New("ds4: session directional steering is not supported by the loaded library (missing symbol)")

ErrSteeringNotSupported is returned by Session.SetDirectionalSteering when the loaded libds4 build does not export ds4_session_set_directional_steering. Callers can check with errors.Is and downgrade to a warning so the rest of the program can continue without dynamic steering.

Functions

func BackendName 

func BackendName(backend Backend) string

BackendName returns ds4's printable name for backend.

func DumpTextTokenization 

func DumpTextTokenization(modelPath, text string, fp File) error

DumpTextTokenization calls ds4_dump_text_tokenization.

func LogIsTTY 

func LogIsTTY(fp File) bool

LogIsTTY calls ds4_log_is_tty for a C FILE*.

func LogString 

func LogString(fp File, typ LogType, msg string)

LogString writes a plain string through ds4_log using a "%s" format.

func RewriteRequiresRebuild 

func RewriteRequiresRebuild(liveLen, canonicalLen, common int) bool

RewriteRequiresRebuild calls ds4_session_rewrite_requires_rebuild.

func SetAbortFunc added in v0.3.0 

func SetAbortFunc(fn AbortFunc) error

SetAbortFunc installs a last-chance libds4 fatal-invariant callback for the default library.

Passing nil restores libds4's default behavior: no callback, then native abort(). The setting is process-global inside libds4; install it once during application startup. Returning from the callback does not recover the engine: libds4 calls abort() immediately afterward.

func SetDefaultLibrary 

func SetDefaultLibrary(lib *Library)

SetDefaultLibrary makes lib the package default library.

func SetStderrFd added in v0.5.0 

func SetStderrFd(fd int) error

SetStderrFd redirects libds4's diagnostic stream to fd for the default library. Pass -1 to restore the native stderr.

libds4 dups fd internally and writes its diagnostics there unbuffered, so the caller may close its own descriptor after this call. The redirect target is a process-global inside libds4; install it once during application startup, before generation is active.

func ThinkMaxMinContext 

func ThinkMaxMinContext() uint32

ThinkMaxMinContext returns the minimum context size ds4 recommends for ThinkMax.

func ThinkMaxPrefix 

func ThinkMaxPrefix() string

ThinkMaxPrefix returns ds4's maximum-effort thinking prompt prefix.

func ThinkModeEnabled 

func ThinkModeEnabled(mode ThinkMode) bool

ThinkModeEnabled reports whether mode emits thinking markers.

func ThinkModeName 

func ThinkModeName(mode ThinkMode) string

ThinkModeName returns ds4's printable name for mode.

Types

type AbortFunc added in v0.3.0 

type AbortFunc func(msg string)

AbortFunc receives a libds4 fatal-invariant message immediately before libds4 aborts the process.

libds4 calls this from ds4_die and allocation-guard failures after routing the same text through the log callback as LogError. Returning from AbortFunc does not recover the engine: libds4 calls abort() immediately afterward. Use this hook only for last-chance crash telemetry, flushing logs, or deliberate process termination. The callback may be invoked from native worker threads, so it must be concurrency-safe, quick, and must not call back into ds4go/libds4 APIs; doing so can deadlock during an active native call.

type ArgmaxGenerateOptions 

type ArgmaxGenerateOptions struct {
	// NPredict is the number of tokens to generate.
	NPredict int
	// CtxSize is the context size used for this generation.
	CtxSize int
	// OnToken streams generated tokens.
	OnToken TokenEmitFunc
	// OnDone is called by ds4 when generation is complete.
	OnDone GenerationDoneFunc
	// OnProgress receives ds4 progress events.
	OnProgress ProgressFunc
}

ArgmaxGenerateOptions controls ds4_engine_generate_argmax.

type Backend 

type Backend int32

Backend selects the accelerator implementation compiled into libds4. 

const (
	// BackendMetal selects the Metal backend.
	BackendMetal Backend = iota
	// BackendCUDA selects the CUDA backend.
	BackendCUDA
	// BackendCPU selects the CPU reference backend.
	BackendCPU
)

type CancelFunc added in v0.5.1 

type CancelFunc func() bool

CancelFunc is polled by ds4 at cooperative cancellation points. Returning true asks ds4_session_sync to stop at a valid checkpoint boundary.

type ContextMemory 

type ContextMemory struct {
	// TotalBytes is the estimated total context memory.
	TotalBytes uint64
	// RawBytes is the raw KV-cache memory estimate.
	RawBytes uint64
	// CompressedBytes is the compressed KV-cache memory estimate.
	CompressedBytes uint64
	// ScratchBytes is the temporary scratch memory estimate.
	ScratchBytes uint64
	// PrefillCap is the prefill capacity.
	PrefillCap uint32
	// RawCap is the raw KV-cache row capacity.
	RawCap uint32
	// CompCap is the compressed KV-cache row capacity.
	CompCap uint32
}

ContextMemory is ds4_context_memory.

func ContextMemoryEstimate 

func ContextMemoryEstimate(backend Backend, ctxSize int) ContextMemory

ContextMemoryEstimate estimates ds4 context memory for a backend and context size. libds4 derives the estimate from the active model shape selected by ds4_engine_open, so the result is only meaningful while at least one engine is open. Prefer Engine.ContextMemoryEstimate when you have an engine handle.

func ContextMemoryEstimateWithPrefill added in v0.5.1 

func ContextMemoryEstimateWithPrefill(backend Backend, ctxSize int, prefillChunk uint32) ContextMemory

ContextMemoryEstimateWithPrefill estimates ds4 context memory for a backend, context size, and prefill chunk size.

type DistributedLayers added in v0.5.0 

type DistributedLayers struct {
	Start     uint32
	End       uint32
	HasOutput bool
	Set       bool
}

DistributedLayers defines the layer slice bounds for a distributed node.

type DistributedOptions added in v0.5.0 

type DistributedOptions struct {
	Role            DistributedRole
	Layers          DistributedLayers
	ListenHost      string
	ListenPort      int
	CoordinatorHost string
	CoordinatorPort int
	PrefillChunk    uint32
	PrefillWindow   uint32
	ActivationBits  uint32
	ReplayCheck     bool
	Debug           bool
}

DistributedOptions configures the distributed inference data/control plane.

type DistributedRole added in v0.5.0 

type DistributedRole int32

DistributedRole defines the distributed execution mode of the engine. 

const (
	// DistributedRoleNone disables distributed execution.
	DistributedRoleNone DistributedRole = 0
	// DistributedRoleCoordinator acts as the entrypoint coordinator.
	DistributedRoleCoordinator DistributedRole = 1
	// DistributedRoleWorker executes a subset of layer computations.
	DistributedRoleWorker DistributedRole = 2
)

type Engine 

type Engine struct {
	// contains filtered or unexported fields
}

Engine wraps a ds4_engine.

func NewEngine 

func NewEngine(opts EngineOptions) (*Engine, error)

NewEngine opens a ds4 engine using the default shared library.

func (*Engine) ChatAppendAssistantPrefix 

func (e *Engine) ChatAppendAssistantPrefix(tokens *Tokens, thinkMode ThinkMode) error

ChatAppendAssistantPrefix appends the assistant prefix for generation.

Passing ThinkHigh or ThinkMax appends the assistant marker followed by the normal <think> marker. Passing ThinkMax here does not append the maximum-effort prompt text; for incremental prompts, call ChatAppendMaxEffortPrefix once near the beginning of the prompt when the effective thinking mode is ThinkMax. The two methods compose and should not be treated as mutually exclusive for ThinkMax.

func (*Engine) ChatAppendMaxEffortPrefix 

func (e *Engine) ChatAppendMaxEffortPrefix(tokens *Tokens) error

ChatAppendMaxEffortPrefix appends ds4's maximum-effort thinking prompt text.

Use this when constructing a chat prompt incrementally with ChatBegin, ChatAppendMessage, and ChatAppendAssistantPrefix, and the effective thinking mode is ThinkMax. To match ds4_encode_chat_prompt, append it once after ChatBegin and before the system/message turns. It does not append the assistant marker or <think> marker; call ChatAppendAssistantPrefix with the same effective thinking mode at the end of the prompt.

Do not call this in addition to EncodeChatPrompt: ds4_encode_chat_prompt already includes this prefix when thinkMode is ThinkMax.

func (*Engine) ChatAppendMessage 

func (e *Engine) ChatAppendMessage(tokens *Tokens, role, content string) error

ChatAppendMessage appends a rendered role/content chat message.

func (*Engine) ChatBegin 

func (e *Engine) ChatBegin(tokens *Tokens) error

ChatBegin appends ds4's chat preamble to tokens.

func (*Engine) Close 

func (e *Engine) Close()

Close releases the underlying ds4_engine.

func (*Engine) CollectIMatrix 

func (e *Engine) CollectIMatrix(datasetPath, outputPath string, ctxSize, maxPrompts, maxTokens int) error

CollectIMatrix calls ds4_engine_collect_imatrix.

func (*Engine) ContextMemoryEstimate added in v0.4.0 

func (e *Engine) ContextMemoryEstimate(backend Backend, ctxSize int) ContextMemory

ContextMemoryEstimate calls ds4_context_memory_estimate using the active model shape selected by the underlying ds4_engine_open. Prefer this over the package-level ContextMemoryEstimate when an engine is open, since the libds4 estimate now depends on Flash-vs-Pro dimensions.

func (*Engine) ContextMemoryEstimateWithPrefill added in v0.5.1 

func (e *Engine) ContextMemoryEstimateWithPrefill(backend Backend, ctxSize int, prefillChunk uint32) ContextMemory

ContextMemoryEstimateWithPrefill calls ds4_context_memory_estimate_with_prefill using the active model shape and prefill chunk size.

func (*Engine) DumpTokens 

func (e *Engine) DumpTokens(tokens *Tokens) error

DumpTokens calls ds4_engine_dump_tokens.

func (*Engine) EncodeChatPrompt 

func (e *Engine) EncodeChatPrompt(system, prompt string, thinkMode ThinkMode) (*Tokens, error)

EncodeChatPrompt encodes a system and user prompt with ds4's chat template.

func (*Engine) FirstTokenTest 

func (e *Engine) FirstTokenTest(prompt *Tokens) error

FirstTokenTest calls ds4_engine_first_token_test.

func (*Engine) GenerateArgmax 

func (e *Engine) GenerateArgmax(prompt *Tokens, opts ArgmaxGenerateOptions) ([]int, error)

GenerateArgmax calls ds4_engine_generate_argmax.

func (*Engine) HasMTP 

func (e *Engine) HasMTP() bool

HasMTP reports whether this engine has an MTP draft model.

func (*Engine) HasOutputHead added in v0.5.0 

func (e *Engine) HasOutputHead() bool

HasOutputHead reports whether the loaded GGUF includes the output head. For a distributed split this is false on the coordinator half (e.g. layers 0:30) and true on the worker half that owns the tail (e.g. 31:output).

func (*Engine) HeadTest 

func (e *Engine) HeadTest(prompt *Tokens) error

HeadTest calls ds4_engine_head_test.

func (*Engine) LayerCompressRatio added in v0.5.0 

func (e *Engine) LayerCompressRatio(layer int) int

LayerCompressRatio returns the compression/quantization ratio of the specified layer, or 0 if not supported.

func (*Engine) LayerCount added in v0.5.0 

func (e *Engine) LayerCount() int

LayerCount returns the total number of layers in the model, or 0 if not supported by the loaded library.

func (*Engine) MTPDraftTokens 

func (e *Engine) MTPDraftTokens() int

MTPDraftTokens returns the configured MTP draft length.

func (*Engine) MetalGraphFullTest 

func (e *Engine) MetalGraphFullTest(prompt *Tokens) error

MetalGraphFullTest calls ds4_engine_metal_graph_full_test.

func (*Engine) MetalGraphPromptTest 

func (e *Engine) MetalGraphPromptTest(prompt *Tokens, ctxSize int) error

MetalGraphPromptTest calls ds4_engine_metal_graph_prompt_test.

func (*Engine) MetalGraphTest 

func (e *Engine) MetalGraphTest(prompt *Tokens) error

MetalGraphTest calls ds4_engine_metal_graph_test.

func (*Engine) ModelID added in v0.4.0 

func (e *Engine) ModelID() int

ModelID returns ds4_engine_model_id: a stable id for cache compatibility. 0 is the original Flash shape; Pro and later shapes use nonzero ids.

func (*Engine) ModelName added in v0.4.0 

func (e *Engine) ModelName() string

ModelName returns ds4_engine_model_name: the printable name of the opened model shape (e.g. "Flash", "Pro").

func (*Engine) NewSession 

func (e *Engine) NewSession(ctxSize int) (*Session, error)

NewSession creates a ds4 session for this engine and context size.

func (*Engine) NewTokens 

func (e *Engine) NewTokens(ids []int) (*Tokens, error)

NewTokens creates a libds4-owned token vector associated with this engine's library.

func (*Engine) Power added in v0.4.0 

func (e *Engine) Power() int

Power returns ds4_engine_power: the current power-throttle duty cycle percentage (1..100). 100 means no throttling.

func (*Engine) RoutedQuantBits 

func (e *Engine) RoutedQuantBits() int

RoutedQuantBits returns the routed expert quantization bits used by the engine.

func (*Engine) SetPower added in v0.4.0 

func (e *Engine) SetPower(powerPercent int) error

SetPower calls ds4_engine_set_power. powerPercent must be in 1..100.

func (*Engine) Summary 

func (e *Engine) Summary() error

Summary prints ds4's engine summary to its configured output.

func (*Engine) TokenAssistant 

func (e *Engine) TokenAssistant() int

TokenAssistant returns ds4's assistant-role token id.

func (*Engine) TokenEOS 

func (e *Engine) TokenEOS() int

TokenEOS returns ds4's end-of-sequence token id.

func (*Engine) TokenText 

func (e *Engine) TokenText(token int) (string, error)

TokenText decodes one token to text and frees the C allocation returned by ds4.

func (*Engine) TokenUser 

func (e *Engine) TokenUser() int

TokenUser returns ds4's user-role token id.

func (*Engine) TokenizeRenderedChat 

func (e *Engine) TokenizeRenderedChat(text string) (*Tokens, error)

TokenizeRenderedChat tokenizes a rendered chat prompt.

func (*Engine) TokenizeText 

func (e *Engine) TokenizeText(text string) (*Tokens, error)

TokenizeText tokenizes plain text with ds4_tokenize_text.

func (*Engine) VocabSize added in v0.4.0 

func (e *Engine) VocabSize() int

VocabSize returns ds4_engine_vocab_size: the model vocabulary size.

type EngineOptions 

type EngineOptions struct {
	// ModelPath is the path to the DeepSeek V4 Flash GGUF model.
	ModelPath string
	// MTPPath is the optional MTP draft model path.
	MTPPath string
	// Backend selects Metal, CUDA, or CPU according to the libds4 build.
	Backend Backend
	// NThreads controls CPU worker threads when the backend uses them.
	NThreads int
	// MTPDraftTokens controls speculative draft length.
	MTPDraftTokens int
	// MTPMargin controls speculative acceptance confidence.
	MTPMargin float32
	// DirectionalSteeringFile points at an optional directional steering file.
	DirectionalSteeringFile string
	// DirectionalSteeringAttn scales directional steering in attention blocks.
	DirectionalSteeringAttn float32
	// DirectionalSteeringFFN scales directional steering in FFN blocks.
	DirectionalSteeringFFN float32
	// PowerPercent throttles GPU work to roughly this duty cycle (1..100).
	// 0 or 100 disables throttling. Maps to ds4_engine_options.power_percent.
	PowerPercent int
	// PrefillChunk controls the prefill chunk size.
	PrefillChunk uint32
	// ExpertProfilePath is the path to the optional expert profile.
	ExpertProfilePath string
	// SSDStreamingCacheExperts is the number of routed experts to keep in VRAM.
	SSDStreamingCacheExperts uint32
	// SSDStreamingCacheBytes is the byte budget for the SSD streaming expert cache.
	SSDStreamingCacheBytes uint64
	// SSDStreamingPreloadExperts is the number of experts to preload during startup.
	SSDStreamingPreloadExperts uint32
	// SimulateUsedMemoryBytes simulates a specific amount of used GPU memory in bytes.
	SimulateUsedMemoryBytes uint64
	// WarmWeights asks ds4 to warm model weights after load.
	WarmWeights bool
	// Quality requests ds4's quality-oriented execution path where supported.
	Quality bool
	// SSDStreaming enables SSD streaming of experts.
	SSDStreaming bool
	// SSDStreamingCold enables SSD streaming of experts with cold cache.
	SSDStreamingCold bool
	// InspectOnly opens the model for inspection without preparing the engine
	// for generation. Maps to ds4_engine_options.inspect_only.
	InspectOnly bool
	// LoadSlice asks ds4 to load only a subset of the model's layers.
	LoadSlice bool
	// LoadLayerStart is the starting layer index to load (inclusive).
	LoadLayerStart uint32
	// LoadLayerEnd is the ending layer index to load (inclusive).
	LoadLayerEnd uint32
	// LoadOutput indicates whether the output vocab projection head should be loaded.
	LoadOutput bool
	// Distributed configures the distributed inference mesh network options.
	Distributed DistributedOptions
}

EngineOptions configures ds4_engine_open.

type File 

type File uintptr

File is an opaque C FILE* used by ds4 APIs that accept FILE pointers.

func OpenFile 

func OpenFile(path, mode string) (File, error)

OpenFile opens a C FILE* with fopen for ds4 FILE*-based APIs.

func (File) Close 

func (f File) Close() error

Close closes a C FILE* opened by OpenFile.

type GenerationDoneFunc 

type GenerationDoneFunc func()

GenerationDoneFunc is called after ds4 completes generation.

type Library 

type Library struct {
	// contains filtered or unexported fields
}

Library is a loaded libds4 shared library.

func DefaultLibrary 

func DefaultLibrary() (*Library, error)

DefaultLibrary returns the lazily loaded default library.

func Load 

func Load(path string) (*Library, error)

Load loads libds4 from path and registers all ds4.h symbols.

Passing an empty path uses DS4_LIB, then searches common local library locations. Higher-level ds4go runtime path policy lives in the module root.

func NewMockLibrary added in v0.2.2 

func NewMockLibrary() *Library

NewMockLibrary returns a Library whose C symbols are backed by trivial in-memory state. The mock supports engine/session lifecycle, tokenization, deterministic generation, and optional MTP metadata.

func (*Library) NewEngine 

func (l *Library) NewEngine(opts EngineOptions) (*Engine, error)

NewEngine opens a ds4 engine using this shared library.

func (*Library) Path 

func (l *Library) Path() string

Path returns the filesystem path used to load this library.

func (*Library) SetAbortFunc added in v0.3.0 

func (l *Library) SetAbortFunc(fn AbortFunc) error

SetAbortFunc installs a last-chance libds4 fatal-invariant callback for this loaded library.

libds4 invokes the callback from ds4_die and allocation-guard failures after logging the same message at LogError and immediately before abort(). Passing nil restores the default behavior. This hook is for crash telemetry, flushing diagnostics, or deliberate process termination; it is not a normal recovery mechanism. If the callback returns, libds4 still aborts. The callback may be invoked from native worker threads and must be concurrency-safe.

func (*Library) SetStderrFd added in v0.5.0 

func (l *Library) SetStderrFd(fd int) error

SetStderrFd redirects this library's diagnostic stream to fd. Pass -1 to restore the native stderr.

libds4 dups fd internally (taking ownership of the dup) and writes its diagnostics there unbuffered, so the caller may close its own descriptor afterward. The redirect target is a process-global inside libds4, not a per-engine setting; install it once at startup, before generation, since it is not synchronized against concurrent logging from native worker threads.

func (*Library) SupportsDistributed added in v0.5.0 

func (l *Library) SupportsDistributed() bool

SupportsDistributed reports whether the loaded library exports the distributed inference and layer-slice entry points. When false, the distributed session methods return ErrDistributedNotSupported and callers should fall back to single-node execution.

func (*Library) SupportsDynamicSteering added in v0.4.0 

func (l *Library) SupportsDynamicSteering() bool

SupportsDynamicSteering reports whether the loaded library exports ds4_session_set_directional_steering. When false, callers should rely on engine-level (static) steering via EngineOptions.DirectionalSteering* fields.

func (*Library) SupportsSessionCancel added in v0.5.1 

func (l *Library) SupportsSessionCancel() bool

SupportsSessionCancel reports whether the loaded library exports ds4_session_set_cancel. When false, Session.SetCancel returns ErrCancelNotSupported.

type LogType 

type LogType int32

LogType is the category used by ds4_log. 

const (
	// LogDefault is the default ds4 log style.
	LogDefault LogType = iota
	// LogPrefill marks prefill messages.
	LogPrefill
	// LogGeneration marks generation messages.
	LogGeneration
	// LogKVCache marks KV-cache messages.
	LogKVCache
	// LogTool marks tool-calling messages.
	LogTool
	// LogWarning marks warnings.
	LogWarning
	// LogTiming marks timing messages.
	LogTiming
	// LogOK marks successful status messages.
	LogOK
	// LogError marks errors.
	LogError
)

type ProgressFunc 

type ProgressFunc func(event string, current, total int)

ProgressFunc receives ds4 progress events.

type Session 

type Session struct {
	// contains filtered or unexported fields
}

Session wraps a ds4_session.

func (*Session) Argmax 

func (s *Session) Argmax() int

Argmax returns the argmax token id for the current logits.

func (*Session) ArgmaxExcluding 

func (s *Session) ArgmaxExcluding(excludedID int) int

ArgmaxExcluding returns the argmax token id excluding one token.

func (*Session) Close 

func (s *Session) Close()

Close releases the underlying ds4_session.

func (*Session) CommonPrefix 

func (s *Session) CommonPrefix(prompt *Tokens) int

CommonPrefix returns the common prefix length between the live session and prompt.

func (*Session) CopyLogits added in v0.4.0 

func (s *Session) CopyLogits() ([]float32, error)

CopyLogits returns a copy of the current logits vector for the session. The slice length equals the engine vocabulary size.

func (*Session) Ctx 

func (s *Session) Ctx() int

Ctx returns the session context size.

func (*Session) DistributedRouteReady added in v0.5.0 

func (s *Session) DistributedRouteReady() (bool, error)

DistributedRouteReady returns 1 when the coordinator has a complete worker route, 0 when workers are still missing, and -1 for internal errors.

func (*Session) Eval 

func (s *Session) Eval(token int) error

Eval evaluates one token and advances the session.

func (*Session) EvalLayerSlice added in v0.5.0 

func (s *Session) EvalLayerSlice(
	tokens []int32,
	pos0 uint32,
	layerStart uint32,
	layerEnd uint32,
	inputHC []float32,
	outputHC []float32,
	outputLogits bool,
	logits []float32,
) error

EvalLayerSlice runs forward pass computations for a contiguous subset of model layers.

func (*Session) EvalOutputHeadFromHC added in v0.5.0 

func (s *Session) EvalOutputHeadFromHC(hiddenHC []float32, nTokens uint32, logits []float32) error

EvalOutputHeadFromHC computes vocabulary logits from final hidden activations.

func (*Session) EvalSpeculativeArgmax 

func (s *Session) EvalSpeculativeArgmax(firstToken, maxTokens, eosToken int) ([]int, error)

EvalSpeculativeArgmax calls ds4_session_eval_speculative_argmax.

func (*Session) Invalidate 

func (s *Session) Invalidate()

Invalidate invalidates the live session state.

func (*Session) IsDistributed added in v0.5.0 

func (s *Session) IsDistributed() bool

IsDistributed returns whether the session is operating in distributed mode.

func (*Session) LayerPayloadBytes added in v0.5.0 

func (s *Session) LayerPayloadBytes(layerStart uint32, layerEnd uint32) uint64

LayerPayloadBytes returns the serialization size of a layer slice KV cache payload.

func (*Session) LayerSliceReset added in v0.5.0 

func (s *Session) LayerSliceReset() error

LayerSliceReset resets the graph and KV cache state of a distributed layer slice.

func (*Session) LoadLayerPayload added in v0.5.0 

func (s *Session) LoadLayerPayload(fp uintptr, payloadBytes uint64, tokens []int32, layerStart uint32, layerEnd uint32) error

LoadLayerPayload reads a layer slice KV cache payload from fp.

func (*Session) LoadPayload 

func (s *Session) LoadPayload(fp File, payloadBytes uint64) error

LoadPayload reads a DS4-specific session payload from fp.

func (*Session) LoadPayloadFile 

func (s *Session) LoadPayloadFile(path string, payloadBytes uint64) error

LoadPayloadFile reads a DS4-specific session payload from path.

func (*Session) LoadSnapshot 

func (s *Session) LoadSnapshot(data []byte) error

LoadSnapshot restores a session snapshot previously returned by SaveSnapshot.

func (*Session) PayloadBytes 

func (s *Session) PayloadBytes() uint64

PayloadBytes returns ds4_session_payload_bytes.

func (*Session) Pos 

func (s *Session) Pos() int

Pos returns the current session token position.

func (*Session) Power added in v0.4.0 

func (s *Session) Power() int

Power returns ds4_session_power. Sessions share the engine power setting, so this returns the engine's current duty-cycle percentage.

func (*Session) Rewind 

func (s *Session) Rewind(pos int)

Rewind rewinds the session to token position pos.

func (*Session) RewriteFromCommon 

func (s *Session) RewriteFromCommon(prompt *Tokens, common int) (SessionRewriteResult, error)

RewriteFromCommon rewrites a session from a known common prefix length.

func (*Session) Sample 

func (s *Session) Sample(temperature float32, topK int, topP, minP float32, rng *uint64) int

Sample samples the next token from current logits.

func (*Session) SaveLayerPayload added in v0.5.0 

func (s *Session) SaveLayerPayload(fp uintptr, layerStart uint32, layerEnd uint32) error

SaveLayerPayload writes a layer slice KV cache payload to fp.

func (*Session) SavePayload 

func (s *Session) SavePayload(fp File) error

SavePayload writes the DS4-specific session payload to fp.

func (*Session) SavePayloadFile 

func (s *Session) SavePayloadFile(path string) error

SavePayloadFile writes the DS4-specific session payload to path.

func (*Session) SaveSnapshot 

func (s *Session) SaveSnapshot() ([]byte, error)

SaveSnapshot serializes a session snapshot to a Go byte slice.

func (*Session) SetCancel added in v0.5.1 

func (s *Session) SetCancel(fn CancelFunc) error

SetCancel sets a persistent cooperative cancellation callback for ds4_session_set_cancel. ds4_session_sync polls this callback only at safe boundaries where the live checkpoint remains valid.

func (*Session) SetDirectionalSteering added in v0.4.0 

func (s *Session) SetDirectionalSteering(file string, mode SteeringMode, ffn float32, attn float32, threshold float32, scope SteeringScope) error

SetDirectionalSteering dynamically updates the directional steering configurations for this session. If the underlying libds4 shared library does not support dynamic steering, this returns an error.

func (*Session) SetDisplayProgress added in v0.4.0 

func (s *Session) SetDisplayProgress(fn ProgressFunc) error

SetDisplayProgress sets a persistent UI-only progress callback for ds4_session_set_display_progress. It may report fine-grained progress inside a prefill chunk; callers must not treat it as a durable KV checkpoint boundary. Use SetProgress for checkpoint-anchored events.

func (*Session) SetPower added in v0.4.0 

func (s *Session) SetPower(powerPercent int) error

SetPower calls ds4_session_set_power. powerPercent must be in 1..100. This updates the engine-wide setting as well as the session's GPU graph, so concurrent sessions sharing the same engine see the new value.

func (*Session) SetProgress 

func (s *Session) SetProgress(fn ProgressFunc) error

SetProgress sets a persistent progress callback for ds4_session_set_progress.

func (*Session) Sync 

func (s *Session) Sync(prompt []int) error

Sync synchronizes the live session to a full prompt token prefix.

func (*Session) SyncTokens 

func (s *Session) SyncTokens(prompt *Tokens) error

SyncTokens synchronizes the live session to a full prompt token prefix.

func (*Session) SyncTokensWithCancel added in v0.5.1 

func (s *Session) SyncTokensWithCancel(prompt *Tokens, fn CancelFunc) error

SyncTokensWithCancel synchronizes the live session to a full prompt token prefix while polling fn for cooperative cancellation. Any persistent SetCancel callback is restored before SyncTokensWithCancel returns.

func (*Session) SyncWithCancel added in v0.5.1 

func (s *Session) SyncWithCancel(prompt []int, fn CancelFunc) error

SyncWithCancel synchronizes the live session to a full prompt token prefix while polling fn for cooperative cancellation. Any persistent SetCancel callback is restored before SyncWithCancel returns.

func (*Session) TokenLogprob 

func (s *Session) TokenLogprob(token int) (TokenScore, error)

TokenLogprob returns the score for a specific token.

func (*Session) Tokens 

func (s *Session) Tokens() *Tokens

Tokens returns a borrowed snapshot of ds4_session_tokens.

func (*Session) TopLogprobs 

func (s *Session) TopLogprobs(k int) ([]TokenScore, error)

TopLogprobs returns the top k token scores for the current logits.

type SessionRewriteResult 

type SessionRewriteResult int32

SessionRewriteResult is returned by ds4 session rewrite helpers. 

const (
	// SessionRewriteError means the rewrite failed.
	SessionRewriteError SessionRewriteResult = -1
	// SessionRewriteOK means the rewrite completed in place.
	SessionRewriteOK SessionRewriteResult = 0
	// SessionRewriteRebuildNeeded means the caller should restore or rebuild the session state.
	SessionRewriteRebuildNeeded SessionRewriteResult = 1
)

type SteeringMode added in v0.4.0 

type SteeringMode int32

SteeringMode defines the formula used to steer session activations. 

const (
	// SteeringAblation projects out activations along the steering direction.
	SteeringAblation SteeringMode = 0
	// SteeringThreshold applies activation steering only above a projection threshold (CAST).
	SteeringThreshold SteeringMode = 1
	// SteeringAdditive projects activations along the steering direction (Golden Gate).
	SteeringAdditive SteeringMode = 2
)

type SteeringScope added in v0.4.0 

type SteeringScope int32

SteeringScope defines the lifetime/scope of the dynamic steering settings. 

const (
	// SteeringScopeNextMessage reverts the steering setting automatically after one message.
	SteeringScopeNextMessage SteeringScope = 0
	// SteeringScopeUntilRevert keeps the steering setting active until explicitly reverted or changed.
	SteeringScopeUntilRevert SteeringScope = 1
	// SteeringScopeOff disables dynamic steering.
	SteeringScopeOff SteeringScope = 2
)

type ThinkMode 

type ThinkMode int32

ThinkMode controls ds4's rendered chat thinking mode. 

const (
	// ThinkNone disables thinking markers in chat prompts.
	ThinkNone ThinkMode = iota
	// ThinkHigh enables ordinary high-effort thinking.
	ThinkHigh
	// ThinkMax requests maximum-effort thinking. ds4_think_mode_for_context
	// may downgrade it to ThinkHigh when the context is below ThinkMaxMinContext.
	ThinkMax
)

func ThinkModeForContext 

func ThinkModeForContext(mode ThinkMode, ctxSize int) ThinkMode

ThinkModeForContext returns the effective thinking mode for a context size.

type TokenEmitFunc 

type TokenEmitFunc func(token int)

TokenEmitFunc is called when ds4 emits a generated token.

type TokenScore 

type TokenScore struct {
	// ID is the token id.
	ID int
	// Logit is the raw model logit.
	Logit float32
	// Logprob is the log probability for the token.
	Logprob float32
}

TokenScore is ds4_token_score.

type Tokens 

type Tokens struct {
	// contains filtered or unexported fields
}

Tokens owns a ds4_tokens value allocated by libds4.

func NewTokens 

func NewTokens(ids []int) (*Tokens, error)

NewTokens creates a libds4-owned token vector from ids.

func (*Tokens) Cap 

func (t *Tokens) Cap() int

Cap returns the token vector capacity.

func (*Tokens) Copy 

func (t *Tokens) Copy() *Tokens

Copy returns a deep copy of this token vector.

func (*Tokens) Free 

func (t *Tokens) Free()

Free releases memory owned by this token vector.

func (*Tokens) Len 

func (t *Tokens) Len() int

Len returns the number of tokens.

func (*Tokens) Push 

func (t *Tokens) Push(token int)

Push appends one token id to the vector.

func (*Tokens) Slice 

func (t *Tokens) Slice() []int

Slice returns a copy of the token ids.

func (*Tokens) StartsWith 

func (t *Tokens) StartsWith(prefix *Tokens) bool

StartsWith reports whether t begins with prefix.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.