needle

package module
v0.0.0-...-cc9977f Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 27, 2026 License: MIT Imports: 22 Imported by: 0

README

needle-go

needle-go is a Go library for running the Cactus Needle on-device tool-calling model.

It provides:

  • CGO-free native engine loading
  • Needle 2 and Needle 3, with Needle 3 as the default
  • automatic, checksum-verified engine and base-weight downloads
  • high-level and manual completion loops
  • typed Go tool handlers
  • structured response extraction
  • Needle 3 text embeddings
  • opt-in Whistle speech transcription, live streaming and audio tool calling

This project is in early development and its API may change.

Install

Requires Go 1.25 or newer.

go get github.com/zbiljic/needle-go

Engine builds are available for macOS, Linux (glibc and musl), and Windows on amd64 and arm64.

Quick Start

package main

import (
	"context"
	"fmt"
	"log"

	needle "github.com/zbiljic/needle-go"
)

type weatherArguments struct {
	City string `json:"city" jsonschema:"description=City whose weather should be returned."`
}

func main() {
	ctx := context.Background()
	weather := needle.NewTool(
		"get_weather",
		"Get the current weather for a city.",
		func(_ context.Context, arguments weatherArguments) (string, error) {
			return "Clear in " + arguments.City, nil
		},
	)

	agent, err := needle.New(ctx, needle.Config{Tools: []needle.Tool{weather}})
	if err != nil {
		log.Fatal(err)
	}
	response, err := agent.Run(
		ctx,
		"What is the weather in Lagos?",
		needle.DefaultMaxSteps,
		needle.DefaultMaxNewTokens,
	)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(response.Results)
}

By default, needle.New downloads and caches the pinned Needle 3 engine (3.3.1) and its needle3.cact base weights (~35 MB) from the Needle 3 model repository. Subsequent runs use the cached files. No Python installation is required.

Tool triggers

Opt into regex routing by assigning triggers after NewTool, using the weather tool above:

weather.Schema.Triggers = []string{`\bweather\b`, `\bforecast\b`}

The native engine interprets the patterns and matches them case-insensitively. Matching triggers can bypass the engine's confidence floor; application confidence policies and response validation still apply.

Independent requests

Agents retain conversation context by default. Set Stateless: true for independent requests:

agent, err := needle.New(ctx, needle.Config{Tools: tools, Stateless: true})

Each Complete or Run starts with a fresh conversation; Run retains context between its tool rounds. Keep the default stateful mode for manual tool loops that feed results back through Complete, and call Reset between independent queries.

Closing

Close releases an agent's response buffer and native binding; later calls return ErrClosed. The engine and loaded weights stay in the process for other agents. Whistle has the same Close.

Text embeddings

Needle 3 agents expose text features as []float32:

vector, err := agent.Embed(ctx, "turn on the kitchen lights")

Embed does not reset the conversation, including in stateless mode. Switching agents still follows the conversation rules below. Needle 2 and engines without needle_embed return an error; both legacy and audio-capable Needle 3 ABIs are supported.

The base model uses confidence-head probe features rather than a trained contrastive retrieval head. Evaluate similarity on your own data; this API does not enable automatic tool retrieval. See the cosine-ranking example and upstream embedding notes.

To check finite, repeatable vectors and a tool-result continuation with an embedding between turns against the real engine:

NEEDLE_TEST_NATIVE=1 go test -run '^TestNativeEmbeddings$' -v .

Speech

NewWhistle adds optional speech transcription, live streaming transcription and audio tool calling with Needle 3. It accepts 16 kHz mono []float32 PCM of any length. Speech weights are downloaded only when requested.

See the speech guide for the API, supported languages and offline setup, or try the PCM example.

Model generations

Use Generation: 2 for the embedded Needle 2 model (engine 2.0.4):

agent, err := needle.New(ctx, needle.Config{
	Generation: 2,
	Tools:      tools,
})

Generation: 0 and Generation: 3 select Needle 3. A custom WeightsPath selects the compatible engine from the .cact header, taking precedence over Generation. Custom Needle 3 archives retain the engine's confidence scores when their metadata carries a recognized confidence head, including platform fine-tunes and the published base archive supplied through WeightsPath. Custom Needle 2 weights and recognized text archives without a valid confidence head report Confidence == nil. Unrecognized model-kind metadata is rejected before loading. Automatically loaded base weights retain the engine's confidence scores.

Both generations can run in the same process, with separate native runtimes. Within one generation, switching agents reinitializes the active conversation. After loading custom Needle 2 weights, returning to its embedded base model requires a separate process. Needle 3 can reload its base archive.

Use LibraryPath for an existing, trusted engine matching the selected generation. When LibraryPath is empty, environment overrides are NEEDLE2_LIB_PATH and NEEDLE3_LIB_PATH; the legacy NEEDLE_LIB_PATH is a fallback for Needle 2 only. A Needle 3 library override still downloads missing base weights unless WeightsPath is supplied.

Native loading detects the audio-capable ABI from its exported capability symbols. Trusted older Needle 2 and Needle 3 libraries retain their legacy completion ABI; incomplete capability sets are rejected before inference. Engines exposing needle_last_error provide detailed native failure messages.

FetchEngine prepares the library and required base weights for offline use. FetchOptions.Generation selects the generation. Default caches are under ~/.cache/cactus-needle/<engine-version>/<platform>/; CacheDir selects an exact directory. V2 uses libneedle.*, while v3 uses libneedle3.*, so the generations can share a custom cache directory for the same platform. For offline use, fetch for the target platform and generation, preserve the .sha256 marker files, and use the same cache directory at runtime. Markers contain the downloaded artifact hash and installed file hash; cached downloads verify both. CachedEngine checks only for the library and trusts the caller's file.

Errors

Errors wrap sentinels for errors.Is: ErrValidation (also wrapped by Run), ErrUnsupportedGeneration, ErrEmbeddingUnsupported, ErrSpeechUnsupported, ErrUnsupportedABI, ErrUnsupportedPlatform, ErrEngineNotFound, ErrStreamActive and ErrClosed. A failed engine call is a *NativeError with the operation, the native code and any engine diagnostic, for errors.As.

Native engine diagnostics

The optional needlez command can fetch the engine and verify that it loads:

go run github.com/zbiljic/needle-go/cmd/needlez@latest fetch
go run github.com/zbiljic/needle-go/cmd/needlez@latest doctor --smoke
go run github.com/zbiljic/needle-go/cmd/needlez@latest test

To run the checks against Needle 2 from a repository checkout:

go run ./cmd/needlez fetch --generation 2
go run ./cmd/needlez doctor --generation 2 --smoke
go run ./cmd/needlez test --generation 2

Responses preserve suppressed_calls for inspection. Run executes only function_calls; it never executes suppressed calls automatically.

Examples

License

Licensed under the MIT License.

Documentation

Overview

Package needle provides Go access to the Needle tool-calling model.

Index

Constants

View Source
const (
	// EngineVersion is the default (Needle 3) native engine version.
	EngineVersion = "3.3.1"
	// EnvLibraryPath overrides Needle 2 engine discovery for legacy clients.
	EnvLibraryPath = "NEEDLE_LIB_PATH"
)
View Source
const (
	// DefaultGeneration is used when no generation or custom weights are selected.
	DefaultGeneration = 3
	// DefaultMaxSteps is the default maximum number of tool-calling rounds.
	DefaultMaxSteps = 8
	// DefaultMaxNewTokens is the default generation limit for each completion.
	DefaultMaxNewTokens = 512
	// DefaultBufferSize is the default native response buffer size.
	DefaultBufferSize = 64 * 1024
)
View Source
const (
	// SpeechSampleRate is the required rate for mono float32 PCM.
	SpeechSampleRate = 16000
	// DefaultSpeechBufferSize accommodates transcripts with word timestamps for
	// clips up to about a minute; longer clips grow the buffer.
	DefaultSpeechBufferSize = 1 << 18
)

Variables

View Source
var (
	// ErrUnsupportedPlatform indicates that no shared-library backend or engine
	// build exists for the target.
	ErrUnsupportedPlatform = errors.New("needle: unsupported platform")
	// ErrEngineNotFound indicates that a cached engine library is unavailable.
	ErrEngineNotFound = errors.New("needle: engine library not found")
	// ErrUnsupportedGeneration indicates a generation other than 2 or 3, or a
	// feature the selected generation lacks.
	ErrUnsupportedGeneration = errors.New("needle: unsupported generation")
	// ErrUnsupportedABI indicates a native library with an incomplete
	// capability set.
	ErrUnsupportedABI = errors.New("needle: unsupported native ABI")
	// ErrEmbeddingUnsupported indicates that the model or engine has no
	// embeddings.
	ErrEmbeddingUnsupported = errors.New("needle: embeddings unavailable")
	// ErrSpeechUnsupported indicates that the engine has no speech ABI.
	ErrSpeechUnsupported = errors.New("needle: speech unavailable")
	// ErrValidation is wrapped by ValidateResponse, and by Run, when the engine
	// flags a response.
	ErrValidation = errors.New("needle: response validation")
	// ErrStreamActive indicates that a live transcription is open on the
	// engine, so other speech calls must wait for its Stop.
	ErrStreamActive = errors.New("needle: a live transcription is in progress")
	// ErrClosed is returned by an Agent or Whistle after Close.
	ErrClosed = errors.New("needle: closed")
)

Sentinel errors for errors.Is. Returned errors wrap them with detail.

Functions

func CachedEngine

func CachedEngine(options FetchOptions) (string, error)

CachedEngine returns the expected cached library path without downloading anything. It accepts libraries installed by needle-go or another client.

func EngineVersionFor

func EngineVersionFor(generation int) (string, error)

EngineVersionFor returns the pinned engine version for a generation (zero means 3).

func Extract

func Extract[T any](response Response) (T, error)

Extract decodes a structured response containing exactly one function call into a value of type T. It does not validate engine warnings; callers that act on the value should call ValidateResponse first.

func FetchEngine

func FetchEngine(ctx context.Context, options FetchOptions) (string, error)

FetchEngine downloads, verifies, and caches the shared library and required base weights for one desktop platform. It returns the library path.

func FetchWhistle

func FetchWhistle(ctx context.Context, options FetchOptions) (string, error)

FetchWhistle caches the Needle 3 engine and Whistle speech weights for offline use. It returns the library path and does not download text weights. Generation must be zero or 3; Platform and CacheDir follow FetchEngine.

func ValidateResponse

func ValidateResponse(response Response) error

ValidateResponse checks only engine validation warnings, accepting absent metadata. It rejects negation or any ungrounded field path, including possible false positives; a matching number elsewhere in the prompt does not exempt a warning. Unflagged output still needs application validation.

func WeightsGeneration

func WeightsGeneration(path string) (int, error)

WeightsGeneration reads a .cact header and returns its model generation. It does not validate the rest of the archive.

Types

type Agent

type Agent struct {
	// contains filtered or unexported fields
}

Agent is a Needle conversation bound to a process-global native runtime. Create one with New. Methods are safe for concurrent use; native calls for one generation are serialized.

func New

func New(ctx context.Context, config Config) (*Agent, error)

New initializes an Agent, downloading the selected engine and any required base weights when they are not cached. Generation 3 is the default.

func (*Agent) Close

func (a *Agent) Close() error

Close releases the agent's native binding and response buffer. Later calls return ErrClosed. Close is idempotent and does not unload the engine or its weights, which remain available to other agents in the process.

func (*Agent) Complete

func (a *Agent) Complete(ctx context.Context, text string, maxNewTokens int) (Response, error)

Complete performs raw inference without applying response validation.

func (*Agent) Embed

func (a *Agent) Embed(ctx context.Context, text string) ([]float32, error)

Embed returns text features from Needle 3 without resetting the conversation. Engines without embedding support return an error.

func (*Agent) Reset

func (a *Agent) Reset(ctx context.Context) error

Reset rewinds the conversation.

func (*Agent) Run

func (a *Agent) Run(ctx context.Context, query string, maxSteps, maxNewTokens int) (Response, error)

Run completes a query, rejects engine validation warnings, and executes accepted tool calls until the model returns a final response. It uses ValidateResponse to reject an entire flagged turn before executing any tool calls, without retrying, and returns the raw flagged response together with results from earlier completed rounds.

type AudioOptions

type AudioOptions struct {
	// Language is empty for detection or en, de, fr, es, it, nl or pl.
	Language       string
	Keywords       []string
	WordTimestamps bool
}

AudioOptions controls language detection, keyword biasing and word timestamps.

type Config

type Config struct {
	Tools  []Tool
	System string
	// Stateless resets the conversation before each Complete or Run request.
	// Run preserves context between tool rounds. The default is stateful; use
	// stateful mode for manual tool loops that continue through Complete.
	Stateless bool
	// Generation selects 2 or 3. Zero defaults to 3; WeightsPath takes precedence.
	Generation int
	// WeightsPath selects custom .cact weights and determines the generation.
	// Custom Needle 3 weights retain confidence when their archive carries a
	// recognized confidence head; other custom weights report nil. The header
	// and model-kind metadata are checked before loading. Remaining model bytes
	// and engine revision compatibility are validated by the native loader.
	WeightsPath   string
	ToolIndexPath string
	BufferSize    int

	// LibraryPath selects a trusted shared library for the selected generation.
	// When empty, New checks NEEDLE2_LIB_PATH or NEEDLE3_LIB_PATH, then fetches
	// the engine. NEEDLE_LIB_PATH is a legacy override for generation 2 only.
	LibraryPath string
	// CacheDir overrides the engine and base-weight download directory.
	CacheDir string
}

Config configures an Agent.

type FetchOptions

type FetchOptions struct {
	// Generation selects 2 or 3; zero defaults to 3.
	Generation int
	Platform   Platform
	CacheDir   string
	Client     *http.Client
}

FetchOptions configures an engine download. An empty Platform selects the current process platform. CacheDir is the exact destination directory.

type FunctionCall

type FunctionCall struct {
	Name      string          `json:"name"`
	Arguments json.RawMessage `json:"arguments"`
}

FunctionCall is a tool invocation selected by the model.

type NativeError

type NativeError struct {
	Operation string
	Code      int32
	Detail    string
}

NativeError reports a failed engine call. Detail carries the engine's diagnostic when one is available.

func (*NativeError) Error

func (e *NativeError) Error() string

type Platform

type Platform string

Platform identifies a downloadable desktop engine build.

const (
	PlatformDarwinAMD64    Platform = "darwin-amd64"
	PlatformDarwinARM64    Platform = "darwin-arm64"
	PlatformLinuxAMD64     Platform = "linux-amd64"
	PlatformLinuxARM64     Platform = "linux-arm64"
	PlatformLinuxAMD64Musl Platform = "linux-amd64-musl"
	PlatformLinuxARM64Musl Platform = "linux-arm64-musl"
	PlatformWindowsAMD64   Platform = "windows-amd64"
	PlatformWindowsARM64   Platform = "windows-arm64"
)

func CurrentPlatform

func CurrentPlatform() (Platform, error)

CurrentPlatform returns the engine build matching the running process.

func SupportedPlatforms

func SupportedPlatforms() []Platform

SupportedPlatforms returns the desktop engine builds known to this version.

type Response

type Response struct {
	Type          ResponseType   `json:"type"`
	Success       bool           `json:"success"`
	Error         *string        `json:"error"`
	ErrorCode     *string        `json:"error_code"`
	FunctionCalls []FunctionCall `json:"function_calls"`
	// SuppressedCalls are withheld by the engine and must not be executed automatically.
	SuppressedCalls []FunctionCall  `json:"suppressed_calls,omitempty"`
	Reasoning       string          `json:"reasoning"`
	Confidence      *float64        `json:"confidence"`
	PrefillTPS      float64         `json:"prefill_tps"`
	DecodeTPS       float64         `json:"decode_tps"`
	PeakRAMMB       float64         `json:"peak_ram_mb"`
	Results         []any           `json:"results,omitempty"`
	Validation      *Validation     `json:"validation,omitempty"`
	AudioText       *string         `json:"audio_text,omitempty"`
	AudioLanguage   *string         `json:"audio_language,omitempty"`
	AudioWords      []WordTimestamp `json:"audio_words,omitempty"`
	AudioTTFTMS     *float64        `json:"audio_ttft_ms,omitempty"`
	AudioDecodeTPS  *float64        `json:"audio_decode_tps,omitempty"`
}

Response is the structured response envelope returned by Needle.

type ResponseType

type ResponseType string

ResponseType identifies the action returned by Needle.

const (
	ResponseCall    ResponseType = "call"
	ResponseRespond ResponseType = "respond"
	ResponseRefuse  ResponseType = "refuse"
	ResponseText    ResponseType = "text"
)

type Stream

type Stream struct {
	// contains filtered or unexported fields
}

Stream is a live transcription started by Whistle.Stream. Feed about one second of PCM per Process call and finish with Stop, which commits the tail. The engine holds one stream per process: while a stream is open, speech calls other than its own Process and Stop return ErrStreamActive. A failed Process ends the stream. Closing the Whistle also ends it.

func (*Stream) Process

func (s *Stream) Process(ctx context.Context, pcm []float32) (StreamTranscript, error)

Process appends PCM to the stream and returns the words it committed.

func (*Stream) Stop

func (s *Stream) Stop(ctx context.Context) (StreamTranscript, error)

Stop ends the stream and returns the words committed from its tail. A stopped stream cannot be resumed; start another with Whistle.Stream.

type StreamTranscript

type StreamTranscript struct {
	Text     string          `json:"text"`
	Words    []WordTimestamp `json:"words"`
	Pending  string          `json:"pending"`
	Language string          `json:"language"`
	Received float64         `json:"received"`
	PassMS   float64         `json:"pass_ms"`
}

StreamTranscript is one step of live transcription. Text and Words are the words the step committed, with times in seconds from the start of the stream, and Pending is the unconfirmed tail. Received counts the seconds of audio so far and PassMS the time the step took. Join the texts of consecutive steps with a space.

type Tool

type Tool struct {
	Schema  ToolSchema
	Handler ToolHandler
}

Tool pairs a JSON schema with an optional Go implementation. A tool without a handler can be used with Complete, while Run reports it as unknown if the model selects it.

func NewTool

func NewTool[Arguments, Result any](
	name string,
	description string,
	handler func(context.Context, Arguments) (Result, error),
) Tool

NewTool builds a Tool whose parameter schema is derived from Arguments. Model-produced JSON arguments are decoded before the typed handler is called.

A nil handler produces a schema-only tool suitable for use with Complete.

type ToolHandler

type ToolHandler func(context.Context, json.RawMessage) (any, error)

ToolHandler executes a model-selected function call.

type ToolSchema

type ToolSchema struct {
	Name        string         `json:"name"`
	Description string         `json:"description,omitempty"`
	Parameters  map[string]any `json:"parameters"`
	// Triggers opt into native regex routing and can bypass the confidence floor.
	Triggers []string `json:"triggers,omitempty"`
}

ToolSchema describes a function that the model may call.

func SchemaFor

func SchemaFor[T any](name, description string) ToolSchema

SchemaFor builds a tool schema whose parameters are derived from T. JSON field names and optionality follow encoding/json tags; JSON Schema annotations can be supplied with jsonschema tags.

type Transcript

type Transcript struct {
	Text      string          `json:"text"`
	Language  string          `json:"language"`
	Words     []WordTimestamp `json:"words,omitempty"`
	TTFTMS    float64         `json:"ttft_ms"`
	DecodeTPS float64         `json:"decode_tps"`
}

Transcript is a Whistle response. Silence has empty Text and Language.

type Validation

type Validation struct {
	Ungrounded []string `json:"ungrounded"`
	Negation   bool     `json:"negation"`
}

Validation contains warnings produced by the Needle engine.

type Whistle

type Whistle struct {
	// contains filtered or unexported fields
}

Whistle transcribes caller-provided 16 kHz mono float32 PCM in [-1, 1]. Speech and Needle 3 text models share a serialized process-global engine.

func NewWhistle

func NewWhistle(ctx context.Context, config WhistleConfig) (*Whistle, error)

NewWhistle loads speech weights without downloading or initializing text weights.

func (*Whistle) Close

func (w *Whistle) Close() error

Close ends an open stream of this Whistle and releases the response buffer. Later calls return ErrClosed. Loaded speech weights stay in the engine for other Whistle values.

func (*Whistle) Complete

func (w *Whistle) Complete(ctx context.Context, textAgent *Agent, pcm []float32, options AudioOptions, maxNewTokens int) (Response, error)

Complete transcribes PCM and completes a raw Needle 3 turn inside the engine. It returns tool calls without executing handlers or applying ValidateResponse. textAgent must be a Needle 3 Agent created by New that shares this engine.

func (*Whistle) Stream

func (w *Whistle) Stream(options AudioOptions) (*Stream, error)

Stream prepares a live transcription with the language and keywords of options; WordTimestamps is ignored because every step carries word times. Nothing reaches the engine until the first Process.

func (*Whistle) Transcribe

func (w *Whistle) Transcribe(ctx context.Context, pcm []float32, options AudioOptions) (Transcript, error)

Transcribe accepts PCM of any length: up to 30 seconds runs as one pass, and longer audio runs as consecutive 30 second windows with word times counted from the start of the clip. Empty audio and silence return an empty transcript. Calls are serialized with all text and speech native work.

type WhistleConfig

type WhistleConfig struct {
	WeightsPath string
	LibraryPath string
	CacheDir    string
	BufferSize  int
}

WhistleConfig selects speech weights and a Needle 3.1-compatible engine. LibraryPath and CacheDir follow Config. Empty WeightsPath fetches Whistle.

type WordTimestamp

type WordTimestamp struct {
	Word        string  `json:"word"`
	Start       float64 `json:"start"`
	End         float64 `json:"end"`
	Probability float64 `json:"probability"`
}

WordTimestamp describes a recognized word, with times in seconds.

Directories

Path Synopsis
cmd
needlez command
examples
embeddings command
fetch-engine command
manual-loop command
tool-call command
transcribe command
Transcribe a file of little-endian float32 mono samples recorded at 16000 Hz, in one pass or, with -stream, live in one-second chunks.
Transcribe a file of little-endian float32 mono samples recorded at 16000 Hz, in one pass or, with -stream, live in one-second chunks.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL