core

package
v0.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 4, 2026 License: MIT Imports: 20 Imported by: 0

Documentation

Overview

Package core holds the provider contract and the data types shared by every layer of the graycode-router client: adapters, middleware, caching, embeddings, and the client facade itself.

core is a leaf package — it must not import any other graycode-router/client subpackage. The conversation DTOs below are aliases to the canonical eagle/llm definitions; core re-exports them so subpackages share the contract without an import cycle through the facade. The public names remain available as aliases in github.com/GrayCodeAI/graycode-router/client, which is the API consumers should keep importing.

See plans/client-package-decomposition.md for the migration plan.

Index

Constants

View Source 
const (
	// DefaultMaxRetries is the default number of request retries on transient errors.
	DefaultMaxRetries = 3
	// DefaultBaseDelay is the default initial backoff between retries.
	DefaultBaseDelay = 500 * time.Millisecond
	// DefaultMaxDelay is the default maximum backoff cap.
	DefaultMaxDelay = 30 * time.Second
	// DefaultRetryStatusCodes are the HTTP status codes retried by default.
	DefaultRetryStatusCodes = "429,500,502,503,529"
)

Default retry configuration.

View Source 
const (
	// DefaultRequestTimeout is the default overall per-request timeout.
	DefaultRequestTimeout = 120 * time.Second
	// DefaultHandshakeTimeout is the default TLS handshake timeout.
	DefaultHandshakeTimeout = 10 * time.Second
	// DefaultIdleConnTimeout is the default idle connection keep-alive window.
	DefaultIdleConnTimeout = 90 * time.Second
)

Default transport timeouts.

View Source 
const (
	// DefaultCooldownDuration is how long a provider stays "cooling down" after a 429/5xx.
	DefaultCooldownDuration = 5 * time.Second
)

Default cooldown windows.

View Source 
const DefaultTimeout = 10 * time.Minute

DefaultTimeout is the default end-to-end HTTP timeout for provider clients.

View Source 
const (

	// StreamChannelBuffer is the default buffer size for provider event channels.
	StreamChannelBuffer = 256
)

SSE stream constants.

Variables

View Source 
var Version = "dev"

Version is set by the root graycode-router package's init() from the VERSION file (via the client facade's SetVersion). Default is "dev".

Functions

func ApplyGuardrails 

func ApplyGuardrails(ctx context.Context, resp *GraycodeRouterResponse, g *Guardrails) error

ApplyGuardrails runs guardrail checks on the response and applies redactions. This is called by provider Chat methods after receiving the LLM response.

func ApplyRedactions 

func ApplyRedactions(response string, violations []GuardrailViolation) string

ApplyRedactions takes the response text and violations, replacing redacted matches with their redaction markers. Non-redact violations are left intact. Match positions are used directly from the violations (captured during Check) so the correct instance of each match is redacted even when the matched text appears multiple times in the response.

func AudioFormatToMediaType 

func AudioFormatToMediaType(format string) string

AudioFormatToMediaType converts a short audio format string to a full MIME type.

func CloseIdleConnections 

func CloseIdleConnections()

CloseIdleConnections closes idle connections in the shared pool. Call during graceful shutdown.

func DoWithRetry 

func DoWithRetry(ctx context.Context, httpClient *http.Client, req *http.Request, rc RetryConfig, logger *slog.Logger) (*http.Response, error)

DoWithRetry executes an HTTP request with retry logic.

Note: DoWithRetry operates at the transport layer, before formatAPIError constructs *GraycodeRouterError. Structured-error awareness lives in the fallback chain (fallback.go:240-244) where *GraycodeRouterError.IsRetriable() / IsAuthError() drive provider rotation. DoWithRetry only needs the raw transport status code and the underlying network error to decide whether to retry the same request.

func Emit 

func Emit(ctx context.Context, ch chan<- GraycodeRouterStreamEvent, evt GraycodeRouterStreamEvent)

func FormatAPIError 

func FormatAPIError(provider, op string, statusCode int, requestID string, d ProviderErrorDetail, inner error) error

FormatAPIError builds the *GraycodeRouterError used across every provider request path (chat, stream, embeddings). It always includes the provider name, the operation (e.g. "chat", "stream"), the HTTP status, the upstream correlation id (for support tickets), a classified actionable hint when one applies, and the raw detail.

Returning *GraycodeRouterError (rather than a plain fmt.Errorf) lets downstream code use errors.As to dispatch on the structured type — retry middleware checks IsRetriable(), fallback chains check IsRetriable()/IsAuthError(), observability code can pull the provider/status/request-id without re-parsing the message.

inner is the read error from ParseProviderError (when the body could not be read). It is attached via GraycodeRouterError.Err so errors.Is(err, io.EOF) and similar checks succeed; pass nil when the body was read cleanly.

func IsRetriableError 

func IsRetriableError(err error) bool

IsRetriableError determines whether a fallback to the next provider should be attempted. It delegates to types.IsTransient for known error patterns but diverges on unknown errors: where IsTransient is conservative (returns false for unrecognized errors), IsRetriableError is optimistic (returns true).

Rationale: in a fallback chain, trying the next provider is cheap and may succeed even if the current provider failed with an unexpected error type. In contrast, retry middleware (which uses IsTransient) should be conservative to avoid wasting requests on errors that won't resolve with a retry.

*GraycodeRouterError (returned by FormatAPIError and friends) is preferred over the string-based heuristic: it carries the structured status code so the classification is exact rather than regex-parsed.

func NewPooledHTTPClient 

func NewPooledHTTPClient(timeout time.Duration) *http.Client

NewPooledHTTPClient creates an *http.Client with the shared connection pool transport and the given timeout. All providers should use this instead of constructing raw *http.Client literals.

func NormalizeImageSource 

func NormalizeImageSource(src string) (mediaType, data string, isBase64 bool, err error)

NormalizeImageSource turns any of the three image source forms into a canonical representation for provider clients:

  • data:<mediaType>;base64,<data> → returned as base64 (mediaType, data, true)
  • http(s)://… → returned as a pass-through URL ("", url, false)
  • a local filesystem path → read, MIME-sniffed by extension, and base64 encoded → (mediaType, data, true)

It is the single entry point for image handling so the provider clients and hawk no longer each carry their own divergent encoder. Local files and data-URLs are validated against supportedImageMediaTypes; HTTP URLs are left for the provider to fetch (avoiding an SSRF surface inside graycode-router).

func OpenAIImageURL 

func OpenAIImageURL(src string) string

OpenAIImageURL renders an image source into the single-string form the OpenAI-compatible image_url field expects: an http(s) URL or a data URL are passed through; a local file is read and encoded into a data URL. On any normalization error it falls back to the raw input so the request is not dropped (the provider will surface a clearer error than graycode-router can here).

func ParseImageString 

func ParseImageString(img string) (mediaType, data string, isBase64 bool)

ParseImageString is the backward-compatible shim retained for existing call sites. It now routes through NormalizeImageSource so local paths are encoded and formats are validated. On error it falls back to treating the input as a pass-through URL, preserving the previous lenient behavior.

func ParseSSEStream 

func ParseSSEStream(ctx context.Context, body io.ReadCloser, logger *slog.Logger) <-chan SSEEvent

ParseSSEStream reads an SSE stream and sends events to a channel. The goroutine closes the channel and body when done or context is cancelled. Scanner errors are emitted as SSEEvent with Event="error" so callers can detect truncation.

func ProcessAnthropicStream 

func ProcessAnthropicStream(ctx context.Context, sseEvents <-chan SSEEvent, logger *slog.Logger) <-chan GraycodeRouterStreamEvent

ProcessAnthropicStream converts Anthropic SSE events to GraycodeRouterStreamEvents. Handles text, tool_use (with input_json_delta), and thinking blocks.

func ProcessAnthropicStreamWithOpts 

func ProcessAnthropicStreamWithOpts(ctx context.Context, sseEvents <-chan SSEEvent, logger *slog.Logger, start time.Time) <-chan GraycodeRouterStreamEvent

func ProcessOpenAIStream 

func ProcessOpenAIStream(ctx context.Context, sseEvents <-chan SSEEvent, logger *slog.Logger) <-chan GraycodeRouterStreamEvent

ProcessOpenAIStream converts OpenAI SSE events to GraycodeRouterStreamEvents. Handles text deltas and tool call streaming by index. Also instruments TTFT (time-to-first-token) and runs a RepeatDetector to synthesise a finish_reason:"repeat" event when the stream loops.

func ProcessOpenAIStreamWithOpts 

func ProcessOpenAIStreamWithOpts(ctx context.Context, sseEvents <-chan SSEEvent, logger *slog.Logger, repeat *RepeatDetector, start time.Time) <-chan GraycodeRouterStreamEvent

func ResponseHasContent 

func ResponseHasContent(resp *GraycodeRouterResponse) bool

healthFromResponse classifies a non-streaming GraycodeRouterResponse. Because the non-streaming response shape does not carry a reasoning field today, the caller passes whether reasoning was observed (e.g. from a thinking field on the raw provider payload); when unknown, pass false. ResponseHasContent reports whether a non-streaming response carries usable text.

func SetVersion 

func SetVersion(v string)

SetVersion wires the canonical version into this package.

func UserAgent 

func UserAgent() string

UserAgent returns the User-Agent string for HTTP requests.

Types

type ChatOptions 

type ChatOptions = llm.ChatOptions

ChatOptions holds options for a chat request.

type ClientOption 

type ClientOption struct {
	// contains filtered or unexported fields
}

ClientOption configures clients. Options built with the constructors below apply to any Configurable adapter; options built with NewGraycodeRouterOption apply to the top-level GraycodeRouterClient.

func NewGraycodeRouterOption 

func NewGraycodeRouterOption(fn func(GraycodeRouterConfigurable)) ClientOption

NewGraycodeRouterOption builds a ClientOption from an GraycodeRouterClient-level apply function.

func NewOption 

func NewOption(fn func(Configurable)) ClientOption

NewOption builds a ClientOption from an adapter-level apply function.

func WithAPIKey 

func WithAPIKey(key string) ClientOption

WithAPIKey sets the API key.

func WithBaseURL 

func WithBaseURL(url string) ClientOption

WithBaseURL sets the base URL.

func WithGuardrailType 

func WithGuardrailType(types ...GuardrailType) ClientOption

WithGuardrailType attaches output guardrails using built-in rules for the specified types. For example, WithGuardrailType(GuardrailPII, GuardrailSecretLeak) enables PII redaction and secret leak blocking with default patterns.

func WithGuardrails 

func WithGuardrails(rules ...GuardrailRule) ClientOption

WithGuardrails attaches output guardrails to the client. Guardrails run after the LLM response but before returning to the caller. Blocked responses are replaced with an error; redacted responses have matches replaced with asterisks.

func WithHTTPClient 

func WithHTTPClient(hc *http.Client) ClientOption

WithHTTPClient sets a custom HTTP client.

func WithLogger 

func WithLogger(l *slog.Logger) ClientOption

WithLogger sets the logger.

func WithMaxTokens 

func WithMaxTokens(n int) ClientOption

WithMaxTokens sets the default max tokens for requests.

func WithMimoAuth 

func WithMimoAuth() ClientOption

WithMimoAuth uses api-key header per MiMo documentation (OpenAI + Anthropic compat).

func WithModel 

func WithModel(model string) ClientOption

WithModel sets the default model for requests.

func WithProviderName 

func WithProviderName(name string) ClientOption

WithProviderName sets the OpenAI client provider name for errors/logging. No-op for the Anthropic adapter, which reports a fixed provider name.

func WithRetry 

func WithRetry(rc RetryConfig) ClientOption

WithRetry sets retry configuration.

func WithTemperature 

func WithTemperature(t float64) ClientOption

WithTemperature sets the default temperature for requests.

func WithTimeout 

func WithTimeout(d time.Duration) ClientOption

WithTimeout sets the HTTP client timeout.

func (ClientOption) Apply 

func (o ClientOption) Apply(c Configurable)

Apply runs the option against an adapter. No-op for GraycodeRouterClient-level options.

func (ClientOption) ApplyGraycodeRouter 

func (o ClientOption) ApplyGraycodeRouter(e GraycodeRouterConfigurable)

ApplyGraycodeRouter runs the option against the top-level client. No-op for adapter-level options.

type Configurable 

type Configurable interface {
	SetTimeout(d time.Duration)
	SetHTTPClient(hc *http.Client)
	SetRetry(rc RetryConfig)
	SetLogger(l *slog.Logger)
	SetAPIKey(key string)
	SetBaseURL(url string)
	SetDefaultModel(model string)
	SetDefaultMaxTokens(n int)
	SetDefaultTemperature(t float64)
	SetGuardrails(g *Guardrails)
	SetProviderName(name string)
	SetMimoAuth()
}

Configurable is the setter surface protocol adapters expose to the functional-options system. It decouples ClientOption from concrete adapter types so the adapters can live in their own package.

type ContentPart 

type ContentPart = llm.ContentPart

ContentPart represents a piece of content in a multi-modal message. Use the helper types (TextPart, ImagePart, AudioPart) to construct these.

type ContinuationConfig 

type ContinuationConfig = llm.ContinuationConfig

ContinuationConfig controls output continuation behavior.

func DefaultContinuationConfig 

func DefaultContinuationConfig() ContinuationConfig

DefaultContinuationConfig returns sensible defaults.

type Embedder 

type Embedder interface {
	CreateEmbedding(ctx context.Context, req EmbeddingRequest) (*EmbeddingResponse, error)
}

Embedder is the interface for creating embeddings. It lives in core (rather than client/embeddings) because protocol adapters implement it — keeping the "subpackages import core only" layering rule intact.

type EmbeddingParams 

type EmbeddingParams struct {
	Indexing map[string]string `json:"indexing,omitempty"`
	Query    map[string]string `json:"query,omitempty"`
}

EmbeddingParams holds asymmetric params for indexing vs query.

type EmbeddingRequest 

type EmbeddingRequest struct {
	Model  string            `json:"model"`
	Input  []string          `json:"input"`
	Params map[string]string `json:"params,omitempty"` // indexing or query params
}

EmbeddingRequest represents an embedding API call.

type EmbeddingResponse 

type EmbeddingResponse struct {
	Embeddings [][]float32          `json:"embeddings"`
	Model      string               `json:"model"`
	Usage      *GraycodeRouterUsage `json:"usage,omitempty"`
}

EmbeddingResponse holds embedding results.

type GraycodeRouterConfig 

type GraycodeRouterConfig = llm.GraycodeRouterConfig

GraycodeRouterConfig holds client configuration.

type GraycodeRouterConfigurable 

type GraycodeRouterConfigurable interface {
	SetCoalescingTTL(ttl time.Duration)
}

GraycodeRouterConfigurable is the setter surface the top-level GraycodeRouterClient exposes for options that configure the universal client rather than an adapter.

type GraycodeRouterError 

type GraycodeRouterError struct {
	Provider   string
	Op         string // operation that failed (e.g. "chat", "stream", "ping")
	StatusCode int
	RequestID  string
	Message    string
	Err        error
}

GraycodeRouterError is a structured error that preserves provider context, HTTP metadata, and request identification for debugging.

func (*GraycodeRouterError) Error 

func (e *GraycodeRouterError) Error() string

func (*GraycodeRouterError) IsAuthError 

func (e *GraycodeRouterError) IsAuthError() bool

IsAuthError returns true if the error indicates an authentication/authorization problem.

func (*GraycodeRouterError) IsRateLimited 

func (e *GraycodeRouterError) IsRateLimited() bool

IsRateLimited returns true if the error indicates rate limiting.

func (*GraycodeRouterError) IsRetriable 

func (e *GraycodeRouterError) IsRetriable() bool

IsRetriable returns true if the error is likely transient and retrying may help.

func (*GraycodeRouterError) Unwrap 

func (e *GraycodeRouterError) Unwrap() error

type GraycodeRouterMessage 

type GraycodeRouterMessage = llm.GraycodeRouterMessage

GraycodeRouterMessage represents a chat message. For simple text messages, set Content directly. For multi-modal messages (images, audio), use ContentParts. When ContentParts is non-empty, it takes precedence over Content and Images. The Images field is retained for backward compatibility.

func SanitizeMessages 

func SanitizeMessages(messages []GraycodeRouterMessage) []GraycodeRouterMessage

SanitizeMessages inspects messages for orphaned tool_use blocks (assistant messages with tool calls that lack matching tool_result blocks) and injects synthetic error results to prevent 400 errors from providers. This is critical for session resume and compaction scenarios.

type GraycodeRouterResponse 

type GraycodeRouterResponse = llm.GraycodeRouterResponse

GraycodeRouterResponse is the response from a chat call.

func CopyResponse 

func CopyResponse(resp *GraycodeRouterResponse) *GraycodeRouterResponse

CopyResponse returns a deep copy of an GraycodeRouterResponse so that callers cannot mutate the cached version.

type GraycodeRouterStreamEvent 

type GraycodeRouterStreamEvent = llm.GraycodeRouterStreamEvent

GraycodeRouterStreamEvent is a streaming event.

type GraycodeRouterTool 

type GraycodeRouterTool = llm.GraycodeRouterTool

GraycodeRouterTool represents a tool definition.

type GraycodeRouterUsage 

type GraycodeRouterUsage = llm.GraycodeRouterUsage

GraycodeRouterUsage tracks token usage.

type GuardrailAction 

type GuardrailAction string

GuardrailAction determines what happens when a rule matches. 

const (
	// GuardrailBlock prevents the response from being returned to the caller.
	GuardrailBlock GuardrailAction = "block"
	// GuardrailRedact replaces the matched content with a redaction marker.
	GuardrailRedact GuardrailAction = "redact"
	// GuardrailWarn allows the response but records the violation.
	GuardrailWarn GuardrailAction = "warn"
)

type GuardrailError 

type GuardrailError struct {
	Violations []GuardrailViolation `json:"violations"`
	Message    string               `json:"message"`
}

GuardrailError is returned when a guardrail blocks a response.

func (*GuardrailError) Error 

func (e *GuardrailError) Error() string

type GuardrailRule 

type GuardrailRule struct {
	Type     GuardrailType     `json:"type"`
	Name     string            `json:"name"`
	Pattern  string            `json:"pattern"`
	Action   GuardrailAction   `json:"action"`
	Severity GuardrailSeverity `json:"severity"`
	// contains filtered or unexported fields
}

GuardrailRule defines a single guardrail check.

func AllDefaultRules 

func AllDefaultRules() []GuardrailRule

AllDefaultRules returns all built-in guardrail rules (PII, secrets, prompt injection, and harmful content).

func DefaultHarmfulContentRules 

func DefaultHarmfulContentRules() []GuardrailRule

DefaultHarmfulContentRules returns built-in rules for detecting harmful content patterns.

func DefaultPIIRules 

func DefaultPIIRules() []GuardrailRule

DefaultPIIRules returns built-in rules for detecting PII in responses.

func DefaultPromptInjectionRules 

func DefaultPromptInjectionRules() []GuardrailRule

DefaultPromptInjectionRules returns built-in rules for detecting prompt injection in responses.

func DefaultSecretLeakRules 

func DefaultSecretLeakRules() []GuardrailRule

DefaultSecretLeakRules returns built-in rules for detecting leaked secrets.

func RulesForType 

func RulesForType(t GuardrailType) []GuardrailRule

RulesForType returns the built-in rules for a single GuardrailType.

type GuardrailSeverity 

type GuardrailSeverity string

GuardrailSeverity indicates how critical a violation is. 

const (
	SeverityLow      GuardrailSeverity = "low"
	SeverityMedium   GuardrailSeverity = "medium"
	SeverityHigh     GuardrailSeverity = "high"
	SeverityCritical GuardrailSeverity = "critical"
)

type GuardrailType 

type GuardrailType string

GuardrailType classifies a guardrail rule. 

const (
	// GuardrailPII detects personally identifiable information.
	GuardrailPII GuardrailType = "pii"
	// GuardrailPromptInjection detects prompt injection attempts in responses.
	GuardrailPromptInjection GuardrailType = "prompt_injection"
	// GuardrailHarmfulContent detects harmful or dangerous content patterns.
	GuardrailHarmfulContent GuardrailType = "harmful_content"
	// GuardrailSecretLeak detects leaked secrets, API keys, tokens, and passwords.
	GuardrailSecretLeak GuardrailType = "secret_leak"
	// GuardrailCustom is a user-defined rule with a custom pattern.
	GuardrailCustom GuardrailType = "custom"
)

type GuardrailViolation 

type GuardrailViolation struct {
	Rule           GuardrailRule `json:"rule"`
	MatchedText    string        `json:"matched_text"`
	RedactedResult string        `json:"redacted_result,omitempty"`
	// contains filtered or unexported fields
}

GuardrailViolation records a single rule match in the response.

type Guardrails 

type Guardrails struct {
	// contains filtered or unexported fields
}

Guardrails holds registered rules and runs them against LLM responses.

func NewGuardrails 

func NewGuardrails(rules ...GuardrailRule) *Guardrails

NewGuardrails creates a Guardrails instance with the given rules.

func NewGuardrailsSafe 

func NewGuardrailsSafe(rules ...GuardrailRule) (*Guardrails, error)

NewGuardrailsSafe creates a Guardrails instance and returns an error if any rule has an invalid pattern. Use this when rules may come from untrusted sources; use NewGuardrails for programmatic rules where invalid patterns indicate a programmer error (matching regexp.MustCompile convention).

func (*Guardrails) AddRule 

func (g *Guardrails) AddRule(r GuardrailRule)

AddRule registers a guardrail rule. It panics if the pattern is invalid. This follows the regexp.MustCompile convention for programmatic rules where an invalid pattern indicates a programmer error. For rules that may originate from untrusted sources (config files, user input), use AddRuleSafe instead.

func (*Guardrails) AddRuleSafe 

func (g *Guardrails) AddRuleSafe(r GuardrailRule) error

AddRuleSafe registers a guardrail rule and returns an error if the pattern is invalid, instead of panicking. Use this when rules may come from untrusted sources (config files, user input).

func (*Guardrails) Check 

func (g *Guardrails) Check(ctx context.Context, response string) ([]GuardrailViolation, error)

Check evaluates all rules against the response text. It returns violations and an error only if a rule with Action=Block matches. For Redact actions, the redacted result is populated in the violation. For Warn actions, the violation is recorded but no error is returned.

func (*Guardrails) Rules 

func (g *Guardrails) Rules() []GuardrailRule

Rules returns a snapshot of the currently registered rules.

type ImageURLPart 

type ImageURLPart = llm.ImageURLPart

ImageURLPart represents an image content part. URL can be an HTTP(S) URL or a data URI (data:image/png;base64,...).

type InputAudioPart 

type InputAudioPart = llm.InputAudioPart

InputAudioPart represents an audio content part (base64 encoded).

type Provider 

type Provider interface {
	// Chat sends a non-streaming chat request.
	Chat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*GraycodeRouterResponse, error)
	// StreamChat sends a streaming chat request.
	// The caller must call Close() on the returned StreamResult when done.
	StreamChat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*StreamResult, error)
	// Ping checks connectivity and authentication.
	Ping(ctx context.Context) error
	// Name returns the provider name (e.g. "anthropic", "openai").
	Name() string
}

Provider is the core interface for LLM providers. Implementations must be safe for concurrent use.

type ProviderErrorDetail 

type ProviderErrorDetail struct {
	Message string // human-readable message from the body
	Type    string // provider error type, e.g. "invalid_request_error"
	Code    string // provider error code, e.g. "invalid_api_key", "model_not_found"
	Raw     string // raw body (truncated) when nothing structured was found
}

ProviderErrorDetail holds the structured fields graycode-router can extract from a provider's error body. Providers vary (OpenAI nests under "error", some put a top-level "code"); the parser is lenient and fills what it can.

func ParseProviderError 

func ParseProviderError(body io.ReadCloser) (ProviderErrorDetail, error)

ParseProviderError reads and classifies an error response body. It never returns a zero detail: on a read failure the detail is filled with a placeholder message and the read error is returned alongside so callers can attach it to the structured *GraycodeRouterError.

type RepeatDetector 

type RepeatDetector struct {

	// MinLength is the minimum accumulated rune count before detection fires.
	// Default 100.
	MinLength int
	// Threshold is the repeatness score below which the stream is aborted.
	// Default 0.5.
	Threshold float64
	// contains filtered or unexported fields
}

RepeatDetector detects streaming repetition using a suffix automaton. GetRepeatness returns the ratio of unique substrings to total possible substrings; when it falls below the threshold after enough tokens have accumulated, the stream is considered stuck in a loop.

Algorithm: SuffixAutomaton from moonpalace/detector/repeat (MIT). uniqueSubstrings = sum of (len - link.len) across all states. total possible = n*(n+1)/2 where n = len(accumulated text). A perfectly non-repetitive string scores 1.0; a fully repeated one scores near 0.0.

func DefaultRepeatDetector 

func DefaultRepeatDetector() *RepeatDetector

DefaultRepeatDetector returns a RepeatDetector with the moonpalace defaults.

func (*RepeatDetector) Add 

func (d *RepeatDetector) Add(delta string) bool

Add feeds a content delta. Returns true when repetition is detected and the stream should be aborted.

func (*RepeatDetector) Feed 

func (d *RepeatDetector) Feed(chunk string)

Feed appends a chunk of text. It does not return a detection signal; call IsRepeating or GetRepeatness after feeding to query state.

func (*RepeatDetector) GetRepeatness 

func (d *RepeatDetector) GetRepeatness() float64

GetRepeatness returns the unique-substring ratio in [0, 1]. A score near 1.0 means highly varied text; near 0.0 means heavy repetition.

func (*RepeatDetector) IsRepeating 

func (d *RepeatDetector) IsRepeating() bool

IsRepeating returns true when at least MinLength runes have been accumulated and GetRepeatness is below Threshold.

type ResponseFormat 

type ResponseFormat = llm.ResponseFormat

ResponseFormat specifies the desired output format for the model response.

type ResponseHealth 

type ResponseHealth string

ResponseHealth classifies the outcome of a model response so graycode-router can turn a confusing "the agent did nothing" symptom into a precise, named diagnostic. It is most valuable for reasoning-capable models behind OpenAI-compatible providers, where a misconfiguration commonly yields thinking tokens but no usable answer. 

const (
	// ResponseOK means the response carried usable content and/or tool calls.
	ResponseOK ResponseHealth = "ok"
	// ResponseErrorOnlyReasoning means the model emitted reasoning/thinking
	// tokens but produced zero content and zero tool calls — usually a sign the
	// provider is dropping the post-reasoning answer (wrong thinking-format
	// config, truncated stream, or a reasoning-only model used as a chat model).
	ResponseErrorOnlyReasoning ResponseHealth = "error_only_reasoning"
	// ResponseEmpty means there was no reasoning, no content, and no tool calls.
	ResponseEmpty ResponseHealth = "empty_response"
	// ResponseMalformedStream means the stream ended abnormally (a stream-level
	// error, or no terminal "done"/finish_reason was observed).
	ResponseMalformedStream ResponseHealth = "malformed_stream"
)

func DetectResponseHealth 

func DetectResponseHealth(sig ResponseSignals) ResponseHealth

DetectResponseHealth classifies a completed response. Order matters: a stream-level error dominates, then the reasoning-only case, then plain empty.

func (ResponseHealth) Diagnostic 

func (h ResponseHealth) Diagnostic() string

Diagnostic returns a human-readable, actionable description for a non-OK health value, or "" when the response was OK.

func (ResponseHealth) Err 

func (h ResponseHealth) Err() error

Err returns a non-nil error for a non-OK health value, suitable for surfacing to callers/operators. OK returns nil.

type ResponseSignals 

type ResponseSignals struct {
	SawReasoning bool // any thinking/reasoning tokens were produced
	ContentLen   int  // length of usable assistant content
	ToolCalls    int  // number of tool calls produced
	FinishReason string
	StreamErr    bool // a stream-level error event was seen
	StreamEnded  bool // a terminal done/finish event was observed
}

ResponseSignals are the minimal observations needed to classify health. They are cheap to gather from both the streaming path (counts of thinking/content/ tool-call events) and the non-streaming path (response field lengths).

type RetryConfig 

type RetryConfig struct {
	types.RetryConfig
	RetryOn []int // HTTP status codes to retry on
}

RetryConfig controls retry behavior for HTTP clients. It embeds types.RetryConfig for the core fields and adds RetryOn for HTTP-status-code–driven retry decisions.

func DefaultRetryConfig 

func DefaultRetryConfig() RetryConfig

DefaultRetryConfig returns sensible defaults.

func NewRetryConfig 

func NewRetryConfig(maxRetries int, baseDelay, maxDelay time.Duration, retryOn ...int) RetryConfig

NewRetryConfig constructs a RetryConfig from core fields and optional HTTP status codes to retry on.

func (RetryConfig) ShouldRetry 

func (rc RetryConfig) ShouldRetry(statusCode int) bool

ShouldRetry checks if a status code is retryable.

type SSEEvent 

type SSEEvent struct {
	Event string
	Data  string
}

SSEEvent represents a single Server-Sent Event.

type StreamGuardrailConfig 

type StreamGuardrailConfig struct {
	// Enabled turns streaming guardrails on or off.
	Enabled bool `json:"enabled"`
	// MaxChunkSize is the maximum allowed size of a single chunk (in bytes).
	// Chunks exceeding this limit are split. A value of 0 disables splitting.
	MaxChunkSize int `json:"max_chunk_size,omitempty"`
	// AccumulateForPII buffers chunks before running PII detection, since
	// patterns like SSNs or credit-card numbers may span chunk boundaries.
	AccumulateForPII bool `json:"accumulate_for_pii"`
	// BlockOnInjection causes an immediate block when a prompt-injection
	// pattern is detected in any chunk, without waiting for accumulation.
	BlockOnInjection bool `json:"block_on_injection"`
}

StreamGuardrailConfig controls how streaming guardrails behave.

type StreamGuardrailResult 

type StreamGuardrailResult struct {
	// Chunk is the original chunk text.
	Chunk string `json:"chunk"`
	// Blocked indicates that the chunk (or accumulated content) was blocked
	// by a guardrail with Action=Block.
	Blocked bool `json:"blocked"`
	// Violations lists any rules matched during this chunk's processing.
	Violations []GuardrailViolation `json:"violations,omitempty"`
	// ModifiedChunk is the chunk after redactions have been applied.
	// When no redactions are needed, it equals Chunk.
	ModifiedChunk string `json:"modified_chunk"`
}

StreamGuardrailResult is returned after processing a single chunk.

type StreamGuardrails 

type StreamGuardrails struct {
	// contains filtered or unexported fields
}

StreamGuardrails validates LLM output chunks incrementally as they arrive, rather than waiting for the full response. It is safe for concurrent use.

func NewStreamGuardrails 

func NewStreamGuardrails(guardrails *Guardrails, config StreamGuardrailConfig) *StreamGuardrails

NewStreamGuardrails creates a StreamGuardrails with the given guardrail rules and configuration. guardrails must not be nil.

func (*StreamGuardrails) Flush 

func (sg *StreamGuardrails) Flush() []GuardrailViolation

Flush checks the accumulated buffer for violations that may span chunk boundaries (primarily PII patterns when AccumulateForPII is enabled). It returns all violations found during the entire stream session so far that were not already reported. Call Flush after the last chunk has been processed.

func (*StreamGuardrails) IsBlocked 

func (sg *StreamGuardrails) IsBlocked() bool

IsBlocked reports whether any guardrail with Action=Block has been triggered during the stream session.

func (*StreamGuardrails) ProcessChunk 

func (sg *StreamGuardrails) ProcessChunk(chunk string) StreamGuardrailResult

ProcessChunk validates a single streaming chunk against the registered guardrails. If AccumulateForPII is enabled, the chunk is appended to an internal buffer and PII checks are deferred until Flush. If BlockOnInjection is enabled, prompt-injection rules are evaluated immediately on the chunk.

The returned StreamGuardrailResult contains the (possibly redacted) chunk and any violations found.

func (*StreamGuardrails) Reset 

func (sg *StreamGuardrails) Reset()

Reset clears the internal buffer and accumulated violations.

type StreamResult 

type StreamResult = llm.StreamResult

StreamResult wraps a streaming response with cleanup. Callers must call Close() when done reading events, or cancel the context.

StreamResult is aliased to the canonical contract type; its Close() method and canonical constructor (NewStreamResult) live in github.com/GrayCodeAI/graycode-router/llm.

type ToolCall 

type ToolCall = llm.ToolCall

ToolCall represents a tool invocation.

func ParseInlineToolCalls 

func ParseInlineToolCalls(text string) (cleanText string, toolCalls []ToolCall)

ParseInlineToolCalls detects and extracts tool calls embedded in text content. Three text-embedded formats are recognized, tried in order:

  • Moonshot/kimi (canopywave): <|tool_calls_section_begin|> <|tool_call_begin|> functions.ToolName:0 <|tool_call_argument_begin|> {"arg":"val"} <|tool_call_end|> <|tool_calls_section_end|>
  • Hermes/Nous (Qwen, and most OpenAI-compatible local models served via vLLM/SGLang/Ollama): each call is a JSON object {"name":...,"arguments":{...}} wrapped in <tool_call>...</tool_call> XML tags, with parallel calls emitted as repeated tag pairs. This is the format Qwen2.5/QwQ Emit; Qwen3-Coder uses a structurally similar but distinct "qwen3_xml" variant not handled here.
  • Bare JSON brace-match (last resort): a {"name":...,"arguments":{...}} object found via strings.Index(text, `{"`) / strings.LastIndex(text, "}") without any wrapper tags. Used by some fine-tuned models and direct JSON outputs.

Providers that already surface tool calls through the structured tool_calls channel never reach this path — it is a fallback for models that inline them.

type ToolChoiceOption 

type ToolChoiceOption = llm.ToolChoiceOption

ToolChoiceOption controls how the model uses tools (Anthropic).

type ToolResult 

type ToolResult = llm.ToolResult

ToolResult represents the result of a tool execution.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.