llm

package
v0.5.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 3, 2026 License: MIT Imports: 11 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func GetAllMetrics added in v0.4.0 

func GetAllMetrics() map[string]MetricsStats

GetAllMetrics returns metrics for all providers

func RegisterMetrics added in v0.4.0 

func RegisterMetrics(providerName string, metrics *Metrics)

RegisterMetrics registers metrics for a provider

Types

type CallingAIProvider added in v0.4.0 

type CallingAIProvider struct {
	// contains filtered or unexported fields
}

CallingAIProvider is an LLM provider for MCP mode that simulates responses

ARCHITECTURE NOTE: In MCP mode, the calling AI (Claude, Codex, Gemini) is already a capable LLM. Rather than requiring a separate LLM backend, we provide simplified behavior:

- CoVe: Returns facts without verification (calling AI is reliable)

For advanced use cases requiring full CoVe in MCP mode: 1. Configure an external LLM (Ollama) - use external.Client

This provider exists to make CoVe "work" in MCP mode without external deps, but with reduced functionality compared to Proxy mode with full LLM backend.

func NewCallingAIProvider added in v0.4.0 

func NewCallingAIProvider() *CallingAIProvider

NewCallingAIProvider creates a new CallingAI provider for MCP mode

func (*CallingAIProvider) ChatCompletions added in v0.4.0 

func (p *CallingAIProvider) ChatCompletions(ctx context.Context, req ChatCompletionRequest) (*ChatCompletionResponse, error)

ChatCompletions in CallingAI mode returns a pass-through response For CoVe: Returns a simple "verified" response without actual verification

This is intentionally simplified. For full CoVe functionality in MCP mode, users should configure an external LLM or wait for embedded text generation support.

func (*CallingAIProvider) Close added in v0.4.0 

func (p *CallingAIProvider) Close() error

Close releases any resources (no resources to release)

func (*CallingAIProvider) GetMetrics added in v0.4.0 

func (p *CallingAIProvider) GetMetrics() *Metrics

GetMetrics returns the current metrics for this provider

func (*CallingAIProvider) ProviderName added in v0.4.0 

func (p *CallingAIProvider) ProviderName() string

ProviderName returns the name of this provider

type ChatCompletionRequest 

type ChatCompletionRequest struct {
	Model       string    `json:"model"`
	Messages    []Message `json:"messages"`
	Stream      bool      `json:"stream"`
	Temperature float64   `json:"temperature,omitempty"`
	TopP        float64   `json:"top_p,omitempty"`
}

ChatCompletionRequest represents a request to the chat completion API

type ChatCompletionResponse 

type ChatCompletionResponse struct {
	ID      string   `json:"id"`
	Object  string   `json:"object"`
	Created int64    `json:"created"`
	Model   string   `json:"model"`
	Choices []Choice `json:"choices"`
}

ChatCompletionResponse represents a response from the chat completion API

type Choice 

type Choice struct {
	Index        int     `json:"index"`
	Message      Message `json:"message"`
	FinishReason *string `json:"finish_reason,omitempty"`
}

Choice represents a choice in the chat completion response

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client handles communication with LLM services

func NewClient 

func NewClient(cfg *config.Config) *Client

NewClient creates a new LLM client

func NewClientWithConfig 

func NewClientWithConfig(baseURL, apiKey string) *Client

NewClientWithConfig creates a new LLM client with proper configuration

func (*Client) ChatCompletions 

func (c *Client) ChatCompletions(ctx context.Context, req ChatCompletionRequest) (*ChatCompletionResponse, error)

ChatCompletions sends a chat completion request

func (*Client) ChatCompletionsRaw 

func (c *Client) ChatCompletionsRaw(ctx context.Context, req ChatCompletionRequest) (*http.Response, error)

ChatCompletionsRaw sends a chat completion request and returns the raw HTTP response.

func (*Client) Close added in v0.4.0 

func (c *Client) Close() error

Close releases any resources (HTTP client has no resources to release)

func (*Client) ConvertStreamToText 

func (c *Client) ConvertStreamToText(ctx context.Context, chunks <-chan StreamChunk, errors <-chan error) (<-chan string, <-chan error)

ConvertStreamToText converts a stream of chunks to text

func (*Client) GetMetrics added in v0.4.0 

func (c *Client) GetMetrics() *Metrics

GetMetrics returns the current metrics for this provider

func (*Client) ProviderName added in v0.4.0 

func (c *Client) ProviderName() string

ProviderName returns the name of this provider

func (*Client) StreamChatCompletions 

func (c *Client) StreamChatCompletions(ctx context.Context, req ChatCompletionRequest) (<-chan StreamChunk, <-chan error)

StreamChatCompletions sends a streaming chat completion request

type Message 

type Message struct {
	Role    string `json:"role"`
	Content string `json:"content"`
}

Message represents a chat message

type Metrics added in v0.4.0 

type Metrics struct {
	// Counters
	TotalRequests   atomic.Int64
	SuccessfulCalls atomic.Int64
	FailedCalls     atomic.Int64

	// Token usage
	TotalTokens      atomic.Int64
	PromptTokens     atomic.Int64
	CompletionTokens atomic.Int64

	// Provider info
	ProviderName string
	StartTime    time.Time
	// contains filtered or unexported fields
}

Metrics tracks LLM provider performance and usage

func GetMetrics added in v0.4.0 

func GetMetrics(providerName string) *Metrics

GetMetrics retrieves metrics for a provider

func NewMetrics added in v0.4.0 

func NewMetrics(providerName string) *Metrics

NewMetrics creates a new metrics tracker

func (*Metrics) GetStats added in v0.4.0 

func (m *Metrics) GetStats() MetricsStats

GetStats returns current statistics

func (*Metrics) RecordRequest added in v0.4.0 

func (m *Metrics) RecordRequest(success bool, latencyMs int64, tokens int)

RecordRequest records a request with its outcome

type MetricsStats added in v0.4.0 

type MetricsStats struct {
	ProviderName     string
	TotalRequests    int64
	SuccessfulCalls  int64
	FailedCalls      int64
	SuccessRate      float64 // Percentage
	AvgLatencyMs     int64
	MinLatencyMs     int64
	MaxLatencyMs     int64
	TotalTokens      int64
	PromptTokens     int64
	CompletionTokens int64
	Uptime           time.Duration
}

MetricsStats holds snapshot of metrics

type Provider added in v0.4.0 

type Provider interface {
	// ChatCompletions sends a chat completion request and returns a response
	ChatCompletions(ctx context.Context, req ChatCompletionRequest) (*ChatCompletionResponse, error)

	// ProviderName returns the name of the provider (for logging/debugging)
	ProviderName() string

	// GetMetrics returns the current metrics for this provider
	GetMetrics() *Metrics

	// Close releases any resources held by the provider
	Close() error
}

Provider defines the interface for LLM providers Implementations include External (HTTP), CallingAI (MCP mode), and Local (embedded)

type StreamChoice 

type StreamChoice struct {
	Index        int     `json:"index"`
	Delta        Message `json:"delta"`
	FinishReason *string `json:"finish_reason,omitempty"`
}

StreamChoice represents a choice in a streaming response

type StreamChunk 

type StreamChunk struct {
	ID      string         `json:"id"`
	Object  string         `json:"object"`
	Created int64          `json:"created"`
	Model   string         `json:"model"`
	Choices []StreamChoice `json:"choices"`
}

StreamChunk represents a chunk in a streaming response

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.