Documentation
¶
Overview ¶
Package vectorizer provides unified interfaces and implementations for converting text into vector embeddings using different AI providers.
This package abstracts away the differences between various embedding APIs, allowing applications to switch between providers with minimal code changes. It supports both OpenAI and Google AI embedding models with configurable dimensions optimized for RAG (Retrieval-Augmented Generation) systems.
Basic Usage ¶
Create a vectorizer and generate embeddings:
import (
"context"
"log"
"github.com/dmitrymomot/foundation/pkg/vectorizer"
)
func main() {
ctx := context.Background()
// OpenAI implementation with default settings
openaiVectorizer, err := vectorizer.NewOpenAI("your-openai-api-key")
if err != nil {
log.Fatal(err)
}
// Single text embedding
embedding, err := openaiVectorizer.Embed(ctx, "Hello world")
if err != nil {
log.Fatal(err)
}
fmt.Printf("Embedding dimensions: %d\n", len(embedding))
// Batch embeddings (more efficient)
texts := []string{"Hello", "World", "AI"}
embeddings, err := openaiVectorizer.EmbedBatch(ctx, texts)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Generated %d embeddings\n", len(embeddings))
}
Provider Configuration ¶
## OpenAI Configuration
OpenAI supports text-embedding-3-small and text-embedding-3-large models:
// High-quality embeddings with custom dimensions
openaiVectorizer, err := vectorizer.NewOpenAI("api-key",
vectorizer.WithOpenAIModel(vectorizer.OpenAITextEmbedding3Large),
vectorizer.WithOpenAIDimensions(3072), // Maximum quality
vectorizer.WithOpenAIMaxBatchSize(500), // Custom batch size
)
// Cost-optimized embeddings
costOptimized, err := vectorizer.NewOpenAI("api-key",
vectorizer.WithOpenAIModel(vectorizer.OpenAITextEmbedding3Small),
vectorizer.WithOpenAIDimensions(512), // Smaller, cheaper
)
## Google AI Configuration
Google AI supports Gemini API and Vertex AI backends:
// Gemini API (simpler setup) googleVectorizer, err := vectorizer.NewGoogle(ctx, "your-gemini-api-key", vectorizer.WithGoogleModel(vectorizer.GoogleTextEmbedding005), vectorizer.WithGoogleDimensions(768), ) // Vertex AI (enterprise features) vertexVectorizer, err := vectorizer.NewGoogleVertexAI(ctx, "your-project", "us-central1", vectorizer.WithGoogleModel(vectorizer.GoogleTextMultilingualEmbedding002), vectorizer.WithGoogleDimensions(1536), )
Model Capabilities and Dimensions ¶
## OpenAI Models:
- text-embedding-3-small: 512 or 1536 dimensions (default: 1536)
- text-embedding-3-large: 256, 1024, or 3072 dimensions (default: 3072)
## Google Models:
- text-embedding-005: 256, 768, 1536, or 3072 dimensions (default: 768)
- text-multilingual-embedding-002: 256, 768, 1536, or 3072 dimensions (default: 768)
Advanced Usage Examples ¶
## RAG System Implementation
type RAGSystem struct {
vectorizer vectorizer.Vectorizer
store VectorStore // Your vector database
}
func (r *RAGSystem) IndexDocuments(ctx context.Context, docs []string) error {
// Process in batches for efficiency
batchSize := 50
for i := 0; i < len(docs); i += batchSize {
end := min(i+batchSize, len(docs))
batch := docs[i:end]
embeddings, err := r.vectorizer.EmbedBatch(ctx, batch)
if err != nil {
return fmt.Errorf("failed to embed batch: %w", err)
}
if err := r.store.Store(ctx, batch, embeddings); err != nil {
return fmt.Errorf("failed to store embeddings: %w", err)
}
}
return nil
}
## Provider Switching
func createVectorizer(provider string) (vectorizer.Vectorizer, error) {
switch provider {
case "openai":
return vectorizer.NewOpenAI(os.Getenv("OPENAI_API_KEY"))
case "google":
return vectorizer.NewGoogle(ctx, os.Getenv("GOOGLE_API_KEY"))
default:
return nil, fmt.Errorf("unsupported provider: %s", provider)
}
}
Error Handling ¶
The package defines specific error types for robust error handling:
embedding, err := v.Embed(ctx, text)
if err != nil {
switch {
case errors.Is(err, vectorizer.ErrInvalidAPIKey):
// Handle authentication issues
log.Fatal("Invalid API key provided")
case errors.Is(err, vectorizer.ErrRateLimitExceeded):
// Implement retry with backoff
time.Sleep(time.Second * 5)
return v.Embed(ctx, text)
case errors.Is(err, vectorizer.ErrTextTooLong):
// Split or truncate text
return v.Embed(ctx, text[:maxLength])
default:
return nil, fmt.Errorf("embedding failed: %w", err)
}
}
Common error types:
- ErrInvalidAPIKey: Missing or invalid API credentials
- ErrModelNotSupported: Unsupported model specified
- ErrInvalidDimensions: Invalid dimension count for the model
- ErrBatchTooLarge: Batch size exceeds provider limits
- ErrRateLimitExceeded: API rate limit exceeded
- ErrTextTooLong: Input text exceeds token limits
Performance Considerations ¶
## Batch Processing
Always use batch operations when processing multiple texts:
// Efficient: Single API call
embeddings, err := v.EmbedBatch(ctx, texts)
// Inefficient: Multiple API calls
for _, text := range texts {
embedding, err := v.Embed(ctx, text)
// ...
}
## Batch Size Limits
Configure batch sizes based on provider limits and your use case:
- OpenAI: Up to 2048 texts per batch (default: 100)
- Google: Up to 100 texts per batch (default: 100)
## Dimension Selection
Choose dimensions based on your requirements:
- Lower dimensions: Faster processing, less storage, lower costs
- Higher dimensions: Better semantic understanding, higher accuracy
- RAG systems: 768-1536 dimensions provide good balance
- High-precision tasks: Use maximum dimensions available
## Cost Optimization
For cost-sensitive applications:
// Use smaller, cheaper models
economical, err := vectorizer.NewOpenAI("api-key",
vectorizer.WithOpenAIModel(vectorizer.OpenAITextEmbedding3Small),
vectorizer.WithOpenAIDimensions(512), // Minimum dimensions
)
API Rate Limits and Constraints ¶
## OpenAI Limits
- Rate limits: Varies by subscription tier
- Token limits: ~8191 tokens per request
- Batch size: Up to 2048 texts
## Google AI Limits
- Rate limits: Varies by quota settings
- Token limits: Model-dependent
- Batch size: Up to 100 texts
## Best Practices
- Implement exponential backoff for rate limit errors
- Monitor API usage and costs
- Cache embeddings when possible
- Use appropriate batch sizes for your workload
- Consider using multiple API keys for higher throughput
Index ¶
- Constants
- Variables
- type Google
- type GoogleOption
- func WithGoogleBackend(backend genai.Backend) GoogleOption
- func WithGoogleDimensions(dims int) GoogleOption
- func WithGoogleLocation(location string) GoogleOption
- func WithGoogleMaxBatchSize(size int) GoogleOption
- func WithGoogleModel(model string) GoogleOption
- func WithGoogleProject(project string) GoogleOption
- type OpenAI
- type OpenAIOption
- type Vectorizer
Constants ¶
const ( GoogleTextEmbedding005 = "text-embedding-005" GoogleTextMultilingualEmbedding002 = "text-multilingual-embedding-002" )
Google model constants.
const ( OpenAITextEmbedding3Small = "text-embedding-3-small" OpenAITextEmbedding3Large = "text-embedding-3-large" )
OpenAI model constants.
Variables ¶
var ( // ErrInvalidDimensions indicates invalid dimensions for the model. ErrInvalidDimensions = errors.New("invalid dimensions for model") // ErrModelNotSupported indicates the model is not supported. ErrModelNotSupported = errors.New("model not supported") // ErrRateLimitExceeded indicates the API rate limit was exceeded. ErrRateLimitExceeded = errors.New("rate limit exceeded") // ErrTextTooLong indicates the input text exceeds the token limit. ErrTextTooLong = errors.New("input text exceeds token limit") // ErrBatchTooLarge indicates the batch size exceeds the limit. ErrBatchTooLarge = errors.New("batch size exceeds limit") // ErrInvalidAPIKey indicates an invalid or missing API key. ErrInvalidAPIKey = errors.New("invalid or missing API key") // ErrEmbeddingFailed indicates a failure in creating embeddings. ErrEmbeddingFailed = errors.New("failed to create embedding") // ErrNoEmbeddingReturned indicates no embedding was returned by the API. ErrNoEmbeddingReturned = errors.New("no embedding returned") // ErrEmbeddingCountMismatch indicates the number of embeddings returned doesn't match the input. ErrEmbeddingCountMismatch = errors.New("embedding count mismatch") // ErrEmptyEmbedding indicates an empty embedding was returned. ErrEmptyEmbedding = errors.New("empty embedding returned") // ErrClientCreationFailed indicates a failure in creating the API client. ErrClientCreationFailed = errors.New("failed to create API client") )
Functions ¶
This section is empty.
Types ¶
type Google ¶
type Google struct {
// contains filtered or unexported fields
}
Google implements the Vectorizer interface using Google's Generative AI API.
func NewGoogle ¶
NewGoogle creates a new Google vectorizer with Gemini API and API key authentication.
func NewGoogleVertexAI ¶
func NewGoogleVertexAI(ctx context.Context, project, location string, opts ...GoogleOption) (*Google, error)
NewGoogleVertexAI creates a new Google vectorizer using Vertex AI with project and location.
func (*Google) Dimensions ¶
Dimensions returns the vector size this implementation produces.
type GoogleOption ¶
type GoogleOption func(*Google)
GoogleOption is a functional option for configuring Google.
func WithGoogleBackend ¶
func WithGoogleBackend(backend genai.Backend) GoogleOption
WithGoogleBackend sets the backend to use (Gemini API or Vertex AI).
func WithGoogleDimensions ¶
func WithGoogleDimensions(dims int) GoogleOption
WithGoogleDimensions sets the output dimensions for the embeddings. Supported values: 256, 768, 1536, 3072.
func WithGoogleLocation ¶
func WithGoogleLocation(location string) GoogleOption
WithGoogleLocation sets the GCP location/region for Vertex AI.
func WithGoogleMaxBatchSize ¶
func WithGoogleMaxBatchSize(size int) GoogleOption
WithGoogleMaxBatchSize sets the maximum batch size for batch operations.
func WithGoogleModel ¶
func WithGoogleModel(model string) GoogleOption
WithGoogleModel sets the model to use.
func WithGoogleProject ¶
func WithGoogleProject(project string) GoogleOption
WithGoogleProject sets the GCP project ID for Vertex AI.
type OpenAI ¶
type OpenAI struct {
// contains filtered or unexported fields
}
OpenAI implements the Vectorizer interface using OpenAI's API.
func NewOpenAI ¶
func NewOpenAI(apiKey string, opts ...OpenAIOption) (*OpenAI, error)
NewOpenAI creates a new OpenAI vectorizer.
func (*OpenAI) Dimensions ¶
Dimensions returns the vector size this implementation produces.
type OpenAIOption ¶
type OpenAIOption func(*OpenAI)
OpenAIOption is a functional option for configuring OpenAI.
func WithOpenAIDimensions ¶
func WithOpenAIDimensions(dims int) OpenAIOption
WithOpenAIDimensions sets the output dimensions for the embeddings. Only applicable to text-embedding-3-* models.
func WithOpenAIHTTPClient ¶
func WithOpenAIHTTPClient(client *http.Client) OpenAIOption
WithOpenAIHTTPClient sets a custom HTTP client.
func WithOpenAIMaxBatchSize ¶
func WithOpenAIMaxBatchSize(size int) OpenAIOption
WithOpenAIMaxBatchSize sets the maximum batch size for batch operations.
func WithOpenAIModel ¶
func WithOpenAIModel(model string) OpenAIOption
WithOpenAIModel sets the model to use.
type Vectorizer ¶
type Vectorizer interface {
// Embed converts a single text to vector embedding.
Embed(ctx context.Context, text string) ([]float32, error)
// EmbedBatch converts multiple texts to vector embeddings.
// Returns embeddings in the same order as input texts.
EmbedBatch(ctx context.Context, texts []string) ([][]float32, error)
// Dimensions returns the vector size this implementation produces.
Dimensions() int
}
Vectorizer converts text to embeddings.
Source Files