embedding

README

SyncBase vector embedding

고정된 multilingual-E5 ONNX 추론, tokenizer, artifact checksum, 동시성 제한과 모델 다운로드 도구를 소유한다. WAS 도메인 타입에 의존하지 않으며 최소 Profile과 Provider 계약만 공개한다. P0 구현체는 E5 하나이며, Worker는 passage embedding, MCP는 query embedding에 같은 immutable profile을 사용한다.

go test ./...
ops/model/fetch-e5-small.sh /absolute/model-dir
ops/model/fetch-onnxruntime.sh /absolute/runtime-dir linux-amd64

License

SyncBase vector embedding의 자체 소스는 Apache License 2.0 (Apache-2.0)으로 배포합니다. multilingual-e5-small, ONNX Runtime, tokenizer 구현체 등 외부 구성요소는 각자의 라이선스를 따르며 세부 출처는 THIRD_PARTY_NOTICES.md에 기록합니다.

Documentation

Overview

Package embedding adapts the pinned multilingual-E5 ONNX model to SyncBase.

Index

• Constants
• Variables
• type Config
• type E5
  • func New(config Config) (*E5, error)
  • func (e *E5) Close() error
  • func (e *E5) CountTokens(text string) (int, error)
  • func (e *E5) EmbedPassages(ctx context.Context, passages []string, profile Profile) ([][]float32, error)
  • func (e *E5) EmbedQuery(ctx context.Context, query string, profile Profile) ([]float32, error)
  • func (e *E5) Ready(ctx context.Context) error
• type Profile
• type Provider

Constants

const (

	// ProviderLocalONNX identifies the only P0 embedding implementation.
	ProviderLocalONNX = "local-onnx"
	// ChunkSizeTokens is the target passage size for the pinned E5 profile.
	ChunkSizeTokens = 384
	// ChunkOverlapTokens is retained when a long sentence needs forced splitting.
	ChunkOverlapTokens = 64
	// VectorDimension is the only vector size supported by the pinned E5 model.
	VectorDimension = 384
)

Variables

var (
	// ErrInvalidArgument reports invalid text or artifact configuration.
	ErrInvalidArgument = errors.New("invalid argument")
	// ErrProfileMismatch reports an incompatible model or vector contract.
	ErrProfileMismatch = errors.New("embedding profile mismatch")
	// ErrTemporarilyUnavailable reports bounded local inference saturation.
	ErrTemporarilyUnavailable = errors.New("embedding temporarily unavailable")
)

Types

type Config

type Config struct {
	ModelPath          string
	ModelSHA256        string
	TokenizerPath      string
	TokenizerSHA256    string
	RuntimeLibraryPath string
	RuntimeSHA256      string
}

Config identifies the pinned model, tokenizer, and ONNX Runtime artifacts.

type E5

type E5 struct {
	// contains filtered or unexported fields
}

E5 owns one ONNX Runtime session that creates deterministic query and passage vectors. Worker and MCP processes use separate instances.

func New

func New(config Config) (*E5, error)

New verifies pinned artifacts and initializes the bounded ONNX runtime.

func (*E5) Close

func (e *E5) Close() error

Close releases the ONNX runtime and model resources.

func (*E5) CountTokens

func (e *E5) CountTokens(text string) (int, error)

CountTokens returns the pinned tokenizer's encoded length for one passage.

func (*E5) EmbedPassages

func (e *E5) EmbedPassages(ctx context.Context, passages []string, profile Profile) ([][]float32, error)

EmbedPassages creates one normalized vector per passage for the active profile.

func (*E5) EmbedQuery

func (e *E5) EmbedQuery(ctx context.Context, query string, profile Profile) ([]float32, error)

EmbedQuery creates one normalized query vector for the active profile.

func (*E5) Ready

func (e *E5) Ready(ctx context.Context) error

Ready reports whether the pinned tokenizer and ONNX session can accept work.

type Profile

type Profile struct {
	Provider           string
	EmbeddingModelID   string
	VectorDimension    int
	Distance           string
	ChunkSizeTokens    int
	ChunkOverlapTokens int
}

Profile is the immutable processing contract required by the E5 engine. The provider validates every field before inference so a query can never use a model contract different from the one that produced stored chunks.

type Provider

type Provider interface {
	EmbedQuery(context.Context, string, Profile) ([]float32, error)
	EmbedPassages(context.Context, []string, Profile) ([][]float32, error)
	CountTokens(string) (int, error)
	Ready(context.Context) error
	Close() error
}

Provider creates query and passage vectors for one immutable embedding profile. Callers own the provider lifecycle and must call Close when they no longer need local inference resources.