README
¶
core
The foundation types every other package builds on: documents, chunks, the
Value metadata type, and sentinel errors. No dependencies outside the
standard library.
Types
| Type | Purpose |
|---|---|
Document |
A source document (ID, title, source, namespace, metadata). Create with NewDocument(id, title, source). |
Chunk |
A chunked piece of a document with its embedding, index, and metadata. |
Value |
A typed JSON-like metadata value (String, Number, Boolean, URI, Literal). Convert with ToString / ToBool / ToFloat64. |
ErrNotFound, ErrInvalidDocument, ErrInvalidChunk |
Sentinel errors; match with errors.Is. |
Minimal example
doc := core.NewDocument("doc-1", "My Doc", "file.txt")
doc.Metadata["lang"] = core.String("en")
if v, ok := core.ToString(doc.Metadata["lang"]); ok {
fmt.Println(v) // en
}
Used by
chunker, embedder, index, store, loader, and the rest of the
pipeline all exchange core.Document / core.Chunk values.
Documentation
¶
Overview ¶
Package core provides the fundamental data types for the recall knowledge store.
Index ¶
Constants ¶
const MetadataKeyNamespace = "namespace"
MetadataKeyNamespace is the chunk metadata key under which stores record the namespace a chunk was uploaded into (see store.MemoryStore and store.SQLiteStore Upload). Services that enforce namespace-scoped credentials (e.g. api.ScopedAPIKeyAuth) filter chunks by this key.
Variables ¶
var ( // ErrNotFound is returned when a requested entity does not exist. ErrNotFound = errors.New("recall: not found") // ErrDuplicate is returned when attempting to add a duplicate entity. ErrDuplicate = errors.New("recall: duplicate") // ErrInvalidChunk is returned when a chunk has invalid content. ErrInvalidChunk = errors.New("recall: invalid chunk") // ErrInvalidDocument is returned when a document is missing or invalid. ErrInvalidDocument = errors.New("recall: invalid document") // ErrInvalidEmbedding is returned when an embedding has invalid dimensions. ErrInvalidEmbedding = errors.New("recall: invalid embedding") // ErrEmbeddingMismatch is returned when embedding dimensions don't match the index. ErrEmbeddingMismatch = errors.New("recall: embedding dimension mismatch") // ErrNamespaceNotFound is returned when a namespace does not exist. ErrNamespaceNotFound = errors.New("recall: namespace not found") // ErrNamespaceExists is returned when a namespace already exists. ErrNamespaceExists = errors.New("recall: namespace exists") // ErrEmptyQuery is returned when a query has no search criteria. ErrEmptyQuery = errors.New("recall: empty query") // ErrContextCancelled is returned when the context is cancelled during a long operation. ErrContextCancelled = errors.New("recall: context cancelled") )
Functions ¶
Types ¶
type Chunk ¶
type Chunk struct {
// ID is a unique identifier for this chunk.
ID string
// Content is the text content of this chunk.
Content string
// DocumentRef identifies the source document this chunk belongs to.
DocumentRef string
// ChunkIndex is the zero-based index of this chunk within its document.
ChunkIndex int
// Embedding is the vector embedding of this chunk's content.
// May be nil if the chunk has not yet been embedded.
Embedding []float32
// Metadata contains arbitrary key-value metadata about this chunk.
// Common keys include: source, author, date, tags, title.
Metadata map[string]Value
// CreatedAt is when this chunk was added to the store.
CreatedAt time.Time
// UpdatedAt is when this chunk was last updated.
UpdatedAt time.Time
}
Chunk represents a unit of text content with associated metadata. Chunks are the primary data unit for retrieval in RAG pipelines.
func (*Chunk) EmbeddingDimension ¶
EmbeddingDimension returns the dimension of this chunk's embedding.
func (*Chunk) GetMetadata ¶
GetMetadata returns the value for a metadata key, or nil if not present.
func (*Chunk) GetMetadataString ¶
GetMetadataString returns the string value for a metadata key.
func (*Chunk) HasEmbedding ¶
HasEmbedding returns true if this chunk has a non-nil embedding.
type Document ¶
type Document struct {
// ID is a unique identifier for this document.
ID string
// Title is a human-readable title for the document.
Title string
// Author is the author or creator of the document.
Author string
// Source is the source location (file path, URL, etc.) of the document.
Source string
// Namespace optionally overrides the store's default namespace for this
// document's chunks. When empty, the store decides (its configured
// namespace). Supported by the Memory and SQLite stores; search spans
// all namespaces present in a store.
Namespace string
// Tags are arbitrary labels for categorizing the document.
Tags []string
// Metadata contains additional arbitrary metadata.
Metadata map[string]Value
// CreatedAt is when the document was first added to the store.
CreatedAt time.Time
// UpdatedAt is when the document was last updated.
UpdatedAt time.Time
// ChunkCount is the number of chunks this document has been split into.
ChunkCount int
}
Document represents a source document that can be chunked and stored.
func NewDocument ¶
NewDocument creates a new Document with the given ID and content metadata.
type Literal ¶
type Literal struct {
Value string
}
Literal wraps an arbitrary string literal value.
type Value ¶
type Value interface {
// Kind returns the type kind of this value.
Kind() ValueKind
// String returns the string representation of this value.
String() string
}
Value represents a typed value that can be stored in the knowledge store. It supports strings, numbers, booleans, URIs, and arbitrary literals.
Source Files