documents

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 15, 2026 License: MIT Imports: 22 Imported by: 0

Documentation

Overview

Package documents implements Aura's document ingestion domain. Postgres owns job state, Neo4j owns searchable document/chunk data, and extractor sidecars own file-format parsing.

Index

Constants

View Source 
const DefaultMaxIngestBytes int64 = 50 << 20

DefaultMaxIngestBytes is the fallback per-file ingestion size ceiling.

Variables

View Source 
var ErrFileTooLarge = errors.New("document file too large")

ErrFileTooLarge is returned when a file exceeds the configured ingest ceiling.

Functions

func ChunkHash 

func ChunkHash(text string, locator Locator) (string, error)

ChunkHash derives a stable hash from normalized text and source locator.

func ChunkID 

func ChunkID(documentID string, index int) string

ChunkID derives a stable chunk id from a document id and chunk index.

func ContentHashPath 

func ContentHashPath(path string) (string, error)

ContentHashPath returns the SHA-256 hex digest for a local file path.

func ContentHashReader 

func ContentHashReader(r io.Reader) (string, error)

ContentHashReader returns the SHA-256 hex digest for bytes read from r.

func DocumentID 

func DocumentID(contentHash, sourceID string) string

DocumentID derives Aura's stable document id from content and source identity.

func NormalizeText 

func NormalizeText(text string) string

NormalizeText collapses internal whitespace and trims document text.

Types

type Chunk 

type Chunk struct {
	ID          string
	DocumentID  string
	SourceID    string
	ContentHash string
	ChunkHash   string
	ChunkIndex  int
	ChunkCount  int
	Kind        string
	Text        string
	Locator     Locator
	HeadingPath []string
}

Chunk is one searchable text unit derived from an extracted document.

type Clock 

type Clock func() time.Time

Clock returns the current time; tests inject it for deterministic timestamps.

type CreateJobParams 

type CreateJobParams struct {
	SourceID     string
	SourceKind   string
	DocumentID   string
	ContentHash  string
	OriginalPath string
	FileName     string
	MIMEType     string
	SizeBytes    int64
	Status       JobStatus
}

CreateJobParams carries the fields needed to create a document ingestion job.

type EmbedQueue 

type EmbedQueue interface {
	Enqueue(ctx context.Context, doc ExtractedDocument) error
}

EmbedQueue accepts extracted documents for asynchronous embedding.

type EmbeddedChunk 

type EmbeddedChunk struct {
	ID        string
	Embedding []float64
}

EmbeddedChunk carries a chunk id with its generated vector embedding.

type EmbeddingClient 

type EmbeddingClient struct {
	BaseURL    string
	Model      string
	Client     *http.Client
	Dimensions int
}

EmbeddingClient calls Aura's OpenAI-compatible local embedding sidecar.

func (*EmbeddingClient) Embed 

func (c *EmbeddingClient) Embed(ctx context.Context, texts []string) ([][]float64, error)

Embed generates embeddings for texts and validates the configured dimensions.

type EmbeddingGenerator 

type EmbeddingGenerator interface {
	Embed(ctx context.Context, texts []string) ([][]float64, error)
}

EmbeddingGenerator generates vector embeddings for document chunk text.

type EmbeddingIndexer 

type EmbeddingIndexer interface {
	UpsertEmbeddings(ctx context.Context, documentID string, chunks []EmbeddedChunk, status JobStatus, embeddedCount int) (int, error)
}

EmbeddingIndexer stores generated embeddings for indexed chunks.

type EmbeddingWorker 

type EmbeddingWorker struct {
	Jobs       JobStore
	Generator  EmbeddingGenerator
	Indexer    EmbeddingIndexer
	BatchSize  int
	MaxRetries int
	Backoff    time.Duration
}

EmbeddingWorker asynchronously embeds document chunks after sparse indexing.

func (*EmbeddingWorker) Enqueue 

func (w *EmbeddingWorker) Enqueue(ctx context.Context, doc ExtractedDocument) error

Enqueue starts background embedding for an extracted document.

func (*EmbeddingWorker) Process 

func (w *EmbeddingWorker) Process(ctx context.Context, doc ExtractedDocument) error

Process embeds and indexes one extracted document synchronously.

type ExtractClient 

type ExtractClient struct {
	BaseURL string
	Client  *http.Client
}

ExtractClient posts local document files to the markitdown extractor sidecar.

func (*ExtractClient) ExtractFile 

func (c *ExtractClient) ExtractFile(ctx context.Context, path string, req IngestRequest) (*ExtractorResponse, error)

ExtractFile streams path to the extractor sidecar and decodes normalized chunks.

type ExtractedChunk 

type ExtractedChunk struct {
	Kind        string   `json:"kind"`
	Text        string   `json:"text"`
	Locator     Locator  `json:"locator"`
	HeadingPath []string `json:"heading_path"`
}

ExtractedChunk is one raw chunk returned by the extractor sidecar.

type ExtractedDocument 

type ExtractedDocument struct {
	ID          string
	SourceID    string
	SourceKind  string
	FileName    string
	MIMEType    string
	SizeBytes   int64
	ContentHash string
	Title       string
	Chunks      []Chunk
	CreatedAt   time.Time
}

ExtractedDocument is the normalized, chunked representation sent to indexing.

func BuildExtractedDocument 

func BuildExtractedDocument(req IngestRequest, contentHash string, resp *ExtractorResponse, createdAt time.Time) (ExtractedDocument, error)

BuildExtractedDocument converts extractor output into Aura's indexed document model.

type Extractor 

type Extractor interface {
	ExtractFile(ctx context.Context, path string, req IngestRequest) (*ExtractorResponse, error)
}

Extractor converts a source file into normalized document chunks.

type ExtractorResponse 

type ExtractorResponse struct {
	Title    string           `json:"title"`
	MIMEType string           `json:"mime_type"`
	Stats    ExtractorStats   `json:"stats"`
	Chunks   []ExtractedChunk `json:"chunks"`
}

ExtractorResponse is the normalized JSON payload returned by the extractor sidecar.

type ExtractorStats 

type ExtractorStats struct {
	Pages      int `json:"pages,omitempty"`
	Sheets     int `json:"sheets,omitempty"`
	Sections   int `json:"sections,omitempty"`
	Chunks     int `json:"chunks,omitempty"`
	Characters int `json:"characters,omitempty"`
}

ExtractorStats summarizes the structure and size of an extractor response.

type Indexer 

type Indexer struct {
	Client    KnowledgeClient
	BatchSize int
}

Indexer writes extracted documents, chunks, and embeddings into Neo4j.

func (*Indexer) UpsertEmbeddings 

func (i *Indexer) UpsertEmbeddings(ctx context.Context, documentID string, chunks []EmbeddedChunk, status JobStatus, embeddedCount int) (int, error)

UpsertEmbeddings stores vector embeddings for already indexed chunks.

func (*Indexer) UpsertSparse 

func (i *Indexer) UpsertSparse(ctx context.Context, doc ExtractedDocument) (int, error)

UpsertSparse stores document metadata and searchable text chunks.

type IngestRequest 

type IngestRequest struct {
	SourceID     string
	SourceKind   string
	OriginalPath string
	FileName     string
	MIMEType     string
	SizeBytes    int64
}

IngestRequest describes one file ingestion request before extraction.

type Job 

type Job struct {
	ID             string    `json:"id"`
	SourceID       string    `json:"source_id"`
	SourceKind     string    `json:"source_kind"`
	DocumentID     string    `json:"document_id"`
	ContentHash    string    `json:"content_hash"`
	OriginalPath   string    `json:"original_path"`
	FileName       string    `json:"file_name"`
	MIMEType       string    `json:"mime_type"`
	SizeBytes      int64     `json:"size_bytes"`
	Status         JobStatus `json:"status"`
	SparseChunks   int       `json:"sparse_chunks"`
	EmbeddedChunks int       `json:"embedded_chunks"`
	Error          string    `json:"error,omitempty"`
	CreatedAt      time.Time `json:"created_at,omitempty"`
	UpdatedAt      time.Time `json:"updated_at,omitempty"`
	SearchableAt   time.Time `json:"searchable_at,omitempty"`
	CompletedAt    time.Time `json:"completed_at,omitempty"`
}

Job is the API-facing view of a persisted document ingestion job.

type JobStatus 

type JobStatus string

JobStatus is the durable lifecycle state for a document ingestion job. 

const (
	JobAccepted   JobStatus = "accepted"
	JobExtracting JobStatus = "extracting"
	JobSearchable JobStatus = "searchable"
	JobEmbedding  JobStatus = "embedding"
	JobComplete   JobStatus = "complete"
	JobFailed     JobStatus = "failed"
	JobRefused    JobStatus = "refused"
	JobCanceled   JobStatus = "canceled"
)

Document ingestion job statuses.

type JobStore 

type JobStore interface {
	Create(ctx context.Context, params CreateJobParams) (Job, error)
	Get(ctx context.Context, id string) (Job, error)
	GetByDocumentID(ctx context.Context, documentID string) (Job, error)
	UpdateStatus(ctx context.Context, id string, status JobStatus, message string) (Job, error)
	UpdateProgress(ctx context.Context, id string, status JobStatus, sparseChunks, embeddedChunks int) (Job, error)
	ListRecent(ctx context.Context, limit int) ([]Job, error)
}

JobStore persists document ingestion job state.

type KnowledgeClient 

type KnowledgeClient interface {
	Read(ctx context.Context, query string, params map[string]any) ([]map[string]any, error)
	Write(ctx context.Context, query string, params map[string]any) ([]map[string]any, error)
}

KnowledgeClient is the subset of Neo4j knowledge operations used by indexing.

type Locator 

type Locator struct {
	Page      int    `json:"page,omitempty"`
	Sheet     string `json:"sheet,omitempty"`
	RowStart  int    `json:"row_start,omitempty"`
	RowEnd    int    `json:"row_end,omitempty"`
	Section   string `json:"section,omitempty"`
	Paragraph int    `json:"paragraph,omitempty"`
}

Locator identifies where a chunk came from inside the source document.

type PostgresJobStore 

type PostgresJobStore struct {
	// contains filtered or unexported fields
}

PostgresJobStore implements JobStore with sqlc-generated Postgres queries.

func NewPostgresJobStore 

func NewPostgresJobStore(pool *pgxpool.Pool) *PostgresJobStore

NewPostgresJobStore builds a Postgres-backed document ingestion job store.

func (*PostgresJobStore) Create 

func (s *PostgresJobStore) Create(ctx context.Context, params CreateJobParams) (Job, error)

Create inserts a new document ingestion job.

func (*PostgresJobStore) Get 

func (s *PostgresJobStore) Get(ctx context.Context, id string) (Job, error)

Get returns a document ingestion job by id.

func (*PostgresJobStore) GetByDocumentID 

func (s *PostgresJobStore) GetByDocumentID(ctx context.Context, documentID string) (Job, error)

GetByDocumentID returns the latest job for a document id.

func (*PostgresJobStore) ListRecent 

func (s *PostgresJobStore) ListRecent(ctx context.Context, limit int) ([]Job, error)

ListRecent returns recent document ingestion jobs, newest first.

func (*PostgresJobStore) UpdateProgress 

func (s *PostgresJobStore) UpdateProgress(ctx context.Context, id string, status JobStatus, sparseChunks, embeddedChunks int) (Job, error)

UpdateProgress updates a job status and indexed chunk counters.

func (*PostgresJobStore) UpdateStatus 

func (s *PostgresJobStore) UpdateStatus(ctx context.Context, id string, status JobStatus, message string) (Job, error)

UpdateStatus updates a job lifecycle status and optional error message.

type SearchBackend 

type SearchBackend interface {
	Search(ctx context.Context, req SearchRequest) ([]SearchHit, error)
}

SearchBackend executes document search requests.

type SearchHit 

type SearchHit struct {
	DocumentID  string   `json:"document_id"`
	ChunkID     string   `json:"chunk_id"`
	FileName    string   `json:"file_name"`
	Score       float64  `json:"score"`
	Text        string   `json:"text"`
	Locator     Locator  `json:"locator"`
	HeadingPath []string `json:"heading_path"`
}

SearchHit is one ranked result from document search.

type SearchRequest 

type SearchRequest struct {
	Query      string
	DocumentID string
	Limit      int
}

SearchRequest describes a sparse document search query.

type Searcher 

type Searcher struct {
	Client KnowledgeClient
}

Searcher runs sparse full-text search against indexed document chunks.

func (*Searcher) Search 

func (s *Searcher) Search(ctx context.Context, req SearchRequest) ([]SearchHit, error)

Search returns ranked chunk hits for a document search request.

type Service 

type Service struct {
	Jobs      JobStore
	Extractor Extractor
	Indexer   SparseIndexer
	Searcher  SearchBackend
	Embedder  EmbedQueue
	Clock     Clock
	MaxBytes  int64
}

Service coordinates document extraction, sparse indexing, search, and embedding.

func (*Service) GetJob 

func (s *Service) GetJob(ctx context.Context, id string) (*Job, error)

GetJob returns one document ingestion job by id.

func (*Service) IngestPath 

func (s *Service) IngestPath(ctx context.Context, req IngestRequest, path string) (*Job, error)

IngestPath ingests a local document file and returns once sparse search is ready.

func (*Service) ListJobs 

func (s *Service) ListJobs(ctx context.Context, limit int) ([]Job, error)

ListJobs returns recent document ingestion jobs.

func (*Service) Search 

func (s *Service) Search(ctx context.Context, req SearchRequest) ([]SearchHit, error)

Search delegates to the configured document search backend.

type SparseIndexer 

type SparseIndexer interface {
	UpsertSparse(ctx context.Context, doc ExtractedDocument) (int, error)
}

SparseIndexer stores extracted document text for immediate sparse search.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.