Documentation
¶
Overview ¶
Package indexer implements the core indexing engine. It maintains an in-memory inverted index backed by on-disk segments that are periodically flushed. Searches fan out across the memory index and all segment readers, and results are deduplicated before being returned.
Index ¶
- type Engine
- func (e *Engine) Close() error
- func (e *Engine) Flush() error
- func (e *Engine) GetAvgDocLength() float64
- func (e *Engine) GetDocLength(docID string) int
- func (e *Engine) GetTotalDocs() int64
- func (e *Engine) IndexDocument(docID string, title string, body string) error
- func (e *Engine) ReloadSegments() int
- func (e *Engine) Search(term string) (index.PostingList, error)
- func (e *Engine) StartFlushLoop(ctx context.Context)
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Engine ¶
type Engine struct {
// contains filtered or unexported fields
}
Engine is the primary indexing data structure. It buffers documents in a MemoryIndex and flushes them to immutable on-disk segments when the configured size threshold is reached.
func NewEngine ¶
func NewEngine(cfg config.IndexerConfig) (*Engine, error)
NewEngine creates a new Engine, creating the data directory if necessary and loading any previously-flushed segments from disk.
func (*Engine) Flush ¶
Flush writes the current memory index snapshot to a new on-disk segment and opens a reader for it.
func (*Engine) GetAvgDocLength ¶
GetAvgDocLength returns the average document length across all indexed docs.
func (*Engine) GetDocLength ¶
GetDocLength returns the token count for the given document.
func (*Engine) GetTotalDocs ¶
GetTotalDocs returns the total number of documents indexed by this engine.
func (*Engine) IndexDocument ¶
IndexDocument tokenises the document and adds it to the memory index. If the memory index exceeds SegmentMaxSize the buffer is flushed to disk.
func (*Engine) ReloadSegments ¶
ReloadSegments re-scans the data directory for .spdx segment files and opens any that are not already loaded. This allows a searcher process to pick up segments flushed by a separate indexer process sharing the same data volume.
func (*Engine) Search ¶
func (e *Engine) Search(term string) (index.PostingList, error)
Search tokenises the query term, queries the memory index and all segment readers, and returns deduplicated postings.
func (*Engine) StartFlushLoop ¶
StartFlushLoop starts a background goroutine that flushes the memory index at the configured interval. It performs a final flush when ctx is cancelled.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package consumer reads ingestion events from Kafka and indexes them via the indexer engine, optionally routing documents through the shard router for partitioned indexing.
|
Package consumer reads ingestion events from Kafka and indexes them via the indexer engine, optionally routing documents through the shard router for partitioned indexing. |
|
Package index defines the in-memory inverted-index data structures used by the indexer.
|
Package index defines the in-memory inverted-index data structures used by the indexer. |
|
Package segment implements a custom binary segment file format (.spdx) for persisting inverted-index data to disk.
|
Package segment implements a custom binary segment file format (.spdx) for persisting inverted-index data to disk. |
|
Package shard provides hash-based shard routing for index engines.
|
Package shard provides hash-based shard routing for index engines. |
|
Package tokenizer provides text tokenisation for the search engine.
|
Package tokenizer provides text tokenisation for the search engine. |