search

package
v0.8.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 29, 2026 License: MIT Imports: 27 Imported by: 0

Documentation

Overview

Package search provides format-neutral, revision-bound search over immutable UTF-8 byte sources. Persistent indexes are disposable candidate accelerators: every indexed hit is verified against the bound Source.

Index

Constants

View Source 
const (
	DefaultBlockBytes                int64 = 64 << 10
	MaximumBlockBytes                int64 = 64 << 20
	DefaultOverlapBytes              int64 = 64
	MaximumOverlapBytes              int64 = 1 << 20
	DefaultMaxBlocks                       = 1 << 20
	MaximumBlocks                          = 16 << 20
	DefaultMaxAnalyzerTokensPerBlock       = 65_536
	MaximumAnalyzerTokensPerBlock          = 1_048_576
	DefaultMaxAnalyzerTermBytes            = 4 << 10
	MaximumAnalyzerTermBytes               = 1 << 20
	MaximumAnalyzerIdentityBytes           = 4 << 10
)
View Source 
const (
	DefaultReadBufferBytes    int64 = 64 << 10
	MaximumReadBufferBytes    int64 = 4 << 20
	DefaultMaxScannedBytes    int64 = 64 << 20
	MaximumScannedBytes       int64 = 1 << 30
	DefaultMaxPatternBytes          = 1 << 20
	MaximumPatternBytes             = 16 << 20
	DefaultMaxResults               = 1_000
	MaximumResults                  = 1_000_000
	DefaultMaxCandidateBlocks       = 100_000
	MaximumCandidateBlocks          = 10_000_000
)
View Source 
const MaximumAnalyzerQueryTerms = 1_024

Variables

View Source 
var (
	ErrInvalidContext     = errors.New("search: nil context")
	ErrInvalidSource      = errors.New("search: invalid source")
	ErrInvalidQuery       = errors.New("search: invalid query")
	ErrInvalidOptions     = errors.New("search: invalid options")
	ErrInvalidUTF8        = errors.New("search: source is not UTF-8")
	ErrSourceInconsistent = errors.New("search: source length or content changed")
	ErrClosed             = errors.New("search: index closed")
	ErrIndexUnavailable   = errors.New("search: persistent index unavailable")
	ErrIndexStale         = errors.New("search: persistent index is stale")
	ErrIndexCorrupt       = errors.New("search: persistent index is corrupt")
	ErrAnalyzerMismatch   = errors.New("search: analyzer identity mismatch")
	ErrLineageMismatch    = errors.New("search: index lineage mismatch")
	ErrInvalidIndex       = errors.New("search: invalid previous index")
)

Functions

func Collect 

func Collect(ctx context.Context, directory string) error

Collect removes unpublished build directories and retired generations that have no live Index in this or another process. Unknown files are untouched.

Types

type Analyzer 

type Analyzer interface {
	Identity() string
	RequiredOverlapBytes() int64
	Analyze(context.Context, []byte) ([]Token, error)
	QueryTerms(context.Context, string) ([]string, error)
}

Analyzer injects language or host-specific "word" policy without teaching the core any document format. Analyze must return deterministic tokens for the same UTF-8 block. QueryTerms returns normalized alternatives for one Term query.

RequiredOverlapBytes must cover both token length and any boundary context the Analyzer needs. Build stores only tokens whose Start lies in a block's non-overlapping core.

type CaseMode 

type CaseMode uint8

CaseMode controls literal comparison. Regexp case behavior is expressed by the Go regexp itself. Term behavior belongs to the injected Analyzer. 

const (
	CaseExact CaseMode = iota + 1
	CaseUnicodeFold
)

type IncompleteReason 

type IncompleteReason uint32

IncompleteReason is a bit set because more than one budget may truncate one query. A zero value means complete. 

const (
	IncompleteScanLimit IncompleteReason = 1 << iota
	IncompleteResultLimit
	IncompleteCandidateLimit
	IncompleteFallbackDisabled
)

func (IncompleteReason) Has 

func (r IncompleteReason) Has(reason IncompleteReason) bool

type Index 

type Index struct {
	// contains filtered or unexported fields
}

Index is permanently bound to one immutable Source revision and one published candidate generation.

func Build 

func Build(ctx context.Context, source Source, revision uint64, options IndexOptions) (*Index, error)

Build creates and atomically publishes a complete persistent generation. The caller retains Source ownership.

func BuildOwned 

func BuildOwned(ctx context.Context, source OwnedSource, revision uint64, options IndexOptions) (*Index, error)

BuildOwned transfers Source ownership to the returned Index.

func Open 

func Open(ctx context.Context, source Source, revision uint64, options IndexOptions) (*Index, error)

Open validates a published generation against the complete immutable Source.

func OpenOrBuild 

func OpenOrBuild(ctx context.Context, source Source, revision uint64, options IndexOptions) (*Index, error)

OpenOrBuild opens an exact generation or replaces a missing, stale, corrupt, or analyzer-incompatible cache with a complete rebuild.

func OpenOrBuildOwned 

func OpenOrBuildOwned(ctx context.Context, source OwnedSource, revision uint64, options IndexOptions) (*Index, error)

func OpenOwned 

func OpenOwned(ctx context.Context, source OwnedSource, revision uint64, options IndexOptions) (*Index, error)

func Rebuild 

func Rebuild(ctx context.Context, source Source, previous *Index, changes coordinate.ChangeMap) (*Index, error)

Rebuild publishes the current Source revision. When deterministic block boundaries are unchanged, it clones the immutable previous database and rewrites only blocks whose overlap-inclusive content hash changed. Any unprovable case falls back to a complete Build.

func RebuildOwned 

func RebuildOwned(ctx context.Context, source OwnedSource, previous *Index, changes coordinate.ChangeMap) (*Index, error)

func (*Index) BelongsTo 

func (i *Index) BelongsTo(lineage *Lineage) bool

func (*Index) Close 

func (i *Index) Close() error

func (*Index) CloseContext 

func (i *Index) CloseContext(ctx context.Context) error

CloseContext starts one shared shutdown, cancels admitted queries, and may stop waiting without abandoning database and Source cleanup.

func (*Index) Search 

func (i *Index) Search(ctx context.Context, query Query, options Options) (Report, error)

Search uses the persistent generation when it can prove a complete candidate plan. Otherwise it runs the same bounded Scan baseline.

func (*Index) Stats 

func (i *Index) Stats() Stats

type IndexOptions 

type IndexOptions struct {
	Directory string

	BlockBytes   int64
	OverlapBytes int64
	MaxBlocks    int

	MaxAnalyzerTokensPerBlock int
	MaxAnalyzerTermBytes      int
	Analyzer                  Analyzer

	// Lineage is an opaque identity inherited by Rebuild and overridden by
	// Session integrations.
	Lineage *Lineage
}

type Lineage 

type Lineage struct {
	// contains filtered or unexported fields
}

Lineage is an opaque identity shared by indexes derived from one trusted Source history. Pointer identity is intentional.

func NewLineage 

func NewLineage() *Lineage

type Match 

type Match struct {
	Revision  uint64
	Start     int64
	End       int64
	SourceKey string
}

Match is a half-open byte range in one immutable revision.

type Options 

type Options struct {
	ReadBufferBytes    int64
	MaxScannedBytes    int64
	MaxPatternBytes    int
	MaxResults         int
	MaxCandidateBlocks int
	// DisableFallback prevents an Index from falling back to Scan when no
	// complete candidate plan fits the configured budgets.
	DisableFallback bool
	// SourceKey is an opaque host value copied into every Match.
	SourceKey string
}

Options are hard per-query limits. Zero selects a bounded default.

type OwnedSource 

type OwnedSource interface {
	Source
	io.Closer
}

OwnedSource transfers its lifetime to an Index.

type Query 

type Query struct {
	Kind QueryKind
	Text string
	Case CaseMode
}

Query is immutable by value. Text is literal text, a Go RE2 pattern, or an analyzer query depending on Kind.

func Literal 

func Literal(text string, mode CaseMode) Query

func Regexp 

func Regexp(pattern string) Query

func Term 

func Term(text string) Query

type QueryKind 

type QueryKind uint8

QueryKind identifies one format-neutral search operation. 

const (
	QueryLiteral QueryKind = iota + 1
	QueryRegexp
	QueryTerm
)

type Report 

type Report struct {
	Revision        uint64
	Generation      uint64
	Matches         []Match
	Complete        bool
	Incomplete      IncompleteReason
	UsedIndex       bool
	ScannedBytes    int64
	CandidateBlocks int
}

Report describes both results and the work used to obtain them.

func Scan 

func Scan(ctx context.Context, source Source, revision uint64, query Query, options Options) (Report, error)

Scan is the correctness baseline used when no persistent candidate index is available. Literal queries stream through a fixed-size buffer. Regexp queries buffer only the explicitly bounded scanned prefix because Go's RE2 engine does not expose resumable automaton state.

type Source 

type Source interface {
	io.ReaderAt
	Len() int64
}

Source is an immutable UTF-8 byte source.

type Stats 

type Stats struct {
	Revision      uint64
	Generation    uint64
	ByteLength    int64
	BlockCount    int
	BlockBytes    int64
	OverlapBytes  int64
	AnalyzerID    string
	Directory     string
	DatabaseBytes int64
	ReusedBlocks  int
	IndexedBlocks int
	Closed        bool
}

Stats is an immutable view of one published search generation.

type Token 

type Token struct {
	Term       string
	Start, End int
}

Token is an analyzer-defined, format-neutral term and byte range relative to the supplied block. The search package never interprets Term.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.