Documentation
¶
Overview ¶
Package bwtsearch provides a stable public API for building and querying FM-index based full-text search indexes.
Index ¶
- func Check(pattern string) error
- type BiIndex
- func BuildBi(text []byte) *BiIndex
- func BuildBiFromFiles(texts [][]byte, separator []byte) *BiIndex
- func BuildBiFromFilesWithConfig(texts [][]byte, separator []byte, algo SuffixArrayAlgorithm, occ OccStructure, ...) *BiIndex
- func BuildBiFromFilesWithOptions(texts [][]byte, separator []byte, algo SuffixArrayAlgorithm, occ OccStructure) *BiIndex
- func BuildBiWithConfig(text []byte, algo SuffixArrayAlgorithm, occ OccStructure, ...) *BiIndex
- func BuildBiWithOptions(text []byte, algo SuffixArrayAlgorithm, occ OccStructure) *BiIndex
- func LoadBi(path string) (*BiIndex, error)
- func ReadBiFrom(r io.Reader) (*BiIndex, error)
- func (idx *BiIndex) ContextAround(pos, patLen, ctxSize int) string
- func (idx *BiIndex) Count(pattern []byte) int
- func (idx *BiIndex) ExtendLeft(bi BiInterval, c byte) BiInterval
- func (idx *BiIndex) ExtendRight(bi BiInterval, c byte) BiInterval
- func (idx *BiIndex) FullInterval() BiInterval
- func (idx *BiIndex) Locate(pattern []byte, limit int) []int
- func (idx *BiIndex) Save(path string) error
- func (idx *BiIndex) TextLen() int
- func (idx *BiIndex) WriteTo(w io.Writer) (int64, error)
- type BiInterval
- type Index
- func Build(text []byte) *Index
- func BuildFromFiles(texts [][]byte, separator []byte) *Index
- func BuildFromFilesWithConfig(texts [][]byte, separator []byte, algo SuffixArrayAlgorithm, occ OccStructure, ...) *Index
- func BuildFromFilesWithOptions(texts [][]byte, separator []byte, algo SuffixArrayAlgorithm, occ OccStructure) *Index
- func BuildWithAlgorithm(text []byte, algo SuffixArrayAlgorithm) *Index
- func BuildWithConfig(text []byte, algo SuffixArrayAlgorithm, occ OccStructure, ...) *Index
- func BuildWithOptions(text []byte, algo SuffixArrayAlgorithm, occ OccStructure) *Index
- func Load(path string) (*Index, error)
- func ReadFrom(r io.Reader) (*Index, error)
- func (idx *Index) AlphabetSize() int
- func (idx *Index) Append(text []byte) error
- func (idx *Index) BWT() []byte
- func (idx *Index) ContextAround(pos, patLen, ctxSize int) string
- func (idx *Index) Count(pattern []byte) int
- func (idx *Index) Locate(pattern []byte, limit int) []int
- func (idx *Index) NumBWTRuns() int
- func (idx *Index) OccStorage() OccStorageOptions
- func (idx *Index) OccType() OccStructure
- func (idx *Index) SAAt(i int) int
- func (idx *Index) SALen() int
- func (idx *Index) Save(path string) error
- func (idx *Index) TextLen() int
- func (idx *Index) WheelerGraphMermaid(maxNodes int) string
- func (idx *Index) WriteTo(w io.Writer) (int64, error)
- type Interval
- type OccExternalStrategy
- type OccStorageMode
- type OccStorageOptions
- type OccStructure
- type SearchResult
- type StdlibIndex
- func (idx *StdlibIndex) ContextAround(pos, patLen, ctxSize int) string
- func (idx *StdlibIndex) Count(pattern []byte) int
- func (idx *StdlibIndex) Locate(pattern []byte, limit int) []int
- func (idx *StdlibIndex) Save(path string) error
- func (idx *StdlibIndex) TextLen() int
- func (idx *StdlibIndex) WriteTo(w io.Writer) (int64, error)
- type SuffixArrayAlgorithm
- type UnsupportedError
- type ViolationError
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type BiIndex ¶
type BiIndex struct {
// contains filtered or unexported fields
}
BiIndex is a bidirectional FM-index supporting both left- and right-extension of a search interval.
func BuildBiFromFiles ¶
BuildBiFromFiles concatenates texts with separator and builds a single BiIndex over the combined corpus. If separator is nil a newline (\n) is used. The separator must not contain 0x00.
BuildBiFromFiles panics if separator contains the byte 0x00.
func BuildBiFromFilesWithConfig ¶
func BuildBiFromFilesWithConfig(texts [][]byte, separator []byte, algo SuffixArrayAlgorithm, occ OccStructure, storage OccStorageOptions) *BiIndex
BuildBiFromFilesWithConfig concatenates texts with separator and builds a BiIndex using the specified logical and physical occ options.
func BuildBiFromFilesWithOptions ¶
func BuildBiFromFilesWithOptions(texts [][]byte, separator []byte, algo SuffixArrayAlgorithm, occ OccStructure) *BiIndex
BuildBiFromFilesWithOptions concatenates texts with separator and builds a BiIndex using the specified algorithm and occurrence-array structure.
BuildBiFromFilesWithOptions panics if separator contains the byte 0x00.
func BuildBiWithConfig ¶
func BuildBiWithConfig(text []byte, algo SuffixArrayAlgorithm, occ OccStructure, storage OccStorageOptions) *BiIndex
BuildBiWithConfig constructs a BiIndex with explicit logical and physical occurrence-array options.
func BuildBiWithOptions ¶
func BuildBiWithOptions(text []byte, algo SuffixArrayAlgorithm, occ OccStructure) *BiIndex
BuildBiWithOptions constructs a BiIndex with explicit suffix-array construction algorithm and occurrence-array structure.
func ReadBiFrom ¶
ReadBiFrom deserialises a BiIndex from r. The reader must be positioned at the start of a BIDX001 stream.
func (*BiIndex) ContextAround ¶
ContextAround returns a snippet of text centred on pos. A nil *BiIndex returns the empty string.
func (*BiIndex) Count ¶
Count returns the number of occurrences of pattern in the indexed text. A nil *BiIndex returns 0.
func (*BiIndex) ExtendLeft ¶
func (idx *BiIndex) ExtendLeft(bi BiInterval, c byte) BiInterval
ExtendLeft narrows bi by prepending character c (left extension). It uses the forward FM-index to update the forward interval and derives the reverse interval from the count of characters < c in the BWT range. A nil *BiIndex returns the zero (empty) BiInterval.
func (*BiIndex) ExtendRight ¶
func (idx *BiIndex) ExtendRight(bi BiInterval, c byte) BiInterval
ExtendRight narrows bi by appending character c (right extension). It uses the reverse FM-index to update the reverse interval and derives the forward interval accordingly. A nil *BiIndex returns the zero (empty) BiInterval.
func (*BiIndex) FullInterval ¶
func (idx *BiIndex) FullInterval() BiInterval
FullInterval returns the initial BiInterval covering the entire SA. A nil *BiIndex returns the zero BiInterval.
func (*BiIndex) Locate ¶
Locate returns up to limit text positions where pattern begins. When limit <= 0 all positions are returned. A nil *BiIndex returns nil.
type BiInterval ¶
BiInterval is a paired suffix-array interval used during bidirectional search. [LoFwd, HiFwd) is the interval in the forward FM-index (of T). [LoRev, HiRev) is the interval in the reverse FM-index (of T^R).
func (BiInterval) Size ¶
func (bi BiInterval) Size() int
Size returns the number of occurrences represented by this interval.
type Index ¶
type Index struct {
// contains filtered or unexported fields
}
Index is a wrapper around the internal FM-index implementation.
func Build ¶
Build constructs an index from text using the default SA-IS algorithm and RLBWT occurrence structure. Passing nil or an empty slice is valid and returns an index over the empty text (sentinel only): Count reports 0 for any non-empty pattern, Search finds no matches, TextLen is 0, and SALen is 1.
Example ¶
idx := Build([]byte("abracadabra"))
fmt.Println(idx.Count([]byte("abra")))
fmt.Println(idx.Count([]byte("xyz")))
Output: 2 0
Example (Japanese) ¶
ExampleBuild_japanese demonstrates building and searching an FM-index over UTF-8 Japanese text. "上杉謙信" (Uesugi Kenshin) is a famous warlord of the Sengoku period; the index finds both occurrences in the sentence.
// 上杉謙信 appears twice: once at byte 15, once at byte 63.
text := "武田信玄と上杉謙信は戦国時代の名将である。上杉謙信は越後の虎と呼ばれた。"
idx := Build([]byte(text))
fmt.Println(idx.Count([]byte("上杉謙信")))
fmt.Println(idx.Count([]byte("徳川家康")))
Output: 2 0
func BuildFromFiles ¶
BuildFromFiles concatenates texts with separator and builds a single FM-index over the combined corpus. If separator is nil a newline (\n) is used. The separator must not contain 0x00, which is reserved as the FM-index sentinel.
BuildFromFiles panics if separator contains the byte 0x00.
func BuildFromFilesWithConfig ¶
func BuildFromFilesWithConfig(texts [][]byte, separator []byte, algo SuffixArrayAlgorithm, occ OccStructure, storage OccStorageOptions) *Index
BuildFromFilesWithConfig concatenates texts with separator and builds a single FM-index using the specified logical and physical occ options. If separator is nil a newline (\n) is used. The separator must not contain 0x00, which is reserved as the FM-index sentinel.
func BuildFromFilesWithOptions ¶
func BuildFromFilesWithOptions(texts [][]byte, separator []byte, algo SuffixArrayAlgorithm, occ OccStructure) *Index
BuildFromFilesWithOptions concatenates texts with separator and builds a single FM-index using the specified algorithm and occurrence-array structure. If separator is nil a newline (\n) is used. The separator must not contain 0x00, which is reserved as the FM-index sentinel.
BuildFromFilesWithOptions panics if separator contains the byte 0x00.
func BuildWithAlgorithm ¶
func BuildWithAlgorithm(text []byte, algo SuffixArrayAlgorithm) *Index
BuildWithAlgorithm constructs an index from text with an explicit algorithm.
Example ¶
idx := BuildWithAlgorithm([]byte("banana"), AlgorithmSAIS)
fmt.Println(idx.Count([]byte("ana")))
Output: 2
func BuildWithConfig ¶
func BuildWithConfig(text []byte, algo SuffixArrayAlgorithm, occ OccStructure, storage OccStorageOptions) *Index
BuildWithConfig constructs an index with explicit logical and physical occurrence-array options.
func BuildWithOptions ¶
func BuildWithOptions(text []byte, algo SuffixArrayAlgorithm, occ OccStructure) *Index
BuildWithOptions constructs an index with an explicit suffix-array construction algorithm and an explicit occurrence-array structure.
Example ¶
ExampleBuildWithOptions demonstrates building an FM-index with an explicit suffix-array algorithm and a Wavelet Tree occurrence array.
idx := BuildWithOptions([]byte("abracadabra"), AlgorithmSAIS, OccWaveletTree)
fmt.Println(idx.Count([]byte("abra")))
fmt.Println(idx.Count([]byte("xyz")))
Output: 2 0
func Load ¶
Load reads an index from a file.
Example ¶
idx := Build([]byte("banana"))
file, err := os.CreateTemp("", "bwtsearch-example-*.idx")
if err != nil {
fmt.Println("error")
return
}
path := file.Name()
file.Close()
defer os.Remove(path)
if err := idx.Save(path); err != nil {
fmt.Println("error")
return
}
loaded, err := Load(path)
if err != nil {
fmt.Println("error")
return
}
fmt.Println(loaded.TextLen())
Output: 6
func (*Index) AlphabetSize ¶
AlphabetSize returns the number of distinct characters in the text. A nil *Index returns 0.
func (*Index) Append ¶
Append incrementally appends text to the index (RopeBWT style). Existing build options (suffix-array algorithm / occurrence structure) are preserved.
Example ¶
idx := Build([]byte("hello"))
_ = idx.Append([]byte(" world"))
fmt.Println(idx.Count([]byte("hello world")))
Output: 1
func (*Index) ContextAround ¶
ContextAround returns a snippet around a match position. A nil *Index returns the empty string.
func (*Index) Locate ¶
Locate returns up to limit positions where pattern begins. A nil *Index returns nil.
func (*Index) NumBWTRuns ¶
NumBWTRuns returns the number of equal-character runs in the BWT. This is the r parameter of the r-index: smaller values indicate more repetitive texts and a more compact RLBWT representation. A nil *Index returns 0.
func (*Index) OccStorage ¶
func (idx *Index) OccStorage() OccStorageOptions
OccStorage returns the physical storage options used by this index. A nil *Index returns in-memory defaults.
func (*Index) OccType ¶
func (idx *Index) OccType() OccStructure
OccType returns the occurrence-array structure used by this index. A nil *Index returns OccBitvectors (the zero value).
func (*Index) SAAt ¶
SAAt returns the text position stored at suffix-array index i. A nil *Index returns 0.
func (*Index) SALen ¶
SALen returns the suffix-array length (text + sentinel). A nil *Index returns 0.
func (*Index) Save ¶
Save writes the index to path.
Example ¶
idx := Build([]byte("abracadabra"))
file, err := os.CreateTemp("", "bwtsearch-example-*.idx")
if err != nil {
fmt.Println("error")
return
}
path := file.Name()
file.Close()
defer os.Remove(path)
if err := idx.Save(path); err != nil {
fmt.Println("error")
return
}
loaded, err := Load(path)
if err != nil {
fmt.Println("error")
return
}
fmt.Println(loaded.Count([]byte("abra")))
Output: 2
func (*Index) WheelerGraphMermaid ¶
WheelerGraphMermaid returns a Mermaid graph representation. A nil *Index returns the empty string.
type OccExternalStrategy ¶
type OccExternalStrategy int
OccExternalStrategy selects the external on-disk backend used when OccStorageOptions.Mode is OccStorageExternal.
const ( // OccExternalStrategyLSM uses an append-and-compact strategy. OccExternalStrategyLSM OccExternalStrategy = OccExternalStrategy(fmindex.OccExternalStrategyLSM) // OccExternalStrategyBPlusTree uses fixed pages with B+-tree-like lookup. OccExternalStrategyBPlusTree OccExternalStrategy = OccExternalStrategy(fmindex.OccExternalStrategyBPlusTree) // OccExternalStrategyInvertedSegments uses shard+segment inverted postings. OccExternalStrategyInvertedSegments OccExternalStrategy = OccExternalStrategy(fmindex.OccExternalStrategyInvertedSegments) )
type OccStorageMode ¶
type OccStorageMode int
OccStorageMode selects the physical storage strategy for occ structures.
const ( // OccStorageInMemory stores occ structures fully in memory. OccStorageInMemory OccStorageMode = OccStorageMode(fmindex.OccStorageInMemory) // OccStorageExternal stores supported occ structures with external memory. // Currently, this mode is supported for OccWaveletTree and OccBitvectors. OccStorageExternal OccStorageMode = OccStorageMode(fmindex.OccStorageExternal) )
type OccStorageOptions ¶
type OccStorageOptions struct {
Mode OccStorageMode
DiskBlockSize int
ExternalStrategy OccExternalStrategy
}
OccStorageOptions controls physical storage parameters for occ structures.
type OccStructure ¶
type OccStructure int
OccStructure selects the occurrence-array implementation inside the FM-index.
const ( // OccBitvectors uses one succinct bit-vector per distinct character // representation. On-disk magic uses FMI + occ-id(01) + persist-id. OccBitvectors OccStructure = OccStructure(fmindex.OccBitvectors) // OccWaveletTree uses a Wavelet Tree over the BWT, providing O(log σ) // rank queries and O(n log σ) total space. This is advantageous when // the alphabet is large or nearly all 256 byte values appear. // On-disk magic uses FMI + occ-id(02) + persist-id. OccWaveletTree OccStructure = OccStructure(fmindex.OccWaveletTree) // OccWaveletMatrix uses a Wavelet Matrix over the BWT. It provides the // same O(log σ) rank complexity as OccWaveletTree but with a flat, // cache-friendly memory layout. On-disk magic uses occ-id(03). OccWaveletMatrix OccStructure = OccStructure(fmindex.OccWaveletMatrix) // OccRLBWT uses a run-length encoded BWT for rank queries. The BWT is // stored as a compact sequence of equal-character runs, which is the // foundation of r-index style compressed indexes. Rank queries run in // O(log r) time where r is the number of BWT runs. This is the default // space-efficient for highly repetitive texts. // On-disk magic uses occ-id(04). OccRLBWT OccStructure = OccStructure(fmindex.OccRLBWT) // OccRRR uses one Raman-Raman-Rao (RRR) bit-vector per distinct character // over the BWT. On-disk magic uses occ-id(05). OccRRR OccStructure = OccStructure(fmindex.OccRRR) // OccEliasFano uses one Elias-Fano encoded position list per distinct // character over the BWT. On-disk magic uses occ-id(06). OccEliasFano OccStructure = OccStructure(fmindex.OccEliasFano) // OccPoppy uses one interleaved RRR (Poppy-style) bit-vector per distinct // character over the BWT. On-disk magic uses occ-id(07). OccPoppy OccStructure = OccStructure(fmindex.OccPoppy) // OccDynamicBitvectors uses one dynamic bit-vector per distinct character // over the BWT. On-disk magic uses occ-id(08). OccDynamicBitvectors OccStructure = OccStructure(fmindex.OccDynamicBitvectors) // OccExternalWaveletTree uses an external-memory Wavelet Tree over the BWT. // Node bit-vectors are stored in temporary files while rank summaries stay // in memory. This constant is kept for backward compatibility. // Prefer OccWaveletTree with OccStorageExternal. OccExternalWaveletTree OccStructure = OccStructure(fmindex.OccExternalWaveletTree) )
type SearchResult ¶
SearchResult holds the outcome of a star-free regex search.
func Search ¶
func Search(idx *Index, pattern string, limit int) (*SearchResult, error)
Search runs a star-free regex search over idx. It returns a *ViolationError if pattern violates the star-free constraint, or a *UnsupportedError for unsupported regex constructs.
Example ¶
idx := Build([]byte("abracadabra"))
res, err := Search(idx, "abra|cad", 0)
if err != nil {
fmt.Println("error")
return
}
fmt.Println(res.TotalCount)
fmt.Println(len(res.Positions(idx)))
Output: 3 3
func (*SearchResult) Positions ¶
func (sr *SearchResult) Positions(idx *Index) []int
Positions resolves all match positions in the indexed text.
type StdlibIndex ¶
type StdlibIndex struct {
// contains filtered or unexported fields
}
StdlibIndex wraps Go's standard-library suffix array for full-text search. Unlike Index (which is FM-index based), StdlibIndex only supports literal (non-regex) patterns.
func BuildStdlib ¶
func BuildStdlib(text []byte) *StdlibIndex
BuildStdlib constructs a StdlibIndex from text.
func BuildStdlibFromFiles ¶
func BuildStdlibFromFiles(texts [][]byte, separator []byte) *StdlibIndex
BuildStdlibFromFiles concatenates texts with separator and builds a single StdlibIndex over the combined corpus. If separator is nil a newline (\n) is used. The separator must not contain 0x00, for consistency with BuildFromFiles and BuildBiFromFiles.
BuildStdlibFromFiles panics if separator contains the byte 0x00.
func LoadStdlib ¶
func LoadStdlib(path string) (*StdlibIndex, error)
LoadStdlib reads a StdlibIndex from a file.
func ReadStdlibFrom ¶
func ReadStdlibFrom(r io.Reader) (*StdlibIndex, error)
ReadStdlibFrom deserialises a StdlibIndex from r. The reader must be positioned at the start of a SAIDX01 stream.
func (*StdlibIndex) ContextAround ¶
func (idx *StdlibIndex) ContextAround(pos, patLen, ctxSize int) string
ContextAround returns a human-readable snippet of text centred on position pos, showing ctxSize bytes on each side plus patLen bytes of the match itself. A nil *StdlibIndex returns the empty string.
func (*StdlibIndex) Count ¶
func (idx *StdlibIndex) Count(pattern []byte) int
Count returns the number of occurrences of pattern in the indexed text. A nil *StdlibIndex returns 0.
func (*StdlibIndex) Locate ¶
func (idx *StdlibIndex) Locate(pattern []byte, limit int) []int
Locate returns up to limit positions where pattern begins (0-indexed). When limit <= 0 all positions are returned. A nil *StdlibIndex returns nil.
func (*StdlibIndex) Save ¶
func (idx *StdlibIndex) Save(path string) error
Save writes the index to path.
func (*StdlibIndex) TextLen ¶
func (idx *StdlibIndex) TextLen() int
TextLen returns the original text length. A nil *StdlibIndex returns 0.
type SuffixArrayAlgorithm ¶
type SuffixArrayAlgorithm int
SuffixArrayAlgorithm selects the suffix-array construction algorithm.
const ( // AlgorithmDoubling uses the prefix-doubling algorithm. AlgorithmDoubling SuffixArrayAlgorithm = SuffixArrayAlgorithm(fmindex.AlgorithmDoubling) // AlgorithmSAIS uses the SA-IS algorithm. AlgorithmSAIS SuffixArrayAlgorithm = SuffixArrayAlgorithm(fmindex.AlgorithmSAIS) )
type UnsupportedError ¶
type UnsupportedError struct {
// Op is a human-readable name of the unsupported operator.
Op string
// SubExpr is the string form of the offending sub-expression.
SubExpr string
}
UnsupportedError is returned by Check and Search when the pattern contains a regex construct that this FM-index matcher does not support (for example, position anchors).
func (*UnsupportedError) Error ¶
func (e *UnsupportedError) Error() string
type ViolationError ¶
type ViolationError struct {
// Op is a human-readable name of the offending operator (e.g. "Kleene star (*)").
Op string
// SubExpr is the string form of the offending sub-expression.
SubExpr string
}
ViolationError is returned by Check and Search when the pattern contains a construct that violates the star-free constraint (e.g. Kleene star, +, or unbounded repetition). Callers can use errors.As to inspect the details.
func (*ViolationError) Error ¶
func (e *ViolationError) Error() string
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
textindex
command
|
|
|
internal
|
|
|
bitvector
Package bitvector provides a succinct bit vector with O(1) Rank1/Rank0 queries.
|
Package bitvector provides a succinct bit vector with O(1) Rank1/Rank0 queries. |
|
fmindex
Package fmindex implements an FM-index, a compressed full-text index that forms a Wheeler graph over the text.
|
Package fmindex implements an FM-index, a compressed full-text index that forms a Wheeler graph over the text. |
|
rindex
Package rindex implements a run-length BWT (RLBWT) based occurrence structure.
|
Package rindex implements a run-length BWT (RLBWT) based occurrence structure. |
|
starfree
Package starfree provides validation and FM-index search for star-free regular expressions.
|
Package starfree provides validation and FM-index search for star-free regular expressions. |
|
wavelet
Package wavelet provides a Wavelet Tree for rank queries over byte sequences.
|
Package wavelet provides a Wavelet Tree for rank queries over byte sequences. |
|
waveletmatrix
Package waveletmatrix provides a Wavelet Matrix for rank queries over byte sequences.
|
Package waveletmatrix provides a Wavelet Matrix for rank queries over byte sequences. |