zap

package
v0.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 10, 2026 License: Apache-2.0, Apache-2.0 Imports: 25 Imported by: 0

README

zapx-bluge segment core

This package is derived from Bleve zapx v17.2.0. The default build remains pure Go and does not compile FAISS, while vector section identifiers, address lookups, caches, and the public backend extension boundary are retained.

The on-disk format is owned by Bluge and is identified by segment type zapx-bluge, version 2. It is not a Bleve zap v17 segment. Version 2 keeps the exact field statistics introduced by v1 and adds an impact offset to every general postings header, after the frequency and location offsets. An offset of zero means the term has no impact table. High-cardinality terms store 64-posting blocks containing:

  1. The delta-encoded inclusive final document number for the block.
  2. The block's non-dominated (frequency, raw norm) impact pairs.

The pairs let built-in BM25 scorers calculate exact block upper bounds using the same arithmetic as document scoring. Terms below 128 postings omit the table and use the ordinary iterator path.

Every field metadata record also stores these unsigned varints after the section address pairs:

  1. Number of documents containing at least one indexed term for the field.
  2. Sum of all term frequencies for the field.

These values let Bluge construct exact BM25 collection statistics without scanning the term dictionary during query setup. Version 2 does not read v1 segments. Any future change to postings headers, field metadata, or the footer layout must increment the zapx-bluge version.

Segment construction consumes Bluge's analyzed token frequencies through the internal blugeidx representation. Native fields reuse their token maps; custom segment.Field implementations use an iteration fallback.

Documentation

Index

Constants

View Source 
const (
	SectionInvertedTextIndex = iota
	// SectionVectorIndex stores a backend-tagged native vector payload. The
	// old name is retained as a source compatibility alias because the slot was
	// originally reserved by the Bleve FAISS integration.
	SectionVectorIndex
	SectionSynonymIndex

	// Add new sections above this line.
	// NumSections automatically reflects the total number of sections
	// and is used to track how many sections can be registered.
	NumSections
)
View Source 
const DefaultFileCallbackId = ""

the default id to use for file callbacks that indicates a no op

View Source 
const DocNum1HitFinished = math.MaxUint64
View Source 
const FSTValEncoding1Hit = uint64(0x8000000000000000)
View Source 
const FSTValEncodingGeneral = uint64(0x0000000000000000)
View Source 
const FSTValEncodingMask = uint64(0xc000000000000000)
View Source 
const FooterSize = 4 + 4 + 4 + 4 + 8 + 8 + 8

FooterSize is the size of the footer record in bytes crc + id length + ver + chunk + sectionsIndexOffset + stored offset + num docs Does not include the length of the id because it is variable length

View Source 
const SectionFaissVectorIndex = SectionVectorIndex
View Source 
const Type string = "zapx-bluge"
View Source 
const Version uint32 = 2

Variables

View Source 
var DefaultChunkMode uint32 = 1026

DefaultChunkMode is the most recent improvement to chunking and should be used by default.

View Source 
var DefaultFileMergerBufferSize = 1024 * 1024
View Source 
var ErrChunkSizeZero = errors.New("chunk size is zero")
View Source 
var ErrVectorPayloadNotFound = errors.New("vector payload was not found")
View Source 
var LegacyChunkMode uint32 = 1024

LegacyChunkMode was the original chunk mode (always chunk size 1024) this mode is still used for chunking doc values.

View Source 
var NewSegmentBufferAvgBytesPerDocFactor float64 = 1.0
View Source 
var NewSegmentBufferNumResultsBump int = 100
View Source 
var NewSegmentBufferNumResultsFactor float64 = 1.0
View Source 
var NormBits1Hit = uint64(1)
View Source 
var SizeOfBool int
View Source 
var SizeOfFloat32 int
View Source 
var SizeOfFloat64 int
View Source 
var SizeOfInt int
View Source 
var SizeOfMap int
View Source 
var SizeOfPtr int
View Source 
var SizeOfSlice int
View Source 
var SizeOfString int
View Source 
var SizeOfUint16 int
View Source 
var SizeOfUint32 int
View Source 
var SizeOfUint64 int
View Source 
var SizeOfUint8 int
View Source 
var ValidateDocFields = func(field *blugeidx.Field) error {
	return nil
}

ValidateDocFields can be set by applications to perform additional checks on fields in a document being added to a new segment, by default it does nothing. This API is experimental and may be removed at any time.

Functions

func FSTValDecode1Hit 

func FSTValDecode1Hit(v uint64) (docNum uint64, normBits uint64)

func FSTValEncode1Hit 

func FSTValEncode1Hit(docNum uint64, normBits uint64) uint64

func LoadBytes 

func LoadBytes(data []byte) (seg.Segment, error)

LoadBytes opens a zapx segment from bytes owned by the caller.

func LoadBytesUsing 

func LoadBytesUsing(data []byte, config map[string]interface{}) (seg.Segment, error)

LoadBytesUsing opens a zapx segment from bytes owned by the caller.

func MergeToWriter 

func MergeToWriter(segments []seg.Segment, drops []*roaring.Bitmap, w io.Writer,
	closeCh chan struct{}) ([][]uint64, uint64, error)

MergeToWriter merges zapx segments and writes the new zapx segment to w.

func MergeToWriterUsing 

func MergeToWriterUsing(segments []seg.Segment, drops []*roaring.Bitmap, w io.Writer,
	closeCh chan struct{}, stats seg.StatsReporter, config map[string]interface{}) (
	[][]uint64, uint64, error)

MergeToWriterUsing merges zapx segments and writes the new zapx segment to w.

func New 

func New(results []*blugeidx.Document) (seg.Segment, uint64, error)

New creates an in-memory zapx text segment.

func NewWithConfig added in v0.6.0 

func NewWithConfig(results []*blugeidx.Document, normCalc func(string, int) float32,
	config map[string]interface{}) (seg.Segment, uint64, error)

NewWithConfig creates a segment while passing opaque section configuration to the segment builder. The ordinary text-only path remains unchanged when config is nil.

func NewWithNormCalc 

func NewWithNormCalc(results []*blugeidx.Document, normCalc func(string, int) float32) (
	seg.Segment, uint64, error)

NewWithNormCalc creates a segment using the caller's field norm encoding.

func PersistSegmentBase 

func PersistSegmentBase(sb *SegmentBase, path string) error

PersistSegmentBase persists SegmentBase in the zap file format.

func PostingsIteratorFrom1Hit 

func PostingsIteratorFrom1Hit(docNum1Hit uint64,
	includeFreqNorm, includeLocs bool) (segment.PostingsIterator, error)

PostingsIteratorFrom1Hit constructs a PostingsIterator given a 1-hit docNum.

func PostingsIteratorFromBitmap 

func PostingsIteratorFromBitmap(bm *roaring.Bitmap,
	includeFreqNorm, includeLocs bool) (segment.PostingsIterator, error)

PostingsIteratorFromBitmap constructs a PostingsIterator given an "actual" bitmap.

func ReadDocValueBoundary 

func ReadDocValueBoundary(chunk int, metaHeaders []MetaData) (uint64, uint64)

ReadDocValueBoundary elicits the start, end offsets from a metaData header slice

Types

type CountHashWriter 

type CountHashWriter struct {
	// contains filtered or unexported fields
}

CountHashWriter is a wrapper around a Writer which counts the number of bytes which have been written and computes a crc32 hash

func NewCountHashWriter 

func NewCountHashWriter(w io.Writer) *CountHashWriter

NewCountHashWriter returns a CountHashWriter which wraps the provided Writer

func NewCountHashWriterWithStatsReporter 

func NewCountHashWriterWithStatsReporter(w io.Writer, s segment.StatsReporter) *CountHashWriter

func (*CountHashWriter) Count 

func (c *CountHashWriter) Count() int

Count returns the number of bytes written

func (*CountHashWriter) Sum32 

func (c *CountHashWriter) Sum32() uint32

Sum32 returns the CRC-32 hash of the content written to this writer

func (*CountHashWriter) Write 

func (c *CountHashWriter) Write(b []byte) (int, error)

Write writes the provided bytes to the wrapped writer and counts the bytes

type Dictionary 

type Dictionary struct {
	// contains filtered or unexported fields
}

Dictionary is the zap representation of the term dictionary

func (*Dictionary) AutomatonIterator 

func (d *Dictionary) AutomatonIterator(a segment.Automaton,
	startKeyInclusive, endKeyExclusive []byte) segment.DictionaryIterator

AutomatonIterator returns an iterator which only visits terms having the the vellum automaton and start/end key range

func (*Dictionary) BytesRead 

func (d *Dictionary) BytesRead() uint64

func (*Dictionary) BytesWritten 

func (d *Dictionary) BytesWritten() uint64

func (*Dictionary) Cardinality 

func (d *Dictionary) Cardinality() int

func (*Dictionary) Contains 

func (d *Dictionary) Contains(key []byte) (bool, error)

func (*Dictionary) PostingsList 

func (d *Dictionary) PostingsList(term []byte, except *roaring.Bitmap,
	prealloc segment.PostingsList) (segment.PostingsList, error)

PostingsList returns the postings list for the specified term

func (*Dictionary) ResetBytesRead 

func (d *Dictionary) ResetBytesRead(val uint64)

type DictionaryIterator 

type DictionaryIterator struct {
	// contains filtered or unexported fields
}

DictionaryIterator is an iterator for term dictionary

func (*DictionaryIterator) Next 

func (i *DictionaryIterator) Next() (*index.DictEntry, error)

Next returns the next entry in the dictionary

type FileReader 

type FileReader struct {
	// contains filtered or unexported fields
}

FileReader wraps a reader callback to be applied to data read from a file.

func NewFileReader 

func NewFileReader(id string, context []byte) (*FileReader, error)

NewFileReader creates a FileReader with the reader callback identified by the context. The id is used to identify which callback to use when reading data.

type FileWriter 

type FileWriter struct {
	// contains filtered or unexported fields
}

FileWriter wraps a CountHashWriter and applies a user provided writer callback to the data being written.

func NewFileWriter 

func NewFileWriter(c *CountHashWriter, context []byte) (*FileWriter, error)

NewFileWriter creates a FileWriter with the provided CountHashWriter and applies the writer callback identified by the context.

func NewFileWriterEmpty 

func NewFileWriterEmpty(c *CountHashWriter) *FileWriter

creates an empty FileWriter with no callback. Used when we are writing data that is not going to be persisted

func (*FileWriter) Count 

func (w *FileWriter) Count() int

func (*FileWriter) Sum32 

func (w *FileWriter) Sum32() uint32

func (*FileWriter) Write 

func (w *FileWriter) Write(data []byte) (int, error)

type Location 

type Location struct {
	// contains filtered or unexported fields
}

Location represents the location of a single occurrence

func (*Location) ArrayPositions 

func (l *Location) ArrayPositions() []uint64

ArrayPositions returns the array position vector associated with this occurrence

func (*Location) End 

func (l *Location) End() uint64

End returns the end byte offset of this occurrence

func (*Location) Field 

func (l *Location) Field() string

Field returns the name of the field (useful in composite fields to know which original field the value came from)

func (*Location) Pos 

func (l *Location) Pos() uint64

Pos returns the 1-based phrase position of this occurrence

func (*Location) Size 

func (l *Location) Size() int

func (*Location) Start 

func (l *Location) Start() uint64

Start returns the start byte offset of this occurrence

type MetaData 

type MetaData struct {
	DocNum      uint64 // docNum of the data inside the chunk
	DocDvOffset uint64 // offset of data inside the chunk for the given docid
}

MetaData represents the data information inside a chunk.

type Posting 

type Posting struct {
	// contains filtered or unexported fields
}

Posting is a single entry in a postings list

func (*Posting) Frequency 

func (p *Posting) Frequency() uint64

Frequency returns the frequencies of occurrence of this term in this doc/field

func (*Posting) Locations 

func (p *Posting) Locations() []segment.Location

Locations returns the location information for each occurrence

func (*Posting) Norm 

func (p *Posting) Norm() float64

Norm returns the normalization factor for this posting

func (*Posting) NormUint64 

func (p *Posting) NormUint64() uint64

NormUint64 returns the norm value as uint64

func (*Posting) Number 

func (p *Posting) Number() uint64

Number returns the document number of this posting in this segment

func (*Posting) Size 

func (p *Posting) Size() int

type PostingsIterator 

type PostingsIterator struct {
	Actual   roaring.IntPeekable
	ActualBM *roaring.Bitmap
	// contains filtered or unexported fields
}

PostingsIterator provides a way to iterate through the postings list

func (*PostingsIterator) ActualBitmap 

func (p *PostingsIterator) ActualBitmap() *roaring.Bitmap

ActualBitmap returns the underlying actual bitmap which can be used up the stack for optimizations

func (*PostingsIterator) Advance 

func (i *PostingsIterator) Advance(docNum uint64) (segment.Posting, error)

Advance returns the posting at the specified docNum or it is not present the next posting, or if the end is reached, nil

func (*PostingsIterator) AdvanceShallow 

func (i *PostingsIterator) AdvanceShallow(docNum uint64) (
	uint64, []blugeseg.Impact, bool, error)

AdvanceShallow advances only the block metadata cursor. It does not consume the postings iterator, so callers can skip the block with Advance(blockEnd+1).

func (*PostingsIterator) BytesRead 

func (i *PostingsIterator) BytesRead() uint64

func (*PostingsIterator) BytesWritten 

func (i *PostingsIterator) BytesWritten() uint64

func (*PostingsIterator) DocNum1Hit 

func (p *PostingsIterator) DocNum1Hit() (uint64, bool)

DocNum1Hit returns the docNum and true if this is "1-hit" optimized and the docNum is available.

func (*PostingsIterator) HasImpacts 

func (i *PostingsIterator) HasImpacts() bool

HasImpacts reports whether this term has v2 block impact metadata.

func (*PostingsIterator) Next 

func (i *PostingsIterator) Next() (segment.Posting, error)

Next returns the next posting on the postings list, or nil at the end

func (*PostingsIterator) ReplaceActual 

func (p *PostingsIterator) ReplaceActual(abm *roaring.Bitmap)

ReplaceActual replaces the ActualBM with the provided bitmap

func (*PostingsIterator) ResetBytesRead 

func (i *PostingsIterator) ResetBytesRead(val uint64)

Implements the segment.DiskStatsReporter interface The purpose of this implementation is to get the bytes read from the disk which includes the freqNorm and location specific information of a hit

func (*PostingsIterator) Size 

func (i *PostingsIterator) Size() int

type PostingsList 

type PostingsList struct {
	// contains filtered or unexported fields
}

PostingsList is an in-memory representation of a postings list

func (*PostingsList) BytesRead 

func (p *PostingsList) BytesRead() uint64

func (*PostingsList) BytesWritten 

func (p *PostingsList) BytesWritten() uint64

func (*PostingsList) Count 

func (p *PostingsList) Count() uint64

Count returns the number of items on this postings list

func (*PostingsList) Iterator 

func (p *PostingsList) Iterator(includeFreq, includeNorm, includeLocs bool,
	prealloc segment.PostingsIterator) segment.PostingsIterator

Iterator returns an iterator for this postings list

func (*PostingsList) OrInto 

func (p *PostingsList) OrInto(receiver *roaring.Bitmap)

func (*PostingsList) ResetBytesRead 

func (p *PostingsList) ResetBytesRead(val uint64)

Implements the segment.DiskStatsReporter interface The purpose of this implementation is to get the bytes read from the postings lists stored on disk, while querying

func (*PostingsList) Size 

func (p *PostingsList) Size() int

type Segment 

type Segment struct {
	SegmentBase
	// contains filtered or unexported fields
}

Segment implements a persisted segment.Segment interface, by embedding an mmap()'ed SegmentBase.

func (*Segment) AddRef 

func (s *Segment) AddRef()

func (*Segment) BytesRead 

func (s *Segment) BytesRead() uint64

func (*Segment) BytesWritten 

func (s *Segment) BytesWritten() uint64

func (*Segment) CRC 

func (s *Segment) CRC() uint32

CRC returns the CRC value stored in the file footer

func (*Segment) ChunkMode 

func (s *Segment) ChunkMode() uint32

ChunkFactor returns the chunk factor in the file footer

func (*Segment) Close 

func (s *Segment) Close() (err error)

Close releases all resources associated with this segment

func (*Segment) Data 

func (s *Segment) Data() []byte

Data returns the underlying mmaped data slice

func (*Segment) DecRef 

func (s *Segment) DecRef() (err error)

func (*Segment) DictAddr 

func (s *Segment) DictAddr(field string) (uint64, error)

DictAddr is a helper function to compute the file offset where the dictionary is stored for the specified field.

func (*Segment) EdgeListAddr 

func (s *Segment) EdgeListAddr() (uint64, error)

EdgeListAddr is the exported helper function to compute the file offset where the edge list is stored.

func (*Segment) NumDocs 

func (s *Segment) NumDocs() uint64

NumDocs returns the number of documents in the file footer

func (*Segment) Path 

func (s *Segment) Path() string

Path returns the path of this segment on disk

func (*Segment) ResetBytesRead 

func (s *Segment) ResetBytesRead(val uint64)

Implements the segment.DiskStatsReporter interface Only the persistedSegment type implments the interface, as the intention is to retrieve the bytes read from the on-disk segment as part of the current query.

func (*Segment) SectionsIndexOffset 

func (s *Segment) SectionsIndexOffset() uint64

SectionsIndexOffset returns the sections index offset in the file footer

func (*Segment) Size 

func (s *Segment) Size() int

func (*Segment) StoredIndexOffset 

func (s *Segment) StoredIndexOffset() uint64

StoredIndexOffset returns the stored value index offset in the file footer

func (*Segment) ThesaurusAddr 

func (s *Segment) ThesaurusAddr(name string) (uint64, error)

ThesaurusAddr is a helper function to compute the file offset where the thesaurus is stored with the specified name.

func (*Segment) VectorAddr 

func (s *Segment) VectorAddr(name string) (uint64, error)

VectorAddr is a helper function to compute the file offset where the vector index is stored for the specified field.

func (*Segment) Version 

func (s *Segment) Version() uint32

Version returns the file version in the file footer

type SegmentBase 

type SegmentBase struct {
	// contains filtered or unexported fields
}

SegmentBase is a memory only, read-only implementation of the segment.Segment interface, using zap's data representation.

func InitSegmentBase 

func InitSegmentBase(mem []byte, memCRC uint32, chunkMode uint32, numDocs uint64,
	storedIndexOffset uint64, sectionsIndexOffset uint64,
	config map[string]interface{}) (*SegmentBase, error)

func (*SegmentBase) AddNestedDocuments 

func (sb *SegmentBase) AddNestedDocuments(drops *roaring.Bitmap) *roaring.Bitmap

AddNestedDocuments returns a bitmap containing the original root document numbers in drops, plus any descendant document numbers for each dropped root document. The drops parameter represents a set of root document numbers to be dropped, and the returned bitmap includes both the original drops and all their descendants (if any). NOTE: This method MODIFIES the drops bitmap in place. NOTE: This method EXPECTS that the drops bitmap contains ONLY root document numbers.

func (*SegmentBase) AddRef 

func (sb *SegmentBase) AddRef()

func (*SegmentBase) Ancestors 

func (sb *SegmentBase) Ancestors(docNum uint64, prealloc []index.AncestorID) []index.AncestorID

Ancestors returns a slice of document numbers representing the ancestors of the specified document (docNum) within the segment. If the document has no ancestors, a slice containing only the document number itself is returned. The prealloc parameter allows for reusing a preallocated slice to avoid additional allocations.

func (*SegmentBase) BytesRead 

func (sb *SegmentBase) BytesRead() uint64

func (*SegmentBase) BytesWritten 

func (sb *SegmentBase) BytesWritten() uint64

func (*SegmentBase) CallbackId 

func (sb *SegmentBase) CallbackId() string

func (*SegmentBase) Close 

func (sb *SegmentBase) Close() (err error)

func (*SegmentBase) Count 

func (sb *SegmentBase) Count() uint64

Count returns the number of documents in this segment.

func (*SegmentBase) CountRoot 

func (sb *SegmentBase) CountRoot(deleted *roaring.Bitmap) uint64

CountRoot returns the number of root documents in the segment, excluding any documents that are marked as deleted in the provided bitmap. The deleted bitmap may contain both root and sub-document numbers, and the method ensures that only root documents are counted.

func (*SegmentBase) DecRef 

func (sb *SegmentBase) DecRef() (err error)

func (*SegmentBase) Dictionary 

func (sb *SegmentBase) Dictionary(field string) (segment.TermDictionary, error)

Dictionary returns the term dictionary for the specified field

func (*SegmentBase) DocID 

func (sb *SegmentBase) DocID(num uint64) ([]byte, error)

DocID returns the value of the _id field for the given docNum

func (*SegmentBase) DocNumbers 

func (sb *SegmentBase) DocNumbers(ids []string) (*roaring.Bitmap, error)

DocNumbers returns a bitset corresponding to the doc numbers of all the provided _id strings

func (*SegmentBase) FieldStats 

func (sb *SegmentBase) FieldStats(field string) (documentCount, sumTotalTermFrequency uint64, ok bool)

FieldStats returns the persisted collection statistics for field.

func (*SegmentBase) Fields 

func (sb *SegmentBase) Fields() []string

Fields returns the field names used in this segment

func (*SegmentBase) FormatVersion 

func (sb *SegmentBase) FormatVersion() uint32

FormatVersion returns the zapx-bluge on-disk format version.

func (*SegmentBase) GetUpdatedFields 

func (s *SegmentBase) GetUpdatedFields() map[string]*index.UpdateFieldInfo

Getter method to retrieve updateFieldInfo within segment base

func (*SegmentBase) Persist 

func (sb *SegmentBase) Persist(path string) error

func (*SegmentBase) ResetBytesRead 

func (sb *SegmentBase) ResetBytesRead(val uint64)

func (*SegmentBase) SetUpdatedFields 

func (s *SegmentBase) SetUpdatedFields(updatedFields map[string]*index.UpdateFieldInfo)

Setter method to store updateFieldInfo within segment base

func (*SegmentBase) Size 

func (sb *SegmentBase) Size() int

func (*SegmentBase) Thesaurus 

func (sb *SegmentBase) Thesaurus(name string) (segment.Thesaurus, error)

Thesaurus returns the thesaurus with the specified name, or an empty thesaurus if not found.

func (*SegmentBase) VectorPayload added in v0.6.0 

func (sb *SegmentBase) VectorPayload(field string) (VectorPayload, error)

VectorPayload returns the opaque native payload for one field. It is kept on SegmentBase so a higher-level backend can open the native index without making zapx depend on that backend's runtime.

func (*SegmentBase) VisitDocValues 

func (sb *SegmentBase) VisitDocValues(localDocNum uint64, fields []string,
	visitor index.DocValueVisitor, dvsIn segment.DocVisitState) (
	segment.DocVisitState, error)

VisitDocValues is an implementation of the DocValueVisitable interface

func (*SegmentBase) VisitStoredFields 

func (sb *SegmentBase) VisitStoredFields(num uint64, visitor segment.StoredFieldValueVisitor) error

VisitStoredFields invokes the StoredFieldValueVisitor for each stored field for the specified doc number

func (*SegmentBase) VisitableDocValueFields 

func (sb *SegmentBase) VisitableDocValueFields() ([]string, error)

VisitableDocValueFields returns the list of fields with persisted doc value terms ready to be visitable using the VisitDocumentFieldTerms method.

func (*SegmentBase) WriteTo 

func (sb *SegmentBase) WriteTo(w io.Writer) (int64, error)

WriteTo is an implementation of io.WriterTo interface.

type Synonym 

type Synonym struct {
	// contains filtered or unexported fields
}

Synonym represents a single synonym, containing the term, synonymID, and document number.

func (*Synonym) Number 

func (s *Synonym) Number() uint32

Number returns the document number of the Synonym.

func (*Synonym) Size 

func (p *Synonym) Size() int

Size returns the memory size of the Synonym, including the length of the term string.

func (*Synonym) Term 

func (s *Synonym) Term() string

Term returns the term of the Synonym.

type SynonymsIterator 

type SynonymsIterator struct {
	Actual   roaring64.IntPeekable64
	ActualBM *roaring64.Bitmap
	// contains filtered or unexported fields
}

SynonymsIterator provides a way to iterate through the synonyms list.

func (*SynonymsIterator) Next 

func (i *SynonymsIterator) Next() (segment.Synonym, error)

Next returns the next Synonym in the iteration or an error if the end is reached.

func (*SynonymsIterator) Size 

func (i *SynonymsIterator) Size() int

type SynonymsList 

type SynonymsList struct {
	// contains filtered or unexported fields
}

SynonymsList represents a list of synonyms for a term, stored in a Roaring64 bitmap.

func (*SynonymsList) Iterator 

func (s *SynonymsList) Iterator(prealloc segment.SynonymsIterator) segment.SynonymsIterator

Iterator creates and returns a SynonymsIterator for the SynonymsList. If the synonyms bitmap is nil, it returns an empty iterator.

func (*SynonymsList) Size 

func (p *SynonymsList) Size() int

type Thesaurus 

type Thesaurus struct {
	// contains filtered or unexported fields
}

Thesaurus is the zap representation of a Thesaurus

func (*Thesaurus) AutomatonIterator 

func (t *Thesaurus) AutomatonIterator(a segment.Automaton,
	startKeyInclusive, endKeyExclusive []byte) segment.ThesaurusIterator

AutomatonIterator returns an iterator which only visits terms having the the vellum automaton and start/end key range

func (*Thesaurus) Contains 

func (t *Thesaurus) Contains(key []byte) (bool, error)

func (*Thesaurus) SynonymsList 

func (t *Thesaurus) SynonymsList(term []byte, except *roaring.Bitmap, prealloc segment.SynonymsList) (segment.SynonymsList, error)

SynonymsList returns the synonyms list for the specified term

type ThesaurusIterator 

type ThesaurusIterator struct {
	// contains filtered or unexported fields
}

ThesaurusIterator is an iterator for term dictionary

func (*ThesaurusIterator) Next 

func (i *ThesaurusIterator) Next() (*index.ThesaurusEntry, error)

Next returns the next entry in the dictionary

type VectorMergeInput added in v0.6.0 

type VectorMergeInput struct {
	Payload    VectorPayload
	NewDocNums []uint64
}

type VectorPayload added in v0.6.0 

type VectorPayload struct {
	Backend    string
	Dimensions uint32
	Similarity string
	DocIDs     []uint32
	Data       []byte
}

VectorPayload is the backend-neutral envelope stored in a segment. The Data field is opaque to zapx; the backend owns its serialization format.

type VectorRecord added in v0.6.0 

type VectorRecord struct {
	DocNum     uint32
	Values     []float32
	Similarity string
}

VectorRecord is one vector value in a segment-local document number.

type VectorSegmentBackend added in v0.6.0 

type VectorSegmentBackend interface {
	SegmentVectorBackend()
	Name() string
	BuildVectorPayload(field string, records []VectorRecord) (VectorPayload, error)
	MergeVectorPayload(field string, inputs []VectorMergeInput) (VectorPayload, error)
}

VectorSegmentBackend is implemented by a native vector backend that can build and merge one serialized index per text segment.

type ZapPlugin 

type ZapPlugin struct{}

ZapPlugin implements the Plugin interface of the blevesearch/scorch_segment_api pkg

func (*ZapPlugin) Merge 

func (z *ZapPlugin) Merge(segments []seg.Segment, drops []*roaring.Bitmap, path string,
	closeCh chan struct{}, s seg.StatsReporter) (
	[][]uint64, uint64, error)

Merge takes a slice of segments and bit masks describing which documents may be dropped, and creates a new segment containing the remaining data. This new segment is built at the specified path.

func (*ZapPlugin) MergeUsing 

func (z *ZapPlugin) MergeUsing(segments []seg.Segment, drops []*roaring.Bitmap, path string,
	closeCh chan struct{}, s seg.StatsReporter, config map[string]interface{}) (
	[][]uint64, uint64, error)

func (*ZapPlugin) New 

func (z *ZapPlugin) New(results []*blugeidx.Document) (
	segment.Segment, uint64, error)

New creates an in-memory zap-encoded SegmentBase from a set of Documents

func (*ZapPlugin) NewUsing 

func (z *ZapPlugin) NewUsing(results []*blugeidx.Document, config map[string]interface{}) (
	segment.Segment, uint64, error)

func (*ZapPlugin) Open 

func (z *ZapPlugin) Open(path string) (segment.Segment, error)

Open returns a zap impl of a segment

func (*ZapPlugin) OpenUsing 

func (z *ZapPlugin) OpenUsing(path string, config map[string]interface{}) (segment.Segment, error)

OpenUsing returns a zap impl of a segment which tracks some config values during the its lifetime.

func (*ZapPlugin) Type 

func (*ZapPlugin) Type() string

func (*ZapPlugin) Version 

func (*ZapPlugin) Version() uint32

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.