Version: v0.0.0-...-c88c75b Latest Latest

This package is not in the latest version of its module.

Go to latest
Published: Mar 24, 2022 License: BSD-3-Clause Imports: 21 Imported by: 0




This section is empty.


This section is empty.


func ConcatN

func ConcatN(destdir string, srcdirs []string) error

func FiveLines

func FiveLines(b []byte, pos int) [5]string

FiveLines returns five \n-separated lines surrounding pos. The first two lines are context above the line containing pos (which is always element [2]), the last two lines are context above that line.


type DeltaReader

type DeltaReader struct {
	// contains filtered or unexported fields

A DeltaReader reads up to 256 deltas at a time (i.e. one TurboPFor block), which is useful for copying index data in a windowed fashion (for merging).

func NewDeltaReader

func NewDeltaReader() *DeltaReader

func (*DeltaReader) Read

func (dr *DeltaReader) Read() []uint32

Read returns up to 256 uint32 deltas.

When all deltas have been read, Read returns nil.

The first Read call after Reset returns a non-nil result.

func (*DeltaReader) Reset

func (dr *DeltaReader) Reset(meta *MetaEntry, data []byte)

Reset positions the reader on a posting list.

type DocidReader

type DocidReader struct {
	Count int
	// contains filtered or unexported fields

func (*DocidReader) All

func (dr *DocidReader) All() io.Reader

func (*DocidReader) Close

func (dr *DocidReader) Close() error

func (*DocidReader) Lookup

func (dr *DocidReader) Lookup(docid uint32) (string, error)

type ErrFunc

type ErrFunc func(path string, info os.FileInfo, err error) error

An ErrFunc is called for each error adding a file to the index, allowing the caller to take action (e.g. delete the not added files to conserve disk space). If the ErrFunc returns a non-nil error, indexing is aborted.

type IgnoreFunc

type IgnoreFunc func(info os.FileInfo, dir string, fn string) error

An IgnoreFunc returns a non-nil error describing the reason why the file should not be added to the index, or nil if it should be added.

type Index

type Index struct {
	DocidMap *DocidReader  // docid → filename mapping
	Docid    *PForReader   // docids for all trigrams
	Pos      *PForReader   // positions for all trigrams
	Posrel   *PosrelReader // position relationships for all trigrams
	// contains filtered or unexported fields

func Open

func Open(dir string) (*Index, error)

func (*Index) Close

func (i *Index) Close() error

func (*Index) Matches

func (i *Index) Matches(t Trigram) ([]Match, error)

func (*Index) PostingQuery

func (i *Index) PostingQuery(q *Query) (docids []uint32)

func (*Index) QueryPositional

func (i *Index) QueryPositional(query string) ([]Match, error)

type Match

type Match struct {
	Docid    uint32
	Position uint32 // byte offset of the trigram within the document

type MetaEntry

type MetaEntry struct {
	Trigram    Trigram
	Entries    uint32 // number of entries (excluding padding)
	OffsetData int64  // delta offset within the corresponding .data or .turbopfor file

A MetaEntry specifies the offset (in the corresponding data file) and number of entries for each trigram.

func (*MetaEntry) Marshal

func (me *MetaEntry) Marshal(b []byte)

func (*MetaEntry) Unmarshal

func (me *MetaEntry) Unmarshal(b []byte)

type PForReader

type PForReader struct {
	// contains filtered or unexported fields

func (*PForReader) Close

func (sr *PForReader) Close() error

func (*PForReader) Data

func (sr *PForReader) Data(t Trigram) (data io.Reader, entries int, _ error)

Streams returns a reader for the specified trigram data.

func (*PForReader) Deltas

func (sr *PForReader) Deltas(t Trigram) ([]uint32, error)

func (*PForReader) MetaEntry

func (sr *PForReader) MetaEntry(trigram Trigram) (*MetaEntry, error)

type PosrelReader

type PosrelReader struct {
	// contains filtered or unexported fields

func (*PosrelReader) Close

func (pr *PosrelReader) Close() error

func (*PosrelReader) DataBytes

func (pr *PosrelReader) DataBytes(t Trigram) ([]byte, error)

func (*PosrelReader) MetaEntry

func (pr *PosrelReader) MetaEntry(trigram Trigram) (*MetaEntry, error)

type Query

type Query struct {
	Op      QueryOp
	Trigram []string
	Sub     []*Query

A Query is a matching machine, like a regular expression, that matches some text and not other text. When we compute a Query from a regexp, the Query is a conservative version of the regexp: it matches everything the regexp would match, and probably quite a bit more. We can then filter target files by whether they match the Query (using a trigram index) before running the comparatively more expensive regexp machinery.

func RegexpQuery

func RegexpQuery(re *syntax.Regexp) *Query

RegexpQuery returns a Query for the given regexp.

func (*Query) String

func (q *Query) String() string

type QueryOp

type QueryOp int
const (
	QAll  QueryOp = iota // Everything matches
	QNone                // Nothing matches
	QAnd                 // All in Sub and Trigram must match
	QOr                  // At least one in Sub or Trigram must match

type SuccessFunc

type SuccessFunc func(path string, info os.FileInfo) error

A SuccessFunc is called for each file which was successfully added to the index.

type Trigram

type Trigram uint32

type Writer

type Writer struct {
	// contains filtered or unexported fields

func Create

func Create(dir string) (*Writer, error)

func (*Writer) AddDir

func (w *Writer) AddDir(dir string, trimPrefix string, ignored IgnoreFunc, ef ErrFunc, sf SuccessFunc) error

func (*Writer) AddFile

func (w *Writer) AddFile(fn, name string) error

func (*Writer) Flush

func (w *Writer) Flush() error

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL