tantivy_go

package module
v0.1.9-jieba Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 6, 2024 License: MIT Imports: 5 Imported by: 1

README

Go Tantivy Bindings

This project provides Go bindings for the Tantivy search engine library. Tantivy is a full-text search engine library written in Rust, and this project aims to make its powerful search capabilities available to Go developers.

The library is thread safe and can be used in a concurrent environment

Disclaimer

This project is still in development and might not be ready for production use. The API is subject to change, and the project may contain bugs. Please use with caution.

Installation

go get github.com/anyproto/tantivy-go

Ensure your libraries are in your ld path.

Example Run
  • Run make download-tantivy-all inside the rust folder
  • Run main.go in the example folder

Development

Development and compilation are done on MacBooks and for Apple platforms. Therefore, the development steps provided are for macOS.

Install environment
  • Install rustup
  • Install Rust architectures: make setup
  • Add Android libraries to your path: export PATH=$PATH:$ANDROID_HOME/tools:$ANDROID_HOME/emulator:$ANDROID_HOME/platform-tools:$ANDROID_HOME/ndk/25.2.9519653/toolchains/llvm/prebuilt/darwin-x86_64/bin
  • Install Windows compiler: brew install mingw-w64
  • Install musl: brew tap messense/macos-cross-toolchains && brew install x86_64-unknown-linux-musl
Install rust libraries

Run inside the rust folder:

make install-all - install release versions for all platforms

make install-debug-all - install debug versions for all platforms

make install-ARCH-GOOS - install release version for ARCH GOOS

make install-debug-ARCH-GOOS - install debug version for ARCH GOOS

GCC support

To be done

Validate min macos version

otool -l libtantivy_go.a | rg LC_BUILD_VERSION -A4 | rg minos | sort | uniq -c Expected output:

 880     minos 11.0
Possible troubleshooting

If you experience SIGSEGV issues with musl or windows, try adding these flags to the linker:

-extldflags '-static -Wl,-z stack-size=1000000'
Nix

flake.nix currently provides two versions of devShell: musl and gcc.

This command will make a bash shell with all required build dependencies:

nix develop .

Each devShell also contains a script which:

  • builds rust into .a lib
  • copies it to ../anytype-heart
  • builds anytype-heart grpcServer
  • copies grpcServer to ../anytype-ts anytypeHelper

[!TIP] To enable musl, set musl = true; in flake.nix.

If you want to debug tantivy from anytype-ts, with musl or gcc, this scripts automates all the flow.

All together it would look like:

nix develop .
tantivy_compile_all_gcc
# or
tantivy_compile_all_musl

To check that it works, run anytype-ts and try to search something.

[!NOTE] MacOS (Darwin) nix shell is not supported yet

Documentation

Index

Constants

View Source 
const (
	// IndexRecordOptionBasic specifies that only basic indexing information should be used.
	IndexRecordOptionBasic = iota
	// IndexRecordOptionWithFreqs specifies that indexing should include term frequencies.
	IndexRecordOptionWithFreqs
	// IndexRecordOptionWithFreqsAndPositions specifies that indexing should include term frequencies and term positions.
	IndexRecordOptionWithFreqsAndPositions
)
View Source 
const (
	Arabic     = "ar"
	Danish     = "da"
	Dutch      = "nl"
	English    = "en"
	Finnish    = "fi"
	French     = "fr"
	German     = "de"
	Greek      = "el"
	Hungarian  = "hu"
	Italian    = "it"
	Norwegian  = "no"
	Portuguese = "pt"
	Romanian   = "ro"
	Russian    = "ru"
	Spanish    = "es"
	Swedish    = "sv"
	Tamil      = "ta"
	Turkish    = "tr"
)
View Source 
const DefaultTokenizer = "default"
View Source 
const TokenizerEdgeNgram = "edge_ngram"
View Source 
const TokenizerJieba = "jieba"
View Source 
const TokenizerNgram = "ngram"
View Source 
const TokenizerRaw = "raw"
View Source 
const TokenizerSimple = "simple"

Variables

This section is empty.

Functions

func GetSearchResults 

func GetSearchResults[T any](
	searchResult *SearchResult,
	schema *Schema,
	f func(json string) (T, error),
	includeFields ...string,
) ([]T, error)

GetSearchResults extracts search results from a SearchResult and converts them into a slice of models.

Parameters:

  • searchResult (*SearchResult): The search results to process.
  • schema (*Schema): The schema to use for converting documents to models.
  • f (func(json string) (T, error)): A function to convert JSON strings to models.
  • includeFields (...string): Optional list of fields to include in the result.

Returns:

  • ([]T, error): A slice of models obtained from the search results, and an error if something goes wrong.

func LibInit 

func LibInit(cleanOnPanic bool, directive ...string) error

LibInit initializes the library with an optional directive.

Parameters:

  • directive: A variadic parameter that allows specifying an initialization directive. If no directive is provided, the default value "info" is used.

Returns: - An error if the initialization fails.

func ToModel 

func ToModel[T any](doc *Document, schema *Schema, includeFields []string, f func(json string) (T, error)) (T, error)

ToModel converts a document to a model of type T using the provided schema and a conversion function.

Parameters:

  • doc: the document to convert
  • schema: the schema to use for converting the document to JSON
  • includeFields: optional fields to include in the JSON output
  • f: a function that takes a JSON string and converts it to a model of type T

Returns:

  • T: the model of type T resulting from the conversion
  • error: an error if the conversion fails, or nil if the operation is successful

Types

type Document 

type Document struct {
	// contains filtered or unexported fields
}

func NewDocument 

func NewDocument() *Document

NewDocument creates a new instance of Document.

Returns:

  • *Document: a pointer to a newly created Document instance.

func (*Document) AddField 

func (d *Document) AddField(fieldName, fieldValue string, tc *TantivyContext) error

AddField adds a field with the specified name and value to the document using the given index. Returns an error if adding the field fails.

Parameters:

  • fieldName: the name of the field to add
  • fieldValue: the value of the field to add
  • index: the index to use for adding the field

Returns:

  • error: an error if adding the field fails, or nil if the operation is successful

func (*Document) Free 

func (d *Document) Free()

func (*Document) FreeStrings added in v0.2.0 

func (d *Document) FreeStrings()

func (*Document) ToJson 

func (d *Document) ToJson(schema *Schema, includeFields ...string) (string, error)

ToJson converts the document to its JSON representation based on the provided schema. Optionally, specific fields can be included in the JSON output.

Parameters:

  • schema: the schema to use for converting the document to JSON
  • includeFields: optional variadic parameter specifying the fields to include in the JSON output

Returns:

  • string: the JSON representation of the document
  • error: an error if the conversion fails, or nil if the operation is successful

type Schema 

type Schema struct {
	// contains filtered or unexported fields
}

type SchemaBuilder 

type SchemaBuilder struct {
	// contains filtered or unexported fields
}

func NewSchemaBuilder 

func NewSchemaBuilder() (*SchemaBuilder, error)

NewSchemaBuilder creates a new SchemaBuilder instance. Returns a pointer to the SchemaBuilder and an error if creation fails.

func (*SchemaBuilder) AddTextField 

func (b *SchemaBuilder) AddTextField(
	name string,
	stored bool,
	isText bool,
	isFast bool,
	indexRecordOption int,
	tokenizer string,
) error

AddTextField adds a text field to the schema being built.

Parameters: - name: The name of the field. - stored: Whether the field should be stored in the index. - isText: Whether the field should be treated as tantivy text or string for full-text search. - isFast: Whether the field should be indexed as tantivy quick field. - indexRecordOption: The indexing option to be used (e.g., basic, with frequencies, with frequencies and positions). - tokenizer: The name of the tokenizer to be used for the field.

Returns an error if the field could not be added.

func (*SchemaBuilder) BuildSchema 

func (b *SchemaBuilder) BuildSchema() (*Schema, error)

BuildSchema finalizes the schema building process and returns the resulting Schema. Returns a pointer to the Schema and an error if the schema could not be built.

type SearchContext added in v0.2.0 

type SearchContext interface {
	// GetQuery returns the search query string.
	GetQuery() string
	// GetDocsLimit returns the document limit as a uintptr.
	GetDocsLimit() uintptr
	// WithHighlights returns true if highlights are enabled.
	WithHighlights() bool
	// GetFieldAndWeights returns slices of field names and their corresponding weights.
	GetFieldAndWeights() ([]string, []float32)
}

SearchContext defines the interface for searchContext

type SearchContextBuilder added in v0.2.0 

type SearchContextBuilder struct {
	// contains filtered or unexported fields
}

SearchContextBuilder is a builder structure for creating searchContext.

func NewSearchContextBuilder added in v0.2.0 

func NewSearchContextBuilder() *SearchContextBuilder

NewSearchContextBuilder creates a new instance of SearchContextBuilder.

func (*SearchContextBuilder) AddField added in v0.2.0 

func (b *SearchContextBuilder) AddField(field string, weight float32) *SearchContextBuilder

AddField adds a field with the specified weight to searchContext.

func (*SearchContextBuilder) AddFieldDefaultWeight added in v0.2.0 

func (b *SearchContextBuilder) AddFieldDefaultWeight(field string) *SearchContextBuilder

AddFieldDefaultWeight adds a field with a default weight of 1.0 to searchContext.

func (*SearchContextBuilder) Build added in v0.2.0 

func (b *SearchContextBuilder) Build() SearchContext

Build returns the constructed searchContext as an interface.

func (*SearchContextBuilder) SetDocsLimit added in v0.2.0 

func (b *SearchContextBuilder) SetDocsLimit(limit uintptr) *SearchContextBuilder

SetDocsLimit sets the docsLimit for searchContext.

func (*SearchContextBuilder) SetQuery added in v0.2.0 

func (b *SearchContextBuilder) SetQuery(query string) *SearchContextBuilder

SetQuery sets the query for searchContext.

func (*SearchContextBuilder) SetWithHighlights added in v0.2.0 

func (b *SearchContextBuilder) SetWithHighlights(withHighlights bool) *SearchContextBuilder

SetWithHighlights sets the withHighlights flag for searchContext.

type SearchResult 

type SearchResult struct {
	// contains filtered or unexported fields
}

func (*SearchResult) Free 

func (r *SearchResult) Free()

func (*SearchResult) Get 

func (r *SearchResult) Get(index uint64) (*Document, error)

Get retrieves a document from the search result at the specified index.

Parameters: - index: The index of the document to retrieve.

Returns: - A pointer to the Document if successful, or nil if not found. - An error if there was an issue retrieving the document.

func (*SearchResult) GetSize 

func (r *SearchResult) GetSize() (uint64, error)

GetSize returns the number of documents in the search result.

Returns: - The size of the search result if successful. - An error if there was an issue getting the size.

type TantivyContext added in v0.1.0 

type TantivyContext struct {
	// contains filtered or unexported fields
}

func NewTantivyContextWithSchema added in v0.1.0 

func NewTantivyContextWithSchema(path string, schema *Schema) (*TantivyContext, error)

NewTantivyContextWithSchema creates a new instance of TantivyContext with the provided schema.

Parameters:

  • path: The path to the index as a string.
  • schema: A pointer to the Schema to be used.

Returns:

  • *TantivyContext: A pointer to a newly created TantivyContext instance.
  • error: An error if the index creation fails.

func (*TantivyContext) AddAndConsumeDocuments added in v0.1.0 

func (tc *TantivyContext) AddAndConsumeDocuments(docs ...*Document) error

AddAndConsumeDocuments adds and consumes the provided documents to the index.

Parameters:

  • docs: A variadic parameter of pointers to Document to be added and consumed.

Returns:

  • error: An error if adding and consuming the documents fails.

func (*TantivyContext) DeleteDocuments added in v0.1.0 

func (tc *TantivyContext) DeleteDocuments(field string, deleteIds ...string) error

DeleteDocuments deletes documents from the index based on the specified field and IDs.

Parameters:

  • field: The field name to match against the document IDs.
  • deleteIds: A variadic parameter of document IDs to be deleted.

Returns:

  • error: An error if deleting the documents fails.

func (*TantivyContext) Free added in v0.1.0 

func (tc *TantivyContext) Free()

func (*TantivyContext) NumDocs added in v0.1.0 

func (tc *TantivyContext) NumDocs() (uint64, error)

NumDocs returns the number of documents in the index.

Returns:

  • uint64: The number of documents.
  • error: An error if retrieving the document count fails.

func (*TantivyContext) RegisterTextAnalyzerEdgeNgram added in v0.1.0 

func (tc *TantivyContext) RegisterTextAnalyzerEdgeNgram(tokenizerName string, minGram, maxGram uintptr, limit uintptr) error

RegisterTextAnalyzerEdgeNgram registers a text analyzer using edge n-grams with the index.

Parameters:

  • tokenizerName (string): The name of the tokenizer to be used.
  • minGram (uintptr): The minimum length of the edge n-grams.
  • maxGram (uintptr): The maximum length of the edge n-grams.
  • limit (uintptr): The maximum number of edge n-grams to generate.

Returns:

  • error: An error if the registration fails.

func (*TantivyContext) RegisterTextAnalyzerJieba added in v0.2.0 

func (tc *TantivyContext) RegisterTextAnalyzerJieba(tokenizerName string, textLimit uintptr) error

RegisterTextAnalyzerJieba registers a jieba text analyzer with the index.

Parameters:

  • tokenizerName (string): The name of the tokenizer to be used.
  • textLimit (uintptr): The limit on the length of the text to be analyzed.

Returns:

  • error: An error if the registration fails.

func (*TantivyContext) RegisterTextAnalyzerNgram added in v0.1.0 

func (tc *TantivyContext) RegisterTextAnalyzerNgram(tokenizerName string, minGram, maxGram uintptr, prefixOnly bool) error

RegisterTextAnalyzerNgram registers a text analyzer using N-grams with the index.

Parameters:

  • tokenizerName (string): The name of the tokenizer to be used.
  • minGram (uintptr): The minimum length of the n-grams.
  • maxGram (uintptr): The maximum length of the n-grams.
  • prefixOnly (bool): Whether to generate only prefix n-grams.

Returns:

  • error: An error if the registration fails.

func (*TantivyContext) RegisterTextAnalyzerRaw added in v0.1.0 

func (tc *TantivyContext) RegisterTextAnalyzerRaw(tokenizerName string) error

RegisterTextAnalyzerRaw registers a raw text analyzer with the index.

Parameters:

  • tokenizerName (string): The name of the raw tokenizer to be used.

Returns:

  • error: An error if the registration fails.

func (*TantivyContext) RegisterTextAnalyzerSimple added in v0.1.0 

func (tc *TantivyContext) RegisterTextAnalyzerSimple(tokenizerName string, textLimit uintptr, lang string) error

RegisterTextAnalyzerSimple registers a simple text analyzer with the index.

Parameters:

  • tokenizerName (string): The name of the tokenizer to be used.
  • textLimit (uintptr): The limit on the length of the text to be analyzed.
  • lang (string): The language code for the text analyzer.

Returns:

  • error: An error if the registration fails.

func (*TantivyContext) Search added in v0.1.0 

func (tc *TantivyContext) Search(sCtx SearchContext) (*SearchResult, error)

Search performs a search query on the index and returns the search results.

Parameters:

  • sCtx (SearchContext): The context for the search, containing query string, document limit, highlight option, and field weights.

Returns:

  • *SearchResult: A pointer to the SearchResult containing the search results.
  • error: An error if the search fails.

Source Files

View all Source files

Directories

Path Synopsis
go module

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.