postprocessors

Documentation

Overview

Package postprocessors is basically the same as package transformers, but used at a different stage of the RAG pipeline

Index

• Variables
• type CohereRerankPostprocessor
  • func (c *CohereRerankPostprocessor) Transform(ctx context.Context, query string, docs []vs.Document) ([]vs.Document, error)
• type ContentFilterPostprocessor
  • func (c *ContentFilterPostprocessor) Transform(ctx context.Context, _ string, docs []vs.Document) ([]vs.Document, error)
• type ContentSubstringFilterPostprocessor
  • func (c *ContentSubstringFilterPostprocessor) Transform(_ context.Context, _ string, docs []vs.Document) ([]vs.Document, error)
• type Postprocessor
  • func GetPostprocessor(name string) (Postprocessor, error)
• type SimilarityPostprocessor
  • func (s *SimilarityPostprocessor) Transform(_ context.Context, query string, docs []vs.Document) ([]vs.Document, error)
• type TransformerWrapper
  • func NewTransformerWrapper(transformer types.DocumentTransformer) *TransformerWrapper
  • func (t *TransformerWrapper) Transform(ctx context.Context, query string, docs []vs.Document) ([]vs.Document, error)

Variables

var PostprocessorMap = map[string]Postprocessor{
	"extra_metadata":                  NewTransformerWrapper(&transformers.ExtraMetadata{}),
	"keywords":                        NewTransformerWrapper(&transformers.KeywordExtractor{}),
	"filter_markdown_docs_no_content": NewTransformerWrapper(&transformers.FilterMarkdownDocsNoContent{}),
	"similarity":                      &SimilarityPostprocessor{},
	"content_substring_filter":        &ContentSubstringFilterPostprocessor{},
	"content_filter":                  &ContentFilterPostprocessor{},
	"cohere_rerank":                   &CohereRerankPostprocessor{},
}

Types

type CohereRerankPostprocessor

type CohereRerankPostprocessor struct {
	ApiKey string `json:"apiKey" yaml:"apiKey"`
	Model  string
	TopN   int
}

func (*CohereRerankPostprocessor) Transform

func (c *CohereRerankPostprocessor) Transform(ctx context.Context, query string, docs []vs.Document) ([]vs.Document, error)

type ContentFilterPostprocessor

type ContentFilterPostprocessor struct {
	Question string // Question about the content, that can be answered with yes or no
	Include  bool   // Whether to include or exclude the documents for which the answer is yes
	LLM      llm.LLM
}

func (*ContentFilterPostprocessor) Transform

func (c *ContentFilterPostprocessor) Transform(ctx context.Context, _ string, docs []vs.Document) ([]vs.Document, error)

type ContentSubstringFilterPostprocessor

type ContentSubstringFilterPostprocessor struct {
	Contains    []string
	NotContains []string
}

func (*ContentSubstringFilterPostprocessor) Transform

func (c *ContentSubstringFilterPostprocessor) Transform(_ context.Context, _ string, docs []vs.Document) ([]vs.Document, error)

type Postprocessor

type Postprocessor interface {
	Transform(ctx context.Context, query string, docs []vs.Document) ([]vs.Document, error)
}

Postprocessor is similar to types.DocumentTransformer, but can take into account the retrieval query

func GetPostprocessor

func GetPostprocessor(name string) (Postprocessor, error)

type SimilarityPostprocessor

type SimilarityPostprocessor struct {
	Threshold float32
}

func (*SimilarityPostprocessor) Transform

func (s *SimilarityPostprocessor) Transform(_ context.Context, query string, docs []vs.Document) ([]vs.Document, error)

type TransformerWrapper

type TransformerWrapper struct {
	types.DocumentTransformer
}

func NewTransformerWrapper

func NewTransformerWrapper(transformer types.DocumentTransformer) *TransformerWrapper

func (*TransformerWrapper) Transform

func (t *TransformerWrapper) Transform(ctx context.Context, query string, docs []vs.Document) ([]vs.Document, error)