internal

Documentation

Index

• func Compact(max int, instructions []int, contents [][]int) []int
• type Encoder
• type StopFilter
  • func NewStopFilter(stops ...string) *StopFilter
  • func (f *StopFilter) Filter(content string) (string, bool)
  • func (f *StopFilter) String() string

Functions

func Compact

func Compact(max int, instructions []int, contents [][]int) []int

Compact will use the provided encoder to encode the combination of prefix, content and suffix, producing a slice of tokens that has the following properties:

1. The number of tokens is less than or equal to max. If this is not possible, nil is returned.
2. All of the tokens from the prefix and suffix are included.
3. As many tokens as possible from the content are included.

This is done by dropping lines of text from the front of the content until it fits. Some assumptions are made, here: that text is peppered with newlines, that newlines have a single unambiguous encoding, and that the encoder will produce the same encoding for the same input. This is true for LLaMA encoders.

Types

type Encoder

type Encoder interface {
	Encode(string) []int
}

An Encoder knows how to encode a string into a slice of tokens.

type StopFilter

type StopFilter struct {
	// contains filtered or unexported fields
}

A StopFilter buffers content that may contain one of the stops provided to its constructor.

func NewStopFilter

func NewStopFilter(stops ...string) *StopFilter

NewStopFilter constructs a stop filter with the provided set of case sensitive stop strings.

func (*StopFilter) Filter

func (f *StopFilter) Filter(content string) (string, bool)

Filter will append content to its internal buffer and return the portion of the buffer that cannot contain any of its stops. Filter will return true if the buffer contains any of its stops, false otherwise. Once a stop has been fully matched, the stop filter buffer will be emptied.

func (*StopFilter) String

func (f *StopFilter) String() string

String returns the content of the internal buffer, which might partially match one of the stops.