ingest

Documentation

Overview

Package ingest turns uploaded documents (PDF, DOCX, XLSX, Markdown, text) into plain text and splits that text into coherent chunks suitable for storing as memories.

Index

• Constants
• Variables
• func Chunk(text string, maxRunes int) []string
• func Extract(filename string, data []byte) (string, error)
• func Supported() []string

Constants

const DefaultChunkRunes = 1200

DefaultChunkRunes is the target chunk size when the caller passes 0.

Variables

var ErrUnsupported = errors.New("unsupported document type")

ErrUnsupported is returned when a file's extension is not one we can extract.

Functions

func Chunk

func Chunk(text string, maxRunes int) []string

Chunk splits text into chunks of up to maxRunes runes, preferring paragraph boundaries and falling back to whitespace within an oversized paragraph so each chunk stays a coherent unit of meaning.

func Extract

func Extract(filename string, data []byte) (string, error)

Extract pulls plain text out of a document based on its filename extension.

func Supported

func Supported() []string

Supported returns the file extensions Extract can handle.