chunks

Documentation

Index

• func ChunkText(text string, chunkSize, overlap int) []string
• func ChunkWithMarkdownHierarchy(content string) []string
• func SplitMarkdownBySections(markdown string) []string
• func SplitTextWithDelimiter(text string, delimiter string) []string
• type MarkdownChunk
  • func ParseMarkdownHierarchy(content string) []MarkdownChunk

Functions

func ChunkText

func ChunkText(text string, chunkSize, overlap int) []string

ChunkText takes a text string and divides it into chunks of a specified size with a given overlap. It returns a slice of strings, where each string represents a chunk of the original text.

Parameters:

• text: The input text to be chunked.
• chunkSize: The size of each chunk.
• overlap: The amount of overlap between consecutive chunks.

Returns:

• []string: A slice of strings representing the chunks of the original text.

func ChunkWithMarkdownHierarchy

func ChunkWithMarkdownHierarchy(content string) []string

ChunkWithMarkdownHierarchy processes markdown content into formatted chunks with hierarchical context

func SplitMarkdownBySections

func SplitMarkdownBySections(markdown string) []string

SplitMarkdownBySections splits markdown content into sections at header boundaries

func SplitTextWithDelimiter

func SplitTextWithDelimiter(text string, delimiter string) []string

SplitTextWithDelimiter splits the given text using the specified delimiter and returns a slice of strings.

Parameters:

• text: The text to be split.
• delimiter: The delimiter used to split the text.

Returns:

• []string: A slice of strings containing the split parts of the text.

Types

type MarkdownChunk

type MarkdownChunk struct {
	Header         string
	Content        string
	Level          int
	Prefix         string
	ParentLevel    int
	ParentHeader   string
	ParentPrefix   string
	Hierarchy      string
	SimpleMetaData string                 // Additional metadata if needed
	Metadata       map[string]interface{} // additional metadata
	KeyWords       []string               // Keywords that could be extracted from the content
}

MarkdownChunk represents a parsed markdown section with hierarchical context

func ParseMarkdownHierarchy

func ParseMarkdownHierarchy(content string) []MarkdownChunk

ParseMarkdownHierarchy parses the given markdown content and returns a slice of MarkdownChunk structs preserving the hierarchical context