pipeline

Documentation

Index

• type ChunkedFile
• type EmbeddedChunk
• type Result
  • func Run(source string, workers int, chunkSize int, chunkOverlap int, maxMem int, ...) ([]Result, error)

Types

type ChunkedFile

type ChunkedFile struct {
	Path   string
	Chunks []EmbeddedChunk
}

ChunkedFile represents a file that has been chunked and embedded.

type EmbeddedChunk

type EmbeddedChunk struct {
	Content string
	Index   int
	Vector  []float32
}

EmbeddedChunk holds a chunk's text and its vector embedding.

type Result

type Result struct {
	Path       string
	ChunkCount int
	Embedded   bool
}

Result holds the output of the full pipeline for a single file.

func Run

func Run(source string, workers int, chunkSize int, chunkOverlap int, maxMem int, emb embedder.Embedder, db *storage.DB) ([]Result, error)

Run executes the crawl-chunk-embed-storage pipeline. If emb is nil, it skips the embedding step.