README
¶
loader
Reads documents from local file sources into a uniform loader.Document
(ID, title, source, content, metadata) ready for store.Upload or the
ingest pipeline. Loaders are pure readers: they do not chunk, embed, or
store.
Loaders
| Loader | Format | Dependencies |
|---|---|---|
TextLoader |
.txt |
stdlib |
MarkdownLoader |
.md |
stdlib (heading-aware document splitting) |
CSVLoader |
.csv |
stdlib |
JSONLoader |
.json |
stdlib |
HTMLLoader |
.html |
golang.org/x/net |
PDFLoader |
.pdf |
ledongthuc/pdf (pure Go) |
DocxLoader |
.docx |
stdlib (zip + XML) |
DirectoryLoader |
walks a tree, dispatches by extension | — |
ForExtension(ext) returns the default loader for an extension
(UnsupportedExtError otherwise). DirectoryLoader takes an explicit
extension list, a recursive flag, and optional per-extension overrides:
l, err := loader.NewDirectoryLoader([]string{".md", ".txt"}, true, nil)
docs, err := l.Load(ctx, "/path/to/corpus")
ExtractHTML is exported for direct HTML text extraction.
Documentation
¶
Overview ¶
Package loader reads documents from file-based sources into a uniform in-memory representation ready for chunking and upload.
Loaders are pure readers: they do not embed, chunk, or store anything. The stdlib-only loaders (text, markdown, CSV, JSON, directory) cover the common local-file sources; binary formats (PDF, DOCX, HTML) are separate concerns that may require additional dependencies.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type CSVLoader ¶
type CSVLoader struct {
// Separator overrides the default comma.
Separator rune
// HeaderRow controls whether the first row names the columns.
// Default true.
HeaderRow bool
// IDColumn selects the column used as the document ID. When empty,
// IDs are generated as "<path>:row<N>" (1-based data row).
IDColumn string
// ContentColumns lists the columns joined into the document content.
// When empty, all columns except IDColumn are included.
ContentColumns []string
// Join separates content column values. Default " | ".
Join string
}
CSVLoader loads a CSV file as one document per data row. The first row is treated as a header unless HeaderRow is false.
type DirectoryLoader ¶
type DirectoryLoader struct {
// Extensions lists the file extensions to load, e.g. [".txt", ".md"].
// Required. Matching is case-insensitive.
Extensions []string
// Recursive controls whether subdirectories are descended into.
// Default true.
Recursive bool
// Loaders optionally overrides the loader used for a given extension.
// Unlisted extensions fall back to ForExtension defaults.
Loaders map[string]Loader
}
DirectoryLoader walks a directory tree and loads files by extension, dispatching each file to a Loader. Matching files that fail to load are reported via errors.Join while successfully loaded documents are still returned, so one bad file does not discard the whole directory.
func NewDirectoryLoader ¶
func NewDirectoryLoader(extensions []string, recursive bool, loaders map[string]Loader) (*DirectoryLoader, error)
NewDirectoryLoader validates and normalizes a DirectoryLoader config.
type Document ¶
type Document struct {
// ID is a stable, source-scoped identifier for this document.
ID string
// Title is a human-readable title.
Title string
// Source is the origin the document was loaded from (file path, etc.).
Source string
// Content is the raw text content.
Content string
// Metadata carries loader-specific structured attributes (heading,
// row/column fields, ...).
Metadata map[string]core.Value
}
Document is the unit produced by a Loader: one piece of source content with stable identity and metadata. It is designed to feed directly into store.Upload (Content) with core.Document fields mapped one-to-one.
func NewDocument ¶
NewDocument creates a loader.Document with empty metadata initialized.
type DocxLoader ¶
type DocxLoader struct {
// Parts limits how many ZIP entries are opened; 0 means no limit.
Parts int
}
DocxLoader loads a .docx file (a ZIP of OOXML parts) as a single document by extracting the paragraph text of word/document.xml. Only the standard library is used: no third-party Office dependency is required for text extraction.
type HTMLLoader ¶
type HTMLLoader struct {
// KeepChrome, when true, does not drop nav/header/footer/aside/form
// elements (their text is kept).
KeepChrome bool
}
HTMLLoader loads an HTML file as a single document with its visible text extracted: script/style/nav/header/footer content is dropped, block elements become line breaks, and whitespace runs are normalized.
type JSONLoader ¶
type JSONLoader struct {
// IDField selects the field used as the document ID. When empty or
// absent, IDs are "<path>:<index>".
IDField string
// ContentField selects the field used as the document content. It must
// hold a string. Default "content".
ContentField string
}
JSONLoader loads a JSON file whose top level is either a single object or an array of objects. Each object becomes one document; field names are addressable via dotted paths (e.g. "meta.title") to reach nested values.
type Loader ¶
Loader reads documents from a source. The ref parameter is loader-specific (a file path for most loaders, a directory for DirectoryLoader).
func ForExtension ¶
ForExtension returns the default Loader for a file extension such as ".txt" or ".md". Extensions are case-insensitive. It returns an error for extensions with no default loader; callers can register their own via DirectoryLoader.Loaders.
type MarkdownLoader ¶
type MarkdownLoader struct {
// IncludeHeading, when true, prepends the heading line to each section's
// content so embedded text is self-describing. Default false.
IncludeHeading bool
}
MarkdownLoader loads a Markdown file, splitting it into one document per ATX heading section. The section before the first heading (if non-empty) is emitted as an "intro" document. Each document carries its heading, level, and breadcrumb path in metadata, and gets a slug-based ID suitable for cross-run stability.
type PDFLoader ¶
type PDFLoader struct{}
PDFLoader loads a PDF file as a single document by extracting its plain text. Encrypted or scanned (image-only) PDFs yield an error respectively.
type TextLoader ¶
type TextLoader struct {
// MaxBytes caps the number of bytes read; 0 means no cap.
MaxBytes int64
}
TextLoader loads a plain text file as a single document.
type UnsupportedExtError ¶
type UnsupportedExtError struct {
Ext string
}
UnsupportedExtError is returned when no loader is registered for an extension.
func (*UnsupportedExtError) Error ¶
func (e *UnsupportedExtError) Error() string
Source Files