sources

package
v0.0.0-...-d7aa690 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 18, 2026 License: MIT Imports: 40 Imported by: 0

Documentation

Overview

Package sources defines interfaces and implementations for document sources.

Index

Constants

View Source 
const IngestionScopeMetadata = "mindcli_ingestion_scope"

IngestionScopeMetadata is attached to documents owned by a reconciled multi-document artifact. It lets the indexer remove records that disappear from a later parse without treating their virtual paths as local files.

Variables

This section is empty.

Functions

func NormalizeWebURL 

func NormalizeWebURL(raw string) string

NormalizeWebURL returns the stable HTTP(S) identity shared by browser indexing and deliberate URL captures.

func ParseDocuments 

func ParseDocuments(ctx context.Context, source Source, file FileInfo) ([]*storage.Document, error)

ParseDocuments adapts a Source to the multi-document ingestion contract.

Types

type BrowserOptions 

type BrowserOptions struct {
	IncludeContent   bool
	AllowedDomains   []string
	DeniedDomains    []string
	MaxResponseBytes int64
	RequestTimeout   time.Duration
	FetchConcurrency int
	MaxPages         int
	RetentionDays    int
}

BrowserOptions bounds browser history retention and optional reader-mode page fetching. IncludeContent is false by default.

func DefaultBrowserOptions 

func DefaultBrowserOptions() BrowserOptions

DefaultBrowserOptions returns privacy-preserving, bounded browser defaults.

type BrowserSource 

type BrowserSource struct {
	// contains filtered or unexported fields
}

BrowserSource indexes browser history and bookmarks.

func NewBrowserSource 

func NewBrowserSource(browsers []string) *BrowserSource

NewBrowserSource creates a new browser history source.

func (*BrowserSource) IsDocumentInScope 

func (b *BrowserSource) IsDocumentInScope(file FileInfo, doc *storage.Document) bool

IsDocumentInScope reports whether doc belongs to the scanned browser profile. The fallback recognizes aggregate documents created by older MindCLI versions so the first new index pass removes them.

func (*BrowserSource) MatchesPath 

func (b *BrowserSource) MatchesPath(path string) bool

MatchesPath reports whether this source is configured to handle the path.

func (*BrowserSource) Name 

func (b *BrowserSource) Name() storage.Source

Name returns the source name.

func (*BrowserSource) Parse 

func (b *BrowserSource) Parse(ctx context.Context, file FileInfo) (*storage.Document, error)

Parse reads browser history and returns a document with all entries. Deprecated: browser indexing uses ParseDocuments so each page is searchable independently. Parse remains as a compatibility shim for Source callers.

func (*BrowserSource) ParseDocuments 

func (b *BrowserSource) ParseDocuments(ctx context.Context, file FileInfo) ([]*storage.Document, error)

ParseDocuments returns one independently searchable document per normalized URL in the browser profile.

func (*BrowserSource) ReconciliationScope 

func (b *BrowserSource) ReconciliationScope(file FileInfo) string

ReconciliationScope identifies all virtual documents owned by one browser profile, regardless of whether History or Bookmarks was the scanned path.

func (*BrowserSource) Scan 

func (b *BrowserSource) Scan(ctx context.Context) (<-chan FileInfo, <-chan error)

Scan finds browser profile data and returns one artifact per browser profile. Chrome history and bookmarks are coalesced so a URL that appears in both is emitted as one document.

func (*BrowserSource) SetOptions 

func (b *BrowserSource) SetOptions(options BrowserOptions)

SetOptions configures retention and page fetching for a browser source.

type ClipboardSource 

type ClipboardSource struct {
	// contains filtered or unexported fields
}

ClipboardSource indexes clipboard history. It polls the system clipboard and stores unique text entries.

func NewClipboardSource 

func NewClipboardSource(db *storage.DB, retentionDays int, skipPasswords bool) *ClipboardSource

NewClipboardSource creates a new clipboard source.

func (*ClipboardSource) MatchesPath 

func (c *ClipboardSource) MatchesPath(path string) bool

MatchesPath reports whether this source is configured to handle the path.

func (*ClipboardSource) Name 

func (c *ClipboardSource) Name() storage.Source

Name returns the source name.

func (*ClipboardSource) Parse 

func (c *ClipboardSource) Parse(ctx context.Context, file FileInfo) (*storage.Document, error)

Parse creates a document from the current clipboard content.

func (*ClipboardSource) Scan 

func (c *ClipboardSource) Scan(ctx context.Context) (<-chan FileInfo, <-chan error)

Scan returns the current clipboard content as a single file-like entry. For clipboard, each unique clip is treated as a "file".

type CodeSource 

type CodeSource struct {
	// contains filtered or unexported fields
}

CodeSource indexes bounded, recognized UTF-8 source files from configured repository roots without following symbolic links.

func NewCodeSource 

func NewCodeSource(paths, ignore []string, maxFileBytes int64, maxFiles int) *CodeSource

func (*CodeSource) MatchesPath 

func (c *CodeSource) MatchesPath(path string) bool

func (*CodeSource) Name 

func (c *CodeSource) Name() storage.Source

func (*CodeSource) Parse 

func (c *CodeSource) Parse(ctx context.Context, file FileInfo) (*storage.Document, error)

func (*CodeSource) Scan 

func (c *CodeSource) Scan(ctx context.Context) (<-chan FileInfo, <-chan error)

type DOCXSource 

type DOCXSource struct {
	// contains filtered or unexported fields
}

DOCXSource indexes bounded Office Open XML word-processing documents.

func NewDOCXSource 

func NewDOCXSource(paths, ignore []string, maxFileBytes, maxDecompressedBytes int64) *DOCXSource

func (*DOCXSource) MatchesPath 

func (d *DOCXSource) MatchesPath(path string) bool

func (*DOCXSource) Name 

func (d *DOCXSource) Name() storage.Source

func (*DOCXSource) Parse 

func (d *DOCXSource) Parse(ctx context.Context, file FileInfo) (*storage.Document, error)

func (*DOCXSource) Scan 

func (d *DOCXSource) Scan(ctx context.Context) (<-chan FileInfo, <-chan error)

type EPUBSource 

type EPUBSource struct {
	// contains filtered or unexported fields
}

EPUBSource indexes each spine item in an EPUB as an independent document.

func NewEPUBSource 

func NewEPUBSource(paths, ignore []string, maxFileBytes, maxDecompressedBytes int64) *EPUBSource

func (*EPUBSource) IsDocumentInScope 

func (e *EPUBSource) IsDocumentInScope(file FileInfo, doc *storage.Document) bool

func (*EPUBSource) MatchesPath 

func (e *EPUBSource) MatchesPath(filePath string) bool

func (*EPUBSource) Name 

func (e *EPUBSource) Name() storage.Source

func (*EPUBSource) Parse 

func (e *EPUBSource) Parse(ctx context.Context, file FileInfo) (*storage.Document, error)

func (*EPUBSource) ParseDocuments 

func (e *EPUBSource) ParseDocuments(ctx context.Context, file FileInfo) ([]*storage.Document, error)

func (*EPUBSource) ReconciliationScope 

func (e *EPUBSource) ReconciliationScope(file FileInfo) string

func (*EPUBSource) Scan 

func (e *EPUBSource) Scan(ctx context.Context) (<-chan FileInfo, <-chan error)

type EmailAttachmentOptions 

type EmailAttachmentOptions struct {
	Enabled              bool
	MaxAttachmentBytes   int64
	MaxDecompressedBytes int64
	MaxArchiveDepth      int
}

EmailAttachmentOptions bounds the optional local extraction of textual attachments. ArchiveDepth 1 permits one DOCX/EPUB container; zero disables archive-backed attachment formats.

func DefaultEmailAttachmentOptions 

func DefaultEmailAttachmentOptions() EmailAttachmentOptions

type EmailSource 

type EmailSource struct {
	// contains filtered or unexported fields
}

EmailSource indexes email archives (mbox, maildir, emlx).

func NewEmailSource 

func NewEmailSource(paths, formats []string) *EmailSource

NewEmailSource creates a new email source.

func (*EmailSource) IsDocumentInScope 

func (e *EmailSource) IsDocumentInScope(file FileInfo, doc *storage.Document) bool

func (*EmailSource) MatchesPath 

func (e *EmailSource) MatchesPath(path string) bool

MatchesPath reports whether this source is configured to handle the path.

func (*EmailSource) Name 

func (e *EmailSource) Name() storage.Source

Name returns the source name.

func (*EmailSource) Parse 

func (e *EmailSource) Parse(ctx context.Context, file FileInfo) (*storage.Document, error)

Parse reads an email file and returns the parsed document. For mbox files, the first message is used as the document.

func (*EmailSource) ParseDocuments 

func (e *EmailSource) ParseDocuments(ctx context.Context, file FileInfo) ([]*storage.Document, error)

ParseDocuments returns the email body plus independently searchable textual attachments when extraction is explicitly enabled.

func (*EmailSource) ReconciliationScope 

func (e *EmailSource) ReconciliationScope(file FileInfo) string

func (*EmailSource) Scan 

func (e *EmailSource) Scan(ctx context.Context) (<-chan FileInfo, <-chan error)

Scan walks configured paths and returns email files to index.

func (*EmailSource) SetAttachmentOptions 

func (e *EmailSource) SetAttachmentOptions(options EmailAttachmentOptions)

SetAttachmentOptions enables and bounds attachment extraction.

func (*EmailSource) SetIgnore 

func (e *EmailSource) SetIgnore(patterns []string)

SetIgnore configures path exclusion patterns.

func (*EmailSource) SetMaskSensitivePreview 

func (e *EmailSource) SetMaskSensitivePreview(enabled bool)

SetMaskSensitivePreview controls redaction in preview/metadata fields.

type FileInfo 

type FileInfo struct {
	Path       string
	ModifiedAt int64 // Unix timestamp
	Size       int64
}

FileInfo contains information about a file to be indexed.

type HTMLSource 

type HTMLSource struct {
	// contains filtered or unexported fields
}

HTMLSource indexes HTML pages, MIME web archives, and Safari webarchives.

func NewHTMLSource 

func NewHTMLSource(paths, ignore []string, maxFileBytes, maxDecompressedBytes int64) *HTMLSource

NewHTMLSource creates a bounded local HTML source.

func (*HTMLSource) MatchesPath 

func (h *HTMLSource) MatchesPath(path string) bool

func (*HTMLSource) Name 

func (h *HTMLSource) Name() storage.Source

func (*HTMLSource) Parse 

func (h *HTMLSource) Parse(ctx context.Context, file FileInfo) (*storage.Document, error)

func (*HTMLSource) Scan 

func (h *HTMLSource) Scan(ctx context.Context) (<-chan FileInfo, <-chan error)

type MarkdownSource 

type MarkdownSource struct {
	// contains filtered or unexported fields
}

MarkdownSource indexes markdown files.

func NewMarkdownSource 

func NewMarkdownSource(paths, extensions, ignore []string) *MarkdownSource

NewMarkdownSource creates a new markdown source.

func (*MarkdownSource) MatchesPath 

func (m *MarkdownSource) MatchesPath(path string) bool

MatchesPath reports whether this source is configured to handle the path.

func (*MarkdownSource) Name 

func (m *MarkdownSource) Name() storage.Source

Name returns the source name.

func (*MarkdownSource) Parse 

func (m *MarkdownSource) Parse(ctx context.Context, file FileInfo) (*storage.Document, error)

Parse reads and parses a markdown file into a Document.

func (*MarkdownSource) Scan 

func (m *MarkdownSource) Scan(ctx context.Context) (<-chan FileInfo, <-chan error)

Scan walks configured paths and returns markdown files.

type MultiDocumentSource 

type MultiDocumentSource interface {
	Source

	// ParseDocuments reads a scanned artifact and returns its documents. Every
	// document must have a stable ID so it can be updated independently.
	ParseDocuments(ctx context.Context, file FileInfo) ([]*storage.Document, error)
}

MultiDocumentSource is implemented by sources where one scanned artifact can contain multiple independently searchable documents. Parse remains part of Source so ordinary file-backed sources keep their simple one-file/one-document contract; the indexer prefers ParseDocuments when this interface is present.

type OrgSource 

type OrgSource struct {
	// contains filtered or unexported fields
}

OrgSource indexes top-level Org-mode sections independently.

func NewOrgSource 

func NewOrgSource(paths, ignore []string, maxFileBytes int64) *OrgSource

func (*OrgSource) IsDocumentInScope 

func (o *OrgSource) IsDocumentInScope(file FileInfo, doc *storage.Document) bool

func (*OrgSource) MatchesPath 

func (o *OrgSource) MatchesPath(path string) bool

func (*OrgSource) Name 

func (o *OrgSource) Name() storage.Source

func (*OrgSource) Parse 

func (o *OrgSource) Parse(ctx context.Context, file FileInfo) (*storage.Document, error)

func (*OrgSource) ParseDocuments 

func (o *OrgSource) ParseDocuments(ctx context.Context, file FileInfo) ([]*storage.Document, error)

func (*OrgSource) ReconciliationScope 

func (o *OrgSource) ReconciliationScope(file FileInfo) string

func (*OrgSource) Scan 

func (o *OrgSource) Scan(ctx context.Context) (<-chan FileInfo, <-chan error)

type PDFOCROptions 

type PDFOCROptions struct {
	Enabled       bool
	Command       string
	Renderer      string
	Languages     []string
	MaxPages      int
	Timeout       time.Duration
	MinTextChars  int
	RenderDPI     int
	MaxRenderedMB int64
}

PDFOCROptions configures the optional local OCR fallback. OCR is only used when ordinary PDF extraction yields fewer than MinTextChars visible chars.

func DefaultPDFOCROptions 

func DefaultPDFOCROptions() PDFOCROptions

type PDFSource 

type PDFSource struct {
	// contains filtered or unexported fields
}

PDFSource indexes PDF files.

func NewPDFSource 

func NewPDFSource(paths, ignore []string) *PDFSource

NewPDFSource creates a new PDF source.

func (*PDFSource) MatchesPath 

func (p *PDFSource) MatchesPath(path string) bool

MatchesPath reports whether this source is configured to handle the path.

func (*PDFSource) Name 

func (p *PDFSource) Name() storage.Source

Name returns the source name.

func (*PDFSource) Parse 

func (p *PDFSource) Parse(ctx context.Context, file FileInfo) (*storage.Document, error)

Parse reads a PDF file and returns the parsed document.

func (*PDFSource) Scan 

func (p *PDFSource) Scan(ctx context.Context) (<-chan FileInfo, <-chan error)

Scan walks configured paths and returns PDF files to index.

func (*PDFSource) SetOCROptions 

func (p *PDFSource) SetOCROptions(options PDFOCROptions)

SetOCROptions enables or tunes the local OCR fallback.

type ParsedMarkdown 

type ParsedMarkdown struct {
	Title       string
	Body        string
	Frontmatter map[string]string
	Headings    []string
	Tags        []string
	Links       []string
}

ParsedMarkdown contains parsed markdown content.

type ReaderPage 

type ReaderPage struct {
	Content     string
	FinalURL    string
	ContentType string
}

ReaderPage is bounded textual content extracted from one public web URL.

func FetchReaderPage 

func FetchReaderPage(ctx context.Context, rawURL string, options BrowserOptions) (ReaderPage, error)

FetchReaderPage reuses the browser source's cookie-free HTTP client, domain policy, redirect checks, content-type restrictions, and response bounds for deliberate URL capture.

type ReconciledMultiDocumentSource 

type ReconciledMultiDocumentSource interface {
	MultiDocumentSource

	ReconciliationScope(file FileInfo) string
	IsDocumentInScope(file FileInfo, doc *storage.Document) bool
}

ReconciledMultiDocumentSource describes a multi-document source whose returned set is authoritative for a scope. The indexer removes documents in that scope when their stable IDs are absent from the latest successful parse.

type ScanConfig 

type ScanConfig struct {
	Paths      []string
	Extensions []string
	Ignore     []string
}

ScanConfig configures the file scanner.

type Scanner 

type Scanner struct {
	// contains filtered or unexported fields
}

Scanner walks directories and returns matching files.

func NewScanner 

func NewScanner(config ScanConfig) *Scanner

NewScanner creates a new file scanner.

func (*Scanner) MatchesPath 

func (s *Scanner) MatchesPath(path string) bool

MatchesPath reports whether a path is included by this scanner's config.

func (*Scanner) Scan 

func (s *Scanner) Scan(ctx context.Context) (<-chan FileInfo, <-chan error)

Scan walks all configured paths and sends file info to the returned channel.

type Source 

type Source interface {
	// Name returns the source name (e.g., "markdown", "pdf").
	Name() storage.Source

	// Scan walks the configured paths and returns files to index.
	Scan(ctx context.Context) (<-chan FileInfo, <-chan error)

	// MatchesPath reports whether this source is configured to handle the given path.
	MatchesPath(path string) bool

	// Parse reads a file and returns the parsed document.
	Parse(ctx context.Context, file FileInfo) (*storage.Document, error)
}

Source represents a document source that can be indexed.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.