generic

Documentation

Overview

Package generic provides a fallback parser for non-code files (text, images).

Index

• Constants
• func CreateTopicNodes(topics []string, docNodeID string) ([]*graph.Node, []*graph.Edge)
• func EnsureDirectoryHierarchy(filePath string, seen map[string]bool) ([]*graph.Node, []*graph.Edge)
• func ExtractDocument(filePath string, content []byte) (string, error)
• func ExtractText(filePath string, content []byte) string
• func NormalizeTopic(topic string) string
• type FileClass
  • func Classify(filePath string, excludeExts []string) FileClass
  • func ClassifyContent(content []byte) FileClass
• type GenericParser
  • func NewGenericParser(excludeExts []string, docsProvider docs.Provider, docsCache *docs.Cache, ...) *GenericParser
  • func (p *GenericParser) Extensions() []string
  • func (p *GenericParser) Language() parser.Language
  • func (p *GenericParser) ParseFile(filePath string, content []byte) (*parser.ParseResult, error)

Constants

const (
	// LangGeneric is the language identifier for generic (non-code) files.
	LangGeneric parser.Language = "generic"
)

Functions

func CreateTopicNodes

func CreateTopicNodes(topics []string, docNodeID string) ([]*graph.Node, []*graph.Edge)

CreateTopicNodes creates NodeTopic nodes and EdgeHasTopic edges from an extraction result, linking topics to the given document node ID. Topics are normalized and deduplicated within the result.

func EnsureDirectoryHierarchy

func EnsureDirectoryHierarchy(filePath string, seen map[string]bool) ([]*graph.Node, []*graph.Edge)

EnsureDirectoryHierarchy creates NodeDirectory nodes and EdgeContains edges for all ancestor directories of the given file path. It skips directories that already exist in the seen set to avoid duplicates within a parse session. Returns all created nodes and edges.

func ExtractDocument

func ExtractDocument(filePath string, content []byte) (string, error)

ExtractDocument extracts plain text from a document file based on its extension. Supports OOXML (DOCX, PPTX, XLSX), ODF (ODT, ODS, ODP), and PDF formats.

func ExtractText

func ExtractText(filePath string, content []byte) string

ExtractText processes raw file content based on the file extension. For most files, returns the content as-is. For specific formats, performs basic extraction (e.g., SVG tag stripping).

func NormalizeTopic

func NormalizeTopic(topic string) string

NormalizeTopic normalizes a topic name for deduplication. Lowercase, trim whitespace, collapse multiple spaces.

Types

type FileClass

type FileClass int

FileClass represents how a file should be processed.

const (
	// FileClassText indicates a text-based file to read and extract.
	FileClassText FileClass = iota
	// FileClassImage indicates an image to describe with a vision model.
	FileClassImage
	// FileClassDocument indicates a document format (DOCX, PPTX, XLSX, ODT, ODS, ODP, PDF)
	// that requires format-specific text extraction from binary containers.
	FileClassDocument
	// FileClassSkip indicates a file to skip (excluded extension or binary).
	FileClassSkip
)

func Classify

func Classify(filePath string, excludeExts []string) FileClass

Classify determines how to process a file based on its extension and content. Design: maximally inclusive — any file not explicitly excluded is a candidate.

func ClassifyContent

func ClassifyContent(content []byte) FileClass

ClassifyContent uses file content to distinguish text from binary. Reads the first 512 bytes and checks for null bytes. Returns FileClassSkip for binary files, FileClassText otherwise.

type GenericParser

type GenericParser struct {
	// contains filtered or unexported fields
}

GenericParser handles non-code files (text and images) that have no registered language parser. It creates NodeDocument nodes with raw text as DocComment and builds directory hierarchy nodes.

When a docs.Provider is available, it uses LLM-based topic extraction to create NodeTopic nodes and EdgeHasTopic edges.

func NewGenericParser

func NewGenericParser(excludeExts []string, docsProvider docs.Provider, docsCache *docs.Cache, maxImageRes int) *GenericParser

NewGenericParser creates a new GenericParser. docsProvider and docsCache may be nil (graceful degradation to raw text). maxImageRes is the max longest-edge resolution for image downscaling (default 1024).

func (*GenericParser) Extensions

func (p *GenericParser) Extensions() []string

Extensions returns an empty slice — the generic parser is used as a fallback, not matched by extension.

func (*GenericParser) Language

func (p *GenericParser) Language() parser.Language

Language returns the language identifier.

func (*GenericParser) ParseFile

func (p *GenericParser) ParseFile(filePath string, content []byte) (*parser.ParseResult, error)

ParseFile parses a non-code file and returns document + directory nodes.