preprocessors

package
v2.3.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 21, 2026 License: Apache-2.0 Imports: 21 Imported by: 0

README

Ferret Scan Preprocessors

Preprocessors extract content from various file types before validation. This enables scanning of binary files like PDFs, Office documents, images, audio, and video files.

Architecture

The preprocessor system uses a specialized architecture where each file type has its own dedicated preprocessor:

  • ImageMetadataPreprocessor: Handles image files (.jpg, .jpeg, .tiff, .tif, .png, .gif, .bmp, .webp)
  • PDFMetadataPreprocessor: Handles PDF documents (.pdf)
  • OfficeMetadataPreprocessor: Handles Office documents (.docx, .xlsx, .pptx, .odt, .ods, .odp)
  • AudioMetadataPreprocessor: Handles audio files (.mp3, .flac, .wav, .m4a)
  • VideoMetadataPreprocessor: Handles video files (.mp4, .m4v, .mov)

Features

  • Specialized Metadata Extraction: Dedicated preprocessors for each file type
  • ProcessorType Identification: Each preprocessor sets a unique ProcessorType for validator decision-making
  • Modular Design: Each preprocessor can be enabled/disabled independently
  • Single Responsibility: Each preprocessor focuses on one file type for better maintainability
  • Automatic Registration: All preprocessors are automatically registered through the router system
  • Backward Compatibility: Maintains identical behavior to the previous monolithic system

Supported File Types

Images
  • Extensions: .jpg, .jpeg, .tiff, .tif, .png, .gif, .bmp, .webp
  • ProcessorType: image_metadata
  • Extracts: EXIF data, camera information, GPS coordinates, creation dates
PDF Documents
  • Extensions: .pdf
  • ProcessorType: pdf_metadata
  • Extracts: Document metadata, author information, creation/modification dates, embedded media
Office Documents
  • Extensions: .docx, .xlsx, .pptx, .odt, .ods, .odp
  • ProcessorType: office_metadata
  • Extracts: Document properties, author information, embedded media, revision history
Audio Files
  • Extensions: .mp3, .flac, .wav, .m4a
  • ProcessorType: audio_metadata
  • Extracts: ID3 tags, artist information, album details, duration, bitrate
Video Files
  • Extensions: .mp4, .m4v, .mov
  • ProcessorType: video_metadata
  • Extracts: Video metadata, codec information, duration, resolution, creation dates

Usage

Preprocessors are automatically used by the Ferret Scan system when processing files. No manual configuration is required.

Command Line Usage
# Process a file with metadata extraction
ferret-scan --file document.pdf

# Process with preprocessing only (no validation)
ferret-scan --file image.jpg --preprocess-only

# Process with debug information
ferret-scan --file audio.mp3 --debug
Programmatic Usage
// Preprocessors are automatically registered
router := router.NewFileRouter(true)
router.RegisterDefaultPreprocessors(router)

// Process a file
result, err := router.ProcessFile("example.jpg")
if err != nil {
    log.Fatal(err)
}

// Check which preprocessor was used
fmt.Printf("Processed by: %s\n", result.ProcessorType)
Extract text from PDF documents
go run pdftext.go /path/to/document.pdf

Individual Preprocessor Documentation

Each specialized preprocessor has its own detailed documentation:

Supported File Types

Images (Metadata)
  • JPEG (.jpg, .jpeg) - EXIF metadata
  • TIFF (.tif, .tiff) - EXIF metadata
  • PNG (.png) - Basic metadata
  • GIF (.gif) - Basic metadata
  • BMP (.bmp) - Basic metadata
  • WEBP (.webp) - Basic metadata
PDF Documents (Metadata + Text)
  • PDF (.pdf) - Document metadata + text extraction
Office Documents (Metadata + Text)
  • Microsoft Word (.docx) - Document properties + text content
  • Microsoft Excel (.xlsx) - Document properties + text content
  • Microsoft PowerPoint (.pptx) - Document properties + text content
  • OpenDocument Text (.odt) - Document properties + text content
  • OpenDocument Spreadsheet (.ods) - Document properties + text content
  • OpenDocument Presentation (.odp) - Document properties + text content

Preprocessor Architecture

The preprocessing system is organized into specialized, modular components following the single responsibility principle:

Core Components
  • Router System: Automatically routes files to appropriate specialized preprocessors
  • Preprocessor Interface: Common interface implemented by all preprocessors
  • ProcessedContent: Standardized output format with ProcessorType identification
  • Shared Utilities: Common error handling, resource management, and observability
Specialized Metadata Preprocessors

Each specialized preprocessor handles one file type:

  • ImageMetadataPreprocessor: Image files (JPEG, PNG, TIFF, etc.)
  • PDFMetadataPreprocessor: PDF documents
  • OfficeMetadataPreprocessor: Office documents (DOCX, XLSX, PPTX, etc.)
  • AudioMetadataPreprocessor: Audio files (MP3, FLAC, WAV, M4A)
  • VideoMetadataPreprocessor: Video files (MP4, M4V, MOV)
Metadata Extraction Libraries
  • meta-extract-exiflib: EXIF metadata from images
  • meta-extract-pdflib: Metadata from PDF documents
  • meta-extract-officelib: Metadata from Office documents
  • meta-extract-audiolib: Metadata from audio files
  • meta-extract-videolib: Metadata from video files
Text Extractors
  • text-extract-pdftextlib: Text from PDF documents
  • text-extract-officetextlib: Text from Office documents
ProcessorType Identification

Each specialized preprocessor sets a unique ProcessorType value:

  • "image_metadata" - ImageMetadataPreprocessor
  • "pdf_metadata" - PDFMetadataPreprocessor
  • "office_metadata" - OfficeMetadataPreprocessor
  • "audio_metadata" - AudioMetadataPreprocessor
  • "video_metadata" - VideoMetadataPreprocessor

This allows validators to identify which preprocessor was used and make appropriate processing decisions.

Integration

Each preprocessor can be enabled/disabled independently and integrates seamlessly with the validation pipeline. The router system automatically selects the appropriate preprocessor based on file extension.

ProcessorType Usage for Validator Selection

The ProcessorType field in ProcessedContent allows validators to make informed decisions based on the preprocessing method used:

Validator Integration
// Example validator logic using ProcessorType
func (v *MyValidator) Validate(content *ProcessedContent) ([]Match, error) {
    switch content.ProcessorType {
    case "image_metadata":
        // Apply image-specific validation rules
        return v.validateImageMetadata(content)
    case "pdf_metadata":
        // Apply PDF-specific validation rules
        return v.validatePDFMetadata(content)
    case "office_metadata":
        // Apply Office document-specific validation rules
        return v.validateOfficeMetadata(content)
    case "audio_metadata":
        // Apply audio-specific validation rules
        return v.validateAudioMetadata(content)
    case "video_metadata":
        // Apply video-specific validation rules
        return v.validateVideoMetadata(content)
    default:
        // Apply generic validation rules
        return v.validateGeneric(content)
    }
}
Benefits for Validators
  • Specialized Rules: Apply file-type-specific validation logic
  • Confidence Scoring: Adjust confidence based on metadata source
  • Error Handling: Handle file-type-specific validation errors
  • Performance: Skip irrelevant validations for certain file types

Dependencies

This project uses minimal external dependencies:

  • Standard Go libraries for most functionality
  • github.com/ledongthuc/pdf for PDF text extraction

Limitations

  • EXIF extraction only works with images that contain EXIF data
  • PDF text extraction may not work with all PDF formats
  • Office document extraction works best with newer formats (DOCX, XLSX, PPTX)
  • Text formatting and layout are not preserved

License

This project is licensed under the MIT License - see the LICENSE file for details.

Documentation

Index

Constants

View Source 
const (
	ProcessorTypeImageMetadata  = "image_metadata"
	ProcessorTypePDFMetadata    = "pdf_metadata"
	ProcessorTypeOfficeMetadata = "office_metadata"
	ProcessorTypeAudioMetadata  = "audio_metadata"
	ProcessorTypeVideoMetadata  = "video_metadata"
)

ProcessorType constants for specialized metadata preprocessors

View Source 
const (
	FormatImageMetadata  = "image_metadata"
	FormatPDFMetadata    = "pdf_metadata"
	FormatOfficeMetadata = "office_metadata"
	FormatAudioMetadata  = "audio_metadata"
	FormatVideoMetadata  = "video_metadata"
)

Format constants for processed content

View Source 
const (
	PreprocessorNameImage  = "image_metadata_preprocessor"
	PreprocessorNamePDF    = "pdf_metadata_preprocessor"
	PreprocessorNameOffice = "office_metadata_preprocessor"
	PreprocessorNameAudio  = "audio_metadata_preprocessor"
	PreprocessorNameVideo  = "video_metadata_preprocessor"
)

Preprocessor name constants

View Source 
const (
	FileTypeImage  = "image"
	FileTypePDF    = "pdf"
	FileTypeOffice = "office"
	FileTypeAudio  = "audio"
	FileTypeVideo  = "video"
)

File type constants for error handling and logging

View Source 
const (
	MetadataFieldTitle              = "Title"
	MetadataFieldAuthor             = "Author"
	MetadataFieldSubject            = "Subject"
	MetadataFieldKeywords           = "Keywords"
	MetadataFieldCreator            = "Creator"
	MetadataFieldProducer           = "Producer"
	MetadataFieldCreationDate       = "CreationDate"
	MetadataFieldModificationDate   = "ModificationDate"
	MetadataFieldPageCount          = "PageCount"
	MetadataFieldWordCount          = "WordCount"
	MetadataFieldCharacterCount     = "CharacterCount"
	MetadataFieldDescription        = "Description"
	MetadataFieldCategory           = "Category"
	MetadataFieldApplication        = "Application"
	MetadataFieldApplicationVersion = "ApplicationVersion"
	MetadataFieldCompany            = "Company"
	MetadataFieldLastModifiedBy     = "LastModifiedBy"
	MetadataFieldManager            = "Manager"
	MetadataFieldComments           = "Comments"
	MetadataFieldContentStatus      = "ContentStatus"
	MetadataFieldIdentifier         = "Identifier"
	MetadataFieldLanguage           = "Language"
	MetadataFieldRevision           = "Revision"
	MetadataFieldPDFVersion         = "PDFVersion"
	MetadataFieldEncrypted          = "Encrypted"
)

Common metadata field names for consistent formatting

View Source 
const (
	MetadataDateFormat = "2006:01:02 15:04:05-07:00"
)

Date format constant for consistent date formatting

Variables

View Source 
var (
	CommonDateExclusionKeys = []string{
		"CreationDate",
		"ModificationDate",
	}
)

Common exclusion keys for properties formatting

View Source 
var ErrEmbeddedTooDeep = embedded.ErrTooDeep

ErrEmbeddedTooDeep is returned by RouterInterface.ProcessEmbedded when a container nests deeper than the router's bound.

A sentinel rather than a formatted string so the caller can branch on it with errors.Is and tell "too deep" (coverage was cut short — say so) apart from "this child failed to parse" (already handled by the ordinary error path).

Aliased to embedded.ErrTooDeep rather than declared independently: the redaction side raises the same condition, and two distinct sentinels would make errors.Is fail across the halves — a caller branching on the read side's value would not recognise the write side's, and the "coverage was cut short" disclosure would be silently downgraded to a generic failure. Existing references to this name keep working.

Functions

func CalculateAbsoluteOffset 

func CalculateAbsoluteOffset(text string, line, charPos int) int

CalculateAbsoluteOffset calculates the absolute character offset for a text position

func DecodeToUTF8 added in v2.1.2 

func DecodeToUTF8(raw []byte, enc TextEncoding) (string, bool)

DecodeToUTF8 decodes raw file bytes to a UTF-8 string according to enc. Decoding is total: malformed sequences (lone surrogates, a truncated final code unit) decode to U+FFFD rather than failing — a corrupt tail must not hide the readable remainder of a file from scanning. The second return is false only when enc is not a transcodable encoding (caller keeps raw).

func EncodeFromUTF8 added in v2.1.2 

func EncodeFromUTF8(s string, enc TextEncoding) []byte

EncodeFromUTF8 re-encodes a UTF-8 string back to enc, restoring the BOM where the source had one. Used by the redaction write path so a redacted copy of a UTF-16 file is still a valid UTF-16 file (a re-importable .reg export, a PowerShell transcript, ...) rather than silently becoming UTF-8.

func IsFileSizeError 

func IsFileSizeError(err error) bool

IsFileSizeError checks if the error is a file size limit error

func IsTimeoutError 

func IsTimeoutError(err error) bool

IsTimeoutError checks if the error is a timeout error

func LooksLikeText added in v2.1.1 

func LooksLikeText(buf []byte) bool

LooksLikeText reports whether buf (a null-free prefix of the file) is text. Exported because the FileRouter maintains a second sniff site (isTextFile in internal/router) that must apply identical semantics — the two copies of the old byte-ratio heuristic drifted into the same UTF-8 bug independently.

UTF-8 first: every byte of a multi-byte UTF-8 sequence is >= 0x80, so the old ASCII-printable ratio counted EVERY non-ASCII character against the file. A short line with a ™ (3 bytes) or an em-dash, a name with accents, or any non-Latin-script document fell below the 95% bar and the file was silently skipped as "binary" — a recall hole across ALL validators in file mode (stdin mode never sniffs, which is how the gap hid). Genuinely binary data essentially never forms long runs of valid UTF-8, so utf8.Valid is both the safer and the stricter signal; the ASCII-ratio heuristic remains only as the fallback for legacy single-byte encodings (Latin-1 etc.), which are not valid UTF-8 but are still text someone may want scanned.

func ShouldPreprocess 

func ShouldPreprocess(filePath string) bool

ShouldPreprocess checks if a file should be preprocessed based on its extension Now returns true for all files since we have a plain text preprocessor that handles all file types

Types

type AudioMetadataPreprocessor 

type AudioMetadataPreprocessor struct {
	// contains filtered or unexported fields
}

AudioMetadataPreprocessor extracts metadata from audio files

func NewAudioMetadataPreprocessor 

func NewAudioMetadataPreprocessor() *AudioMetadataPreprocessor

NewAudioMetadataPreprocessor creates a new audio metadata preprocessor

func (*AudioMetadataPreprocessor) CanProcess 

func (amp *AudioMetadataPreprocessor) CanProcess(filePath string) bool

CanProcess checks if this preprocessor can handle the given file

func (*AudioMetadataPreprocessor) GetName 

func (amp *AudioMetadataPreprocessor) GetName() string

GetName returns the name of this preprocessor

func (*AudioMetadataPreprocessor) GetSupportedExtensions 

func (amp *AudioMetadataPreprocessor) GetSupportedExtensions() []string

GetSupportedExtensions returns the file extensions this preprocessor supports

func (*AudioMetadataPreprocessor) Process 

func (amp *AudioMetadataPreprocessor) Process(filePath string) (*ProcessedContent, error)

Process extracts metadata from audio files

func (*AudioMetadataPreprocessor) SetObserver 

func (amp *AudioMetadataPreprocessor) SetObserver(observer observability.Observer)

SetObserver sets the observability component

type BaseMetadataPreprocessor 

type BaseMetadataPreprocessor struct {
	// contains filtered or unexported fields
}

BaseMetadataPreprocessor provides common functionality for all specialized metadata preprocessors

func NewBaseMetadataPreprocessor 

func NewBaseMetadataPreprocessor(name, processorType string) *BaseMetadataPreprocessor

NewBaseMetadataPreprocessor creates a new base metadata preprocessor

func (*BaseMetadataPreprocessor) AddRetryDelay 

func (bmp *BaseMetadataPreprocessor) AddRetryDelay(attemptCount int)

AddRetryDelay adds a small delay before retry attempts

func (*BaseMetadataPreprocessor) ApplyConfig 

func (bmp *BaseMetadataPreprocessor) ApplyConfig(config *MetadataProcessingConfig)

ApplyConfig applies configuration to the base preprocessor

func (*BaseMetadataPreprocessor) BuildErrorContent 

func (bmp *BaseMetadataPreprocessor) BuildErrorContent(filePath, format string, err error) *ProcessedContent

BuildErrorContent creates a failed ProcessedContent structure

func (*BaseMetadataPreprocessor) BuildSuccessContent 

func (bmp *BaseMetadataPreprocessor) BuildSuccessContent(filePath, text, format string, pageCount int) *ProcessedContent

BuildSuccessContent creates a successful ProcessedContent structure

func (*BaseMetadataPreprocessor) CreateProcessingContext 

func (bmp *BaseMetadataPreprocessor) CreateProcessingContext() (context.Context, context.CancelFunc)

CreateProcessingContext creates a context with timeout for processing

func (*BaseMetadataPreprocessor) GetName 

func (bmp *BaseMetadataPreprocessor) GetName() string

GetName returns the name of this preprocessor

func (*BaseMetadataPreprocessor) GetObserver 

func (bmp *BaseMetadataPreprocessor) GetObserver() observability.Observer

GetObserver returns the observer instance

func (*BaseMetadataPreprocessor) GetRouter 

func (bmp *BaseMetadataPreprocessor) GetRouter() RouterInterface

GetRouter returns the router instance

func (*BaseMetadataPreprocessor) GetUtilities 

func (bmp *BaseMetadataPreprocessor) GetUtilities() *SharedUtilities

GetUtilities returns the shared utilities instance

func (*BaseMetadataPreprocessor) HandleError 

func (bmp *BaseMetadataPreprocessor) HandleError(filePath, fileType string, err error) *ProcessedContent

HandleError handles processing errors with comprehensive error handling

func (*BaseMetadataPreprocessor) LogDebugInfo 

func (bmp *BaseMetadataPreprocessor) LogDebugInfo(message string)

LogDebugInfo logs debug information if observer is available

func (*BaseMetadataPreprocessor) LogFileSystemInfo 

func (bmp *BaseMetadataPreprocessor) LogFileSystemInfo(filename string, fileSize int64, mimeType string)

LogFileSystemInfo logs file system information for observability (excluded from validator content)

func (*BaseMetadataPreprocessor) LogRetryAttempt 

func (bmp *BaseMetadataPreprocessor) LogRetryAttempt(filePath string, attemptCount int)

LogRetryAttempt logs retry attempt information

func (*BaseMetadataPreprocessor) LogSuccessfulProcessing 

func (bmp *BaseMetadataPreprocessor) LogSuccessfulProcessing(filename string, fileSize int64, mimeType string)

LogSuccessfulProcessing logs successful processing information

func (*BaseMetadataPreprocessor) ProcessEmbeddedMedia 

func (bmp *BaseMetadataPreprocessor) ProcessEmbeddedMedia(originalFilePath string, embeddedMedia []EmbeddedMedia) (string, []ContentSection, []string)

ProcessEmbeddedMedia processes embedded media through the router if available.

It returns the text to append to the container's own metadata text, AND one ContentSection per embedded item describing that text out of band. The sections' LineOffset values are relative to the START OF THE RETURNED TEXT; the caller shifts them by the length of whatever it puts in front.

The sections matter because an embedded item is a section INSIDE one preprocessor's output, and it routes to a DIFFERENT metadata rule set than its container: a .wav inside a .docx carries an "Artist:" field, which is on the audio rule list but not the office one. Without a declared sub-section the whole blob would be labelled office_metadata and that field would report nothing. Measured: the AUTHOR_INFO finding for an embedded clip's artist address disappeared until these sections were carried. ProcessEmbeddedMedia now also returns WARNINGS: notes about embedded items it could not descend into. An item skipped silently is undisclosed missing coverage, which is the failure mode this whole area keeps producing.

func (*BaseMetadataPreprocessor) ProcessWithRetry 

func (bmp *BaseMetadataPreprocessor) ProcessWithRetry(filePath string, processFunc func() (*ProcessedContent, error)) (*ProcessedContent, error)

ProcessWithRetry provides a generic retry mechanism for metadata processing

func (*BaseMetadataPreprocessor) SetObserver 

func (bmp *BaseMetadataPreprocessor) SetObserver(observer observability.Observer)

SetObserver sets the observability component

func (*BaseMetadataPreprocessor) SetRouter 

func (bmp *BaseMetadataPreprocessor) SetRouter(router RouterInterface)

SetRouter sets the router instance for reprocessing embedded media

func (*BaseMetadataPreprocessor) ShouldRetry 

func (bmp *BaseMetadataPreprocessor) ShouldRetry(err error, attemptCount int) bool

ShouldRetry determines if processing should be retried based on error type and attempt count

func (*BaseMetadataPreprocessor) ValidateFileSize 

func (bmp *BaseMetadataPreprocessor) ValidateFileSize(filePath string, isVideo bool) error

ValidateFileSize validates file size based on file type

type BoundingBox 

type BoundingBox struct {
	// X is the left coordinate (normalized 0.0-1.0 or absolute pixels)
	X float64 `json:"x"`

	// Y is the top coordinate (normalized 0.0-1.0 or absolute pixels)
	Y float64 `json:"y"`

	// Width is the width of the box
	Width float64 `json:"width"`

	// Height is the height of the box
	Height float64 `json:"height"`

	// Unit indicates the coordinate system ("normalized", "pixels", "points")
	Unit string `json:"unit,omitempty"`
}

BoundingBox represents a rectangular area in a document

type ContentSection added in v2.2.1 

type ContentSection struct {
	// Name is the producing preprocessor's GetName(), e.g. "Text Extractor" or
	// "office_metadata". This is the string the flat separator used to spell out.
	Name string

	// Kind is the routing decision: SectionKindBody or SectionKindMetadata.
	Kind SectionKind

	// Type is the metadata preprocessor type ("office_metadata",
	// "image_metadata", ...) for a metadata section, and empty for a body
	// section. It is what the METADATA validator keys its per-preprocessor
	// field rules off.
	Type string

	// SourceFile is the path to attribute findings in this section to. It is the
	// scanned file for a normal section, and the "container.docx -> image1.jpg"
	// form for a section extracted from embedded media.
	SourceFile string

	// Text is this section's bytes. It is a substring of ProcessedContent.Text
	// (Go strings share backing storage, so this is a reference, not a copy).
	Text string

	// LineOffset is the 0-based line index at which Text begins within
	// ProcessedContent.Text, so a finding's line number inside a section can be
	// reported against the whole extracted document.
	LineOffset int
}

ContentSection describes one contiguous run of ProcessedContent.Text and where it came from. Every field is derived from OUR OWN code — the preprocessor's GetName() and the router's classification of it — never from the scanned document's bytes. That provenance is the entire point: it is what makes a section boundary unforgeable by a document author.

type DocumentPosition 

type DocumentPosition struct {
	// Page is the page number (1-based, 0 for single-page documents)
	Page int `json:"page"`

	// BoundingBox defines the rectangular area in the document (for PDFs, images, etc.)
	BoundingBox *BoundingBox `json:"bounding_box,omitempty"`

	// TextRun is the text run identifier (for structured documents)
	TextRun int `json:"text_run,omitempty"`

	// CharOffset is the character offset within the original document
	CharOffset int `json:"char_offset"`

	// LineNumber is the line number in the original document (if applicable)
	LineNumber int `json:"line_number,omitempty"`
}

DocumentPosition represents a position in the original document

type EmbeddedMedia 

type EmbeddedMedia struct {
	OriginalName string
	TempFilePath string
	MediaType    string
}

EmbeddedMedia represents embedded media extracted from documents

type ErrorClassifier 

type ErrorClassifier struct{}

ErrorClassifier classifies errors into appropriate types

func NewErrorClassifier 

func NewErrorClassifier() *ErrorClassifier

NewErrorClassifier creates a new error classifier

func (*ErrorClassifier) ClassifyError 

func (ec *ErrorClassifier) ClassifyError(err error) ErrorType

ClassifyError classifies an error into an appropriate ErrorType

type ErrorLogger 

type ErrorLogger struct {
	// contains filtered or unexported fields
}

ErrorLogger provides structured logging for media processing errors

func NewErrorLogger 

func NewErrorLogger(level LogLevel) *ErrorLogger

NewErrorLogger creates a new error logger

func (*ErrorLogger) LogError 

func (el *ErrorLogger) LogError(err *MediaProcessingError)

LogError logs a media processing error with appropriate level

type ErrorRecoveryManager 

type ErrorRecoveryManager struct {
	// contains filtered or unexported fields
}

ErrorRecoveryManager manages error recovery strategies

func NewErrorRecoveryManager 

func NewErrorRecoveryManager() *ErrorRecoveryManager

NewErrorRecoveryManager creates a new error recovery manager

func (*ErrorRecoveryManager) GetMaxRetries 

func (erm *ErrorRecoveryManager) GetMaxRetries() int

GetMaxRetries returns the maximum number of retries

func (*ErrorRecoveryManager) GetRecoveryStrategy 

func (erm *ErrorRecoveryManager) GetRecoveryStrategy(errorType ErrorType) RecoveryStrategy

GetRecoveryStrategy returns the recovery strategy for an error type

func (*ErrorRecoveryManager) SetMaxRetries 

func (erm *ErrorRecoveryManager) SetMaxRetries(maxRetries int)

SetMaxRetries sets the maximum number of retries

func (*ErrorRecoveryManager) ShouldRetry 

func (erm *ErrorRecoveryManager) ShouldRetry(errorType ErrorType, attemptCount int) bool

ShouldRetry determines if an error should trigger a retry

type ErrorType 

type ErrorType string

ErrorType represents different types of processing errors 

const (
	// File-related errors
	ErrorTypeFileAccess    ErrorType = "file_access"
	ErrorTypeFileSize      ErrorType = "file_size"
	ErrorTypeFileCorrupted ErrorType = "file_corrupted"

	// Format-related errors
	ErrorTypeUnsupportedFormat ErrorType = "unsupported_format"
	ErrorTypeInvalidFormat     ErrorType = "invalid_format"
	ErrorTypeFormatCorrupted   ErrorType = "format_corrupted"

	// Processing-related errors
	ErrorTypeTimeout          ErrorType = "timeout"
	ErrorTypeMemoryLimit      ErrorType = "memory_limit"
	ErrorTypeParsingFailed    ErrorType = "parsing_failed"
	ErrorTypeExtractionFailed ErrorType = "extraction_failed"

	// Context-related errors
	ErrorTypeCancelled ErrorType = "cancelled"

	// Unknown errors
	ErrorTypeUnknown ErrorType = "unknown"
)

type FileExtensionValidator 

type FileExtensionValidator struct {
	// contains filtered or unexported fields
}

FileExtensionValidator provides common file extension validation functions

func NewFileExtensionValidator 

func NewFileExtensionValidator() *FileExtensionValidator

NewFileExtensionValidator creates a new file extension validator

func (*FileExtensionValidator) GetAudioExtensions 

func (fev *FileExtensionValidator) GetAudioExtensions() []string

GetAudioExtensions returns all supported audio extensions

func (*FileExtensionValidator) GetFileExtension 

func (fev *FileExtensionValidator) GetFileExtension(filePath string) string

GetFileExtension returns the lowercase file extension

func (*FileExtensionValidator) GetFileName 

func (fev *FileExtensionValidator) GetFileName(filePath string) string

GetFileName extracts the filename from a file path

func (*FileExtensionValidator) GetImageExtensions 

func (fev *FileExtensionValidator) GetImageExtensions() []string

GetImageExtensions returns all supported image extensions

func (*FileExtensionValidator) GetOfficeExtensions 

func (fev *FileExtensionValidator) GetOfficeExtensions() []string

GetOfficeExtensions returns all supported Office extensions

func (*FileExtensionValidator) GetPDFExtensions 

func (fev *FileExtensionValidator) GetPDFExtensions() []string

GetPDFExtensions returns all supported PDF extensions

func (*FileExtensionValidator) GetVideoExtensions 

func (fev *FileExtensionValidator) GetVideoExtensions() []string

GetVideoExtensions returns all supported video extensions

func (*FileExtensionValidator) IsAudioFile 

func (fev *FileExtensionValidator) IsAudioFile(filePath string) bool

IsAudioFile checks if the file is an audio file

func (*FileExtensionValidator) IsImageFile 

func (fev *FileExtensionValidator) IsImageFile(filePath string) bool

IsImageFile checks if the file is an image file

func (*FileExtensionValidator) IsOfficeFile 

func (fev *FileExtensionValidator) IsOfficeFile(filePath string) bool

IsOfficeFile checks if the file is an Office document

func (*FileExtensionValidator) IsPDFFile 

func (fev *FileExtensionValidator) IsPDFFile(filePath string) bool

IsPDFFile checks if the file is a PDF file

func (*FileExtensionValidator) IsVideoFile 

func (fev *FileExtensionValidator) IsVideoFile(filePath string) bool

IsVideoFile checks if the file is a video file

type GracefulDegradationHandler 

type GracefulDegradationHandler struct {
	// contains filtered or unexported fields
}

GracefulDegradationHandler handles graceful degradation for different error types

func NewGracefulDegradationHandler 

func NewGracefulDegradationHandler() *GracefulDegradationHandler

NewGracefulDegradationHandler creates a new graceful degradation handler

func (*GracefulDegradationHandler) HandleError 

func (gdh *GracefulDegradationHandler) HandleError(filePath, fileType string, err error) *ProcessedContent

HandleError handles an error with graceful degradation

type ImageMetadataPreprocessor 

type ImageMetadataPreprocessor struct {
	// contains filtered or unexported fields
}

ImageMetadataPreprocessor extracts metadata from image files

func NewImageMetadataPreprocessor 

func NewImageMetadataPreprocessor() *ImageMetadataPreprocessor

NewImageMetadataPreprocessor creates a new image metadata preprocessor

func (*ImageMetadataPreprocessor) CanProcess 

func (imp *ImageMetadataPreprocessor) CanProcess(filePath string) bool

CanProcess checks if this preprocessor can handle the given file

func (*ImageMetadataPreprocessor) GetName 

func (imp *ImageMetadataPreprocessor) GetName() string

GetName returns the name of this preprocessor

func (*ImageMetadataPreprocessor) GetSupportedExtensions 

func (imp *ImageMetadataPreprocessor) GetSupportedExtensions() []string

GetSupportedExtensions returns the file extensions this preprocessor supports

func (*ImageMetadataPreprocessor) Process 

func (imp *ImageMetadataPreprocessor) Process(filePath string) (*ProcessedContent, error)

Process extracts metadata from image files

func (*ImageMetadataPreprocessor) SetObserver 

func (imp *ImageMetadataPreprocessor) SetObserver(observer observability.Observer)

SetObserver sets the observability component

type LogLevel 

type LogLevel int

LogLevel represents different log levels 

const (
	LogLevelDebug LogLevel = iota
	LogLevelInfo
	LogLevelWarn
	LogLevelError
)

type MediaProcessingError 

type MediaProcessingError struct {
	FilePath    string
	FileType    string
	ErrorType   ErrorType
	Message     string
	Cause       error
	Recoverable bool
	Context     map[string]interface{}
}

MediaProcessingError represents a comprehensive error during media processing

func NewMediaProcessingError 

func NewMediaProcessingError(filePath, fileType string, errorType ErrorType, message string, cause error) *MediaProcessingError

NewMediaProcessingError creates a new media processing error

func (*MediaProcessingError) Error 

func (mpe *MediaProcessingError) Error() string

Error implements the error interface

func (*MediaProcessingError) GetContext 

func (mpe *MediaProcessingError) GetContext() map[string]interface{}

GetContext returns the error context

func (*MediaProcessingError) GetErrorType 

func (mpe *MediaProcessingError) GetErrorType() ErrorType

GetErrorType returns the error type

func (*MediaProcessingError) IsRecoverable 

func (mpe *MediaProcessingError) IsRecoverable() bool

IsRecoverable returns whether the error is recoverable

func (*MediaProcessingError) Unwrap 

func (mpe *MediaProcessingError) Unwrap() error

Unwrap returns the underlying error

func (*MediaProcessingError) WithContext 

func (mpe *MediaProcessingError) WithContext(key string, value interface{}) *MediaProcessingError

WithContext adds context to the error

type MediaResourceManager 

type MediaResourceManager struct {
	// contains filtered or unexported fields
}

MediaResourceManager manages resource limits for media file processing

func NewMediaResourceManager 

func NewMediaResourceManager() *MediaResourceManager

NewMediaResourceManager creates a new resource manager with default limits

func NewMediaResourceManagerWithLimits 

func NewMediaResourceManagerWithLimits(limits *ResourceLimits) *MediaResourceManager

NewMediaResourceManagerWithLimits creates a new resource manager with custom limits

func (*MediaResourceManager) CreateProcessingContext 

func (rm *MediaResourceManager) CreateProcessingContext() (context.Context, context.CancelFunc)

CreateProcessingContext creates a context with timeout for processing

func (*MediaResourceManager) GetLimits 

func (rm *MediaResourceManager) GetLimits() *ResourceLimits

GetLimits returns the current resource limits

func (*MediaResourceManager) SetLimits 

func (rm *MediaResourceManager) SetLimits(limits *ResourceLimits)

SetLimits updates the resource limits

func (*MediaResourceManager) ValidateFileSize 

func (rm *MediaResourceManager) ValidateFileSize(filePath string, isVideo bool) error

ValidateFileSize checks if the file size is within limits

type MetadataFormatter 

type MetadataFormatter struct{}

MetadataFormatter provides common metadata formatting functions

func NewMetadataFormatter 

func NewMetadataFormatter() *MetadataFormatter

NewMetadataFormatter creates a new metadata formatter

func (*MetadataFormatter) CalculateTextMetrics 

func (mf *MetadataFormatter) CalculateTextMetrics(text string) (wordCount, charCount, lineCount int)

CalculateTextMetrics calculates word count, character count, and line count for text

func (*MetadataFormatter) FormatBooleanField 

func (mf *MetadataFormatter) FormatBooleanField(key string, value bool) string

FormatBooleanField formats a boolean field, only including it if true

func (*MetadataFormatter) FormatDateField 

func (mf *MetadataFormatter) FormatDateField(key string, date time.Time) string

FormatDateField formats a date field with consistent formatting

func (*MetadataFormatter) FormatMetadataField 

func (mf *MetadataFormatter) FormatMetadataField(key, value string) string

FormatMetadataField formats a metadata field with proper key-value formatting

func (*MetadataFormatter) FormatNumericField 

func (mf *MetadataFormatter) FormatNumericField(key string, value int) string

FormatNumericField formats a numeric field, only including it if greater than zero

func (*MetadataFormatter) FormatPropertiesMap 

func (mf *MetadataFormatter) FormatPropertiesMap(properties map[string]string, excludeKeys []string) string

FormatPropertiesMap formats a map of additional properties

type MetadataProcessingConfig 

type MetadataProcessingConfig struct {
	EnableRetry          bool
	MaxRetries           int
	EnableResourceLimits bool
	EnableObservability  bool
}

MetadataProcessingConfig holds configuration for metadata processing

func DefaultMetadataProcessingConfig 

func DefaultMetadataProcessingConfig() *MetadataProcessingConfig

DefaultMetadataProcessingConfig returns default configuration

type OfficeMetadataPreprocessor 

type OfficeMetadataPreprocessor struct {
	*BaseMetadataPreprocessor
}

OfficeMetadataPreprocessor extracts metadata from Office documents

func NewOfficeMetadataPreprocessor 

func NewOfficeMetadataPreprocessor() *OfficeMetadataPreprocessor

NewOfficeMetadataPreprocessor creates a new Office metadata preprocessor

func (*OfficeMetadataPreprocessor) CanProcess 

func (omp *OfficeMetadataPreprocessor) CanProcess(filePath string) bool

CanProcess checks if this preprocessor can handle the given file

func (*OfficeMetadataPreprocessor) GetSupportedExtensions 

func (omp *OfficeMetadataPreprocessor) GetSupportedExtensions() []string

GetSupportedExtensions returns the file extensions this preprocessor supports

func (*OfficeMetadataPreprocessor) Process 

func (omp *OfficeMetadataPreprocessor) Process(filePath string) (*ProcessedContent, error)

Process extracts metadata from Office documents

func (*OfficeMetadataPreprocessor) SetObserver 

func (omp *OfficeMetadataPreprocessor) SetObserver(observer observability.Observer)

SetObserver sets the observability component

func (*OfficeMetadataPreprocessor) SetRouter 

func (omp *OfficeMetadataPreprocessor) SetRouter(router RouterInterface)

SetRouter sets the router instance for reprocessing embedded media

type PDFMetadataPreprocessor 

type PDFMetadataPreprocessor struct {
	*BaseMetadataPreprocessor
}

PDFMetadataPreprocessor extracts metadata from PDF documents

func NewPDFMetadataPreprocessor 

func NewPDFMetadataPreprocessor() *PDFMetadataPreprocessor

NewPDFMetadataPreprocessor creates a new PDF metadata preprocessor

func (*PDFMetadataPreprocessor) CanProcess 

func (pmp *PDFMetadataPreprocessor) CanProcess(filePath string) bool

CanProcess checks if this preprocessor can handle the given file

func (*PDFMetadataPreprocessor) GetSupportedExtensions 

func (pmp *PDFMetadataPreprocessor) GetSupportedExtensions() []string

GetSupportedExtensions returns the file extensions this preprocessor supports

func (*PDFMetadataPreprocessor) Process 

func (pmp *PDFMetadataPreprocessor) Process(filePath string) (*ProcessedContent, error)

Process extracts metadata from the PDF file

func (*PDFMetadataPreprocessor) SetObserver 

func (pmp *PDFMetadataPreprocessor) SetObserver(observer observability.Observer)

SetObserver sets the observability component

func (*PDFMetadataPreprocessor) SetRouter 

func (pmp *PDFMetadataPreprocessor) SetRouter(router RouterInterface)

SetRouter sets the router instance for reprocessing embedded media

type PlainTextPreprocessor 

type PlainTextPreprocessor struct {
	// contains filtered or unexported fields
}

PlainTextPreprocessor handles plain text files by passing their content through This ensures text files are processed through the same pipeline as other file types

func NewPlainTextPreprocessor 

func NewPlainTextPreprocessor() *PlainTextPreprocessor

NewPlainTextPreprocessor creates a new plain text preprocessor

func NewPlainTextPreprocessorWithConfig 

func NewPlainTextPreprocessorWithConfig(enableRedaction bool) *PlainTextPreprocessor

NewPlainTextPreprocessorWithConfig creates a new plain text preprocessor with redaction configuration

func (*PlainTextPreprocessor) CanProcess 

func (ptp *PlainTextPreprocessor) CanProcess(filePath string) bool

CanProcess checks if this preprocessor can handle the given file

func (*PlainTextPreprocessor) GetName 

func (ptp *PlainTextPreprocessor) GetName() string

GetName returns the name of this preprocessor

func (*PlainTextPreprocessor) GetSupportedExtensions 

func (ptp *PlainTextPreprocessor) GetSupportedExtensions() []string

GetSupportedExtensions returns the file extensions this preprocessor supports

func (*PlainTextPreprocessor) Process 

func (ptp *PlainTextPreprocessor) Process(filePath string) (*ProcessedContent, error)

Process extracts text content from the file

func (*PlainTextPreprocessor) SetObserver 

func (ptp *PlainTextPreprocessor) SetObserver(observer observability.Observer)

SetObserver sets the observability component

type PositionMapping 

type PositionMapping struct {
	// ExtractedPosition is the position in the extracted text
	ExtractedPosition TextPosition `json:"extracted_position"`

	// OriginalPosition is the corresponding position in the original document
	OriginalPosition DocumentPosition `json:"original_position"`

	// ConfidenceScore is the confidence in this position mapping (0.0 to 1.0)
	ConfidenceScore float64 `json:"confidence_score"`

	// Context is surrounding text for verification
	Context string `json:"context,omitempty"`

	// Method describes how this mapping was determined
	Method string `json:"method"`
}

PositionMapping represents a mapping between extracted text positions and original document positions

type Preprocessor 

type Preprocessor interface {
	// CanProcess checks if this preprocessor can handle the given file
	CanProcess(filePath string) bool

	// Process extracts content from the file
	Process(filePath string) (*ProcessedContent, error)

	// GetName returns the name of this preprocessor
	GetName() string

	// GetSupportedExtensions returns the file extensions this preprocessor supports
	GetSupportedExtensions() []string

	// SetObserver sets the observability component
	SetObserver(observer observability.Observer)
}

Preprocessor interface defines methods for preprocessing files

type PreprocessorManager 

type PreprocessorManager struct {
	// contains filtered or unexported fields
}

PreprocessorManager manages all available preprocessors

func NewPreprocessorManager 

func NewPreprocessorManager() *PreprocessorManager

NewPreprocessorManager creates a new preprocessor manager

func (*PreprocessorManager) GetAvailablePreprocessors 

func (pm *PreprocessorManager) GetAvailablePreprocessors() []Preprocessor

GetAvailablePreprocessors returns all registered preprocessors

func (*PreprocessorManager) GetPreprocessor 

func (pm *PreprocessorManager) GetPreprocessor(filePath string) Preprocessor

GetPreprocessor returns the appropriate preprocessor for a file, or nil if none found

func (*PreprocessorManager) ProcessFile 

func (pm *PreprocessorManager) ProcessFile(filePath string) (*ProcessedContent, error)

ProcessFile processes a file with all appropriate preprocessors

func (*PreprocessorManager) RegisterPreprocessor 

func (pm *PreprocessorManager) RegisterPreprocessor(p Preprocessor)

RegisterPreprocessor adds a preprocessor to the manager

type ProcessedContent 

type ProcessedContent struct {
	// Original file information
	OriginalPath string
	Filename     string

	// Extracted content
	Text string

	// Content metadata
	Format     string
	PageCount  int
	WordCount  int
	CharCount  int
	LineCount  int
	Paragraphs int

	// Processing information
	ProcessorType string
	Success       bool
	Error         error

	// ExtractionWarning is a short, payload-free note that extraction SUCCEEDED
	// but produced suspiciously nothing — e.g. a container whose format carries a
	// document body yielded no body text. It is deliberately not an Error: the
	// file was read and whatever was extracted is valid, so the scan continues and
	// the findings stand. It exists because "extracted nothing" and "the document
	// is empty" used to be indistinguishable — both were Success with textLen 0 —
	// which made a skipped document body look like a clean scan.
	ExtractionWarning string

	// Position mapping information for redaction
	PositionMappings []PositionMapping `json:"position_mappings,omitempty"`

	// Position tracking metadata
	PositionTrackingEnabled bool                   `json:"position_tracking_enabled"`
	PositionConfidence      float64                `json:"position_confidence"`
	PositionMetadata        map[string]interface{} `json:"position_metadata,omitempty"`

	// Additional metadata for embedded media and other extensions
	Metadata map[string]interface{}

	// Sections carries the structure of Text OUT OF BAND: one entry per
	// preprocessor whose output was concatenated into Text, in the same order.
	//
	// It exists because the structure used to be carried IN BAND. The file router
	// flattens every extractor's output into Text with literal "\n\n--- name ---\n"
	// separators, and the content router then re-parsed that text to recover which
	// bytes came from which extractor. Document authors control the text, so they
	// controlled the recovered "structure": a paragraph typed as
	// "--- office_metadata ---" became a section boundary. That is in-band
	// signalling, the same class of defect as SQL injection, and it has the same
	// answer — carry the structure alongside the data instead of re-deriving it
	// from the data.
	//
	// Text stays the flat concatenation, byte for byte, so every existing consumer
	// is unaffected; this field is purely additive. A nil/empty Sections means "no
	// structure was declared", which consumers must treat as "all of Text is
	// document body" — the safe direction, since the document path runs the full
	// validator set and the metadata path runs one field-name scanner.
	Sections []ContentSection
}

ProcessedContent represents content that has been processed by a preprocessor

func (*ProcessedContent) AddPositionMapping 

func (pc *ProcessedContent) AddPositionMapping(mapping PositionMapping)

AddPositionMapping adds a position mapping to the processed content

func (*ProcessedContent) AddPositionMetadata 

func (pc *ProcessedContent) AddPositionMetadata(key string, value interface{})

AddPositionMetadata adds metadata related to position tracking

func (*ProcessedContent) EnablePositionTracking 

func (pc *ProcessedContent) EnablePositionTracking()

EnablePositionTracking enables position tracking for this content

func (*ProcessedContent) GetPositionMappingsForRange 

func (pc *ProcessedContent) GetPositionMappingsForRange(startLine, startChar, endLine, endChar int) []PositionMapping

GetPositionMappingsForRange returns position mappings that overlap with the given text range

func (*ProcessedContent) SetPositionConfidence 

func (pc *ProcessedContent) SetPositionConfidence(confidence float64)

SetPositionConfidence sets the overall confidence for position mappings

type ProcessedContentBuilder 

type ProcessedContentBuilder struct {
	// contains filtered or unexported fields
}

ProcessedContentBuilder helps build ProcessedContent structures consistently

func NewProcessedContentBuilder 

func NewProcessedContentBuilder() *ProcessedContentBuilder

NewProcessedContentBuilder creates a new processed content builder

func (*ProcessedContentBuilder) BuildErrorContent 

func (pcb *ProcessedContentBuilder) BuildErrorContent(filePath, format, processorType string, err error) *ProcessedContent

BuildErrorContent creates a failed ProcessedContent structure

func (*ProcessedContentBuilder) BuildSuccessContent 

func (pcb *ProcessedContentBuilder) BuildSuccessContent(filePath, text, format, processorType string, pageCount int) *ProcessedContent

BuildSuccessContent creates a successful ProcessedContent structure

type ProcessingError 

type ProcessingError struct {
	FilePath string
	FileType string
	Reason   string
	Err      error
}

ProcessingError represents an error that occurred during media processing

func NewProcessingError 

func NewProcessingError(filePath, fileType, reason string, err error) *ProcessingError

NewProcessingError creates a new processing error

func (*ProcessingError) Error 

func (pe *ProcessingError) Error() string

Error implements the error interface

func (*ProcessingError) Unwrap 

func (pe *ProcessingError) Unwrap() error

Unwrap returns the underlying error

type RecoveryStrategy 

type RecoveryStrategy int

RecoveryStrategy represents different recovery strategies 

const (
	RecoveryStrategyNone RecoveryStrategy = iota
	RecoveryStrategyRetry
	RecoveryStrategyFallback
	RecoveryStrategySkip
)

type ResourceLimits 

type ResourceLimits struct {
	MaxVideoFileSize  int64         // Maximum video file size in bytes
	MaxAudioFileSize  int64         // Maximum audio file size in bytes
	ProcessingTimeout time.Duration // Maximum processing time per file
	MaxMemoryUsage    int64         // Maximum memory usage per file
}

ResourceLimits defines limits for media file processing

func DefaultResourceLimits 

func DefaultResourceLimits() *ResourceLimits

DefaultResourceLimits returns the default resource limits for media processing

type RouterIntegrationHelper 

type RouterIntegrationHelper struct{}

RouterIntegrationHelper provides utilities for router integration

func NewRouterIntegrationHelper 

func NewRouterIntegrationHelper() *RouterIntegrationHelper

NewRouterIntegrationHelper creates a new router integration helper

func (*RouterIntegrationHelper) CreateEmbeddedMediaPath 

func (rih *RouterIntegrationHelper) CreateEmbeddedMediaPath(originalFilePath, embeddedFileName string) string

CreateEmbeddedMediaPath creates a path showing the relationship between original file and embedded media

func (*RouterIntegrationHelper) FormatEmbeddedMediaSection 

func (rih *RouterIntegrationHelper) FormatEmbeddedMediaSection(index int, mediaName, content string) string

FormatEmbeddedMediaSection formats embedded media content for inclusion in metadata text.

The "--- Embedded Media N (name) ---" line is now DISPLAY ONLY: it tells a human reading the extracted text where an embedded item begins, and nothing parses it back. Two readers used to, and both let a document author choose the reported source of a finding just by typing this line as body text — the content router's extractEmbeddedMediaPath (deleted with the rest of the text-sniffing read side) and the METADATA validator's line loop. The real provenance travels out of band in ContentSection.SourceFile, set by the preprocessor that actually opened the archive member.

So do not reintroduce a parser for this text, and do not "fix" a wrong attribution by escaping the parentheses or the name: an author still controls whatever a parser would read, and the structure carries the answer already.

type RouterInterface 

type RouterInterface interface {
	ProcessFile(filePath string, context interface{}) (*ProcessedContent, error)

	// ProcessEmbedded processes a child file that was extracted OUT OF parentPath,
	// enforcing a nesting-depth bound.
	//
	// Separate from ProcessFile because the router is the only component that can own
	// the depth: the preprocessor instance is shared across concurrent workers (so it
	// cannot hold per-call state) and Process takes no context to thread one through.
	// The preprocessor knows its own path — that is the argument to Process — so
	// passing it as parentPath is enough for the router to compute depth without any
	// change to the Preprocessor interface.
	//
	// Returns ErrEmbeddedTooDeep when the bound is reached. Callers must DISCLOSE
	// that rather than skipping quietly: refusing to descend is incomplete coverage,
	// and an undisclosed gap reads as a clean result.
	ProcessEmbedded(childPath, parentPath string) (*ProcessedContent, error)

	// CanProcessFile reports whether the router can process a file at all, and why
	// not when it cannot.
	//
	// Needed so embedded parts are admitted by CAPABILITY rather than by a private
	// extension list. The list version excluded 19% of the embedded parts in a real
	// corpus, including .svg, which scans perfectly well as a standalone file. Asking
	// the router means a preprocessor added anywhere -- including the byte-sniffing
	// text fallback -- extends embedded coverage for free instead of leaving a
	// second list to maintain.
	CanProcessFile(filePath string, enablePreprocessors bool) (bool, string)
}

RouterInterface defines the router functionality the metadata preprocessors need (set via SetRouter). Defined here alongside its only consumer, BaseMetadataPreprocessor and the specialized preprocessors that embed it.

type SectionKind added in v2.2.1 

type SectionKind int

SectionKind is the routing classification of a ContentSection. 

const (
	// SectionKindBody is document body text: the full validator set applies.
	SectionKindBody SectionKind = iota
	// SectionKindMetadata is extracted metadata fields: the METADATA validator's
	// per-preprocessor field rules apply. Note this is about LABELLING, not
	// coverage — the document path scans the union of all sections regardless.
	SectionKindMetadata
)

func ClassifySection added in v2.2.1 

func ClassifySection(preprocessorName string) (SectionKind, string)

ClassifySection maps a preprocessor's GetName() to a section Kind and metadata Type. It lives next to the metadata_constants.go names it matches, and it is a CLOSED switch on the seven names this repo's preprocessors actually return (verified against every GetName() implementation) rather than a substring search.

A closed switch is the security-relevant part. The name it classifies is our own code's constant, so an exact match is always possible; substring matching was how the old text-parsing path turned a document paragraph named "--- Employee Metadata ---" into a metadata section. Anything unrecognized classifies as BODY, which is the fail-closed direction: the document path runs the full validator set, so a preprocessor added later without updating this switch loses a label, not coverage.

type SharedUtilities 

type SharedUtilities struct {
	Formatter          *MetadataFormatter
	ExtensionValidator *FileExtensionValidator
	ContentBuilder     *ProcessedContentBuilder
	RouterHelper       *RouterIntegrationHelper
}

SharedUtilities provides a centralized access point for all shared utilities

func NewSharedUtilities 

func NewSharedUtilities() *SharedUtilities

NewSharedUtilities creates a new shared utilities instance

type TextEncoding added in v2.1.2 

type TextEncoding int

TextEncoding identifies the on-disk encoding of a text file, as detected from its leading bytes. Only encodings ferret-scan can transparently decode to UTF-8 (and re-encode on the redaction write path) are enumerated; everything else is EncodingUnknown and handled by the legacy byte-level heuristics. 

const (
	// EncodingUTF8 is plain UTF-8 / ASCII — no transform needed.
	EncodingUTF8 TextEncoding = iota
	// EncodingUTF8BOM is UTF-8 with a leading BOM (Windows Notepad default).
	// The BOM is stripped on decode and restored on encode.
	EncodingUTF8BOM
	// EncodingUTF16LE is UTF-16 little-endian with BOM (PowerShell 5
	// Out-File default, regedit .reg exports, many Windows logs).
	EncodingUTF16LE
	// EncodingUTF16BE is UTF-16 big-endian with BOM.
	EncodingUTF16BE
	// EncodingUTF16LENoBOM is BOM-less UTF-16 LE, detected by the
	// alternating-null heuristic.
	EncodingUTF16LENoBOM
	// EncodingUTF16BENoBOM is BOM-less UTF-16 BE.
	EncodingUTF16BENoBOM
	// EncodingUnknown means "not a transcodable encoding we recognize" —
	// callers fall back to treating the bytes as-is.
	EncodingUnknown
)

func DetectTextEncoding added in v2.1.2 

func DetectTextEncoding(buf []byte) TextEncoding

DetectTextEncoding inspects the leading bytes of buf (any prefix of the file, e.g. the 512-byte sniff window or the whole content) and identifies the encoding. Detection order matters: BOMs are unambiguous and checked first; the BOM-less UTF-16 heuristic runs only when the buffer contains null bytes in the tell-tale alternating pattern (UTF-16-encoded ASCII/Latin text has a 0x00 in every other byte, which is exactly why the legacy null-byte check classified such files as binary).

func (TextEncoding) String added in v2.1.2 

func (e TextEncoding) String() string

String returns a short name for metadata/debug surfaces.

type TextPosition 

type TextPosition struct {
	// Line is the line number (1-based)
	Line int `json:"line"`

	// StartChar is the starting character position in the line (0-based)
	StartChar int `json:"start_char"`

	// EndChar is the ending character position in the line (0-based)
	EndChar int `json:"end_char"`

	// AbsoluteOffset is the absolute character offset from the beginning of the text
	AbsoluteOffset int `json:"absolute_offset"`
}

TextPosition represents a position in extracted text

type TextPreprocessor 

type TextPreprocessor struct {
	// contains filtered or unexported fields
}

TextPreprocessor handles text extraction from various document formats

func NewTextPreprocessor 

func NewTextPreprocessor() *TextPreprocessor

NewTextPreprocessor creates a new text preprocessor

func (*TextPreprocessor) CanProcess 

func (tp *TextPreprocessor) CanProcess(filePath string) bool

CanProcess checks if this preprocessor can handle the given file

func (*TextPreprocessor) GetName 

func (tp *TextPreprocessor) GetName() string

GetName returns the name of this preprocessor

func (*TextPreprocessor) GetSupportedExtensions 

func (tp *TextPreprocessor) GetSupportedExtensions() []string

GetSupportedExtensions returns the file extensions this preprocessor supports

func (*TextPreprocessor) Process 

func (tp *TextPreprocessor) Process(filePath string) (*ProcessedContent, error)

Process extracts text content from the file

func (*TextPreprocessor) SetObserver 

func (tp *TextPreprocessor) SetObserver(observer observability.Observer)

SetObserver sets the observability component

type VideoMetadataPreprocessor 

type VideoMetadataPreprocessor struct {
	*BaseMetadataPreprocessor
}

VideoMetadataPreprocessor extracts metadata from video files

func NewVideoMetadataPreprocessor 

func NewVideoMetadataPreprocessor() *VideoMetadataPreprocessor

NewVideoMetadataPreprocessor creates a new video metadata preprocessor

func (*VideoMetadataPreprocessor) CanProcess 

func (vmp *VideoMetadataPreprocessor) CanProcess(filePath string) bool

CanProcess checks if this preprocessor can handle the given file

func (*VideoMetadataPreprocessor) GetSupportedExtensions 

func (vmp *VideoMetadataPreprocessor) GetSupportedExtensions() []string

GetSupportedExtensions returns the file extensions this preprocessor supports

func (*VideoMetadataPreprocessor) Process 

func (vmp *VideoMetadataPreprocessor) Process(filePath string) (*ProcessedContent, error)

Process extracts metadata from video files with comprehensive error handling and retry logic

func (*VideoMetadataPreprocessor) SetObserver 

func (vmp *VideoMetadataPreprocessor) SetObserver(observer observability.Observer)

SetObserver sets the observability component

func (*VideoMetadataPreprocessor) SetRouter 

func (vmp *VideoMetadataPreprocessor) SetRouter(router RouterInterface)

SetRouter sets the router instance (not used for video metadata but required by interface)

Source Files

View all Source files

Directories

Path Synopsis
meta-extractors
meta-extract-audiolib
meta-extract-exiflib
meta-extract-officelib
meta-extract-pdflib
meta-extract-videolib
text-extractors
text-extract-officetextlib
text-extract-pdftextlib

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.