multimodal

package
v1.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 5, 2026 License: MIT Imports: 23 Imported by: 0

Documentation

Overview

Package multimodal — audio plugin client (Bug #1).

Routes audio files to an external transcription service that exposes the standard Gleann plugin contract: 

POST /convert (multipart file=<audio>) → { "markdown": "<transcript>" }

This matches the gleann-plugin-sound API so installing the whisper.cpp plugin transparently enables high-quality multilingual transcription without code changes.

Package multimodal — content-hash cache layer.

Bug #3 fix: avoid re-invoking the VLM for files that have not changed. We hash file bytes (SHA-256) plus model+prompt-version and cache the result JSON under ~/.gleann/cache/multimodal/<sha>.json. Subsequent runs short-circuit ProcessFile to return the cached description.

Package multimodal provides model-native multimodal processing for gleann.

Package multimodal — file metadata extractor (Bug #9).

Provides format-aware metadata that complements the VLM description so downstream consumers (graph indexer, search filters) can query by dimensions, timestamp, or exiftool-derived camera fields.

The pure-Go path uses image.DecodeConfig (registered with all stdlib image formats) to obtain dimensions cheaply without decoding the full pixel buffer. When `exiftool` is on PATH and `GLEANN_EXIFTOOL=1` is set the extractor additionally collects camera/GPS/datetime tags.

Package multimodal provides model-native multimodal processing for gleann. Instead of using external plugins for audio/image, it leverages Ollama's multimodal models (Gemma4, Qwen3-VL) to describe media content as text, which can then be indexed and searched like any other document.

Package multimodal provides model-native multimodal processing for gleann.

Package multimodal — streaming VLM (Bug #8).

Adds Processor.ProcessFileStream which mirrors ProcessFile but invokes a per-token callback as Ollama produces output. The result still passes through the content-hash cache so subsequent calls return instantly via a single synthetic on-token chunk.

Package multimodal provides model-native multimodal processing for gleann.

Package multimodal — vision plugin client (Layer B/C, Bug #4/#7/#11).

This file defines the v2 plugin schema used to enrich an IndexableItem with structured signals from a dedicated computer-vision plugin (gleann-plugin-vision): a CLIP joint-space embedding, OCR text, an object/entity list, and an EXIF map. Each field is optional so older plugins that only return markdown continue to work unchanged.

The schema is intentionally JSON-stable to allow third-party plugins to implement it without depending on the Go module.

Index

Constants

View Source 
const PromptVersion = "v2"

PromptVersion is bumped whenever the prompt templates change so that cached entries from older Gleann versions are invalidated automatically.

Variables

This section is empty.

Functions

func AutoDetectModel 

func AutoDetectModel(ollamaHost string) string

AutoDetectModel queries Ollama for available multimodal models and returns the best one.

func CacheStats added in v1.5.0 

func CacheStats() (hits, misses int64)

CacheStats returns observed hit/miss counters since process start.

func CleanupFrames added in v1.5.0 

func CleanupFrames(frames []ExtractedFrame)

CleanupFrames removes the temporary frame directory.

func CleanupPDFPages added in v1.5.0 

func CleanupPDFPages(pages []PDFPageResult)

CleanupPDFPages removes the temporary page image directory.

func IsMultimodal 

func IsMultimodal(path string) bool

IsMultimodal returns true if the file is an audio, image, or video file that can be processed by a multimodal model.

func RenderPDFPageToBase64 added in v1.6.0 

func RenderPDFPageToBase64(pdfPath string, pageNum int, dpi int) (string, error)

RenderPDFPageToBase64 renders a specific page of a PDF and returns it as a base64 string.

func ResetCacheStats added in v1.5.0 

func ResetCacheStats()

ResetCacheStats is a test helper.

func VisionPluginHealthy added in v1.5.0 

func VisionPluginHealthy(pluginURL string) bool

VisionPluginHealthy reports whether the vision plugin /health endpoint returns 200 within a small timeout. Useful for the TUI status panel and ProcessDirectory pre-checks.

func VisionPluginURL added in v1.5.0 

func VisionPluginURL() string

VisionPluginURL returns the configured base URL of the vision plugin, or empty if disabled. Honours GLEANN_VISION_PLUGIN_URL=off as an explicit opt-out so users can keep the plugin installed but bypass it.

Types

type Chart added in v1.5.0 

type Chart struct {
	Type        string           `json:"type"`        // "bar", "line", "pie", "scatter", "diagram", "other"
	Title       string           `json:"title"`       // chart title
	Description string           `json:"description"` // detailed description
	DataPoints  []ChartDataPoint `json:"data_points"` // extracted data if possible
	Labels      []string         `json:"labels"`      // axis labels or legend items
}

Chart represents a single extracted chart/figure.

type ChartDataPoint added in v1.5.0 

type ChartDataPoint struct {
	Label string  `json:"label"`
	Value float64 `json:"value"`
}

ChartDataPoint represents a single data point from a chart.

type ChartExtractionResult added in v1.5.0 

type ChartExtractionResult struct {
	PageNum int     `json:"page_num"`
	Charts  []Chart `json:"charts"`
	RawText string  `json:"raw_text"`
}

ChartExtractionResult holds structured data from a chart/figure.

type ExtractedFrame added in v1.5.0 

type ExtractedFrame struct {
	Path      string  // Path to the extracted frame image.
	Timestamp float64 // Timestamp in seconds.
	Index     int     // Frame index (0-based).
}

ExtractedFrame represents a single frame extracted from a video.

func ExtractFrames added in v1.5.0 

func ExtractFrames(videoPath string, cfg FrameExtractionConfig) ([]ExtractedFrame, error)

ExtractFrames extracts keyframes from a video file using ffmpeg. Returns paths to extracted frame images in a temp directory. Requires ffmpeg to be installed.

type FaithfulnessCheck added in v1.5.0 

type FaithfulnessCheck struct {
	Rule     string `json:"rule"` // what was checked
	Type     string `json:"type"` // "omission" or "hallucination"
	Passed   bool   `json:"passed"`
	Evidence string `json:"evidence"` // relevant text snippet
}

FaithfulnessCheck represents a single content faithfulness check.

type FaithfulnessResult added in v1.5.0 

type FaithfulnessResult struct {
	Score         float64             `json:"score"`               // 0-100 overall faithfulness
	OmissionCount int                 `json:"omission_count"`      // content in source but missing from extraction
	HallucinCount int                 `json:"hallucination_count"` // content in extraction but not in source
	TotalChecks   int                 `json:"total_checks"`
	Details       []FaithfulnessCheck `json:"details"`
}

FaithfulnessResult holds content faithfulness analysis results.

func CheckFaithfulness added in v1.5.0 

func CheckFaithfulness(sourceText, extractedText string) *FaithfulnessResult

CheckFaithfulness compares extracted text against source content using rule-based heuristics for omission and hallucination detection.

type FileMetadata added in v1.5.0 

type FileMetadata struct {
	Width     int               `json:"width,omitempty"`
	Height    int               `json:"height,omitempty"`
	Format    string            `json:"format,omitempty"`
	SizeBytes int64             `json:"size_bytes,omitempty"`
	ModTime   time.Time         `json:"mod_time,omitempty"`
	Exif      map[string]string `json:"exif,omitempty"` // optional, exiftool when enabled
}

FileMetadata captures lightweight, format-aware facts about a media file.

func ExtractFileMetadata added in v1.5.0 

func ExtractFileMetadata(path string) (FileMetadata, error)

ExtractFileMetadata returns metadata for the given file. The function never returns an error for missing optional data; instead it populates only the fields it could read. A missing file or unreadable header is reported via the returned error.

type FrameExtractionConfig added in v1.5.0 

type FrameExtractionConfig struct {
	MaxFrames      int     // Maximum number of frames to extract (default: 8).
	FPS            float64 // Frames per second to sample (0 = auto-calculate from MaxFrames).
	Width          int     // Resize width (0 = original).
	Quality        int     // JPEG quality 1-100 (default: 85).
	SceneThreshold float64 // Bug #5: 0 = fixed-FPS, >0 = ffmpeg scene-change filter (0.3–0.5 typical).
}

FrameExtractionConfig controls how frames are sampled from a video.

func DefaultFrameConfig added in v1.5.0 

func DefaultFrameConfig() FrameExtractionConfig

DefaultFrameConfig returns sensible defaults for frame extraction. SceneThreshold defaults to 0.4 so meaningful cuts are preferred over uniform sampling; set to 0 to revert to the legacy fixed-FPS behaviour.

type IndexableItem added in v1.5.0 

type IndexableItem struct {
	Source      string // Original file path.
	Text        string // LLM-generated description text.
	MediaType   MediaType
	Description string // Same as Text, kept for clarity.

	// Layer C / Bug #4 #7 #11 enrichments. All optional; nil/empty when
	// the vision plugin is not configured or did not return that field.
	OCRText       string         `json:"ocr_text,omitempty"`
	CLIPEmbedding []float32      `json:"clip_embedding,omitempty"`
	Entities      []VisionEntity `json:"entities,omitempty"`
	Metadata      *FileMetadata  `json:"metadata,omitempty"`
}

IndexableItem represents a multimodal file converted to text for vector indexing.

type MediaType 

type MediaType int

MediaType classifies a file into a processing category. 

const (
	MediaTypeUnknown MediaType = iota
	MediaTypeImage
	MediaTypeAudio
	MediaTypeVideo
	MediaTypeDocument // handled by plugins, not multimodal
)

func DetectMediaType 

func DetectMediaType(path string) MediaType

DetectMediaType returns the media category for a file path.

type ModelCapabilities 

type ModelCapabilities struct {
	Name       string
	Vision     bool
	Audio      bool
	Multimodal bool
}

ModelCapabilities describes what a model can handle.

func DetectCapabilities 

func DetectCapabilities(ollamaHost, modelName string) ModelCapabilities

DetectCapabilities queries Ollama for a model's multimodal support.

type PDFAnalysis added in v1.5.0 

type PDFAnalysis struct {
	SourcePath string
	Pages      []PDFPageResult
	TotalPages int
}

PDFAnalysis holds the complete analysis of a PDF document.

type PDFPageResult added in v1.5.0 

type PDFPageResult struct {
	PageNum     int                    // 1-based page number.
	ImagePath   string                 // Path to rendered page image.
	Description string                 // VLM-generated description.
	HasTable    bool                   // Whether a table was detected.
	HasChart    bool                   // Whether a chart/figure was detected.
	Tables      *TableExtractionResult // Extracted tables (nil if no tables detected).
	Charts      *ChartExtractionResult // Extracted charts (nil if no charts detected).
	MarkerText  string                 // Text from marker plugin (if available).
	Error       error
}

PDFPageResult holds the analysis of a single PDF page.

type PDFVisionConfig added in v1.5.0 

type PDFVisionConfig struct {
	DPI        int  // Render DPI for PDF pages (default: 150).
	MaxPages   int  // Max pages to process (0 = all).
	UseMarker  bool // Try gleann-plugin-marker first, VLM fallback for tables/charts.
	MarkerOnly bool // Bug #6: when marker returns text for a page, skip the VLM entirely.
}

PDFVisionConfig controls the PDF vision pipeline.

func DefaultPDFConfig added in v1.5.0 

func DefaultPDFConfig() PDFVisionConfig

DefaultPDFConfig returns sensible defaults.

type ProcessResult 

type ProcessResult struct {
	FilePath    string
	MediaType   MediaType
	Description string // Text description of the media content.
	Error       error
}

ProcessResult holds the output of multimodal processing.

type Processor 

type Processor struct {
	OllamaHost string
	Model      string
	Lang       string // "en" (default) or "tr" — Bug #10
}

Processor handles multimodal file processing via Ollama.

func NewProcessor 

func NewProcessor(ollamaHost, model string) *Processor

NewProcessor creates a multimodal processor. If model is empty, it tries to auto-detect from GLEANN_MULTIMODAL_MODEL env. Language defaults to GLEANN_MULTIMODAL_LANG ("en" if unset).

func (*Processor) AnalyzePDF added in v1.5.0 

func (p *Processor) AnalyzePDF(pdfPath string, cfg PDFVisionConfig) (*PDFAnalysis, error)

AnalyzePDF processes a PDF using a hybrid pipeline: 1. If marker plugin is available and UseMarker is true, get text extraction first 2. Render pages to images using pdftoppm/mutool 3. Send page images to VLM for table/chart detection and description

func (*Processor) AnalyzeVideo added in v1.5.0 

func (p *Processor) AnalyzeVideo(videoPath string, cfg FrameExtractionConfig) (*VideoAnalysis, error)

AnalyzeVideo extracts frames from a video, processes each with the multimodal model, and returns a combined analysis with per-frame descriptions and a summary.

func (*Processor) CanProcess 

func (p *Processor) CanProcess(path string) bool

CanProcess returns true if the processor is configured and the file is multimodal.

func (*Processor) ExtractCharts added in v1.5.0 

func (p *Processor) ExtractCharts(pageImagePath string, pageNum int) (*ChartExtractionResult, error)

ExtractCharts sends a page image to the VLM with a chart-focused prompt and parses the response into structured chart data.

func (*Processor) ExtractTables added in v1.5.0 

func (p *Processor) ExtractTables(pageImagePath string, pageNum int) (*TableExtractionResult, error)

ExtractTables sends a page image to the VLM with a table-focused prompt and parses the response into structured table data.

func (*Processor) ProcessDirectory added in v1.5.0 

func (p *Processor) ProcessDirectory(dir string, skipExts []string, progressFn func(int, int, string)) ([]IndexableItem, error)

ProcessDirectory scans a directory for multimodal files (images, audio, video) and generates text descriptions for each using the configured Ollama model. The returned items can be passed directly to LeannBuilder.Build().

skipExts optionally lists extensions to skip (e.g., ".svg" which tree-sitter handles). progressFn is called after each file with (current, total, path).

Bug #2 fix: files are processed by a worker pool (default 4 workers, override with GLEANN_MULTIMODAL_WORKERS) so a large directory does not pay the full Ollama latency sequentially. Ordering of the returned slice matches the discovered file ordering for reproducibility.

func (*Processor) ProcessFile 

func (p *Processor) ProcessFile(path string) ProcessResult

ProcessFile sends a file to the multimodal model and returns a text description.

func (*Processor) ProcessFileStream added in v1.5.0 

func (p *Processor) ProcessFileStream(path string, onToken TokenCallback) ProcessResult

ProcessFileStream is the streaming sibling of ProcessFile. It returns the same ProcessResult, but additionally invokes onToken for every chunk so that TUIs and APIs can render incremental output.

On cache hit the cached description is forwarded as a single token so callers can treat the streaming path uniformly.

type Table added in v1.5.0 

type Table struct {
	Caption  string     `json:"caption,omitempty"`
	Headers  []string   `json:"headers"`
	Rows     [][]string `json:"rows"`
	Markdown string     `json:"markdown"` // markdown representation
}

Table represents a single extracted table.

type TableExtractionResult added in v1.5.0 

type TableExtractionResult struct {
	PageNum int     `json:"page_num"`
	Tables  []Table `json:"tables"`
	RawText string  `json:"raw_text"` // VLM's raw description
}

TableExtractionResult holds structured table data extracted from a page image.

type TokenCallback added in v1.5.0 

type TokenCallback func(token string) error

TokenCallback receives streamed tokens from the VLM. Errors returned by the callback abort the stream.

type VideoAnalysis added in v1.5.0 

type VideoAnalysis struct {
	SourcePath   string
	Frames       []ExtractedFrame
	Descriptions []string // One per frame, from multimodal model.
	Summary      string   // Combined summary of all frames.
	Duration     float64  // Video duration in seconds.
}

VideoAnalysis holds the results of video frame extraction and analysis.

type VisionEntity added in v1.5.0 

type VisionEntity struct {
	Type       string    `json:"type"`           // "Object" | "Person" | "Logo" | "Text"
	Label      string    `json:"label"`          // e.g. "laptop", "stop-sign"
	Confidence float32   `json:"confidence"`     // 0..1
	BBox       []float32 `json:"bbox,omitempty"` // [x, y, w, h] normalised
}

VisionEntity is a single detection from an object/entity recogniser.

type VisionResult added in v1.5.0 

type VisionResult struct {
	SchemaVersion int               `json:"schema_version"`
	Markdown      string            `json:"markdown,omitempty"`
	OCRText       string            `json:"ocr_text,omitempty"`
	CLIPEmbedding []float32         `json:"clip_embedding,omitempty"`
	Entities      []VisionEntity    `json:"entities,omitempty"`
	Exif          map[string]string `json:"exif,omitempty"`
	Width         int               `json:"width,omitempty"`
	Height        int               `json:"height,omitempty"`
	Capabilities  []string          `json:"capabilities,omitempty"`
}

VisionResult is the parsed /convert response from a v2 vision plugin. All fields except SchemaVersion are optional.

func CallVisionPlugin added in v1.5.0 

func CallVisionPlugin(pluginURL, path string) (*VisionResult, error)

CallVisionPlugin uploads an image to the vision plugin's /convert endpoint and returns the parsed VisionResult. Errors are returned so callers can decide whether to fall back to VLM-only mode.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.