Documentation
¶
Overview ¶
Package vision turns images into searchable text: a vision LLM describes an image, and the resulting title/description/transcription is indexed as ordinary markdown by every backend (full-text, vector, metadata filtering).
The package is deliberately independent of the ingestion pipeline: it is used by convert/vision to index standalone image files, and by the markdown enrichment of embedded images. Callers pass an llm.Client already decorated with llmx.NewRetryClient (retry + rate limit), exactly as the HyDE and Judge retrieval stages do.
Index ¶
Constants ¶
const DefaultMaxImageBytes int64 = 10 << 20 // 10 MiB
DefaultMaxImageBytes bounds, by default, the size of an image submitted to the vision model. Providers reject oversized payloads anyway, and the base64 encoding inflates the request by a third.
const DefaultPrompt = `` /* 201-byte string literal not displayed */
DefaultPrompt asks for a dense, factual description. The description is what gets indexed and searched, so it must be rich in the vocabulary a user would actually type: named entities, object names, chart values, UI labels.
It is deliberately terse, and the detailed field requirements live in descriptionSchema instead. Spelling them out here made an OCR-specialized model (glm-ocr) — whose instinct is to transcribe whatever text it is handed — copy the instructions themselves into the answer, which would then have been indexed verbatim for every image. Keep any change to this prompt short and free of enumerations, and re-run the vision integration test: it asserts that no fragment of the prompt comes back in the answer.
const MaxTitleRunes = 120
MaxTitleRunes is the length beyond which a "title" is not one. The title becomes the heading of the emitted markdown, so a model answering with a whole paragraph — or with the instructions it was given — must not turn it into a monstrous `#` line.
const PromptVersion = "1"
PromptVersion identifies the default prompt. It participates in the cache key (see CachingDescriber): bump it whenever DefaultPrompt changes, so descriptions produced by an older prompt are not served from the cache.
Variables ¶
var ErrImageTooLarge = errors.New("image too large")
ErrImageTooLarge is returned when an image exceeds the configured size limit. It is returned *before* any call to the model.
Functions ¶
Types ¶
type CachingDescriber ¶
type CachingDescriber struct {
// contains filtered or unexported fields
}
CachingDescriber decorates a Describer with a persistent on-disk cache keyed by the *content* of the image — sha256(promptVersion, namespace, bytes) — sharded on the first hex byte below dir/vision, the same layout as llmx.CachingClient.
A dedicated cache is required: llmx.CachingClient deliberately refuses to cache any chat completion carrying an attachment (serializing the image into a JSON key would be wasteful), so vision calls would otherwise never be cached. Keying on the bytes rather than on the file path also means a renamed, moved or duplicated image is described only once, and a full re-index costs nothing.
The namespace must identify the vision model *and* its prompt — build it with Namespace(model, prompt) — otherwise switching model would serve descriptions produced by the previous one.
func NewCachingDescriber ¶
func NewCachingDescriber(describer Describer, dir, namespace string) (*CachingDescriber, error)
NewCachingDescriber wraps describer with a cache rooted at dir (created if missing), keyed under namespace.
func (*CachingDescriber) Describe ¶
func (c *CachingDescriber) Describe(ctx context.Context, mimeType string, data []byte) (*Description, error)
Describe implements Describer.
func (*CachingDescriber) MaxImageBytes ¶
func (c *CachingDescriber) MaxImageBytes() int64
MaxImageBytes reports the limit of the wrapped describer, when it exposes one, so the decorator stays transparent to size-aware callers.
func (*CachingDescriber) Stats ¶
func (c *CachingDescriber) Stats() (hits, misses int64)
Stats returns the number of cache hits and misses served so far.
type Describer ¶
type Describer interface {
Describe(ctx context.Context, mimeType string, data []byte) (*Description, error)
}
Describer produces a textual description of an image.
type Description ¶
type Description struct {
// Title is a short, one-line title for the image.
Title string `json:"title"`
// Description is the detailed markdown description: content, context,
// notable elements, data of charts and diagrams.
Description string `json:"description"`
// Text is the text visible in the image, transcribed verbatim (implicit
// OCR). Empty when the image carries no text.
Text string `json:"text"`
}
Description is the structured result of describing an image.
func (*Description) IsEmpty ¶
func (d *Description) IsEmpty() bool
IsEmpty reports whether the description carries no usable text at all.
type LLMDescriber ¶
type LLMDescriber struct {
// contains filtered or unexported fields
}
LLMDescriber describes images with a vision-capable chat model.
func NewLLMDescriber ¶
func NewLLMDescriber(client llm.Client, funcs ...OptionFunc) *LLMDescriber
NewLLMDescriber builds a Describer on top of client, which must be a vision-capable chat model and is expected to be already decorated with llmx.NewRetryClient by the caller.
func (*LLMDescriber) Describe ¶
func (d *LLMDescriber) Describe(ctx context.Context, mimeType string, data []byte) (*Description, error)
Describe implements Describer.
func (*LLMDescriber) MaxImageBytes ¶
func (d *LLMDescriber) MaxImageBytes() int64
MaxImageBytes reports the configured image size limit, so callers can bound their reads before handing over the bytes.
func (*LLMDescriber) Prompt ¶
func (d *LLMDescriber) Prompt() string
Prompt reports the effective description prompt (used to derive the cache namespace).
type OptionFunc ¶
type OptionFunc func(opts *Options)
func WithMaxImageBytes ¶
func WithMaxImageBytes(maxBytes int64) OptionFunc
WithMaxImageBytes bounds the size of an accepted image; <= 0 keeps the default (DefaultMaxImageBytes).
func WithPrompt ¶
func WithPrompt(prompt string) OptionFunc
WithPrompt replaces the default description prompt. A custom prompt must be reflected in the cache namespace (see Namespace).
func WithStructuredOutput ¶
func WithStructuredOutput(enabled bool) OptionFunc
WithStructuredOutput toggles the JSON response schema.
type Options ¶
type Options struct {
Prompt string
// MaxImageBytes is the largest image accepted; above it Describe fails
// with ErrImageTooLarge without calling the model.
MaxImageBytes int64
// StructuredOutput requests a JSON response matching Description. Disable
// it for providers that reject a response schema; the whole reply then
// lands in Description.
StructuredOutput bool
}
Options configures an LLMDescriber.
func NewOptions ¶
func NewOptions(funcs ...OptionFunc) *Options
Source Files