Documentation
¶
Overview ¶
Package vision turns images into searchable text: a vision LLM describes an image, and the resulting title/description/transcription is indexed as ordinary markdown by every backend (full-text, vector, metadata filtering).
The package is deliberately independent of the ingestion pipeline: it is used by convert/vision to index standalone image files, and by the markdown enrichment of embedded images. Callers pass an llm.Client already decorated with llmx.NewRetryClient (retry + rate limit), exactly as the HyDE and Judge retrieval stages do.
Index ¶
Constants ¶
const DefaultMaxImageBytes int64 = 10 << 20 // 10 MiB
DefaultMaxImageBytes bounds, by default, the size of an image submitted to the vision model. Providers reject oversized payloads anyway, and the base64 encoding inflates the request by a third.
const DefaultMaxSourceBytes int64 = 64 << 20 // 64 MiB
DefaultMaxSourceBytes bounds the size of an image *before* shrinking: an image between MaxImageBytes and this limit is re-encoded to fit (see Shrink), a larger one is refused outright. It exists because shrinking decodes the pixels into memory — a 300 MiB PNG is not worth the RAM of a whole ingestion worker.
const DefaultPrompt = `` /* 201-byte string literal not displayed */
DefaultPrompt asks for a dense, factual description. The description is what gets indexed and searched, so it must be rich in the vocabulary a user would actually type: named entities, object names, chart values, UI labels.
It is deliberately terse, and the detailed field requirements live in descriptionSchema instead. Spelling them out here made an OCR-specialized model (glm-ocr) — whose instinct is to transcribe whatever text it is handed — copy the instructions themselves into the answer, which would then have been indexed verbatim for every image. Keep any change to this prompt short and free of enumerations, and re-run the vision integration test: it asserts that no fragment of the prompt comes back in the answer.
const MaxTitleRunes = 120
MaxTitleRunes is the length beyond which a "title" is not one. The title becomes the heading of the emitted markdown, so a model answering with a whole paragraph — or with the instructions it was given — must not turn it into a monstrous `#` line.
const PromptVersion = "1"
PromptVersion identifies the default prompt. It participates in the cache key (see CachingDescriber): bump it whenever DefaultPrompt changes, so descriptions produced by an older prompt are not served from the cache.
const ShrinkMaxDimension = 2048
ShrinkMaxDimension caps the longest side of a re-encoded image. The vision providers downscale anything larger anyway (Anthropic works at ~1568px), so carrying more pixels costs bytes without buying any detail.
const ShrinkQuality = 85
ShrinkQuality is the JPEG quality used when re-encoding an oversized image. High enough to keep the small text of a screenshot or a dashboard legible, which is precisely what the description has to transcribe.
Variables ¶
var ErrImageTooLarge = errors.New("image too large")
ErrImageTooLarge is returned when an image exceeds the configured size limit and could not be shrunk below it (see Shrink). It is returned *before* any call to the model.
var ErrUnsupportedImageFormat = errors.New("unsupported image format")
ErrUnsupportedImageFormat is returned when the media type of an image is not one the vision providers accept. Like ErrImageTooLarge, it is returned *before* any call to the model: the provider client rejects such an image while building its request parameters, and reports it as a generic "unavailable" error that is indistinguishable from a transient outage — so it would otherwise be retried, pointlessly, on every image.
var SupportedMimeTypes = []string{
"image/png",
"image/jpeg",
"image/webp",
"image/gif",
}
SupportedMimeTypes are the media types accepted by the mainstream vision providers (and the only ones the OpenAI-compatible client of genai lets through). It mirrors convert/vision.DefaultExtensions.
Functions ¶
func IsSupportedMimeType ¶ added in v0.11.0
IsSupportedMimeType reports whether a vision provider accepts mimeType. Any media type parameter (`; charset=...`, as returned by http.DetectContentType) is ignored, and the comparison is case-insensitive.
func Namespace ¶
Namespace derives a cache namespace identifying both the vision model and the prompt it was given: two descriptions may only share a cache entry when both match. Pass it to NewCachingDescriber.
func Shrink ¶ added in v0.11.0
Shrink re-encodes data as JPEG so that it fits within maxBytes, and reports the media type of the result. Data already under the limit is returned untouched, with its original media type.
It is what keeps a big screenshot — a Grafana dashboard exported as a 12 MiB PNG is the typical case — from failing the indexation of the file that carries it: those images are large because PNG stores them losslessly, not because they hold that much detail.
An image that cannot be decoded (no Go decoder for the format) or that cannot be brought under the limit without falling below a legible size is an error: the caller decides whether that is fatal.
Types ¶
type CachingDescriber ¶
type CachingDescriber struct {
// contains filtered or unexported fields
}
CachingDescriber decorates a Describer with a persistent on-disk cache keyed by the *content* of the image — sha256(promptVersion, namespace, bytes) — sharded on the first hex byte below dir/vision, the same layout as llmx.CachingClient.
A dedicated cache is required: llmx.CachingClient deliberately refuses to cache any chat completion carrying an attachment (serializing the image into a JSON key would be wasteful), so vision calls would otherwise never be cached. Keying on the bytes rather than on the file path also means a renamed, moved or duplicated image is described only once, and a full re-index costs nothing.
The namespace must identify the vision model *and* its prompt — build it with Namespace(model, prompt) — otherwise switching model would serve descriptions produced by the previous one.
func NewCachingDescriber ¶
func NewCachingDescriber(describer Describer, dir, namespace string) (*CachingDescriber, error)
NewCachingDescriber wraps describer with a cache rooted at dir (created if missing), keyed under namespace.
func (*CachingDescriber) Describe ¶
func (c *CachingDescriber) Describe(ctx context.Context, mimeType string, data []byte) (*Description, error)
Describe implements Describer.
func (*CachingDescriber) MaxImageBytes ¶
func (c *CachingDescriber) MaxImageBytes() int64
MaxImageBytes reports the limit of the wrapped describer, when it exposes one, so the decorator stays transparent to size-aware callers.
func (*CachingDescriber) MaxSourceBytes ¶ added in v0.11.0
func (c *CachingDescriber) MaxSourceBytes() int64
MaxSourceBytes reports the source limit of the wrapped describer, when it exposes one, so the decorator stays transparent to size-aware callers.
func (*CachingDescriber) Stats ¶
func (c *CachingDescriber) Stats() (hits, misses int64)
Stats returns the number of cache hits and misses served so far.
type Describer ¶
type Describer interface {
Describe(ctx context.Context, mimeType string, data []byte) (*Description, error)
}
Describer produces a textual description of an image.
type Description ¶
type Description struct {
// Title is a short, one-line title for the image.
Title string `json:"title"`
// Description is the detailed markdown description: content, context,
// notable elements, data of charts and diagrams.
Description string `json:"description"`
// Text is the text visible in the image, transcribed verbatim (implicit
// OCR). Empty when the image carries no text.
Text string `json:"text"`
}
Description is the structured result of describing an image.
func (*Description) IsEmpty ¶
func (d *Description) IsEmpty() bool
IsEmpty reports whether the description carries no usable text at all.
type LLMDescriber ¶
type LLMDescriber struct {
// contains filtered or unexported fields
}
LLMDescriber describes images with a vision-capable chat model.
func NewLLMDescriber ¶
func NewLLMDescriber(client llm.Client, funcs ...OptionFunc) *LLMDescriber
NewLLMDescriber builds a Describer on top of client, which must be a vision-capable chat model and is expected to be already decorated with llmx.NewRetryClient by the caller.
func (*LLMDescriber) Describe ¶
func (d *LLMDescriber) Describe(ctx context.Context, mimeType string, data []byte) (*Description, error)
Describe implements Describer.
func (*LLMDescriber) MaxImageBytes ¶
func (d *LLMDescriber) MaxImageBytes() int64
MaxImageBytes reports the configured image size limit, so callers can bound their reads before handing over the bytes.
func (*LLMDescriber) MaxSourceBytes ¶ added in v0.11.0
func (d *LLMDescriber) MaxSourceBytes() int64
MaxSourceBytes reports the largest image Describe accepts before shrinking, so callers can bound their reads accordingly.
func (*LLMDescriber) Prompt ¶
func (d *LLMDescriber) Prompt() string
Prompt reports the effective description prompt (used to derive the cache namespace).
type OptionFunc ¶
type OptionFunc func(opts *Options)
func WithMaxImageBytes ¶
func WithMaxImageBytes(maxBytes int64) OptionFunc
WithMaxImageBytes bounds the size of an accepted image; <= 0 keeps the default (DefaultMaxImageBytes).
func WithMaxSourceBytes ¶ added in v0.11.0
func WithMaxSourceBytes(maxBytes int64) OptionFunc
WithMaxSourceBytes bounds the size of an image accepted for shrinking; <= 0 keeps the default (DefaultMaxSourceBytes). It is always raised to at least MaxImageBytes.
func WithPrompt ¶
func WithPrompt(prompt string) OptionFunc
WithPrompt replaces the default description prompt. A custom prompt must be reflected in the cache namespace (see Namespace).
func WithStructuredOutput ¶
func WithStructuredOutput(enabled bool) OptionFunc
WithStructuredOutput toggles the JSON response schema.
type Options ¶
type Options struct {
Prompt string
// MaxImageBytes is the largest image submitted to the model; a larger one
// is re-encoded to fit (see Shrink).
MaxImageBytes int64
// MaxSourceBytes is the largest image Describe accepts at all: between
// MaxImageBytes and this limit an image is shrunk, above it Describe fails
// with ErrImageTooLarge without decoding it.
MaxSourceBytes int64
// StructuredOutput requests a JSON response matching Description. Disable
// it for providers that reject a response schema; the whole reply then
// lands in Description.
StructuredOutput bool
}
Options configures an LLMDescriber.
func NewOptions ¶
func NewOptions(funcs ...OptionFunc) *Options
Source Files