vision

Documentation

Overview

Package vision is goinfer's pure-Go image preprocessing for vision-language models: decode → resize → normalize → pixel_values, the tensor a vision encoder (SigLIP / ViT) consumes. Stdlib-only (no cgo), and it bounds the decoded pixel count BEFORE decoding so a hostile/corrupt image yields a typed error, never an OOM (the campaign Track-2 posture, extended to image bytes).

Parity note: the resize here is bilinear (half-pixel centers). HF/PIL Gemma 3 uses BICUBIC, so this is NOT pixel-exact yet — per goinfer's multimodal.md §2 the end-to-end gate runs on precomputed pixel_values, and a PIL-exact separable resampler is a follow-on. This file is the structure + the security guard.

"multimodal.md" refers to https://github.com/townsendmerino/goinfer/blob/main/docs/multimodal.md.

Index

• func RegisterResident(f func(*Encoder) (ResidentEncoder, error))
• type Config
  • func Gemma3() Config
• type Encoder
  • func LoadEncoder(dir string, quant bool) (*Encoder, error)
  • func (e *Encoder) Close()
  • func (e *Encoder) EnableResident() error
  • func (e *Encoder) Forward(pixels []float32) ([]float32, error)
  • func (e *Encoder) GPUWeights() (GPUWeights, error)
  • func (e *Encoder) GridPatches(pixels []float32) ([]float32, error)
• type EncoderConfig
• type GPULayer
• type GPUMat
• type GPUWeights
• type PixelValues
  • func Preprocess(data []byte, cfg Config) (*PixelValues, error)
• type QwenEncoderConfig
• type QwenVisionEncoder
  • func LoadQwenVisionEncoder(dir string, quant bool) (*QwenVisionEncoder, error)
  • func (e *QwenVisionEncoder) Forward(pixelValues []float32, gridTHW [][3]int) ([]float32, error)
  • func (e *QwenVisionEncoder) ForwardViT(pixelValues []float32, gridTHW [][3]int) ([]float32, error)
• type ResidentEncoder

Functions

func RegisterResident

func RegisterResident(f func(*Encoder) (ResidentEncoder, error))

RegisterResident plugs in a device-resident encoder backend. Called by the gpu module's init() under -tags gpu — the core never imports the GPU package.

Types

type Config

type Config struct {
	Size      int        // target square side (Gemma 3 SigLIP: 896)
	Mean      [3]float32 // per-channel normalization mean (applied after /255)
	Std       [3]float32 // per-channel normalization std
	MaxPixels int        // reject a decoded image with more than this many pixels (W*H)
}

Config parameterizes preprocessing per model family.

func Gemma3

func Gemma3() Config

Gemma3 is the SigLIP preprocessing Gemma 3 / 4 use: 896×896, mean=std=0.5 on all channels (so the output lands in [-1, 1]).

type Encoder

type Encoder struct {
	Cfg EncoderConfig
	// contains filtered or unexported fields
}

Encoder is a loaded SigLIP vision tower.

func LoadEncoder

func LoadEncoder(dir string, quant bool) (*Encoder, error)

LoadEncoder reads a SigLIP vision checkpoint (config.json + model.safetensors) and returns a ready Encoder. Weights are copied out, so the safetensors file is closed before return (no retained mmap).

func (*Encoder) Close

func (e *Encoder) Close()

Close releases a resident backend if one is attached (no-op otherwise).

func (*Encoder) EnableResident

func (e *Encoder) EnableResident() error

EnableResident attaches a device-resident encoder (the WebGPU tower). Requires int8 matmul weights (LoadEncoder quant=true) and a -tags gpu build; Forward then runs on the device. Returns an error (and leaves the CPU path intact) if no backend is registered or the upload fails.

func (*Encoder) Forward

func (e *Encoder) Forward(pixels []float32) ([]float32, error)

Forward runs the encoder on pixel_values [NumChannels*ImageSize*ImageSize] (a single image, CHW order — the preprocess output) and returns last_hidden_state [numPatches * HiddenSize], row-major over patches in (row, col) grid order.

Concurrent Forward calls are safe, but NOT concurrent with EnableResident or Close: those write e.resident without synchronization, so a Forward racing one may read a torn pointer. Enable/close the resident backend before sharing the Encoder across goroutines.

func (*Encoder) GPUWeights

func (e *Encoder) GPUWeights() (GPUWeights, error)

GPUWeights exports the tower for the GPU resident encoder. Requires int8 matmul weights (LoadEncoder quant=true) — errors otherwise.

func (*Encoder) GridPatches

func (e *Encoder) GridPatches(pixels []float32) ([]float32, error)

GridPatches runs the CPU im2col patch extraction (the pure-Go preprocess step that stays on the host) and returns patches [NumPatches * (C*P*P)] for the GPU patch-embed matmul.

type EncoderConfig

type EncoderConfig struct {
	HiddenSize        int     `json:"hidden_size"`
	IntermediateSize  int     `json:"intermediate_size"`
	NumHiddenLayers   int     `json:"num_hidden_layers"`
	NumAttentionHeads int     `json:"num_attention_heads"`
	NumChannels       int     `json:"num_channels"`
	ImageSize         int     `json:"image_size"`
	PatchSize         int     `json:"patch_size"`
	LayerNormEps      float64 `json:"layer_norm_eps"`
}

EncoderConfig mirrors the SiglipVisionConfig fields the forward needs.

type GPULayer

type GPULayer struct {
	LN1w, LN1b     []float32
	Qw, Kw, Vw, Ow GPUMat
	Qb, Kb, Vb, Ob []float32
	LN2w, LN2b     []float32
	FC1w, FC2w     GPUMat
	FC1b, FC2b     []float32
}

GPULayer is one SigLIP transformer layer's weights for the GPU encoder.

type GPUMat

type GPUMat struct {
	Q          []int8
	Scales     []float32
	Rows, Cols int
}

GPUMat is one matmul weight in W8A8 form: int8 rows [Rows,Cols] + per-row scales.

type GPUWeights

type GPUWeights struct {
	Hidden, Inter, NumLayers, NumHeads, HeadDim int
	NumPatches, Grid, PatchSize, NumChannels    int
	Eps                                         float32
	PatchW, PatchB, PosEmb                      []float32 // patch-embed conv (f32) + positional
	PostLNw, PostLNb                            []float32
	Layers                                      []GPULayer
}

GPUWeights is the whole tower, flattened for the GPU encoder.

type PixelValues

type PixelValues struct {
	Data []float32 // channel-major: Data[c*Size*Size + y*Size + x]
	Size int
}

PixelValues is normalized image data in CHW order ([3*Size*Size]) plus its spatial size, ready for a vision encoder's patch-embed conv.

func Preprocess

func Preprocess(data []byte, cfg Config) (*PixelValues, error)

Preprocess decodes image bytes and produces normalized pixel_values. It reads the header first (image.DecodeConfig) and rejects an oversized image before decoding — a decompression bomb (tiny file, huge declared dimensions) errors here, it never allocates the full bitmap.

type QwenEncoderConfig

type QwenEncoderConfig struct {
	Depth               int    `json:"depth"`
	HiddenSize          int    `json:"hidden_size"`
	IntermediateSize    int    `json:"intermediate_size"`
	NumHeads            int    `json:"num_heads"`
	InChans             int    `json:"in_chans"`
	PatchSize           int    `json:"patch_size"`
	SpatialMergeSize    int    `json:"spatial_merge_size"`
	TemporalPatchSize   int    `json:"temporal_patch_size"`
	OutHiddenSize       int    `json:"out_hidden_size"`
	WindowSize          int    `json:"window_size"`
	FullattBlockIndexes []int  `json:"fullatt_block_indexes"`
	HiddenAct           string `json:"hidden_act"`
}

QwenEncoderConfig mirrors the HF Qwen2_5_VLVisionConfig fields the forward needs.

type QwenVisionEncoder

type QwenVisionEncoder struct {
	Cfg QwenEncoderConfig
	// contains filtered or unexported fields
}

QwenVisionEncoder is a loaded Qwen2.5-VL vision tower (dynamic resolution).

func LoadQwenVisionEncoder

func LoadQwenVisionEncoder(dir string, quant bool) (*QwenVisionEncoder, error)

LoadQwenVisionEncoder reads a Qwen2.5-VL checkpoint (config.json vision_config + safetensors) and returns a ready encoder. Weights are copied out, so the safetensors file is closed before return (no retained mmap). quant wraps the projections as int8 W8A8 (the patch-embed matmul stays f32); the fp32 parity gate runs quant=false.

func (*QwenVisionEncoder) Forward

func (e *QwenVisionEncoder) Forward(pixelValues []float32, gridTHW [][3]int) ([]float32, error)

Forward runs the ViT + merger on pre-patchified pixel_values [n_patches, patch_dim] (patch_dim = in_chans*temporal*patch*patch) with per-image grids (t,h,w in patch units, h/w multiples of spatial_merge_size). It returns the merged image embeddings [n_merged, out_hidden_size] in ORIGINAL patch order — the embeddings that replace the decoder's <image> placeholders. n_merged = Σ t*h*w / merge².

func (*QwenVisionEncoder) ForwardViT

func (e *QwenVisionEncoder) ForwardViT(pixelValues []float32, gridTHW [][3]int) ([]float32, error)

ForwardViT exposes the pre-merge hidden state for stage-isolated parity tests.

type ResidentEncoder

type ResidentEncoder interface {
	ForwardPatches(patches []float32) ([]float32, error)
	Close()
}

ResidentEncoder is a device-resident SigLIP forward — the GPU path in goinfer/gpu (vision_encoder.go), which uploads the tower once and runs every op on-device. vision stays pure-Go: it delegates Forward here when a resident backend is attached (EnableResident), and the gpu module plugs the factory in from its init() under -tags gpu. ForwardPatches consumes GridPatches output.