vision

Documentation

Overview

Package vision provides vision-related neural network layers.

Package vision provides vision-related neural network layers.

Stability: beta

Index

• type CLIPEncoder
  • func NewCLIPEncoder[T tensor.Numeric](name string, engine compute.Engine[T], ops numeric.Arithmetic[T], ...) (*CLIPEncoder[T], error)
  • func (e *CLIPEncoder[T]) Attributes() map[string]interface{}
  • func (e *CLIPEncoder[T]) Backward(_ context.Context, _ types.BackwardMode, _ *tensor.TensorNumeric[T], ...) ([]*tensor.TensorNumeric[T], error)
  • func (e *CLIPEncoder[T]) Forward(ctx context.Context, inputs ...*tensor.TensorNumeric[T]) (*tensor.TensorNumeric[T], error)
  • func (e *CLIPEncoder[T]) OpType() string
  • func (e *CLIPEncoder[T]) OutputShape() []int
  • func (e *CLIPEncoder[T]) Parameters() []*graph.Parameter[T]
• type CLIPEncoderConfig
  • func (c CLIPEncoderConfig) NumPatches() int
• type LTXVAEDecoderConfig
• type LTXVAEDecoderSkeleton
  • func NewLTXVAEDecoderSkeleton[T tensor.Float](engine compute.Engine[T], ops numeric.Arithmetic[T], cfg LTXVAEDecoderConfig) (*LTXVAEDecoderSkeleton[T], error)
  • func (d *LTXVAEDecoderSkeleton[T]) Forward(ctx context.Context, latent *tensor.TensorNumeric[T]) (*tensor.TensorNumeric[T], error)
  • func (d *LTXVAEDecoderSkeleton[T]) OutputShape() []int

Types

type CLIPEncoder

type CLIPEncoder[T tensor.Numeric] struct {
	// contains filtered or unexported fields
}

CLIPEncoder implements a CLIP ViT (Vision Transformer) encoder.

Architecture:

PatchEmbed -> [CLS] + PatchEmbeddings + PosEmbed -> [LN -> SelfAttn -> Add -> LN -> FFN(QuickGELU) -> Add] x N -> LN

Input shape: [batch, channels, height, width] (pixel values normalized to [-1, 1]) Output shape: [batch, numPatches+1, hiddenDim]

func NewCLIPEncoder

func NewCLIPEncoder[T tensor.Numeric](
	name string,
	engine compute.Engine[T],
	ops numeric.Arithmetic[T],
	cfg CLIPEncoderConfig,
) (*CLIPEncoder[T], error)

NewCLIPEncoder creates a new CLIP ViT encoder.

func (*CLIPEncoder[T]) Attributes

func (e *CLIPEncoder[T]) Attributes() map[string]interface{}

func (*CLIPEncoder[T]) Backward

func (e *CLIPEncoder[T]) Backward(_ context.Context, _ types.BackwardMode, _ *tensor.TensorNumeric[T], _ ...*tensor.TensorNumeric[T]) ([]*tensor.TensorNumeric[T], error)

func (*CLIPEncoder[T]) Forward

func (e *CLIPEncoder[T]) Forward(ctx context.Context, inputs ...*tensor.TensorNumeric[T]) (*tensor.TensorNumeric[T], error)

Forward runs the CLIP vision encoder. Input: [batch, channels, height, width] pixel values. Output: [batch, numPatches+1, hiddenDim] vision embeddings.

func (*CLIPEncoder[T]) OpType

func (e *CLIPEncoder[T]) OpType() string

func (*CLIPEncoder[T]) OutputShape

func (e *CLIPEncoder[T]) OutputShape() []int

func (*CLIPEncoder[T]) Parameters

func (e *CLIPEncoder[T]) Parameters() []*graph.Parameter[T]

Parameters returns all trainable parameters from the CLIP encoder.

type CLIPEncoderConfig

type CLIPEncoderConfig struct {
	ImageSize   int // Input image size (square, e.g. 224).
	PatchSize   int // Patch size for patch embedding (e.g. 14).
	HiddenDim   int // Hidden dimension throughout the encoder.
	NumHeads    int // Number of attention heads per transformer block.
	NumLayers   int // Number of transformer encoder blocks.
	NumChannels int // Number of input channels (default 3 for RGB).
}

CLIPEncoderConfig holds configuration for a CLIP vision encoder.

func (CLIPEncoderConfig) NumPatches

func (c CLIPEncoderConfig) NumPatches() int

NumPatches returns the number of patches (excluding class token).

type LTXVAEDecoderConfig

type LTXVAEDecoderConfig struct {
	LatentChannels int // input channels (e.g. 128 for LTX-2)
	MidChannels    int // working channels (must be divisible by NumGroups)
	OutChannels    int // output channels (3 for RGB)
	NumGroups      int
	Epsilon        float64
}

LTXVAEDecoderConfig parameterizes the skeleton.

type LTXVAEDecoderSkeleton

type LTXVAEDecoderSkeleton[T tensor.Float] struct {
	// contains filtered or unexported fields
}

LTXVAEDecoderSkeleton is a SKELETON of the LTX-2 video VAE decoder (E127 T127.4.4). It wires the E127 convolutional primitives -- Conv3d, ConvTranspose3d (zerfoo#896) and GroupNormalization -- together with GELU into a representative residual + 2x-upsample decode forward, to prove the primitive set composes end-to-end and produces correctly shaped output.

It is intentionally NOT the weight-accurate LTX-2 VAE: the real decoder's block_out_channels = [256,512,1024,2048], 32x spatial / 8x temporal upsample, stacked ResNet3D blocks, mid-block 3D attention, and the actual converted weights are the remainder of T127.4.4. Treat this as scaffolding/integration surface, not a correctness reference. The weights here are small deterministic fixtures so the forward runs and is finite.

Pipeline (latent [N, Clatent, D, H, W] -> [N, OutChannels, 2D, 2H, 2W]):

in-conv (1x1x1)
-> residual{ GroupNorm -> GELU -> Conv3d(3x3x3, same) } (+ skip)
-> ConvTranspose3d(2x2x2, stride 2)  [2x upsample]
-> GroupNorm -> GELU -> out-conv(3x3x3, same -> OutChannels)

func NewLTXVAEDecoderSkeleton

func NewLTXVAEDecoderSkeleton[T tensor.Float](
	engine compute.Engine[T],
	ops numeric.Arithmetic[T],
	cfg LTXVAEDecoderConfig,
) (*LTXVAEDecoderSkeleton[T], error)

NewLTXVAEDecoderSkeleton builds the skeleton with deterministic fixture weights.

func (*LTXVAEDecoderSkeleton[T]) Forward

func (d *LTXVAEDecoderSkeleton[T]) Forward(ctx context.Context, latent *tensor.TensorNumeric[T]) (*tensor.TensorNumeric[T], error)

Forward decodes a latent [N, LatentChannels, D, H, W] to [N, OutChannels, 2D, 2H, 2W].

func (*LTXVAEDecoderSkeleton[T]) OutputShape

func (d *LTXVAEDecoderSkeleton[T]) OutputShape() []int

OutputShape returns the last forward output shape.