functional

Documentation

Overview

Package functional provides stateless, tensor-in tensor-out wrappers for common neural-network operations. Unlike the graph-aware layer types in sibling packages (e.g. layers/normalization), the functions here carry no parameter state, do no graph registration, and have no backward pass — they are pure forward-only computations suitable for scripting, testing, and composition.

Index

• func GELU[T tensor.Float](ctx context.Context, engine compute.Engine[T], ops numeric.Arithmetic[T], ...) (*tensor.TensorNumeric[T], error)
• func GELUBackward[T tensor.Float](ctx context.Context, engine compute.Engine[T], ops numeric.Arithmetic[T], ...) (*tensor.TensorNumeric[T], error)
• func LayerNorm[T tensor.Numeric](ctx context.Context, engine compute.Engine[T], ...) (*tensor.TensorNumeric[T], error)
• func LayerNormBackward[T tensor.Numeric](ctx context.Context, engine compute.Engine[T], ...) (dInput, dScale, dBias *tensor.TensorNumeric[T], err error)
• func Linear[T tensor.Numeric](ctx context.Context, engine compute.Engine[T], ...) (*tensor.TensorNumeric[T], error)
• func LinearBackward[T tensor.Numeric](ctx context.Context, engine compute.Engine[T], ...) (dInput, dWeight, dBias *tensor.TensorNumeric[T], err error)
• func MLPBackward[T tensor.Float](ctx context.Context, engine compute.Engine[T], ops numeric.Arithmetic[T], ...) (dInput, dWeight1, dBias1, dWeight2, dBias2 *tensor.TensorNumeric[T], err error)
• func MultiHeadAttention[T tensor.Numeric](ctx context.Context, engine compute.Engine[T], ...) (*tensor.TensorNumeric[T], error)
• func MultiHeadAttentionBackward[T tensor.Float](ctx context.Context, engine compute.Engine[T], ops numeric.Arithmetic[T], ...) (dQ, dK, dV *tensor.TensorNumeric[T], err error)
• func RMSNorm[T tensor.Numeric](ctx context.Context, engine compute.Engine[T], ...) (*tensor.TensorNumeric[T], error)
• func ReLU[T tensor.Numeric](ctx context.Context, engine compute.Engine[T], ops numeric.Arithmetic[T], ...) (*tensor.TensorNumeric[T], error)
• func SiLU[T tensor.Float](ctx context.Context, engine compute.Engine[T], ops numeric.Arithmetic[T], ...) (*tensor.TensorNumeric[T], error)
• func Sigmoid[T tensor.Numeric](ctx context.Context, engine compute.Engine[T], ops numeric.Arithmetic[T], ...) (*tensor.TensorNumeric[T], error)
• func Softmax[T tensor.Numeric](ctx context.Context, engine compute.Engine[T], x *tensor.TensorNumeric[T], ...) (*tensor.TensorNumeric[T], error)
• func SoftmaxBackward[T tensor.Float](ctx context.Context, engine compute.Engine[T], ops numeric.Arithmetic[T], ...) (*tensor.TensorNumeric[T], error)

Functions

func GELU

func GELU[T tensor.Float](ctx context.Context, engine compute.Engine[T], ops numeric.Arithmetic[T],
	x *tensor.TensorNumeric[T]) (*tensor.TensorNumeric[T], error)

GELU applies the Gaussian Error Linear Unit activation using the tanh approximation: 0.5 * x * (1 + tanh(sqrt(2/pi) * (x + 0.044715*x^3))). Thin wrapper that delegates to the canonical activations.Gelu Node so there is a single source of truth for the math (T124.2.2).

func GELUBackward

func GELUBackward[T tensor.Float](ctx context.Context, engine compute.Engine[T], ops numeric.Arithmetic[T],
	dOutput, input *tensor.TensorNumeric[T]) (*tensor.TensorNumeric[T], error)

GELUBackward computes the gradient of the GELU activation. dOutput: gradient from upstream input: original input to GELU Returns: dInput (same shape as input)

Thin wrapper that delegates to the canonical activations.Gelu Node's Backward (T124.2.2). Since the ADR 006 (T2.3) migration the node's Backward recomputes everything from the live input it receives, so no forward-seeding of a cache is needed (or possible).

func LayerNorm

func LayerNorm[T tensor.Numeric](ctx context.Context, engine compute.Engine[T],
	x, scale, bias *tensor.TensorNumeric[T], eps T) (*tensor.TensorNumeric[T], error)

LayerNorm applies layer normalization to x using the provided scale (gamma) and bias (beta) tensors. Normalization is performed over the last dimension.

output = (x - mean) / sqrt(variance + eps) * scale + bias

func LayerNormBackward

func LayerNormBackward[T tensor.Numeric](ctx context.Context, engine compute.Engine[T],
	dOutput, input, scale *tensor.TensorNumeric[T], eps T) (dInput, dScale, dBias *tensor.TensorNumeric[T], err error)

LayerNormBackward computes gradients for layer normalization. dOutput: gradient from upstream [*, features] input: original input [*, features] scale: gamma [features] eps: epsilon used in forward Returns: dInput [*, features], dScale [features], dBias [features]

func Linear

func Linear[T tensor.Numeric](ctx context.Context, engine compute.Engine[T],
	x, weight *tensor.TensorNumeric[T], bias *tensor.TensorNumeric[T]) (*tensor.TensorNumeric[T], error)

Linear computes y = x @ weight^T + bias. If bias is nil, computes y = x @ weight^T. x: [*, in_features], weight: [out_features, in_features], bias: [out_features]

func LinearBackward

func LinearBackward[T tensor.Numeric](ctx context.Context, engine compute.Engine[T],
	dOutput, input, weight *tensor.TensorNumeric[T]) (dInput, dWeight, dBias *tensor.TensorNumeric[T], err error)

LinearBackward computes gradients for y = x @ weight^T + bias. dOutput: gradient from upstream [batch, out_features] input: original input [batch, in_features] weight: weight matrix [out_features, in_features] Returns: dInput [batch, in_features], dWeight [out_features, in_features], dBias [out_features]

func MLPBackward

func MLPBackward[T tensor.Float](ctx context.Context, engine compute.Engine[T], ops numeric.Arithmetic[T],
	dOutput, input, weight1, bias1, weight2, bias2, hidden, activated *tensor.TensorNumeric[T],
	activation string) (dInput, dWeight1, dBias1, dWeight2, dBias2 *tensor.TensorNumeric[T], err error)

MLPBackward computes gradients for a 2-layer MLP: y = Linear2(activation(Linear1(x))). dOutput: gradient from upstream [batch, out_features] input: original input [batch, in_features] weight1: [hidden, in_features], bias1: [hidden] weight2: [out_features, hidden], bias2: [out_features] hidden: output of Linear1 (pre-activation) [batch, hidden] activated: output after activation [batch, hidden] activation: "relu" or "gelu" Returns: dInput, dWeight1, dBias1, dWeight2, dBias2

func MultiHeadAttention

func MultiHeadAttention[T tensor.Numeric](ctx context.Context, engine compute.Engine[T],
	q, k, v *tensor.TensorNumeric[T], nHeads int) (*tensor.TensorNumeric[T], error)

MultiHeadAttention computes multi-head scaled dot-product attention. q, k, v: [seq_len, d_model]. nHeads: number of attention heads. Returns: [seq_len, d_model]

func MultiHeadAttentionBackward

func MultiHeadAttentionBackward[T tensor.Float](ctx context.Context, engine compute.Engine[T], ops numeric.Arithmetic[T],
	dOutput, q, k, v *tensor.TensorNumeric[T], nHeads int) (dQ, dK, dV *tensor.TensorNumeric[T], err error)

MultiHeadAttentionBackward computes gradients for multi-head scaled dot-product attention. dOutput: gradient from upstream [seq_len, d_model] q, k, v: original inputs [seq_len, d_model] nHeads: number of attention heads Returns: dQ, dK, dV [seq_len, d_model]

func RMSNorm

func RMSNorm[T tensor.Numeric](ctx context.Context, engine compute.Engine[T],
	x, scale *tensor.TensorNumeric[T], eps T) (*tensor.TensorNumeric[T], error)

RMSNorm applies root-mean-square normalization to x using the provided scale (gain) tensor. Normalization is performed over the last dimension.

output = x * rsqrt(mean(x^2) + eps) * scale

func ReLU

func ReLU[T tensor.Numeric](ctx context.Context, engine compute.Engine[T], ops numeric.Arithmetic[T],
	x *tensor.TensorNumeric[T]) (*tensor.TensorNumeric[T], error)

ReLU applies the Rectified Linear Unit activation: max(0, x).

func SiLU

func SiLU[T tensor.Float](ctx context.Context, engine compute.Engine[T], ops numeric.Arithmetic[T],
	x *tensor.TensorNumeric[T]) (*tensor.TensorNumeric[T], error)

SiLU applies the Sigmoid Linear Unit (SiLU / Swish) activation: x * sigmoid(x). All arithmetic is routed through Engine[T] primitives.

func Sigmoid

func Sigmoid[T tensor.Numeric](ctx context.Context, engine compute.Engine[T], ops numeric.Arithmetic[T],
	x *tensor.TensorNumeric[T]) (*tensor.TensorNumeric[T], error)

Sigmoid applies the sigmoid activation: exp(x) / (1 + exp(x)). Thin wrapper that delegates to the canonical activations.Sigmoid Node (T124.2.2).

func Softmax

func Softmax[T tensor.Numeric](ctx context.Context, engine compute.Engine[T],
	x *tensor.TensorNumeric[T], axis int) (*tensor.TensorNumeric[T], error)

Softmax applies the softmax function along the given axis.

func SoftmaxBackward

func SoftmaxBackward[T tensor.Float](ctx context.Context, engine compute.Engine[T], ops numeric.Arithmetic[T],
	dOutput, softmaxOutput *tensor.TensorNumeric[T]) (*tensor.TensorNumeric[T], error)

SoftmaxBackward computes the gradient of the softmax function. dOutput: gradient from upstream [*, features] softmaxOutput: output of softmax forward pass [*, features] (already computed s_i values) Returns: dInput [*, features]

For each row: dInput_i = s_i * (dOutput_i - sum_j(dOutput_j * s_j))