initializers

Documentation

Overview

Package initializers include several weight initializers, to be used with a context.Context.

They construct computation.VariableInitializer closures.

Index

• Variables
• func One(graph *Graph, shape shapes.Shape) *Node
• func Zero(graph *Graph, shape shapes.Shape) *Node
• type VariableInitializer
  • func BroadcastTensorToShape(baseValue *tensors.Tensor) VariableInitializer
  • func GlorotUniformFn(ctx *context.Context) VariableInitializer
  • func HeFn(ctx *context.Context) VariableInitializer
  • func RandomNormalFn(ctx *context.Context, stddev float64) VariableInitializer
  • func RandomUniformFn(ctx *context.Context, min, max float64) VariableInitializer
  • func XavierNormalFn(ctx *context.Context) VariableInitializer
  • func XavierUniformFn(ctx *context.Context) VariableInitializer

Variables

var (
	// ParamInitialSeed is the key for the hyperparameter to use for initial seed (int64). The default is 0,
	// which makes it non-deterministic. Set it to a value different from 0 for a deterministic initialization
	// (as long as the model doesn't change).
	//
	// It is an alias to context.ParamInitialSeed.
	ParamInitialSeed = context.ParamInitialSeed
)

Functions

func One

func One(graph *Graph, shape shapes.Shape) *Node

One initializes variables with one.

func Zero

func Zero(graph *Graph, shape shapes.Shape) *Node

Zero initializes variables with zero.

Types

type VariableInitializer

type VariableInitializer = context.VariableInitializer

VariableInitializer builds a node that returns a value to initialize a variable of the given shape. It is defined in the Context.

func BroadcastTensorToShape

func BroadcastTensorToShape(baseValue *tensors.Tensor) VariableInitializer

BroadcastTensorToShape is an initializer that takes a constant tensor as baseValue, and during initialization it broadcast it to the requested variable shape.

The broadcasting happens only on the prefix dimensions (using graph.BroadcastPrefix), so the baseValue tensor's shape must match the last dimensions of the variable's shape.

The baseValue can have a different dtype, in which case it is converted (using graph.ConvertDType) to the requested variable dtype.

It also works with a scalar baseValue, which translates to constant value initializer.

func GlorotUniformFn

func GlorotUniformFn(ctx *context.Context) VariableInitializer

GlorotUniformFn return a Glorot uniform initializer, also called Xavier uniform initializer.

It can be set to a context with `ctx.WithInitializer(GlorotUniformFn(ctx))`, where `initialSeed` can be 0 for a random seed to be generated.

For float and complex values, it draws samples from a uniform distribution within `[-limit, limit]`, where `limit = sqrt(3 / ((fan_in + fan_out)/2))` (`fan_in` is the number of input units in the weight tensor and fan_out is the number of output units).

Since it doesn't have semantic information about the variables being created, it makes some assumptions about the shapes of the variables: it assumes either these are weights for biases, matrix multiplications or 2D or 3D convolutions. Using it for different types of shapes may not get the expected result.

It uses the context's ParamInitialSeed hyperparameter to initialize the random number generator -- only the first time it is used for a graph. If it is set to 0 (NoSeed, the default), a random seed is instead generated (from the nanosecond clock).

It initializes biases (anything with rank <= 1) to zeros.

Non-float and non-complex variables are initialized with zero instead.

func HeFn

func HeFn(ctx *context.Context) VariableInitializer

HeFn returns the initializer that tries to preserve the variance of 1, calculated for the Relu activation functions.

It initializes biases (anything with rank <= 1) to zeros.

It uses the context's ParamInitialSeed hyperparameter to initialize the random number generator -- only the first time it is used for a graph. If it is set to 0 (NoSeed, the default), a random seed is instead generated (from the nanosecond clock).

[1] https://medium.com/@tylernisonoff/weight-initialization-for-cnns-a-deep-dive-into-he-initialization-50b03f37f53d [2] https://arxiv.org/pdf/1502.01852

func RandomNormalFn

func RandomNormalFn(ctx *context.Context, stddev float64) VariableInitializer

RandomNormalFn returns an initializer that generates random normal values with the given standard deviation and mean set to 0.

It uses the context's ParamInitialSeed hyperparameter to initialize the random number generator -- only the first time it is used for a graph. If it is set to 0 (NoSeed, the default), a random seed is instead generated (from the nanosecond clock).

Non-float and non-complex variables are initialized with zero instead.

func RandomUniformFn

func RandomUniformFn(ctx *context.Context, min, max float64) VariableInitializer

RandomUniformFn return an initializer that generates random uniform values from [min, max).

It uses the context's ParamInitialSeed hyperparameter to initialize the random number generator -- only the first time it is used for a graph. If it is set to 0 (NoSeed, the default), a random seed is instead generated (from the nanosecond clock).

Non-float and non-complex variables are initialized with zero instead.

func XavierNormalFn

func XavierNormalFn(ctx *context.Context) VariableInitializer

XavierNormalFn returns an initializer that generates random values with a normal distribution with mean in 0 and stddev of sqrt(2 / (fanIn+fanOut)). See description in https://paperswithcode.com/method/xavier-initialization

It uses the context's ParamInitialSeed hyperparameter to initialize the random number generator -- only the first time it is used for a graph. If it is set to 0 (NoSeed, the default), a random seed is instead generated (from the nanosecond clock).

It initializes biases (anything with rank <= 1) to zeros.

Non-float and non-complex variables are initialized with zero instead.

func XavierUniformFn

func XavierUniformFn(ctx *context.Context) VariableInitializer

XavierUniformFn returns an initializer that generates random values with a uniform distribution with a range defined by +/- sqrt(6 / (fanIn+fanOut)). See description in https://paperswithcode.com/method/xavier-initialization

It uses the context's ParamInitialSeed hyperparameter to initialize the random number generator -- only the first time it is used for a graph. If it is set to 0 (NoSeed, the default), a random seed is instead generated (from the nanosecond clock).

It initializes biases (anything with rank <= 1) to zeros.

Non-float and non-complex variables are initialized with zero instead.