initializers

Documentation

Overview

Package initializers include several weight initializers, to be used with context. They implement computation.VariableInitializer type.

Index

• Constants
• Variables
• func Finalize()
• func One(graph *Graph, shape shapes.Shape) *Node
• func UseRngState(g *Graph, initialSeed int64, fn func(rngState *Node) (newRngState *Node))
• func Zero(graph *Graph, shape shapes.Shape) *Node
• type VariableInitializer
  • func BroadcastTensorToShape(baseValue *tensors.Tensor) VariableInitializer
  • func GlorotUniformFn(ctx *context.Context) VariableInitializer
  • func HeFn(ctx *context.Context) VariableInitializer
  • func RandomNormalFn(ctx *context.Context, stddev float64) VariableInitializer
  • func RandomUniformFn(ctx *context.Context, min, max float64) VariableInitializer
  • func XavierNormalFn(ctx *context.Context) VariableInitializer
  • func XavierUniformFn(ctx *context.Context) VariableInitializer

Constants

const NoSeed = int64(0)

NoSeed is the default seed value for ParamInitialSeed, and it means a seed is randomly generated -- which also means initialization is not deterministic.

Variables

var (
	// ParamInitialSeed is the key for the hyperparameter to use for initial seed (int64). The default is 0,
	// which makes it non-deterministic. Set it to a value different from 0 for a deterministic (as long
	// as the model doesn't change) initialization.
	//
	// If you set this hyperparameter, remember to configure a new default initializer in your context (context.Context.WithInitializer),
	// so that it is used.
	//
	// More details in initializers.UseRngState.
	ParamInitialSeed = "initializers_seed"
)

Functions

func Finalize

func Finalize()

Finalize will clear the global state kept alive and free up the memory. Namely, the random number generator states.

Used for testing and debugging.

func One

func One(graph *Graph, shape shapes.Shape) *Node

One initializes variables with one.

func UseRngState

func UseRngState(g *Graph, initialSeed int64, fn func(rngState *Node) (newRngState *Node))

UseRngState will provide a random-number generator state for the current graph, to be used for initialization.

If all initializers of a model uses random from this, in a deterministic order, then the initialization of the model will be deterministic and can be replicated exactly.

The initialSeed is only used the first time the function is called for a Graph. If the initialSeed is 0, a random seed is generated -- in which case initialization is not deterministic. the returned updated state.

See examples usage in RandomNormalFn and RandomUniformFn.

It locks the state, so only one call to it will be happening at a time.

func Zero

func Zero(graph *Graph, shape shapes.Shape) *Node

Zero initializes variables with zero.

Types

type VariableInitializer

type VariableInitializer = context.VariableInitializer

VariableInitializer builds a node that returns a value to initialize a variable of the given shape. It is defined in the Context.

func BroadcastTensorToShape

func BroadcastTensorToShape(baseValue *tensors.Tensor) VariableInitializer

BroadcastTensorToShape is an initializer that takes a constant tensor as baseValue and during initialization it broadcast it to the requested variable shape.

The broadcasting happens only on the prefix dimensions (using graph.BroadcastPrefix), so the shape of the baseValue tensor mush match the last dimensions of the variables shape.

The baseValue can have a different dtype, in which case it is converted (using graph.ConvertDType) to the requested variable dtype.

It also works with a scalar baseValue, which translates to constant value initializer.

func GlorotUniformFn

func GlorotUniformFn(ctx *context.Context) VariableInitializer

GlorotUniformFn return a Glorot uniform initializer, also called Xavier uniform initializer.

It can be set to a context with `ctx.WithInitializer(GlorotUniformFn(ctx))`, where `initialSeed` can be 0 for a random seed to be generated.

For float and complex values, it draws samples from a uniform distribution within `[-limit, limit]`, where `limit = sqrt(3 / ((fan_in + fan_out)/2))` (`fan_in` is the number of input units in the weight tensor and fan_out is the number of output units).

Since it doesn't have semantic information about the variables being created, it makes some assumptions about the shapes of the variables: it assumes either these are weights for biases, matrix multiplications or 2D or 3D convolutions. Using it for different types of shapes may not get the expected result.

It uses the context's ParamInitialSeed hyperparameter to initialize the random number generator -- only the first time it is used for a graph. If it is set to 0 (NoSeed, the default), a random seed is instead generated (from the nanosecond clock).

It initializes biases (anything with rank <= 1) to zeros.

Non-float and non-complex variables are initialized with zero instead.

func HeFn

func HeFn(ctx *context.Context) VariableInitializer

HeFn returns the initializer that tries to preserve the variance of 1, calculated for the Relu activation functions.

It initializes biases (anything with rank <= 1) to zeros.

It uses the context's ParamInitialSeed hyperparameter to initialize the random number generator -- only the first time it is used for a graph. If it is set to 0 (NoSeed, the default), a random seed is instead generated (from the nanosecond clock).

[1] https://medium.com/@tylernisonoff/weight-initialization-for-cnns-a-deep-dive-into-he-initialization-50b03f37f53d [2] https://arxiv.org/pdf/1502.01852

func RandomNormalFn

func RandomNormalFn(ctx *context.Context, stddev float64) VariableInitializer

RandomNormalFn returns an initializer that generates random normal values with the given standard deviation and mean set to 0.

It uses the context's ParamInitialSeed hyperparameter to initialize the random number generator -- only the first time it is used for a graph. If it is set to 0 (NoSeed, the default), a random seed is instead generated (from the nanosecond clock).

Non-float and non-complex variables are initialized with zero instead.

func RandomUniformFn

func RandomUniformFn(ctx *context.Context, min, max float64) VariableInitializer

RandomUniformFn return an initializer that generates a random uniform values from [min, max).

It uses the context's ParamInitialSeed hyperparameter to initialize the random number generator -- only the first time it is used for a graph. If it is set to 0 (NoSeed, the default), a random seed is instead generated (from the nanosecond clock).

Non-float and non-complex variables are initialized with zero instead.

func XavierNormalFn

func XavierNormalFn(ctx *context.Context) VariableInitializer

XavierNormalFn returns an initializer that generates random values with a normal distribution with mean in 0 and stddev of sqrt(2 / (fanIn+fanOut)). See description in https://paperswithcode.com/method/xavier-initialization

It uses the context's ParamInitialSeed hyperparameter to initialize the random number generator -- only the first time it is used for a graph. If it is set to 0 (NoSeed, the default), a random seed is instead generated (from the nanosecond clock).

It initializes biases (anything with rank <= 1) to zeros.

Non-float and non-complex variables are initialized with zero instead.

func XavierUniformFn

func XavierUniformFn(ctx *context.Context) VariableInitializer

XavierUniformFn returns an initializer that generates random values with an uniform distribution with a range defined by +/- sqrt(6 / (fanIn+fanOut)). See description in https://paperswithcode.com/method/xavier-initialization

It uses the context's ParamInitialSeed hyperparameter to initialize the random number generator -- only the first time it is used for a graph. If it is set to 0 (NoSeed, the default), a random seed is instead generated (from the nanosecond clock).

It initializes biases (anything with rank <= 1) to zeros.

Non-float and non-complex variables are initialized with zero instead.