initializers

Documentation

Overview

Package initializers include several weight initializers, to be used with context. They implement computation.VariableInitializer type.

Index

• Constants
• func Finalize()
• func One(graph *Graph, shape shapes.Shape) *Node
• func Zero(graph *Graph, shape shapes.Shape) *Node
• type VariableInitializer
  • func GlorotUniformFn(initialSeed int64) VariableInitializer
  • func HeFn(initialSeed int64) VariableInitializer
  • func RandomNormalFn(initialSeed int64, stddev float64) VariableInitializer
  • func RandomUniformFn(initialSeed int64, min, max float64) VariableInitializer
  • func XavierNormalFn(initialSeed int64) VariableInitializer
  • func XavierUniformFn(initialSeed int64) VariableInitializer

Constants

const NoSeed = int64(0)

Functions

func Finalize

func Finalize()

Finalize will clear the global state kept alive and free up the memory. Namely, the random number generator states.

Used for testing and debugging.

func One

func One(graph *Graph, shape shapes.Shape) *Node

One initializes variables with one.

func Zero

func Zero(graph *Graph, shape shapes.Shape) *Node

Zero initializes variables with zero.

Types

type VariableInitializer

type VariableInitializer func(graph *Graph, shape shapes.Shape) *Node

VariableInitializer builds a node that returns a value to initialize a variable of the given shape. It is defined in the Context.

func GlorotUniformFn

func GlorotUniformFn(initialSeed int64) VariableInitializer

GlorotUniformFn return a Glorot uniform initializer, also called Xavier uniform initializer.

It can be set to a context with `ctx.WithInitializer(GlorotUniformFn(initialSeed))`, where `initialSeed` can be 0 for a random seed to be generated.

For float and complex values, it draws samples from a uniform distribution within `[-limit, limit]`, where `limit = sqrt(3 / ((fan_in + fan_out)/2))` (`fan_in` is the number of input units in the weight tensor and fan_out is the number of output units).

Since it doesn't have semantic information about the variables being created, it makes some assumptions about the shapes of the variables: it assumes either these are weights for biases, matrix multiplications or 2D or 3D convolutions. Using it for different types of shapes may not get the expected result.

The parameter `initialSeed` is used to initialize the random number generator -- only the first time it is used for a graph. If it is set to 0 (NoSeed), a random seed is instead generated (from the nanosecond clock).

It initializes biases (anything with rank <= 1) to zeros.

Non-float and non-complex variables are initialized with zero instead.

func HeFn

func HeFn(initialSeed int64) VariableInitializer

HeFn returns the initializer that tries to preserve the variance of 1, calculated for the Relu activation functions.

It initializes biases (anything with rank <= 1) to zeros.

[1] https://medium.com/@tylernisonoff/weight-initialization-for-cnns-a-deep-dive-into-he-initialization-50b03f37f53d [2] https://arxiv.org/pdf/1502.01852

func RandomNormalFn

func RandomNormalFn(initialSeed int64, stddev float64) VariableInitializer

RandomNormalFn returns an initializer that generates random normal values with the given standard deviation and mean set to 0.

The parameter `initialSeed` is used to initialize the random number generator -- only the first time it is used for a graph, later it continues to pool from the same rng state shared by all initializers. If it is set to 0 (NoSeed), a random seed is instead generated (from the nanosecond clock).

Non-float and non-complex variables are initialized with zero instead.

func RandomUniformFn

func RandomUniformFn(initialSeed int64, min, max float64) VariableInitializer

RandomUniformFn return an initializer that generates a random uniform values from [min, max).

The parameter `initialSeed` is used to initialize the random number generator -- only the first time it is used for a graph, later it continues to pool from the same rng state shared by all initializers. If it is set to 0 (NoSeed), a random seed is instead generated (from the nanosecond clock).

Non-float and non-complex variables are initialized with zero instead.

func XavierNormalFn

func XavierNormalFn(initialSeed int64) VariableInitializer

XavierNormalFn returns an initializer that generates random values with a normal distribution with mean in 0 and stddev of sqrt(2 / (fanIn+fanOut)). See description in https://paperswithcode.com/method/xavier-initialization

The parameter `initialSeed` is used to initialize the random number generator -- only the first time it is used for a graph, later it continues to pool from the same rng state shared by all initializers. If it is set to 0 (NoSeed), a random seed is instead generated (from the nanosecond clock).

It initializes biases (anything with rank <= 1) to zeros.

Non-float and non-complex variables are initialized with zero instead.

func XavierUniformFn

func XavierUniformFn(initialSeed int64) VariableInitializer

XavierUniformFn returns an initializer that generates random values with an uniform distribution with a range defined by +/- sqrt(6 / (fanIn+fanOut)). See description in https://paperswithcode.com/method/xavier-initialization

The parameter `initialSeed` is used to initialize the random number generator -- only the first time it is used for a graph, later it continues to pool from the same rng state shared by all initializers. If it is set to 0 (NoSeed), a random seed is instead generated (from the nanosecond clock).

It initializes biases (anything with rank <= 1) to zeros.

Non-float and non-complex variables are initialized with zero instead.