deep

package module
v0.0.0-...-797df6e Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 6, 2022 License: MIT Imports: 4 Imported by: 0

README

This is an edited version of the go-deep library except it has been converted to 32-bit for better performance and some extra activation functions have been added (Elu, Mish and Swish, RootX, MulDiv and DoubleRoot)

Update: concurrency is now used more to increase performance and enabled multiple activation functions in the one network. (one activation function type per layer)

neural-net

Feed forward/backpropagation neural network implementation. Currently supports:

  • Activation functions: sigmoid, hyperbolic, ReLU, Elu, Mish, Swish, also activations I created (RootX, DivX, DoublePow, DoubleRoot and DoubleDiv).. RootX is particularly effective.
  • Double Root is looks like a combination of the sqrt function and tanh (double Div is similar except using division), they can be used to squash numbers inside your network to prevent the network from exploding... Boom!
  • I designed DoubleDiv, DoublePow & DoubleRoot to help the neural networks solve mathematical equations, usually used with the linear activation function
  • RootX (combining sqrt with relu) seems to solve problems facter than Mish and Swish... Still testing DivX (combining division with relu) but should produce similar results to RootX
  • Solvers: SGD, SGD with momentum/nesterov, Adam
  • Classification modes: regression, multi-class, multi-label, binary
  • Supports batch training in parallel
  • Bias nodes

Networks are modeled as a set of neurons connected through synapses. No GPU computations - don't use this for any large scale applications.

Install

go get -u github.com/nathanleary/neural-net

Usage

Import the go-deep package

import (
	"fmt"
	deep "github.com/nathanleary/neural-net"
	"github.com/nathanleary/neural-net/training"
)

Define some data...

var data = training.Examples{
	{[]float32{2.7810836, 2.550537003}, []float32{0}},
	{[]float32{1.465489372, 2.362125076}, []float32{0}},
	{[]float32{3.396561688, 4.400293529}, []float32{0}},
	{[]float32{1.38807019, 1.850220317}, []float32{0}},
	{[]float32{7.627531214, 2.759262235}, []float32{1}},
	{[]float32{5.332441248, 2.088626775}, []float32{1}},
	{[]float32{6.922596716, 1.77106367}, []float32{1}},
	{[]float32{8.675418651, -0.242068655}, []float32{1}},
}

Create a network with two hidden layers of size 2 and 2 respectively:

n := deep.NewNeural(&deep.Config{
	/* Input dimensionality */
	Inputs: 2,
	/* Three hidden layers consisting of two neurons each, and a single output */
	Layout: []int{2, 2, 2, 2, 1},
	/* Activation functions: Sigmoid, Tanh, ReLU, Linear, Elu, Mish, Swish, RootX, DoubleRoot */
	/*Defining the three hidden layer's Activation function*/
	Activation: []deep.ActivationType{
				deep.ActivationMulDiv,
				deep.ActivationRootX,
				deep.ActivationDoubleRoot,
				deep.ActivationMish,
			},
	/* Determines output layer activation & loss function: 
	ModeRegression: linear outputs with MSE loss
	ModeMultiClass: softmax output with Cross Entropy loss
	ModeMultiLabel: sigmoid output with Cross Entropy loss
	ModeBinary: sigmoid output with binary CE loss */
	Mode: deep.ModeBinary,
	/* Weight initializers: {deep.NewNormal(μ, σ), deep.NewUniform(μ, σ)} */
	Weight: deep.NewNormal(1.0, 0.0),
	/* Apply bias */
	Bias: true,
})

Train:

// params: learning rate, momentum, alpha decay, nesterov
optimizer := training.NewSGD(0.05, 0.1, 1e-6, true)
// params: optimizer, verbosity (print stats at every 50th iteration)
trainer := training.NewTrainer(optimizer, 50)

training, heldout := data.Split(0.5)
trainer.Train(n, training, heldout, 1000) // training, validation, iterations

resulting in:

Epochs        Elapsed       Error         
---           ---           ---           
5             12.938µs      0.36438       
10            125.691µs     0.02261       
15            177.194µs     0.00404       
...     
1000          10.703839ms   0.00000

Finally, make some predictions:

fmt.Println(data[0].Input, "=>", n.Predict(data[0].Input))
fmt.Println(data[5].Input, "=>", n.Predict(data[5].Input))

Alternatively, batch training can be performed in parallell:

optimizer := NewAdam(0.001, 0.9, 0.999, 1e-8)
// params: optimizer, verbosity (print info at every n:th iteration), batch-size, number of workers
trainer := training.NewBatchTrainer(optimizer, 1, 200, 4)

training, heldout := data.Split(0.75)
trainer.Train(n, training, heldout, 1000) // training, validation, iterations

Examples

See training/trainer_test.go for a variety of toy examples of regression, multi-class classification, binary classification, etc.

See examples/ for more realistic examples:

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ArgMax 

func ArgMax(xx []float32) int

ArgMax is the index of the largest element

func Dot 

func Dot(xx, yy []float32) float32

Dot product

func Logistic 

func Logistic(x, a float32) float32

Logistic is the logistic function

func Max 

func Max(xx []float32) float32

Max is the largest element

func Mean 

func Mean(xx []float32) float32

Mean of xx

func Min 

func Min(xx []float32) float32

Min is the smallest element

func Normal 

func Normal(stdDev, mean float32) float32

Normal samples a value from N(μ, σ)

func Normalize 

func Normalize(xx []float32)

Normalize scales to (0,1)

func Round 

func Round(x float32) float32

Round to nearest integer

func SetCustomDf 

func SetCustomDf(Df func(float32, float32) float32)

func SetCustomF 

func SetCustomF(F func(float32) float32)

func Sgn 

func Sgn(x float32) float32

Sgn is signum

func Softmax 

func Softmax(xx []float32) []float32

Softmax is the softmax function

func Sqrt 

func Sqrt(N float32) float32

func StandardDeviation 

func StandardDeviation(xx []float32) float32

StandardDeviation of xx

func Standardize 

func Standardize(xx []float32)

Standardize (z-score) shifts distribution to μ=0 σ=1

func Sum 

func Sum(xx []float32) (sum float32)

Sum is sum

func Uniform 

func Uniform(stdDev, mean float32) float32

Uniform samples a value from u(mean-stdDev/2,mean+stdDev/2)

func Variance 

func Variance(xx []float32) float32

Variance of xx

Types

type ActivationType 

type ActivationType int

ActivationType is represents a neuron activation function 

const (
	// ActivationNone is no activation
	ActivationNone ActivationType = 0
	// ActivationSigmoid is a sigmoid activation
	ActivationSigmoid ActivationType = 1
	// ActivationTanh is hyperbolic activation
	ActivationTanh ActivationType = 2
	// ActivationReLU is rectified linear unit activation
	ActivationReLU ActivationType = 3
	// ActivationLinear is linear activation
	ActivationLinear ActivationType = 4
	// ActivationSoftmax is a softmax activation (per layer)
	ActivationSoftmax ActivationType = 5
	// ActivationELU is a Elu activation
	ActivationELU ActivationType = 6
	// ActivationSwish is a Swish activation
	ActivationSwish ActivationType = 7
	// ActivationMish is a Mish activation
	ActivationMish ActivationType = 8
	// ActivationCustom is a Custom activation
	ActivationCustom ActivationType = 9
	// ActivationCustom is a Custom activation
	ActivationDoubleRoot ActivationType = 10
	// ActivationCustom is a Custom activation
	ActivationRootX ActivationType = 11
	// ActivationMulDiv is a Custom activation
	ActivationDivX ActivationType = 12
	// ActivationMulDiv is a Custom activation
	ActivationDoubleDiv ActivationType = 13
	// ActivationMulDiv is a Custom activation
	ActivationRootPow ActivationType = 14
	// ActivationMulDiv is a Custom activation
	ActivationDoublePow ActivationType = 15
	// ActivationMulDiv is a Custom activation
	ActivationRootSwish ActivationType = 16
)

func OutputActivation 

func OutputActivation(c Mode) ActivationType

OutputActivation returns activation corresponding to prediction mode

type BinaryCrossEntropy 

type BinaryCrossEntropy struct{}

BinaryCrossEntropy is binary CE loss

func (BinaryCrossEntropy) Df 

func (l BinaryCrossEntropy) Df(estimate, ideal, activation float32) float32

Df is CE'(...)

func (BinaryCrossEntropy) F 

func (l BinaryCrossEntropy) F(estimate, ideal [][]float32) float32

F is CE(...)

type Config 

type Config struct {

	// Number of inputs
	Inputs int
	// Defines topology:
	// For instance, [5 3 3] signifies a network with two hidden layers
	// containing 5 and 3 nodes respectively, followed an output layer
	// containing 3 nodes.
	Layout []int
	// Activation functions: {ActivationTanh, ActivationReLU, ActivationSigmoid}
	Activation []ActivationType
	// Solver modes: {ModeRegression, ModeBinary, ModeMultiClass, ModeMultiLabel}
	Mode Mode
	// Initializer for weights: {NewNormal(σ, μ), NewUniform(σ, μ)}
	Weight WeightInitializer `json:"-"`
	// Loss functions: {LossCrossEntropy, LossBinaryCrossEntropy, LossMeanSquared}
	Loss LossType
	// Apply bias nodes
	Bias bool
}

Config defines the network topology, activations, losses etc

type CrossEntropy 

type CrossEntropy struct{}

CrossEntropy is CE loss

func (CrossEntropy) Df 

func (l CrossEntropy) Df(estimate, ideal, activation float32) float32

Df is CE'(...)

func (CrossEntropy) F 

func (l CrossEntropy) F(estimate, ideal [][]float32) float32

F is CE(...)

type Custom 

type Custom struct {
	Mem map[float32]float32
}

func (Custom) Df 

func (a Custom) Df(y float32) float32

Df is Custom'(y), where y = Custom(x)

func (Custom) F 

func (a Custom) F(x float32, training bool) float32

F is Custom(x)

type Differentiable 

type Differentiable interface {
	F(float32, bool) float32
	Df(float32) float32
}

Differentiable is an activation function and its first order derivative, where the latter is expressed as a function of the former for efficiency

func GetActivation 

func GetActivation(act ActivationType) Differentiable

GetActivation returns the concrete activation given an ActivationType

type DivX 

type DivX struct {
	Mem map[float32]float32
}

MulDiv is a logistic activator in the special case of a = 1

func (DivX) Df 

func (a DivX) Df(x float32) float32

Df is MulDiv'(y), where y = MulDiv(x)

func (DivX) F 

func (a DivX) F(x float32, training bool) float32

F is MulDiv(x)

type DoubleDiv 

type DoubleDiv struct {
	Mem map[float32]float32
}

MulDiv is a logistic activator in the special case of a = 1

func (DoubleDiv) Df 

func (a DoubleDiv) Df(x float32) float32

Df is MulDiv'(y), where y = MulDiv(x)

func (DoubleDiv) F 

func (a DoubleDiv) F(x float32, training bool) float32

F is MulDiv(x)

type DoublePow 

type DoublePow struct {
	Mem map[float32]float32
}

RootX is a logistic activator in the special case of a = 1

func (DoublePow) Df 

func (a DoublePow) Df(x float32) float32

Df is DoubleRoot'(y), where y = DoubleRoot(x)

func (DoublePow) F 

func (a DoublePow) F(x float32, training bool) float32

F is RootX(x)

type DoubleRoot 

type DoubleRoot struct {
	Mem map[float32]float32
}

DoubleRoot is a logistic activator in the special case of a = 1

func (DoubleRoot) Df 

func (a DoubleRoot) Df(x float32) float32

Df is DoubleRoot'(y), where y = DoubleRoot(x)

func (DoubleRoot) F 

func (a DoubleRoot) F(x float32, training bool) float32

F is DoubleRoot(x)

type Dump 

type Dump struct {
	Config  *Config
	Weights [][][]float32
}

Dump is a neural network dump

type Layer 

type Layer struct {
	Neurons []*Neuron
	A       ActivationType
}

Layer is a set of neurons and corresponding activation

func NewLayer 

func NewLayer(n int, activation ActivationType) *Layer

NewLayer creates a new layer with n nodes

func (*Layer) ApplyBias 

func (l *Layer) ApplyBias(weight WeightInitializer) []*Synapse

ApplyBias creates and returns a bias synapse for each neuron in l

func (*Layer) Connect 

func (l *Layer) Connect(next *Layer, weight WeightInitializer)

Connect fully connects layer l to next, and initializes each synapse with the given weight function

func (Layer) String 

func (l Layer) String() string

type Linear 

type Linear struct {
	Mem map[float32]float32
}

Linear is a linear activator

func (Linear) Df 

func (a Linear) Df(x float32) float32

Df is constant

func (Linear) F 

func (a Linear) F(x float32, training bool) float32

F is the identity function

type Loss 

type Loss interface {
	F(estimate, ideal [][]float32) float32
	Df(estimate, ideal, activation float32) float32
}

Loss is satisfied by loss functions

func GetLoss 

func GetLoss(loss LossType) Loss

GetLoss returns a loss function given a LossType

type LossType 

type LossType int

LossType represents a loss function 

const (
	// LossNone signifies unspecified loss
	LossNone LossType = 0
	// LossCrossEntropy is cross entropy loss
	LossCrossEntropy LossType = 1
	// LossBinaryCrossEntropy is the special case of binary cross entropy loss
	LossBinaryCrossEntropy LossType = 2
	// LossMeanSquared is MSE
	LossMeanSquared LossType = 3
)

func (LossType) String 

func (l LossType) String() string

type MeanSquared 

type MeanSquared struct{}

MeanSquared in MSE loss

func (MeanSquared) Df 

func (l MeanSquared) Df(estimate, ideal, activation float32) float32

Df is MSE'(...)

func (MeanSquared) F 

func (l MeanSquared) F(estimate, ideal [][]float32) float32

F is MSE(...)

type Mish 

type Mish struct {
	Mem map[float32]float32
}

func (Mish) Df 

func (a Mish) Df(y float32) float32

Df is Mish'(y), where y = Mish(x)

func (Mish) F 

func (a Mish) F(x float32, training bool) float32

F is Mish(x)

type Mode 

type Mode int

Mode denotes inference mode 

const (
	// ModeDefault is unspecified mode
	ModeDefault Mode = 0
	// ModeMultiClass is for one-hot encoded classification, applies softmax output layer
	ModeMultiClass Mode = 1
	// ModeRegression is regression, applies linear output layer
	ModeRegression Mode = 2
	// ModeBinary is binary classification, applies sigmoid output layer
	ModeBinary Mode = 3
	// ModeMultiLabel is for multilabel classification, applies sigmoid output layer
	ModeMultiLabel Mode = 4
)

type Neural 

type Neural struct {
	// 	Shift        []float32
	// 	Significance []float32
	Layers []*Layer
	Biases [][]*Synapse
	Config *Config
}

Neural is a neural network

func FromDump 

func FromDump(dump *Dump) *Neural

FromDump restores a Neural from a dump

func NewNeural 

func NewNeural(c *Config) *Neural

NewNeural returns a new neural network

func Unmarshal 

func Unmarshal(bytes []byte) (*Neural, error)

Unmarshal restores network from a JSON blob

func (*Neural) ApplyWeights 

func (n *Neural) ApplyWeights(weights [][][]float32)

ApplyWeights sets the weights from a three-dimensional slice

func (Neural) Dump 

func (n Neural) Dump() *Dump

Dump generates a network dump

func (*Neural) Forward 

func (n *Neural) Forward(input []float32, training bool) error

Forward computes a forward pass

func (Neural) Marshal 

func (n Neural) Marshal() ([]byte, error)

Marshal marshals to JSON from network

func (*Neural) NumWeights 

func (n *Neural) NumWeights() (num int)

NumWeights returns the number of weights in the network

func (*Neural) Predict 

func (n *Neural) Predict(input []float32) []float32

Predict computes a forward pass and returns a prediction

func (*Neural) String 

func (n *Neural) String() string

func (Neural) Weights 

func (n Neural) Weights() [][][]float32

Weights returns all weights in sequence

type Neuron 

type Neuron struct {
	A     ActivationType `json:"-"`
	In    []*Synapse
	Out   []*Synapse
	Value float32 `json:"-"`
}

Neuron is a neural network node

func NewNeuron 

func NewNeuron(activation ActivationType) *Neuron

NewNeuron returns a neuron with the given activation

func (*Neuron) Activate 

func (n *Neuron) Activate(x float32, training bool) float32

Activate applies the neurons activation

func (*Neuron) DActivate 

func (n *Neuron) DActivate(x float32) float32

DActivate applies the derivative of the neurons activation

type ReLU 

type ReLU struct {
	Mem map[float32]float32
}

ReLU is a rectified linear unit activator

func (ReLU) Df 

func (a ReLU) Df(y float32) float32

Df is ReLU'(y), where y = ReLU(x)

func (ReLU) F 

func (a ReLU) F(x float32, training bool) float32

F is ReLU(x)

type RootPow 

type RootPow struct {
	Mem map[float32]float32
}

RootX is a logistic activator in the special case of a = 1

func (RootPow) Df 

func (a RootPow) Df(x float32) float32

Df is DoubleRoot'(y), where y = DoubleRoot(x)

func (RootPow) F 

func (a RootPow) F(x float32, training bool) float32

F is RootX(x)

type RootSwish 

type RootSwish struct {
	Mem map[float32]float32
}

func (RootSwish) Df 

func (a RootSwish) Df(y float32) float32

Df is swish'(y), where y = Swish(x)

func (RootSwish) F 

func (a RootSwish) F(x float32, training bool) float32

F is Swish(x)

type RootX 

type RootX struct {
	Mem map[float32]float32
}

RootX is a logistic activator in the special case of a = 1

func (RootX) Df 

func (a RootX) Df(x float32) float32

Df is DoubleRoot'(y), where y = DoubleRoot(x)

func (RootX) F 

func (a RootX) F(x float32, training bool) float32

F is RootX(x)

type Sigmoid 

type Sigmoid struct {
	Mem map[float32]float32
}

Sigmoid is a logistic activator in the special case of a = 1

func (Sigmoid) Df 

func (a Sigmoid) Df(y float32) float32

Df is Sigmoid'(y), where y = Sigmoid(x)

func (Sigmoid) F 

func (a Sigmoid) F(x float32, training bool) float32

F is Sigmoid(x)

type Swish 

type Swish struct {
	Mem map[float32]float32
}

func (Swish) Df 

func (a Swish) Df(y float32) float32

Df is swish'(y), where y = Swish(x)

func (Swish) F 

func (a Swish) F(x float32, training bool) float32

F is Swish(x)

type Synapse 

type Synapse struct {
	Weight  float32
	In, Out float32 `json:"-"`
	IsBias  bool
}

Synapse is an edge between neurons

func NewSynapse 

func NewSynapse(weight float32) *Synapse

NewSynapse returns a synapse with the specified initialized weight

type Tanh 

type Tanh struct {
	Mem map[float32]float32
}

Tanh is a hyperbolic activator

func (Tanh) Df 

func (a Tanh) Df(y float32) float32

Df is Tanh'(y), where y = Tanh(x)

func (Tanh) F 

func (a Tanh) F(x float32, training bool) float32

F is Tanh(x)

type WeightInitializer 

type WeightInitializer func() float32

A WeightInitializer returns a (random) weight

func NewNormal 

func NewNormal(stdDev, mean float32) WeightInitializer

NewNormal returns a normal weight generator

func NewUniform 

func NewUniform(stdDev, mean float32) WeightInitializer

NewUniform returns a uniform weight generator

Source Files

View all Source files

Directories

Path Synopsis
examples
mnist
wines

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.