README
¶
go-deep
Feed forward/backpropagation neural network implementation. Currently supports:
- Activation functions: sigmoid, hyperbolic, ReLU
- Solvers: SGD, SGD with momentum/nesterov, Adam
- Classification modes: regression, multi-class, multi-label, binary
- Supports batch training in parallel
- Bias nodes
Networks are modeled as a set of neurons connected through synapses. No GPU computations - don't use this for any large scale applications.
Todo:
- Dropout
- Batch normalization
Install
go get -u github.com/patrikeh/go-deep
Usage
Import the go-deep package
import (
"fmt"
deep "github.com/patrikeh/go-deep"
"github.com/patrikeh/go-deep/training"
)
Define some data...
var data = training.Examples{
{[]float64{2.7810836, 2.550537003}, []float64{0}},
{[]float64{1.465489372, 2.362125076}, []float64{0}},
{[]float64{3.396561688, 4.400293529}, []float64{0}},
{[]float64{1.38807019, 1.850220317}, []float64{0}},
{[]float64{7.627531214, 2.759262235}, []float64{1}},
{[]float64{5.332441248, 2.088626775}, []float64{1}},
{[]float64{6.922596716, 1.77106367}, []float64{1}},
{[]float64{8.675418651, -0.242068655}, []float64{1}},
}
Create a network with two hidden layers of size 2 and 2 respectively:
n := deep.NewNeural(&deep.Config{
/* Input dimensionality */
Inputs: 2,
/* Two hidden layers consisting of two neurons each, and a single output */
Layout: []int{2, 2, 1},
/* Activation functions: Sigmoid, Tanh, ReLU, Linear */
Activation: deep.ActivationSigmoid,
/* Determines output layer activation & loss function:
ModeRegression: linear outputs with MSE loss
ModeMultiClass: softmax output with Cross Entropy loss
ModeMultiLabel: sigmoid output with Cross Entropy loss
ModeBinary: sigmoid output with binary CE loss */
Mode: deep.ModeBinary,
/* Weight initializers: {deep.NewNormal(μ, σ), deep.NewUniform(μ, σ)} */
Weight: deep.NewNormal(1.0, 0.0),
/* Apply bias */
Bias: true,
})
Train:
// params: learning rate, momentum, alpha decay, nesterov
optimizer := training.NewSGD(0.05, 0.1, 1e-6, true)
// params: optimizer, verbosity (print stats at every 50th iteration)
trainer := training.NewTrainer(optimizer, 50)
training, heldout := data.Split(0.5)
trainer.Train(n, training, heldout, 1000) // training, validation, iterations
resulting in:
Epochs Elapsed Error
--- --- ---
5 12.938µs 0.36438
10 125.691µs 0.02261
15 177.194µs 0.00404
...
1000 10.703839ms 0.00000
Finally, make some predictions:
fmt.Println(data[0].Input, "=>", n.Predict(data[0].Input))
fmt.Println(data[5].Input, "=>", n.Predict(data[5].Input))
Alternatively, batch training can be performed in parallell:
optimizer := NewAdam(0.001, 0.9, 0.999, 1e-8)
// params: optimizer, verbosity (print info at every n:th iteration), batch-size, number of workers
trainer := training.NewBatchTrainer(optimizer, 1, 200, 4)
training, heldout := data.Split(0.75)
trainer.Train(n, training, heldout, 1000) // training, validation, iterations
Examples
See training/trainer_test.go
for a variety of toy examples of regression, multi-class classification, binary classification, etc.
See examples/
for more realistic examples:
Dataset | Topology | Epochs | Accuracy |
---|---|---|---|
wines | [5 5] | 10000 | ~98% |
mnist | [50] | 25 | ~97% |
Documentation
¶
Index ¶
- func ArgMax(xx []float64) int
- func Dot(xx, yy []float64) float64
- func Logistic(x, a float64) float64
- func Max(xx []float64) float64
- func Mean(xx []float64) float64
- func Min(xx []float64) float64
- func Normal(stdDev, mean float64) float64
- func Normalize(xx []float64)
- func Round(x float64) float64
- func Sgn(x float64) float64
- func Softmax(xx []float64) []float64
- func StandardDeviation(xx []float64) float64
- func Standardize(xx []float64)
- func Sum(xx []float64) (sum float64)
- func Uniform(stdDev, mean float64) float64
- func Variance(xx []float64) float64
- type ActivationType
- type BinaryCrossEntropy
- type Config
- type CrossEntropy
- type Differentiable
- type Dump
- type Layer
- type Linear
- type Loss
- type LossType
- type MeanSquared
- type Mode
- type Neural
- func (n *Neural) ApplyWeights(weights [][][]float64)
- func (n Neural) Dump() *Dump
- func (n *Neural) Forward(input []float64) error
- func (n Neural) Marshal() ([]byte, error)
- func (n *Neural) NumWeights() (num int)
- func (n *Neural) Predict(input []float64) []float64
- func (n *Neural) String() string
- func (n Neural) Weights() [][][]float64
- type Neuron
- type ReLU
- type Sigmoid
- type Synapse
- type Tanh
- type WeightInitializer
Constants ¶
Variables ¶
Functions ¶
func Standardize ¶
func Standardize(xx []float64)
Standardize (z-score) shifts distribution to μ=0 σ=1
Types ¶
type ActivationType ¶
type ActivationType int
ActivationType is represents a neuron activation function
const ( // ActivationNone is no activation ActivationNone ActivationType = 0 // ActivationSigmoid is a sigmoid activation ActivationSigmoid ActivationType = 1 // ActivationTanh is hyperbolic activation ActivationTanh ActivationType = 2 // ActivationReLU is rectified linear unit activation ActivationReLU ActivationType = 3 // ActivationLinear is linear activation ActivationLinear ActivationType = 4 // ActivationSoftmax is a softmax activation (per layer) ActivationSoftmax ActivationType = 5 )
func OutputActivation ¶
func OutputActivation(c Mode) ActivationType
OutputActivation returns activation corresponding to prediction mode
type BinaryCrossEntropy ¶
type BinaryCrossEntropy struct{}
BinaryCrossEntropy is binary CE loss
func (BinaryCrossEntropy) Df ¶
func (l BinaryCrossEntropy) Df(estimate, ideal, activation float64) float64
Df is CE'(...)
func (BinaryCrossEntropy) F ¶
func (l BinaryCrossEntropy) F(estimate, ideal [][]float64) float64
F is CE(...)
type Config ¶
type Config struct { // Number of inputs Inputs int // Defines topology: // For instance, [5 3 3] signifies a network with two hidden layers // containing 5 and 3 nodes respectively, followed an output layer // containing 3 nodes. Layout []int // Activation functions: {ActivationTanh, ActivationReLU, ActivationSigmoid} Activation ActivationType // Solver modes: {ModeRegression, ModeBinary, ModeMultiClass, ModeMultiLabel} Mode Mode // Initializer for weights: {NewNormal(σ, μ), NewUniform(σ, μ)} Weight WeightInitializer `json:"-"` // Loss functions: {LossCrossEntropy, LossBinaryCrossEntropy, LossMeanSquared} Loss LossType // Apply bias nodes Bias bool }
Config defines the network topology, activations, losses etc
type CrossEntropy ¶
type CrossEntropy struct{}
CrossEntropy is CE loss
func (CrossEntropy) Df ¶
func (l CrossEntropy) Df(estimate, ideal, activation float64) float64
Df is CE'(...)
type Differentiable ¶
Differentiable is an activation function and its first order derivative, where the latter is expressed as a function of the former for efficiency
func GetActivation ¶
func GetActivation(act ActivationType) Differentiable
GetActivation returns the concrete activation given an ActivationType
type Layer ¶
type Layer struct { Neurons []*Neuron A ActivationType }
Layer is a set of neurons and corresponding activation
func NewLayer ¶
func NewLayer(n int, activation ActivationType) *Layer
NewLayer creates a new layer with n nodes
func (*Layer) ApplyBias ¶
func (l *Layer) ApplyBias(weight WeightInitializer) []*Synapse
ApplyBias creates and returns a bias synapse for each neuron in l
func (*Layer) Connect ¶
func (l *Layer) Connect(next *Layer, weight WeightInitializer)
Connect fully connects layer l to next, and initializes each synapse with the given weight function
type Loss ¶
type Loss interface { F(estimate, ideal [][]float64) float64 Df(estimate, ideal, activation float64) float64 }
Loss is satisfied by loss functions
type LossType ¶
type LossType int
LossType represents a loss function
const ( // LossNone signifies unspecified loss LossNone LossType = 0 // LossCrossEntropy is cross entropy loss LossCrossEntropy LossType = 1 // LossBinaryCrossEntropy is the special case of binary cross entropy loss LossBinaryCrossEntropy LossType = 2 // LossMeanSquared is MSE LossMeanSquared LossType = 3 )
type MeanSquared ¶
type MeanSquared struct{}
MeanSquared in MSE loss
func (MeanSquared) Df ¶
func (l MeanSquared) Df(estimate, ideal, activation float64) float64
Df is MSE'(...)
type Mode ¶
type Mode int
Mode denotes inference mode
const ( // ModeDefault is unspecified mode ModeDefault Mode = 0 // ModeMultiClass is for one-hot encoded classification, applies softmax output layer ModeMultiClass Mode = 1 // ModeRegression is regression, applies linear output layer ModeRegression Mode = 2 // ModeBinary is binary classification, applies sigmoid output layer ModeBinary Mode = 3 // ModeMultiLabel is for multilabel classification, applies sigmoid output layer ModeMultiLabel Mode = 4 )
type Neural ¶
Neural is a neural network
func (*Neural) ApplyWeights ¶
ApplyWeights sets the weights from a three-dimensional slice
func (*Neural) NumWeights ¶
NumWeights returns the number of weights in the network
type Neuron ¶
type Neuron struct { A ActivationType `json:"-"` In []*Synapse Out []*Synapse Value float64 `json:"-"` }
Neuron is a neural network node
func NewNeuron ¶
func NewNeuron(activation ActivationType) *Neuron
NewNeuron returns a neuron with the given activation
type Synapse ¶
Synapse is an edge between neurons
func NewSynapse ¶
NewSynapse returns a synapse with the specified initialized weight
type WeightInitializer ¶
type WeightInitializer func() float64
A WeightInitializer returns a (random) weight
func NewNormal ¶
func NewNormal(stdDev, mean float64) WeightInitializer
NewNormal returns a normal weight generator
func NewUniform ¶
func NewUniform(stdDev, mean float64) WeightInitializer
NewUniform returns a uniform weight generator