nn

package
v0.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 24, 2021 License: BSD-2-Clause Imports: 15 Imported by: 0

README

Neural Network

The inclusion of neural models in the nn sub-package is mostly arbitrary.

Not all neural models are useful. For instance, I wanted to implement many recurrent networks for the sake of curiosity, but in the end, the LSTM and GRU almost always gave us the best performance in natural language processing tasks.

We might decide - based on your suggestions - to delete some of them to lighten the core package.

Documentation

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

func Affine 

func Affine(g *ag.Graph, xs ...ag.Node) ag.Node

Affine performs an affine transformation over an arbitrary (odd) number of nodes held in the input. The first node is the “bias”, which is added to the output as-is. The remaining nodes of the form "Wx" are multiplied together in pairs, then added. The pairs except the first whose "x" is nil are not considered. y = b + W1x1 + W2x2 + ... + WnXn

func BiAffine 

func BiAffine(g *ag.Graph, w, u, v, b, x1, x2 ag.Node) ag.Node

BiAffine performs a biaffine transformation.

func BiLinear 

func BiLinear(g *ag.Graph, w, x1, x2 ag.Node) ag.Node

BiLinear performs a bilinear transformation of the type (x_1 W x_2)

func ClearSupport 

func ClearSupport(m Model)

ClearSupport clears the support structure of all model's parameters (including sub-params).

func Conv2D 

func Conv2D(g *ag.Graph, w, x ag.Node, xStride, yStride int) ag.Node

Conv2D performs a 2D convolution.

func DumpParamsVector 

func DumpParamsVector(model Model) mat.Matrix

DumpParamsVector dumps all params of a Model into a single Dense vector.

func ForEachParam 

func ForEachParam(m Model, callback func(param Param))

ForEachParam iterate all the parameters of a model also exploring the sub-parameters recursively.

func ForEachParamStrict 

func ForEachParamStrict(m Model, callback func(param Param))

ForEachParamStrict iterate all the parameters of a model without exploring the sub-models.

func LoadParamsVector 

func LoadParamsVector(model Model, vector mat.Matrix)

LoadParamsVector sets all params of a Model from a previously dumped Dense vector.

func MarshalBinaryParam added in v0.4.0 

func MarshalBinaryParam(p Param, w io.Writer) error

MarshalBinaryParam encodes a Param into binary form.

func Separate 

func Separate(g *ag.Graph, x ag.Node) [][]ag.Node

Separate returns a matrix of Node(s) represented as a slice of slice containing the elements extracted from the input. The dimensions of the resulting matrix are the same of the input.

func SeparateVec 

func SeparateVec(g *ag.Graph, x ag.Node) []ag.Node

SeparateVec returns a slice of Node(s) containing the elements extracted from the input. The size of the vector equals the number of input elements. You can think of this method as the inverse of the ag.Concat operator.

func SplitVec 

func SplitVec(g *ag.Graph, x ag.Node, chunks int) []ag.Node

SplitVec splits the x Node into multiple chunks. It panics if the x Node is not a vector. TODO: optimize, this is extremely inefficient!

func ToNode added in v0.2.0 

func ToNode(i interface{}) ag.Node

ToNode coerces the given value to an ag.Node.

If the underlying value is already of type ag.Node, it is simply returned unmodified. If the value is a slice of nodes []ag.Node which contains exactly one item, the method returns just that node. If the value is a slice of zero, two or more nodes, the method panics. In case of any other type, the method panics as well.

Example 
package main

import (
	"fmt"
	"github.com/nlpodyssey/spago/pkg/ml/ag"
	"github.com/nlpodyssey/spago/pkg/ml/nn"
)

func main() {
	g := ag.NewGraph()

	a := g.NewScalar(1)
	b := []ag.Node{g.NewScalar(2)}

	fmt.Println(nn.ToNode(a).Value())
	fmt.Println(nn.ToNode(b).Value())

}

Output:

[1]
[2]

func ToNodes added in v0.2.0 

func ToNodes(i interface{}) []ag.Node

ToNodes coerces the given value to a slice of nodes []ag.Node.

If the value is already of type []ag.Node, it is simply returned unmodified. If the value is a single ag.Node, the method returns a new slice of nodes, containing just that node. In case of any other type, the method panics.

Example 
package main

import (
	"fmt"
	"github.com/nlpodyssey/spago/pkg/ml/ag"
	"github.com/nlpodyssey/spago/pkg/ml/nn"
)

func main() {
	g := ag.NewGraph()

	a := g.NewScalar(1)
	b := []ag.Node{g.NewScalar(2), g.NewScalar(3)}

	fmt.Printf("len: %d, first value: %v\n", len(nn.ToNodes(a)), nn.ToNodes(a)[0].Value())
	fmt.Printf("len: %d, first value: %v\n", len(nn.ToNodes(b)), nn.ToNodes(b)[0].Value())

}

Output:

len: 1, first value: [1]
len: 2, first value: [2]

func UnmarshalBinaryParamWithReceiver added in v0.4.0 

func UnmarshalBinaryParamWithReceiver(r io.Reader, dest Param) error

UnmarshalBinaryParamWithReceiver decodes a Param from binary form into the receiver.

func ZeroGrad 

func ZeroGrad(m Model)

ZeroGrad set the gradients of all model's parameters (including sub-params) to zeros.

Types

type BaseModel added in v0.2.0 

type BaseModel struct {
	// G is the computational graph on which the (reified) model operates.
	G *ag.Graph
	// ProcessingMode is the processing mode for the model (training or inference).
	ProcessingMode ProcessingMode
}

BaseModel satisfies some methods of the Model interface. Don't use it directly; it is meant to be embedded in other processors to reduce the amount of boilerplate code.

func (*BaseModel) Close added in v0.6.0 

func (m *BaseModel) Close()

Close can be used to close or finalize model structures.

func (*BaseModel) Graph added in v0.2.0 

func (m *BaseModel) Graph() *ag.Graph

Graph returns the computational graph on which the (reified) model operates. It panics if the Graph is nil.

func (*BaseModel) InitProcessor added in v0.2.0 

func (m *BaseModel) InitProcessor()

InitProcessor is used to initialize structures and data useful for the Forward(). nn.Reify() automatically invokes InitProcessor() for any sub-models.

func (*BaseModel) IsProcessor added in v0.2.0 

func (m *BaseModel) IsProcessor() bool

IsProcessor returns whether the model has been reified (i.e., contextualized to operate on a graph) and can perform the Forward().

func (*BaseModel) Mode added in v0.2.0 

func (m *BaseModel) Mode() ProcessingMode

Mode returns whether the (reified) model is being used for training or inference.

type DefaultParamsIterator 

type DefaultParamsIterator struct {
	// contains filtered or unexported fields
}

DefaultParamsIterator is spaGO default implementation of a ParamsGetter.

func NewDefaultParamsIterator 

func NewDefaultParamsIterator(models ...Model) *DefaultParamsIterator

NewDefaultParamsIterator returns a new DefaultParamsIterator.

func (*DefaultParamsIterator) Params added in v0.2.0 

func (i *DefaultParamsIterator) Params() []Param

Params returns a slice with all Param elements from all models held by the DefaultParamsIterator.

type Model 

type Model interface {
	// Graph returns the computational graph on which the model operates (can be nil).
	Graph() *ag.Graph
	// Mode returns whether the model is being used for training or inference.
	Mode() ProcessingMode
	// IsProcessor returns whether the model has been reified (i.e., contextualized to operate
	// on a graph) and can perform the Forward().
	IsProcessor() bool
	// InitProcessor is used to initialize structures and data useful for the Forward().
	// nn.Reify() automatically invokes InitProcessor() for any sub-models.
	InitProcessor()
	// Close can be used to close or finalize model structures.
	// For example, embeddings.Model needs to close the underlying key-value store.
	Close()
}

Model is implemented by all neural network architectures. You can assign parameters (i.e. nn.Param) as regular attributes (if any). A Model can also contain other Model(s), allowing to nest them in a tree structure. Through "reification" (i.e. nn.Reify()), a Model operates as a "processor" using the computational graph. The Forward() operation can only be performed on a reified model (a.k.a. processor).

func MakeNewModels 

func MakeNewModels(n int, callback func(i int) Model) []Model

MakeNewModels return n new models. The callback is delegated to return a new model for each i-item.

func Reify added in v0.2.0 

func Reify(m Model, g *ag.Graph, mode ProcessingMode) Model

Reify returns a new "reified" model (a.k.a. processor) to execute the forward step.

func ReifyForInference added in v0.6.0 

func ReifyForInference(m Model, g *ag.Graph) Model

ReifyForInference returns a new reified model (a.k.a. processor) with the given Graph, setting the mode to Inference.

func ReifyForTraining added in v0.6.0 

func ReifyForTraining(m Model, g *ag.Graph) Model

ReifyForTraining returns a new reified model (a.k.a. processor) with the given Graph, setting the mode to Training.

type Param 

type Param interface {
	ag.Node // it implies fn.Operand and ag.GradValue too

	// Name returns the params name (can be empty string).
	Name() string
	// SetName set the params name (can be empty string).
	SetName(name string)
	// Type returns the params type (weights, biases, undefined).
	Type() ParamsType
	// SetType set the params type (weights, biases, undefined).
	SetType(pType ParamsType)
	// SetRequiresGrad set whether the param requires gradient, or not.
	SetRequiresGrad(value bool)
	// ReplaceValue replaces the value of the parameter and clears the support structure.
	ReplaceValue(value mat.Matrix)
	// ApplyDelta updates the value of the underlying storage applying the delta.
	ApplyDelta(delta mat.Matrix)
	// Payload returns the optimizer support structure (can be nil).
	Payload() *Payload
	// SetPayload is a thread safe operation to set the given Payload on the
	// receiver Param.
	SetPayload(payload *Payload)
	// ClearPayload clears the support structure.
	ClearPayload()
}

Param is the interface for a Model parameter.

func NewParam 

func NewParam(value mat.Matrix, opts ...ParamOption) Param

NewParam returns a new param.

func UnmarshalBinaryParam added in v0.4.0 

func UnmarshalBinaryParam(r io.Reader) (Param, error)

UnmarshalBinaryParam decodes a Param from binary form. TODO: add a "withBacking" optional argument to remove the need of UnmarshalBinaryParamWithReceiver().

type ParamOption 

type ParamOption func(*param)

ParamOption allows to configure a new Param with your specific needs.

func RequiresGrad 

func RequiresGrad(value bool) ParamOption

RequiresGrad is an option to specify whether a Param should be trained or not.

func SetStorage 

func SetStorage(storage *kvdb.KeyValueDB) ParamOption

SetStorage is an option to specify a kvdb.KeyValueDB storage. This is useful, for example, for a memory-efficient embeddings Param implementation.

type Params added in v0.2.0 

type Params []Param

Params extends a slice of Param with Nodes() method.

func (Params) Nodes added in v0.2.0 

func (ps Params) Nodes() []ag.Node

Nodes converts the slice of Param into a slice of ag.Node.

type ParamsGetter added in v0.2.0 

type ParamsGetter interface {
	Params() []Param
}

ParamsGetter is implemented by any value that has the ParamsList method, which should return the list of parameters of one or more models.

type ParamsType 

type ParamsType uint8

ParamsType is the enumeration-like type used for the set of parameter (Param) types of a neural network Model. 

const (
	// Weights identifies a Param containing weights.
	Weights ParamsType = iota
	// Biases identifies a Param containing biases.
	Biases
	// Undefined identifies a generic Param, which cannot be described
	// with other ParamsType values.
	Undefined
)

func (ParamsType) String 

func (t ParamsType) String() string

type Payload 

type Payload struct {
	Label int
	Data  []mat.Matrix
}

Payload contains the support data used for example by the optimization methods

func NewPayload added in v0.4.0 

func NewPayload() *Payload

NewPayload returns an empty support structure, not connected to any optimization method.

func (Payload) MarshalBinary added in v0.4.0 

func (p Payload) MarshalBinary() ([]byte, error)

MarshalBinary encodes the Payload into binary form.

func (*Payload) UnmarshalBinary added in v0.4.0 

func (p *Payload) UnmarshalBinary(data []byte) error

UnmarshalBinary decodes a Payload from binary form.

type ProcessingMode 

type ProcessingMode uint8

ProcessingMode regulates the different usage of some operations (e.g. Dropout, BatchNorm, etc.), depending on whether you're doing training or inference. Failing to set the right mode will yield inconsistent inference results. 

const (
	// Training is to be used during the training phase of a model. For example, dropouts are enabled.
	Training ProcessingMode = iota
	// Inference keeps weights fixed while using the model and disables some operations (e.g. skip dropout).
	Inference
)

type StandardForwarder added in v0.2.0 

type StandardForwarder interface {
	// Forward executes the forward step for each input and returns the result.
	// Recurrent networks, treats the input nodes as a sequence. Differently, feed-forward
	// networks are stateless so every computation is independent and possibly concurrent.
	Forward(xs ...ag.Node) []ag.Node
}

StandardForwarder consists of a Forward variadic function that accepts ag.Node and returns a slice of ag.Node. It is called StandardForwarder since this is the most frequent forward method among all implemented neural models.

type StandardModel added in v0.2.0 

type StandardModel interface {
	Model
	StandardForwarder
}

StandardModel consists of a model that implements StandardForwarder.

Source Files

View all Source files

Directories

Path Synopsis
lshattention
Package lshattention provides an implementation of the LSH-Attention model, as describe in `Reformer: The Efficient Transformer` by N. Kitaev, Ł. Kaiser, A. Levskaya (https://arxiv.org/pdf/2001.04451.pdf).
 Package lshattention provides an implementation of the LSH-Attention model, as describe in `Reformer: The Efficient Transformer` by N. Kitaev, Ł. Kaiser, A. Levskaya (https://arxiv.org/pdf/2001.04451.pdf).
multiheadattention
selfattention
syntheticattention
Package syntheticattention provides an implementation of the Synthetic Attention described in: "SYNTHESIZER: Rethinking Self-Attention in Transformer Models" by Tay et al., 2020.
 Package syntheticattention provides an implementation of the Synthetic Attention described in: "SYNTHESIZER: Rethinking Self-Attention in Transformer Models" by Tay et al., 2020.
Package birnncrf provides an implementation of a Bidirectional Recurrent Neural Network (BiRNN) with a Conditional Random Fields (CRF) on tom.
 Package birnncrf provides an implementation of a Bidirectional Recurrent Neural Network (BiRNN) with a Conditional Random Fields (CRF) on tom.
Package bls provides an implementation of the Broad Learning System (BLS) described in "Broad Learning System: An Effective and Efficient Incremental Learning System Without the Need for Deep Architecture" by C. L. Philip Chen and Zhulin Liu, 2017.
 Package bls provides an implementation of the Broad Learning System (BLS) described in "Broad Learning System: An Effective and Efficient Incremental Learning System Without the Need for Deep Architecture" by C. L. Philip Chen and Zhulin Liu, 2017.
Package conv1x1 implements a 1-dimensional 1-kernel convolution model
 Package conv1x1 implements a 1-dimensional 1-kernel convolution model
Package gmlp implements a model composed by basic MLP layers with gating mechanism.
 Package gmlp implements a model composed by basic MLP layers with gating mechanism.
gnn
slstm
Package slstm implements a Sentence-State LSTM graph neural network.
 Package slstm implements a Sentence-State LSTM graph neural network.
startransformer
Package startransformer provides a variant implementation of the Star-Transformer model introduced by Qipeng Guo, Xipeng Qiu et al.
 Package startransformer provides a variant implementation of the Star-Transformer model introduced by Qipeng Guo, Xipeng Qiu et al.
normalization
adanorm
Package adanorm implements the Adaptive Normalization (AdaNorm) method.
 Package adanorm implements the Adaptive Normalization (AdaNorm) method.
batchnorm
fixnorm
Package fixnorm implements the fixnorm normalization method.
 Package fixnorm implements the fixnorm normalization method.
layernorm
Package layernorm implements the Layer Normalization (LayerNorm) i method.
 Package layernorm implements the Layer Normalization (LayerNorm) i method.
layernormsimple
Package layernormsimple implements a simple version of LayerNorm (LayerNorm-simple).
 Package layernormsimple implements a simple version of LayerNorm (LayerNorm-simple).
rmsnorm
Package rmsnorm implements the Root Mean Square Layer Normalization method.
 Package rmsnorm implements the Root Mean Square Layer Normalization method.
scalenorm
Package rae provides an implementation of the recursive auto-encoder strategy described in "Towards Lossless Encoding of Sentences" by Prato et al., 2019.
 Package rae provides an implementation of the recursive auto-encoder strategy described in "Towards Lossless Encoding of Sentences" by Prato et al., 2019.
Package rc contains built-in Residual Connections (RC).
 Package rc contains built-in Residual Connections (RC).
recurrent
cfn
deltarnn
fsmn
gru
horn
Package horn provides an implementation of Higher Order Recurrent Neural Networks (HORN).
 Package horn provides an implementation of Higher Order Recurrent Neural Networks (HORN).
indrnn
lstm
lstmsc
Package lstmsc provides an implementation of LSTM enriched with a PolicyGradient to enable Dynamic Skip Connections.
 Package lstmsc provides an implementation of LSTM enriched with a PolicyGradient to enable Dynamic Skip Connections.
ltm
mist
Package mist provides an implementation of the MIST (MIxed hiSTory) recurrent network as described in "Analyzing and Exploiting NARX Recurrent Neural Networks for Long-Term Dependencies" by Di Pietro et al., 2018 (https://arxiv.org/pdf/1702.07805.pdf).
 Package mist provides an implementation of the MIST (MIxed hiSTory) recurrent network as described in "Analyzing and Exploiting NARX Recurrent Neural Networks for Long-Term Dependencies" by Di Pietro et al., 2018 (https://arxiv.org/pdf/1702.07805.pdf).
nru
Package nru provides an implementation of the NRU (Non-Saturating Recurrent Units) recurrent network as described in "Towards Non-Saturating Recurrent Units for Modelling Long-Term Dependencies" by Chandar et al., 2019.
 Package nru provides an implementation of the NRU (Non-Saturating Recurrent Units) recurrent network as described in "Towards Non-Saturating Recurrent Units for Modelling Long-Term Dependencies" by Chandar et al., 2019.
ran
rla
Package rla provides an implementation of RLA (Recurrent Linear Attention).
 Package rla provides an implementation of RLA (Recurrent Linear Attention).
srn
srnn
Package srnn implements the SRNN (Shuffling Recurrent Neural Networks) by Rotman and Wolf, 2020.
 Package srnn implements the SRNN (Shuffling Recurrent Neural Networks) by Rotman and Wolf, 2020.
tpr
Package sgu implements the Spatial Gating Unit (SGU).
 Package sgu implements the Spatial Gating Unit (SGU).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.